Repository: django-extensions/django-extensions

Issue:
sqldiff: MySQL backend fails to introspect unsigned fields 
If I create a MySQL database using `manage.py syncdb` and then immediately run `sqldiff`, differences are detectd for all fields of type `models.PositiveIntegerField()`. The error looks like:

```
...
|--+ field 'rank' parameters differ: db='integer', model='integer UNSIGNED'
...
```

When looking at the actual database I see:

```
mysql> SHOW CREATE TABLE ...
...
`rank` int(10) unsigned NOT NULL,
...
```

So, the table and model DO match, but `sqldiff` can't detect that the database field is in fact `unsigned`. (As I only develop against MySQL, I am not sure if this is an issue with all backends or just MySQL.)

Upon investigation, it looks like this fails because the Django `FieldInfo` tuple identifies the type with `type_code`. This code does not contain information about `unsigned`. Just simply identifies the field as an integer. Should this be an upstream bug? Should the `FieldInfo` contain information about unsigned? I am not familiar enough to know this answer, but if it should be developed upstream I can do so there.

To get this information directly from MySQL, one can query the `information_schema.columns` and parse the `column_type` field which will look like `int(10) unsigned`. I can do this in  `MySQLDiff.get_field_db_type()` if you think this is the way to go.

Let me know your thoughts and I'll prepare a proper patch.



Discussion:
I think your approach is correct. Let's get the information directly from MySQL, specially if its possible in some nice generic way (cross version / cross storage engine), we do this for PostGreSQL as well for a bunch of things. (nullable columns, constraints, etc)

To see if other backend have a similar problem I created a very small test app with a PostitiveIntegerField ran syncdb and directly after sqldiff.

On Sqlite3 and PostGreSQL (easiest to test for me) this works as expected. You found that MySQL does not work and Oracle is very hard to test. (and not very widely used I guess)

Django creates unsigned integers some backends and integers in others:

```
MYSQL:        'PositiveIntegerField': 'integer UNSIGNED',
Sqlite3:        'PositiveIntegerField': 'integer unsigned',
Postgresql:  'PositiveIntegerField': 'integer',
Oracle:         'PositiveIntegerField': 'NUMBER(11)',
```

Semi related: http://stackoverflow.com/questions/20810134/why-unsigned-integer-is-not-available-in-postgresql


Resolving patch:
diff --git a/django_extensions/management/commands/sqldiff.py b/django_extensions/management/commands/sqldiff.py
--- a/django_extensions/management/commands/sqldiff.py
+++ b/django_extensions/management/commands/sqldiff.py
@@ -117,6 +117,8 @@ class SQLDiff(object):
     SQL_TABLE_MISSING_IN_DB = lambda self, style, qn, args: style.NOTICE('-- Table missing: %s' % args[0])
 
     can_detect_notnull_differ = False
+    can_detect_unsigned_differ = False
+    unsigned_suffix = None
 
     def __init__(self, app_models, options):
         self.has_differences = None
@@ -137,6 +139,7 @@ def __init__(self, app_models, options):
         self.unknown_db_fields = {}
         self.new_db_fields = set()
         self.null = {}
+        self.unsigned = {}
 
         self.DIFF_SQL = {
             'error': self.SQL_ERROR,
@@ -158,9 +161,15 @@ def __init__(self, app_models, options):
         if self.can_detect_notnull_differ:
             self.load_null()
 
+        if self.can_detect_unsigned_differ:
+            self.load_unsigned()
+
     def load_null(self):
         raise NotImplementedError("load_null functions must be implemented if diff backend has 'can_detect_notnull_differ' set to True")
 
+    def load_unsigned(self):
+        raise NotImplementedError("load_unsigned function must be implemented if diff backend has 'can_detect_unsigned_differ' set to True")
+
     def add_app_model_marker(self, app_label, model_name):
         self.differences.append((app_label, model_name, []))
 
@@ -255,6 +264,13 @@ def get_field_db_type(self, description, field=None, table_name=None):
             field_db_type = getattr(module, package_name)(**kwargs).db_type(connection=connection)
         else:
             field_db_type = getattr(models, reverse_type)(**kwargs).db_type(connection=connection)
+
+        tablespace = field.db_tablespace
+        if not tablespace:
+            tablespace = "public"
+        if self.unsigned.get((tablespace, table_name, field.column), False):
+            field_db_type = '%s %s' % (field_db_type, self.unsigned_suffix)
+
         return field_db_type
 
     def get_field_db_type_lookup(self, type_code):
@@ -359,7 +375,7 @@ def find_field_type_differ(self, meta, table_description, table_name, func=None)
             description = db_fields[field.name]
 
             model_type = self.get_field_model_type(field)
-            db_type = self.get_field_db_type(description, field)
+            db_type = self.get_field_db_type(description, field, table_name)
 
             # use callback function if defined
             if func:
@@ -478,6 +494,10 @@ def print_diff_text(self, style):
             print(style.NOTICE("# Detecting notnull changes not implemented for this database backend"))
             print("")
 
+        if not self.can_detect_unsigned_differ:
+            print(style.NOTICE("# Detecting unsigned changes not implemented for this database backend"))
+            print("")
+
         cur_app_label = None
         for app_label, model_name, diffs in self.differences:
             if not diffs:
@@ -531,17 +551,34 @@ class GenericSQLDiff(SQLDiff):
 
 class MySQLDiff(SQLDiff):
     can_detect_notnull_differ = False
+    can_detect_unsigned_differ = True
+    unsigned_suffix = 'UNSIGNED'
+
+    def load_unsigned(self):
+        tablespace = 'public'
+        for table_name in self.db_tables:
+            result = self.sql_to_dict("""
+                SELECT column_name
+                FROM information_schema.columns
+                WHERE table_schema = DATABASE()
+                    AND table_name = %s
+                    AND column_type LIKE '%%unsigned'""", [table_name])
+            for table_info in result:
+                key = (tablespace, table_name, table_info['column_name'])
+                self.unsigned[key] = True
 
     # All the MySQL hacks together create something of a problem
     # Fixing one bug in MySQL creates another issue. So just keep in mind
     # that this is way unreliable for MySQL atm.
     def get_field_db_type(self, description, field=None, table_name=None):
         from MySQLdb.constants import FIELD_TYPE
-        db_type = super(MySQLDiff, self).get_field_db_type(description)
+        db_type = super(MySQLDiff, self).get_field_db_type(description, field, table_name)
         if not db_type:
             return
         if field:
-            if field.primary_key and (db_type == 'integer' or db_type == 'bigint'):
+            parts = db_type.split(' ', 1)
+            db_data_type = parts.pop(0)
+            if field.primary_key and (db_data_type == 'integer' or db_type == 'bigint'):
                 db_type += ' AUTO_INCREMENT'
             # MySQL isn't really sure about char's and varchar's like sqlite
             field_type = self.get_field_model_type(field)
@@ -553,7 +590,8 @@ def get_field_db_type(self, description, field=None, table_name=None):
             if db_type == 'integer' and description[1] == FIELD_TYPE.TINY and description[4] == 1:
                 db_type = 'bool'
             if db_type == 'integer' and description[1] == FIELD_TYPE.SHORT:
-                db_type = 'smallint UNSIGNED'  # FIXME: what about if it's not UNSIGNED ?
+                parts.insert(0, 'smallint')
+                db_type = ' '.join(parts)
         return db_type
 
 
@@ -592,7 +630,7 @@ def find_index_missing_in_model(self, meta, table_indexes, table_constraints, ta
         pass
 
     def get_field_db_type(self, description, field=None, table_name=None):
-        db_type = super(SqliteSQLDiff, self).get_field_db_type(description)
+        db_type = super(SqliteSQLDiff, self).get_field_db_type(description, field, table_name)
         if not db_type:
             return
         if field:
@@ -605,6 +643,7 @@ def get_field_db_type(self, description, field=None, table_name=None):
 
 class PostgresqlSQLDiff(SQLDiff):
     can_detect_notnull_differ = True
+    can_detect_unsigned_differ = True
 
     DATA_TYPES_REVERSE_OVERRIDE = {
         1042: 'CharField',
@@ -650,6 +689,11 @@ def load_null(self):
             key = (dct['nspname'], dct['relname'], dct['attname'])
             self.null[key] = not dct['attnotnull']
 
+    def load_unsigned(self):
+        # PostgreSQL does not support unsigned, so no columns are
+        # unsigned. Nothing to do.
+        pass
+
     def load_constraints(self):
         for dct in self.sql_to_dict(self.SQL_LOAD_CONSTRAINTS, []):
             key = (dct['nspname'], dct['relname'], dct['attname'])
@@ -744,10 +788,7 @@ def get_constraints(self, cursor, table_name, introspection):
         return constraints
 
     def get_field_db_type(self, description, field=None, table_name=None):
-        if getattr(field, 'geography', False):
-            db_type = super(PostgresqlSQLDiff, self).get_field_db_type(description, field=field)
-        else:
-            db_type = super(PostgresqlSQLDiff, self).get_field_db_type(description)
+        db_type = super(PostgresqlSQLDiff, self).get_field_db_type(description, field, table_name)
         if not db_type:
             return
         if field:


Test patch: