Repository: googleapis/python-spanner-django

Issue:
django: fix 'ALTER TABLE ... ADD CONSTRAINT' syntax
When starting up a Django app, and running for the first time
```shell
python3 manage.py migrate
```

I get back this long stacktrace below

<details>

```shell
$ GOOGLE_APPLICATION_CREDENTIALS=~/Downloads/appdev-soda-spanner-staging-b842e628aa41-orijtech.json python3 manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial...Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Errors parsing Spanner DDL statement: ALTER TABLE django_content_type ADD CONSTRAINT django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model) : Syntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'\nSyntax error on line 1, column 48: Encountered \'django_content_type_app_label_model_76bd3d3b_uniq\' while parsing: column_type\nSyntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'"
	debug_error_string = "{"created":"@1573584883.490410000","description":"Error received from peer ipv4:172.217.4.138:443","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"Errors parsing Spanner DDL statement: ALTER TABLE django_content_type ADD CONSTRAINT django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model) : Syntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'\nSyntax error on line 1, column 48: Encountered \'django_content_type_app_label_model_76bd3d3b_uniq\' while parsing: column_type\nSyntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 76, in execute
    self.__do_update_ddl(sql)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 191, in __do_update_ddl
    return self.__db_handle.update_ddl(ddl_statements)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/spanner/dbapi/connection.py", line 65, in update_ddl
    return self.__dbhandle.update_ddl(ddl_statements)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/cloud/spanner_v1/database.py", line 277, in update_ddl
    self.name, ddl_statements, operation_id=operation_id, metadata=metadata
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/cloud/spanner_admin_database_v1/gapic/database_admin_client.py", line 606, in update_database_ddl
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/retry.py", line 277, in retry_wrapped_func
    on_error=on_error,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Errors parsing Spanner DDL statement: ALTER TABLE django_content_type ADD CONSTRAINT django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model) : Syntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'\nSyntax error on line 1, column 48: Encountered \'django_content_type_app_label_model_76bd3d3b_uniq\' while parsing: column_type\nSyntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 88, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
spanner.dbapi.exceptions.ProgrammingError: 400 Errors parsing Spanner DDL statement: ALTER TABLE django_content_type ADD CONSTRAINT django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model) : Syntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'\nSyntax error on line 1, column 48: Encountered \'django_content_type_app_label_model_76bd3d3b_uniq\' while parsing: column_type\nSyntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 234, in handle
    fake_initial=fake_initial,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/migrations/operations/models.py", line 530, in database_forwards
    getattr(new_model._meta, self.option_name, set()),
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 367, in alter_unique_together
    self.execute(self._create_unique_sql(model, columns))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 137, in execute
    cursor.execute(sql, params)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 99, in execute
    return super().execute(sql, params)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/spanner/dbapi/cursor.py", line 88, in execute
    raise ProgrammingError(e.details if hasattr(e, 'details') else e)
django.db.utils.ProgrammingError: 400 Errors parsing Spanner DDL statement: ALTER TABLE django_content_type ADD CONSTRAINT django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model) : Syntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'\nSyntax error on line 1, column 48: Encountered \'django_content_type_app_label_model_76bd3d3b_uniq\' while parsing: column_type\nSyntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'
```
</details>

but the problem is with
```shell
django.db.utils.ProgrammingError: 400 Errors parsing Spanner DDL statement: ALTER TABLE django_content_type ADD CONSTRAINT django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model) : Syntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'\nSyntax error on line 1, column 48: Encountered \'django_content_type_app_label_model_76bd3d3b_uniq\' while parsing: column_type\nSyntax error on line 1, column 48: Expecting \'EOF\' but found \'django_content_type_app_label_model_76bd3d3b_uniq\'
```

of which the statement that's failing is
```sql
ALTER TABLE django_content_type ADD CONSTRAINT 
django_content_type_app_label_model_76bd3d3b_uniq UNIQUE (app_label, model) 
```


Discussion:
It looks like the [proper SQL](https://cloud.google.com/spanner/docs/secondary-indexes) should be something like:
`CREATE UNIQUE INDEX django_content_type_app_label_model_76bd3d3b_uniq ON django_content_type (app_label, model);`
This can be customized on the `SchemaEditor`.
Oh wow, thanks for the catch! I definitely didn’t think it could be done.
Could you please send a PR for it? Thank you.

On Tue, Nov 12, 2019 at 2:59 PM Tim Graham <notifications@github.com> wrote:

> It looks like the proper SQL
> <https://cloud.google.com/spanner/docs/secondary-indexes> should be
> something like:
> CREATE UNIQUE INDEX django_content_type_app_label_model_76bd3d3b_uniq ON
> django_content_type (app_label, model);
> This can be customized on the SchemaEditor.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/orijtech/spanner-orm/issues/37?email_source=notifications&email_token=ABFL3V2L5TNH2J6RQ7BQKPDQTMYLNA5CNFSM4JMIOBA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED4H6AQ#issuecomment-553156354>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABFL3V73MSDO6FE5QYLM2J3QTMYLNANCNFSM4JMIOBAQ>
> .
>


Resolving patch:
diff --git a/spanner/dbapi/cursor.py b/spanner/dbapi/cursor.py
--- a/spanner/dbapi/cursor.py
+++ b/spanner/dbapi/cursor.py
@@ -67,6 +67,10 @@ def execute(self, sql, args=None):
         if not self.__session:
             raise ProgrammingError('Cursor is not connected to the database')
 
+        # param_types doesn't seem required except as an empty dict to avoid
+        # ValueError("Specify 'param_types' when passing 'params'.").
+        # See https://github.com/orijtech/spanner-orm/issues/35
+        param_types = {} if args else None
         # Classify whether this is a read-only SQL statement.
         try:
             if classify_stmt(sql) == STMT_DDL:
@@ -74,31 +78,17 @@ def execute(self, sql, args=None):
                 # by invoking `execute_sql` or `execute_update`, we must run
                 # DDL updates on the database handle itself.
                 self.__do_update_ddl(sql)
-
             elif classify_stmt(sql) == STMT_NON_UPDATING:
                 self.__do_execute_non_update(
                     sql, args or None,
-                    # param_types aren't actually required but an empty dict must be passed
-                    # to avoid
-                    #   "ValueError("Specify 'param_types' when passing 'params'.").
-                    # See https://github.com/orijtech/spanner-orm/issues/35
-                    # Instead of examining every parameter and making
-                    # a corresponding protobuf cast, we can just pass
-                    # in param_types={}
-                    param_types={})
-
+                    param_types=param_types,
+                )
             else:
                 self.__session.run_in_transaction(
                     self.__do_execute_update,
                     sql, args or None,
-                    # param_types aren't actually required but an empty dict must be
-                    # passed to avoid
-                    #   "ValueError("Specify 'param_types' when passing 'params'.").
-                    # See https://github.com/orijtech/spanner-orm/issues/35
-                    # Instead of examining every parameter and making
-                    # a corresponding protobuf cast, we can just pass
-                    # in param_types={}
-                    param_types={})
+                    param_types=param_types,
+                )
 
         except grpc_exceptions.AlreadyExists as e:
             raise IntegrityError(e.details if hasattr(e, 'details') else e)
diff --git a/spanner/django/schema.py b/spanner/django/schema.py
--- a/spanner/django/schema.py
+++ b/spanner/django/schema.py
@@ -3,6 +3,7 @@
 
 class DatabaseSchemaEditor(BaseDatabaseSchemaEditor):
     sql_create_table = "CREATE TABLE %(table)s (%(definition)s) PRIMARY KEY(%(primary_key)s)"
+    sql_create_unique = "CREATE UNIQUE INDEX %(name)s ON %(table)s (%(columns)s)"
 
     def create_model(self, model):
         """


Test patch: