Repository: googleapis/python-spanner-django

Issue:
spanner/django: waiting on long running operations for every update_ddl slows down operations
After PR #75 which was meant to fix #74 it now takes about 11min to 14min to invoke 
```shell
python3 manage.py migrate
```

```shell
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying polls.0001_initial... OK
  Applying sessions.0001_initial... OK

real	10m46.339s
user	0m1.445s
sys	0m0.313s
```
and
```shell
real	13m6.239s
user	0m1.419s
sys	0m0.321s
```

However, many of the operations can be performed asynchronously such as creating indices and ALTER-ations. We definitely need `CREATE TABLE` to be synchronous since in any migrate, the next command could refer to a table that doesn't exist, and I've seen such errors before e.g.
```shell
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Table not found: django_migrations
```

### After prototyping
I prototyped a change that will only synchronously wait on `CREATE TABLE` and the result was a little better
```diff
diff --git a/spanner/dbapi/connection.py b/spanner/dbapi/connection.py
index ac9d98c..5e24971 100644
--- a/spanner/dbapi/connection.py
+++ b/spanner/dbapi/connection.py
@@ -64,6 +64,4 @@ class Connection(object):
         Returns:
             google.api_core.operation.Operation
         """
-        lro = self.__dbhandle.update_ddl(ddl_statements)
-        # Synchronously wait on the operation's completion.
-        return lro.result()
+        return self.__dbhandle.update_ddl(ddl_statements)
diff --git a/spanner/dbapi/cursor.py b/spanner/dbapi/cursor.py
index 76002db..d4db8a2 100644
--- a/spanner/dbapi/cursor.py
+++ b/spanner/dbapi/cursor.py
@@ -230,7 +230,18 @@ class Cursor(object):
         if not self.__db_handle:
             raise ProgrammingError('Trying to run an DDL update but no database handle')
 
-        return self.__db_handle.update_ddl(ddl_statements)
+        lro = self.__db_handle.update_ddl(ddl_statements)
+        if has_create_stmt(ddl_statements):
+            return lro.result()
+        else:
+            return lro
+
+
+def has_create_stmt(ddl_statements):
+    for ddl_stmt in ddl_statements:
+        if 'CREATE TABLE' in ddl_stmt:
+            return True
+    return False
 
 
 class Column:
```

and that took about 4minutes.
```shell
real	4m1.102s
user	0m1.284s
sys	0m0.254s
```

more than a 50% reduction in time, but we could still do better, but for now this suffices.


Discussion:

Resolving patch:
diff --git a/spanner/dbapi/connection.py b/spanner/dbapi/connection.py
--- a/spanner/dbapi/connection.py
+++ b/spanner/dbapi/connection.py
@@ -64,6 +64,4 @@ def update_ddl(self, ddl_statements):
         Returns:
             google.api_core.operation.Operation
         """
-        lro = self.__dbhandle.update_ddl(ddl_statements)
-        # Synchronously wait on the operation's completion.
-        return lro.result()
+        return self.__dbhandle.update_ddl(ddl_statements)
diff --git a/spanner/dbapi/cursor.py b/spanner/dbapi/cursor.py
--- a/spanner/dbapi/cursor.py
+++ b/spanner/dbapi/cursor.py
@@ -230,7 +230,22 @@ def __do_update_ddl(self, *ddl_statements):
         if not self.__db_handle:
             raise ProgrammingError('Trying to run an DDL update but no database handle')
 
-        return self.__db_handle.update_ddl(ddl_statements)
+        lro = self.__db_handle.update_ddl(ddl_statements)
+
+        if has_create_table_stmt(ddl_statements):
+            # We only care about waiting on the result of CREATE TABLE DDL
+            # statements because tables could be referenced, whereas indices are
+            # most likely to be used not immediately, same thing for ALTER statements.
+            return lro.result()
+        else:
+            return lro
+
+
+def has_create_table_stmt(ddl_statements):
+    for ddl_stmt in ddl_statements:
+        if ddl_stmt.upper().startswith('CREATE TABLE'):
+            return True
+    return False
 
 
 class Column:


Test patch: