Repository: googleapis/python-spanner-django

Issue:
spanner/django: fields with Google SQL keywords cause exceptions
Just while running the test suite, I got this exception after this DDL was passed in
```sql
CREATE TABLE validation_uniquefordatemodel (
       id INT64 NOT NULL,
       start_date DATE NOT NULL,
       end_date TIMESTAMP NOT NULL,
       count INT64 NOT NULL,
       order INT64 NOT NULL,
       name STRING(100) NOT NULL
) PRIMARY KEY(id)
```

fails with
```shell
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Errors parsing Spanner DDL statement: CREATE TABLE validation_uniquefordatemodel (id INT64 NOT NULL, start_date DATE NOT NULL, end_date TIMESTAMP NOT NULL, count INT64 NOT NULL, order INT64 NOT NULL, name STRING(100) NOT NULL) PRIMARY KEY(id) : Syntax error on line 1, column 141: Expecting \')\' but found \'order\'\nSyntax error on line 1, column 141: Expecting \'PRIMARY\' but found \'order\'\nSyntax error on line 1, column 141: Expecting \'KEY\' but found \'order\'\nSyntax error on line 1, column 141: Expecting \'(\' but found \'order\'\nSyntax error on line 1, column 141: Expecting \')\' but found \'order\'\nSyntax error on line 1, column 141: Expecting \'EOF\' but found \'order\'"
```

I don't think that there is much we can do here if Cloud Spanner is the one tripping out with field names or the parser on Cloud Spanner's backend can't recognize the context of it being a field /cc @skuruppu as our Cloud Spanner engineering contact.

@timgraham users will get failures anyways in their Django projects so perhaps we'll just need to document and skip these tests.


Discussion:
Okay the advisory at https://cloud.google.com/spanner/docs/data-definition-language#reserved_keywords says that if we need to use reserved keywords, they SHOULD be enclosed in backticks
![Screen Shot 2019-12-04 at 12 57 23 AM](https://user-images.githubusercontent.com/4898263/70128068-1a611500-1631-11ea-8c2d-27738ca72aa4.png)

so basically we'll need to parse through each field name and backtick escape it if it includes any reserved keyword with a listing of reserved keywords at https://cloud.google.com/spanner/docs/lexical#reserved-keywords

Changing the status of this from a documentation relegation to something we must implement.
This failure also blocks the modeladmin testsuite
```shell
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Errors parsing Spanner DDL statement: CREATE TABLE modeladmin_validationtestmodel (id INT64 NOT NULL, name STRING(100) NOT NULL, slug STRING(50) NOT NULL, state STRING(2) NOT NULL, is_active BOOL NOT NULL, pub_date TIMESTAMP NOT NULL, band_id INT64 NOT NULL, best_friend_id INT64 NOT NULL, no INT64) PRIMARY KEY(id) : Syntax error on line 1, column 253: Expecting \')\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'PRIMARY\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'KEY\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'(\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \')\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'EOF\' but found \'no\'"
	debug_error_string = "{"created":"@1575449691.131220000","description":"Error received from peer ipv4:172.217.5.74:443","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"Errors parsing Spanner DDL statement: CREATE TABLE modeladmin_validationtestmodel (id INT64 NOT NULL, name STRING(100) NOT NULL, slug STRING(50) NOT NULL, state STRING(2) NOT NULL, is_active BOOL NOT NULL, pub_date TIMESTAMP NOT NULL, band_id INT64 NOT NULL, best_friend_id INT64 NOT NULL, no INT64) PRIMARY KEY(id) : Syntax error on line 1, column 253: Expecting \')\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'PRIMARY\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'KEY\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'(\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \')\' but found \'no\'\nSyntax error on line 1, column 253: Expecting \'EOF\' but found \'no\'","grpc_status":3}"
>
```
Duplicate of https://github.com/orijtech/spanner-orm/issues/110.

Resolving patch:
diff --git a/spanner/dbapi/parse_utils.py b/spanner/dbapi/parse_utils.py
--- a/spanner/dbapi/parse_utils.py
+++ b/spanner/dbapi/parse_utils.py
@@ -445,3 +445,112 @@ def insert_key_in_param_types(key, param_types, spanner_type):
         param_types[key] = spanner_type
 
     return param_types
+
+
+SPANNER_RESERVED_KEYWORDS = {
+    'ALL',
+    'AND',
+    'ANY',
+    'ARRAY',
+    'AS',
+    'ASC',
+    'ASSERT_ROWS_MODIFIED',
+    'AT',
+    'BETWEEN',
+    'BY',
+    'CASE',
+    'CAST',
+    'COLLATE',
+    'CONTAINS',
+    'CREATE',
+    'CROSS',
+    'CUBE',
+    'CURRENT',
+    'DEFAULT',
+    'DEFINE',
+    'DESC',
+    'DISTINCT',
+    'ELSE',
+    'END',
+    'ENUM',
+    'ESCAPE',
+    'EXCEPT',
+    'EXCLUDE',
+    'EXISTS',
+    'EXTRACT',
+    'FALSE',
+    'FETCH',
+    'FOLLOWING',
+    'FOR',
+    'FROM',
+    'FULL',
+    'GROUP',
+    'GROUPING',
+    'GROUPS',
+    'HASH',
+    'HAVING',
+    'IF',
+    'IGNORE',
+    'IN',
+    'INNER',
+    'INTERSECT',
+    'INTERVAL',
+    'INTO',
+    'IS',
+    'JOIN',
+    'LATERAL',
+    'LEFT',
+    'LIKE',
+    'LIMIT',
+    'LOOKUP',
+    'MERGE',
+    'NATURAL',
+    'NEW',
+    'NO',
+    'NOT',
+    'NULL',
+    'NULLS',
+    'OF',
+    'ON',
+    'OR',
+    'ORDER',
+    'OUTER',
+    'OVER',
+    'PARTITION',
+    'PRECEDING',
+    'PROTO',
+    'RANGE',
+    'RECURSIVE',
+    'RESPECT',
+    'RIGHT',
+    'ROLLUP',
+    'ROWS',
+    'SELECT',
+    'SET',
+    'SOME',
+    'STRUCT',
+    'TABLESAMPLE',
+    'THEN',
+    'TO',
+    'TREAT',
+    'TRUE',
+    'UNBOUNDED',
+    'UNION',
+    'UNNEST',
+    'USING',
+    'WHEN',
+    'WHERE',
+    'WINDOW',
+    'WITH',
+    'WITHIN',
+}
+
+
+def escape_name(name):
+    """
+    Escape name by applying backticks to value that either
+    contain '-' or are any of Cloud Spanner's reserved keywords.
+    """
+    if '-' in name or name.upper() in SPANNER_RESERVED_KEYWORDS:
+        return '`' + name + '`'
+    return name
diff --git a/spanner/django/operations.py b/spanner/django/operations.py
--- a/spanner/django/operations.py
+++ b/spanner/django/operations.py
@@ -3,7 +3,7 @@
 from django.conf import settings
 from django.db.backends.base.operations import BaseDatabaseOperations
 from django.utils import timezone
-from spanner.dbapi.parse_utils import DateStr, TimestampStr
+from spanner.dbapi.parse_utils import DateStr, TimestampStr, escape_name
 
 
 class DatabaseOperations(BaseDatabaseOperations):
@@ -17,9 +17,7 @@ class DatabaseOperations(BaseDatabaseOperations):
     }
 
     def quote_name(self, name):
-        if '-' in name:
-            return '`' + name + '`'
-        return name
+        return escape_name(name)
 
     def bulk_insert_sql(self, fields, placeholder_rows):
         placeholder_rows_sql = (", ".join(row) for row in placeholder_rows)


Test patch:
diff --git a/tests/spanner/dbapi/test_parse_utils.py b/tests/spanner/dbapi/test_parse_utils.py
--- a/tests/spanner/dbapi/test_parse_utils.py
+++ b/tests/spanner/dbapi/test_parse_utils.py
@@ -18,8 +18,8 @@
 from spanner.dbapi.exceptions import Error
 from spanner.dbapi.parse_utils import (
     STMT_DDL, STMT_NON_UPDATING, DateStr, TimestampStr, classify_stmt,
-    ensure_where_clause, extract_connection_params, infer_param_types,
-    parse_insert, parse_spanner_url, reINSTANCE_CONFIG,
+    ensure_where_clause, escape_name, extract_connection_params,
+    infer_param_types, parse_insert, parse_spanner_url, reINSTANCE_CONFIG,
     rows_for_insert_or_update, sql_pyformat_args_to_spanner,
     validate_instance_config,
 )
@@ -406,3 +406,16 @@ def test_ensure_where_clause(self):
             with self.subTest(sql=sql):
                 got = ensure_where_clause(sql)
                 self.assertEqual(got, want)
+
+    def test_escape_name(self):
+        cases = [
+                ('SELECT', '`SELECT`'),
+                ('id', 'id'),
+                ('', ''),
+                ('dashed-value', '`dashed-value`'),
+        ]
+
+        for name, want in cases:
+            with self.subTest(name=name):
+                got = escape_name(name)
+                self.assertEqual(got, want)