internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: googleapis/python-spanner-django

Issue:
django-test-suite: constraints failure due to CHECK CONSTRAINT being added yet unsupported by Cloud Spanner
When I run the constraints test suite for Django, I encountered this error
```shell
django.db.utils.ProgrammingError: 400 Errors parsing Spanner DDL statement:
CREATE TABLE constraints_product (id INT64 NOT NULL, name STRING(255) NOT NULL,
color STRING(32), price INT64, discounted_price INT64,
CONSTRAINT price_gt_discounted_price CHECK (price > (discounted_price)), CONSTRAINT name_color_uniq UNIQUE (name, color)) PRIMARY KEY(id) : Syntax error on line 1, column 147: Expecting \'EOF\' but found \'price_gt_discounted_price\'
Syntax error on line 1, column 147: Encountered \'price_gt_discounted_price\' while parsing: column_type
Syntax error on line 1, column 147: Expecting \')\' but found \'price_gt_discounted_price\'
Syntax error on line 1, column 147: Expecting \'PRIMARY\' but found \'price_gt_discounted_price\'
Syntax error on line 1, column 147: Expecting \'KEY\' but found \'price_gt_discounted_price\'
Syntax error on line 1, column 147: Expecting \'(\' but found \'price_gt_discounted_price\'
Syntax error on line 1, column 173: Expecting \')\' but found \'CHECK\'
Syntax error on line 1, column 173: Expecting \'EOF\' ...
```
and for clarity here is the SQL statement
```sql
CREATE TABLE constraints_product (
    id INT64 NOT NULL,
    name STRING(255) NOT NULL,
    color STRING(32), price INT64,
    discounted_price INT64,
    CONSTRAINT price_gt_discounted_price CHECK (price > (discounted_price)),
    CONSTRAINT name_color_uniq UNIQUE (name, color)
) PRIMARY KEY(id)
```
but Cloud Spanner says it doesn't support CONSTRAINTs and advises that any checks SHOULD be moved to the actual application https://cloud.google.com/spanner/docs/migrating-postgres-spanner#foreign_keys_and_referential_integrity
![Screen Shot 2019-12-04 at 1 45 48 AM](https://user-images.githubusercontent.com/4898263/70131832-d291bc00-1637-11ea-9456-4112bf810800.png)


Discussion:

Resolving patch:
diff --git a/spanner/django/features.py b/spanner/django/features.py
--- a/spanner/django/features.py
+++ b/spanner/django/features.py
@@ -4,6 +4,8 @@
 class DatabaseFeatures(BaseDatabaseFeatures):
     supports_foreign_keys = False
     supports_transactions = False
+    supports_column_check_constraints = False
+    supports_table_check_constraints = False
 
     # Django tests that aren't supported by Spanner.
     skip_tests = (


Test patch:
Repository: googleapis/python-spanner-django

Issue:
spanner/django: CREATE TABLE with field of STRING and BYTES without a length is an error on Cloud Spanner
While running the test suite, encountered the exception
```shell
google.api_core.exceptions.InvalidArgument: 400 Errors parsing Spanner DDL statement: CREATE TABLE serializers_binarydata (id INT64 NOT NULL, data BYTES) PRIMARY KEY(id) : Syntax error on line 1, column 67: Expecting \'(\' but found \')\'\nSyntax error on line 1, column 67: Expecting \'EOF\' but found \')\'\nSyntax error on line 1, column 67: Encountered \')\' while parsing: column_length\nSyntax error on line 1, column 69: Expecting \')\' but found \'PRIMARY\'
```
where the actual error is
```shell
google.api_core.exceptions.InvalidArgument: 400
Errors parsing Spanner DDL statement: CREATE TABLE serializers_binarydata (id INT64 NOT NULL, data BYTES) PRIMARY KEY(id) : Syntax error on line 1, column 67: Expecting \'(\' but found \')\'
Syntax error on line 1, column 67: Expecting \'EOF\' but found \')\'
Syntax error on line 1, column 67: Encountered \')\' while parsing: column_length
Syntax error on line 1, column 69: Expecting \')\' but found \'PRIMARY\'
```

and the CREATE TABLE statement is
```sql
CREATE TABLE serializers_binarydata (
    id INT64 NOT NULL,
    data BYTES
) PRIMARY KEY(id)
```
What perhaps we should do is detect the lack of definition value and instead insert MAX so for example fix that to
```sql
CREATE TABLE serializers_binarydata (
    id INT64 NOT NULL,
    data BYTES(MAX)
) PRIMARY KEY(id)
```


Discussion:

Resolving patch:
diff --git a/spanner/django/base.py b/spanner/django/base.py
--- a/spanner/django/base.py
+++ b/spanner/django/base.py
@@ -33,7 +33,7 @@ class DatabaseWrapper(BaseDatabaseWrapper):
     data_types = {
         'AutoField': 'INT64',
         'BigAutoField': 'INT64',
-        'BinaryField': 'BYTES',
+        'BinaryField': 'BYTES(MAX)',
         'BooleanField': 'BOOL',
         'CharField': 'STRING(%(max_length)s)',
         'DateField': 'DATE',
@@ -66,7 +66,7 @@ class DatabaseWrapper(BaseDatabaseWrapper):
         'SlugField': 'STRING(%(max_length)s)',
         'SmallAutoField': 'INT64',
         'SmallIntegerField': 'INT64',
-        'TextField': 'STRING(10000)',
+        'TextField': 'STRING(MAX)',
         # With or without the time zone, Spanner expects the time field as a string.
         'TimeField': 'STRING(50)',
         # A UUID4 like 'd9c388b1-184d-4511-a818-3d598cc2f847', 16 bytes, with 4 dashes.


Test patch:
Repository: googleapis/python-spanner-django

Issue:
dbapi: set arraysize to 1 as recommended by PEP-0249
PEP-0249's arraysize section recommends that we set it to 1, but we had it unimplemented. It should also be readable and writable instead of only readable.


Discussion:

Resolving patch:
diff --git a/spanner/dbapi/connection.py b/spanner/dbapi/connection.py
--- a/spanner/dbapi/connection.py
+++ b/spanner/dbapi/connection.py
@@ -32,6 +32,7 @@ def __raise_if_already_closed(self):
     def close(self):
         self.commit()
         self.__raise_if_already_closed()
+        self._clear_all_sessions()
         self.__dbhandle = None
         self.__closed = True
 
@@ -63,11 +64,19 @@ def append_to_batch_stack(self, op, table, columns, values):
         self.__ops.append((op, table, columns, values))
 
     def cursor(self):
-        session = self.__dbhandle.session()
-        if not session.exists():
-            session.create()
+        session = self._new_session()
         return Cursor(session, self)
 
+    def _new_session(self):
+        return self.__dbhandle._pool.get()
+
+    def _done_with_session(self, session):
+        if session:
+            self.__dbhandle._pool.put(session)
+
+    def _clear_all_sessions(self):
+        return self.__dbhandle._pool.clear()
+
     def update_ddl(self, ddl_statements):
         """
         Runs the list of Data Definition Language (DDL) statements on the specified
diff --git a/spanner/dbapi/cursor.py b/spanner/dbapi/cursor.py
--- a/spanner/dbapi/cursor.py
+++ b/spanner/dbapi/cursor.py
@@ -53,10 +53,7 @@ def rowcount(self):
     def close(self):
         self.__commit_preceding_batch()
 
-        if not self.__session:
-            return
-
-        self.__session.delete()
+        self.__db_handle._done_with_session(self.__session)
         self.__session = None
 
     def execute(self, sql, args=None):


Test patch:
Repository: googleapis/python-spanner-django

Issue:
dbapi: set arraysize to 1 as recommended by PEP-0249
PEP-0249's arraysize section recommends that we set it to 1, but we had it unimplemented. It should also be readable and writable instead of only readable.


Discussion:

Resolving patch:
diff --git a/spanner/dbapi/cursor.py b/spanner/dbapi/cursor.py
--- a/spanner/dbapi/cursor.py
+++ b/spanner/dbapi/cursor.py
@@ -34,6 +34,11 @@ def __init__(self, db_handle=None):
         self.__row_count = _UNSET_COUNT
         self.__db_handle = db_handle
 
+        # arraysize is a readable and writable property mandated
+        # by PEP-0249 https://www.python.org/dev/peps/pep-0249/#arraysize
+        # It determines the results of .fetchmany
+        self.arraysize = 1
+
     @property
     def description(self):
         if not (self.__res and self.__res.metadata):
@@ -220,7 +225,7 @@ def fetchone(self):
     def fetchall(self):
         return list(self.__iter__())
 
-    def fetchmany(self, size=50):
+    def fetchmany(self, size=None):
         """
         Fetch the next set of rows of a query result, returning a sequence of sequences.
         An empty sequence is returned when no more rows are available.
@@ -233,6 +238,9 @@ def fetchmany(self, size=50):
             Error if the previous call to .execute*() did not produce any result set
             or if no call was issued yet.
         """
+        if size is None:
+            size = self.arraysize
+
         items = []
         for i in range(size):
             try:
@@ -242,10 +250,6 @@ def fetchmany(self, size=50):
 
         return items
 
-    @property
-    def arraysize(self):
-        raise ProgrammingError('Unimplemented')
-
     @property
     def lastrowid(self):
         return None


Test patch:
Repository: googleapis/python-spanner-django

Issue:
django: store DurationField as INT64 (seconds) rather than STRING
See https://github.com/django/django/commit/57554442fe3e209c135e15dda4ea45123e579e58 for the implementation on other databases like MySQL and Oracle.


Discussion:

Resolving patch:
diff --git a/spanner/django/base.py b/spanner/django/base.py
--- a/spanner/django/base.py
+++ b/spanner/django/base.py
@@ -39,18 +39,7 @@ class DatabaseWrapper(BaseDatabaseWrapper):
         'DateField': 'DATE',
         'DateTimeField': 'TIMESTAMP',
         'DecimalField': 'FLOAT64',
-        # Django extracts this field with parse_duration by invoking
-        #   https://docs.djangoproject.com/en/2.2/_modules/django/utils/dateparse/#parse_duration
-        # starting from
-        #   https://docs.djangoproject.com/en/2.2/_modules/django/db/models/fields/#DurationField
-        # expecting format "[DD] [HH:[MM:]]ss[.uuuuuu]"
-        # also see
-        #   https://cloud.google.com/spanner/docs/migrating-postgres-spanner#data_types
-        # which recommends either:
-        #   "INT64" if storing the value in milliseconds
-        #  OR
-        #   "STRING" if storing the value in an application-defined interval format.
-        'DurationField': 'STRING(50)',
+        'DurationField': 'INT64',
         'EmailField': 'STRING(%(max_length)s)',
         'FileField': 'STRING(%(max_length)s)',
         'FilePathField': 'STRING(%(max_length)s)',
diff --git a/spanner/django/features.py b/spanner/django/features.py
--- a/spanner/django/features.py
+++ b/spanner/django/features.py
@@ -17,6 +17,7 @@ class DatabaseFeatures(BaseDatabaseFeatures):
         'basic.tests.SelectOnSaveTests.test_select_on_save_lying_update',
         # spanner.django monkey patches AutoField to have a default value.
         'basic.tests.ModelTest.test_hash',
+        'model_fields.test_durationfield.TestSerialization.test_dumping',
         'model_fields.test_uuid.TestSerialization.test_dumping',
         # Tests that assume a serial pk.
         'admin_views.tests.AdminViewPermissionsTest.test_history_view',


Test patch:
Repository: googleapis/python-spanner-django

Issue:
dbapi: ensure that Cursor.__commit_preceding_batch flushes everything on DQL
If we encounter a DQL statement such as SELECT, we MUST flush any previously batched operations and run them immediately otherwise we encounter a situation where duplicate table create statements will be issued because the command issued right after DDL will report to the batcher that the last operation is DDL. The second DQL statement will finally return results.

Noticed from https://travis-ci.com/orijtech/spanner-orm/jobs/267962237#L489 by error
```shell
google.api_core.exceptions.GoogleAPICallError: None Duplicate name in schema: django_migrations.
```

and from debugging offline and with various tests that were failing.


Discussion:

Resolving patch:
diff --git a/spanner/dbapi/cursor.py b/spanner/dbapi/cursor.py
--- a/spanner/dbapi/cursor.py
+++ b/spanner/dbapi/cursor.py
@@ -165,7 +165,14 @@ def __execute_insert_no_params(self, transaction, sql):
     def __commit_preceding_batch(self, op=None):
         last_op = self.__last_op
         self.__last_op = op
-        self.__db_handle.commit(last_op)
+        if op is OP_DQL:
+            # Unconditionally flush all operations
+            # before any DQL runs to ensure that
+            # any stale batched data that hasn't yet been uploaded
+            # to Cloud Spanner doesn't linger. See issue #213.
+            return self.__db_handle.commit(OP_DQL)
+        else:
+            return self.__db_handle.commit(last_op)
 
     def __handle_DQL(self, sql, params, param_types=None):
         self.__commit_preceding_batch(OP_DQL)


Test patch:
Repository: googleapis/python-spanner-django

Issue:
dbapi: remove automagic instance and database creation
So to bootstrap this library and for convenience, I made the design decision to automagically create instances and libraries. However, @timgraham raised a great point that if someone makes a single typo, the can't catch it and might deploy to production without any loud failures to explicitly tell them a database/instance don't exist.

The relevant code that does the automagic creation is:
https://github.com/orijtech/spanner-orm/blob/cbe4822b965651a85f306ac4447a9357e8bdf1f5/spanner/dbapi/__init__.py#L96-L111


Discussion:
/cc @larkee @skuruppu 

Resolving patch:
diff --git a/spanner/dbapi/__init__.py b/spanner/dbapi/__init__.py
--- a/spanner/dbapi/__init__.py
+++ b/spanner/dbapi/__init__.py
@@ -23,7 +23,6 @@
 )
 from .parse_utils import (
     extract_connection_params, infer_param_types, parse_spanner_url,
-    validate_instance_config,
 )
 from .types import (
     BINARY, DATETIME, NUMBER, ROWID, STRING, Binary, Date, DateFromTicks, Time,
@@ -95,27 +94,11 @@ def connect(spanner_url, credentials_uri=None):
 
     instance = client.instance(instance_name)
     if not instance.exists():
-        # Attempt to create the instance if it doesn't yet exist.
-        instance_config = conn_params.get('instance_config')
-        if not instance_config:
-            raise Error("instance '%s' does not yet exist yet no 'default_zone' was set" % instance_name)
-
-        err_msg = validate_instance_config(instance_config)
-        if err_msg:
-            raise Error(err_msg)
-
-        instance.configuration_name = instance_config
-        lro = instance.create()
-        # Synchronously wait on the operation's completion.
-        # TODO: Report the long-running operation result.
-        _ = lro.result()
+        raise ProgrammingError("instance '%s' does not yet exist" % instance_name)
 
     db = instance.database(db_name)
     if not db.exists():
-        lro = db.create()
-        # Synchronously wait on the operation's completion.
-        # TODO: Report the long-running operation result.
-        _ = lro.result()
+        raise ProgrammingError("database '%s' does not yet exist" % db_name)
 
     return Connection(db)
 


Test patch:
Repository: googleapis/python-spanner-django

Issue:
dbapi: space invalid in column name
Trying to run
```
CREATE TABLE aggregation_regress_entries (
`Entry ID` INT64 NOT NULL,
Entry STRING(50) NOT NULL,
`Exclude` BOOL NOT NULL)
PRIMARY KEY(`Entry ID`)
```
gives `GoogleAPICallError: None Column name not valid: aggregation_regress_entries.Entry ID.`

It seems to [contradict the documentation](https://cloud.google.com/spanner/docs/lexical) which says "Quoted identifiers are identifiers enclosed by backtick (`) characters and can contain any character, such as spaces or symbols."

We could fix this by having the dbapi replace spaces with underscores, for example, even if that risks a collision with another column name. Django typically detects such collisions, but that protection won't work if the dbapi modifies names. 


Discussion:

Resolving patch:
diff --git a/spanner/dbapi/parse_utils.py b/spanner/dbapi/parse_utils.py
--- a/spanner/dbapi/parse_utils.py
+++ b/spanner/dbapi/parse_utils.py
@@ -536,6 +536,6 @@ def escape_name(name):
     Escape name by applying backticks to value that either
     contain '-' or are any of Cloud Spanner's reserved keywords.
     """
-    if '-' in name or name.upper() in SPANNER_RESERVED_KEYWORDS:
+    if '-' in name or ' ' in name or name.upper() in SPANNER_RESERVED_KEYWORDS:
         return '`' + name + '`'
     return name
diff --git a/spanner/django/operations.py b/spanner/django/operations.py
--- a/spanner/django/operations.py
+++ b/spanner/django/operations.py
@@ -1,3 +1,4 @@
+import os
 import re
 from base64 import b64decode
 from datetime import datetime, time
@@ -24,6 +25,13 @@ class DatabaseOperations(BaseDatabaseOperations):
     }
 
     def quote_name(self, name):
+        # Spanner says "column name not valid" if spaces are present (although
+        # according the docs, they should be allowed). Replace spaces when
+        # running the Django tests to prevent crashes. (Don't modify names in
+        # normal operation to prevent the possibility of colliding with another
+        # column.) https://github.com/orijtech/spanner-orm/issues/204
+        if os.environ.get('RUNNING_SPANNER_BACKEND_TESTS') == '1':
+            name = name.replace(' ', '_')
         return escape_name(name)
 
     def bulk_insert_sql(self, fields, placeholder_rows):


Test patch:
diff --git a/tests/spanner/dbapi/test_parse_utils.py b/tests/spanner/dbapi/test_parse_utils.py
--- a/tests/spanner/dbapi/test_parse_utils.py
+++ b/tests/spanner/dbapi/test_parse_utils.py
@@ -399,6 +399,7 @@ def test_escape_name(self):
                 ('id', 'id'),
                 ('', ''),
                 ('dashed-value', '`dashed-value`'),
+                ('with space', '`with space`'),
         ]
 
         for name, want in cases:
Repository: googleapis/python-spanner-django

Issue:
dbapi: fix paramstyle naming to match how we translate
Brain fart on my end, but @bvandiver carefully noticed in design doc review that I wrongly interpreted the paramstyle variable as per
https://github.com/orijtech/django-spanner/blob/e53ba7c2aa8175f0e581d60e889dca0620af4279/spanner/dbapi/__init__.py#L25-L28

which on re-reading out loud https://www.python.org/dev/peps/pep-0249/#paramstyle, yes the Cursor expects '%s' params and under the hood we translate them to '@' thus the paramstyle should be 'format' for `ANSI C style format codes e.g. '%s'`


Discussion:

Resolving patch:
diff --git a/spanner/dbapi/__init__.py b/spanner/dbapi/__init__.py
--- a/spanner/dbapi/__init__.py
+++ b/spanner/dbapi/__init__.py
@@ -22,10 +22,9 @@
 
 # Globals that MUST be defined ###
 apilevel = "2.0"  # Implements the Python Database API specification 2.0 version.
-# '@' is used by Cloud Spanner as the param style but that style isn't listed
-# in any of the options  https://www.python.org/dev/peps/pep-0249/#paramstyle
-# so we are going with a custom named paramstyle.
-paramstyle = 'at-named'
+# We accept arguments in the format '%s' aka ANSI C print codes.
+# as per https://www.python.org/dev/peps/pep-0249/#paramstyle
+paramstyle = 'format'
 # Threads may not share the module. This is a paranoid threadsafety level, but it is
 # necessary for starters to use when debugging failures. Eventually once transactions
 # are working properly, we'll update the threadsafety level.


Test patch:
diff --git a/tests/spanner/dbapi/test_globals.py b/tests/spanner/dbapi/test_globals.py
--- a/tests/spanner/dbapi/test_globals.py
+++ b/tests/spanner/dbapi/test_globals.py
@@ -12,5 +12,5 @@
 class DBAPIGlobalsTests(TestCase):
     def test_apilevel(self):
         self.assertEqual(apilevel, '2.0', 'We implement PEP-0249 version 2.0')
-        self.assertEqual(paramstyle, 'at-named', 'Cloud Spanner uses @param')
+        self.assertEqual(paramstyle, 'format', 'Cloud Spanner uses @param')
         self.assertEqual(threadsafety, 0, 'Downgraded to the least concurrency')
Repository: googleapis/python-spanner-django

Issue:
django: disable column defaults since Spanner doesn't support them
While running the code for https://healthchecks.io/ from https://github.com/healthchecks/healthchecks/ and running `python3 manage.py migrate`, I encountered this issued statement
```shell
SQL:  ALTER TABLE accounts_profile ADD COLUMN ping_log_limit INT64 DEFAULT %s NOT NULL
Args: [100]
```

but we seem to be handling ALTERs for not null with
https://github.com/orijtech/django-spanner/blob/d0f73005d4e98b217f9b48129aac44164d6f1f07/spanner/django/schema.py#L19


The problem seems to when changing keys to become default and here is that the original DDL looked like
```sql
CREATE TABLE accounts_profile (
    id INT64 NOT NULL,
    next_report_date TIMESTAMP,
    reports_allowed BOOL NOT NULL,
    user_id INT64
) PRIMARY KEY(id)
```
and we tried to ALTER the table by adding a column with `DEFAULT <VALUE>`

Anyways Cloud Spanner's lack of support for DEFAULT constraint should be recognized by us and at least explicitly called out by the Cloud Spanner docs.


Discussion:
It looks like there are two issues here. First, Spanner doesn't support column defaults. Second, Spanner can't add NOT NULL columns to existing tables. The second issue is tracked in #84, so I'm going to repurpose this issue for the first one.

Resolving patch:
diff --git a/spanner/django/schema.py b/spanner/django/schema.py
--- a/spanner/django/schema.py
+++ b/spanner/django/schema.py
@@ -101,19 +101,6 @@ def column_sql(self, model, field, include_default=False):
             return None, None
         # Work out nullability
         null = field.null
-        # If we were told to include a default value, do so
-        include_default = include_default and not self.skip_default(field)
-        if include_default:
-            default_value = self.effective_default(field)
-            if default_value is not None:
-                if self.connection.features.requires_literal_defaults:
-                    # Some databases can't take defaults as a parameter (oracle)
-                    # If this is the case, the individual schema backend should
-                    # implement prepare_default
-                    sql += " DEFAULT %s" % self.prepare_default(default_value)
-                else:
-                    sql += " DEFAULT %s"
-                    params += [default_value]
         # Oracle treats the empty string ('') as null, so coerce the null
         # option whenever '' is a possible value.
         if (field.empty_strings_allowed and not field.primary_key and
@@ -143,3 +130,7 @@ def _unique_sql(self, model, fields, name, condition=None):
         if sql:
             self.deferred_sql.append(sql)
         return None
+
+    def skip_default(self, field):
+        """Cloud Spanner doesn't support column defaults."""
+        return True


Test patch: