internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: googleapis/google-cloud-python

Issue:
Shouldn't gcloud.dataset.transaction.Transaction rollback on error exit from context manager?
If the application raises inside a 'with transaction:' block, the transaction should be rolled back, rather than blindlin committed.



Discussion:

Resolving patch:
diff --git a/gcloud/datastore/transaction.py b/gcloud/datastore/transaction.py
--- a/gcloud/datastore/transaction.py
+++ b/gcloud/datastore/transaction.py
@@ -21,18 +21,16 @@ class Transaction(object):
     ...   entity1.save()
     ...   entity2.save()
 
-  To rollback a transaction if there is an error::
+  By default, the transaction is rolled back if the transaction block
+  exits with an error::
 
     >>> from gcloud import datastore
     >>> dataset = datastore.get_dataset('dataset-id', email, key_path)
     >>> with dataset.transaction() as t:
-    ...   try:
-    ...     do_some_work()
-    ...     entity1.save()
-    ...   except:
-    ...     t.rollback()
+    ...   do_some_work()
+    ...   raise Exception() # rolls back
 
-  If the transaction isn't rolled back,
+  If the transaction block exists without an exception,
   it will commit by default.
 
   .. warning::
@@ -249,4 +247,7 @@ def __enter__(self):
     return self
 
   def __exit__(self, exc_type, exc_val, exc_tb):
-    self.commit()
+    if exc_type is None:
+      self.commit()
+    else:
+      self.rollback()


Test patch:
diff --git a/gcloud/datastore/test_transaction.py b/gcloud/datastore/test_transaction.py
--- a/gcloud/datastore/test_transaction.py
+++ b/gcloud/datastore/test_transaction.py
@@ -112,7 +112,6 @@ def test_context_manager_no_raise(self):
         self.assertEqual(xact.id(), None)
 
     def test_context_manager_w_raise(self):
-        # See https://github.com/GoogleCloudPlatform/gcloud-python/issues/136
         class Foo(Exception):
             pass
         _DATASET = 'DATASET'
@@ -127,13 +126,10 @@ class Foo(Exception):
                 self.assertTrue(connection._xact is xact)
                 raise Foo()
         except Foo:
-            pass  # XXX
-            # self.assertEqual(xact.id(), None)
-            # self.assertEqual(connection._rolled_back, _DATASET))
-            # self.assertEqual(connection._xact, None)
-        # XXX should *not* have committed
-        self.assertEqual(connection._committed, (_DATASET, mutation))
-        # self.assertEqual(connection._committed, None)
+            self.assertEqual(xact.id(), None)
+            self.assertEqual(connection._rolled_back, _DATASET)
+            self.assertEqual(connection._xact, None)
+        self.assertEqual(connection._committed, None)
         self.assertTrue(connection._xact is None)
         self.assertEqual(xact.id(), None)
Repository: googleapis/google-cloud-python

Issue:
Int64 CheckValue in datastore.helpers returns None instead of the value.
We current do "return INT64(val)", however INT64 returns nothing. We need to run this method, and _then_ return the value.

See: https://code.google.com/p/protobuf/source/browse/trunk/python/google/protobuf/internal/type_checkers.py?r=163#100 returns None. We need to _check_ and then return "val".

and

https://github.com/GoogleCloudPlatform/gcloud-python/blob/1779aeccdea1f2557e5e2cf80e448d0b64f48527/gcloud/datastore/helpers.py#L59



Discussion:

Resolving patch:
diff --git a/gcloud/datastore/helpers.py b/gcloud/datastore/helpers.py
--- a/gcloud/datastore/helpers.py
+++ b/gcloud/datastore/helpers.py
@@ -7,7 +7,7 @@
 
 from gcloud.datastore.key import Key
 
-INT64 = Int64ValueChecker().CheckValue
+INT_VALUE_CHECKER = Int64ValueChecker()
 
 
 def get_protobuf_attribute_and_value(val):
@@ -56,7 +56,8 @@ def get_protobuf_attribute_and_value(val):
     elif isinstance(val, float):
         name, value = 'double', val
     elif isinstance(val, (int, long)):
-        name, value = 'integer', INT64(val)
+        INT_VALUE_CHECKER.CheckValue(val)  # This will raise an exception if invalid.
+        name, value = 'integer', long(val)  # Always cast to a long.
     elif isinstance(val, basestring):
         name, value = 'string', val
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -16,7 +16,7 @@
 REQUIREMENTS = [
     'httplib2',
     'oauth2client',
-    'protobuf',
+    'protobuf >= 2.5.0',
     'pycrypto',
     'pyopenssl',
     'pytz',


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Make sure deep copy is correctly performed in datastore.query.Query._clone
From TODO:

``` python
    # TODO(jjg): Double check that this makes sense...
```

See [`datastore.query.Query._clone`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/1f2aaf3606802f4f02eeb30e22e1082ef7f4594e/gcloud/datastore/query.py#L63) for relevant code and #146 for some context.

`git blame` output

```
dec0e7c6 (JJ Geewax 2014-01-28 15:15:40 -0500  63)     # TODO(jjg): Double check that this makes sense...
```

Make sure deep copy is correctly performed in datastore.query.Query._clone
From TODO:

``` python
    # TODO(jjg): Double check that this makes sense...
```

See [`datastore.query.Query._clone`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/1f2aaf3606802f4f02eeb30e22e1082ef7f4594e/gcloud/datastore/query.py#L63) for relevant code and #146 for some context.

`git blame` output

```
dec0e7c6 (JJ Geewax 2014-01-28 15:15:40 -0500  63)     # TODO(jjg): Double check that this makes sense...
```



Discussion:

Resolving patch:
diff --git a/gcloud/datastore/query.py b/gcloud/datastore/query.py
--- a/gcloud/datastore/query.py
+++ b/gcloud/datastore/query.py
@@ -1,7 +1,6 @@
 """Create / interact with gcloud datastore queries."""
 
 import base64
-import copy
 
 from gcloud.datastore import datastore_v1_pb2 as datastore_pb
 from gcloud.datastore import _helpers
@@ -67,8 +66,9 @@ def _clone(self):
         :rtype: :class:`gcloud.datastore.query.Query`
         :returns: a copy of 'self'.
         """
-        clone = copy.deepcopy(self)
-        clone._dataset = self._dataset  # Shallow copy the dataset.
+        clone = self.__class__(dataset=self._dataset)
+        clone._pb.CopyFrom(self._pb)
+        clone._cursor = self._cursor
         return clone
 
     def to_protobuf(self):


Test patch:
diff --git a/gcloud/datastore/test_query.py b/gcloud/datastore/test_query.py
--- a/gcloud/datastore/test_query.py
+++ b/gcloud/datastore/test_query.py
@@ -33,14 +33,17 @@ def test__clone(self):
 
         _DATASET = 'DATASET'
         _KIND = 'KIND'
+        _CURSOR = 'DEADBEEF'
         dataset = Dataset(_DATASET)
         query = self._makeOne(_KIND, dataset)
+        query._cursor = _CURSOR
         clone = query._clone()
         self.assertFalse(clone is query)
         self.assertTrue(isinstance(clone, self._getTargetClass()))
         self.assertTrue(clone.dataset() is dataset)
         kq_pb, = list(clone.kind())
         self.assertEqual(kq_pb.name, _KIND)
+        self.assertEqual(clone._cursor, _CURSOR)
 
     def test_to_protobuf_empty(self):
         query = self._makeOne()
Repository: googleapis/google-cloud-python

Issue:
Support list values in entities
Right now, we support timestamps, keys, booleans, doubles, integers, and strings, but not lists. I think it'd be nice if we could figure out how to support saving lists in a nice way.

Ref: https://cloud.google.com/datastore/docs/apis/v1beta2/entity



Discussion:
See https://github.com/Tagtoo/gcloud-python/commit/0475d5ee2ac53ecd9a711c747916fade6e2d9c02 (accidentally included in PR #126).

Still missing support for marshalling lists _to_ protobufs.


Resolving patch:
diff --git a/gcloud/datastore/_helpers.py b/gcloud/datastore/_helpers.py
--- a/gcloud/datastore/_helpers.py
+++ b/gcloud/datastore/_helpers.py
@@ -66,6 +66,8 @@ def _get_protobuf_attribute_and_value(val):
         name, value = 'string', val
     elif isinstance(val, Entity):
         name, value = 'entity', val
+    elif isinstance(val, list):
+        name, value = 'list', val
     else:
         raise ValueError("Unknown protobuf attr type %s" % type(val))
 
@@ -167,5 +169,10 @@ def _set_protobuf_value(value_pb, val):
             p_pb = e_pb.property.add()
             p_pb.name = item_key
             _set_protobuf_value(p_pb.value, value)
+    elif attr == 'list_value':
+        l_pb = value_pb.list_value
+        for item in val:
+            i_pb = l_pb.add()
+            _set_protobuf_value(i_pb, item)
     else:  # scalar, just assign
         setattr(value_pb, attr, val)


Test patch:
diff --git a/gcloud/datastore/test__helpers.py b/gcloud/datastore/test__helpers.py
--- a/gcloud/datastore/test__helpers.py
+++ b/gcloud/datastore/test__helpers.py
@@ -90,6 +90,12 @@ def test_entity(self):
         self.assertEqual(name, 'entity_value')
         self.assertTrue(value is entity)
 
+    def test_list(self):
+        values = ['a', 0, 3.14]
+        name, value = self._callFUT(values)
+        self.assertEqual(name, 'list_value')
+        self.assertTrue(value is values)
+
     def test_object(self):
         self.assertRaises(ValueError, self._callFUT, object())
 
@@ -301,3 +307,13 @@ def test_entity_w_key(self):
         self.assertEqual(len(props), 1)
         self.assertEqual(props[0].name, 'foo')
         self.assertEqual(props[0].value.string_value, 'Foo')
+
+    def test_list(self):
+        pb = self._makePB()
+        values = ['a', 0, 3.14]
+        self._callFUT(pb, values)
+        marshalled = pb.list_value
+        self.assertEqual(len(marshalled), len(values))
+        self.assertEqual(marshalled[0].string_value, values[0])
+        self.assertEqual(marshalled[1].integer_value, values[1])
+        self.assertEqual(marshalled[2].double_value, values[2])
Repository: googleapis/google-cloud-python

Issue:
The value of `more_results` from storage.Connection.run_query should not be thrown away.
I'm currently writing a regression test that checks for more items after a fetch with a limit set and this boolean would determine if the next fetch (at the final cursor) contains all the remaining results.

As a side note, the Node [regression](https://github.com/GoogleCloudPlatform/gcloud-node/blob/029f3a52e05a3b59bd07f69d7d2032b7a61fe331/regression/datastore.js#L237) needlessly re-queries with a cursor which (should) point to the end of the kind.

/cc @silvolu 



Discussion:
@dhermes  the reason for that is described in [this issue](https://github.com/GoogleCloudPlatform/gcloud-node/issues/5) (basically it doesn't really work as you'd expect).

@silvolu Are you saying that GoogleCloudPlatform/gcloud-node#5 says that `more_results` is not a value that can be consistently trusted? Only @rakyll's issue description mentions the value of `more_results`.

UPDATE: FWIW I tested this on #281 and `more_results` was always equal to `2`, irrespective of the query, which is the enum value of [`MORE_RESULTS_AFTER_LIMIT`](https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/6073e01540264db58ee50c30a1e9f006677eb794/proto/datastore_v1.proto#L354).

The backend keeps returning MORE_RESULTS_AFTER_LIMIT even when there are none.

Cool. I'd like to add this to the code as a comment and a reference to this issue. SG?

SGTM


Resolving patch:
diff --git a/gcloud/datastore/connection.py b/gcloud/datastore/connection.py
--- a/gcloud/datastore/connection.py
+++ b/gcloud/datastore/connection.py
@@ -217,12 +217,14 @@ def run_query(self, dataset_id, query_pb, namespace=None):
         Using the `fetch`` method...
 
         >>> query.fetch()
-        [<list of Entity Protobufs>], cursor, more_results, skipped_results
+        [<list of Entity unmarshalled from protobuf>]
+        >>> query.cursor()
+        <string containing cursor where fetch stopped>
 
         Under the hood this is doing...
 
         >>> connection.run_query('dataset-id', query.to_protobuf())
-        [<list of Entity Protobufs>]
+        [<list of Entity Protobufs>], cursor, more_results, skipped_results
 
         :type dataset_id: string
         :param dataset_id: The ID of the dataset over which to run the query.
diff --git a/gcloud/datastore/query.py b/gcloud/datastore/query.py
--- a/gcloud/datastore/query.py
+++ b/gcloud/datastore/query.py
@@ -333,6 +333,13 @@ def fetch(self, limit=None):
             dataset_id=self.dataset().id(),
             namespace=self._namespace,
             )
+        # NOTE: `query_results` contains two extra values that we don't use,
+        #       namely `more_results` and `skipped_results`. The value of
+        #       `more_results` is unusable because it always returns an enum
+        #       value of MORE_RESULTS_AFTER_LIMIT even if there are no more
+        #       results. See
+        #       https://github.com/GoogleCloudPlatform/gcloud-python/issues/280
+        #       for discussion.
         entity_pbs, end_cursor = query_results[:2]
 
         self._cursor = end_cursor


Test patch:
Repository: googleapis/google-cloud-python

Issue:
storage: Switch to JSON API v1
Google Cloud Storage v1beta2 is deprecated and will be shut down in a couple of months. Please upgrade to v1:

https://developers.google.com/storage/docs/json_api/release_notes#april302014



Discussion:
The migration guide appears to have no issues which affect our use of the API, beyond changing the API base URL:

  https://developers.google.com/storage/docs/json_api/v1/how-tos/migrate

This only occurs twice as of 1f2aaf3606802f4f02eeb30e22e1082ef7f4594e:

``` sh
$ git grep -i v1beta2
gcloud/datastore/connection.py:  API_VERSION = 'v1beta2'
gcloud/storage/connection.py:  API_VERSION = 'v1beta2'
```

Shouldn't we also put this constant in some globals/settings module somewhere? We could even allow people to override it by setting an environment variable or something similar.

Hmm, it is a global really:  the 'API_VERSION' is a class attribute of the Connection class in both cases, and is used only by the 'build_api_url()' method of that class as a default when the caller does not pass an argument for 'api_version'.

I can't imagine that the library will continue to work against arbitrary other versions of the API;  maybe I'm just not imaginative enough. :)

Touche. I suppose allowing folks to tweak it is a bit silly.


Resolving patch:
diff --git a/gcloud/storage/connection.py b/gcloud/storage/connection.py
--- a/gcloud/storage/connection.py
+++ b/gcloud/storage/connection.py
@@ -68,7 +68,7 @@ class Connection(connection.Connection):
       True
     """
 
-    API_VERSION = 'v1beta2'
+    API_VERSION = 'v1'
     """The version of the API, used in building the API call's URL."""
 
     API_URL_TEMPLATE = '{api_base_url}/storage/{api_version}{path}'


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Figure out and documents exceptions thrown in storage.connection.Connection.create_bucket
From TODO:

``` python
    # TODO: Which exceptions will this raise?
```

See [`storage.connection.Connection.create_bucket`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/1f2aaf3606802f4f02eeb30e22e1082ef7f4594e/gcloud/storage/connection.py#L320) for relevant code and #146 for some context.

`git blame` output

```
b7f1b774 (JJ Geewax   2014-01-29 18:02:29 -0500 320)     # TODO: Which exceptions will this raise?
```

Figure out and documents exceptions thrown in storage.connection.Connection.create_bucket
From TODO:

``` python
    # TODO: Which exceptions will this raise?
```

See [`storage.connection.Connection.create_bucket`](https://github.com/GoogleCloudPlatform/gcloud-python/blob/1f2aaf3606802f4f02eeb30e22e1082ef7f4594e/gcloud/storage/connection.py#L320) for relevant code and #146 for some context.

`git blame` output

```
b7f1b774 (JJ Geewax   2014-01-29 18:02:29 -0500 320)     # TODO: Which exceptions will this raise?
```



Discussion:
Same question asked in #159 about documenting exceptions where: should we document exceptions that are not directly raised by the method?

Same question asked in #159 about documenting exceptions where: should we document exceptions that are not directly raised by the method?


Resolving patch:
diff --git a/gcloud/storage/connection.py b/gcloud/storage/connection.py
--- a/gcloud/storage/connection.py
+++ b/gcloud/storage/connection.py
@@ -338,6 +338,8 @@ def create_bucket(self, bucket):
 
         :rtype: :class:`gcloud.storage.bucket.Bucket`
         :returns: The newly created bucket.
+        :raises: :class:`gcloud.storage.exceptions.ConnectionError` if
+                 there is a confict (bucket already exists, invalid name, etc.)
         """
         bucket = self.new_bucket(bucket)
         response = self.api_request(method='POST', path='/b',


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Document "text" vs. "bytes" values for Entities
Docs need to explain clearly how users can ensure that the values they set onto entities get mapped as they expect:  "text" values should be mapped to/from 'string_value' entries on the protobufs, and "bytes" values should be mapped to/from 'blob_value' entries.

We are to have to educate folks that the native 'str' is "bytes" on Python2, but "text" on Python3, and mandate that they be explicit if they want to control how such values get stored.

Document "text" vs. "bytes" values for Entities
Docs need to explain clearly how users can ensure that the values they set onto entities get mapped as they expect:  "text" values should be mapped to/from 'string_value' entries on the protobufs, and "bytes" values should be mapped to/from 'blob_value' entries.

We are to have to educate folks that the native 'str' is "bytes" on Python2, but "text" on Python3, and mandate that they be explicit if they want to control how such values get stored.



Discussion:
It may be helpful to explain to Python2 users that they need to encode their strings with unicode in order to map it correctly. I understood that ""text" values should be mapped to/from 'string_value' entries on the protobufs" but it took me a bit of time to figure out HOW to do that.

It may be helpful to explain to Python2 users that they need to encode their strings with unicode in order to map it correctly. I understood that ""text" values should be mapped to/from 'string_value' entries on the protobufs" but it took me a bit of time to figure out HOW to do that.


Resolving patch:
diff --git a/gcloud/datastore/entity.py b/gcloud/datastore/entity.py
--- a/gcloud/datastore/entity.py
+++ b/gcloud/datastore/entity.py
@@ -68,6 +68,16 @@ class Entity(dict):
     >>> dict(entity)
     {'age': 20, 'name': 'JJ'}
 
+    .. note::
+
+       When saving an entity to the backend, values which are "text"
+       ('unicode' in Python2, 'str' in Python3) will be saved using
+       the 'text_value' field, after being encoded to UTF-8.  When
+       retrieved from the back-end, such values will be decoded to "text"
+       again.  Values which are "bytes" ('str' in Python2, 'bytes' in
+       Python3), will be saved using the 'blob_value' field, without
+       any decoding / encoding step.
+
     """
 
     def __init__(self, dataset=None, kind=None):


Test patch:
Repository: googleapis/google-cloud-python

Issue:
asn1 error in reading credentials on windows
Hi,

I reckon line 39 in glcoud.credentials should open the private key with read binary on windows:
        private_key=open(private_key_path, 'rb').read(),

cheers,
David



Discussion:

Resolving patch:
diff --git a/gcloud/credentials.py b/gcloud/credentials.py
--- a/gcloud/credentials.py
+++ b/gcloud/credentials.py
@@ -34,5 +34,5 @@ def get_for_service_account(client_email, private_key_path, scope=None):
     """
     return client.SignedJwtAssertionCredentials(
         service_account_name=client_email,
-        private_key=open(private_key_path).read(),
+        private_key=open(private_key_path, 'rb').read(),
         scope=scope)


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Key name w/ slash quoted twice during upload
The fix for #354 in #364 mistakenly quotes the `Key.name` when adding it the the `query_params` for the upload API call, but those params are already going to be quoted inside `Connection.make_api_request` (by way of `urllib.urlencode`).



Discussion:

Resolving patch:
diff --git a/gcloud/storage/key.py b/gcloud/storage/key.py
--- a/gcloud/storage/key.py
+++ b/gcloud/storage/key.py
@@ -285,7 +285,7 @@ def upload_from_file(self, file_obj, rewind=False, size=None,
 
         query_params = {
             'uploadType': 'resumable',
-            'name': urllib.quote_plus(self.name),
+            'name': self.name,
         }
 
         upload_url = self.connection.build_api_url(


Test patch:
diff --git a/gcloud/storage/test_key.py b/gcloud/storage/test_key.py
--- a/gcloud/storage/test_key.py
+++ b/gcloud/storage/test_key.py
@@ -298,7 +298,7 @@ def test_upload_from_file_w_slash_in_name(self):
         self.assertEqual(netloc, 'example.com')
         self.assertEqual(path, '/b/name/o')
         self.assertEqual(dict(parse_qsl(qs)),
-                         {'uploadType': 'resumable', 'name': 'parent%2Fchild'})
+                         {'uploadType': 'resumable', 'name': 'parent/child'})
         self.assertEqual(rq[0]['headers'],
                          {'X-Upload-Content-Length': 6,
                           'X-Upload-Content-Type': 'application/unknown'})
Repository: googleapis/google-cloud-python

Issue:
Add long_description to setup.py
See #434 for context.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,14 +1,14 @@
-import sys
+import os
 
 
-try:
-    from setuptools import setup, find_packages
-except ImportError:
-    from distutils.core import setup, find_packages
+from setuptools import setup
+from setuptools import find_packages
 
+here = os.path.abspath(os.path.dirname(__file__))
 
-if sys.version_info <= (2, 5):
-    raise Exception('Requires Python Version 2.6 or above... exiting.')
+
+with open(os.path.join(here, 'README.rst')) as f:
+    README = f.read()
 
 
 REQUIREMENTS = [
@@ -27,6 +27,7 @@
     description='API Client library for Google Cloud',
     author='JJ Geewax',
     author_email='jj@geewax.org',
+    long_description=README,
     scripts=[],
     url='https://github.com/GoogleCloudPlatform/gcloud-python',
     packages=find_packages(),


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Build in regression test case retries
Add in one retry for network failures to reduce (eliminate?) flaky / intermittent failures.

For example
https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/46346592
after #509 failed on code that didn't even change in the PR.



Discussion:
See https://github.com/dhermes/gcloud-python/tree/retry-flaky-regression-cases, will submit PR

Another regression failure:
https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/46219691

This one is actually a test failure (`F`) instead of an error (`E`) caused by a network 404 as above.


Resolving patch:
diff --git a/regression/storage.py b/regression/storage.py
--- a/regression/storage.py
+++ b/regression/storage.py
@@ -39,10 +39,20 @@ def setUpModule():
         SHARED_BUCKETS['test_bucket'] = connection.create_bucket(bucket_name)
 
 
+def safe_delete(bucket):
+    for key in bucket:
+        try:
+            key.delete()
+        except storage.exceptions.NotFound:
+            print('Delete failed with 404: %r' % (key,))
+
+    # Passing force=False does not try to delete the contained files.
+    bucket.delete(force=False)
+
+
 def tearDownModule():
     for bucket in SHARED_BUCKETS.values():
-        # Passing force=True also deletes all files.
-        bucket.delete(force=True)
+        safe_delete(bucket)
 
 
 class TestStorage(unittest2.TestCase):


Test patch: