internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: googleapis/google-cloud-python

Issue:
Google cloud storage needs requests >= 2.18.0 but the setup.py only requires >= 2.4.0
According to the requests [changelog](http://docs.python-requests.org/en/latest/community/updates/) the Response class was turned into a context manager only in version 2.18.0. However in google/resumable_media/requests/download.py line 62 (inside _write_to_stream) the code tries to use "response" as a context manager. Some digging reveals that this response makes its way from Client._http which is defined in google/cloud/client.py and gets the Response object from google/auth/transport/requests.py:AuthorizedSession.

So... it definitely feels like something in here needs to require requests >= 2.18.0 (maybe all of them?). 

Thoughts?

Detailed info:

1. OS type and version: N/A
2. Python version and virtual environment information: 2.7 but I don't think it matters.
3. google-cloud-python version: google-cloud-storage v 1.3.1
4. Stacktrace if available: below.
5. Steps to reproduce: Install a version of the requests library _less than 2.18.0_. Get some blob that exists. Try blob.download_as_string(). I personally re-pro'd this with version 2.10.0 to check. Note that the setup.py only requires 2.0.0 for storage and 2.4.0 for client, etc...

-- PG.


Discussion:

Resolving patch:
diff --git a/bigquery/setup.py b/bigquery/setup.py
--- a/bigquery/setup.py
+++ b/bigquery/setup.py
@@ -54,7 +54,7 @@
     'google-cloud-core >= 0.26.0, < 0.27dev',
     'google-auth >= 1.0.0',
     'google-resumable-media >= 0.2.1',
-    'requests >= 2.0.0',
+    'requests >= 2.18.0',
 ]
 
 setup(
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -54,7 +54,7 @@
     'googleapis-common-protos >= 1.3.4',
     'protobuf >= 3.0.0',
     'google-auth >= 0.4.0, < 2.0.0dev',
-    'requests >= 2.4.0, < 3.0.0dev',
+    'requests >= 2.18.0, < 3.0.0dev',
     'setuptools >= 34.0.0',
     'six',
     'tenacity >= 4.0.0, <5.0.0dev'
diff --git a/storage/setup.py b/storage/setup.py
--- a/storage/setup.py
+++ b/storage/setup.py
@@ -54,7 +54,7 @@
     'google-cloud-core >= 0.26.0, < 0.27dev',
     'google-auth >= 1.0.0',
     'google-resumable-media >= 0.2.3',
-    'requests >= 2.0.0',
+    'requests >= 2.18.0',
 ]
 
 setup(


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Unhelpful documentation for Table.upload_from_file.
The [docs](https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-table.html#google.cloud.bigquery.table.Table.upload_from_file) for `Table.upload_from_file` are not very helpful. 

Non of the actual default values are shown. They all show "foo=None". Also the docs indicate that "source_format" can only be JSON or CSV, but if you check the entry for "configuration.load.sourceFormat" [here.](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs) You will see that AVRO is also accepted.


Discussion:

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/table.py b/bigquery/google/cloud/bigquery/table.py
--- a/bigquery/google/cloud/bigquery/table.py
+++ b/bigquery/google/cloud/bigquery/table.py
@@ -1045,9 +1045,10 @@ def upload_from_file(self,
         :param file_obj: A file handle opened in binary mode for reading.
 
         :type source_format: str
-        :param source_format: one of 'CSV' or 'NEWLINE_DELIMITED_JSON'.
-                              job configuration option; see
-                              :meth:`google.cloud.bigquery.job.LoadJob`
+        :param source_format: Any supported format. The full list of supported
+            formats is documented under the
+            ``configuration.extract.destinationFormat`` property on this page:
+            https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
 
         :type rewind: bool
         :param rewind: If True, seek to the beginning of the file handle before


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Incorrect link in documentation
There is an incorrect link at https://googlecloudplatform.github.io/google-cloud-python/latest/bigquery/dataset.html#google.cloud.bigquery.dataset.Dataset.delete.  It links to https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/delete, but it should link to https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/delete.


Discussion:

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/dataset.py b/bigquery/google/cloud/bigquery/dataset.py
--- a/bigquery/google/cloud/bigquery/dataset.py
+++ b/bigquery/google/cloud/bigquery/dataset.py
@@ -536,7 +536,7 @@ def delete(self, client=None):
         """API call:  delete the dataset via a DELETE request.
 
         See
-        https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/delete
+        https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/delete
 
         :type client: :class:`~google.cloud.bigquery.client.Client` or
                       ``NoneType``


Test patch:
Repository: googleapis/google-cloud-python

Issue:
[runtimeconfig] typo: "config.get_variable(...)"
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/bee32c0ae68b1e6cf224f9017c758a8e31e8be83/runtimeconfig/google/cloud/runtimeconfig/config.py#L190
[runtimeconfig] Should be "client.config(...)" not "client.get_config(...)
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/bee32c0ae68b1e6cf224f9017c758a8e31e8be83/runtimeconfig/google/cloud/runtimeconfig/config.py#L189


Discussion:

Thanks, the [client class](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/bee32c0ae68b1e6cf224f9017c758a8e31e8be83/runtimeconfig/google/cloud/runtimeconfig/client.py#L56) does indeed have a `config` method (rather than `get_config`).

In addition, that example has a typo, `get_varialbe`.

Resolving patch:
diff --git a/runtimeconfig/google/cloud/runtimeconfig/config.py b/runtimeconfig/google/cloud/runtimeconfig/config.py
--- a/runtimeconfig/google/cloud/runtimeconfig/config.py
+++ b/runtimeconfig/google/cloud/runtimeconfig/config.py
@@ -186,8 +186,8 @@ def get_variable(self, variable_name, client=None):
 
           >>> from google.cloud import runtimeconfig
           >>> client = runtimeconfig.Client()
-          >>> config = client.get_config('my-config')
-          >>> print(config.get_varialbe('variable-name'))
+          >>> config = client.config('my-config')
+          >>> print(config.get_variable('variable-name'))
           <Variable: my-config, variable-name>
           >>> print(config.get_variable('does-not-exist'))
           None


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Spelling: s/obselete/obsolete
`core/google/cloud/obselete.py`


Discussion:

Resolving patch:
diff --git a/core/google/cloud/obselete.py b/core/google/cloud/obsolete.py
similarity index 91%
rename from core/google/cloud/obselete.py
rename to core/google/cloud/obsolete.py
--- a/core/google/cloud/obselete.py
+++ b/core/google/cloud/obsolete.py
@@ -26,12 +26,12 @@ def complain(distribution_name):
     rather than just send a warning.
 
     Args:
-        distribution_name (str): The name of the obselete distribution.
+        distribution_name (str): The name of the obsolete distribution.
     """
     try:
         pkg_resources.get_distribution(distribution_name)
         warnings.warn(
-            'The {pkg} distribution is now obselete. '
+            'The {pkg} distribution is now obsolete. '
             'Please `pip uninstall {pkg}`. '
             'In the future, this warning will become an ImportError.'.format(
                 pkg=distribution_name,


Test patch:
diff --git a/core/tests/unit/test_obselete.py b/core/tests/unit/test_obsolete.py
similarity index 87%
rename from core/tests/unit/test_obselete.py
rename to core/tests/unit/test_obsolete.py
--- a/core/tests/unit/test_obselete.py
+++ b/core/tests/unit/test_obsolete.py
@@ -16,16 +16,16 @@
 
 import mock
 
-from google.cloud import obselete
+from google.cloud import obsolete
 
 
 def test_complain_noop():
     with mock.patch.object(warnings, 'warn', autospec=True) as warn:
-        obselete.complain('bogus_package')
+        obsolete.complain('bogus_package')
         assert warn.call_count == 0
 
 
 def test_complain():
     with mock.patch.object(warnings, 'warn', autospec=True) as warn:
-        obselete.complain('google-cloud-core')
+        obsolete.complain('google-cloud-core')
         warn.assert_called_once_with(mock.ANY, DeprecationWarning)
Repository: googleapis/google-cloud-python

Issue:
Support a query over an external data source / federated data source
Bigquery supports running queries over csv files, but I don't see this option in this bigquery api. 

See https://cloud.google.com/bigquery/external-data-sources
and
https://cloud.google.com/bigquery/external-data-cloud-storage

I see example for reading csv files into a table, but I would rather not have to manage creating and deleting temporary tables.


Discussion:
AFAIK this **is** supported out of the box, but I'm not sure how.

@tswast Do you have any docs to this effect?
No samples yet for this. It's on my todo list.
@tswast Is there anything you'd like to see from us, e.g. a `Client.from_foo()` factory constructor that allows specifying scopes for the federated sources? (Or is that necessary?)
Yeah, it is necessary, thus: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3067 The Google Cloud Platform scope covers Cloud Storage, but not Drive or Sheets. (Not sure about Bigtable)

Regarding  those `Client.from_foo()` functions, I'd want us to make sure we pass through all the supported arguments in the google-auth library.

As far is the external data source functionality goes, the Python client libs are missing a bit it seems.

Looks like [QueryJob](https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-job.html#google.cloud.bigquery.job.QueryJob) is missing the [tableDefinitions parameter](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions).

It also looks like [Table](https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-table.html) is missing [external data configuration](https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration).

You can create an external data table in the UI and then query it fine, though. Once the external table definition is defined you can query it like any other table.
I cannot use the UI. I'm looking to replace Google's Datalab python/bigquery api with this one.
Hey all,
While triaging today, I deteremined that we have two issues for this, the other being #2372. I am closing this one in favor of that one.
@lukesneeringer Agree. Both this one and #2372 are BigQuery specific. We should probably have a more generic: "non-default scopes?" issue

Resolving patch:
diff --git a/spanner/setup.py b/spanner/setup.py
--- a/spanner/setup.py
+++ b/spanner/setup.py
@@ -52,6 +52,7 @@
 
 REQUIREMENTS = [
     'google-cloud-core >= 0.27.0, < 0.28dev',
+    'google-auth >= 1.1.0',
     'grpcio >= 1.2.0, < 2.0dev',
     'gapic-google-cloud-spanner-v1 >= 0.15.0, < 0.16dev',
     'gapic-google-cloud-spanner-admin-database-v1 >= 0.15.0, < 0.16dev',


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Spelling Error (_medical_likeliehood)
_medical_likeliehood -> _medical_likelihood

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/b28a4eb667ae08c3f4dcf9af891ed4931884989c/vision/google/cloud/vision/safe_search.py#L43


Discussion:
@jnhasty Thank you for the report!

Resolving patch:
diff --git a/vision/google/cloud/vision/safe_search.py b/vision/google/cloud/vision/safe_search.py
--- a/vision/google/cloud/vision/safe_search.py
+++ b/vision/google/cloud/vision/safe_search.py
@@ -40,7 +40,7 @@ def __init__(self, adult_likelihood, spoof_likelihood, medical_likelihood,
                  violence_likelihood):
         self._adult_likelihood = adult_likelihood
         self._spoof_likelihood = spoof_likelihood
-        self._medical_likeliehood = medical_likelihood
+        self._medical_likelihood = medical_likelihood
         self._violence_likelihood = violence_likelihood
 
     @classmethod
@@ -104,7 +104,7 @@ def medical(self):
         :rtype: :class:`~google.cloud.vision.likelihood.Likelihood`
         :returns: The ``Likelihood`` that the image is medical in origin.
         """
-        return self._medical_likeliehood
+        return self._medical_likelihood
 
     @property
     def violence(self):


Test patch:
Repository: googleapis/google-cloud-python

Issue:
Spanner: IndexError: pop from empty list when merge_array
```
return merger(lhs, rhs, type_)
  File ".../google/cloud/spanner_v1/streamed.py", line 259, in _merge_array
    first = rhs.pop(0)
IndexError: pop from empty list
```


Discussion:

Resolving patch:
diff --git a/spanner/google/cloud/spanner_v1/streamed.py b/spanner/google/cloud/spanner_v1/streamed.py
--- a/spanner/google/cloud/spanner_v1/streamed.py
+++ b/spanner/google/cloud/spanner_v1/streamed.py
@@ -256,6 +256,12 @@ def _merge_array(lhs, rhs, type_):
         lhs.list_value.values.extend(rhs.list_value.values)
         return lhs
     lhs, rhs = list(lhs.list_value.values), list(rhs.list_value.values)
+
+    # Sanity check: If either list is empty, short-circuit.
+    # This is effectively a no-op.
+    if not len(lhs) or not len(rhs):
+        return Value(list_value=ListValue(values=(lhs + rhs)))
+
     first = rhs.pop(0)
     if first.HasField('null_value'):  # can't merge
         lhs.append(first)


Test patch:
diff --git a/spanner/tests/unit/test_streamed.py b/spanner/tests/unit/test_streamed.py
--- a/spanner/tests/unit/test_streamed.py
+++ b/spanner/tests/unit/test_streamed.py
@@ -314,6 +314,22 @@ def test__merge_chunk_array_of_float(self):
         self.assertEqual(merged, expected)
         self.assertIsNone(streamed._pending_chunk)
 
+    def test__merge_chunk_array_of_string_with_empty(self):
+        iterator = _MockCancellableIterator()
+        streamed = self._make_one(iterator)
+        FIELDS = [
+            self._make_array_field('name', element_type_code='STRING'),
+        ]
+        streamed._metadata = self._make_result_set_metadata(FIELDS)
+        streamed._pending_chunk = self._make_list_value([u'A', u'B', u'C'])
+        chunk = self._make_list_value([])
+
+        merged = streamed._merge_chunk(chunk)
+
+        expected = self._make_list_value([u'A', u'B', u'C'])
+        self.assertEqual(merged, expected)
+        self.assertIsNone(streamed._pending_chunk)
+
     def test__merge_chunk_array_of_string(self):
         iterator = _MockCancellableIterator()
         streamed = self._make_one(iterator)
Repository: googleapis/google-cloud-python

Issue:
PubSub: Default thread-based subscriber Policy's on_response() implementation degenerates into series of synchronous calls because of debug statement
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/4e17158c0f14e0e91cbeb74c3e8483c5a231e9ac/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py#L147

While testing out the new PubSub API (v 0.28.3), I noticed that the default thread-based `Policy`, which uses a `ThreadPoolExecutor(max_workers=10)` seemed to be executing the subscriber callbacks sequentially, instead of asynchronously.  Upon introspection of the source, it looks like the above referenced code:
```
...
future = self._executor.submit(self._callback, message)
logger.debug('Result: %s' % future.result())
```
will block until `self._callback` completes, because of the call to `future.result()`.  Or am I losing my mind?  Thanks!


Discussion:
You're totally right. That should be done via add_done_callback instead.

On Wed, Oct 11, 2017, 10:21 PM Jeff Payne <notifications@github.com> wrote:

>
> https://github.com/GoogleCloudPlatform/google-cloud-python/blob/4e17158c0f14e0e91cbeb74c3e8483c5a231e9ac/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py#L147
>
> While testing out the new PubSub API (v 0.28.3), I noticed that the
> default thread-based Policy, which uses a
> ThreadPoolExecutor(max_workers=10) seemed to be executing the subscriber
> callbacks sequentially, instead of asynchronously. Upon introspection of
> the source, it looks like the above referenced code:
>
> ...
> future = self._executor.submit(self._callback, message)
> logger.debug('Result: %s' % future.result())
>
> will block until self._callback completes, because of the call to
> future.result(). Or am I losing my mind? Thanks!
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4167>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAPUcyH9glRDri8lfWWL-oFringJqy4Xks5sraHRgaJpZM4P2exY>
> .
>


Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py b/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
@@ -30,6 +30,11 @@
 logger = logging.getLogger(__name__)
 
 
+def _callback_completed(future):
+    """Simple callback that just logs a `Future`'s result."""
+    logger.debug('Result: %s', future.result())
+
+
 class Policy(base.BasePolicy):
     """A consumer class based on :class:`threading.Thread`.
 
@@ -144,4 +149,4 @@ def on_response(self, response):
             logger.debug(self._callback)
             message = Message(msg.message, msg.ack_id, self._request_queue)
             future = self._executor.submit(self._callback, message)
-            logger.debug('Result: %s' % future.result())
+            future.add_done_callback(_callback_completed)


Test patch:
Repository: googleapis/google-cloud-python

Issue:
GCS test failures for 'headers'
Was: https://github.com/GoogleCloudPlatform/google-resumable-media-python/issues/41


Discussion:

Resolving patch:
diff --git a/storage/setup.py b/storage/setup.py
--- a/storage/setup.py
+++ b/storage/setup.py
@@ -53,7 +53,7 @@
 REQUIREMENTS = [
     'google-cloud-core >= 0.27.0, < 0.28dev',
     'google-auth >= 1.0.0',
-    'google-resumable-media >= 0.3.0',
+    'google-resumable-media >= 0.3.1',
     'requests >= 2.18.0',
 ]
 


Test patch:
diff --git a/storage/tests/unit/test_blob.py b/storage/tests/unit/test_blob.py
--- a/storage/tests/unit/test_blob.py
+++ b/storage/tests/unit/test_blob.py
@@ -441,7 +441,9 @@ def _mock_requests_response(
         response.status_code = status_code
         response.headers.update(headers)
         if stream:
-            response.raw = io.BytesIO(content)
+            raw = io.BytesIO(content)
+            raw.headers = headers
+            response.raw = raw
             response._content = False
         else:
             response.raw = None
@@ -677,15 +679,18 @@ def test_download_to_filename_corrupted(self):
         empty_hash = base64.b64encode(
             hashlib.md5(b'').digest()).decode(u'utf-8')
         headers = {'x-goog-hash': 'md5=' + empty_hash}
+        mock_raw = mock.Mock(headers=headers, spec=['headers'])
         response = mock.MagicMock(
             headers=headers,
             status_code=http_client.OK,
+            raw=mock_raw,
             spec=[
                 '__enter__',
                 '__exit__',
                 'headers',
                 'iter_content',
                 'status_code',
+                'raw',
             ],
         )
         # i.e. context manager returns ``self``.
Repository: googleapis/google-cloud-python

Issue:
Logging: grace_period & batch_size needs to get passed to _Worker in BackgroundThreadTransport
grace_period & batch_size needs to get passed to _Worker:
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/6c602823e7101b40dfe5bfc614da633beb8accfb/logging/google/cloud/logging/handlers/transports/background_thread.py#L258

Ran into this issue because we're using BackgroundThreadTransport and I'm trying to override the default grace period on shutdown.


Discussion:

Resolving patch:
diff --git a/logging/google/cloud/logging/handlers/transports/background_thread.py b/logging/google/cloud/logging/handlers/transports/background_thread.py
--- a/logging/google/cloud/logging/handlers/transports/background_thread.py
+++ b/logging/google/cloud/logging/handlers/transports/background_thread.py
@@ -255,7 +255,9 @@ def __init__(self, client, name, grace_period=_DEFAULT_GRACE_PERIOD,
                  batch_size=_DEFAULT_MAX_BATCH_SIZE):
         self.client = client
         logger = self.client.logger(name)
-        self.worker = _Worker(logger)
+        self.worker = _Worker(logger,
+                              grace_period=grace_period,
+                              max_batch_size=batch_size)
         self.worker.start()
 
     def send(self, record, message, resource=None, labels=None):


Test patch:
diff --git a/logging/tests/unit/handlers/transports/test_background_thread.py b/logging/tests/unit/handlers/transports/test_background_thread.py
--- a/logging/tests/unit/handlers/transports/test_background_thread.py
+++ b/logging/tests/unit/handlers/transports/test_background_thread.py
@@ -76,6 +76,22 @@ def test_flush(self):
 
         transport.worker.flush.assert_called()
 
+    def test_worker(self):
+        client = _Client(self.PROJECT)
+        name = 'python_logger'        
+        batch_size = 30
+        grace_period = 20.
+        transport, worker = self._make_one(client,
+                                           name,
+                                           grace_period=grace_period,
+                                           batch_size=batch_size)
+        worker_grace_period = worker.call_args[1]['grace_period']  # **kwargs.
+        worker_batch_size = worker.call_args[1]['max_batch_size']
+        self.assertEqual(worker_grace_period,
+                         grace_period)
+        self.assertEqual(worker_batch_size,
+                         batch_size)
+
 
 class Test_Worker(unittest.TestCase):
     NAME = 'python_logger'