Repository: googleapis/google-cloud-python

Issue:
BigQuery: Add BadGateway to DEFAULT_RETRY predicate
OS: Linux dc32b7e8763a 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux
Python version: Python 2.7.6
google-cloud-bigquery: 1.5.0

I'm getting BadGateway exceptions periodically when running queries with the default retry object. Here's an example (relevant part of the stack trace):

```
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 2605, in done
    location=self.location)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 579, in _get_query_results
    retry, method='GET', path=path, query_params=extra_params)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 331, in _call_api
    return call()
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
BadGateway: 502 GET https://www.googleapis.com/bigquery/v2/projects/packard-campbell-synth/queries/c4af3c68-d204-455e-877f-9f81e1c77d68?maxResults=0: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
```

It would be useful to add this to the DEFAULT_RETRY object in https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/retry.py.


Discussion:
I've patched the library at commit f3e00adb4abec400ecc51636cd2c5632b44af5ea, and we're still getting BadGateways.

I think it's because the BadGateway is raised before the job is done, and therefore looking at the job's errors isn't going to show BadGateway. So I believe you'll need to catch BadGateway using something like the if_exception_type method (https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/api_core/google/api_core/retry.py#L78).
@bencaine1 Thanks for the followup.  The issue is that the `BadGateway` you report is being raised from `Client._get_query_results`, which doesn't take a `retry` argument.
Oops, `Client._get_query_results` **does** [take a `retry` argument](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/64caa5f0a319313ccc5dbe515ae91c4029548447/bigquery/google/cloud/bigquery/client.py#L538), and `QueryJob.result` [passes it](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/64caa5f0a319313ccc5dbe515ae91c4029548447/bigquery/google/cloud/bigquery/job.py#L2647).
@bencaine1 Can you manage to trap the code when the `BadGateway` is raised, and print out the full response body?

@tswast Any clues from your side how we should be handling this?
It could be that `errors` or `reason` isn't getting populated? I think 502 comes from the layer before it actually hit's BigQuery, so maybe the BigQuery team's advice to look at the body rather than error codes shouldn't apply for 502?
CC @shollyman 
From the error message, it looks like the body is HTML, not JSON, so probably the `reason` field isn't populated.

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/64caa5f0a319313ccc5dbe515ae91c4029548447/api_core/google/api_core/exceptions.py#L362-L388
Yes, this is likely a GFE response, which means we're not going to get a structured error back.

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/retry.py b/bigquery/google/cloud/bigquery/retry.py
--- a/bigquery/google/cloud/bigquery/retry.py
+++ b/bigquery/google/cloud/bigquery/retry.py
@@ -13,16 +13,23 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from google.api_core import exceptions
 from google.api_core import retry
 
 
 _RETRYABLE_REASONS = frozenset([
-    'backendError',
     'rateLimitExceeded',
+    'backendError',
     'internalError',
     'badGateway',
 ])
 
+_UNSTRUCTURED_RETRYABLE_TYPES = (
+    exceptions.TooManyRequests,
+    exceptions.InternalServerError,
+    exceptions.BadGateway,
+)
+
 
 def _should_retry(exc):
     """Predicate for determining when to retry.
@@ -32,8 +39,11 @@ def _should_retry(exc):
     """
     if not hasattr(exc, 'errors'):
         return False
+
     if len(exc.errors) == 0:
-        return False
+        # Check for unstructured error returns, e.g. from GFE
+        return isinstance(exc, _UNSTRUCTURED_RETRYABLE_TYPES)
+
     reason = exc.errors[0]['reason']
     return reason in _RETRYABLE_REASONS
 


Test patch:
diff --git a/bigquery/tests/unit/test_retry.py b/bigquery/tests/unit/test_retry.py
--- a/bigquery/tests/unit/test_retry.py
+++ b/bigquery/tests/unit/test_retry.py
@@ -47,12 +47,30 @@ def test_w_rateLimitExceeded(self):
             errors=[{'reason': 'rateLimitExceeded'}], spec=['errors'])
         self.assertTrue(self._call_fut(exc))
 
+    def test_w_unstructured_too_many_requests(self):
+        from google.api_core.exceptions import TooManyRequests
+
+        exc = TooManyRequests('testing')
+        self.assertTrue(self._call_fut(exc))
+
     def test_w_internalError(self):
         exc = mock.Mock(
             errors=[{'reason': 'internalError'}], spec=['errors'])
         self.assertTrue(self._call_fut(exc))
 
+    def test_w_unstructured_internal_server_error(self):
+        from google.api_core.exceptions import InternalServerError
+
+        exc = InternalServerError('testing')
+        self.assertTrue(self._call_fut(exc))
+
     def test_w_badGateway(self):
         exc = mock.Mock(
             errors=[{'reason': 'badGateway'}], spec=['errors'])
         self.assertTrue(self._call_fut(exc))
+
+    def test_w_unstructured_bad_gateway(self):
+        from google.api_core.exceptions import BadGateway
+
+        exc = BadGateway('testing')
+        self.assertTrue(self._call_fut(exc))