Repository: googleapis/google-cloud-python

Issue:
Core: add retry on 503 errors.
The google-api-python-client and gcloud CLI  both retry on 503 SERVICE UNAVAILABLE errors yet the current version of google-cloud-python doesn't. 

https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L94


We seem to have coded a retry on 500 and 429 , but not 503.  

```python
if_transient_error = if_exception_type((
    exceptions.InternalServerError,
    exceptions.TooManyRequests))
```

Please consider adding 503 as retryable. 

I'm running 2.7.11 but I expect same behavior with 3

```python
503 The service is currently unavailable.
  File "./create_policies.py", line 48, in <module>
    <class 'google.api_core.exceptions.ServiceUnavailable'>
print('Created ' +  future.result().name)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462, in result
    return self.__get_result()
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "./create_policies.py", line 11, in create_policy
    policy = alert_client.create_alert_policy(project, policy)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/cloud/monitoring_v3/gapic/alert_policy_service_client.py", line 399, in create_alert_policy
    request, retry=retry, timeout=timeout, metadata=metadata)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 263, in retry_wrapped_func
    on_error=on_error,
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 178, in retry_target
    return target()
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/six.py", line 737, in raise_from
    raise value
google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.
```

```
---
Metadata-Version: 2.0
Name: google-cloud
Version: 0.22.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: jjg+google-cloud-python@google.com
License: Apache 2.0
Location: /home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages
Requires: google-cloud-resource-manager, google-cloud-monitoring, google-cloud-happybase, google-cloud-language, google-cloud-storage, google-cloud-translate, google-cloud-error-reporting, google-cloud-vision, google-cloud-runtimeconfig, google-cloud-core, google-cloud-bigtable, google-cloud-bigquery, google-cloud-pubsub, google-cloud-datastore, google-cloud-logging, google-cloud-dns
```


Discussion:
@bmenasha Thanks for the report!  @theacodes retrying 503 responses seems reasonable to me.

One quick note:  the `google-cloud` umbrella package is deprecated, and will stop including any of the API-specific packages as dependencies in its next release (about three weeks from now).  After that point, you will need to install the API-specific packages directly.
A storage copy operation fenced by Retry just failed for me by 503, too. Also, the `polling.RETRY_PREDICATE` contains `BadGateway` HTTP 502 exception.

Please, fix this simple issue, add `ServiceUnavailable` and `BadGateway` to `if_transient_error` and uniry it with `polling.RETRY_PREDICATE`.


Ugly hack until the issue is solved:
```py
orig_if_transient_error = retry.if_transient_error
retry.if_transient_error = lambda ex: orig_if_transient_error(ex) \
                                      or isinstance(ex, (exceptions.ServiceUnavailable, exceptions.BadGateway))
retry.Retry.__init__ = functools.partialmethod(retry.Retry.__init__, predicate = retry.if_transient_error)
```


Resolving patch:
diff --git a/api_core/google/api_core/retry.py b/api_core/google/api_core/retry.py
--- a/api_core/google/api_core/retry.py
+++ b/api_core/google/api_core/retry.py
@@ -98,7 +98,9 @@ def if_exception_type_predicate(exception):
 # Pylint sees this as a constant, but it is also an alias that should be
 # considered a function.
 if_transient_error = if_exception_type(
-    (exceptions.InternalServerError, exceptions.TooManyRequests)
+    exceptions.InternalServerError,
+    exceptions.TooManyRequests,
+    exceptions.ServiceUnavailable,
 )
 """A predicate that checks if an exception is a transient API error.
 
@@ -107,6 +109,7 @@ def if_exception_type_predicate(exception):
 - :class:`google.api_core.exceptions.InternalServerError` - HTTP 500, gRPC
     ``INTERNAL(13)`` and its subclasses.
 - :class:`google.api_core.exceptions.TooManyRequests` - HTTP 429
+- :class:`google.api_core.exceptions.ServiceUnavailable` - HTTP 503
 - :class:`google.api_core.exceptions.ResourceExhausted` - gRPC
     ``RESOURCE_EXHAUSTED(8)``
 """
diff --git a/api_core/setup.py b/api_core/setup.py
--- a/api_core/setup.py
+++ b/api_core/setup.py
@@ -39,7 +39,7 @@
     'futures >= 3.2.0; python_version < "3.2"',
 ]
 extras = {
-    "grpc": "grpcio >= 1.8.2",
+    "grpc": "grpcio >= 1.8.2, < 2.0dev",
     "grpcgcp": "grpcio-gcp >= 0.2.2",
     "grpcio-gcp": "grpcio-gcp >= 0.2.2",
 }
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -29,7 +29,7 @@
 # 'Development Status :: 5 - Production/Stable'
 release_status = "Development Status :: 4 - Beta"
 dependencies = ["google-api-core >= 1.11.0, < 2.0.0dev"]
-extras = {"grpc": "grpcio >= 1.8.2"}
+extras = {"grpc": "grpcio >= 1.8.2, < 2.0dev"}
 
 
 # Setup boilerplate below this line.


Test patch:
diff --git a/api_core/tests/unit/test_retry.py b/api_core/tests/unit/test_retry.py
--- a/api_core/tests/unit/test_retry.py
+++ b/api_core/tests/unit/test_retry.py
@@ -41,6 +41,7 @@ def test_if_exception_type_multiple():
 def test_if_transient_error():
     assert retry.if_transient_error(exceptions.InternalServerError(""))
     assert retry.if_transient_error(exceptions.TooManyRequests(""))
+    assert retry.if_transient_error(exceptions.ServiceUnavailable(""))
     assert not retry.if_transient_error(exceptions.InvalidArgument(""))