Repository: googleapis/google-cloud-python

Issue:
Subscribe experimental doesn't appear to be retrying on retryable exceptions, also doesn't surface the error.
Full context at https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4841#issuecomment-386988007 downwards.

gist with logs: https://gist.github.com/psalaberria002/e91956d94e82d706d65da7111a40ca24

Possible culprits:

1. `send()` and `recv()` expectedly error on `next(self.call)`, but if that was a *retryable* error that somehow re-bubbled, it would not trigger any callbacks.
2. `bidi` uses `rpc._wrapped` because `api_core` doesn't appropriately expose `add_done_callback`. This means that bidi unwraps all of the retry logic.

This doesn't explain *why* retrying a perfectly normal looking error somehow still leads us to a broken stream, unless the newly opened stream also has an error. But it's unclear why that wouldn't happen when calling the initial RPC.


Discussion:
Can anyone tell me how I can auto kill the subscriber in case this happens? I mean, coz, restarting the process works, killing it incase such event happens and starting it will do the trick!
Just need to know how does my code know that this exception arouse.
As noted above, the core issue here is that the lower-level code *is not* surfacing this error in a way that can be handled by the user. There is no workaround for now. When this is fixed, the `future` will appropriate raise the exception.
@theacodes Any update on this? It's stopping every 2 hours now. Very weird. Could you explain the meaning of the log? I just couldn't understand it as to why it's happening. I'm curious
The log: 
```
Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 439, in _thread_main
    response = self._bidi_rpc.recv()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 365, in recv
    super(ResumableBidiRpc, self).recv)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 357, in _recoverable
    return method(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv
    return next(self.call)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 350, in next
    return self._next()
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/usr/local/lib/python2.7/dist-packages/six.py", line 737, in raise_from
    raise value
ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]
[2018-05-15 14:39:07] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py:455} INFO - Thread-ConsumeBidirectionalStream exiting
[2018-05-15 14:39:08] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py:110} DEBUG - Empty queue and inactive call, exiting request generator.
[2018-05-15 14:39:08] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/leaser.py:164} INFO - Thread-LeaseMaintainer exiting.
```
I find `subscriber` in **0.32.0** more stable than `subscriber_experimental` in **0.34.0**. That stops too, but it happens after a day or two not unlike `subscriber_experimental` which goes offline every two hours.
I'm encountering exactly the same issue. What is currently the recommended approach to build an application which continuously awaits messages and processes them, until the exact cause is identified and solved? Are there older versions we can use?
Hi folks - I am actively working on this. Please keep in mind that this is a *beta* library that is using a *beta* feature of the Pub/Sub API. Issues like this are why we haven't yet declared this library (and the endpoint) GA. We appreciate your feedback and testing!

In the meantime, you can try using the `google-api-python-client` and the non-streaming `pull` method. You won't get anywhere near the throughput as you would with streaming pull, but for many use cases this is totally acceptable.

* API Client Docs: https://developers.google.com/api-client-library/python/
* Pub/Sub Pull Docs: https://developers.google.com/resources/api-libraries/documentation/pubsub/v1/python/latest/pubsub_v1.projects.subscriptions.html#pull

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
@@ -170,8 +170,8 @@ def __init__(self, start_rpc, initial_request=None):
         self._request_queue = queue.Queue()
         self._request_generator = None
         self._is_active = False
-        self.call = None
         self._callbacks = []
+        self.call = None
 
     def add_done_callback(self, callback):
         """Adds a callback that will be called when the RPC terminates.
@@ -311,14 +311,25 @@ def __init__(self, start_rpc, should_recover, initial_request=None):
         super(ResumableBidiRpc, self).__init__(start_rpc, initial_request)
         self._should_recover = should_recover
         self._operational_lock = threading.Lock()
+        self._finalized = False
+        self._finalize_lock = threading.Lock()
+
+    def _finalize(self, result):
+        with self._finalize_lock:
+            if self._finalized:
+                return
+
+            for callback in self._callbacks:
+                callback(result)
+
+            self._finalized = True
 
     def _on_call_done(self, future):
         # Unlike the base class, we only execute the callbacks on a terminal
         # error, not for errors that we can recover from. Note that grpc's
         # "future" here is also a grpc.RpcError.
         if not self._should_recover(future):
-            for callback in self._callbacks:
-                callback(future)
+            self._finalize(future)
 
     def _reopen(self):
         with self._operational_lock:
@@ -330,7 +341,14 @@ def _reopen(self):
             # Request generator should exit cleanly since the RPC its bound to
             # has exited.
             self.request_generator = None
-            self.open()
+
+            try:
+                self.open()
+            # If re-opening fails, consider this a terminal error and finalize
+            # the object.
+            except Exception as exc:
+                self._finalize(exc)
+                raise
 
     def _recoverable(self, method, *args, **kwargs):
         """Wraps a method to recover the stream and retry on error.


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py
@@ -310,7 +310,7 @@ def test_send_recover(self):
             grpc.StreamStreamMultiCallable,
             instance=True,
             side_effect=[call_1, call_2])
-        should_recover = mock.Mock(autospec=['__call__'], return_value=True)
+        should_recover = mock.Mock(spec=['__call__'], return_value=True)
         bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
 
         bidi_rpc.open()
@@ -331,7 +331,7 @@ def test_send_failure(self):
             grpc.StreamStreamMultiCallable,
             instance=True,
             return_value=call)
-        should_recover = mock.Mock(autospec=['__call__'], return_value=False)
+        should_recover = mock.Mock(spec=['__call__'], return_value=False)
         bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
 
         bidi_rpc.open()
@@ -355,7 +355,7 @@ def test_recv_recover(self):
             grpc.StreamStreamMultiCallable,
             instance=True,
             side_effect=[call_1, call_2])
-        should_recover = mock.Mock(autospec=['__call__'], return_value=True)
+        should_recover = mock.Mock(spec=['__call__'], return_value=True)
         bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
 
         bidi_rpc.open()
@@ -412,7 +412,7 @@ def test_recv_failure(self):
             grpc.StreamStreamMultiCallable,
             instance=True,
             return_value=call)
-        should_recover = mock.Mock(autospec=['__call__'], return_value=False)
+        should_recover = mock.Mock(spec=['__call__'], return_value=False)
         bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
 
         bidi_rpc.open()
@@ -426,6 +426,48 @@ def test_recv_failure(self):
         assert bidi_rpc.is_active is False
         assert call.cancelled is True
 
+    def test_reopen_failure_on_rpc_restart(self):
+        error1 = ValueError('1')
+        error2 = ValueError('2')
+        call = CallStub([error1])
+        # Invoking start RPC a second time will trigger an error.
+        start_rpc = mock.create_autospec(
+            grpc.StreamStreamMultiCallable,
+            instance=True,
+            side_effect=[call, error2])
+        should_recover = mock.Mock(spec=['__call__'], return_value=True)
+        callback = mock.Mock(spec=['__call__'])
+
+        bidi_rpc = bidi.ResumableBidiRpc(start_rpc, should_recover)
+        bidi_rpc.add_done_callback(callback)
+
+        bidi_rpc.open()
+
+        with pytest.raises(ValueError) as exc_info:
+            bidi_rpc.recv()
+
+        assert exc_info.value == error2
+        should_recover.assert_called_once_with(error1)
+        assert bidi_rpc.call is None
+        assert bidi_rpc.is_active is False
+        callback.assert_called_once_with(error2)
+
+    def test_finalize_idempotent(self):
+        error1 = ValueError('1')
+        error2 = ValueError('2')
+        callback = mock.Mock(spec=['__call__'])
+        should_recover = mock.Mock(spec=['__call__'], return_value=False)
+
+        bidi_rpc = bidi.ResumableBidiRpc(
+            mock.sentinel.start_rpc, should_recover)
+
+        bidi_rpc.add_done_callback(callback)
+
+        bidi_rpc._on_call_done(error1)
+        bidi_rpc._on_call_done(error2)
+
+        callback.assert_called_once_with(error1)
+
 
 class TestBackgroundConsumer(object):
     def test_consume_once_then_exit(self):