Repository: googleapis/google-cloud-python

Issue:
Pubsub 0.35.3 stop consuming messages - ConsumeBidirectionalStream exited RPC
After an ~hour, the rpc connection is closed:

```
[2018-06-06 11:02:36,258 - INFO - Thread-14 - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,259 - INFO - Thread-ConsumeBidirectionalStream - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,260 - INFO - Thread-14 - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,260 - **ERROR - Thread-ConsumeBidirectionalStream - bidi] The bidirectional RPC unexpectedly exited. <google.api_core.grpc_helpers._StreamingResponseIterator object at 0x7fb758946090>** 
```

I need to cancel and start the subscriber again.


OS type and version - "Debian GNU/Linux 8 (jessie)"
Version - Python 2.7.9
4. pip freeze | grep pubsub
google-cloud-pubsub==0.35.3

Code example:

```
        def subscriber_handler(message):
            path = message.attributes.get("type")
            self.handler_function(path=path, body=message.data)  
            message.ack()
        self.listening_subscriber = self.subscriber.subscribe(subscription_path, callback=subscriber_handler, flow_control=flow_control)
```

All the other info is the same as in https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5325.




Discussion:
Thanks, @shomratalon! Can you provide me a bit more information?

1. What does your callback do?
2. How many messages per second are you processing?
3. How are you authenticating?
4. What are your flow-control settings?
Okay, I think I may have identified one potential cause of this issue. I've yet to be able to reproduce this so if y'all could try this out and let me know that would be really awesome.

You can install the dev version by either:

```
pip install --upgrade http://temp.theadora.io/google_cloud_pubsub-0.35.3-py2.py3-none-any.whl
```

or

```
git clone https://github.com/GoogleCloudPlatform/google-cloud-python.git
cd google-cloud-python
git checkout pubsub-lock-active
pip install --upgrade .
```

I would **highly** recommend doing this in a *fresh* virtualenv.
@jam182 please take a look at https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5444#issuecomment-395146942.
I reproduced the bug

```
DEBUG | 19:16:04 | google.cloud.pubsub_v1.subscriber._protocol.bidi | waiting for recv.
DEBUG | 19:16:04 | google.cloud.pubsub_v1.subscriber._protocol.dispatcher | Handling 14 batched requests
DEBUG | 19:16:04 | google.cloud.pubsub_v1.subscriber._protocol.bidi | recved response.
DEBUG | 19:16:04 | google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager | Scheduling callbacks for 2 messages.
INFO | 19:16:04 | google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager | Observed recoverable stream error 504 Deadline Exceeded
DEBUG | 19:16:05 | google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager | Sent request(s) over unary RPC.
DEBUG | 19:16:05 | google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager | Sent request(s) over unary RPC.
ERROR | 19:16:05 | google.cloud.pubsub_v1.subscriber._protocol.bidi | The bidirectional RPC unexpectedly exited. <google.api_core.grpc_helpers._StreamingResponseIterator object at 0x7fd536b71908>
INFO | 19:16:05 | google.cloud.pubsub_v1.subscriber._protocol.bidi | Thread-ConsumeBidirectionalStream exiting
DEBUG | 19:16:05 | google.cloud.pubsub_v1.subscriber._protocol.bidi | Empty queue and inactive call, exiting request generator.
INFO | 19:16:06 | google.cloud.pubsub_v1.subscriber._protocol.heartbeater | Thread-Heartbeater exiting.
INFO | 19:16:08 | google.cloud.pubsub_v1.subscriber._protocol.leaser | Thread-LeaseMaintainer exiting.
```
and then it was hanging.

I made sure the `@property` you added is being reached... :(
Try it once more the the latest copy of that branch/file. I had to switch a `Lock` with an `RLock`.
I tried again, I can still reproduce the bug, unfortunately. It looks like with 120 r/s and `info` logging level I am able to reproduce it quite consistently. But I have to wait each time 10 minutes.
@jam182 is it hanging or is it the "the bidirectional RPC unexpectedly closed"?
Update: I was able to reproduce this myself. Investigating now.

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
@@ -235,7 +235,9 @@ def send(self, request):
             raise ValueError(
                 'Can not send() on an RPC that has never been open()ed.')
 
-        if self.is_active:
+        # Don't use self.is_active(), as ResumableBidiRpc will overload it
+        # to mean something semantically different.
+        if self.call.is_active():
             self._request_queue.put(request)
         else:
             # calling next should cause the call to raise.
@@ -330,11 +332,15 @@ def _on_call_done(self, future):
         # "future" here is also a grpc.RpcError.
         if not self._should_recover(future):
             self._finalize(future)
+        else:
+            _LOGGER.debug('Re-opening stream from gRPC callback.')
+            self._reopen()
 
     def _reopen(self):
         with self._operational_lock:
             # Another thread already managed to re-open this stream.
-            if self.is_active:
+            if self.call is not None and self.call.is_active():
+                _LOGGER.debug('Stream was already re-established.')
                 return
 
             self.call = None
@@ -379,10 +385,14 @@ def _recoverable(self, method, *args, **kwargs):
                 return method(*args, **kwargs)
 
             except Exception as exc:
+                _LOGGER.debug('Call to retryable %r caused %s.', method, exc)
                 if not self._should_recover(exc):
                     self.close()
+                    _LOGGER.debug('Not retrying %r due to %s.', method, exc)
+                    self._finalize(exc)
                     raise exc
 
+            _LOGGER.debug('Re-opening stream from retryable %r.', method)
             self._reopen()
 
     def send(self, request):
@@ -398,8 +408,13 @@ def is_active(self):
         """bool: True if this stream is currently open and active."""
         # Use the operational lock. It's entirely possible for something
         # to check the active state *while* the RPC is being retried.
+        # Also, use finalized to track the actual terminal state here.
+        # This is because if the stream is re-established by the gRPC thread
+        # it's technically possible to check this between when gRPC marks the
+        # RPC as inactive and when gRPC executes our callback that re-opens
+        # the stream.
         with self._operational_lock:
-            return self.call is not None and self.call.is_active()
+            return self.call is not None and not self._finalized
 
 
 class BackgroundConsumer(object):
@@ -491,8 +506,8 @@ def _thread_main(self):
 
         else:
             _LOGGER.error(
-                'The bidirectional RPC unexpectedly exited. %s',
-                self._bidi_rpc.call)
+                'The bidirectional RPC unexpectedly exited. This is a truly '
+                'exceptional case. Please file a bug with your logs.')
 
         _LOGGER.info('%s exiting', _BIDIRECTIONAL_CONSUMER_NAME)
 
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
@@ -402,6 +402,7 @@ def _on_response(self, response):
         After the messages have all had their ack deadline updated, execute
         the callback for each message using the executor.
         """
+
         _LOGGER.debug(
             'Scheduling callbacks for %s messages.',
             len(response.received_messages))


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_bidi.py
@@ -285,13 +285,17 @@ def test_initial_state(self):
         assert bidi_rpc.is_active is False
 
     def test_done_callbacks_recoverable(self):
-        bidi_rpc = bidi.ResumableBidiRpc(None, lambda _: True)
+        start_rpc = mock.create_autospec(
+            grpc.StreamStreamMultiCallable, instance=True)
+        bidi_rpc = bidi.ResumableBidiRpc(start_rpc, lambda _: True)
         callback = mock.Mock(spec=['__call__'])
 
         bidi_rpc.add_done_callback(callback)
         bidi_rpc._on_call_done(mock.sentinel.future)
 
         callback.assert_not_called()
+        start_rpc.assert_called_once()
+        assert bidi_rpc.is_active
 
     def test_done_callbacks_non_recoverable(self):
         bidi_rpc = bidi.ResumableBidiRpc(None, lambda _: False)