Repository: googleapis/google-cloud-python

Issue:
Subscribe experimental doesn't appear to be retrying on retryable exceptions, also doesn't surface the error.
Full context at https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4841#issuecomment-386988007 downwards.

gist with logs: https://gist.github.com/psalaberria002/e91956d94e82d706d65da7111a40ca24

Possible culprits:

1. `send()` and `recv()` expectedly error on `next(self.call)`, but if that was a *retryable* error that somehow re-bubbled, it would not trigger any callbacks.
2. `bidi` uses `rpc._wrapped` because `api_core` doesn't appropriately expose `add_done_callback`. This means that bidi unwraps all of the retry logic.

This doesn't explain *why* retrying a perfectly normal looking error somehow still leads us to a broken stream, unless the newly opened stream also has an error. But it's unclear why that wouldn't happen when calling the initial RPC.


Discussion:
Can anyone tell me how I can auto kill the subscriber in case this happens? I mean, coz, restarting the process works, killing it incase such event happens and starting it will do the trick!
Just need to know how does my code know that this exception arouse.
As noted above, the core issue here is that the lower-level code *is not* surfacing this error in a way that can be handled by the user. There is no workaround for now. When this is fixed, the `future` will appropriate raise the exception.
@theacodes Any update on this? It's stopping every 2 hours now. Very weird. Could you explain the meaning of the log? I just couldn't understand it as to why it's happening. I'm curious
The log: 
```
Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 439, in _thread_main
    response = self._bidi_rpc.recv()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 365, in recv
    super(ResumableBidiRpc, self).recv)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 357, in _recoverable
    return method(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv
    return next(self.call)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 350, in next
    return self._next()
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/usr/local/lib/python2.7/dist-packages/six.py", line 737, in raise_from
    raise value
ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]
[2018-05-15 14:39:07] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py:455} INFO - Thread-ConsumeBidirectionalStream exiting
[2018-05-15 14:39:08] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py:110} DEBUG - Empty queue and inactive call, exiting request generator.
[2018-05-15 14:39:08] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/leaser.py:164} INFO - Thread-LeaseMaintainer exiting.
```
I find `subscriber` in **0.32.0** more stable than `subscriber_experimental` in **0.34.0**. That stops too, but it happens after a day or two not unlike `subscriber_experimental` which goes offline every two hours.
I'm encountering exactly the same issue. What is currently the recommended approach to build an application which continuously awaits messages and processes them, until the exact cause is identified and solved? Are there older versions we can use?
Hi folks - I am actively working on this. Please keep in mind that this is a *beta* library that is using a *beta* feature of the Pub/Sub API. Issues like this are why we haven't yet declared this library (and the endpoint) GA. We appreciate your feedback and testing!

In the meantime, you can try using the `google-api-python-client` and the non-streaming `pull` method. You won't get anywhere near the throughput as you would with streaming pull, but for many use cases this is totally acceptable.

* API Client Docs: https://developers.google.com/api-client-library/python/
* Pub/Sub Pull Docs: https://developers.google.com/resources/api-libraries/documentation/pubsub/v1/python/latest/pubsub_v1.projects.subscriptions.html#pull
Hi @theacodes thank you very much. I understand the beta state, thanks for the links. I'm looking forward to test the latest updates.
Hi @theacodes , 

I tested 0.35.0 this morning, using following code:
```python
subscriber = pubsub.SubscriberClient()
subscription = subscriber.subscription_path(options.project, options.subscription)
print("Starting")
while True:
    try:
        future = subscriber.subscribe(subscription, callback)
        future.result()
    except KeyboardInterrupt as e:
        future.cancel()
        break
    except Exception as e:
        print("Reloading subscriber due to faillure ({0}).".format(e.__class__.__name__))
        pass
```
The goal is to re-open whenever something occurs. Unfortunately after about 3 minutes following occurs, which is the same issue encountered with 0.34.0 and was reported in #4841 
```[D 180528 14:36:29 bidi:466] Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code.
    Traceback (most recent call last):
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next
        return six.next(self._wrapped)
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/grpc/_channel.py", line 341, in __next__
        return self._next()
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/grpc/_channel.py", line 335, in _next
        raise self
    grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])>
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 366, in _recoverable
        return method(*args, **kwargs)
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv
        return next(self.call)
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next
        six.raise_from(exceptions.from_grpc_error(exc), exc)
      File "<string>", line 3, in raise_from
    google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next
        return six.next(self._wrapped)
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/grpc/_channel.py", line 341, in __next__
        return self._next()
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/grpc/_channel.py", line 335, in _next
        raise self
    grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])>
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 457, in _thread_main
        response = self._bidi_rpc.recv()
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 383, in recv
        super(ResumableBidiRpc, self).recv)
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 375, in _recoverable
        return method(*args, **kwargs)
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv
        return next(self.call)
      File "/home/javdrher/.virtualenvs/recommender-api-backend/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next
        six.raise_from(exceptions.from_grpc_error(exc), exc)
      File "<string>", line 3, in raise_from
    google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]
[I 180528 14:36:29 bidi:473] Thread-ConsumeBidirectionalStream exiting
[D 180528 14:36:29 bidi:110] Empty queue and inactive call, exiting request generator.
[I 180528 14:36:30 leaser:164] Thread-LeaseMaintainer exiting.
```
No exception seems to reach the handler. 

Python 3.6.3, libraries:
google-api-core==1.2.0
google-auth==1.4.1
google-cloud-pubsub==0.35.0
googleapis-common-protos==1.5.3
grpc-google-iam-v1==0.11.4
grpcio==1.12.0



Thank you for testing this, @anmol1vw13! I still haven't been able to reproduce this myself, but it would be super helpful if you could enable debug level logging and provide the *full* log of everything. If it's too big to send over a github gist, please feel free to email it to me.
Note to (future) self:

This is due to https://github.com/GoogleCloudPlatform/google-cloud-python/blob/26f1a97ec06b5c48df7a3c9eec00ae3985657164/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py#L375 assuming that the method will *always* succeed. For some reason, this is possible to fail as well. 
Assuming you meant to refer me, that log was created with debugging on, the full log can be found here:
https://gist.github.com/javdrher/90ebab7a171466f9b9c457f28d95ac09
it occurs frequently, each time I run it this happens after approximately 3 minutes.
@javdrher interesting. It seems like in your case you never received any messages - is that expected? 

Can you tell me more about your environment? What credentials are you using?
This is indeed the case, I wasn't sending any messages through. Its currently running in a development environment with no live requests passing through it yet. The client was running on my laptop, with standard gcloud authentication. 

I also have an implementation which is now being integrated into our system, which is running on a Kubernetes cluster with a service account with the PubSub publisher role. That implementation currently runs 0.34.0, with following code:
```python
future = subscriber.subscribe_experimental(subscription, callback)
while True:
    if not future._manager._consumer.is_active:
        logging.info("Reloading subscriber due to known faillure.")
        future = subscriber.subscribe_experimental(subscription, loader)
    time.sleep(60)
```
The log of that pod is as follows (it runs logging level INFO, but I know from previous runs the debug log looks the same like the one I posted)
```

sales-loader | May 28, 2018, 7:23:54 PM | [I 180528 17:23:54 loader:39] Reloading subscriber due to known faillure.
sales-loader | May 28, 2018, 7:23:28 PM | [I 180528 17:23:28 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 28, 2018, 7:23:22 PM | [I 180528 17:23:22 bidi:455] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 28, 2018, 7:19:52 PM | [I 180528 17:19:52 loader:39] Reloading subscriber due to known faillure.
sales-loader | May 28, 2018, 7:19:36 PM | [I 180528 17:19:36 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 28, 2018, 7:19:31 PM | [I 180528 17:19:31 bidi:455] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 28, 2018, 7:15:51 PM | [I 180528 17:15:51 loader:39] Reloading subscriber due to known faillure.
sales-loader | May 28, 2018, 7:14:56 PM | [I 180528 17:14:56 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 28, 2018, 7:14:56 PM | [I 180528 17:14:56 bidi:455] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 28, 2018, 7:12:50 PM | [I 180528 17:12:50 loader:39] Reloading subscriber due to known faillure.
sales-loader | May 28, 2018, 7:11:58 PM | [I 180528 17:11:58 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 28, 2018, 7:11:57 PM | [I 180528 17:11:57 bidi:455] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 28, 2018, 7:08:49 PM | [I 180528 17:08:49 loader:39] Reloading subscriber due to known faillure.
sales-loader | May 28, 2018, 7:08:10 PM | [I 180528 17:08:10 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 28, 2018, 7:08:07 PM | [I 180528 17:08:07 bidi:455] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 28, 2018, 7:04:48 PM | [I 180528 17:04:48 loader:39] Reloading subscriber due to known faillure.
sales-loader | May 28, 2018, 7:04:22 PM | [I 180528 17:04:22 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 28, 2018, 7:04:20 PM | [I 180528 17:04:20 bidi:455] Thread-ConsumeBidirectionalStream exiting
...
```
It's very strange to see it fail that often, but that is on `0.34`, so I'd be interested to see how it behaves in your cluster with `0.35`. On my end, I've literally run the thing for 4 days without any fatal errors, so this is a fun one. 

If you try using service account key authentication locally, does that make a difference?
Modified the code as follows for 0.35.0:
```python
future = subscriber.subscribe(subscription, loader)
while True:
    if not future._manager._consumer.is_active:
        print("Reloading subscriber due to known faillure.")
        future = subscriber.subscribe(subscription, callback)
    time.sleep(60)
```

With the service account locally:
https://gist.github.com/javdrher/df44f6521ae352cde463aa7af7d97607

On kubernetes (GKE, version 1.8.10-gke.0), no signs of improvement either:

```
sales-loader | May 29, 2018, 10:25:42 AM | [I 180529 08:25:42 loader:38] Reloading subscriber due to known faillure.
sales-loader | May 29, 2018, 10:25:05 AM | [I 180529 08:25:05 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 29, 2018, 10:25:05 AM | [I 180529 08:25:05 bidi:473] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 29, 2018, 10:22:42 AM | [I 180529 08:22:42 loader:38] Reloading subscriber due to known faillure.
sales-loader | May 29, 2018, 10:22:05 AM | [I 180529 08:22:05 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 29, 2018, 10:21:58 AM | [I 180529 08:21:58 bidi:473] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 29, 2018, 10:18:42 AM | [I 180529 08:18:42 loader:38] Reloading subscriber due to known faillure.
sales-loader | May 29, 2018, 10:18:17 AM | [I 180529 08:18:17 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 29, 2018, 10:18:15 AM | [I 180529 08:18:15 bidi:473] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 29, 2018, 10:14:42 AM | [I 180529 08:14:42 loader:38] Reloading subscriber due to known faillure.
sales-loader | May 29, 2018, 10:14:30 AM | [I 180529 08:14:30 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 29, 2018, 10:14:24 AM | [I 180529 08:14:24 bidi:473] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 29, 2018, 10:11:44 AM | [I 180529 08:11:44 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 29, 2018, 10:11:42 AM | [I 180529 08:11:42 loader:38] Reloading subscriber due to known faillure.
sales-loader | May 29, 2018, 10:11:41 AM | [I 180529 08:11:41 bidi:473] Thread-ConsumeBidirectionalStream exiting
sales-loader | May 29, 2018, 10:08:41 AM | [I 180529 08:08:41 loader:38] Reloading subscriber due to known faillure.
sales-loader | May 29, 2018, 10:08:22 AM | [I 180529 08:08:22 leaser:164] Thread-LeaseMaintainer exiting.
sales-loader | May 29, 2018, 10:08:16 AM | [I 180529 08:08:16 bidi:473] Thread-ConsumeBidirectionalStream exiting
```

I also see this issue in v0.35.0. It happens during a period where no message published to the subscribed topic for roughly 3 to 3.5 minutes. Once the error occurs, no further messages are published until the process is manually killed and restarted. No error bubbles up to the `future`, and likewise, `future.done()` always reports `False`, leading to no simple way of detecting the scenario.

In our use-case, we expect there to be periods of very low activity (particularly overnight), and also one topic usually has messages published at a rate of one message every five minutes. Subscribers to this topic would fail immediately, and subscribers to other topics with higher activity during the day would fail at the end of the day when activity falls.

Logs are at https://gist.github.com/sjagoe/8b360af13e9c4edb3e76f1da735950ce

The code I'm using is taken from the main example in the documentation.

```
def callback(message):
    message.ack()

def run():
    subscriber = pubsub_v1.SubscriberClient()
    subscription_path = subscriber.subscription_path('my-project', 'my-subscription')
    future = subscriber.subscribe(subscription_path, callback)
    future.result()
```

#5402 should address this. Standby for a release.
Thanks, this should mostly fix the propagation of the exception? when released I'll check the stability as well.
Yep
@theacodes This seems to have introduced an error. I now have this at the tail of the logs. Otherwise, the behaviour is the same; the error does not bubble up to the future and the subscriber hangs.

```
Traceback (most recent call last):                                                                                                             
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 456, in _thread_main
    response = self._bidi_rpc.recv()                                                                                                     
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 382, in recv                
    super(ResumableBidiRpc, self).recv)                                                                  
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 373, in _recoverable       
    self._finalize(exc)                                                                                             
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 323, in _finalize           
    callback(result)                                                                                                          
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 424, in _on_rpc_done
    self.close(reason=future)                                                                                                 
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 321, in close
    self._consumer.stop()                                                                                                     
  File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 493, in stop                   
    self._thread.join()                                                                                                       
  File "/usr/local/lib/python3.6/threading.py", line 1053, in join                                                                       
    raise RuntimeError("cannot join current thread")                                                                          
RuntimeError: cannot join current thread                                                                                                 
2018-05-30 06:04:12,489 DEBUG    [google.cloud.pubsub_v1.subscriber._protocol.bidi:120] Cleanly exiting request generator.
2018-05-30 06:04:12,490 INFO     [google.cloud.pubsub_v1.subscriber._protocol.bidi:472] Thread-ConsumeBidirectionalStream exiting
2018-05-30 06:04:12,582 INFO     [google.cloud.pubsub_v1.subscriber._protocol.leaser:164] Thread-LeaseMaintainer exiting.
```
I'm investigating this now.

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py
@@ -342,36 +342,48 @@ def _reopen(self):
             # has exited.
             self.request_generator = None
 
-            self.open()
+            # Note: we do not currently do any sort of backoff here. The
+            # assumption is that re-establishing the stream under normal
+            # circumstances will happen in intervals greater than 60s.
+            # However, it is possible in a degenerative case that the server
+            # closes the stream rapidly which would lead to thrashing here,
+            # but hopefully in those cases the server would return a non-
+            # retryable error.
+
+            try:
+                self.open()
+            # If re-opening or re-calling the method fails for any reason,
+            # consider it a terminal error and finalize the stream.
+            except Exception as exc:
+                self._finalize(exc)
+                raise
+
+            _LOGGER.info('Re-established stream')
 
     def _recoverable(self, method, *args, **kwargs):
         """Wraps a method to recover the stream and retry on error.
 
-        If a recoverable error occurs, this will retry the RPC and retry the
-        method. If a second error occurs while retrying the method, it will
-        bubble up.
+        If a retryable error occurs while making the call, then the stream will
+        be re-opened and the method will be retried. This happens indefinitely
+        so long as the error is a retryable one. If an error occurs while
+        re-opening the stream, then this method will raise immediately and
+        trigger finalization of this object.
 
         Args:
             method (Callable[..., Any]): The method to call.
             args: The args to pass to the method.
             kwargs: The kwargs to pass to the method.
         """
-        try:
-            return method(*args, **kwargs)
+        while True:
+            try:
+                return method(*args, **kwargs)
 
-        except Exception as exc:
-            if not self._should_recover(exc):
-                self.close()
-                raise exc
+            except Exception as exc:
+                if not self._should_recover(exc):
+                    self.close()
+                    raise exc
 
-        try:
             self._reopen()
-            return method(*args, **kwargs)
-        # If re-opening or re-calling the method fails for any reason, consider
-        # it a terminal error and finalize the object.
-        except Exception as exc:
-            self._finalize(exc)
-            raise
 
     def send(self, request):
         return self._recoverable(
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
@@ -33,6 +33,7 @@
 import google.cloud.pubsub_v1.subscriber.scheduler
 
 _LOGGER = logging.getLogger(__name__)
+_RPC_ERROR_THREAD_NAME = 'Thread-OnRpcTerminated'
 _RETRYABLE_STREAM_ERRORS = (
     exceptions.DeadlineExceeded,
     exceptions.ServiceUnavailable,
@@ -414,11 +415,28 @@ def _should_recover(self, exception):
         # If this is in the list of idempotent exceptions, then we want to
         # recover.
         if isinstance(exception, _RETRYABLE_STREAM_ERRORS):
+            logging.info('Observed recoverable stream error %s', exception)
             return True
+        logging.info('Observed non-recoverable stream error %s', exception)
         return False
 
     def _on_rpc_done(self, future):
+        """Triggered whenever the underlying RPC terminates without recovery.
+
+        This is typically triggered from one of two threads: the background
+        consumer thread (when calling ``recv()`` produces a non-recoverable
+        error) or the grpc management thread (when cancelling the RPC).
+
+        This method is *non-blocking*. It will start another thread to deal
+        with shutting everything down. This is to prevent blocking in the
+        background consumer and preventing it from being ``joined()``.
+        """
         _LOGGER.info(
             'RPC termination has signaled streaming pull manager shutdown.')
         future = _maybe_wrap_exception(future)
-        self.close(reason=future)
+        thread = threading.Thread(
+            name=_RPC_ERROR_THREAD_NAME,
+            target=self.close,
+            kwargs={'reason': future})
+        thread.daemon = True
+        thread.start()


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
@@ -420,10 +420,12 @@ def test__should_recover_false():
     assert manager._should_recover(exc) is False
 
 
-def test__on_rpc_done():
+@mock.patch('threading.Thread', autospec=True)
+def test__on_rpc_done(thread):
     manager = make_manager()
 
-    with mock.patch.object(manager, 'close') as close:
-        manager._on_rpc_done(mock.sentinel.error)
+    manager._on_rpc_done(mock.sentinel.error)
 
-    close.assert_called_once_with(reason=mock.sentinel.error)
+    thread.assert_called_once_with(
+        name=mock.ANY, target=manager.close,
+        kwargs={'reason': mock.sentinel.error})