Repository: googleapis/google-cloud-python

Issue:
Pubsub: Pull Subscriber unable to re-connect after a while
#### Environment details

1. API: PubSub (maybe gRPC)
2. OS type and version: Debian GNU/Linux 9 (stretch) container (based on python:2.7.15-slim-stretch) running in 1.12.6-gke.10
3. Python version: 2.7.15
4. google-cloud-pubsub version:  0.40.0

#### Steps to reproduce

  1. Have several PubSub subscribers run for a while (from hours to days) in a Kubernetes container, reconnecting every 10 minutes as normal
  2. One of the subscribers suddenly start seeing reconnection rate go up as much as CPU resources allow (trying every few milliseconds in our case) and repeatedly getting 503 errors
  3. At this point subscriber stops pulling messages and the process needs to be restarted

The issue did happen in a different service with only one subscriber in the container, but having several subscribers with all the others working excludes the possibility of a lot of other factors preventing re-connection (ie DNS resolution, no network, etc etc).

#### Code example

Totally standard Pull subscription using `SubscriberClient` + `create_subscription` + `subscribe`. Can paste code if required though.

#### StackDriver log snippet
```
...
I  2019-05-09 10:51:06.131 [INFO] Re-established stream
I  2019-05-09 10:51:06.134 [DEBUG] Call to retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7f2fde071110>> caused 503 Connect Failed.
I  2019-05-09 10:51:06.135 [INFO] Observed recoverable stream error 503 Connect Failed 
I  2019-05-09 10:51:06.135 [DEBUG] Re-opening stream from retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7f2fde071110>>.
I  2019-05-09 10:51:06.137 [DEBUG] Stream was already re-established.
I  2019-05-09 10:51:06.138 [INFO] Re-established stream
I  2019-05-09 10:51:06.141 [INFO] Observed recoverable stream error 503 Connect Failed
I  2019-05-09 10:51:06.142 [DEBUG] Re-opening stream from gRPC callback.
I  2019-05-09 10:51:06.143 [INFO] Re-established stream 
I  2019-05-09 10:51:06.145 [DEBUG] The current p99 value is 10 seconds.
I  2019-05-09 10:51:06.146 [DEBUG] Snoozing lease management for 7.560536 seconds.
I  2019-05-09 10:51:06.147 [INFO] Observed recoverable stream error 503 Connect Failed
I  2019-05-09 10:51:06.147 [INFO] Re-established stream
...
```

First of all of course I'd be interested to help getting to the bottom of the issue and get it resolved.

But in the meanwhile it would be great to have a workaround detecting lost connection with a subscriber. I went through the public API documentation and couldn't find a way to get to the underlying (gRPC?) client, but it would be great to have a clean(ish) method doing a periodic check on the connection to be able to restart once the issue happens.

Thanks a lot in advance 🙏 


Discussion:
I came across #7709 which might be the root cause. When the new release comes out I'll post an update, but will also try to dig deeper in the meanwhile.
@daaain Thank you for reporting this and doing the initial research!

The linked issue could indeed be the root cause, and [the fix](https://github.com/googleapis/google-cloud-python/pull/7863) for it merged recently makes sure that a clean shutdown of background threads is triggered if the underlying gRPC channel remains in an error state for too long.

The log records posted here are produced by these very threads, thus shutting them down should get rid of this problem.

Looking forward for more info after the next release!

(alternatively, if the nature of your application allows for it, i.e. it is not mission-critical, you could also experiment with the current development version of google-cloud-pubsub).

**Update:**
Whoa, I might have just been (un)lucky enough to actually observe a very similar error locally. While working on another issue and running the subscriber client's streaming pull (`subscriber.subscribe()`), my WiFi connection apparently got unstable, and I saw the following in the log output:

```
...
DEBUG    [2019-05-15 10:06:01,440] Thread-ConsumeBidirectionalStream [_recoverable] Call to retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7fb5eb91d630>> caused 503 DNS resolution failed.
INFO     [2019-05-15 10:06:01,440] Thread-ConsumeBidirectionalStream [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,440] Thread-ConsumeBidirectionalStream [_recoverable] Re-opening stream from retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7fb5eb91d630>>.
INFO     [2019-05-15 10:06:01,441] Thread-ConsumeBidirectionalStream [_reopen] Re-established stream
INFO     [2019-05-15 10:06:01,441] Thread-1                          [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,441] Thread-1                          [_on_call_done] Re-opening stream from gRPC callback.
DEBUG    [2019-05-15 10:06:01,441] Thread-1                          [_reopen] Stream was already re-established.
INFO     [2019-05-15 10:06:01,441] Thread-1                          [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,441] Thread-1                          [_on_call_done] Re-opening stream from gRPC callback.
INFO     [2019-05-15 10:06:01,441] Thread-1                          [_reopen] Re-established stream
DEBUG    [2019-05-15 10:06:01,441] Thread-ConsumeBidirectionalStream [_recoverable] Call to retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7fb5eb91d630>> caused 503 DNS resolution failed.
INFO     [2019-05-15 10:06:01,442] Thread-ConsumeBidirectionalStream [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,442] Thread-ConsumeBidirectionalStream [_recoverable] Re-opening stream from retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7fb5eb91d630>>.
INFO     [2019-05-15 10:06:01,442] Thread-ConsumeBidirectionalStream [_reopen] Re-established stream
INFO     [2019-05-15 10:06:01,442] Thread-1                          [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,442] Thread-1                          [_on_call_done] Re-opening stream from gRPC callback.
DEBUG    [2019-05-15 10:06:01,442] Thread-1                          [_reopen] Stream was already re-established.
INFO     [2019-05-15 10:06:01,442] Thread-1                          [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,442] Thread-1                          [_on_call_done] Re-opening stream from gRPC callback.
INFO     [2019-05-15 10:06:01,443] Thread-1                          [_reopen] Re-established stream
DEBUG    [2019-05-15 10:06:01,443] Thread-ConsumeBidirectionalStream [_recoverable] Call to retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7fb5eb91d630>> caused 503 DNS resolution failed.
INFO     [2019-05-15 10:06:01,443] Thread-ConsumeBidirectionalStream [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,443] Thread-ConsumeBidirectionalStream [_recoverable] Re-opening stream from retryable <bound method ResumableBidiRpc._recv of <google.api_core.bidi.ResumableBidiRpc object at 0x7fb5eb91d630>>.
INFO     [2019-05-15 10:06:01,444] Thread-ConsumeBidirectionalStream [_reopen] Re-established stream
INFO     [2019-05-15 10:06:01,444] Thread-1                          [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,444] Thread-1                          [_on_call_done] Re-opening stream from gRPC callback.
DEBUG    [2019-05-15 10:06:01,444] Thread-1                          [_reopen] Stream was already re-established.
INFO     [2019-05-15 10:06:01,444] Thread-1                          [_should_recover] Observed recoverable stream error 503 DNS resolution failed
DEBUG    [2019-05-15 10:06:01,444] Thread-1                          [_on_call_done] Re-opening stream from gRPC callback.
INFO     [2019-05-15 10:06:01,445] Thread-1                          [_reopen] Re-established stream
...
```
It appears that the [streaming_pull()](https://github.com/googleapis/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/gapic/subscriber_client.py#L1028-L1041) method is being repeatedly called in rapid succession despite the Retry settings that should result in exponentially increasing delays between successive calls.

The issue is reproducible on the current latest `master` branch, which already includes the aforementioned fix for #7709, thus my previous comment does not apply, unfortunately. To reproduce it locally, one must simply disable their internet connection, and then use subscriber client for a streaming pull:
```py
subscriber.subscribe(SUBSCRIPTION_PATH, callback=my_callback)

while True:
    try:
        time.sleep(60)
    except KeyboardInterrupt:
        break
```

This busy _"re-open the stream"_ loop consuming a lot of CPU is definitely not expected behavior.
I think that the "definition of done" for this issue will be to definitively prove that this is a grpc level issue.  If so, we can open up a grpc issue and we can work with the pub/sub team to see if this feature is truly GA blocking.
Hi, 

I've been experiencing something similar but with publishing. When the connection to the cloud is disrupted (error 503), the gRPC in the background continues to attempt to re-establish connectivity with "exponentially increasing delays" as described above. Despite using an async request (check code sample below) this essentially hijacks the normal execution course of the program until the connection is re-established. 
 
This error can be easily replicated by removing / adding routes to the routing table manually to simulate connections and disconnects to the cloud. I've managed to replicate this issue with a sample code from the pubsub documentation as follows:

```python
def callback(message_future, testOne, testTwo):
    # When timeout is unspecified, the exception method waits indefinitely.
    if message_future.exception(timeout=3):
        print('Publishing message on {} threw an Exception {}.'.format(
            TOPIC_NAME, message_future.exception()))
    else:
        print("The number is: {} and the refOne: {} plus refTwo {}".format(message_future.result(), testOne, testTwo))


for n in range(1, 6):
    data = u'Message number {}'.format(n)
    # Data must be a bytestring
    data = data.encode('utf-8')
    # When you publish a message, the client returns a Future.
    message_future = PUBLISHER.publish(TOPIC_PATH, data=data)
    message_future.add_done_callback(partial(callback, testOne=123, testTwo=str(datetime.datetime.now())))

print('Published message IDs:')
```

I am using the latest google.cloud.pubsub libraries available to date (0.41). Any suggestions/advise?

Thanks!

Dan


Thanks to reporting this, @Dan4London. Could it be that this issue is the same as #8036 reporting a similar problem with publisher blocking?
I investigated the issue with the subscriber reconnect attempts skyrocketing. It seems that it has to do with the [api_core.bidi.ResumableBidiRpc](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L282) helper.

If there are network problems, the following can happen:

- The code initiates asynchronous message pulling by calling `subscriber.subscribe()`. That method creates a new [StreamingPullManager](https://github.com/googleapis/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py#L76) instance, and the latter opens the [BackgroundConsumer](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L521-L567) running in its own thread.
- The background consumer thread [opens](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L202-L233) the (resumable) `BidiRpc`.
- `ResumableBidiRpc` [starts the RPC](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L210), which in this case is the [SubscriberClient.straming_pull()](https://github.com/googleapis/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/gapic/subscriber_client.py#L1031-L1043) method.
- The result of this invocation (stored in `call`) is a [StreamingResponseIterator](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/grpc_helpers.py#L64) object wrapping the result object returned by the underlying gRPC.
- A done callback is registered on this result object - the [ResumableBidiRpc._on_call_done()](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L345-L354) method.
- If an error occurs, this callback is invoked. It checks if the error is recoverable ([StreamingPullManager._should_recover()](https://github.com/googleapis/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py#L453-L473)) - and 503 Service Unavailable is - and calls `_reopen()`, which in turn calls the [open()](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L377) method again.
- Go to point 2, rinse and repeat, and consume a lot of CPU in the process.

(if the network error gets resolved in the meantime, the behavior returns back to normal)

---

One might wonder why the [Retry](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/retry.py#L211) wrapper that wraps the `streaming_pull` method does not kick in. The reason probably lies somewhere around the place where the `streaming_pull` actually [gets invoked](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/grpc_helpers.py#L125).

That call does not result in an exception, but instead the `grpcio` library returns a `_Rendezvous` object that seems to represent the result of a call. That object gets wrapped into [StreamingResponseIterator](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/grpc_helpers.py#L64), and the exception would only occur when `next()` is called on it.

Since the latter does not happen in that part of the code, the call returns normally (i.e. without an exception), thus the [Retry](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/retry.py#L211) wrapper around it does not kick in (and exponential backoff does not happen).

Instead, the aforementioned `_on_call_done()` callback is invoked, causing an unwanted retry loop at a different level.
It sounds like there needs to be a backoff of some sort.  This looks very much like [this nodejs-pubsub issue](https://github.com/googleapis/nodejs-pubsub/issues/574).

@busunkim96, this sounds like something we should discuss with you.
@plamut that makes sense.  @busunkim96 should be involved, or at least aware, with any api_core related changes.  @crwilcox mentioned offline that `ResumableBidiRpc` is used by Firestore as well, so I'm hereby pinging @BenWhitehead, since this will also likely affect Firestore.
Thanks for your feedback Plamut. #8036 describes precisely the problem I am experiencing. You can replicate the behaviour by re-executing the for loop before the previous gprc publish() request has timed out. 

Do you have any suggestions when a fix for this would be available? 

Thanks!
@Dan4London Unfortunately not (yet...), because it seems that the bug will have to be fixed in one of the PubSub client dependencies, but that will have to be coordinated with other teams that could be affected by the chance. The bug is high on the priority list, though.

**Edit:** Oh, you were probably asking about the publisher issue?
Depends on the priority assigned to it, @sduskis could probably tell more.

**Edit 2:** If nothing else comes across, I can probably have a look at it tomorrow.

**Edit 3:** This issue has been prioritized over the publisher issue, will look at the latter after this.

Resolving patch:
diff --git a/api_core/google/api_core/bidi.py b/api_core/google/api_core/bidi.py
--- a/api_core/google/api_core/bidi.py
+++ b/api_core/google/api_core/bidi.py
@@ -14,8 +14,11 @@
 
 """Bi-directional streaming RPC helpers."""
 
+import collections
+import datetime
 import logging
 import threading
+import time
 
 from six.moves import queue
 
@@ -134,6 +137,73 @@ def __iter__(self):
             yield item
 
 
+class _Throttle(object):
+    """A context manager limiting the total entries in a sliding time window.
+
+    If more than ``access_limit`` attempts are made to enter the context manager
+    instance in the last ``time window`` interval, the exceeding requests block
+    until enough time elapses.
+
+    The context manager instances are thread-safe and can be shared between
+    multiple threads. If multiple requests are blocked and waiting to enter,
+    the exact order in which they are allowed to proceed is not determined.
+
+    Example::
+
+        max_three_per_second = _Throttle(
+            access_limit=3, time_window=datetime.timedelta(seconds=1)
+        )
+
+        for i in range(5):
+            with max_three_per_second as time_waited:
+                print("{}: Waited {} seconds to enter".format(i, time_waited))
+
+    Args:
+        access_limit (int): the maximum number of entries allowed in the time window
+        time_window (datetime.timedelta): the width of the sliding time window
+    """
+
+    def __init__(self, access_limit, time_window):
+        if access_limit < 1:
+            raise ValueError("access_limit argument must be positive")
+
+        if time_window <= datetime.timedelta(0):
+            raise ValueError("time_window argument must be a positive timedelta")
+
+        self._time_window = time_window
+        self._access_limit = access_limit
+        self._past_entries = collections.deque(maxlen=access_limit)  # least recent first
+        self._entry_lock = threading.Lock()
+
+    def __enter__(self):
+        with self._entry_lock:
+            cutoff_time = datetime.datetime.now() - self._time_window
+
+            # drop the entries that are too old, as they are no longer relevant
+            while self._past_entries and self._past_entries[0] < cutoff_time:
+                self._past_entries.popleft()
+
+            if len(self._past_entries) < self._access_limit:
+                self._past_entries.append(datetime.datetime.now())
+                return 0.0  # no waiting was needed
+
+            to_wait = (self._past_entries[0] - cutoff_time).total_seconds()
+            time.sleep(to_wait)
+
+            self._past_entries.append(datetime.datetime.now())
+            return to_wait
+
+    def __exit__(self, *_):
+        pass
+
+    def __repr__(self):
+        return "{}(access_limit={}, time_window={})".format(
+            self.__class__.__name__,
+            self._access_limit,
+            repr(self._time_window),
+        )
+
+
 class BidiRpc(object):
     """A helper for consuming a bi-directional streaming RPC.
 
@@ -323,15 +393,31 @@ def should_recover(exc):
             whenever an error is encountered on the stream.
         metadata Sequence[Tuple(str, str)]: RPC metadata to include in
             the request.
+        throttle_reopen (bool): If ``True``, throttling will be applied to
+            stream reopen calls. Defaults to ``False``.
     """
 
-    def __init__(self, start_rpc, should_recover, initial_request=None, metadata=None):
+    def __init__(
+        self,
+        start_rpc,
+        should_recover,
+        initial_request=None,
+        metadata=None,
+        throttle_reopen=False,
+    ):
         super(ResumableBidiRpc, self).__init__(start_rpc, initial_request, metadata)
         self._should_recover = should_recover
         self._operational_lock = threading.RLock()
         self._finalized = False
         self._finalize_lock = threading.Lock()
 
+        if throttle_reopen:
+            self._reopen_throttle = _Throttle(
+                access_limit=5, time_window=datetime.timedelta(seconds=10),
+            )
+        else:
+            self._reopen_throttle = None
+
     def _finalize(self, result):
         with self._finalize_lock:
             if self._finalized:
@@ -374,7 +460,11 @@ def _reopen(self):
             # retryable error.
 
             try:
-                self.open()
+                if self._reopen_throttle:
+                    with self._reopen_throttle:
+                        self.open()
+                else:
+                    self.open()
             # If re-opening or re-calling the method fails for any reason,
             # consider it a terminal error and finalize the stream.
             except Exception as exc:
@@ -573,7 +663,7 @@ def start(self):
             thread = threading.Thread(
                 name=_BIDIRECTIONAL_CONSUMER_NAME,
                 target=self._thread_main,
-                args=(ready,)
+                args=(ready,),
             )
             thread.daemon = True
             thread.start()
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
@@ -383,6 +383,7 @@ def open(self, callback, on_callback_error):
             start_rpc=self._client.api.streaming_pull,
             initial_request=self._get_initial_request,
             should_recover=self._should_recover,
+            throttle_reopen=True,
         )
         self._rpc.add_done_callback(self._on_rpc_done)
 


Test patch:
diff --git a/api_core/tests/unit/test_bidi.py b/api_core/tests/unit/test_bidi.py
--- a/api_core/tests/unit/test_bidi.py
+++ b/api_core/tests/unit/test_bidi.py
@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import datetime
 import logging
 import threading
 
@@ -116,6 +117,87 @@ def test_exit_with_stop(self):
         assert items == []
 
 
+class Test_Throttle(object):
+    def test_repr(self):
+        delta = datetime.timedelta(seconds=4.5)
+        instance = bidi._Throttle(access_limit=42, time_window=delta)
+        assert repr(instance) == \
+            "_Throttle(access_limit=42, time_window={})".format(repr(delta))
+
+    def test_raises_error_on_invalid_init_arguments(self):
+        with pytest.raises(ValueError) as exc_info:
+            bidi._Throttle(
+                access_limit=10, time_window=datetime.timedelta(seconds=0.0)
+            )
+        assert "time_window" in str(exc_info.value)
+        assert "must be a positive timedelta" in str(exc_info.value)
+
+        with pytest.raises(ValueError) as exc_info:
+            bidi._Throttle(
+                access_limit=0, time_window=datetime.timedelta(seconds=10)
+            )
+        assert "access_limit" in str(exc_info.value)
+        assert "must be positive" in str(exc_info.value)
+
+    def test_does_not_delay_entry_attempts_under_threshold(self):
+        throttle = bidi._Throttle(
+            access_limit=3, time_window=datetime.timedelta(seconds=1)
+        )
+        entries = []
+
+        for _ in range(3):
+            with throttle as time_waited:
+                entry_info = {
+                    "entered_at": datetime.datetime.now(),
+                    "reported_wait": time_waited,
+                }
+                entries.append(entry_info)
+
+        # check the reported wait times ...
+        assert all(entry["reported_wait"] == 0.0 for entry in entries)
+
+        # .. and the actual wait times
+        delta = entries[1]["entered_at"] - entries[0]["entered_at"]
+        assert delta.total_seconds() < 0.1
+        delta = entries[2]["entered_at"] - entries[1]["entered_at"]
+        assert delta.total_seconds() < 0.1
+
+    def test_delays_entry_attempts_above_threshold(self):
+        throttle = bidi._Throttle(
+            access_limit=3, time_window=datetime.timedelta(seconds=1)
+        )
+        entries = []
+
+        for _ in range(6):
+            with throttle as time_waited:
+                entry_info = {
+                    "entered_at": datetime.datetime.now(),
+                    "reported_wait": time_waited,
+                }
+                entries.append(entry_info)
+
+        # For each group of 4 consecutive entries the time difference between
+        # the first and the last entry must have been greater than time_window,
+        # because a maximum of 3 are allowed in each time_window.
+        for i, entry in enumerate(entries[3:], start=3):
+            first_entry = entries[i - 3]
+            delta = entry["entered_at"] - first_entry["entered_at"]
+            assert delta.total_seconds() > 1.0
+
+        # check the reported wait times
+        # (NOTE: not using assert all(...), b/c the coverage check would complain)
+        for i, entry in enumerate(entries):
+            if i != 3:
+                assert entry["reported_wait"] == 0.0
+
+        # The delayed entry is expected to have been delayed for a significant
+        # chunk of the full second, and the actual and reported delay times
+        # should reflect that.
+        assert entries[3]["reported_wait"] > 0.7
+        delta = entries[3]["entered_at"] - entries[2]["entered_at"]
+        assert delta.total_seconds() > 0.7
+
+
 class _CallAndFuture(grpc.Call, grpc.Future):
     pass
 
@@ -442,6 +524,22 @@ def test_reopen_failure_on_rpc_restart(self):
         assert bidi_rpc.is_active is False
         callback.assert_called_once_with(error2)
 
+    def test_using_throttle_on_reopen_requests(self):
+        call = CallStub([])
+        start_rpc = mock.create_autospec(
+            grpc.StreamStreamMultiCallable, instance=True, return_value=call
+        )
+        should_recover = mock.Mock(spec=["__call__"], return_value=True)
+        bidi_rpc = bidi.ResumableBidiRpc(
+            start_rpc, should_recover, throttle_reopen=True
+        )
+
+        patcher = mock.patch.object(bidi_rpc._reopen_throttle.__class__, "__enter__")
+        with patcher as mock_enter:
+            bidi_rpc._reopen()
+
+        mock_enter.assert_called_once()
+
     def test_send_not_open(self):
         bidi_rpc = bidi.ResumableBidiRpc(None, lambda _: False)
 
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
@@ -428,6 +428,7 @@ def test_open(heartbeater, dispatcher, leaser, background_consumer, resumable_bi
         start_rpc=manager._client.api.streaming_pull,
         initial_request=manager._get_initial_request,
         should_recover=manager._should_recover,
+        throttle_reopen=True,
     )
     resumable_bidi_rpc.return_value.add_done_callback.assert_called_once_with(
         manager._on_rpc_done