Repository: googleapis/google-cloud-python

Issue:
PubSub: `set_exception can only be called once.` can still occur "erroneously"
For example, If I add

```python
_LOGGER.debug('thread.Policy.on_exception: %s', exception)
```

just [before `future.set_exception()`][1], then, when forcing a failure by:

```python
from google.cloud.pubsub_v1.subscriber.policy import base
# Monkey-patch to intentionally miss errors.
base.BasePolicy._RETRYABLE_STREAM_ERRORS = ()
```

I see (focus on **00117233** and **00185066**):

```
...
00116110:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :The current p99 value is 10 seconds.
00116111:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :Renewing lease for 0 ack IDs.
00116111:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :Snoozing lease management for 2.555879 seconds.
00117233:DEBUG:google.cloud.pubsub_v1.subscriber.policy.thread:Consumer helper: consume bidirectional stream:thread.Policy.on_exception: 503 The service was unable to fulfill your request. Please try agai
n. [code=8a75]
00117236:DEBUG:google.cloud.pubsub_v1.subscriber._consumer:Thread-23             :Sending initial request: subscription: "projects/precise-truck-742/subscriptions/s-djh-local-1511822800630"
stream_ack_deadline_seconds: 10

00118670:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :The current p99 value is 10 seconds.
00118670:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :Renewing lease for 0 ack IDs.
00118670:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :Snoozing lease management for 1.883336 seconds.
...
00181929:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :The current p99 value is 10 seconds.
00181929:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :Renewing lease for 0 ack IDs.
00181929:DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Thread-8              :Snoozing lease management for 7.864327 seconds.
00185066:DEBUG:google.cloud.pubsub_v1.subscriber.policy.thread:Consumer helper: consume bidirectional stream:thread.Policy.on_exception: 503 The service was unable to fulfill your request. Please try again. [code=8a75]
Exception in thread Consumer helper: consume bidirectional stream:
Traceback (most recent call last):
  File ".../google/api_core/grpc_helpers.py", line 51, in error_remapped_callable
    return callable_(*args, **kwargs)
  File ".../grpc/_channel.py", line 342, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../google/cloud/pubsub_v1/subscriber/_consumer.py", line 235, in _blocking_consume
    for response in response_generator:
  File ".../grpc/_channel.py", line 348, in __next__
    return self._next()
  File ".../google/api_core/grpc_helpers.py", line 53, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "${HOME}/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "${HOME}/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File ".../google/cloud/pubsub_v1/subscriber/_consumer.py", line 249, in _blocking_consume
    self._policy.on_exception(exc)
  File ".../google/cloud/pubsub_v1/subscriber/policy/thread.py", line 163, in on_exception
    self._future.set_exception(exception)
  File ".../google/cloud/pubsub_v1/futures.py", line 159, in set_exception
    raise RuntimeError('set_exception can only be called once.')
RuntimeError: set_exception can only be called once.
```

This is **PROGRESS**! It shows `67.833` seconds after the **first** `UNAVAILABLE`, a second one occurs and also tries to call `set_exception()`.

This reveals the **real bug**: the subscriber doesn't exit after the first call to `set_exception()`.

[1]: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/f35c1b7c39a39028b706901cbe2ea9b7c5cceade/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py#L161-L162

----

- FOOTNOTE 1: This is what remains after #4234.
- FOOTNOTE 2: The 67 second gap between `UNAVAILABLE` agrees with what we've previously seen. That is, after ~60 seconds of inactivity, a gRPC client will raise a local `UNAVAILABLE` (i.e. no round trip to the backend) and then it will become magically usable immediately after raising.


Discussion:
I also got this if I try to listen on a subscription that doesn't exist, FYI.

```python
Listening for messages on projects/python-docs-samples-tests/subscriptions/my-subs
Exception in thread Consumer helper: consume bidirectional stream:
Traceback (most recent call last):
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 51, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/grpc/_channel.py", line 342, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.NOT_FOUND, Resource not found (resource=my-subs).)>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_consumer.py", line 235, in _blocking_consume
    for response in response_generator:
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/grpc/_channel.py", line 348, in __next__
    return self._next()
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 53, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Resource not found (resource=my-subs).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jonwayne/.pyenv/versions/3.6.0/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Users/jonwayne/.pyenv/versions/3.6.0/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_consumer.py", line 249, in _blocking_consume
    self._policy.on_exception(exc)
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/policy/thread.py", line 162, in on_exception
    self._future.set_exception(exception)
  File "/Users/jonwayne/workspace/python-docs-samples/pubsub/cloud-client/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/futures.py", line 159, in set_exception
    raise RuntimeError('set_exception can only be called once.')
RuntimeError: set_exception can only be called once.
```
Good data, thanks Jon!
@jonparrott your example will trigger two `NotFound` exceptions ~100ms apart. More importantly, this gap is too small for the `active = False` switch to take place and so we still have the "set_exception can only be called once" issue. (Thanks for providing such an easy to use case!)
I think if there's any none retry-able exception occurred, the helper loop should just exit immediately by calling stop_consume. 
@shileiyu I totally agree with you. I am currently wrestling this issue. The problem is that if it calls `stop_comsuming()`, then the current thread will try to `join()` itself. So I am trying to figure out the right thing to do (i.e. set `self.active = False`, set the `self._exiting` event and then signal to some other thread that this one is a zombie)
@dhermes 
Maybe just break out of the loop?

BTW: I don't think the KeyboardInterrupt would ever happen on the helper thread, not only because only main thread would receive the system signals, but also if KeyboardInterrupt ever occurred before, the thread should stuck as well.

> Maybe just break out of the loop?

We can effectively do this by setting `self.active = False` and calling `self._exiting.set()`, but that still leaves a zombie thread.

> BTW: I don't think the KeyboardInterrupt would ever happen

I haven't been able to trigger the `KeyboardInterrupt` branch, so that doesn't surprise me.

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_consumer.py b/pubsub/google/cloud/pubsub_v1/subscriber/_consumer.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_consumer.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_consumer.py
@@ -242,10 +242,10 @@ def _blocking_consume(self):
                 # case, break out of the while loop and exit this thread.
                 _LOGGER.debug('Clean RPC loop exit signalled consumer exit.')
                 break
-            except KeyboardInterrupt:
-                self.stop_consuming()
             except Exception as exc:
-                self.active = self._policy.on_exception(exc)
+                recover = self._policy.on_exception(exc)
+                if not recover:
+                    self.stop_consuming()
 
     def start_consuming(self):
         """Start consuming the stream."""
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_helper_threads.py b/pubsub/google/cloud/pubsub_v1/subscriber/_helper_threads.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_helper_threads.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_helper_threads.py
@@ -40,6 +40,19 @@
 STOP = uuid.uuid4()
 
 
+def _current_thread():
+    """Get the currently active thread.
+
+    This is provided as a test helper so that it can be mocked easily.
+    Mocking ``threading.current_thread()`` directly may have unintended
+    consequences on code that relies on it.
+
+    Returns:
+        threading.Thread: The current thread.
+    """
+    return threading.current_thread()
+
+
 class HelperThreadRegistry(object):
     def __init__(self):
         self._helper_threads = {}
@@ -84,6 +97,16 @@ def stop(self, name):
         if helper_thread is None:
             return
 
+        if helper_thread.thread is _current_thread():
+            # The current thread cannot ``join()`` itself but it can
+            # still send a signal to stop.
+            _LOGGER.debug('Cannot stop current thread %s', name)
+            helper_thread.queue.put(STOP)
+            # We return and stop short of ``pop()``-ing so that the
+            # thread that invoked the current helper can properly stop
+            # it.
+            return
+
         # Join the thread if it is still alive.
         if helper_thread.thread.is_alive():
             _LOGGER.debug('Stopping helper thread %s', name)
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py b/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
@@ -180,12 +180,12 @@ def on_exception(self, exception):
         """Handle the exception.
 
         If the exception is one of the retryable exceptions, this will signal
-        to the consumer thread that it should remain active.
+        to the consumer thread that it should "recover" from the failure.
 
         This will cause the stream to exit when it returns :data:`False`.
 
         Returns:
-            bool: Indicates if the caller should remain active or shut down.
+            bool: Indicates if the caller should recover or shut down.
             Will be :data:`True` if the ``exception`` is "acceptable", i.e.
             in a list of retryable / idempotent exceptions.
         """


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_consumer.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_consumer.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_consumer.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_consumer.py
@@ -76,19 +76,6 @@ def test_blocking_consume():
             assert on_res.mock_calls[1][1][1] == mock.sentinel.B
 
 
-def test_blocking_consume_keyboard_interrupt():
-    consumer = create_consumer()
-    Policy = type(consumer._policy)
-
-    # Establish that we get responses until we are sent the exiting event.
-    with mock.patch.object(Policy, 'call_rpc', autospec=True) as call_rpc:
-        call_rpc.return_value = (mock.sentinel.A, mock.sentinel.B)
-        with mock.patch.object(Policy, 'on_response', autospec=True) as on_res:
-            on_res.side_effect = KeyboardInterrupt
-            consumer._blocking_consume()
-            on_res.assert_called_once_with(consumer._policy, mock.sentinel.A)
-
-
 class OnException(object):
 
     def __init__(self, exiting_event, acceptable=None):
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_helper_threads.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_helper_threads.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_helper_threads.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_helper_threads.py
@@ -36,6 +36,29 @@ def test_stop_noop():
     assert len(registry._helper_threads) == 0
 
 
+@mock.patch.object(
+    _helper_threads, '_current_thread', return_value=mock.sentinel.thread)
+def test_stop_current_thread(_current_thread):
+    registry = _helper_threads.HelperThreadRegistry()
+    queue_ = mock.Mock(spec=('put',))
+
+    name = 'here'
+    registry._helper_threads[name] = _helper_threads._HelperThread(
+        name=name,
+        queue=queue_,
+        thread=_current_thread.return_value,
+    )
+    assert list(registry._helper_threads.keys()) == [name]
+    registry.stop(name)
+    # Make sure it hasn't been removed from the registry ...
+    assert list(registry._helper_threads.keys()) == [name]
+    # ... but it did receive the STOP signal.
+    queue_.put.assert_called_once_with(_helper_threads.STOP)
+
+    # Verify that our mock was only called once.
+    _current_thread.assert_called_once_with()
+
+
 def test_stop_dead_thread():
     registry = _helper_threads.HelperThreadRegistry()
     registry._helper_threads['foo'] = _helper_threads._HelperThread(