Repository: googleapis/google-cloud-python

Issue:
PubSub: dispatch_callback errors after cancel() of a subscription
OS: Linux 4.17.0-3rodete2-amd64, Debian 4.17.17-1rodete2
Python 2.7.13
google-cloud-pubsub 0.35.4

## Stacktrace 1
```
ERROR:google.cloud.pubsub_v1.subscriber._protocol.helper_threads:Error in queue callback worker: 'NoneType' object has no attribute 'is_paused'
Traceback (most recent call last):
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/helper_threads.py", line 112, in __call__
    self._callback(items)
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py", line 101, in dispatch_callback
    self.ack(batched_commands.pop(requests.AckRequest))
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py", line 124, in ack
    self.drop(items)
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py", line 133, in drop
    self._manager.maybe_resume_consumer()
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 224, in maybe_resume_consumer
    if not self._consumer.is_paused:
AttributeError: 'NoneType' object has no attribute 'is_paused'
```

## Stacktrace 2
```
ERROR:google.cloud.pubsub_v1.subscriber._protocol.helper_threads:Error in queue callback worker: 'NoneType' object has no attribute 'remove'
Traceback (most recent call last):
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/helper_threads.py", line 112, in __call__
    self._callback(items)
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py", line 101, in dispatch_callback
    self.ack(batched_commands.pop(requests.AckRequest))
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py", line 124, in ack
    self.drop(items)
  File "/usr/local/google/home/ehudm/virtualenvs/beamenv/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py", line 132, in drop
    self._manager.leaser.remove(items)
AttributeError: 'NoneType' object has no attribute 'remove'
```

## Steps to reproduce
```py
event = threading.Event()
def message_cb(msg):
  msg.ack()
  event.set()
sub_client = pubsub.SubscriberClient()
sub_future = sub_client.subscribe(sub_name, message_cb)
event.wait()
sub_future.cancel()
```

This subscription should be canceled approximately after the first message. In my test there are two messages sent to the topic and `event.set()` is called after the second.

I realize that calling cancel() so soon prevents the ack from being sent. Is there a cleaner way to stop the subscription?


Discussion:
@theacodes I'm not sure how cancellation should work.  Can you clairify?
Cancellation tries to be as "fast" as possible, as waiting for all currently held messages to resolve is basically the halting problem. We should probably update the docs there to mention that.

More serious is the fact that there doesn't seem to be a "clean" shutdown here. I'm going to assign over to @crwilcox to handle, but really the fix should be straightforward (be more defensive when trying to invoke things on the streaming pull manager that could be in the process of shutting down)
@theacodes, @crwilcox Should this be categorized as a bug, then?
@plamut, can you please take a look at this?
The first manifestation of the issue (stacktrace 1) is the issue #5751, which has been fixed and released. However, I was able to reproduce the second issue (stacktrace 2) even with the latest PubSub version (`0.41.0` at the time of writing) - an exception can occur in the background threads upon shutdown, particularly in the callback dispatcher thread.

It appears that some extra checks need to be added when interacting with the streaming pull manager, because the latter could be in the process of shutting down (as @theacodes already mentioned).

Working on a fix.

**Update:** Fixed, opened a PR.

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
@@ -428,12 +428,23 @@ def close(self, reason=None):
             _LOGGER.debug("Stopping scheduler.")
             self._scheduler.shutdown()
             self._scheduler = None
+
+            # Leaser and dispatcher reference each other through the shared
+            # StreamingPullManager instance, i.e. "self", thus do not set their
+            # references to None until both have been shut down.
+            #
+            # NOTE: Even if the dispatcher operates on an inactive leaser using
+            # the latter's add() and remove() methods, these have no impact on
+            # the stopped leaser (the leaser is never again re-started). Ditto
+            # for the manager's maybe_resume_consumer() / maybe_pause_consumer(),
+            # because the consumer gets shut down first.
             _LOGGER.debug("Stopping leaser.")
             self._leaser.stop()
-            self._leaser = None
             _LOGGER.debug("Stopping dispatcher.")
             self._dispatcher.stop()
             self._dispatcher = None
+            # dispatcher terminated, OK to dispose the leaser reference now
+            self._leaser = None
             _LOGGER.debug("Stopping heartbeater.")
             self._heartbeater.stop()
             self._heartbeater = None


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
@@ -13,6 +13,8 @@
 # limitations under the License.
 
 import logging
+import threading
+import time
 import types as stdlib_types
 
 import mock
@@ -511,6 +513,50 @@ def test_close_idempotent():
     assert scheduler.shutdown.call_count == 1
 
 
+class FakeDispatcher(object):
+    def __init__(self, manager, error_callback):
+        self._manager = manager
+        self._error_callback = error_callback
+        self._thread = None
+        self._stop = False
+
+    def start(self):
+        self._thread = threading.Thread(target=self._do_work)
+        self._thread.daemon = True
+        self._thread.start()
+
+    def stop(self):
+        self._stop = True
+        self._thread.join()
+        self._thread = None
+
+    def _do_work(self):
+        while not self._stop:
+            try:
+                self._manager.leaser.add([mock.Mock()])
+            except Exception as exc:
+                self._error_callback(exc)
+            time.sleep(0.1)
+
+        # also try to interact with the leaser after the stop flag has been set
+        try:
+            self._manager.leaser.remove([mock.Mock()])
+        except Exception as exc:
+            self._error_callback(exc)
+
+
+def test_close_no_dispatcher_error():
+    manager, _, _, _, _, _ = make_running_manager()
+    error_callback = mock.Mock(name="error_callback")
+    dispatcher = FakeDispatcher(manager=manager, error_callback=error_callback)
+    manager._dispatcher = dispatcher
+    dispatcher.start()
+
+    manager.close()
+
+    error_callback.assert_not_called()
+
+
 def test_close_callbacks():
     manager, _, _, _, _, _ = make_running_manager()