Repository: googleapis/google-cloud-python

Issue:
PubSub: Sub client not retrieving message again if it doesn't ack
I'm experiencing an issue where messages are not retrieved again when a message is not ack'ed. 

In the below code, the self.sub is opened and the main thread sleeps. When the process method fails, the message is not ack'ed and I am expecting it to be resent in the ack deadline of the subscription (10 seconds this case). However, the message is never receieved again in this subclient, unless I completely restart the subclient and open a new callback on the same subscription. The subscription continues to listen to new messages though.

In other threads (such as this one https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4615) and in the docs, it mentions using `future.result()` or `future.exception()`, but both of these just block and never show a problem has occured.

What is the best way to fix this sort of problem? Additionally, is there a way to get the error to propagate? 

A snippet:
```
import random

from google import auth
from google.auth import exceptions
from google.auth.credentials import AnonymousCredentials
from google.cloud import pubsub


class TestProcessor:

    def process_message(self, message):
        print('processing...')
        if random.randint(0, 1): # Some unforeseen error
            raise ValueError
        print(message)
        message.ack()


class GoogleSubClient:

    def __init__(self, project_id, topic, subscription, message_processor, max_messages=10):
        try:
            credentials, _ = auth.default()
        except exceptions.DefaultCredentialsError:  # For running on emulator
            credentials = AnonymousCredentials()
        self._subscriber = pubsub.SubscriberClient(credentials=credentials)
        self.sub = None
        self.future = None
        self._flow = pubsub.types.FlowControl(max_messages=max_messages)
        self._topic = self._subscriber.topic_path(project=project_id, topic=topic)
        self._subscription = self._subscriber.subscription_path(project=project_id, subscription=subscription)
        self._message_processor = message_processor

    def run(self):
        self.sub = self._subscriber.subscribe(subscription=self._subscription, flow_control=self._flow)
        self.future = self.sub.open(self._message_processor.process_message)
```

pubsub version:
```
Name: google-cloud-pubsub
Version: 0.32.0
```


Discussion:
@kir-titievsky is this expected behavior?

I have a hunch that because we manage leasing messages, a top-level exception means that the client holds onto the message indefinitely.
Right -- since neither the ack nor a nack would ever get called.  If you restart the client, there is no longer anything extending the leases and messages are eventually re-delivered.  

So the "solution" is to: 
<pre>
        if random.randint(0, 1): # Some unforeseen error
            message.nack()
            raise ValueError
</pre>

Is there a better way to handle errors more generically?  Meaning always call message.nack() on callback failure?  

We have a solution posted here: https://cloud.google.com/pubsub/docs/pull#subscriber-error-listener-python but now that I look at it, it appears that may not work. 


Yeah we can actually wrap the user callback in something like this:

```python
try:
    user_callback(message)
except Exception:
    message.nack()
    raise
```

If you think that's the right behavior?
I do.  Otherwise messages get stuck.  I wonder if we need to do something like this in other client libraries.  @mdietz94, thoughts?
I believe we already do this, certainly it's how we did things in Java.
Thanks for the quick responses.

Adding something like:
```
class GoogleSubClient:

    def __init__(self, project_id, topic, subscription, message_processor, max_messages=10):
        try:
            credentials, _ = auth.default()
        except exceptions.DefaultCredentialsError:  # For running on emulator
            credentials = AnonymousCredentials()
        self._subscriber = pubsub.SubscriberClient(credentials=credentials)
        self.sub = None
        self.future = None
        self._flow = pubsub.types.FlowControl(max_messages=max_messages)
        self._topic = self._subscriber.topic_path(project=project_id, topic=topic)  # pylint: disable=no-member
        self._subscription = self._subscriber.subscription_path(project=project_id, subscription=subscription)  # pylint: disable=no-member
        self._message_processor = message_processor

    def callback_wrapper(self, message):
        try:
            self._message_processor.process_message(message)
        except Exception:
            message.nack()
            raise  # Or maybe just log this, otherwise we don't see it

    def run(self):
        self.sub = self._subscriber.subscribe(subscription=self._subscription, flow_control=self._flow)
        self.future = self.sub.open(self.callback_wrapper)
```
Into the subclient by wrapping the callback, fixes the issue of redelivery on exception. However, the redelivery also doesn't occur if `message.ack()` isn't called at all, or isn't called for > messageAckTime.

If the change the processor to:
```
class TestProcessor:

    @staticmethod
    def process_message(message):
        print('processing...')
        if random.randint(0, 1):
            time.sleep(100)
        print(message)
        message.ack()
```
The messages are not redelivered in that sleep time, which is something (maybe incorrectly) that I would expect.
You should not expect that. Potentially we want a max processing time after
which we stop mod asking in case we leak the ack handler, but that should
be several hours. We have users that take an hour to process each message.

On Fri, Mar 9, 2018, 5:15 AM liam-j-bennett <notifications@github.com>
wrote:

> Thanks for the quick responses.
>
> Adding something like:
>
> class GoogleSubClient:
>
>     def __init__(self, project_id, topic, subscription, message_processor, max_messages=10):
>         try:
>             credentials, _ = auth.default()
>         except exceptions.DefaultCredentialsError:  # For running on emulator
>             credentials = AnonymousCredentials()
>         self._subscriber = pubsub.SubscriberClient(credentials=credentials)
>         self.sub = None
>         self.future = None
>         self._flow = pubsub.types.FlowControl(max_messages=max_messages)
>         self._topic = self._subscriber.topic_path(project=project_id, topic=topic)  # pylint: disable=no-member
>         self._subscription = self._subscriber.subscription_path(project=project_id, subscription=subscription)  # pylint: disable=no-member
>         self._message_processor = message_processor
>
>     def callback_wrapper(self, message):
>         try:
>             self._message_processor.process_message(message)
>         except Exception:
>             message.nack()
>             raise  # Or maybe just log this, otherwise we don't see it
>
>     def run(self):
>         self.sub = self._subscriber.subscribe(subscription=self._subscription, flow_control=self._flow)
>         self.future = self.sub.open(self.callback_wrapper)
>
> Into the subclient by wrapping the callback, fixes the issue of redelivery
> on exception. However, the redelivery also doesn't occur if message.ack()
> isn't called at all, or isn't called for > messageAckTime.
>
> If the change the processor to:
>
> class TestProcessor:
>
>     @staticmethod
>     def process_message(message):
>         print('processing...')
>         if random.randint(0, 1):
>             time.sleep(100)
>         print(message)
>         message.ack()
>
> The messages are not redelivered in that sleep time, which is something
> (maybe incorrectly) that I would expect.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5005#issuecomment-371771576>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABvXagjHXCEg0W0LKgRtrHPHAkZkd8M7ks5tclYogaJpZM4SjBiY>
> .
>

Yeah, so `messageAckTime` doesn't necessary mean that it will be redelivered after that. The subscriber "leases" the messages it receives and calls modify ack deadline on them in the background.

So it seems there's two actionable items here:

1. We need to wrap the callback in an error handler to nack them.
2. We need to have some sort of overall deadline where we nack messages instead of leasing them indefinitely.  (Should this be configurable?)

@kir-titievsky @mdietz94 does that sound right to y'all?
Yes. 2 should definitely be configurable, with a default maybe of 2 hours

On Fri, Mar 9, 2018, 12:08 PM Jon Wayne Parrott <notifications@github.com>
wrote:

> Yeah, so messageAckTime doesn't necessary mean that it will be
> redelivered after that. The subscriber "leases" the messages it receives
> and calls modify ack deadline on them in the background.
>
> So it seems there's two actionable items here:
>
>    1. We need to wrap the callback in an error handler to nack them.
>    2. We need to have some sort of overall deadline where we nack
>    messages instead of leasing them indefinitely. (Should this be
>    configurable?)
>
> @kir-titievsky <https://github.com/kir-titievsky> @mdietz94
> <https://github.com/mdietz94> does that sounds right to y'all?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5005#issuecomment-371876996>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABvXavTd1WHQ6HnjbTYKykjoxA_OVa0zks5tcrcbgaJpZM4SjBiY>
> .
>

amen to all, except that it's not obvious to me how I change the max lease
deadline.

On Fri, Mar 9, 2018 at 12:15 PM Max Dietz <notifications@github.com> wrote:

> Yes. 2 should definitely be configurable, with a default maybe of 2 hours
>
> On Fri, Mar 9, 2018, 12:08 PM Jon Wayne Parrott <notifications@github.com>
> wrote:
>
> > Yeah, so messageAckTime doesn't necessary mean that it will be
> > redelivered after that. The subscriber "leases" the messages it receives
> > and calls modify ack deadline on them in the background.
> >
> > So it seems there's two actionable items here:
> >
> > 1. We need to wrap the callback in an error handler to nack them.
> > 2. We need to have some sort of overall deadline where we nack
> > messages instead of leasing them indefinitely. (Should this be
> > configurable?)
> >
> > @kir-titievsky <https://github.com/kir-titievsky> @mdietz94
> > <https://github.com/mdietz94> does that sounds right to y'all?
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > <
> https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5005#issuecomment-371876996
> >,
> > or mute the thread
> > <
> https://github.com/notifications/unsubscribe-auth/ABvXavTd1WHQ6HnjbTYKykjoxA_OVa0zks5tcrcbgaJpZM4SjBiY
> >
> > .
> >
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5005#issuecomment-371879049>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ARrMFlYrkmuLTNdMcYrSNYCI2OrJ935oks5tcriqgaJpZM4SjBiY>
> .
>


-- 
Kir Titievsky | Product Manager | Google Cloud Pub/Sub
<https://cloud.google.com/pubsub/overview>

I'm guessing it makes the most sense to add yet another flow control setting.

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/policy/base.py b/pubsub/google/cloud/pubsub_v1/subscriber/policy/base.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/policy/base.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/policy/base.py
@@ -59,6 +59,10 @@
     ['ack_id', 'byte_size'],
 )
 
+_LeasedMessage = collections.namedtuple(
+    '_LeasedMessage',
+    ['added_time', 'size'])
+
 
 @six.add_metaclass(abc.ABCMeta)
 class BasePolicy(object):
@@ -92,8 +96,6 @@ class BasePolicy(object):
                 your own dictionary class, ensure this assumption holds
                 or you will get strange behavior.
     """
-
-    _managed_ack_ids = None
     _RETRYABLE_STREAM_ERRORS = (
         exceptions.DeadlineExceeded,
         exceptions.ServiceUnavailable,
@@ -109,6 +111,10 @@ def __init__(self, client, subscription,
         self._future = None
         self.flow_control = flow_control
         self.histogram = _histogram.Histogram(data=histogram_data)
+        """.Histogram: the histogram tracking ack latency."""
+        self.leased_messages = {}
+        """dict[str, float]: A mapping of ack IDs to the local time when the
+            ack ID was initially leased in seconds since the epoch."""
 
         # These are for internal flow control tracking.
         # They should not need to be used by subclasses.
@@ -144,17 +150,6 @@ def future(self):
         """
         return self._future
 
-    @property
-    def managed_ack_ids(self):
-        """Return the ack IDs currently being managed by the policy.
-
-        Returns:
-            set: The set of ack IDs being managed.
-        """
-        if self._managed_ack_ids is None:
-            self._managed_ack_ids = set()
-        return self._managed_ack_ids
-
     @property
     def subscription(self):
         """Return the subscription.
@@ -182,7 +177,7 @@ def _load(self):
             float: The load value.
         """
         return max([
-            len(self.managed_ack_ids) / self.flow_control.max_messages,
+            len(self.leased_messages) / self.flow_control.max_messages,
             self._bytes / self.flow_control.max_bytes,
             self._consumer.pending_requests / self.flow_control.max_requests
         ])
@@ -252,11 +247,10 @@ def drop(self, items):
         # Remove the ack ID from lease management, and decrement the
         # byte counter.
         for item in items:
-            if item.ack_id in self.managed_ack_ids:
-                self.managed_ack_ids.remove(item.ack_id)
+            if self.leased_messages.pop(item.ack_id, None) is not None:
                 self._bytes -= item.byte_size
             else:
-                _LOGGER.debug('Item %s wasn\'t managed', item.ack_id)
+                _LOGGER.debug('Item %s was not managed.', item.ack_id)
 
         if self._bytes < 0:
             _LOGGER.debug(
@@ -290,7 +284,7 @@ def get_initial_request(self, ack_queue=False):
         # Any ack IDs that are under lease management and not being acked
         # need to have their deadline extended immediately.
         ack_ids = set()
-        lease_ids = self.managed_ack_ids
+        lease_ids = set(self.leased_messages.keys())
         if ack_queue:
             ack_ids = self._ack_on_resume
             lease_ids = lease_ids.difference(ack_ids)
@@ -321,8 +315,10 @@ def lease(self, items):
         for item in items:
             # Add the ack ID to the set of managed ack IDs, and increment
             # the size counter.
-            if item.ack_id not in self.managed_ack_ids:
-                self.managed_ack_ids.add(item.ack_id)
+            if item.ack_id not in self.leased_messages:
+                self.leased_messages[item.ack_id] = _LeasedMessage(
+                    added_time=time.time(),
+                    size=item.byte_size)
                 self._bytes += item.byte_size
             else:
                 _LOGGER.debug(
@@ -361,22 +357,36 @@ def maintain_leases(self):
             p99 = self.histogram.percentile(99)
             _LOGGER.debug('The current p99 value is %d seconds.', p99)
 
+            # Drop any leases that are well beyond max lease time. This
+            # ensures that in the event of a badly behaving actor, we can
+            # drop messages and allow Pub/Sub to resend them.
+            cutoff = time.time() - self.flow_control.max_lease_duration
+            to_drop = [
+                DropRequest(ack_id, item.size)
+                for ack_id, item
+                in six.iteritems(self.leased_messages)
+                if item.added_time < cutoff]
+
+            if to_drop:
+                _LOGGER.warning(
+                    'Dropping %s items because they were leased too long.',
+                    len(to_drop))
+                self.drop(to_drop)
+
             # Create a streaming pull request.
             # We do not actually call `modify_ack_deadline` over and over
             # because it is more efficient to make a single request.
-            ack_ids = list(self.managed_ack_ids)
-            _LOGGER.debug('Renewing lease for %d ack IDs.', len(ack_ids))
+            ack_ids = list(self.leased_messages.keys())
             if ack_ids:
-                request = types.StreamingPullRequest(
-                    modify_deadline_ack_ids=ack_ids,
-                    modify_deadline_seconds=[p99] * len(ack_ids),
-                )
+                _LOGGER.debug('Renewing lease for %d ack IDs.', len(ack_ids))
+
                 # NOTE: This may not work as expected if ``consumer.active``
                 #       has changed since we checked it. An implementation
                 #       without any sort of race condition would require a
                 #       way for ``send_request`` to fail when the consumer
                 #       is inactive.
-                self._consumer.send_request(request)
+                self.modify_ack_deadline([
+                    ModAckRequest(ack_id, p99) for ack_id in ack_ids])
 
             # Now wait an appropriate period of time and do this again.
             #
diff --git a/pubsub/google/cloud/pubsub_v1/types.py b/pubsub/google/cloud/pubsub_v1/types.py
--- a/pubsub/google/cloud/pubsub_v1/types.py
+++ b/pubsub/google/cloud/pubsub_v1/types.py
@@ -54,7 +54,8 @@
 FlowControl = collections.namedtuple(
     'FlowControl',
     ['max_bytes', 'max_messages', 'resume_threshold', 'max_requests',
-     'max_request_batch_size', 'max_request_batch_latency'],
+     'max_request_batch_size', 'max_request_batch_latency',
+     'max_lease_duration'],
 )
 FlowControl.__new__.__defaults__ = (
     psutil.virtual_memory().total * 0.2,  # max_bytes: 20% of total RAM
@@ -63,6 +64,7 @@
     100,                                  # max_requests: 100
     100,                                  # max_request_batch_size: 100
     0.01,                                 # max_request_batch_latency: 0.01s
+    2 * 60 * 60,                          # max_lease_duration: 2 hours.
 )
 
 


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_base.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_base.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_base.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_base.py
@@ -67,15 +67,15 @@ def test_get_initial_request():
     assert initial_request.stream_ack_deadline_seconds == 10
 
 
-def test_managed_ack_ids():
+def test_leased_messagess():
     policy = create_policy()
 
     # Ensure we always get a set back, even if the property is not yet set.
-    managed_ack_ids = policy.managed_ack_ids
-    assert isinstance(managed_ack_ids, set)
+    leased_messages = policy.leased_messages
+    assert isinstance(leased_messages, dict)
 
     # Ensure that multiple calls give the same actual object back.
-    assert managed_ack_ids is policy.managed_ack_ids
+    assert leased_messages is policy.leased_messages
 
 
 def test_subscription():
@@ -130,25 +130,25 @@ def test_call_rpc():
 
 def test_drop():
     policy = create_policy()
-    policy.managed_ack_ids.add('ack_id_string')
+    policy.leased_messages['ack_id_string'] = 0
     policy._bytes = 20
     policy.drop([base.DropRequest(ack_id='ack_id_string', byte_size=20)])
-    assert len(policy.managed_ack_ids) == 0
+    assert len(policy.leased_messages) == 0
     assert policy._bytes == 0
 
     # Do this again to establish idempotency.
     policy.drop([base.DropRequest(ack_id='ack_id_string', byte_size=20)])
-    assert len(policy.managed_ack_ids) == 0
+    assert len(policy.leased_messages) == 0
     assert policy._bytes == 0
 
 
 @mock.patch.object(base, '_LOGGER', spec=logging.Logger)
 def test_drop_unexpected_negative(_LOGGER):
     policy = create_policy()
-    policy.managed_ack_ids.add('ack_id_string')
+    policy.leased_messages['ack_id_string'] = 0
     policy._bytes = 0
     policy.drop([base.DropRequest(ack_id='ack_id_string', byte_size=20)])
-    assert len(policy.managed_ack_ids) == 0
+    assert len(policy.leased_messages) == 0
     assert policy._bytes == 0
     _LOGGER.debug.assert_called_once_with(
         'Bytes was unexpectedly negative: %d', -20)
@@ -161,7 +161,7 @@ def test_drop_below_threshold():
     the flow control thresholds, it should resume.
     """
     policy = create_policy()
-    policy.managed_ack_ids.add('ack_id_string')
+    policy.leased_messages['ack_id_string'] = 0
     num_bytes = 20
     policy._bytes = num_bytes
     consumer = policy._consumer
@@ -290,15 +290,55 @@ def trigger_inactive(seconds):
         sleep.assert_called()
 
 
+@mock.patch.object(time, 'time', autospec=True)
+@mock.patch.object(time, 'sleep', autospec=True)
+def test_maintain_leases_outdated_items(sleep, time):
+    policy = create_policy()
+    policy._consumer._stopped.clear()
+
+    # Add these items at the beginning of the timeline
+    time.return_value = 0
+    policy.lease([
+        base.LeaseRequest(ack_id='ack1', byte_size=50)])
+
+    # Add another item at towards end of the timeline
+    time.return_value = policy.flow_control.max_lease_duration - 1
+    policy.lease([
+        base.LeaseRequest(ack_id='ack2', byte_size=50)])
+
+    # Now make sure time reports that we are at the end of our timeline.
+    time.return_value = policy.flow_control.max_lease_duration + 1
+
+    # Mock the sleep object.
+    def trigger_inactive(seconds):
+        assert 0 < seconds < 10
+        policy._consumer._stopped.set()
+
+    sleep.side_effect = trigger_inactive
+
+    # Also mock the consumer, which sends the request.
+    with mock.patch.object(policy._consumer, 'send_request') as send:
+        policy.maintain_leases()
+
+    # Only ack2 should be renewed. ack1 should've been dropped
+    send.assert_called_once_with(types.StreamingPullRequest(
+        modify_deadline_ack_ids=['ack2'],
+        modify_deadline_seconds=[10],
+    ))
+    assert len(policy.leased_messages) == 1
+
+    sleep.assert_called()
+
+
 def test_lease():
     policy = create_policy()
     policy.lease([base.LeaseRequest(ack_id='ack_id_string', byte_size=20)])
-    assert len(policy.managed_ack_ids) == 1
+    assert len(policy.leased_messages) == 1
     assert policy._bytes == 20
 
     # Do this again to prove idempotency.
     policy.lease([base.LeaseRequest(ack_id='ack_id_string', byte_size=20)])
-    assert len(policy.managed_ack_ids) == 1
+    assert len(policy.leased_messages) == 1
     assert policy._bytes == 20