Repository: googleapis/google-cloud-python

Issue:
pubsub: implement streaming pull receipt
StreamingPull currently has a "protocol" level bug affecting all languages.

TLDR: client libs should send modify ack deadline for all messages upon receiving them.

## Problem description
When streaming, `StreamingPullResponse` can be buffered, either by gRPC itself or by proxy/firewall.
There might be significant delay between the server sending a response and the client library reading it.
For this reason, by the time client library decides it should send a modify ack deadline ("modack"), it might be too late. The message might have already expired and resent.

Experimenting with Java client, we have seen many messages delivered 3-5 times because of this problem.

## What server-side is doing
To fix this, pubsub team is implementing the "receipt" feature.
The server will not start the expiration timer until it receives a receipt from the client [1]. A receipt here is similar to an ack. A receipt notifies the server that the client has *seen* the message, while an ack notifies that the client has finished with it.
This should make the client and server start expiration timer closer to each other.

[1] There's actually a limit on how long the server is willing to wait, so that a message isn't lost forever if the client doesn't send a receipt.

## What the client needs to do
The client should be modified to send these receipts.
For compatibility, these "receipts" are just modack requests. The client should modack messages after seeing it, in addition to what it is currently doing. These modacks can be batched for performance.

## Few more details for the curious
These modacks/receipts aren't strictly necessary for avoiding duplication, but it will help pubsub server expire messages on time. Without them, messages might expire too late.

Each `StreamingPullResponse` can contain multiple messages. In an implementation about to roll out, a modack for any message in the response will serve as receipts for all messages in that response. Still, client libs are encouraged to actually modack all messages since this behavior might change in the future.


Discussion:
We are also seeing duplicated messages and I believe this might be the cause. Is there anything we can do in the meantime to aleviate the problem? Eg set the "Ack deadline" for the subscriptions to a higher value (eg 30 seconds) ?
ISTM the right place to put this is

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/5c0b432499d03291c5781fc626ce8fc3aaaae262/pubsub/google/cloud/pubsub_v1/subscriber/message.py#L97

or

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/5c0b432499d03291c5781fc626ce8fc3aaaae262/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py#L337

It's unclear to me what the new ack deadline should be?
I believe the ack deadline is "whatever a modack would do if it runs". If a modack is scheduled to run 1 minute from now, maybe the recept-modack should be something like 70 seconds. Does this help you in any way?
> It's unclear to me what the new ack deadline should be?

~10 seconds.~ Whatever the `p99` value is.
@chemelnucfin can you please take a look at his implementation?  Please reach out to @pongad for questions about the Java reference implementation.  @jonparrott on point for Python specific.
@lukesneeringer what's a `p99` value?
oh, nvm, I see it.

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py b/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/policy/thread.py
@@ -329,8 +329,20 @@ def on_exception(self, exception):
     def on_response(self, response):
         """Process all received Pub/Sub messages.
 
-        For each message, schedule a callback with the executor.
+        For each message, send a modified acknowledgement request to the
+        server. This prevents expiration of the message due to buffering by
+        gRPC or proxy/firewall. This makes the server and client expiration
+        timer closer to each other thus preventing the message being
+        redelivered multiple times.
+
+        After the messages have all had their ack deadline updated, execute
+        the callback for each message using the executor.
         """
+        items = [
+            base.ModAckRequest(message.ack_id, self.histogram.percentile(99))
+            for message in response.received_messages
+        ]
+        self.modify_ack_deadline(items)
         for msg in response.received_messages:
             _LOGGER.debug(
                 'Using %s to process message with ack_id %s.',


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_thread.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_thread.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_thread.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_policy_thread.py
@@ -209,9 +209,17 @@ def test_on_response():
         ],
     )
 
-    # Actually run the method and prove that executor.submit and
-    # future.add_done_callback were called in the expected way.
-    policy.on_response(response)
+    # Actually run the method and prove that modack and executor.submit
+    # are called in the expected way.
+    modack_patch = mock.patch.object(
+        policy, 'modify_ack_deadline', autospec=True)
+    with modack_patch as modack:
+        policy.on_response(response)
+
+    modack.assert_called_once_with(
+        [base.ModAckRequest('fack', 10),
+         base.ModAckRequest('back', 10)]
+    )
 
     submit_calls = [m for m in executor.method_calls if m[0] == 'submit']
     assert len(submit_calls) == 2