Repository: googleapis/google-cloud-python

Issue:
Pub/Sub: Send modify_ack_deadline requests in chunks
Currently leaser is limited in total number of messages (<2000?) which is a problem for use cases with many small messages. 

Same issue was solved in nodejs library here: https://github.com/googleapis/nodejs-pubsub/pull/65


Discussion:
More context: lease tries to ack all messages at the same time which results in the following error

```
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 335, in send
    self._send_unary_request(request)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 322, in _send_unary_request
    ack_deadline_seconds=deadline,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/_gapic.py", line 40, in <lambda>
    fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw)  # noqa
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py", line 821, in modify_ack_deadline
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/usr/local/lib/python2.7/dist-packages/six.py", line 737, in raise_from
    raise value
InvalidArgument: 400 Request payload size exceeds the limit: 524288 bytes.
```
@mwytock Thanks for the report!

By default the streaming pull is configured to lease a maximum of 100 messages, or 100 MiB of messages, whichever limit is hit first.
https://github.com/googleapis/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/types.py#L71-L73

Any messages received on top of that are put in a waiting buffer, and the message stream gets paused. Sending 100 or less ACK IDs should work fine, thus I'm wondering if you maybe use custom `FlowControl` settings? Or send ACK and mofiyAckDeadline requests manually?

Also, are there any other interesting entries in the log referring to pausing/resuming the message stream? Things like the following:

- _"Message backlog over load at X.YZ, pausing."_
- _"Current load is X.YZ, resuming consumer."_
- _"Did not resume, current load is X.YZ."_

Any such info would be useful, thanks!
Hi Peter,

Yes you are right - 100 ack IDs is no problem, this only happens when you
modify flow control to allow more simultaneous messages in flight, 2000+.

Perhaps this isn't the intended use for the async pull code path; we
actually worked around the issue by switching to sync pull and managing the
the modify ack request messages ourselves which seems to work a lot better.
If that's the case feel free to close with working as intended - but I
might suggest raising if some crazy person comes along and tries to set max
messages too high in Flow control as we did!




On Tue, Aug 27, 2019, 6:39 AM Peter Lamut <notifications@github.com> wrote:

> @mwytock <https://github.com/mwytock> Thanks for the report!
>
> By default the streaming pull is configured to lease a maximum of 100
> messages, or 100 MiB of messages, whichever limit is hit first.
>
> https://github.com/googleapis/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/types.py#L71-L73
>
> Any messages received on top of that are put in a waiting buffer, and the
> message stream gets paused. Sending 100 or less ACK IDs should work fine,
> thus I'm wondering if you maybe use custom FlowControl settings? Or send
> ACK and mofiyAckDeadline requests manually?
>
> Also, are there any other interesting entries in the log referring to
> pausing/resuming the message stream? Things like the following:
>
>    - *"Message backlog over load at X.YZ, pausing."*
>    - *"Current load is X.YZ, resuming consumer."*
>    - *"Did not resume, current load is X.YZ."*
>
> Any such info would be useful, thanks!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/googleapis/google-cloud-python/issues/9103?email_source=notifications&email_token=AAEQPVW2NEYIXU7JVC3LWODQGUVBDA5CNFSM4IPUQOKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5HYPKA#issuecomment-525305768>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAEQPVVYIXRSI3P7C2J43RDQGUVBDANCNFSM4IPUQOKA>
> .
>

@mwytock Thanks for the quick response.

It seems that the defaults are fine then, but I agree that if a user can customize this to a high number, the resulting error message should be more descriptive, or the error should not occur at all.

Since the max request size limit is imposed by the backend, a quick way around it would indeed be raising an error if the `FlowControl.max_messages` is set to to high a number.

Alternatively, sending the message(s) in smaller batches could be implemented, similarly to what the Node.js client did.
Sounds right to me!

On a more subjective note, I would add that throwing an error if
max_messages is too high is probably the right thing to do. We later
discovered other info in the docs
(https://cloud.google.com/pubsub/docs/pull "StreamingPull:
Dealing with large backlogs of small messages") which convinced us that we
should switch away from StreamingPull and to use Pull for our use case.

On Tue, Aug 27, 2019 at 7:45 AM Peter Lamut <notifications@github.com>
wrote:

> @mwytock <https://github.com/mwytock> Thanks for the quick response.
>
> It seems that the defaults are fine then, but I agree that if a user can
> customize this to a high number, the resulting error message should be more
> descriptive, or the error should not occur at all.
>
> Since the max request size limit is imposed by the backend, a quick way
> around it would indeed be raising an error if the FlowControl.max_messages
> is set to to high a number.
>
> Alternatively, sending the message in smaller batches could be
> implemented, similarly to what the Node.js client did.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/googleapis/google-cloud-python/issues/9103?email_source=notifications&email_token=AAEQPVVWDDIUBNNHJYJ7IC3QGU4ZVA5CNFSM4IPUQOKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5H7SEQ#issuecomment-525334802>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAEQPVRZOYEAGNSCLHOAGJLQGU4ZVANCNFSM4IPUQOKA>
> .
>

I will definitely put this up for consideration to the product team. There is a threshold beyond which the `max_messages` setting is not really sensible anymore for practical purposes, although on the other hand, consistency in client implementations across different languages is also high on the list.

It's good that we now have this issue documented, thus thanks again for reporting it!

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
@@ -13,9 +13,12 @@
 # limitations under the License.
 
 from __future__ import absolute_import
+from __future__ import division
 
 import collections
+import itertools
 import logging
+import math
 import threading
 
 from google.cloud.pubsub_v1 import types
@@ -34,6 +37,18 @@
 """The maximum amount of time in seconds to wait for additional request items
 before processing the next batch of requests."""
 
+_ACK_IDS_BATCH_SIZE = 2500
+"""The maximum number of ACK IDs to send in a single StreamingPullRequest.
+
+The backend imposes a maximum request size limit of 524288 bytes (512 KiB) per
+acknowledge / modifyAckDeadline request. ACK IDs have a maximum size of 164
+bytes, thus we cannot send more than o 524288/176 ~= 2979 ACK IDs in a single
+StreamingPullRequest message.
+
+Accounting for some overhead, we should thus only send a maximum of 2500 ACK
+IDs at a time.
+"""
+
 
 class Dispatcher(object):
     def __init__(self, manager, queue):
@@ -119,9 +134,16 @@ def ack(self, items):
             if time_to_ack is not None:
                 self._manager.ack_histogram.add(time_to_ack)
 
-        ack_ids = [item.ack_id for item in items]
-        request = types.StreamingPullRequest(ack_ids=ack_ids)
-        self._manager.send(request)
+        # We must potentially split the request into multiple smaller requests
+        # to avoid the server-side max request size limit.
+        ack_ids = (item.ack_id for item in items)
+        total_chunks = int(math.ceil(len(items) / _ACK_IDS_BATCH_SIZE))
+
+        for _ in range(total_chunks):
+            request = types.StreamingPullRequest(
+                ack_ids=itertools.islice(ack_ids, _ACK_IDS_BATCH_SIZE)
+            )
+            self._manager.send(request)
 
         # Remove the message from lease management.
         self.drop(items)
@@ -150,13 +172,18 @@ def modify_ack_deadline(self, items):
         Args:
             items(Sequence[ModAckRequest]): The items to modify.
         """
-        ack_ids = [item.ack_id for item in items]
-        seconds = [item.seconds for item in items]
-
-        request = types.StreamingPullRequest(
-            modify_deadline_ack_ids=ack_ids, modify_deadline_seconds=seconds
-        )
-        self._manager.send(request)
+        # We must potentially split the request into multiple smaller requests
+        # to avoid the server-side max request size limit.
+        ack_ids = (item.ack_id for item in items)
+        seconds = (item.seconds for item in items)
+        total_chunks = int(math.ceil(len(items) / _ACK_IDS_BATCH_SIZE))
+
+        for _ in range(total_chunks):
+            request = types.StreamingPullRequest(
+                modify_deadline_ack_ids=itertools.islice(ack_ids, _ACK_IDS_BATCH_SIZE),
+                modify_deadline_seconds=itertools.islice(seconds, _ACK_IDS_BATCH_SIZE),
+            )
+            self._manager.send(request)
 
     def nack(self, items):
         """Explicitly deny receipt of messages.


Test patch:
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_dispatcher.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_dispatcher.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_dispatcher.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_dispatcher.py
@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import collections
 import threading
 
 from google.cloud.pubsub_v1 import types
@@ -95,6 +96,34 @@ def test_ack_no_time():
     manager.ack_histogram.add.assert_not_called()
 
 
+def test_ack_splitting_large_payload():
+    manager = mock.create_autospec(
+        streaming_pull_manager.StreamingPullManager, instance=True
+    )
+    dispatcher_ = dispatcher.Dispatcher(manager, mock.sentinel.queue)
+
+    items = [
+        # use realistic lengths for ACK IDs (max 176 bytes)
+        requests.AckRequest(ack_id=str(i).zfill(176), byte_size=0, time_to_ack=20)
+        for i in range(5001)
+    ]
+    dispatcher_.ack(items)
+
+    calls = manager.send.call_args_list
+    assert len(calls) == 3
+
+    all_ack_ids = {item.ack_id for item in items}
+    sent_ack_ids = collections.Counter()
+
+    for call in calls:
+        message = call.args[0]
+        assert message.ByteSize() <= 524288  # server-side limit (2**19)
+        sent_ack_ids.update(message.ack_ids)
+
+    assert set(sent_ack_ids) == all_ack_ids  # all messages should have been ACK-ed
+    assert sent_ack_ids.most_common(1)[0][1] == 1  # each message ACK-ed exactly once
+
+
 def test_lease():
     manager = mock.create_autospec(
         streaming_pull_manager.StreamingPullManager, instance=True
@@ -153,6 +182,34 @@ def test_modify_ack_deadline():
     )
 
 
+def test_modify_ack_deadline_splitting_large_payload():
+    manager = mock.create_autospec(
+        streaming_pull_manager.StreamingPullManager, instance=True
+    )
+    dispatcher_ = dispatcher.Dispatcher(manager, mock.sentinel.queue)
+
+    items = [
+        # use realistic lengths for ACK IDs (max 176 bytes)
+        requests.ModAckRequest(ack_id=str(i).zfill(176), seconds=60)
+        for i in range(5001)
+    ]
+    dispatcher_.modify_ack_deadline(items)
+
+    calls = manager.send.call_args_list
+    assert len(calls) == 3
+
+    all_ack_ids = {item.ack_id for item in items}
+    sent_ack_ids = collections.Counter()
+
+    for call in calls:
+        message = call.args[0]
+        assert message.ByteSize() <= 524288  # server-side limit (2**19)
+        sent_ack_ids.update(message.modify_deadline_ack_ids)
+
+    assert set(sent_ack_ids) == all_ack_ids  # all messages should have been MODACK-ed
+    assert sent_ack_ids.most_common(1)[0][1] == 1  # each message MODACK-ed exactly once
+
+
 @mock.patch("threading.Thread", autospec=True)
 def test_start(thread):
     manager = mock.create_autospec(