Repository: googleapis/google-cloud-python

Issue:
Pub/Sub: buffered messages delivered multiple times ignoring acknowledgment deadline
I noticed a problem where buffered messages in pubsub library are delivered multiple times ignoring acknowledgment deadline. I have a subscription with acknowledgment deadline set to 5 minutes but messages are delivered multiple times to the same subscriber after about 30 seconds.

My code example starts a subscriber client with `max_messages` set to 1. When two messages are sent to the subscriptions, the second message is always duplicated. This same behavior also happens with higher `max_messages` values. For example, using the default value 100 for `max_messages`, when 150 messages are sent to the subscription, some of the messages are duplicated if the processing of the first 100 messages takes more than 30 seconds.

I understand that with large backlog of small messages, the messages can get redelivered ([Dealing with large backlogs of small messages](https://cloud.google.com/pubsub/docs/pull#streamingpull_dealing_with_large_backlogs_of_small_messages)), however, shouldn't the messages be re-delivered only after the ack_deadline is exceeded?

#### Environment details
- OS type and version: Ubuntu 18.04.3 LTS
- Python version and virtual environment information: Python 3.7.3
- google-cloud-pubsub version: google-cloud-pubsub==1.0.0

#### Steps to reproduce

  1. Create a subscription with a high ack_deadline.
  2. Start subscriber (see code examples): `python subscriber.py -p PROJECT_ID SUBSCRIPTION_NAME`.
  3. Send two messages to the pubsub topic.
  4. The second message is duplicated.

#### Code example
Example subscriber code here: https://gist.github.com/qvik-olli/0bfd4ace2d06def1675a76fbc20493e5

#### Logs
From the subscriber script:
```
2019-09-19 14:10:37,740 INFO ThreadPoolExecutor-ThreadScheduler_0: [652240118977134] got message with content: b'0'
2019-09-19 14:10:37,740 INFO ThreadPoolExecutor-ThreadScheduler_0: [652240118977134] sleeping
2019-09-19 14:11:17,747 INFO ThreadPoolExecutor-ThreadScheduler_0: [652240118977134] done sleeping
2019-09-19 14:11:17,819 INFO ThreadPoolExecutor-ThreadScheduler_1: [652240118977135] got message with content: b'1'
2019-09-19 14:11:17,819 INFO ThreadPoolExecutor-ThreadScheduler_1: [652240118977135] sleeping
2019-09-19 14:11:57,859 INFO ThreadPoolExecutor-ThreadScheduler_1: [652240118977135] done sleeping
2019-09-19 14:11:57,986 ERROR ThreadPoolExecutor-ThreadScheduler_0: [652240118977135] Duplicate message!!!
```

With debug logging from the pubsub library:
```
2019-09-19 14:10:35,175 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:10:35,176 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 6.301498 seconds.
2019-09-19 14:10:37,634 DEBUG Thread-ConsumeBidirectionalStream: recved response.
2019-09-19 14:10:37,634 DEBUG Thread-ConsumeBidirectionalStream: Processing 2 received message(s), currenty on hold 0.
2019-09-19 14:10:37,740 DEBUG Thread-ConsumeBidirectionalStream: Sent request(s) over unary RPC.
2019-09-19 14:10:37,740 DEBUG Thread-ConsumeBidirectionalStream: Message backlog over load at 1.00, pausing.
2019-09-19 14:10:37,740 DEBUG Thread-ConsumeBidirectionalStream: Scheduling callbacks for 1 new messages, new total on hold 1.
2019-09-19 14:10:37,740 INFO ThreadPoolExecutor-ThreadScheduler_0: [652240118977134] got message with content: b'0'
2019-09-19 14:10:37,740 INFO ThreadPoolExecutor-ThreadScheduler_0: [652240118977134] sleeping
2019-09-19 14:10:37,740 DEBUG Thread-ConsumeBidirectionalStream: paused, waiting for waking.
2019-09-19 14:10:41,477 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:10:41,477 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:10:41,539 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:10:41,539 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 5.628947 seconds.
2019-09-19 14:10:47,168 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:10:47,168 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:10:47,238 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:10:47,238 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 1.232672 seconds.
2019-09-19 14:10:48,471 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:10:48,471 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:10:48,584 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:10:48,584 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 0.579531 seconds.
2019-09-19 14:10:49,164 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:10:49,164 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:10:49,242 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:10:49,242 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 5.177725 seconds.
2019-09-19 14:10:54,420 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:10:54,420 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:10:54,477 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:10:54,477 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 6.448523 seconds.
2019-09-19 14:11:00,926 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:11:00,926 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:11:01,002 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:11:01,002 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 7.749161 seconds.
2019-09-19 14:11:02,469 DEBUG Thread-Heartbeater: Sent heartbeat.
2019-09-19 14:11:08,752 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:11:08,752 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:11:08,813 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:11:08,813 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 4.640038 seconds.
2019-09-19 14:11:13,454 DEBUG Thread-LeaseMaintainer: The current p99 value is 10 seconds.
2019-09-19 14:11:13,454 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:11:13,532 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:11:13,532 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 4.966822 seconds.
2019-09-19 14:11:17,747 INFO ThreadPoolExecutor-ThreadScheduler_0: [652240118977134] done sleeping
2019-09-19 14:11:17,748 DEBUG Thread-CallbackRequestDispatcher: Handling 1 batched requests
2019-09-19 14:11:17,818 DEBUG Thread-CallbackRequestDispatcher: Sent request(s) over unary RPC.
2019-09-19 14:11:17,818 DEBUG Thread-CallbackRequestDispatcher: Current load: 0.00
2019-09-19 14:11:17,818 DEBUG Thread-CallbackRequestDispatcher: Released held message to leaser, scheduling callback for it, still on hold 0.
2019-09-19 14:11:17,819 INFO ThreadPoolExecutor-ThreadScheduler_1: [652240118977135] got message with content: b'1'
2019-09-19 14:11:17,819 INFO ThreadPoolExecutor-ThreadScheduler_1: [652240118977135] sleeping
2019-09-19 14:11:17,819 DEBUG Thread-CallbackRequestDispatcher: Did not resume, current load is 1.00.
2019-09-19 14:11:18,499 DEBUG Thread-LeaseMaintainer: The current p99 value is 41 seconds.
2019-09-19 14:11:18,499 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:11:18,575 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:11:18,575 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 18.568330 seconds.
2019-09-19 14:11:32,469 DEBUG Thread-Heartbeater: Sent heartbeat.
2019-09-19 14:11:37,144 DEBUG Thread-LeaseMaintainer: The current p99 value is 41 seconds.
2019-09-19 14:11:37,144 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:11:37,470 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:11:37,470 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 5.602221 seconds.
2019-09-19 14:11:43,072 DEBUG Thread-LeaseMaintainer: The current p99 value is 41 seconds.
2019-09-19 14:11:43,072 DEBUG Thread-LeaseMaintainer: Renewing lease for 1 ack IDs.
2019-09-19 14:11:43,146 DEBUG Thread-LeaseMaintainer: Sent request(s) over unary RPC.
2019-09-19 14:11:43,146 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 22.866717 seconds.
2019-09-19 14:11:57,859 INFO ThreadPoolExecutor-ThreadScheduler_1: [652240118977135] done sleeping
2019-09-19 14:11:57,859 DEBUG Thread-CallbackRequestDispatcher: Handling 1 batched requests
2019-09-19 14:11:57,923 DEBUG Thread-CallbackRequestDispatcher: Sent request(s) over unary RPC.
2019-09-19 14:11:57,924 DEBUG Thread-CallbackRequestDispatcher: Current load: 0.00
2019-09-19 14:11:57,924 DEBUG Thread-CallbackRequestDispatcher: Current load is 0.00, resuming consumer.
2019-09-19 14:11:57,924 DEBUG Thread-ConsumeBidirectionalStream: woken.
2019-09-19 14:11:57,924 DEBUG Thread-ConsumeBidirectionalStream: waiting for recv.
2019-09-19 14:11:57,924 DEBUG Thread-ConsumeBidirectionalStream: recved response.
2019-09-19 14:11:57,924 DEBUG Thread-ConsumeBidirectionalStream: Processing 1 received message(s), currenty on hold 0.
2019-09-19 14:11:57,985 DEBUG Thread-ConsumeBidirectionalStream: Sent request(s) over unary RPC.
2019-09-19 14:11:57,986 DEBUG Thread-ConsumeBidirectionalStream: Message backlog over load at 1.00, pausing.
2019-09-19 14:11:57,986 DEBUG Thread-ConsumeBidirectionalStream: Scheduling callbacks for 1 new messages, new total on hold 0.
2019-09-19 14:11:57,986 ERROR ThreadPoolExecutor-ThreadScheduler_0: [652240118977135] Duplicate message!!!
2019-09-19 14:11:57,986 DEBUG Thread-ConsumeBidirectionalStream: paused, waiting for waking.
2019-09-19 14:11:57,986 DEBUG Thread-CallbackRequestDispatcher: Handling 1 batched requests
2019-09-19 14:11:58,087 DEBUG Thread-CallbackRequestDispatcher: Sent request(s) over unary RPC.
2019-09-19 14:11:58,088 DEBUG Thread-CallbackRequestDispatcher: Current load: 0.00
2019-09-19 14:11:58,088 DEBUG Thread-CallbackRequestDispatcher: Current load is 0.00, resuming consumer.
2019-09-19 14:11:58,088 DEBUG Thread-ConsumeBidirectionalStream: woken.
2019-09-19 14:11:58,088 DEBUG Thread-ConsumeBidirectionalStream: waiting for recv.
2019-09-19 14:12:02,470 DEBUG Thread-Heartbeater: Sent heartbeat.
2019-09-19 14:12:06,013 DEBUG Thread-LeaseMaintainer: The current p99 value is 81 seconds.
2019-09-19 14:12:06,013 DEBUG Thread-LeaseMaintainer: Snoozing lease management for 71.733369 seconds.
```


Discussion:
@qvik-olli Thanks for the detailed report!

This sounds as if the request(s) to modify the message acknowledge deadline do not arrive in time, and the backend re-sends the message. Can you check if the subscription's max ACK deadline is indeed set to 300 seconds (5 minutes), and not just to e.g. the reported 30 seconds because of a typo or something?

**Technical details:**
Received messages that exceed the `max_messages` setting are put into a waiting buffer until the lease manager's load drops under a certain threshold, at which point some of these messages are released to the lease management.

The messages in the buffer are _not_ lease-managed, and if they need to wait there for longer than the acknowledgment deadline, the backend assumes they got lost and re-sends them.

Processing the first message takes ~40 seconds, which exceeds the default subscription's ACK deadline of 10 seconds. However, if the deadline is indeed correctly set to 300 seconds (5 minutes), the reported behavior is actually surprising and would require a deeper look.


Thank you @plamut for your quick response. I'm sure about the ack deadline and I double-checked that it is 300 seconds (5 minutes).

You can forget what I said about 30 seconds. I did some further testing and it seems that there is some dynamic timeout that seems to be changing depending on how long does it take to process a batch of messages. This timeout seems to ignore ack deadline.

I did some tests with larger amount of messages. I used the script you can find in my first message. I set the message handler to sleep 1 second when a message arrived. `max_messages` was set to 1 and ack deadline was set to 600 seconds:

```
2019-09-20 09:09:29,188 INFO MainThread: Subscriber started
2019-09-20 09:09:34,777 INFO ThreadPoolExecutor-ThreadScheduler_0: [652265204686881] got message with content: b'1'
2019-09-20 09:09:34,778 INFO ThreadPoolExecutor-ThreadScheduler_0: [652265204686881] sleeping
2019-09-20 09:09:35,779 INFO ThreadPoolExecutor-ThreadScheduler_0: [652265204686881] done sleeping
2019-09-20 09:09:35,850 INFO ThreadPoolExecutor-ThreadScheduler_1: [652265204686882] got message with content: b'2'
2019-09-20 09:09:35,851 INFO ThreadPoolExecutor-ThreadScheduler_1: [652265204686882] sleeping
2019-09-20 09:09:36,852 INFO ThreadPoolExecutor-ThreadScheduler_1: [652265204686882] done sleeping
---snip---sending 60 messages with each taking 1 second to handle---
2019-09-20 09:10:36,946 INFO ThreadPoolExecutor-ThreadScheduler_9: [652265204686939] got message with content: b'59'
2019-09-20 09:10:36,947 INFO ThreadPoolExecutor-ThreadScheduler_9: [652265204686939] sleeping
2019-09-20 09:10:37,948 INFO ThreadPoolExecutor-ThreadScheduler_9: [652265204686939] done sleeping
2019-09-20 09:10:37,997 INFO ThreadPoolExecutor-ThreadScheduler_6: [652265204686940] got message with content: b'60'
2019-09-20 09:10:37,997 INFO ThreadPoolExecutor-ThreadScheduler_6: [652265204686940] sleeping
2019-09-20 09:10:38,998 INFO ThreadPoolExecutor-ThreadScheduler_6: [652265204686940] done sleeping
2019-09-20 09:10:39,170 ERROR ThreadPoolExecutor-ThreadScheduler_3: [652265204686891] Duplicate message!!!
2019-09-20 09:10:39,259 ERROR ThreadPoolExecutor-ThreadScheduler_0: [652265204686892] Duplicate message!!!
2019-09-20 09:10:39,377 ERROR ThreadPoolExecutor-ThreadScheduler_5: [652265204686893] Duplicate message!!!
---snip---all together 145 duplicate messages---
2019-09-20 09:10:54,913 ERROR ThreadPoolExecutor-ThreadScheduler_8: [652265204686938] Duplicate message!!!
2019-09-20 09:10:55,011 ERROR ThreadPoolExecutor-ThreadScheduler_9: [652265204686939] Duplicate message!!!
2019-09-20 09:10:55,125 ERROR ThreadPoolExecutor-ThreadScheduler_6: [652265204686940] Duplicate message!!!
2019-09-20 09:10:59,245 INFO ThreadPoolExecutor-ThreadScheduler_7: [652265042758581] got message with content: b'1'
2019-09-20 09:10:59,245 INFO ThreadPoolExecutor-ThreadScheduler_7: [652265042758581] sleeping
2019-09-20 09:11:00,246 INFO ThreadPoolExecutor-ThreadScheduler_7: [652265042758581] done sleeping
2019-09-20 09:11:00,318 INFO ThreadPoolExecutor-ThreadScheduler_4: [652265042758582] got message with content: b'2'
2019-09-20 09:11:00,318 INFO ThreadPoolExecutor-ThreadScheduler_4: [652265042758582] sleeping
2019-09-20 09:11:01,320 INFO ThreadPoolExecutor-ThreadScheduler_4: [652265042758582] done sleeping
---snip---sending another 60 messages---
2019-09-20 09:12:00,801 INFO ThreadPoolExecutor-ThreadScheduler_9: [652265042758639] got message with content: b'59'
2019-09-20 09:12:00,801 INFO ThreadPoolExecutor-ThreadScheduler_9: [652265042758639] sleeping
2019-09-20 09:12:01,803 INFO ThreadPoolExecutor-ThreadScheduler_9: [652265042758639] done sleeping
2019-09-20 09:12:01,886 INFO ThreadPoolExecutor-ThreadScheduler_2: [652265042758640] got message with content: b'60'
2019-09-20 09:12:01,886 INFO ThreadPoolExecutor-ThreadScheduler_2: [652265042758640] sleeping
2019-09-20 09:12:02,887 INFO ThreadPoolExecutor-ThreadScheduler_2: [652265042758640] done sleeping
2019-09-20 09:12:03,077 ERROR ThreadPoolExecutor-ThreadScheduler_7: [652265042758640] Duplicate message!!!
2019-09-20 09:12:14,436 INFO ThreadPoolExecutor-ThreadScheduler_4: [652264914144087] got message with content: b'1'
2019-09-20 09:12:14,436 INFO ThreadPoolExecutor-ThreadScheduler_4: [652264914144087] sleeping
2019-09-20 09:12:15,437 INFO ThreadPoolExecutor-ThreadScheduler_4: [652264914144087] done sleeping
2019-09-20 09:12:15,609 INFO ThreadPoolExecutor-ThreadScheduler_8: [652264914144088] got message with content: b'2'
2019-09-20 09:12:15,609 INFO ThreadPoolExecutor-ThreadScheduler_8: [652264914144088] sleeping
2019-09-20 09:12:16,610 INFO ThreadPoolExecutor-ThreadScheduler_8: [652264914144088] done sleeping
---snip---sending another 60 messages---
2019-09-20 09:13:16,882 INFO ThreadPoolExecutor-ThreadScheduler_2: [652264914144145] got message with content: b'59'
2019-09-20 09:13:16,882 INFO ThreadPoolExecutor-ThreadScheduler_2: [652264914144145] sleeping
2019-09-20 09:13:17,883 INFO ThreadPoolExecutor-ThreadScheduler_2: [652264914144145] done sleeping
2019-09-20 09:13:17,948 INFO ThreadPoolExecutor-ThreadScheduler_3: [652264914144146] got message with content: b'60'
2019-09-20 09:13:17,948 INFO ThreadPoolExecutor-ThreadScheduler_3: [652264914144146] sleeping
2019-09-20 09:13:18,949 INFO ThreadPoolExecutor-ThreadScheduler_3: [652264914144146] done sleeping
2019-09-20 09:13:19,194 ERROR ThreadPoolExecutor-ThreadScheduler_4: [652264914144146] Duplicate message!!!
2019-09-20 09:13:46,466 INFO ThreadPoolExecutor-ThreadScheduler_8: [652264749734941] got message with content: b'1'
2019-09-20 09:13:46,466 INFO ThreadPoolExecutor-ThreadScheduler_8: [652264749734941] sleeping
2019-09-20 09:13:47,467 INFO ThreadPoolExecutor-ThreadScheduler_8: [652264749734941] done sleeping
2019-09-20 09:13:47,512 INFO ThreadPoolExecutor-ThreadScheduler_1: [652264749734942] got message with content: b'2'
2019-09-20 09:13:47,512 INFO ThreadPoolExecutor-ThreadScheduler_1: [652264749734942] sleeping
2019-09-20 09:13:48,513 INFO ThreadPoolExecutor-ThreadScheduler_1: [652264749734942] done sleeping
---snip---sending 100 messages---
2019-09-20 09:15:30,324 INFO ThreadPoolExecutor-ThreadScheduler_2: [652264749735038] sleeping
2019-09-20 09:15:31,325 INFO ThreadPoolExecutor-ThreadScheduler_2: [652264749735038] done sleeping
2019-09-20 09:15:31,372 INFO ThreadPoolExecutor-ThreadScheduler_3: [652264749735039] got message with content: b'99'
2019-09-20 09:15:31,372 INFO ThreadPoolExecutor-ThreadScheduler_3: [652264749735039] sleeping
2019-09-20 09:15:32,373 INFO ThreadPoolExecutor-ThreadScheduler_3: [652264749735039] done sleeping
2019-09-20 09:15:32,418 INFO ThreadPoolExecutor-ThreadScheduler_0: [652264749735040] got message with content: b'100'
2019-09-20 09:15:32,418 INFO ThreadPoolExecutor-ThreadScheduler_0: [652264749735040] sleeping
2019-09-20 09:15:33,419 INFO ThreadPoolExecutor-ThreadScheduler_0: [652264749735040] done sleeping
2019-09-20 09:15:33,573 ERROR ThreadPoolExecutor-ThreadScheduler_8: [652264749735001] Duplicate message!!!
2019-09-20 09:15:33,693 ERROR ThreadPoolExecutor-ThreadScheduler_1: [652264749735002] Duplicate message!!!
2019-09-20 09:15:33,811 ERROR ThreadPoolExecutor-ThreadScheduler_6: [652264749735003] Duplicate message!!!
---snip---all together 95 duplicate messages---
2019-09-20 09:15:43,785 ERROR ThreadPoolExecutor-ThreadScheduler_6: [652264749735038] Duplicate message!!!
2019-09-20 09:15:43,858 ERROR ThreadPoolExecutor-ThreadScheduler_7: [652264749735039] Duplicate message!!!
2019-09-20 09:15:43,938 ERROR ThreadPoolExecutor-ThreadScheduler_0: [652264749735040] Duplicate message!!!
```

So when I started the subscriber and sent the first 60 messages, I got 145 duplicates. When I sent another 60 messages, I got just one duplicate message. After that, I sent 100 messages and I got a high amount of duplicates again.
Interesting, I will try to run it today myself and see if I can reproduce it. I do not see anything apparent in the client itself, but I have not looked too deeply yet.

**Update:**
I was able to reproduce the reported behavior, and it's indeed unexpected. The key bit is that both messages need to be delivered in a single server response (so that one of them is temporarily put on hold into a buffer).

This can be achieved by rapidly publishing them one after another, i.e. without waiting on each individual publish future, which results in both messages being sent to the backend in the same publish batch.
After experimenting with various things, I was not able to pinpoint the exact source of the issue (in the client at least).

It appears that if the [delay](https://gist.github.com/qvik-olli/0bfd4ace2d06def1675a76fbc20493e5#file-subscriber-py-L40) in the message handler is considerably shorter than 10 seconds, the issue is not reproducible. The probability of a duplicate message increases when the `time.sleep()` interval is around 10 seconds (8 - 12), and approaches one with delays longer than that.

I also tried disabling the automatic modifications of the ACK deadlines, just in case the related client's logic is flawed:
```patch
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
index 2b257482930..d6b1d2c54d0 100644
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
@@ -152,7 +156,10 @@ class Dispatcher(object):
         """
         ack_ids = [item.ack_id for item in items]
         seconds = [item.seconds for item in items]
-
+        #################
+        _LOGGER.debug(f"\x1b[33m(DISABLED) Modifying ACK deadline to {seconds}, ACK IDs: {ack_ids}\x1b[0m")
+        return
+        ######################
         request = types.StreamingPullRequest(
             modify_deadline_ack_ids=ack_ids, modify_deadline_seconds=seconds
         )
```
The result was the same, which makes me think it might be something with the subscription on the backend (the ACK deadline is set to 300 seconds), and that the deadline expected by the server is actually around 10 seconds, in my case at least.

> I did some further testing and it seems that there is some dynamic timeout that seems to be changing depending on how long does it take to process a batch of messages. This timeout seems to ignore ack deadline.

I got a similar impression. The actual server-side deadline does not seem to be static. If it was really, say, 10 seconds, a sleep delay of 8 seconds should almost definitely not reproduce the issue, and at the same time a delay of 12 seconds should always reproduce it - but the outcome of the test script was not deterministic.

Labeling as a backend issue until further evidence.
I don't think this is a backend issue. Looking at some logs, it appears that the server is delivering the message based on its understanding of the ack deadline with no unexpected duplicates. I think there are two things at play here:

1. The client library sets the [ack deadline on the stream based on the 99th percentile ack latency](https://github.com/googleapis/google-cloud-python/blob/bd1cf0eeb159e0f826dd165fe5b2c14ab4aa4f65/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py#L489). This overrides the ack deadline set on the subscription at the time it is created. By default, this deadline is 10 seconds. We do this because there is a tradeoff with ack deadlines between more time to process and delays in redelivery if a client goes down or becomes unhealthy. The client library changes the ack deadline for the streams it creates and then calls ModifyAckDeadline up to the specified [max_lease_duration](https://github.com/googleapis/google-cloud-python/blob/7e16c08581a059a6c8e40b0403f4af1ebfe0fea6/pubsub/google/cloud/pubsub_v1/types.py#L90) in the FlowControl settings.

2. It looks like these ModifyAckDeadlines are arriving a little late, a second or two beyond the time of the deadline expiry. I suspect this means that the client library is not sending the ModifyAckDeadlines soon enough, possibly because they are buffered or something like that.
> The client library sets the ack deadline on the stream based on the 99th percentile ack latency. 

Indeed, although the issue was reproducible even with these calls disabled. However, it appears that the topic ACK deadline is overridden even before pulling the messages start, i.e. when the stream is established. This key piece of info is missing in the client debug logs and needs to be added (along with the fix for the observed behavior).

Thanks for checking the backend logs, @kamalaboulhosn !

@qvik-olli We have just released a [new version](https://pypi.org/project/google-cloud-pubsub/) of the PubSub client (1.0.1), which includes the fix for this issue. The ACK deadline for the messages should now more closely match the deadline set on the subscription, resulting in fewer duplicate re-deliveries.
@qvik-olli Unfortunately, it turned out that he the fix for this caused a different issue (#9339) which is more severe, and thus had to be reverted in PubSub 1.0.2 (released ~15 minutes ago). Fetching a subscription to read its ACK deadline requires additional permissions, and that broke some the users' applications.

For the time being the default stream ACK deadline is set to 60 seconds. Some of the messages can still time out a bit to soon and will be re-delivered, but the time window is now wider, giving the user code more time to process the received messages that exceed the `FlowControl.max_messages` setting.

A different fix will have to be made, although that will likely require some more substantial changes to the streaming pull core logic.
Okay, I see. Increasing the default stream ACK deadline to 60 seconds is helpful, but I do hope that the issue will get fixed.

Our systems were experiencing this issue because we were controlling concurrency by setting `max_messages` to a low value (less than 10). We have now updated our code to use default `max_messages` and have a custom scheduler to control concurrency. This doesn't prevent the issue from occurring but it removes a lot of the duplicates we were experiencing earlier.

I'm sharing this snippet if someone else is affected by this issue. Creating custom scheduler to control concurrency:
``` python
subscriber = pubsub.SubscriberClient()
executor = concurrent.futures.ThreadPoolExecutor(max_workers=5)
# google.cloud.pubsub_v1.subscriber.scheduler.ThreadScheduler
scheduler = ThreadScheduler(executor=executor)
subscriber.subscribe(subscription_name, callback, scheduler=scheduler)
```

For CPU-intensive computations in callbacks, providing a shallower thread pool for invoking callbacks is indeed one way of limiting the concurrency (the default thread pool size is 10, FWIW).

Nevertheless, I reopened this issue until it gets a proper fix, which will probably require to auto-lease (i.e. auto-extend the ACK deadlines) _all_ received messages, and also to decouple tracking of the current load from the number of actively leased messages.

(the load is used to decide how many messages to dispatch to the callbacks, and when to pause/resume the message stream)

Resolving patch:
diff --git a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
--- a/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
+++ b/pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
@@ -51,13 +51,6 @@
 _RESUME_THRESHOLD = 0.8
 """The load threshold below which to resume the incoming message stream."""
 
-_DEFAULT_STREAM_ACK_DEADLINE = 60
-"""The default message acknowledge deadline in seconds for incoming message stream.
-
-This default deadline is dynamically modified for the messages that are added
-to the lease management.
-"""
-
 
 def _maybe_wrap_exception(exception):
     """Wraps a gRPC exception class, if needed."""
@@ -135,9 +128,15 @@ def __init__(
         # because the FlowControl limits have been hit.
         self._messages_on_hold = queue.Queue()
 
+        # the total number of bytes consumed by the messages currently on hold
+        self._on_hold_bytes = 0
+
         # A lock ensuring that pausing / resuming the consumer are both atomic
         # operations that cannot be executed concurrently. Needed for properly
-        # syncing these operations with the current leaser load.
+        # syncing these operations with the current leaser load. Additionally,
+        # the lock is used to protect modifications of internal data that
+        # affects the load computation, i.e. the count and size of the messages
+        # currently on hold.
         self._pause_resume_lock = threading.Lock()
 
         # The threads created in ``.open()``.
@@ -218,10 +217,18 @@ def load(self):
         if self._leaser is None:
             return 0.0
 
+        # Messages that are temporarily put on hold are not being delivered to
+        # user's callbacks, thus they should not contribute to the flow control
+        # load calculation.
+        # However, since these messages must still be lease-managed to avoid
+        # unnecessary ACK deadline expirations, their count and total size must
+        # be subtracted from the leaser's values.
         return max(
             [
-                self._leaser.message_count / self._flow_control.max_messages,
-                self._leaser.bytes / self._flow_control.max_bytes,
+                (self._leaser.message_count - self._messages_on_hold.qsize())
+                / self._flow_control.max_messages,
+                (self._leaser.bytes - self._on_hold_bytes)
+                / self._flow_control.max_bytes,
             ]
         )
 
@@ -292,13 +299,19 @@ def _maybe_release_messages(self):
             except queue.Empty:
                 break
 
-            self.leaser.add(
-                [requests.LeaseRequest(ack_id=msg.ack_id, byte_size=msg.size)]
-            )
+            self._on_hold_bytes -= msg.size
+
+            if self._on_hold_bytes < 0:
+                _LOGGER.warning(
+                    "On hold bytes was unexpectedly negative: %s", self._on_hold_bytes
+                )
+                self._on_hold_bytes = 0
+
             _LOGGER.debug(
-                "Released held message to leaser, scheduling callback for it, "
-                "still on hold %s.",
+                "Released held message, scheduling callback for it, "
+                "still on hold %s (bytes %s).",
                 self._messages_on_hold.qsize(),
+                self._on_hold_bytes,
             )
             self._scheduler.schedule(self._callback, msg)
 
@@ -392,17 +405,7 @@ def open(self, callback, on_callback_error):
         )
 
         # Create the RPC
-
-        # We must use a fixed value for the ACK deadline, as we cannot read it
-        # from the subscription. The latter would require `pubsub.subscriptions.get`
-        # permission, which is not granted to the default subscriber role
-        # `roles/pubsub.subscriber`.
-        # See also https://github.com/googleapis/google-cloud-python/issues/9339
-        #
-        # When dynamic lease management is enabled for the "on hold" messages,
-        # the default stream ACK deadline should again be set based on the
-        # historic ACK timing data, i.e. `self.ack_histogram.percentile(99)`.
-        stream_ack_deadline_seconds = _DEFAULT_STREAM_ACK_DEADLINE
+        stream_ack_deadline_seconds = self.ack_histogram.percentile(99)
 
         get_initial_request = functools.partial(
             self._get_initial_request, stream_ack_deadline_seconds
@@ -540,40 +543,46 @@ def _on_response(self, response):
         the callback for each message using the executor.
         """
         _LOGGER.debug(
-            "Processing %s received message(s), currenty on hold %s.",
+            "Processing %s received message(s), currenty on hold %s (bytes %s).",
             len(response.received_messages),
             self._messages_on_hold.qsize(),
+            self._on_hold_bytes,
         )
 
+        # Immediately (i.e. without waiting for the auto lease management)
+        # modack the messages we received, as this tells the server that we've
+        # received them.
+        items = [
+            requests.ModAckRequest(message.ack_id, self._ack_histogram.percentile(99))
+            for message in response.received_messages
+        ]
+        self._dispatcher.modify_ack_deadline(items)
+
         invoke_callbacks_for = []
 
         for received_message in response.received_messages:
             message = google.cloud.pubsub_v1.subscriber.message.Message(
                 received_message.message, received_message.ack_id, self._scheduler.queue
             )
-            if self.load < _MAX_LOAD:
-                req = requests.LeaseRequest(
-                    ack_id=message.ack_id, byte_size=message.size
-                )
-                self.leaser.add([req])
-                invoke_callbacks_for.append(message)
-                self.maybe_pause_consumer()
-            else:
-                self._messages_on_hold.put(message)
-
-        # Immediately (i.e. without waiting for the auto lease management)
-        # modack the messages we received and not put on hold, as this tells
-        # the server that we've received them.
-        items = [
-            requests.ModAckRequest(message.ack_id, self._ack_histogram.percentile(99))
-            for message in invoke_callbacks_for
-        ]
-        self._dispatcher.modify_ack_deadline(items)
+            # Making a decision based on the load, and modifying the data that
+            # affects the load -> needs a lock, as that state can be modified
+            # by different threads.
+            with self._pause_resume_lock:
+                if self.load < _MAX_LOAD:
+                    invoke_callbacks_for.append(message)
+                else:
+                    self._messages_on_hold.put(message)
+                    self._on_hold_bytes += message.size
+
+            req = requests.LeaseRequest(ack_id=message.ack_id, byte_size=message.size)
+            self.leaser.add([req])
+            self.maybe_pause_consumer()
 
         _LOGGER.debug(
-            "Scheduling callbacks for %s new messages, new total on hold %s.",
+            "Scheduling callbacks for %s new messages, new total on hold %s (bytes %s).",
             len(invoke_callbacks_for),
             self._messages_on_hold.qsize(),
+            self._on_hold_bytes,
         )
         for msg in invoke_callbacks_for:
             self._scheduler.schedule(self._callback, msg)


Test patch:
diff --git a/pubsub/tests/system.py b/pubsub/tests/system.py
--- a/pubsub/tests/system.py
+++ b/pubsub/tests/system.py
@@ -382,10 +382,6 @@ class CallbackError(Exception):
         with pytest.raises(CallbackError):
             future.result(timeout=30)
 
-    @pytest.mark.xfail(
-        reason="The default stream ACK deadline is static and received messages "
-        "exceeding FlowControl.max_messages are currently not lease managed."
-    )
     def test_streaming_pull_ack_deadline(
         self, publisher, subscriber, project, topic_path, subscription_path, cleanup
     ):
@@ -400,7 +396,7 @@ def test_streaming_pull_ack_deadline(
         # Subscribe to the topic. This must happen before the messages
         # are published.
         subscriber.create_subscription(
-            subscription_path, topic_path, ack_deadline_seconds=240
+            subscription_path, topic_path, ack_deadline_seconds=45
         )
 
         # publish some messages and wait for completion
@@ -408,7 +404,7 @@ def test_streaming_pull_ack_deadline(
 
         # subscribe to the topic
         callback = StreamingPullCallback(
-            processing_time=70,  # more than the default stream ACK deadline (60s)
+            processing_time=13,  # more than the default stream ACK deadline (10s)
             resolve_at_msg_count=3,  # one more than the published messages count
         )
         flow_control = types.FlowControl(max_messages=1)
@@ -416,13 +412,13 @@ def test_streaming_pull_ack_deadline(
             subscription_path, callback, flow_control=flow_control
         )
 
-        # We expect to process the first two messages in 2 * 70 seconds, and
+        # We expect to process the first two messages in 2 * 13 seconds, and
         # any duplicate message that is re-sent by the backend in additional
-        # 70 seconds, totalling 210 seconds (+ overhead) --> if there have been
-        # no duplicates in 240 seconds, we can reasonably assume that there
+        # 13 seconds, totalling 39 seconds (+ overhead) --> if there have been
+        # no duplicates in 60 seconds, we can reasonably assume that there
         # won't be any.
         try:
-            callback.done_future.result(timeout=240)
+            callback.done_future.result(timeout=60)
         except exceptions.TimeoutError:
             # future timed out, because we received no excessive messages
             assert sorted(callback.seen_message_ids) == [1, 2]
diff --git a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
--- a/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
+++ b/pubsub/tests/unit/pubsub_v1/subscriber/test_streaming_pull_manager.py
@@ -233,13 +233,15 @@ def test__maybe_release_messages_on_overload():
     manager = make_manager(
         flow_control=types.FlowControl(max_messages=10, max_bytes=1000)
     )
-    # Ensure load is exactly 1.0 (to verify that >= condition is used)
-    _leaser = manager._leaser = mock.create_autospec(leaser.Leaser)
-    _leaser.message_count = 10
-    _leaser.bytes = 1000
 
     msg = mock.create_autospec(message.Message, instance=True, ack_id="ack", size=11)
     manager._messages_on_hold.put(msg)
+    manager._on_hold_bytes = msg.size
+
+    # Ensure load is exactly 1.0 (to verify that >= condition is used)
+    _leaser = manager._leaser = mock.create_autospec(leaser.Leaser)
+    _leaser.message_count = 10
+    _leaser.bytes = 1000 + msg.size
 
     manager._maybe_release_messages()
 
@@ -254,18 +256,20 @@ def test__maybe_release_messages_below_overload():
     )
     manager._callback = mock.sentinel.callback
 
-    # init leaser message count to 8 to leave room for 2 more messages
+    # Init leaser message count to 11, so that when subtracting the 3 messages
+    # that are on hold, there is still room for another 2 messages before the
+    # max load is hit.
     _leaser = manager._leaser = mock.create_autospec(leaser.Leaser)
-    fake_leaser_add(_leaser, init_msg_count=8, assumed_msg_size=25)
-    _leaser.add = mock.Mock(wraps=_leaser.add)  # to spy on calls
+    fake_leaser_add(_leaser, init_msg_count=11, assumed_msg_size=10)
 
     messages = [
-        mock.create_autospec(message.Message, instance=True, ack_id="ack_foo", size=11),
-        mock.create_autospec(message.Message, instance=True, ack_id="ack_bar", size=22),
-        mock.create_autospec(message.Message, instance=True, ack_id="ack_baz", size=33),
+        mock.create_autospec(message.Message, instance=True, ack_id="ack_foo", size=10),
+        mock.create_autospec(message.Message, instance=True, ack_id="ack_bar", size=10),
+        mock.create_autospec(message.Message, instance=True, ack_id="ack_baz", size=10),
     ]
     for msg in messages:
         manager._messages_on_hold.put(msg)
+        manager._on_hold_bytes = 3 * 10
 
     # the actual call of MUT
     manager._maybe_release_messages()
@@ -274,13 +278,6 @@ def test__maybe_release_messages_below_overload():
     msg = manager._messages_on_hold.get_nowait()
     assert msg.ack_id == "ack_baz"
 
-    assert len(_leaser.add.mock_calls) == 2
-    expected_calls = [
-        mock.call([requests.LeaseRequest(ack_id="ack_foo", byte_size=11)]),
-        mock.call([requests.LeaseRequest(ack_id="ack_bar", byte_size=22)]),
-    ]
-    _leaser.add.assert_has_calls(expected_calls)
-
     schedule_calls = manager._scheduler.schedule.mock_calls
     assert len(schedule_calls) == 2
     for _, call_args, _ in schedule_calls:
@@ -289,6 +286,34 @@ def test__maybe_release_messages_below_overload():
         assert call_args[1].ack_id in ("ack_foo", "ack_bar")
 
 
+def test__maybe_release_messages_negative_on_hold_bytes_warning(caplog):
+    manager = make_manager(
+        flow_control=types.FlowControl(max_messages=10, max_bytes=1000)
+    )
+
+    msg = mock.create_autospec(message.Message, instance=True, ack_id="ack", size=17)
+    manager._messages_on_hold.put(msg)
+    manager._on_hold_bytes = 5  # too low for some reason
+
+    _leaser = manager._leaser = mock.create_autospec(leaser.Leaser)
+    _leaser.message_count = 3
+    _leaser.bytes = 150
+
+    with caplog.at_level(logging.WARNING):
+        manager._maybe_release_messages()
+
+    expected_warnings = [
+        record.message.lower()
+        for record in caplog.records
+        if "unexpectedly negative" in record.message
+    ]
+    assert len(expected_warnings) == 1
+    assert "on hold bytes" in expected_warnings[0]
+    assert "-12" in expected_warnings[0]
+
+    assert manager._on_hold_bytes == 0  # should be auto-corrected
+
+
 def test_send_unary():
     manager = make_manager()
     manager._UNARY_REQUESTS = True
@@ -404,8 +429,6 @@ def test_heartbeat_inactive():
     "google.cloud.pubsub_v1.subscriber._protocol.heartbeater.Heartbeater", autospec=True
 )
 def test_open(heartbeater, dispatcher, leaser, background_consumer, resumable_bidi_rpc):
-    stream_ack_deadline = streaming_pull_manager._DEFAULT_STREAM_ACK_DEADLINE
-
     manager = make_manager()
 
     manager.open(mock.sentinel.callback, mock.sentinel.on_callback_error)
@@ -435,7 +458,7 @@ def test_open(heartbeater, dispatcher, leaser, background_consumer, resumable_bi
     )
     initial_request_arg = resumable_bidi_rpc.call_args.kwargs["initial_request"]
     assert initial_request_arg.func == manager._get_initial_request
-    assert initial_request_arg.args[0] == stream_ack_deadline
+    assert initial_request_arg.args[0] == 10  # the default stream ACK timeout
     assert not manager._client.api.get_subscription.called
 
     resumable_bidi_rpc.return_value.add_done_callback.assert_called_once_with(
@@ -668,13 +691,17 @@ def test__on_response_with_leaser_overload():
     # are called in the expected way.
     manager._on_response(response)
 
-    # only the messages that are added to the lease management and dispatched to
-    # callbacks should have their ACK deadline extended
+    # all messages should be added to the lease management and have their ACK
+    # deadline extended, even those not dispatched to callbacks
     dispatcher.modify_ack_deadline.assert_called_once_with(
-        [requests.ModAckRequest("fack", 10)]
+        [
+            requests.ModAckRequest("fack", 10),
+            requests.ModAckRequest("back", 10),
+            requests.ModAckRequest("zack", 10),
+        ]
     )
 
-    # one message should be scheduled, the leaser capacity allows for it
+    # one message should be scheduled, the flow control limits allow for it
     schedule_calls = scheduler.schedule.mock_calls
     assert len(schedule_calls) == 1
     call_args = schedule_calls[0][1]