Repository: googleapis/google-cloud-ruby (Ruby)

Issue:
Topic publish latencies
I'm using Pubsub as a backend together with ActiveJob for a background job system. I'm running `google-cloud-pubsub (2.15.0)`.

I'm experiencing big latencies when publishing messages. Here's the code:

```ruby
    def publish(job)
      serialized_job = job.serialize
      enqueued_at = Time.zone.parse(serialized_job["enqueued_at"])
      job_id = serialized_job["job_id"]
      topic.publish_async(serialized_job.to_json) do |publish_result|
        publish_time = Time.zone.now
        publish_latency_ms = (publish_time - enqueued_at) * 1000
        LOGGER.info(
          labels: { name: "api.job.published", topic: name },
          job_id: job_id,
          message_id: publish_result.message.message_id,
          enqueued_at: enqueued_at.rfc3339,
          publish_time: publish_time.rfc3339,
          publish_latency: publish_latency_ms,
          succeeded: publish_result.succeeded?,
          error: publish_result.error.to_s
        )
      end
    end
```

I have a process that can publish messages into 6 different topics. The config for each one is the default one (no async options provided). After collecting some data I've found that sometimes I have 10 minutes of `publish_latency`. I think this doesn't make sense as topic batches are published every 0.01s (this is the default value). So how is this possible? The `pubsub.googleapis.com/topic/send_request_latencies` metric is not higher than 200 ms so it doesn't seem to be a network issue. Here are the charts:

- Publish latency from the callback
<img width="822" alt="image" src="https://user-images.githubusercontent.com/7353713/219794869-edbfd789-5bfc-4777-9b8f-44f270a37793.png">

- `pubsub.googleapis.com/topic/send_request_latencies`
<img width="817" alt="image" src="https://user-images.githubusercontent.com/7353713/219794943-cb2c8e1a-0677-42fd-a3af-8354ec1a3778.png">


I can try tweaking the `async` option for the topic, but getting a delay of 10 minutes doesn't make sense to me as it should publish the batch every 0,01 seconds? So I'm kind of lost regarding which one I should modify. Unless I'm missing something.


Discussion: I have added more logs to see if I can find the reason for the messages to be stuck in the topic. I'm logging every 10 seconds this:
```ruby
topic.async_publisher.instance_variable_get(:@batches).each_value do |batch|
  LOGGER.info(
    labels: {
      name: "app.metrics.pubsub.batch",
      topic_id: queue
    },
    items_size: batch.items.size,
    items: batch.items.map { |item| JSON.parse(item.msg.data)["job_id"] },
    publishing: batch.publishing?,
    queue_size: batch.instance_variable_get(:@queue).size,
    queue: batch.instance_variable_get(:@queue).map { |item| JSON.parse(item.msg.data)["job_id"] }
  )
end
```

I've got a new latency spike and this is how the logs look like around that time:
```json
[
  { "timestamp": "2023-02-23T14:04:02.578784734Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:04:12.560695640Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:04:22.570929516Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:04:32.563439248Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:04:42.566222755Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:04:52.565668141Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:05:02.564330148Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:05:12.581279748Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:05:22.599212732Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:05:32.570264296Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:05:42.611339509Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:05:52.573469511Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:06:02.569038032Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:06:12.573697314Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:06:22.565853908Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:06:32.571672048Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:06:42.584346933Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:06:52.574483397Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:07:02.685817811Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:07:12.566093358Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:07:22.571082485Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:07:32.576168969Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:07:42.562463075Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:07:52.564612405Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:08:02.576715042Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:08:12.579005236Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:08:22.621475754Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:08:32.569079662Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:08:42.572310756Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:08:52.568013786Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:09:02.570582950Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:09:12.572859261Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:09:22.572130604Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:09:32.564724745Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:09:42.633640818Z", "items_size": 1, "items": ["3d7c5444-8be1-4377-938e-a492ee57a075"], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:09:52.583241671Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:10:02.571272350Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:10:12.563065833Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:10:22.576786430Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:10:32.566813897Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:10:42.573725492Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:10:52.586781432Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:11:02.569040997Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false },
  { "timestamp": "2023-02-23T14:11:12.567989421Z", "items_size": 0, "items": [], "queue_size": 0, "queue": [], "publishing": false }
]
```

The job was part of the batch for around 5 minutes. What could be the reason for that?
Could there be a race condition around `@published_at`? The variable is accessed and modified in different threads. Attaching more info that might be useful.
```json
[
  { "timestamp": "2023-02-23T14:04:07.572372362Z", "job_id": "26a23e08-3197-4db7-84dd-5d83e9c3a9a1", "labels": { "name": "app.job.enqueued", "queue": "jobs-emails" } },
  { "timestamp": "2023-02-23T14:04:07.573314509Z", "job_id": "3d7c5444-8be1-4377-938e-a492ee57a075", "labels": { "name": "app.job.enqueued", "queue": "jobs-emails" } },
  { "timestamp": "2023-02-23T14:04:07.576407051Z", "job_id": "26a23e08-3197-4db7-84dd-5d83e9c3a9a1", "labels": { "name": "api.job.published", "topic": "jobs-emails" } }
]
```
`app.job.enqueued` is the active job log that you get when you call enqueue. This does not mean that the job was published because the gem does this asynchronously. So the `api.job.publish` is the log line that I've added as part of the callback thread of the `topic.publish_async` call.

As you can see those two jobs were enqueued right at the same time, but only one was published at that moment. The other one got stuck for 5 minutes. Then right around the time the job was actually published I have these logs:
```json
[
  { "timestamp": "2023-02-23T14:09:47.573560254Z", "job_id": "dc9390fd-3c8c-4fea-904e-413c1e527ba8", "labels": { "name": "app.job.enqueued", "queue": "jobs-emails" } },
  { "timestamp": "2023-02-23T14:09:47.574879047Z", "job_id": "3d7c5444-8be1-4377-938e-a492ee57a075", "labels": { "name": "api.job.published", "topic": "jobs-emails" } }
]
```
So a new job was enqueued, releasing the other one immediately. As I mentioned before, my guess would be that there is some kind of race condition around the `@published_at` that makes the AsyncPublisher's background thread to stop until a new message arrives. But I don't know much the code so I might be wrong.

What do you think?
I have added to the batch logs (the one printing the items size and queue size) the `@published_at` value and it's always nil. That explains why it's stuck? The background thread is waiting for a signal to wake up so when a new job arrives it is released?
I'm trying to reproduce this locally (I still can't maybe because I'm using the emulator), but would there be a race condition if a new message arrives after calling `batch.reset!` and [this line](https://github.com/googleapis/google-cloud-ruby/blob/main/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher.rb#L357)?
@blowmage would you mind assessing priority for this request?
@ce07c3 unfortunately I’m no longer active on this gem. @dazuma do you know who else can help here?
Thank you @blowmage. @dazuma please let us know. :)
@alanhala Thanks for reporting this I can see some unexpected behaviour too. Trying to find the actual problem and root cause. Add add more details when found.
@NivedhaSenthil thank you!
@alanhala Thanks for the detailed analysis like you have mentioned the issue is happening in batch `reset!` where it is trying to set the status of publishing while the queue is empty. As in this case though the queue is empty there were messages in items that has to be published and it got struck there. When new message is enqueued `published_at` is renewed and the thread freshly starts publishing and things are back to normal. 

Working on fixing this.
@NivedhaSenthil If the `reset!` method was called then shouldn't the `items` array be empty? [It is the first thing the method does.](https://github.com/googleapis/google-cloud-ruby/blob/main/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher/batch.rb#L172) And if that was the case then the message shouldn't be stuck but instead lost?

Might be wrong though. It was an observation that I wanted to share.
The messages from queue are added to items [here](https://github.com/googleapis/google-cloud-ruby/blob/79404d89e48322d02db06211d6b8f62954cf7315/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher/batch.rb#L183) so when checking items array is not empty but queue is. 
@alanhala Can you please check if the fix [here](https://github.com/googleapis/google-cloud-ruby/pull/20763) solves the problem for you ? 

Still can't seem to find the reasoning behind checking queue emptiness when reseting batch, trying to dig into history for figuring that. Meanwhile if you can check and confirm, it will be helpful.
> The messages from queue are added to items [here](https://github.com/googleapis/google-cloud-ruby/blob/79404d89e48322d02db06211d6b8f62954cf7315/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher/batch.rb#L183) so when checking items array is not empty but queue is.

Yeah, I know. I was assuming that both messages were added to the items and not one to the items and the other to the queue as they arrived at the same time.

@NivedhaSenthil I think by doing that change you will constantly keep publishing messages if they keep added to the queue. I guess the idea of the original code was to move the messages from the queue to the items, stop the publishing of that interval and then publish them in the next interval or when the items list is full.

I don't know what downsides the change has. Because now you would ignore not only the interval the user used for the publisher but also the batch size. I guess if you want to keep that functionality you have to tell the async publsiher that it should stop publishing (like it's done now) but also tell to set the `published_at` to now, so that the countdown for the next batch starts.

Did I get it correctly?
Yes thats correct. I am still exploring other options to fix this, just wanted to confirm if the cause I have mentioned is also the cause for the issue you are facing. 
It feels a bit risky to deploy to production a code pointing to a draft PR with code changes that have some downsides like the ones mentioned in my previous comment.
Yes definitely would not recommend putting it on production. Only if it the issue is replicable locally this would be helpful. 
Thanks for the response.

So the problem is happening as the interval given to publish  batches is not honoured. I am thinking of delegating this responsibility to Batch itself so that when reset happens Batch can say true either if batch is full or the interval is met so that publishing can continue. Will keep it posted here how this works out.  



Fix (patch):
diff --git a/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher/batch.rb b/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher/batch.rb
index e1a7352c9c62..46bc0449374d 100644
--- a/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher/batch.rb
+++ b/google-cloud-pubsub/lib/google/cloud/pubsub/async_publisher/batch.rb
@@ -141,7 +141,7 @@ def rebalance!
 
               until @queue.empty?
                 item = @queue.first
-                if try_add item.msg, item.callback
+                if try_add item.msg, item.callback, item.create_time
                   @queue.shift
                   next
                 end
@@ -178,12 +178,7 @@ def reset!
                 return false
               end
 
-              until @queue.empty?
-                item = @queue.first
-                added = try_add item.msg, item.callback
-                break unless added
-                @queue.shift
-              end
+              refill_items
 
               return false unless @publishing
               if @items.empty?
@@ -191,6 +186,7 @@ def reset!
                 return false
               else
                 return true if stopping?
+                return true if interval_met?(@items.first.create_time) || batch_full_by_count?
                 if @queue.empty?
                   @publishing = false
                   return false
@@ -261,16 +257,33 @@ def total_message_bytes
 
           protected
 
-          def items_add msg, callback
-            item = Item.new msg, callback
+          def interval_met? create_time
+            Time.now - create_time > @publisher.interval
+          end
+
+          def batch_full_by_count?
+            total_message_count == @publisher.max_messages
+          end
+
+          def refill_items
+            until @queue.empty?
+              item = @queue.first
+              added = try_add item.msg, item.callback, item.create_time
+              break unless added
+              @queue.shift
+            end
+          end
+
+          def items_add msg, callback, create_time
+            item = Item.new msg, callback, (create_time || Time.now)
             @items << item
             @total_message_bytes += item.bytesize + 2
           end
 
-          def try_add msg, callback
+          def try_add msg, callback, create_time = nil
             if @items.empty?
               # Always add when empty, even if bytesize is bigger than total
-              items_add msg, callback
+              items_add msg, callback, create_time
               return true
             end
             new_message_count = total_message_count + 1
@@ -279,12 +292,12 @@ def try_add msg, callback
                new_message_bytes >= @publisher.max_bytes
               return false
             end
-            items_add msg, callback
+            items_add msg, callback, create_time
             true
           end
 
           def queue_add msg, callback
-            item = Item.new msg, callback
+            item = Item.new msg, callback, Time.now
             @queue << item
           end
 
@@ -292,7 +305,7 @@ def total_message_count
             @items.count
           end
 
-          Item = Struct.new :msg, :callback do
+          Item = Struct.new :msg, :callback, :create_time do
             def bytesize
               msg.to_proto.bytesize
             end