opened
gabrielgerhardsson
username_0
Reimplement so that we avoid using a seemingly problematic rate limiter implementation.

Instead of using the Guava SmoothWarmingUp implementation, use the
simple SmoothBursty and apply a manual slow start period to it in a
wrapper. Slow start duration is still configurable with
rateLimitSlowStartSeconds. The amount of slow start is hard coded to
factor 4. During startup the rate will be 4 times lower for x seconds.
Fix consumer slow-start mode
issue
created
gabrielgerhardsson
username_0
@username_1 Can we revisit this? I rebased again and pushed now.
comment
created
udoprog
username_1
@username_0 hm, it might be more straight forward if just set a timer at start (using the global `ScheduledExecutorService`) that will update the rate accordingly instead of spinning on `isSlowStartMode` and then racing to call `setRate`.
comment
created
gabrielgerhardsson
username_0
@username_1 That's a good point, thanks. That would be much more straight forward.
I'll look at this when I get the chance.
comment
created
dmichel1
username_2
@username_0 do you think this is still needed? Do you remember what you were observing internally with the rate limiter implementation?
comment
created
gabrielgerhardsson
username_0
@username_2 The were observing spikes of requests going out from consumer nodes to ES clusters, after restart (and after reconnect, if I remember correctly). We believed it was caused by the SmoothWarmingUp rate limiter - looked like it was letting too much traffic through at the very beginning, which was drowning the ES clusters in traffic when they first came back up after being down. Possibly, the ES clusters were extra sensitive to load at the very beginning, perhaps that they were still syncing the shards among nodes etc. So we felt it important to have a proper slow start method.
We decided to try a more simple one instead. The more simple one did look good indeed. 

I'm not able to verify right now that the current code uses the same SmoothWarmingUp. I seem to remember that we switched to a straight rate limit instead (and probably choose a lower rate value).

Using the warmup implementation in this ticket could possibly allow us to increase the write rate to ES per node, while maintaining a slow rate speed at warmup, making us not drown ES. Worth looking into perhaps.
comment
created
dmichel1
username_2
@username_0 thank you for the explanation.

I verified the rate limiter is still using the same `SmoothWarmingUp` even after the upgrade to guava 20 (from 19).

One thing we observed when migrating to Google Pubsub is the cache expiration happening roughly at the same time across all consumers (because they are often restarted or deployed together). This puts a LOT of extra load on the ES clusters until the write-though-cache fills up again.  

![screen shot 2018-09-24 at 1 51 49 pm](https://user-images.githubusercontent.com/677960/45969257-2864e000-c001-11e8-8bff-020e9a07fdf0.png)

We mostly smoothed out this graph with the new consumers by adding a random amount of time to each one.

So we have not specially ran in the issue you are describing but this is very useful information for us to know so we can watch out for it.

I'm going to close this and we can re-open if this comes up again.

Thanks,
Drew
comment