mozilla/browserid — Issue #876: load_gen shows decline in QPS over time
Author: jbonacci
Created: 2012-01-11 20:00:23+00:00
State: open

I have been watching our load_gen on Stage for the last 40 hours.
I have noticed in the main QPS graphs for each service (*_QPS_200 / Stage) that avg QPS continues to decline at a steady (almost linear) rate.
Since it does not seem to be leveling off, I have to assume that at some point, we will approach 0.

Here is a sample graph:
http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead?start=40+hours+ago&duration=&width=800&height=400
(adjust accordingly to fit the time frame of 01-09-2012 through 01-11-2012 PST)

Clicking on Zoom, shows similar graphs for "Summary" and the three webheads, similar for other services.

It's odd, and as @fetep points out, the response times aren't spiking up as throughput goes down

QA will start a new load test, same options to see if we can duplicate the scenario.