mozilla/browserid — Issue #937: load_gen memory growth and utimate out-of-memory failure.
Author: jrgm
Created: 2012-01-23 20:17:10+00:00
State: open

While running bin/load_gen from client4 against stage servers over this past weekend I had noticed that RSS was
into the GB range. (Sorry, I wasn't tracking RSS growth systematically; next time).

Here is the related QPS graph for this period of time:
http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead_wsapi?start=2012-01-20+15%3A00%3A00&duration=48+hours&width=1000&height=400

I suspect that the growth in RSS and the decline in QPS (GH-875) are two sides of the same coin (at least partially).

Utimately the load_gen run died with these last few messages (timestamps provided by a tai64n filter that was tailing the logfile
output):
2012-01-22 13:18:03.938622500 200620.80 153967.20 182433.73 695 R, 24 S (32sp 12rs 6al 12rh 633sn 0iy 0cs)
2012-01-22 13:18:05.714453500 88231.68 140820.09 180863.70 695 R, 37 S (32sp 12rs 6al 12rh 632sn 0iy 1cs)
2012-01-22 13:18:09.957594500 393904.08 191436.89 184414.37 695 R, 53 S (32sp 12rs 6al 12rh 632sn 0iy 1cs)
2012-01-22 13:18:11.682048500 160496.64 185248.84 184015.74 695 R, 0 S (32sp 12rs 6al 12rh 632sn 0iy 1cs)
2012-01-22 13:18:15.943881500 27604.80 153720.03 181408.89 695 R, 15 S (33sp 12rs 6al 12rh 631sn 0iy 1cs)
2012-01-22 13:18:47.097894500 FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory