esnet/esmond — pull request #25 by mitchell-es (2014-07-30 23:25:40+00:00)
Title: Backfill investigation

I've stopped (most) of the backlog events by increasing the number of persister processes from 2 to 8 (2 was not keeping up) and increasing the memory given to memcached. The remaining ones are mostly from the Infinera's which routinely stop responding to SNMP.

I'm not running memcached with the -M option as well which causes it to return failure to the set() calls so we can log when we are having problems. Normally memcached would just silently evict something else to make room. I didn't bother trying to implement any sort of retry. Proper tuning should keep it from happening.

The deployment doc is updated as well.