Router has a suspicion mechanism. bidding_agents do not ignore BYEBYE messages from router.

Concerns RTBOPT-624 as blocked by RTBKIT-486 . From statistics collected by graphite, it appears that agents *do* receive a BYEBYE message when they’re kicked out by the router.  They currently ignore this message, hence turning them into a zombie (they don’t know they’re dead). 
As far as fixing RTBKIT-486, I don’t know what the best action would be. The least we can do is introduce a callback, and expect that once deliver this callback, the only correct message from this very agent would be a DOCONFIG.  Anyone having better ideas please let me know. 

Once this is in place, we can remove this cron job put in place by Marc, forcing a campaign restart every hour.

Now about what’s triggering this condition, I don’t have a clear idea. Everyday from 6 to 9AM, the system is under abnormal I/O (both disk and network) causing its throughput to half. Combined with this machine also running a very fat and resource consuming cloud_runner_logger, this might explain why this situation occurs —> hence my first recommendation: segregate process by their levels of ‘realtimness’ 

Note that fixing RTBKIT-486 will not prevent false positive. As explained in my notes on RTBKIT-624, one easy way to alleviate this, is to introduce a suspect list.