rtbkit/rtbkit — pull request #159 by jsulmont (2014-03-14 09:32:21+00:00)
Title: Router has a suspicion mechanism. bidding_agents do not ignore BYEBYE messages from router.

Concerns RTBOPT-624 as blocked by RTBKIT-486 . From statistics collected by graphite, it appears that agents *do* receive a BYEBYE message when they’re kicked out by the router. They currently ignore this message, hence turning them into a zombie (they don’t know they’re dead).
As far as fixing RTBKIT-486, I don’t know what the best action would be. The least we can do is introduce a callback, and expect that once deliver this callback, the only correct message from this very agent would be a DOCONFIG. Anyone having better ideas please let me know.

Once this is in place, we can remove this cron job put in place by Marc, forcing a campaign restart every hour.

Now about what’s triggering this condition, I don’t have a clear idea. Everyday from 6 to 9AM, the system is under abnormal I/O (both disk and network) causing its throughput to half. Combined with this machine also running a very fat and resource consuming cloud_runner_logger, this might explain why this situation occurs —> hence my first recommendation: segregate process by their levels of ‘realtimness’

Note that fixing RTBKIT-486 will not prevent false positive. As explained in my notes on RTBKIT-624, one easy way to alleviate this, is to introduce a suspect list.