saltstack/salt — pull request #15382 by felskrone (2014-08-29 13:20:46+00:00)
Title: Minions connected cache

This PR replaces the original idea discussed in PR #15005 which resulted in PR #15230.

The code in PR #15230 does its job of keeping the I/O low and reducing the system-load, but it does not work/perform the way we thought it would.

If the filesystem-cache is enabled and the numbers of minions authenticating rises, the MWorkers are effectively DOS'ed reading the grain-data over and over again even though its done through the filesystem-cache. It just does not scale querying all grains for every authenticating minion and checking for the ips within the grains.

Here is a different, working and verified approach. Its a cache that provides the list of currently connected minions to any module at any time.

Its basically the same idea as the FSCache, but instead of caching all grain-data, it caches the minion-ids that have successfully authenticated. After every successfull authentication, the new minion-id is pushed into the cache by each MWorker. Its a very cheap action and the MWorker can not block because its a pub/sub connection (MWorker/pub, ConCache/sub).

Pushing directly in the MWorkers is necessary, to keep up with the minion count for max_minions inbetween cache-cleanups.

The cache is cleaned/renewed regularly by running CkMinions.connected_ids() and updating the cache with the currently connected minions. That way the cache does not get polluted over time with minions that have disconnected inbetween the cleanup-runs.

Its been tested in my environment (6 salt-masters, 14.000 minions) and works like a charm.

For now it just caches connected minion-ids, but its easy to add for example a grains-data, mine-data etc.