saltstack/salt — pull request #6311 by felskrone (2013-07-25 16:40:46+00:00)
Title: changed timeout from 9999 to self.opts['timeout'] which is the user spec...

Changed the timeout from 9999 to self.opts['timeout'] which is the user specified timeout or defaults to 5. With timeout of 9999 the loop (almost) never ends if a previously detected minion vanishes/shuts down during the actual batch run.

Also added a minion_tracker dict which keeps track of the cmd_iter_no_block-iterators and their minions.

* the cmd_iter_no_block-iterators in iters[] are now cleaned up during every run if they have finished (iterating iters[] gets faster).

* minions that did not respond within the given timeout are taken care of by removing them from active[] and updating ret{} with empty returns.

These two additions enable the main while loop to actually finish before 9999 seconds are over because ret{} and self.minions are eventually the same size (with responsive and possibly unresponsive minions).

This also fixes the problem of unresponsive minions blocking the progression of the whole batch run by removing timed out minions from active[].

Example: if batch=10 and 10 minions are unresponsive, active[] never changes size until 9999 are over. That leads to range(bnum - len(active) ) (the main while loop) to always evaluate to zero and no more minions are added to next_.

This worky very nicely here. What im not sure about is returning empty responses if a minion did time out. Your thoughts?

Any comments or suggestions for improving this are welcome.