saltstack/salt — pull request #10063 by mbirtwell (2014-01-30 17:00:35+00:00)
Title: Fix a case when job returns are missed by salt because saltutil.find_job times out

So let's start with my problem.

I was running a salt-call state.sls on my salt-master which made use of salt RC style states. In particular it had two salt.state states in it. The first did a highstate on a minion and the second targeted the same minion and with a particular state sls. The second state was failing with the error that state.highstate was already running.

What was happening was that the call to LocalClient.cmd_iter for the first highstate was timing out, being treated as success and allowing the second to start too early. This occurred when one of the saltutil.find_job calls timed out. The log showed that the find_job call was being processed quickly by the minion but not until after it had already timed out on the master. The minion was delayed starting find_job because it was busy processing refresh_module events. These were being generated by the original highstate. Each one could take as much as 4 seconds to process and if two were queued then it frequently over ran the 2 second time out set in LocalClient.gather_job_info.

My patch changes the salt-minion to use only one poller. This means that the salt-minion can interleave the handling of execution requests and events. Significantly reducing the chance of this problem occurring. It was happening every time for me before this patch and has not happened at all for me yet with it.

I realise that changes to the core of salt-minion might be controversial and I assume whom ever decided that it needed two overlapping polling loops had good reason for it. But I thought I'd give you the patch to see what you think.

Also I don't have any information at the moment whether this fix is relevant to any of the other problems that people have seen with jobs timing out.