Fix the 'No xyz key' problem, improve the polling behaviour, and clean out crud

This request fixes the problem we've seen in testing where monitor output gets garbled. It avoids the race-condition by imposing a minimal delay between successive polls of the same job.

I also re-worked the polling routine to be cleaner and easier to understand, and cleaned out a lot of dead code.