cylc/cylc — pull request #462 by hjoliver (2013-05-11 04:29:12+00:00)
Title: Job poll and kill etc

This addresses the following issues: #67, #299, #350, #381, #461

A new job polling mechanism uses the stored job submit ID and the automatically generated task status files to determine the actual status of tasks that are supposed to be submitted or running.
* new commands ```cylc poll``` and ```cylc kill``` to manually poll or kill real jobs (and update suite status accordingly)
* automatic polling on task submission or execution timeouts
* automatic polling on suite restart, to magically determine what happened to submitted or running tasks that were orphaned when the suite went down.
* polling as an alternative method of tracking task status, for task hosts that do not allow any network socket or ssh communication back to the suite host.

Implemented so far for the **background**, **at**, and **loadleveler** job submission methods (local and remote tasks). Not yet done for **pbs** or **sge** (I can't test these at NIWA). To extend these will be simple: new functions in each job submission module must return strings of shell scripting to (a) determine job queued or running status or (b) kill a job - given the job submission method-specific submit ID - taking the background, at, and loadleveler modules as examples.