reinferio/rapture — pull request #3 by cristicbz (2014-06-20 11:33:38+00:00)
Title: redesigned heartbeat and timeout system

Heartbeats are now handled separately from timeouts, which are semantically different: heartbeats are necessary to ensure a rapture node is not dead, while timeouts are meant to stop subprocesses for running too long (since that is probably a sign of a deadlock or infinite loop). Rapture still kills the process if it doesn't produce progress for longer than the heartbeat interval, but this behaviour can now be disabled.

The implementation uses three queue options now:
- 'heartbeat_secs' -- how often should updates be pushed to the job queue
- 'auto_heartbeat' -- if True, then in addition to progress messages, rapture will send heartbeats automatically every 'heartbeat_secs', to signal that the job is still being processed. If False then subprocesses will be killed if they don't produce output for longer than 'heartbeat_secs'
- 'timeout_secs' -- how long (regardless of progress) should a subprocess be allowed to run in total. A -1 value means no limit.