Provide configuration hook to specify allowable heartmonitor misses from controller checking engine connectivity
Update heartmonitor miss default to be in line with new engine heartbeat monitoring. Shuts down engines if controller fails to ping them for 1 consecutive minute
Improve documentation of default to indicate consecutive and supply logging of misses
Lower default misses to 10 and improve documentation of option
Merge pull request #3290 from chapmanb/master

Improved heartbeat controller to engine monitoring for long running tasks

I'm using Ipython parallel for long running tasks (hours to days) on larger clusters (100+ engines) and have been having trouble losing connectivity with engines. The ipcontroller log would intermittently report a block of 5 to 10 `registration::unregister_engine` messages and remove those engines.

In digging into the problem, it appears as if the current behavior is to ping engines every 3 seconds and kill an engine if it fails to respond to pings twice. The first failed ping results in an engine being put `on_probation` and the second triggers unregistration.

For longer running engines this is too restrictive, and is at odds with the new `EngineFactory.max_heartbeat_misses` which defaults to 50 misses.

This pull request provides a configuration variable `HeartMonitor.max_heartmonitor_misses`, which allows ipcontrollers to specify how many consecutive misses to allow before unregistering an engine. It defaults to failing to contact an engine for 1 minute.

I'm happy to adjust or include different defaults if desired. Thanks much for all your work on IPython.
Brad