Repository: ipython/ipython

Issue:
controller/server load can disrupt heartbeat
For code that uses ZMQStreams for heartbeat (notebook, hub), it is actually possible for server load to disrupt the heartbeat.  I'm not sure if this actually happens, but it is a real possibility.  I expect the most likely case is the Hub, which can have very long blocking calls when using the mongodb/sqlite task stores.

The relevant aspects of the code:
- `stream.send(msg)` _does not_ actually send a message, it registers a message for sending, which will happen at the top of the next loop iteration at the earliest.
- server load can make this loop iteration take an abitrarily long time, inserting a potentially long delay between `stream.send(msg)` and `socket.send(msg)`.
- heartbeat timer starts simultaneously with `stream.send()`, not `socket.send()`, so the above delay is interpreted as a delay in the heartbeat response, and can be longer than the total heartbeat period, causing a heart failure.

Possible solutions:
1. ZMQStreams have an 'on_send' method for registering callbacks for when the actual Socket sends the message.  If we schedule the next heartbeat relative to that, rather than as a PeriodicCallback with stream.send then the heartbeat measure would be accurate.
2. call `stream.flush()` immediately after `stream.send()`, which will cause the socket to send the message, if possible (it will never be impossible with our current patterns).
3. directly call `stream.socket.send()`, bypassing the issue entirely.



Discussion:
I am not sure I have a good sense of which of the solutions would be better.  I would say that 3 seems like the simplest option, but 2 seems almost equivalent.  1 sounds a bit more complex but perhaps it more directly targets the underlying problem.  I don't see any reason why any of these would not work though.


Resolving patch:
diff --git a/IPython/frontend/html/notebook/handlers.py b/IPython/frontend/html/notebook/handlers.py
--- a/IPython/frontend/html/notebook/handlers.py
+++ b/IPython/frontend/html/notebook/handlers.py
@@ -481,6 +481,8 @@ def ping_or_dead():
                 if self._kernel_alive:
                     self._kernel_alive = False
                     self.hb_stream.send(b'ping')
+                    # flush stream to force immediate socket send
+                    self.hb_stream.flush()
                 else:
                     try:
                         callback()
diff --git a/IPython/parallel/controller/heartmonitor.py b/IPython/parallel/controller/heartmonitor.py
--- a/IPython/parallel/controller/heartmonitor.py
+++ b/IPython/parallel/controller/heartmonitor.py
@@ -64,7 +64,7 @@ class HeartMonitor(LoggingConfigurable):
     pongstream: an XREP stream
     period: the period of the heartbeat in milliseconds"""
 
-    period = Integer(1000, config=True,
+    period = Integer(3000, config=True,
         help='The frequency at which the Hub pings the engines for heartbeats '
         '(in ms)',
     )
@@ -124,6 +124,8 @@ def beat(self):
         # print self.on_probation, self.hearts
         # self.log.debug("heartbeat::beat %.3f, %i beating hearts", self.lifetime, len(self.hearts))
         self.pingstream.send(asbytes(str(self.lifetime)))
+        # flush stream to force immediate socket send
+        self.pingstream.flush()
 
     def handle_new_heart(self, heart):
         if self._new_handlers:


Test patch: