Repository: ipython/ipython

Issue:
Add checking/flushing of background output from kernel in mainloop
"""
zmq sockets are not threadsafe. To protect against segfaults, background threads are not allowed to write directly to the IOPub socket. What they do instead is write to a buffer, which the main thread checks periodically. That's why async output from a thread doesn't show up until there is some interaction in the main thread.

It would be easy enough to hook up that checking / flushing to the main eventloop, in which case it would still be threadsafe, but still allow publication while the kernel is idle. But that's now how it works right now.

-MinRK
"""

Sample code:

<pre>
import threading
import time
class ThreadClass(threading.Thread):
    def run(self):
        x = 1
        while True:
            print(x)
            x += 1
            time.sleep(1)
t = ThreadClass()
t.start()
t.join()
</pre>


Then you will see the data for as long as you don't interrupt the
cell. If you comment out the t.join(), then you won't see any output
(or maybe just the first printed).

In the next cell, if you do something such as, say:

<pre>
while True:
    pass
</pre>


and then interrupt the kernel, you'll see some more of the printouts.



Discussion:
I actually already wrote up a fix for this in the kernel, but ran right into #5407 when I did. We'll figure it out eventually.


Resolving patch:
diff --git a/IPython/kernel/zmq/iostream.py b/IPython/kernel/zmq/iostream.py
--- a/IPython/kernel/zmq/iostream.py
+++ b/IPython/kernel/zmq/iostream.py
@@ -1,12 +1,7 @@
-"""wrappers for stdout/stderr forwarding over zmq
-"""
+"""Wrappers for forwarding stdout/stderr over zmq"""
 
-#-----------------------------------------------------------------------------
-#  Copyright (C) 2013  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-----------------------------------------------------------------------------
+# Copyright (c) IPython Development Team.
+# Distributed under the terms of the Modified BSD License.
 
 import os
 import threading
@@ -15,6 +10,7 @@
 from io import StringIO, UnsupportedOperation
 
 import zmq
+from zmq.eventloop.ioloop import IOLoop
 
 from .session import extract_header
 
@@ -130,6 +126,17 @@ def _flush_from_subprocesses(self):
             else:
                 break
     
+    def _schedule_flush(self):
+        """schedule a flush in the main thread
+        
+        only works with a tornado/pyzmq eventloop running
+        """
+        if IOLoop.initialized():
+            IOLoop.instance().add_callback(self.flush)
+        else:
+            # no async loop, at least force the timer
+            self._start = 0
+    
     def flush(self):
         """trigger actual zmq send"""
         if self.pub_socket is None:
@@ -140,9 +147,9 @@ def flush(self):
         if mp_mode != CHILD:
             # we are master
             if not self._is_master_thread():
-                # sub-threads must not trigger flush,
-                # but at least they can force the timer.
-                self._start = 0
+                # sub-threads must not trigger flush directly,
+                # but at least they can schedule an async flush, or force the timer.
+                self._schedule_flush()
                 return
             
             self._flush_from_subprocesses()


Test patch: