Repository: ipython/ipython

Issue:
IPython terminal issue with Qt4Agg on XP SP3
If matplotlib backend in 'matplotlibrc' is set to Qt4Agg, terminal console mode slows down on multiple levels: there is pause on every letter entered; if I paste some code I have to wait approx 1sec for every 2-3 characters!; and overall it seems so unnaturally slow. This does not happen however in IPython qtconsole, and I don't remember similar behavior when I used 10.2 on Windows 7. This also does not happen if matplotlib backend is set to WXAgg for example.

Noticed on:

Windows XP SP3 32 bit
Python 2.6.6 (official from python.org)
PyQt4 4.8.4
matplotlib 1.0.1
IPython 0.11.dev



Discussion:
I'm marking this as critical for 0.11 so that at least we check (I can do it) if one of us can reproduce it.  If this is systemic, it will make 0.11 unusable on Windows with Qt.

I can reproduce this problem on windows 7 64-bit.
python2.6.6 (32bit)
pyqt4 v4.8.3
matplotlib 1.0.1
ipython 0.11.dev master 1144388

However it is not a problem in the qtconsole

It does not behave like this with the tk-backend

/Jörgen

@epatters, this looks pretty nasty. Do you have any inkling as to what could be going on?  I had the impression you'd been testing with Windows a while ago, so I imagined you'd have seen this already.

It definitely is not a problem in Linux, so somehow it must be something with Qt on Windows.  I wonder if it's a Qt 4.8 problem, I think when we did the original work we were testing (I also did some light Windows testing at the time) with Qt 4.6 or 4.7...

@epatters, btw, I just realized my previous message makes it sound like I'm blaming you, which is not the case :)  Just looking for your input/wisdom...

I've been using PySide on Windows, which means that I can't use matplotlib's Qt backend. In other words, I haven't tried this out lately. When I have access to my Windows box tomorrow, I will switch back to PyQt4 and see if I can debug the problem. 

Off the top of my head, though, the behavior described above also occurs when running a PDB session under vanilla Python _if_ `QtCore.pyqtRemoveInputHook()` has not been called. For more information, see:

http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/python_shell.html

Perhaps something is not playing nice with PyQt4's input hook...

fperez skrev 2011-06-23 00:25:

> @epatters, this looks pretty nasty. Do you have any inkling as to what could be going on?  I had the impression you'd been testing with Windows a while ago, so I imagined you'd have seen this already.
> 
> It definitely is not a problem in Linux, so somehow it must be something with Qt on Windows.  I wonder if it's a Qt 4.8 problem, I think when we did the original work we were testing (I also did some light Windows testing at the time) with Qt 4.6 or 4.7...
> 
> I tried to run without pyreadline and then it behaves normally. So there 
> seems to be some kind of interaction between pyreadline and Qt4.

Unfortunately I'm leaving for a short midsummer holiday and will be back 
on sunday evening.

/Jörgen

On Thu, Jun 23, 2011 at 1:30 AM, jstenar
reply@reply.github.com
wrote:

> I tried to run without pyreadline and then it behaves normally. So there
> seems to be some kind of interaction between pyreadline and Qt4.

Thanks for the feedback though, this is already useful.

Have a good wekend!

f

Feel free to contact me with instructions if you don't have Windows XP/7 PC. I have both with IPython 0.11dev

Cheers

Well, I have no idea how I closed this issue as I only commented

huh

The "Comment and Close" button is dangerously close to the "Comment" button. Not a great usability decision there...

That's true.  I use [dotjs](http://defunkt.io/dotjs) (Userscripts/greasemonkey would work too) to float it to the left and fade it a little, so I don't click it accidentally:

``` javascript
function leftClose(){
    // float 'Comment & Close' button on the left, to prevent accidental clicking
    // also fade text to grey to make it stand out less
    // query from from userscripts.org/scripts/105325
    var andClose = document.querySelector('button[name="comment_and_close"]');
    if (andClose){
        // float it on the left
        andClose.style.float='left';
        // fade text
        andClose.style.color="#aaa";
        // or you can just remove the button altogether:
        // andClose.parentNode.removeChild(andClose);
    }
}
leftClose();
```

This is what it looks like to me:
<a target='_blank' title='Screenshot' href='http://imageshack.us/photo/my-images/861/screenshot20110624at104.png/'><img src='http://img861.imageshack.us/img861/2772/screenshot20110624at104.png' border='0'/></a>

Nice one! Thanks.

I noticed same behaviour with visvis. In this example:

<pre>In [1]: import visvis as vv
In [2]: app = vv.use()
In [3]: f = vv.clf()
In [4]: a = vv.cla()</pre>


after input [3] IPython console slows down multiple times.

Obviously regardless matplotlibrc backend settings

Just noticed that visvis default backend is set to Qt4, and when set to other no slowdowns
So this seems like 'expected behavior' then

I confirmed this problem in my Windows 7 VM. I should add that I also experience this on my OS X machine, and though the delay is _much_ less noticeable, I do not think I am imagining it.

Notice this comment from the PyQt4 docs that I linked above: "The installation of an input hook can cause problems for certain applications (particularly those that implement a similar feature using different means)."

I have no idea how to go about making these input hooks play nicely together.

Actually, further testing shows that the delay is just as bad on OS X as Windows. As it stands now, the interaction between pyreadline and PyQt4's input hook makes the terminal IPython effectively useless when using PyQt4 (this includes the Qt pylab backend).

Is there anything we can do about this before 0.11? Do we need to put a warning in the docs (or in the code)?

That's pretty rough, though at least Windows is the only place that pyreadline is normally used.  I don't see any issues when using real readline on OSX.

Hmm. If I run `ipython pylab=qt` on OS X and then plot something, I experience the delays.

On Fri, 1 Jul 2011 12:46:33 -0700 takluyver wrote:

> Is there anything we can do about this before 0.11? Do we need to put a warning in the docs (or in the code)?

Maybe just fallback on Tk or WX (if user has WX installed), as there
seems no point in running ipython console with Qt4 backend on Windows

Weird, I can't cause any problems at all.

What's your Python, matplotlib, Qt, etc. situation?

My configuration is:

Python - 2.7.1
IPython - master branch
matplotlib - 1.0.1
Qt - 4.7.3
PyQt - 4.8.2

OK, this is even weirder than I thought. Only _sometimes_ do I experience the 2-3 second delays. Sometimes it appears to work fine.

I am thoroughly baffled.

The only major difference I have is I'm using system Python 2.6, but I wouldn't expect that to make a difference.  I don 't particularly want to go through the bother of building pyqt for 2.7, but maybe I have to.

I've tried both matplotlib 1.0.1 and git master with no change.
Do you have readline, as installed by 'easy_install readline', or something else?

@epatters, just for my understanding: you're seeing the slowdown happening with normal readline, right? Pyreadline is win32-only, if I remember correctly...

I have a readline module:

<pre>
In [4]: import readline

In [5]: readline
Out[5]: module 'IPython.utils.rlineimpl' from '/Users/epatters/Local/IPython/IPython/utils/rlineimpl.pyc'
</pre>


I do not have pyreadline installed.

That will be reported no matter what readline you have. You need to check `readline._rl` to see the real module if you are in IPython.

@fperez, yes pyreadline is win32 only, but I could swear that I have installed it accidentally on OSX ages ago when I didn't know what name to easy_install.

As for what to do on the matter, since IPython requires pyreadline on Windows, we essentially have to say "don't use the matplotlib Qt backend with IPython until this gets worked out".

We should put a warning describing the issue in the Qt GUI activation if pyreadline is detected.

Thanks, @minrk, that was the source of my slightly confused question.  This looks really mysterious.  I'm quite stumped too...

@minrk, your plan for win32 sounds like the right thing to do, though I'd go as far as adding special code in there that spits out a very loud warning at runtime if people try to use the win32/pyreadline/qt combo, so the normal users who don't read docs still understand what's going on.

As for osx, total mystery...

@fperez by 'warning' I did mean actual warnings.warn inside inputhook.activate_qt4, not a comment for developers.

We will have to investigate further on osx, but I can't seem to reproduce it.

Ah, OK, thanks.  That's what I meant too, just misunderstood you.

I'm experiencing the same problems on a Mac.  Any invocation of Qt gui elements severely hangs/lags the terminal and spikes CPU usage.  This behavior is inconsistent, though, usually but not always happening, and regardless if which ipython threading flags I supply (it also happens in the regular python prompt).  My matplotlib backend is 'TkAgg'.

OS X 10.6.7
Python - 2.6.1 (Apple-supplied)
IPython - master branch
matplotlib - 1.0.0
PyQt - 4.8.4
Qt - 4.7

I'm removing this from being an 0.11 blocker, since for now all we can do is issue the warning that's been put in.  Hopefully we'll understand better what the problem is later on.


Resolving patch:
diff --git a/IPython/lib/inputhook.py b/IPython/lib/inputhook.py
--- a/IPython/lib/inputhook.py
+++ b/IPython/lib/inputhook.py
@@ -15,6 +15,7 @@
 #-----------------------------------------------------------------------------
 
 import ctypes
+import os
 import sys
 import warnings
 
@@ -31,11 +32,58 @@
 GUI_OSX = 'osx'
 GUI_GLUT = 'glut'
 GUI_PYGLET = 'pyglet'
+GUI_NONE = 'none' # i.e. disable
 
 #-----------------------------------------------------------------------------
-# Utility classes
+# Utilities
 #-----------------------------------------------------------------------------
 
+def _stdin_ready_posix():
+    """Return True if there's something to read on stdin (posix version)."""
+    infds, outfds, erfds = select.select([sys.stdin],[],[],0)
+    return bool(infds)
+
+def _stdin_ready_nt():
+    """Return True if there's something to read on stdin (nt version)."""
+    return msvcrt.kbhit()
+
+def _stdin_ready_other():
+    """Return True, assuming there's something to read on stdin."""
+    return True #
+
+
+def _ignore_CTRL_C_posix():
+    """Ignore CTRL+C (SIGINT)."""
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
+
+def _allow_CTRL_C_posix():
+    """Take CTRL+C into account (SIGINT)."""
+    signal.signal(signal.SIGINT, signal.default_int_handler)
+
+def _ignore_CTRL_C_other():
+    """Ignore CTRL+C (not implemented)."""
+    pass
+
+def _allow_CTRL_C_other():
+    """Take CTRL+C into account (not implemented)."""
+    pass
+
+if os.name == 'posix':
+    import select
+    import signal
+    stdin_ready = _stdin_ready_posix
+    ignore_CTRL_C = _ignore_CTRL_C_posix
+    allow_CTRL_C = _allow_CTRL_C_posix
+elif os.name == 'nt':
+    import msvcrt
+    stdin_ready = _stdin_ready_nt
+    ignore_CTRL_C = _ignore_CTRL_C_other
+    allow_CTRL_C = _allow_CTRL_C_other
+else:
+    stdin_ready = _stdin_ready_other
+    ignore_CTRL_C = _ignore_CTRL_C_other
+    allow_CTRL_C = _allow_CTRL_C_other
+
 
 #-----------------------------------------------------------------------------
 # Main InputHookManager class
@@ -70,6 +118,11 @@ def get_pyos_inputhook_as_func(self):
 
     def set_inputhook(self, callback):
         """Set PyOS_InputHook to callback and return the previous one."""
+        # On platforms with 'readline' support, it's all too likely to
+        # have a KeyboardInterrupt signal delivered *even before* an
+        # initial ``try:`` clause in the callback can be executed, so
+        # we need to disable CTRL+C in this situation.
+        ignore_CTRL_C()
         self._callback = callback
         self._callback_pyfunctype = self.PYFUNC(callback)
         pyos_inputhook_ptr = self.get_pyos_inputhook()
@@ -93,6 +146,7 @@ def clear_inputhook(self, app=None):
         pyos_inputhook_ptr = self.get_pyos_inputhook()
         original = self.get_pyos_inputhook_as_func()
         pyos_inputhook_ptr.value = ctypes.c_void_p(None).value
+        allow_CTRL_C()
         self._reset()
         return original
 
@@ -181,33 +235,11 @@ def enable_qt4(self, app=None):
             from PyQt4 import QtCore
             app = QtGui.QApplication(sys.argv)
         """
-        from IPython.external.qt_for_kernel import QtCore, QtGui
-
-        if 'pyreadline' in sys.modules:
-            # see IPython GitHub Issue #281 for more info on this issue
-            # Similar intermittent behavior has been reported on OSX,
-            # but not consistently reproducible
-            warnings.warn("""PyReadline's inputhook can conflict with Qt, causing delays
-            in interactive input. If you do see this issue, we recommend using another GUI
-            toolkit if you can, or disable readline with the configuration option
-            'TerminalInteractiveShell.readline_use=False', specified in a config file or
-            at the command-line""",
-            RuntimeWarning)
-        
-        # PyQt4 has had this since 4.3.1.  In version 4.2, PyOS_InputHook
-        # was set when QtCore was imported, but if it ever got removed,
-        # you couldn't reset it.  For earlier versions we can
-        # probably implement a ctypes version.
-        try:
-            QtCore.pyqtRestoreInputHook()
-        except AttributeError:
-            pass
+        from IPython.lib.inputhookqt4 import create_inputhook_qt4
+        app, inputhook_qt4 = create_inputhook_qt4(self, app)
+        self.set_inputhook(inputhook_qt4)
 
         self._current_gui = GUI_QT4
-        if app is None:
-            app = QtCore.QCoreApplication.instance()
-        if app is None:
-            app = QtGui.QApplication([" "])
         app._in_event_loop = True
         self._apps[GUI_QT4] = app
         return app
@@ -416,8 +448,8 @@ def enable_gui(gui=None, app=None):
     Parameters
     ----------
     gui : optional, string or None
-      If None, clears input hook, otherwise it must be one of the recognized
-      GUI names (see ``GUI_*`` constants in module).
+      If None (or 'none'), clears input hook, otherwise it must be one
+      of the recognized GUI names (see ``GUI_*`` constants in module).
 
     app : optional, existing application object.
       For toolkits that have the concept of a global app, you can supply an
@@ -432,6 +464,7 @@ def enable_gui(gui=None, app=None):
     one.
     """
     guis = {None: clear_inputhook,
+            GUI_NONE: clear_inputhook,
             GUI_OSX: lambda app=False: None,
             GUI_TK: enable_tk,
             GUI_GTK: enable_gtk,
diff --git a/IPython/lib/inputhookqt4.py b/IPython/lib/inputhookqt4.py
new file mode 100644
--- /dev/null
+++ b/IPython/lib/inputhookqt4.py
@@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+"""
+Qt4's inputhook support function
+
+Author: Christian Boos
+"""
+
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2011  The IPython Development Team
+#
+#  Distributed under the terms of the BSD License.  The full license is in
+#  the file COPYING, distributed as part of this software.
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+from IPython.core.interactiveshell import InteractiveShell
+from IPython.external.qt_for_kernel import QtCore, QtGui
+from IPython.lib.inputhook import allow_CTRL_C, ignore_CTRL_C, stdin_ready
+
+#-----------------------------------------------------------------------------
+# Code
+#-----------------------------------------------------------------------------
+
+def create_inputhook_qt4(mgr, app=None):
+    """Create an input hook for running the Qt4 application event loop.
+
+    Parameters
+    ----------
+    mgr : an InputHookManager
+
+    app : Qt Application, optional.
+        Running application to use.  If not given, we probe Qt for an
+        existing application object, and create a new one if none is found.
+
+    Returns
+    -------
+    A pair consisting of a Qt Application (either the one given or the
+    one found or created) and a inputhook.
+
+    Notes
+    -----
+    We use a custom input hook instead of PyQt4's default one, as it
+    interacts better with the readline packages (issue #481).
+
+    The inputhook function works in tandem with a 'pre_prompt_hook'
+    which automatically restores the hook as an inputhook in case the
+    latter has been temporarily disabled after having intercepted a
+    KeyboardInterrupt.
+    """
+
+    if app is None:
+        app = QtCore.QCoreApplication.instance()
+        if app is None:
+            app = QtGui.QApplication([" "])
+
+    # Re-use previously created inputhook if any
+    ip = InteractiveShell.instance()
+    if hasattr(ip, '_inputhook_qt4'):
+        return app, ip._inputhook_qt4
+
+    # Otherwise create the inputhook_qt4/preprompthook_qt4 pair of
+    # hooks (they both share the got_kbdint flag)
+
+    got_kbdint = [False]
+
+    def inputhook_qt4():
+        """PyOS_InputHook python hook for Qt4.
+
+        Process pending Qt events and if there's no pending keyboard
+        input, spend a short slice of time (50ms) running the Qt event
+        loop.
+
+        As a Python ctypes callback can't raise an exception, we catch
+        the KeyboardInterrupt and temporarily deactivate the hook,
+        which will let a *second* CTRL+C be processed normally and go
+        back to a clean prompt line.
+        """
+        try:
+            allow_CTRL_C()
+            app = QtCore.QCoreApplication.instance()
+            if not app: # shouldn't happen, but safer if it happens anyway...
+                return 0
+            app.processEvents(QtCore.QEventLoop.AllEvents, 300)
+            if not stdin_ready():
+                timer = QtCore.QTimer()
+                timer.timeout.connect(app.quit)
+                while not stdin_ready():
+                    timer.start(50)
+                    app.exec_()
+                    timer.stop()
+            ignore_CTRL_C()
+        except KeyboardInterrupt:
+            ignore_CTRL_C()
+            got_kbdint[0] = True
+            print("\nKeyboardInterrupt - qt4 event loop interrupted!"
+                  "\n  * hit CTRL+C again to clear the prompt"
+                  "\n  * use '%gui none' to disable the event loop"
+                  " permanently"
+                  "\n    and '%gui qt4' to re-enable it later")
+            mgr.clear_inputhook()
+        except: # NO exceptions are allowed to escape from a ctypes callback
+            mgr.clear_inputhook()
+            from traceback import print_exc
+            print_exc()
+            print("Got exception from inputhook_qt4, unregistering.")
+        return 0
+
+    def preprompthook_qt4(ishell):
+        """'pre_prompt_hook' used to restore the Qt4 input hook
+
+        (in case the latter was temporarily deactivated after a
+        CTRL+C)
+        """
+        if got_kbdint[0]:
+            mgr.set_inputhook(inputhook_qt4)
+        got_kbdint[0] = False
+
+    ip._inputhook_qt4 = inputhook_qt4
+    ip.set_hook('pre_prompt_hook', preprompthook_qt4)
+
+    return app, inputhook_qt4
diff --git a/IPython/lib/inputhookwx.py b/IPython/lib/inputhookwx.py
--- a/IPython/lib/inputhookwx.py
+++ b/IPython/lib/inputhookwx.py
@@ -24,26 +24,13 @@
 from timeit import default_timer as clock
 import wx
 
-if os.name == 'posix':
-    import select
-elif sys.platform == 'win32':
-    import msvcrt
+from IPython.lib.inputhook import stdin_ready
+
 
 #-----------------------------------------------------------------------------
 # Code
 #-----------------------------------------------------------------------------
 
-def stdin_ready():
-    if os.name == 'posix':
-        infds, outfds, erfds = select.select([sys.stdin],[],[],0)
-        if infds:
-            return True
-        else:
-            return False
-    elif sys.platform == 'win32':
-        return msvcrt.kbhit()
-
-
 def inputhook_wx1():
     """Run the wx event loop by processing pending events only.
 


Test patch: