Repository: spyder-ide/spyder

Issue:
Automatic graphics backend throws an error
## Description of your problem

If graphics backend for Ipython is set to "Automatic", ipython throws a ModuleNotFoundError for qt5 even if tkinter and/or GTK3 backend is installed.

**What steps will reproduce the problem?**

1. Tools -> Preferences ->IPython Console
2. Graphics tab
3. Set graphics backend to automatic

**What is the expected output? What do you see instead?**
Any code that generates graphics output should show output. Instead, there is not graphics output, only error message.

```
Traceback (most recent call last):

  File "<ipython-input-1-9701cca0214f>", line 1, in <module>
    runfile('/home/mukundan/nextcloud/documents/programming/python/misc/plot_test.py', wdir='/home/mukundan/nextcloud/documents/programming/python/misc')

  File "/usr/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "/usr/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/home/mukundan/nextcloud/documents/programming/python/misc/plot_test.py", line 9, in <module>
    import matplotlib.pyplot as plt

  File "/usr/lib64/python3.6/site-packages/matplotlib/pyplot.py", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()

  File "/usr/lib64/python3.6/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
    globals(),locals(),[backend_name],0)

ModuleNotFoundError: No module named 'matplotlib.backends.backend_qt5agg'
```

If I install matplotlib-qt5 and select automatic, I get this - 

```
 Traceback (most recent call last):

  File "<ipython-input-1-64557277dbff>", line 1, in <module>
    get_ipython().run_line_magic('matplotlib', 'qt5')

  File "/usr/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2095, in run_line_magic
    result = fn(*args,**kwargs)

  File "<decorator-gen-107>", line 2, in matplotlib

  File "/usr/lib/python3.6/site-packages/IPython/core/magic.py", line 187, in <lambda>
    call = lambda f, *a, **k: f(*a, **k)

  File "/usr/lib/python3.6/site-packages/IPython/core/magics/pylab.py", line 99, in matplotlib
    gui, backend = self.shell.enable_matplotlib(args.gui)

  File "/usr/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2983, in enable_matplotlib
    self.enable_gui(gui)

  File "/usr/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 482, in enable_gui
    real_enable_gui(gui)

  File "/usr/lib/python3.6/site-packages/ipykernel/eventloops.py", line 284, in enable_gui
    raise RuntimeError("Cannot activate multiple GUI eventloops")

RuntimeError: Cannot activate multiple GUI eventloops
```
It seems like the tk (or other non qt) backend should be detected automatically but is not happening.


**Please provide any additional information below**
If I manually select the graphics backend (inline, qt5, gtk3, tkinter), everything works as expected. Only 'automatic' seems to fail.

## Versions and main components

* Spyder Version: 3.2.4 (same problem exists in 3.2.3 as well)
* Python Version: 3.6.3 (64-bit)
* Qt Version: 5.9.2
* PyQt Version: 5.9
* Operating system: Linux (Fedora 27 beta; problem also exists in Fedora 26).


## Dependencies

Please go to the menu entry `Help > Optional Dependencies` (or
`Help > Dependencies`), press the button `Copy to clipboard`
and paste the contents below:

```
IPython >=4.0    :  6.2.1 (OK)
cython >=0.21    :  None (NOK)
jedi >=0.9.0     :  0.10.2 (OK)
nbconvert >=4.0  :  5.2.1 (OK)
numpy >=1.7      :  1.13.3 (OK)
pandas >=0.13.1  :  0.20.1 (OK)
psutil >=0.3     :  5.2.2 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pyflakes >=0.6.0 :  1.6.0 (OK)
pygments >=2.0   :  2.2.0 (OK)
pylint >=0.25    :  1.7.4 (OK)
qtconsole >=4.2.0:  4.3.0 (OK)
rope >=0.9.4     :  None (NOK)
sphinx >=0.6.6   :  1.6.4 (OK)
sympy >=0.7.3    :  1.1.1 (OK)
```

matplotlib-tk (python 3) is installed and works fine upon manual selection.
```
$ rpm -qa *matplotlib* | sort
python3-matplotlib-2.0.0-3.fc27.x86_64
python3-matplotlib-tk-2.0.0-3.fc27.x86_64
python-matplotlib-data-2.0.0-3.fc27.noarch
python-matplotlib-data-fonts-2.0.0-3.fc27.noarch
```



Discussion:
The `Automatic` option always fallback to `Qt` (4 or 5) because you need one of those to run Spyder. I didn't know that Fedora decided to break the Matplotlib Qt5 support in yet another package. Sorry, this is not our problem.

> If I install matplotlib-qt5 and select automatic

About the error you get here, do you have a `matplotlibrc` file in your home directory that sets the Matplotlib backend?
You are right. /etc/matplotlibrc was set to TkAgg and it messed it up. But, I changed that to Qt5Agg and re-ran the plot code. This is what I get - 

```
runfile('/home/test/test.py', wdir='/home/test')
Traceback (most recent call last):

  File "<ipython-input-1-53e50646982f>", line 1, in <module>
    runfile('/home/test/test.py', wdir='/home/test')

  File "/usr/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "/usr/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/home/test/test.py", line 10, in <module>
    import matplotlib.pyplot as plt

  File "/usr/lib64/python3.6/site-packages/matplotlib/pyplot.py", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()

  File "/usr/lib64/python3.6/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
    globals(),locals(),[backend_name],0)

ModuleNotFoundError: No module named 'matplotlib.backends.backend_qt4agg'
```

This does not look right. Am I missing something?
Hi,
The matplotlibrc file is located in the /etc directory, and there is no .matplotlibrc file in the home directory. 
The is a Fedora 26 installation. 
Regards,
Shadmeister.
> This does not look right. Am I missing something?

It look like an installation issue, not a Spyder problem, sorry.
> The matplotlibrc file is located in the /etc directory

So, this is a problem with Fedora setting a default backend in its Matplotlib package. There's nothing we can do about it, except catching the errors above to prevent these crashes.
Hi,
If possible, can you let me know which file is supposed to be changed when i select "automatic" in the relevant menu ?. 
Although the deployment may be slightly different on Fedora 26, it may be a simple issue such as permissions on a file, or library missing, although i have installed all relevant libraries (see : https://bugzilla.redhat.com/show_bug.cgi?id=1502871)
Thanks and regards,
Shadmeister.
> The Automatic option always fallback to Qt (4 or 5) because you need one of those to run Spyder. I didn't know that Fedora decided to break the Matplotlib Qt5 support in yet another package. Sorry, this is not our problem.

> It look like an installation issue, not a Spyder problem, sorry.

I am not sure I am following here. 

Spyder has a fallback to qt4 or qt5. In the case of Fedora, QT5 is used throughout (I can provide build logs if needed). So, in the case I mentioned above, I am setting matplotlib to QT5Agg but spyder still seems to look for QT4 instead of QT5.

Is there any other option that needs to be set?
Hi! I have more information about this. As long as I have python3-matplotlib-qt5(or -qt4) installed, I can get any plot to work just fine either running this in a terminal 

``` $ python3 plot_test.py ```

Or, by unchecking "Activate support" under graphics tab under Ipython console in the Spyder preferences window.

So, I think there is something that is being imported when spyder activates graphics support that seems to interfere. If you can give me some pointers on how I can trace what is being launched, I can probably provide more information. This would enable me to understand where the issue is (my package or elsewhere).

Thanks!

**test_plot.py**

``` python

#! /usr/bin/env python3
# -*- coding: utf-8 -*-

import matplotlib
matplotlib.use("Qt4Agg")
import matplotlib.pyplot
matplotlib.pyplot.plot([1, 2, 3])
matplotlib.pyplot.show()
```


@ccordoba12 : Could you please help out @nonamedotc with some more information. This bug is affecting me as well, so I have an own (egoisitic ;)) interest to get this bug solved.


Regards
Look, if you want to select backends at runtime (i.e. with `matplotlib.use`), it's obvious you don't need to select *Activate support* + *Automatic* in Spyder, because if you do that, then Spyder will try to set the backend for you, but you're trying to set it yourself in the first place!

And (in case you don't know), it's not possible to switch backends at runtime (with the exception of `inline`).

As a matter of fact, that's the source of the problem here: Fedora Matplotlib packagers decided to set a backend themselves (`TkAgg`), and that's conflicting with the Spyder mechanism. That's (I'd say) bad practice because it generates unexpected and confusing results. Users should set a backend themselves, or at least `inline` should be set by default (as we do), so that people can select a different backend at runtime.

Finally, about @nonamedotc's https://github.com/spyder-ide/spyder/issues/5541#issuecomment-339159363 above, where he says:

> This does not look right. Am I missing something?

I really don't know. It seems Fedora devs broke something by making a package for every Matplotlib backend (which is something crazy to do, and one of the reasons we keep recommending Anaconda above *any* other installation method). In any case, that's not Spyder problem nor bug, so it's not our fault.
Hi,
**I have removed all settings in /etc/matplotlibrc for the backend.** 
I then set the Spyder 3 environment to turn off "Activate Support". 
Restarted Spyder3.
Ran the file which included the line "matplotlib.use('Qt5Agg')"
The first time it was run, the graphic window appeared, and all is ok.
Closed the graphic window. 
The second time the same script was run, no graphic, and the warning :

```
/usr/lib64/python3.6/site-packages/matplotlib/__init__.py:1404: UserWarning:  This call to matplotlib.use() has no effect
because the backend has already been chosen;
matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

  warnings.warn(_use_error_msg)
```
The kernel hangs, so you have to restart the kernel. The call to matplotlib.use is before the stated requirements in the error message. 

As such, is this a Fedora issue or Spyder3 issue ?
Thanks and regards,
Shadmeister.
> **The second time** the same script was run, no graphic, and the warning

I said you *can't* switch backends at runtime, or wasn't I clear enough? That means you can only set your backend *once*, and that's it.

Please google to understand how Matplotlib backends work and read carefully about them before using them, and specially before blaming Spyder about failures that really don't exist.
Hi,
Ok - understood and modified the system. 
Activate Support is not selected.
/etc/matplotlibrc has been set with : 

`backend      : Qt5Agg`

No reference in the python script to **import matplotlib** or **matplotlib.use('Qt5Agg')**
Run the script - graphic appears. 
Close the graphic.
Run the script - graphic appears and the kernel hangs.
Close the graphic.

As such, is this an issue still - no errors, but the kernel hangs ?
Regards,
Shadmeister.

Hi,
For information, using the following page :
[http://www.scipy-lectures.org/intro/matplotlib/matplotlib.html](url)
And specifically the code :
`import numpy as np
import matplotlib.pyplot as plt
X = np.linspace(-np.pi, np.pi, 256, endpoint=True)
C, S = np.cos(X), np.sin(X)
plt.plot(X, C)
plt.plot(X, S)
plt.show()`

Regards,
Shadmeister.

Resolving patch:
diff --git a/spyder/utils/ipython/spyder_kernel.py b/spyder/utils/ipython/spyder_kernel.py
--- a/spyder/utils/ipython/spyder_kernel.py
+++ b/spyder/utils/ipython/spyder_kernel.py
@@ -11,6 +11,8 @@
 # Standard library imports
 import os
 import os.path as osp
+import sys
+import traceback
 
 # Third-party imports
 from ipykernel.datapub import publish_data
@@ -63,9 +65,11 @@ def __init__(self, *args, **kwargs):
         super(SpyderKernel, self).__init__(*args, **kwargs)
 
         self.namespace_view_settings = {}
+
         self._pdb_obj = None
         self._pdb_step = None
         self._do_publish_pdb_state = True
+        self._mpl_backend_error = None
 
         kernel_config = self.config.get('IPKernelApp', None)
         if kernel_config is not None:
@@ -260,7 +264,6 @@ def get_cwd(self):
 
     def get_syspath(self):
         """Return sys.path contents."""
-        import sys
         return sys.path[:]
 
     def get_env(self):
@@ -410,3 +413,49 @@ def _eval(self, text):
             return eval(text, ns), True
         except:
             return None, False
+
+    # --- Others
+    def _set_mpl_backend(self, backend, pylab=False):
+        """
+        Set a backend for Matplotlib.
+
+        backend: A parameter that can be passed to %matplotlib
+                 (e.g. inline or tk).
+        """
+        from IPython.core.getipython import get_ipython
+        generic_error = ("\n"
+                         "NOTE: The following error appeared when setting "
+                         "your Matplotlib backend\n\n"
+                         "{0}")
+
+        error = None
+        try:
+            if pylab:
+                get_ipython().run_line_magic('pylab', backend)
+            else:
+                get_ipython().run_line_magic('matplotlib', backend)
+        except RuntimeError as err:
+            # This catches errors generated by ipykernel when
+            # trying to set a backend. See issue 5541
+            if "GUI eventloops" in str(err):
+                import matplotlib
+                previous_backend = matplotlib.get_backend()
+                error = ("\n"
+                         "NOTE: Spyder *can't* set your selected Matplotlib "
+                         "backend because there is a previous backend already "
+                         "in use.\n\n"
+                         "Your backend will be {0}".format(previous_backend))
+                del matplotlib
+            # This covers other RuntimeError's
+            else:
+                error = generic_error.format(traceback.format_exc())
+        except Exception:
+            error = generic_error.format(traceback.format_exc())
+
+        self._mpl_backend_error = error
+
+    def _show_mpl_backend_errors(self):
+        """Show Matplotlib backend errors after the prompt is ready."""
+        if self._mpl_backend_error is not None:
+            print(self._mpl_backend_error)  # spyder: test-skip
+
diff --git a/spyder/utils/ipython/start_kernel.py b/spyder/utils/ipython/start_kernel.py
--- a/spyder/utils/ipython/start_kernel.py
+++ b/spyder/utils/ipython/start_kernel.py
@@ -105,12 +105,9 @@ def kernel_config():
         # Automatically load Pylab and Numpy, or only set Matplotlib
         # backend
         autoload_pylab_o = CONF.get('ipython_console', 'pylab/autoload')
-        if autoload_pylab_o:
-            spy_cfg.IPKernelApp.exec_lines.append(
-                                              "%pylab {0}".format(mpl_backend))
-        else:
-            spy_cfg.IPKernelApp.exec_lines.append(
-                                         "%matplotlib {0}".format(mpl_backend))
+        command = "get_ipython().kernel._set_mpl_backend('{0}', {1})"
+        spy_cfg.IPKernelApp.exec_lines.append(
+            command.format(mpl_backend, autoload_pylab_o))
 
         # Inline backend configuration
         if mpl_backend == 'inline':
diff --git a/spyder/widgets/ipythonconsole/client.py b/spyder/widgets/ipythonconsole/client.py
--- a/spyder/widgets/ipythonconsole/client.py
+++ b/spyder/widgets/ipythonconsole/client.py
@@ -222,6 +222,10 @@ def configure_shellwidget(self, give_focus=True):
         # To hide the loading page
         self.shellwidget.sig_prompt_ready.connect(self._hide_loading_page)
 
+        # Show possible errors when setting Matplotlib backend
+        self.shellwidget.sig_prompt_ready.connect(
+            self._show_mpl_backend_errors)
+
     def enable_stop_button(self):
         self.stop_button.setEnabled(True)
 
@@ -530,3 +534,12 @@ def _read_stderr(self):
         encoding = get_coding(stderr_text)
         stderr = to_text_string(stderr_text, encoding)
         return stderr
+
+    def _show_mpl_backend_errors(self):
+        """
+        Show possible errors when setting the selected Matplotlib backend.
+        """
+        self.shellwidget.silent_execute(
+            "get_ipython().kernel._show_mpl_backend_errors()")
+        self.shellwidget.sig_prompt_ready.disconnect(
+            self._show_mpl_backend_errors)


Test patch:
diff --git a/spyder/app/tests/test_mainwindow.py b/spyder/app/tests/test_mainwindow.py
--- a/spyder/app/tests/test_mainwindow.py
+++ b/spyder/app/tests/test_mainwindow.py
@@ -18,7 +18,7 @@
 import numpy as np
 from numpy.testing import assert_array_equal
 import pytest
-from qtpy import PYQT5, PYQT_VERSION
+from qtpy import PYQT4, PYQT5, PYQT_VERSION
 from qtpy.QtCore import Qt, QTimer
 from qtpy.QtTest import QTest
 from qtpy.QtWidgets import QApplication, QFileDialog, QLineEdit
@@ -171,7 +171,8 @@ def test_calltip(main_window, qtbot):
 
 
 @flaky(max_runs=3)
-@pytest.mark.skipif(os.name == 'nt' or PY2, reason="It fails sometimes")
+@pytest.mark.skipif(os.name == 'nt' or PY2 or PYQT4,
+                    reason="It fails sometimes")
 def test_move_to_first_breakpoint(main_window, qtbot):
     """Test that we move to the first breakpoint if there's one present."""
     # Wait until the window is fully up