Repository: spyder-ide/spyder

Issue:
IPython console hangs after changing warning to errors
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

Run <F9> the following code:

    import warnings
    warnings.filterwarnings("error")

Ipython then seems to break.



## Versions

* Spyder version: 4.1.3 
* Python version: 3.7.6
* Qt version: 5.9.6
* PyQt5 version: 5.9.2
* Operating System: Windows 10

### Dependencies

```

# Mandatory:
atomicwrites >=1.2.0           :  1.4.0 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  1.4.1 (OK)
diff_match_patch >=20181111    :  20181111 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.13.0 (OK)
jedi =0.15.2                   :  0.15.2 (OK)
nbconvert >=4.0                :  5.6.1 (OK)
numpydoc >=0.6.0               :  0.9.2 (OK)
paramiko >=2.4.0               :  2.7.1 (OK)
parso =0.5.2                   :  0.5.2 (OK)
pexpect >=4.4.0                :  4.8.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.7.0 (OK)
pygments >=2.0                 :  2.6.1 (OK)
pylint >=0.25                  :  2.5.2 (OK)
pyls >=0.31.9;<0.32.0          :  0.31.10 (OK)
qdarkstyle >=2.8               :  2.8.1 (OK)
qtawesome >=0.5.7              :  0.7.0 (OK)
qtconsole >=4.6.0              :  4.7.4 (OK)
qtpy >=1.5.0                   :  1.9.0 (OK)
rtree >=0.8.3                  :  0.9.4 (OK)
sphinx >=0.6.6                 :  3.0.3 (OK)
spyder_kernels >=1.9.1;<1.10.0 :  1.9.1 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  18.1.1 (OK)

# Optional:
cython >=0.21                  :  0.29.17 (OK)
matplotlib >=2.0.0             :  3.1.3 (OK)
numpy >=1.7                    :  1.18.1 (OK)
pandas >=0.13.1                :  1.0.3 (OK)
scipy >=0.17.0                 :  1.4.1 (OK)
sympy >=0.7.3                  :  1.5.1 (OK)
```

IPython console hangs after changing warning to errors
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

Run <F9> the following code:

    import warnings
    warnings.filterwarnings("error")

Ipython then seems to break.



## Versions

* Spyder version: 4.1.3 
* Python version: 3.7.6
* Qt version: 5.9.6
* PyQt5 version: 5.9.2
* Operating System: Windows 10

### Dependencies

```

# Mandatory:
atomicwrites >=1.2.0           :  1.4.0 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  1.4.1 (OK)
diff_match_patch >=20181111    :  20181111 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.13.0 (OK)
jedi =0.15.2                   :  0.15.2 (OK)
nbconvert >=4.0                :  5.6.1 (OK)
numpydoc >=0.6.0               :  0.9.2 (OK)
paramiko >=2.4.0               :  2.7.1 (OK)
parso =0.5.2                   :  0.5.2 (OK)
pexpect >=4.4.0                :  4.8.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.7.0 (OK)
pygments >=2.0                 :  2.6.1 (OK)
pylint >=0.25                  :  2.5.2 (OK)
pyls >=0.31.9;<0.32.0          :  0.31.10 (OK)
qdarkstyle >=2.8               :  2.8.1 (OK)
qtawesome >=0.5.7              :  0.7.0 (OK)
qtconsole >=4.6.0              :  4.7.4 (OK)
qtpy >=1.5.0                   :  1.9.0 (OK)
rtree >=0.8.3                  :  0.9.4 (OK)
sphinx >=0.6.6                 :  3.0.3 (OK)
spyder_kernels >=1.9.1;<1.10.0 :  1.9.1 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  18.1.1 (OK)

# Optional:
cython >=0.21                  :  0.29.17 (OK)
matplotlib >=2.0.0             :  3.1.3 (OK)
numpy >=1.7                    :  1.18.1 (OK)
pandas >=0.13.1                :  1.0.3 (OK)
scipy >=0.17.0                 :  1.4.1 (OK)
sympy >=0.7.3                  :  1.5.1 (OK)
```



Discussion:
Hi @smetterspa, thanks for reporting this issue, I was able to reproduce it. @impact27 @dalthviz, do you know what could be the cause of the console freeze?
I'm unsure why this is happening but when you use the `Clear console` option  you can see this:

![image](https://user-images.githubusercontent.com/16781833/84447182-f9dfcc00-ac0c-11ea-9f4d-51994da275f5.png)

Can't reproduce on macOS:
``` python
Python 3.7.7 (default, Mar 10 2020, 15:43:33) 
Type "copyright", "credits" or "license" for more information.

IPython 7.10.2 -- An enhanced Interactive Python.

In [1]: import warnings
   ...: warnings.filterwarnings("error")

In [2]: 1+1
Out[2]: 2

In [3]: 
```
It is not terribly surprising, if you transform warnings into errors, any warning that would be generated by the kernel gets upgraded to an error which means the kernel can crash really easily.
On windows using `jupyuter console`:
```
In [4]: 1+1[IPKernelApp] ERROR | Exception in message handler:
Traceback (most recent call last):
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 702, in is_complete_request
    reply_content = yield gen.maybe_future(self.do_is_complete(code))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 454, in do_is_complete
    status, indent_spaces = self.shell.input_splitter.check_complete(code)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 503, in input_splitter
    DeprecationWarning, stacklevel=2
DeprecationWarning: `input_splitter` is deprecated since IPython 7.0, prefer `input_transformer_manager`.
```
I would say we can close because this is not a spyder issue and there is realistically not much we can do about that. There will always be some code that might break the kernel. We could however display a warning if the kernel stops responding.
We could print the content of stderr with QFileSystemWatcher. From the internal console:
```python
>>> client = spy.window.ipyconsole.get_current_client()
>>> client.stderr_file
'C:\\Users\\QUENTI~1.PET\\AppData\\Local\\Temp\\spyder\\kernel-a2eddad5daa9.stderr'

>>> with open(client.stderr_file, 'r') as f: print(f.read())
... 
[IPKernelApp] ERROR | Exception in message handler:
Traceback (most recent call last):
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 702, in is_complete_request
    reply_content = yield gen.maybe_future(self.do_is_complete(code))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 454, in do_is_complete
    status, indent_spaces = self.shell.input_splitter.check_complete(code)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 503, in input_splitter
    DeprecationWarning, stacklevel=2
DeprecationWarning: `input_splitter` is deprecated since IPython 7.0, prefer `input_transformer_manager`.

>>> 
```
Hi @smetterspa, thanks for reporting this issue, I was able to reproduce it. @impact27 @dalthviz, do you know what could be the cause of the console freeze?
I'm unsure why this is happening but when you use the `Clear console` option  you can see this:

![image](https://user-images.githubusercontent.com/16781833/84447182-f9dfcc00-ac0c-11ea-9f4d-51994da275f5.png)

Can't reproduce on macOS:
``` python
Python 3.7.7 (default, Mar 10 2020, 15:43:33) 
Type "copyright", "credits" or "license" for more information.

IPython 7.10.2 -- An enhanced Interactive Python.

In [1]: import warnings
   ...: warnings.filterwarnings("error")

In [2]: 1+1
Out[2]: 2

In [3]: 
```
It is not terribly surprising, if you transform warnings into errors, any warning that would be generated by the kernel gets upgraded to an error which means the kernel can crash really easily.
On windows using `jupyuter console`:
```
In [4]: 1+1[IPKernelApp] ERROR | Exception in message handler:
Traceback (most recent call last):
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 702, in is_complete_request
    reply_content = yield gen.maybe_future(self.do_is_complete(code))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 454, in do_is_complete
    status, indent_spaces = self.shell.input_splitter.check_complete(code)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 503, in input_splitter
    DeprecationWarning, stacklevel=2
DeprecationWarning: `input_splitter` is deprecated since IPython 7.0, prefer `input_transformer_manager`.
```
I would say we can close because this is not a spyder issue and there is realistically not much we can do about that. There will always be some code that might break the kernel. We could however display a warning if the kernel stops responding.
We could print the content of stderr with QFileSystemWatcher. From the internal console:
```python
>>> client = spy.window.ipyconsole.get_current_client()
>>> client.stderr_file
'C:\\Users\\QUENTI~1.PET\\AppData\\Local\\Temp\\spyder\\kernel-a2eddad5daa9.stderr'

>>> with open(client.stderr_file, 'r') as f: print(f.read())
... 
[IPKernelApp] ERROR | Exception in message handler:
Traceback (most recent call last):
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 702, in is_complete_request
    reply_content = yield gen.maybe_future(self.do_is_complete(code))
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 454, in do_is_complete
    status, indent_spaces = self.shell.input_splitter.check_complete(code)
  File "C:\Users\Quentin.Peter\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 503, in input_splitter
    DeprecationWarning, stacklevel=2
DeprecationWarning: `input_splitter` is deprecated since IPython 7.0, prefer `input_transformer_manager`.

>>> 
```

Resolving patch:
diff --git a/spyder/plugins/ipythonconsole/widgets/client.py b/spyder/plugins/ipythonconsole/widgets/client.py
--- a/spyder/plugins/ipythonconsole/widgets/client.py
+++ b/spyder/plugins/ipythonconsole/widgets/client.py
@@ -177,6 +177,13 @@ def __init__(self, plugin, id_,
         # Show timer
         self.update_time_label_visibility()
 
+        # Poll for stderr changes
+        self.stderr_mtime = 0
+        self.stderr_timer = QTimer(self)
+        self.stderr_timer.timeout.connect(self.poll_stderr_file_change)
+        self.stderr_timer.setInterval(1000)
+        self.stderr_timer.start()
+
     def __del__(self):
         """Close threads to avoid segfault"""
         if (self.restart_thread is not None
@@ -240,6 +247,22 @@ def get_stderr_contents(self):
             stderr = None
         return stderr
 
+    @Slot()
+    def poll_stderr_file_change(self):
+        """Check if the stderr file just changed"""
+        try:
+            mtime = os.stat(self.stderr_file).st_mtime
+        except Exception:
+            return
+
+        if mtime == self.stderr_mtime:
+            return
+        self.stderr_mtime = mtime
+        stderr = self.get_stderr_contents()
+        if stderr:
+            self.shellwidget._append_plain_text(
+                '\n' + stderr, before_prompt=True)
+
     def configure_shellwidget(self, give_focus=True):
         """Configure shellwidget after kernel is connected."""
         if give_focus:


Test patch:
diff --git a/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py b/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
--- a/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
+++ b/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
@@ -1699,5 +1699,19 @@ def test_kernel_restart_after_manual_restart_and_crash(ipyconsole, qtbot):
             'status'] == 'ready')
 
 
+@flaky(max_runs=3)
+def test_stderr_poll(ipyconsole, qtbot):
+    """Test if the content of stderr is printed to the console."""
+    shell = ipyconsole.get_current_shellwidget()
+    qtbot.waitUntil(lambda: shell._prompt_html is not None,
+                    timeout=SHELL_TIMEOUT)
+    client = ipyconsole.get_current_client()
+    with open(client.stderr_file, 'w') as f:
+        f.write("test_test")
+    # Wait for the poll
+    qtbot.wait(2000)
+    assert "test_test" in ipyconsole.get_focus_widget().toPlainText()
+
+
 if __name__ == "__main__":
     pytest.main()