Repository: spyder-ide/spyder

Issue:
IPython magics no longer work in Spyder 4
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

### Issue Report Checklist

* [x] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [ ] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [x] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [x] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [x] Tried basic troubleshooting (if a bug/error)
    * [x] Restarted Spyder
    * [x] Reset preferences with ``spyder --reset``
    * [x] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [x] Tried the other applicable steps from the Troubleshooting Guide
* [x] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description
If we have ipython magic commands in the editor, running them with shift+enter no longer works with Spyder 4
```
%who print
# %%
```
```

runcell(0, 'C:/users/mark/git/vispy/vispy/app/untitled1.py')
  File "C:\users\mark\git\vispy\vispy\app\untitled1.py", line 1
    %who print
    ^
SyntaxError: invalid syntax
```
![image](https://user-images.githubusercontent.com/90008/70754963-911fa180-1d06-11ea-9003-667896516546.png)


### What is the expected output? What do you see instead?

It used to work in Spyder 3, has the decision been made to revert things?

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version:  4.0.0
* Python version:  3.7.3 64-bit
* Qt version: 5.12.5
* PyQt version: 5.12.3
* Operating System name/version: Windows / Linux



Discussion:
@impact27 @bcolsen ?
IPython magics are not valid Python code, so we decided to not support them anymore in Python files. That will avoid the common problem of files that work in Spyder but not outside of it.

If you want to keep using them, you need to run them like

```python
from IPython import get_ipython
get_ipython().magic('who print')
```

I think @impact27 added support for `.ipy` files, i.e. IPython script files. In those files you can use magics freely.
hmm ok, got it
Thanks for the thorough explication 
Yes you can currently use line magics in .ipy files. 

There is no support for output of the last command to be printed or cellmagics

If you're interested in those features I don't think we have a bug tracking those anymore but you can submit one here:
https://github.com/spyder-ide/spyder-kernels/issues

See more debate about it here: https://github.com/spyder-ide/spyder-kernels/issues/139
Is it possible to go through a deprecation cycle for this? Whereby users are told exactly what you told me about magics not being `python` but rather `ipython specific` and that they should save as `ipy` and rerun their file?
Alternatively we could create ipy files instead of py files by default? The problem is also that magic is detected as an error in the editor.

> On 14 Dec 2019, at 16:06, Mark Harfouche <notifications@github.com> wrote:
> 
> ﻿
> Is it possible to go through a deprecation cycle for this? Whereby users are told exactly what you told me about magics not being python but rather ipython specific and that they should save as ipy and rerun their file?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

> The problem is also that magic is detected as an error in the editor.


By the PyLS ? I could add a fix for that (at least on the editor side for a quick fix)?
@ccordoba12 I would like to request this issue be reopened. While I understand the desire to change the behavior, I feel like this was a key feature of Spyder that has since been removed with no guidance to users that don't fully appreciate the difference between python/ipython.
> Alternatively we could create ipy files instead of py files by default?

I don't agree with this. Magics are more of a convenience for interactive computing than something people should be using in production code.

> I could add a fix for that (at least on the editor side for a quick fix)?

I don't think that's a good idea because magics are invalid Python syntax.
Does PyLS work with ipy files? We could try to detect ipython magic and have an error message like: ipython magic are not valid python code. Please save your file with an ipython extension (.ipy) for them to work.

> On 14 Dec 2019, at 16:31, Gonzalo Peña-Castellanos <notifications@github.com> wrote:
> 
> ﻿
> The problem is also that magic is detected as an error in the editor.
> 
> By the PyLS ? I could add a fix for that?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

>  I would like to request this issue be reopened

Allowing magics will break linting, folding and indent guidelines in the editor because they generate syntax errors. So I'm afraid you'll have to use the code I gave you above to run them.

We can ask @juanis2112 to document that while she's improving our documentation though.
> Does PyLS work with ipy files?

We'd need to extend it to work with them, but I don't think they'll work well with Pylint and Pyflakes.

> We could try to detect ipython magic and have an error message like: ipython magic are not valid python code

Another option would be to parse the Python file sent to the PyLS before passing its contents to the linters, and transform magics to valid Python code.

I think IPython has some machinery to do that already.
Maybe the difference is that pressing "shift + enter" used to copy the code into the console. Which was really good to **see** exactly what was being typed, this made magics work.

So now we have a bunch of Spyder 3 scripts that used to work, and we love the work you have done with Spyder 4. Unfortunately the error message:

```

  File "/home/mark/git/delay_test.py", line 11
    %time
    ^
SyntaxError: invalid syntax
```
isn't very helpful because this wasn't a syntax error **before**. Something like:

```
SyntaxError: invalid syntax. In Spyder 4, we no longer allow iPython magics in .py files. If you want to use iPython magics, please save your file as .ipy the iPython file extension.
```

Would go a long way.
Even something like:

Detecting that the user is trying to use ipython magics upon save/linting in an `.py` file, and issuing them a warning, or option to resave their file as a `.ipy` to enable the magics to work.
> Maybe the difference is that pressing "shift + enter" used to copy the code into the console

Ok, you should have started there. If you want the previous behavior back, you need to go to

    Preferences > Editor > Run code

and activate the option called `Copy full cell contents to the console when running code cells`.
> and activate the option called Copy full cell contents to the console when running code cells.

I guess I'm trying to say that I've seen 4/4 users of Spyder 3 migrate to Spyder 4 and been tripped up by this behavior.

People posting on this thread are to say the least, pretty advanced users of python/spyder. This isn't the norm.

I agree with your desire to help people write better code, but I would consider this a breaking change in the user experience, and some kind of guidance would be helpful.

It seems that you would prefer users not to use `Copy full cell contents to the console when running code cells`, but rather to save their files as `ipy` to keep them portable.

A little guidance will go a long way.

Changing *how* code is excuted (copied or now) can be ignored. The fact that  scripts developed in spyder no longer work with spyder is not really fun.
Thanks, I really appreciate understanding our usecase. I also appreciate the pointer to chaging the editor behavior to copy the contents of the cell.
> It seems that you would prefer users not to use `Copy full cell contents to the console when running code cells`, but rather to save their files as ipy to keep them portable.

The thing is not copying cell contents to the console allows pretty advanced things like debugging cells, which was added by @impact27.

The easiest thing for us to do would be to parse the file and replace magics by valid Python code before evaluating them in the kernel. @impact27, what do you think?

A big problem is allowing magics usage is detrimental because it'll break a lot of stuff in the editor. However, we could make some adjustments in the PyLS to deal with that (again, parsing the file to transform magics). @goanpeca, what do you think?
> The easiest thing for us to do would be to parse the file and replace magics by valid Python code before evaluating them in the kernel. @impact27, what do you think?

You mean for python files as well? It would be very easy but then we need to do the same before sending to PyLS. But then we have the problem that a .py file might not work outside of spyder. 
>A big problem is allowing magics usage is detrimental because it'll break a lot of stuff in the editor. However, we could make some adjustments in the PyLS to deal with that (again, parsing the file to transform magics). @goanpeca, what do you think?

I am a bit torn, the previous behavior was broken cause `% time` is not valid python, so even if it used to work, it does not mean it was right for it to work I mean if it was something like

```python
# %time
```

Then we would not need to really fix anything cause that is a normal comment, but we could add some support for "known" magics and strip the `# ` when sending it to the console, (we could have an option that was ON by default to "parse known ipython magics"

This feels like the best  since we do not need to customize the PYLS, we enforce old users to write VALID python code on the editor (with minimal effort), and provide a workaround on top of this to make life easy for users.

By doing this we could also add a better error message for users to indicate them that `%time` is not valid code, that they should use `# %time` instead.

Thoughts @impact27 , @ccordoba12 @hmaarrfk 
I think for python 4 we should go back to using the run cell mechanism in already in ipython.

This fixes line magics, cell magics (like %%timeit), outputs of the last cell line and indentation errors we are having. This is the way the runcell function was originally implemented.

I think the only disadvantage (@impact27  can correct me here) was that the execution name was a hash instead of the real file name. Maybe debug cell would work with magic but I don't know what one should do in that case.

I think that the current plan we have here for magic in only ipy file is a great way forward for spyder 5 when we have proper support for .ipy in the interface. I would like to see an option in for a new ipy file and an explanation. I would also like to cell by cell execution for .ipy files when I use runfile(F5).

> I think for python 4 we should go back to using the run cell mechanism in already in ipython.

I guess you mean Spyder 4 above. If we decide to do that, we should change the default for 4.0.1 (to be released next week), instead of leaving more time with the current one.

What do other people think about it? I'd prefer to leave the current default (i.e. using `runcell`).
Is there a way to have the new mechanism be transparent?

I.e. make the console "appear" as though things were type in, but in the background use the new "runcell" command?
No, I don't think so. The whole point of `runcell` is to not copy the contents of cells to the console.
Sorry yes Spyder 4.  We would use Spyder kernel runcell() but change out the exec() call to use ipython.run_cell() instead. We will loose filenames in the tracebacks but the links and the line numbers will work. It shouldn't be to hard to do, I had it working before.

I want to reiterate that having a .ipy file in Spyder 5 that treats a text file with code cells like a set of ipython cells would be incredible, but we would have to educate about the change or maybe even make our own .spy Spyder script files since .ipy doesn't actually support cells.
I think we should:
- reenable line magic in .py files
- show a warning and suggest either
    - transforming to python code (we could have a button to do that in the source menu)
    - saving as .ipy
- in spyder5 have the warning become an error 
> On 14 Dec 2019, at 19:03, bcolsen <notifications@github.com> wrote:
> 
> ﻿
> Sorry yes Spyder 4. We would use Spyder kernel runcell() but change out the exec() call to use ipython.run_cell() instead. We will loose filenames in the tracebacks but the links and the line numbers will work. It shouldn't be to hard to do, I had it working before.
> 
> I want to reiterate that having a .ipy file in Spyder 5 that treats a text file with code cells like a set of ipython cells would be incredible, but we would have to educate about the change or maybe even make our own .spy Spyder script files since .ipy doesn't actually support cells.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

So I had a look and I think the work to be done is, in `spyder_kernels/customize/spydercustomize.py::exec_code`: 
 - call the transform manager even if `is_ipython` is false
 - Print a warning if the transform manager made something (Easiest would be to try to compile without the manager and checking if a syntaxerror is raised.
 - fixing the cell magic by not calling `tm.cleanup_transforms = []`. Instead, count the number of blank lines removed and add them back after calling `code = tm.transform_cell(code)` (Otherwise the callback will be messed up.)

Then there is a few outstanding problems that needs to be addressed:
 - `runfile` won't work with cells magic (line magic are fine) as `tm.transform_cell` only transforms a single cell. Maybe we should iterate on cells?
 - `tm.transform_cell` places the entire cell on one line if a cell magic is used, so any traceback and debugging will be messed up. I am not sure how to fix this one but I don't expect that problem to be super commun.
 - We need to call `tm.transform_cell` before sending the python code to PyLS

Any thoughts?
Implemented in https://github.com/spyder-ide/spyder-kernels/pull/190. I also opened https://github.com/spyder-ide/spyder/issues/11090
Thanks for your help @impact27! Since this too much for a simple bugfix release, I'm going to leave this for 4.1.0
#11101 would improve integration of ipython magic in spyder.

Resolving patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/console/start.py b/external-deps/spyder-kernels/spyder_kernels/console/start.py
--- a/external-deps/spyder-kernels/spyder_kernels/console/start.py
+++ b/external-deps/spyder-kernels/spyder_kernels/console/start.py
@@ -264,7 +264,8 @@ def varexp(line):
     except:
         import matplotlib.pyplot as pyplot
     __fig__ = pyplot.figure();
-    __items__ = getattr(pyplot, funcname[2:])(ip.user_ns[name])
+    __items__ = getattr(pyplot, funcname[2:])(
+        ip.kernel._get_current_namespace()[name])
     pyplot.show()
     del __fig__, __items__
 
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
@@ -52,6 +52,7 @@
 # =============================================================================
 IS_EXT_INTERPRETER = os.environ.get('SPY_EXTERNAL_INTERPRETER') == "True"
 HIDE_CMD_WINDOWS = os.environ.get('SPY_HIDE_CMD') == "True"
+SHOW_INVALID_SYNTAX_MSG = True
 
 
 # =============================================================================
@@ -143,6 +144,44 @@ def _getfilesystemencoding_wrapper():
         pass
 
 
+# =============================================================================
+# Patch PyQt4 and PyQt5
+# =============================================================================
+# This saves the QApplication instances so that Python doesn't destroy them.
+# Python sees all the QApplication as differnet Python objects, while
+# Qt sees them as a singleton (There is only one Application!). Deleting one
+# QApplication causes all the other Python instances to become broken.
+# See spyder-ide/spyder/issues/2970
+try:
+    from PyQt5 import QtWidgets
+
+    class SpyderQApplication(QtWidgets.QApplication):
+        def __init__(self, *args, **kwargs):
+            super(SpyderQApplication, self).__init__(*args, **kwargs)
+            # Add reference to avoid destruction
+            # This creates a Memory leak but avoids a Segmentation fault
+            SpyderQApplication._instance_list.append(self)
+
+    SpyderQApplication._instance_list = []
+    QtWidgets.QApplication = SpyderQApplication
+except Exception:
+    pass
+
+try:
+    from PyQt4 import QtGui
+
+    class SpyderQApplication(QtGui.QApplication):
+        def __init__(self, *args, **kwargs):
+            super(SpyderQApplication, self).__init__(*args, **kwargs)
+            # Add reference to avoid destruction
+            # This creates a Memory leak but avoids a Segmentation fault
+            SpyderQApplication._instance_list.append(self)
+
+    SpyderQApplication._instance_list = []
+    QtGui.QApplication = SpyderQApplication
+except Exception:
+    pass
+
 # =============================================================================
 # IPython adjustments
 # =============================================================================
@@ -358,8 +397,34 @@ def get_debugger(filename):
     return debugger, filename
 
 
+def count_leading_empty_lines(cell):
+    """Count the number of leading empty cells."""
+    if PY2:
+        lines = cell.splitlines(True)
+    else:
+        lines = cell.splitlines(keepends=True)
+    if not lines:
+        return 0
+    for i, line in enumerate(lines):
+        if line and not line.isspace():
+            return i
+    return len(lines)
+
+
+def transform_cell(code):
+    """Transform ipython code to python code."""
+    tm = TransformerManager()
+    number_empty_lines = count_leading_empty_lines(code)
+    code = tm.transform_cell(code)
+    if PY2:
+        return code
+    return '\n' * number_empty_lines + code
+
+
 def exec_code(code, filename, ns_globals, ns_locals=None):
     """Execute code and display any exception."""
+    global SHOW_INVALID_SYNTAX_MSG
+
     if PY2:
         filename = encode(filename)
         code = encode(code)
@@ -367,14 +432,35 @@ def exec_code(code, filename, ns_globals, ns_locals=None):
     ipython_shell = get_ipython()
     is_ipython = os.path.splitext(filename)[1] == '.ipy'
     try:
-        if is_ipython:
-            # transform code
-            tm = TransformerManager()
-            if not PY2:
-                # Avoid removing lines
-                tm.cleanup_transforms = []
-            code = tm.transform_cell(code)
-        exec(compile(code, filename, 'exec'), ns_globals, ns_locals)
+        if not is_ipython:
+            # TODO: remove the try-except and let the SyntaxError raise
+            # Because there should not be ipython code in a python file
+            try:
+                compiled = compile(code, filename, 'exec')
+            except SyntaxError as e:
+                try:
+                    compiled = compile(transform_cell(code), filename, 'exec')
+                except SyntaxError:
+                    if PY2:
+                        raise e
+                    else:
+                        # Need to call exec to avoid Syntax Error in Python 2.
+                        # TODO: remove exec when dropping Python 2 support.
+                        exec("raise e from None")
+                else:
+                    if SHOW_INVALID_SYNTAX_MSG:
+                        _print(
+                            "\nWARNING: This is not valid Python code. "
+                            "If you want to use IPython magics, "
+                            "flexible indentation, and prompt removal, "
+                            "please save this file with the .ipy extension. "
+                            "This will be an error in a future version of "
+                            "Spyder.\n")
+                        SHOW_INVALID_SYNTAX_MSG = False
+        else:
+            compiled = compile(transform_cell(code), filename, 'exec')
+
+        exec(compiled, ns_globals, ns_locals)
     except SystemExit as status:
         # ignore exit(0)
         if status.code:
diff --git a/spyder/plugins/ipythonconsole/widgets/debugging.py b/spyder/plugins/ipythonconsole/widgets/debugging.py
--- a/spyder/plugins/ipythonconsole/widgets/debugging.py
+++ b/spyder/plugins/ipythonconsole/widgets/debugging.py
@@ -353,6 +353,7 @@ def _handle_input_request(self, msg):
             if self.is_waiting_pdb_input():
                 self._executing = False
                 self._highlighter.highlighting_on = True
+                self.executed.emit(msg)
 
         if self.is_waiting_pdb_input():
             self._pdb_input_ready = True


Test patch:
diff --git a/spyder/app/tests/test_mainwindow.py b/spyder/app/tests/test_mainwindow.py
--- a/spyder/app/tests/test_mainwindow.py
+++ b/spyder/app/tests/test_mainwindow.py
@@ -1634,15 +1634,19 @@ def test_varexp_magic_dbg(main_window, qtbot):
     control = main_window.ipyconsole.get_focus_widget()
     control.setFocus()
 
-    # Create an object that can be plotted
-    with qtbot.waitSignal(shell.executed):
-        shell.execute('li = [1, 2, 3]')
-
     # Click the debug button
     debug_action = main_window.debug_toolbar_actions[0]
     debug_button = main_window.debug_toolbar.widgetForAction(debug_action)
     qtbot.mouseClick(debug_button, Qt.LeftButton)
-    qtbot.wait(1000)
+    qtbot.waitUntil(
+        lambda: shell._control.toPlainText().split()[-1] == 'ipdb>')
+
+    # Get to an object that can be plotted
+    for _ in range(2):
+        qtbot.keyClicks(control, 'n')
+        qtbot.keyClick(control, Qt.Key_Enter)
+        qtbot.waitUntil(
+            lambda: shell._control.toPlainText().split()[-1] == 'ipdb>')
 
     # Generate the plot from the Variable Explorer
     nsb.editor.plot('li', 'plot')
@@ -2992,5 +2996,55 @@ def test_runcell_after_restart(main_window, qtbot):
     # Make sure no errors are shown
     assert "error" not in shell._control.toPlainText().lower()
 
+@pytest.mark.slow
+@flaky(max_runs=1)
+@pytest.mark.parametrize(
+    "ipython", [True, False])
+@pytest.mark.parametrize(
+    "test_cell_magic", [True, False])
+def test_ipython_magic(main_window, qtbot, tmpdir, ipython, test_cell_magic):
+    """Test the runcell command with cell magic."""
+    # Write code with a cell to a file
+    write_file = tmpdir.mkdir("foo").join("bar.txt")
+    assert not osp.exists(to_text_string(write_file))
+    if test_cell_magic:
+        code = "\n\n%%writefile " + to_text_string(write_file) + "\ntest\n"
+    else:
+        code = "\n\n%debug print()"
+    if ipython:
+        fn = "cell-test.ipy"
+    else:
+        fn = "cell-test.py"
+    p = tmpdir.join(fn)
+    p.write(code)
+    main_window.editor.load(to_text_string(p))
+    shell = main_window.ipyconsole.get_current_shellwidget()
+    qtbot.waitUntil(lambda: shell._prompt_html is not None,
+                    timeout=SHELL_TIMEOUT)
+
+    # Execute runcell
+    with qtbot.waitSignal(shell.executed):
+        shell.execute("runcell(0, r'{}')".format(to_text_string(p)))
+    control = main_window.ipyconsole.get_focus_widget()
+
+    error_text = 'save this file with the .ipy extension'
+    try:
+        if ipython:
+            if test_cell_magic:
+                qtbot.waitUntil(
+                    lambda: 'Writing' in control.toPlainText())
+
+                # Verify that the code was executed
+                assert osp.exists(to_text_string(write_file))
+            else:
+                qtbot.waitUntil(lambda: 'ipdb>' in control.toPlainText())
+            assert error_text not in control.toPlainText()
+        else:
+            qtbot.waitUntil(lambda: error_text in control.toPlainText())
+    finally:
+        if osp.exists(to_text_string(write_file)):
+            os.remove(to_text_string(write_file))
+
+
 if __name__ == "__main__":
     pytest.main()
diff --git a/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py b/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
--- a/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
+++ b/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
@@ -1528,9 +1528,9 @@ def test_pdb_multiline(ipyconsole, qtbot):
 
 
 @flaky(max_runs=3)
-@pytest.mark.skipif(not sys.platform.startswith('linux'),
-                    reason="It only works on Linux")
-def test_pdb_ignore_lib(ipyconsole, qtbot):
+@pytest.mark.parametrize(
+    "show_lib", [True, False])
+def test_pdb_ignore_lib(ipyconsole, qtbot, show_lib):
     """Test that pdb can avoid closed files."""
     shell = ipyconsole.get_current_shellwidget()
     qtbot.waitUntil(lambda: shell._prompt_html is not None,
@@ -1541,7 +1541,7 @@ def test_pdb_ignore_lib(ipyconsole, qtbot):
     control.setFocus()
 
     # Tests assume inline backend
-    CONF.set('run', 'pdb_ignore_lib', False)
+    CONF.set('run', 'pdb_ignore_lib', not show_lib)
     with qtbot.waitSignal(shell.executed):
         shell.execute('%debug print()')
         qtbot.waitUntil(lambda: control.toPlainText().split()[-1] == 'ipdb>')
@@ -1554,26 +1554,11 @@ def test_pdb_ignore_lib(ipyconsole, qtbot):
         qtbot.keyClicks(control, 'q')
         qtbot.keyClick(control, Qt.Key_Enter)
 
-    assert 'iostream.py' in control.toPlainText()
-
-    shell.clear_console()
-    qtbot.wait(500)
-
-    # Tests assume inline backend
+    if show_lib:
+        assert 'iostream.py' in control.toPlainText()
+    else:
+        assert 'iostream.py' not in control.toPlainText()
     CONF.set('run', 'pdb_ignore_lib', True)
-    with qtbot.waitSignal(shell.executed):
-        shell.execute('%debug print()')
-        qtbot.waitUntil(lambda: control.toPlainText().split()[-1] == 'ipdb>')
-
-        qtbot.keyClicks(control, 's')
-        qtbot.keyClick(control, Qt.Key_Enter)
-        qtbot.wait(500)
-        qtbot.waitUntil(lambda: control.toPlainText().split()[-1] == 'ipdb>')
-
-        qtbot.keyClicks(control, 'q')
-        qtbot.keyClick(control, Qt.Key_Enter)
-
-    assert 'iostream.py' not in control.toPlainText()
 
 
 @flaky(max_runs=3)