Repository: spyder-ide/spyder

Issue:
Indentation handling when running code cells (IndentationError: unexpected indent)
### Issue Report Checklist

* [x] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [x] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [ ] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [ ] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [ ] Tried basic troubleshooting (if a bug/error)
    * [x] Restarted Spyder
    * [ ] Reset preferences with ``spyder --reset``
    * [ ] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [ ] Tried the other applicable steps from the Troubleshooting Guide
* [x] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description

A cell with indented code cannot be run as it could before.  In this example the second cell cannot be run:
```
print ('hello')

if 1:
    print('hello again')

#%%
    print ('hello')
```

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->


    Spyder version: 4.1.2
    Python version: 3.7.2 64b
    Qt version: 5.9.6
    PyQt version: 5.9.2
    Operating System name/version: Win7 64b


### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
# Mandatory:
atomicwrites >=1.2.0           :  1.2.1 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  0.6.1 (OK)
diff_match_patch >=20181111    :  20181111 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.2.0 (OK)
jedi =0.15.2                   :  0.15.2 (OK)
nbconvert >=4.0                :  5.3.1 (OK)
numpydoc >=0.6.0               :  0.8.0 (OK)
paramiko >=2.4.0               :  2.4.2 (OK)
parso =0.5.2                   :  0.5.2 (OK)
pexpect >=4.4.0                :  4.7.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.4.8 (OK)
pygments >=2.0                 :  2.3.1 (OK)
pylint >=0.25                  :  2.2.2 (OK)
pyls >=0.31.9;<0.32.0          :  0.31.9 (OK)
qdarkstyle >=2.8               :  2.8 (OK)
qtawesome >=0.5.7              :  0.5.7 (OK)
qtconsole >=4.6.0              :  4.6.0 (OK)
qtpy >=1.5.0                   :  1.5.2 (OK)
rtree >=0.8.3                  :  0.8.3 (OK)
sphinx >=0.6.6                 :  1.8.2 (OK)
spyder_kernels >=1.9.0;<1.10.0 :  1.9.0 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  17.1.2 (OK)

# Optional:
cython >=0.21                  :  0.29.2 (OK)
matplotlib >=2.0.0             :  3.0.2 (OK)
numpy >=1.7                    :  1.15.4 (OK)
pandas >=0.13.1                :  0.24.2 (OK)
scipy >=0.17.0                 :  1.2.1 (OK)
sympy >=0.7.3                  :  1.3 (OK)

```



Discussion:
Hi @adriesse, it seems that the snippet posted on the issue has wrong identation. That should be fixed by correcting the identation of the offending lines.
To run a cell, normally only the indentation within that cell must be correct--at least that's the way it used to work.  Anyway, I've updated the snippet so that it is correct for the whole file as well.  The second cell still does not run.
This is a duplicate of #11023, which should be fixed in spyder-kernels.

I can't reproduce the error here using the same spyder-kernels version as @adriesse 
I apologize for not finding that other issue earlier, but the way I read now it is that it was fixed for .ipy files but not for .py files and nevertheless closed.  I'm afraid I don't quite understand this.
It should work on py and ipy files. I think we are only going to allow ipython magic commands in ipy eventually but code cells will still work in py files.

@impact27 Can you reproduce this one?
With the ipython PR I get:

```
In [1]: runcell(1, '/Users/quentinpeter/Desktop/untitled2.py')

WARNING: This is not valid Python code. If you want to use IPython magics, flexible indentation, and prompt removal, please save this file with the .ipy extension. This will be an error in a future version of Spyder.

hello

In [2]: 
```
I can't reproduce it either. @dalthviz, can you reproduce this on Windows?
@impact27 I get that too.  Off topic but I think that warning is coming up in error. We should check if the `%` is in a comment.  
> I think that warning is coming up in error

Why? Random indentation is not correct python code. Executing:
```
#%%
    print ('hello')
```
Shouldn't work, unless this is a ipython file.
I observe the same behaviour, whether the filename ends in '.py' or '.ipy'.
> Why? Random indentation is not correct python code

It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the `if __name__ = "__main__"` section.
> > Why? Random indentation is not correct python code
> 
> It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the `if __name__ = "__main__"` section.

Fair, then we can use `leading_indent` from `IPython.core.inputtransformer2` (PY3 only) when running cells and the full `TransformerManager` for ipython files. 
I thought this was an editor/ide convenience feature, not part of the Python language.
Have you all decided which behavior Spyder will provide going forward?
Hi @adriesse I think some ideas were indeed proposed by @impact27 what do you think @ccordoba12 ? 

Also, checking this I wasn't able to reproduce the error (at least using Spyder 4.1.3) using `Ctrl + Enter` to run the cells.

![image](https://user-images.githubusercontent.com/16781833/83534927-0fd7e900-a4b7-11ea-99cd-30af9d1397b1.png)

This is what I'm getting with the `Copy full cell contents to the console when running code cell` option enabled (this option is available in the `Preferences dialog > Editor > Run code`):

![image](https://user-images.githubusercontent.com/16781833/83533181-d900d380-a4b4-11ea-8d64-3046a08c71ff.png)

And this is what I'm getting with the option disabled:

![image](https://user-images.githubusercontent.com/16781833/83532945-9939ec00-a4b4-11ea-8e1f-14d40714da00.png)

If you keep experiencing the issue (the raise of an `IndentationError`) could you try to:
* Update to Spyder 4.1.3
* Record a GIF for a better understanding of the way you are running the cells and how an `IndentationError` is being raised

Any new input (info about the error you experienced and/or perspective on how things should be working) is greatly appreciated, thanks :)
> I think some ideas were indeed proposed by @impact27 what do you think @ccordoba12 ?

Let's wait to see what @adriesse says because I can't reproduce this bug either on Linux.
I'm still unclear on what your intended behavior is, and/or whether there was a conscious decision recently to change the intended behavior, so I don't whether it is a bug.  Anyway, here are some screen shots:

![image](https://user-images.githubusercontent.com/9001027/83560990-5ceba780-a517-11ea-9e28-7898156d0a0c.png)

With "copy full cell contents":

![image](https://user-images.githubusercontent.com/9001027/83561084-8a385580-a517-11ea-9781-a53dad40b900.png)

Without "copy full cell contents":

![image](https://user-images.githubusercontent.com/9001027/83561218-c2d82f00-a517-11ea-9424-223b5369edaa.png)


Thanks for the screenshots @adriesse I was able to reproduce this!:

![image](https://user-images.githubusercontent.com/16781833/83562896-f4350880-a4df-11ea-9119-a63806cdfd11.png)

Please update `ipython` (that should enable you to just see a warning instead of the `IndentationError`):

Thanks!  The upgrade changes the error to a warning, and now if I rename the file to `.ipy` the warning goes away.

How will it be in the future?  

As @bcolsen observed above:
> It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the if __name__ = "__main__" section.

What is the motivation for disallowing the execution of indented code blocks?  Has it caused problems?
I think that indentation is a fundamental part of the syntax of Python and taking that into account basically mixed indentation levels are not the correct way to go (the code will not be accepted for the Python interpreter). As an example the following isn't correctly formatted Python code:

![indentation](https://user-images.githubusercontent.com/16781833/83655548-2e57e600-a584-11ea-8066-1420ab3e5d29.png)

Even though currently you can run this inside Spyder (and a warning mentions what I'm saying):

![image](https://user-images.githubusercontent.com/16781833/83655838-74ad4500-a584-11ea-8131-2d381819d0ce.png)

Using `.ipy` and IPython in general provides a layer that can handle, for example, the mixed indentation since IPython has a transformation logic that not only provides the way to handle the semantic of magics but also can handle things like flexible indentation. However, at the end, the transformation logic that IPython uses translates your code into properly formatted Python (for example removing the leading indentation that isn't supported for the interpreter).

Maybe I'm missing some stuff, and please guys correct me if I'm messing up some where in the explanation, but that's why we not encourage using indented blocks if running from `.py` files :)
@dalthviz The only indentation flexibility that ipython has is if code has an indentation in the first line  that indentation is removed from all the other lines. This also isn't corrected in .ipy files because they don't support executing codecells we are tagging that on ourselves.

As @impact27 mentions this, is the solution:

> > > Why? Random indentation is not correct python code
> > 
> > 
> > It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the `if __name__ = "__main__"` section.
> 
> Fair, then we can use `leading_indent` from `IPython.core.inputtransformer2` (PY3 only) when running cells and the full `TransformerManager` for ipython files.

We could even implement it ourselves:
https://github.com/ipython/ipython/blob/0bc26f65bc775082dc36d89edc1657be031b9e79/IPython/core/inputtransformer2.py#L34

I think that allowing codecells with arbitrary initial indent is fine. I don't think we need to be that strict about it. It will still give plenty of indentation errors:

```python
In [4]:     print("hello")
   ...: print("hello")
hello
hello

In [5]:     print("hello")
   ...: print("hello")
   ...:         print("hello")
  File "<ipython-input-5-ed6671ba29f1>", line 3
    print("hello")
    ^
IndentationError: unexpected indent


In [6]:     print("hello")
   ...:         print("hello")
   ...:         print("hello")
  File "<ipython-input-6-e529d318b1ea>", line 2
    print("hello")
    ^
IndentationError: unexpected indent


In [7]: print("hello")
   ...:     print("hello")
   ...:     print("hello")
  File "<ipython-input-7-e529d318b1ea>", line 2
    print("hello")
    ^
IndentationError: unexpected indent


In [8]:     print("hello")
   ...:     print("hello")
   ...:     print("hello")
hello
hello
hello

In [9]:         print("hello")
   ...:     print("hello")
   ...:     print("hello")
  File "<ipython-input-9-e529d318b1ea>", line 2
    print("hello")
    ^
IndentationError: unexpected indent
```

> I think that allowing codecells with arbitrary initial indent is fine.

I agree.

> We could even implement it ourselves

@bcolsen, could you create a PR for this?
On a related note, I have the impression that Spyder is now saving my file (all changed files?) whenever I run a cell, whereas previously it only did so when running the whole file.  As with the previously discussed behavior, it is unclear whether this is by design intent or a side-effect of some other change.  I can see the logic in saving other changed files in case there are dependencies, but saving the current file doesn't seem necessary or desirable--at least to me.  I have now unchecked the editor/run_code/save_files option and I'll how that goes.
> On a related note, I have the impression that Spyder is now saving my file (all changed files?) whenever I run a cell, whereas previously it only did so when running the whole file. As with the previously discussed behavior, it is unclear whether this is by design intent or a side-effect of some other change. I can see the logic in saving other changed files in case there are dependencies, but saving the current file doesn't seem necessary or desirable--at least to me. I have now unchecked the editor/run_code/save_files option and I'll how that goes.

I don't think it should do that. Could you open a new issue for this behaviour?

Fixed by https://github.com/spyder-ide/spyder-kernels/pull/243

Resolving patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
@@ -12,28 +12,31 @@
 #
 
 import bdb
+import cmd
+from distutils.version import LooseVersion
 import io
+import logging
 import os
 import pdb
 import shlex
 import sys
 import time
 import warnings
-import logging
-import cmd
 
+from IPython import __version__ as ipy_version
 from IPython.core.getipython import get_ipython
 
-from spyder_kernels.py3compat import TimeoutError, PY2, _print, encode
 from spyder_kernels.comms.frontendcomm import CommError, frontend_request
 from spyder_kernels.customize.namespace_manager import NamespaceManager
 from spyder_kernels.customize.spyderpdb import SpyderPdb
 from spyder_kernels.customize.umr import UserModuleReloader
+from spyder_kernels.py3compat import TimeoutError, PY2, _print, encode
 
-if not PY2:
-    from IPython.core.inputtransformer2 import TransformerManager
+if LooseVersion(ipy_version) > LooseVersion('7.0.0'):
+    from IPython.core.inputtransformer2 import (
+        TransformerManager, leading_indent, leading_empty_lines)
 else:
-    from IPython.core.inputsplitter import IPythonInputSplitter as TransformerManager
+    from IPython.core.inputsplitter import IPythonInputSplitter
 
 
 logger = logging.getLogger(__name__)
@@ -398,13 +401,25 @@ def count_leading_empty_lines(cell):
     return len(lines)
 
 
-def transform_cell(code):
+def transform_cell(code, indent_only=False):
     """Transform ipython code to python code."""
-    tm = TransformerManager()
     number_empty_lines = count_leading_empty_lines(code)
-    code = tm.transform_cell(code)
-    if PY2:
-        return code
+    if indent_only:
+        # Not implemented for PY2
+        if LooseVersion(ipy_version) < LooseVersion('7.0.0'):
+            return code
+        if not code.endswith('\n'):
+            code += '\n'  # Ensure the cell has a trailing newline
+        lines = code.splitlines(keepends=True)
+        lines = leading_indent(leading_empty_lines(lines))
+        code = ''.join(lines)
+    else:
+        if LooseVersion(ipy_version) < LooseVersion('7.0.0'):
+            tm = IPythonInputSplitter()
+            return tm.transform_cell(code)
+        else:
+            tm = TransformerManager()
+            code = tm.transform_cell(code)
     return '\n' * number_empty_lines + code
 
 
@@ -425,7 +440,8 @@ def exec_code(code, filename, ns_globals, ns_locals=None, post_mortem=False):
             # TODO: remove the try-except and let the SyntaxError raise
             # Because there should not be ipython code in a python file
             try:
-                compiled = compile(code, filename, 'exec')
+                compiled = compile(
+                    transform_cell(code, indent_only=True), filename, 'exec')
             except SyntaxError as e:
                 try:
                     compiled = compile(transform_cell(code), filename, 'exec')
@@ -442,9 +458,8 @@ def exec_code(code, filename, ns_globals, ns_locals=None, post_mortem=False):
                             "\nWARNING: This is not valid Python code. "
                             "If you want to use IPython magics, "
                             "flexible indentation, and prompt removal, "
-                            "please save this file with the .ipy extension. "
-                            "This will be an error in a future version of "
-                            "Spyder.\n")
+                            "we recommend that you save this file with the "
+                            ".ipy extension.\n")
                         SHOW_INVALID_SYNTAX_MSG = False
         else:
             compiled = compile(transform_cell(code), filename, 'exec')


Test patch:
diff --git a/spyder/app/tests/test_mainwindow.py b/spyder/app/tests/test_mainwindow.py
--- a/spyder/app/tests/test_mainwindow.py
+++ b/spyder/app/tests/test_mainwindow.py
@@ -2677,6 +2677,28 @@ def test_runcell(main_window, qtbot, tmpdir, debug):
         pass
 
 
+@pytest.mark.slow
+@flaky(max_runs=3)
+def test_runcell_leading_indent(main_window, qtbot, tmpdir):
+    """Test the runcell command with leading indent."""
+    # Write code with a cell to a file
+    code = ("def a():\n    return\nif __name__ == '__main__':\n"
+            "# %%\n    print(1233 + 1)\n")
+    p = tmpdir.join("cell-test.py")
+    p.write(code)
+    main_window.editor.load(to_text_string(p))
+    shell = main_window.ipyconsole.get_current_shellwidget()
+    qtbot.waitUntil(lambda: shell._prompt_html is not None,
+                    timeout=SHELL_TIMEOUT)
+
+    # Execute runcell
+    with qtbot.waitSignal(shell.executed):
+        shell.execute("runcell(1, r'{}')".format(to_text_string(p)))
+
+    assert "1234" in shell._control.toPlainText()
+    assert "This is not valid Python code" not in shell._control.toPlainText()
+
+
 @pytest.mark.slow
 @flaky(max_runs=3)
 def test_varexp_rename(main_window, qtbot, tmpdir):