Repository: ipython/ipython
Pull request: Fixes 11334/11354 Multi line code blocks
This pull request merges the recent input transformer changes from master.  It builds off of the work by @bartskowron in  #11354 to resolve #11334 .  The changes pass tests locally, including the new tests provided by @Carreau; let's see what CI says.

@takluyver [I commented out an assertion](https://github.com/ipython/ipython/blame/498f5f716d102a572cac3571e13426aa002f8454/IPython/core/inputtransformer2.py#L252).  I am not sure what the statement, is this change ok?

Problem statement:
Cannot make multi-line code blocks in ipython
I just did a `pip install ipython` while teaching a class today and typed an if statement and then hit enter after the first line and the code executed.

This seems like a bug.

I did a `pip install ipython==6.5.0` and hitting Enter in a code block properly showed me the next indented line to type out.

I'm on Ubuntu 18.04 running ipython within tmux, though I doubt tmux is the issue here.

Hints from issue discussion:
you can use `ctrl-o` to force a new line. 
I never would have guessed that one. I'm not sure how to make this more clear, but I do think it warrants some clarification/hand-holding. I tried every shortcut I could think of and ended up forcing a downgrade to fix my issue.

I use IPython to make it easier to type out and edit multi-line blocks in the REPL while teaching live audiences. I suspect others in my situation may have similar issues.

Maybe a warning somewhere that indicates that Ctrl-O can be used to make a multi-line code block? I'm not sure whether there's an appropriate place for that.
It is a bug with a recent refactor, I was just trying telling you that you can use Ctrl-O in the meantime if you wish to use 7.x
Ah great! I'm glad it was just a bug. Thanks @Carreau! 😄
Would you mind giving me a few examples (working and not workig) of what you expect, in order to provide test for the fixes ? 

I have some but just collecting a few and don't want to influence you. 
```
with open('hello.txt', mode='wt') as my_file:
    my_file.write('hi')
    my_file.write('hi again')
```

Can't remember the second example, but a number of loops:

```
numbers = [2, 1, 3, 4, 7, 8, 11]

for n in numbers:
    if n > 0:
        print(n*2)
    else:
        print(n/2)
```
@Carreau is someone working on this issue? I'd be happy to help out
hi @Deborah-Digges, I've had a quick look but no too much so far. 

I've wrote this small test case to see the difference between the old input_splitter and the new input_transformer: 

```
from IPython.core import inputtransformer2 as ipt2 # new way
from IPython.core import inputsplitter #oldl way

occ =  inputsplitter.InputSplitter().check_complete
cc = ipt2.TransformerManager().check_complete

comp = lambda x : (cc(x), occ(x), x)
print(comp('if'))
print(comp('if\n\n'))
print(comp("""
def foo():
    print('Hello')"""
))
print(comp('if True:'))
```

Which result in 
```
(('invalid', None), ('invalid', None), 'if')
(('complete', None), ('invalid', None), 'if\n\n')
(('complete', None), ('incomplete', 4), "\ndef foo():\n    print('Hello')")
(('incomplete', 4), ('incomplete', 4), 'if True:')
```

You see that the 3rd item is the one that interest us. The second was out of curiosity as one of the "feature" of IPython is to force-execute if there is  more than 2 newlines. 

Part of the relevant code is in [shortcut.py:L109-L127](https://github.com/ipython/ipython/blob/194d1d7fe2fcc8547205bd7f54fcf63bee9df4e3/IPython/terminal/shortcuts.py#L109-L127). 

I guess there is some kind of conflation between the role of defining whether a check of code is "complete", or whether we should "execute or add a new line".

I'm _guessing_ that the heuristic that check whether the input is multiline and whether the last character is already a new line or not should be sufficient. 

One of the remaining questions is where to put that fix ? 
 - in `shortcut.py`? if so it will only fix terminal IPython
 - in input_transformer2 ? Ithink that might be the right solution as this but will likely also affect QtConsole. 

Let me know if that's enough to at least get you started. 

Thanks !
Hi @Carreau ! Thanks so much for the detailed explanation and test cases.

I was able to reproduce it locally with the dev version of ipython and will now start looking into the code for `shortcut.py` and `inputtransformer`.
> Would you mind giving me a few examples (working and not workig) of what you expect, in order to provide test for the fixes ?

You don't need a few examples. The bug occurs in very reproducible way.
Multiline works **only** if the line ends with colon. If there is no colon at the end of line then iPython executes code. This looks a bit more trivial with this information but never looked into the iPython source code, so maybe I am just a bit too optimistic here ;)

Commit message: Remove a condition from check_complete

Co-Authored-By: Nicholas Bollweg <bollwyvl@users.noreply.github.com>

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index 09e16146476..cf184a9d993 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,6 +34,7 @@ after_success:
 
 matrix:
     include:
+        - { python: "3.6-dev", dist: xenial, sudo: true }
         - { python: "3.7", dist: xenial, sudo: true }
         - { python: "3.7-dev", dist: xenial, sudo: true }
         - { python: "nightly", dist: xenial, sudo: true }
diff --git a/IPython/core/inputtransformer2.py b/IPython/core/inputtransformer2.py
index 0a01100398d..5777559fbd4 100644
--- a/IPython/core/inputtransformer2.py
+++ b/IPython/core/inputtransformer2.py
@@ -234,6 +234,7 @@ def find(cls, tokens_by_line):
         for line in tokens_by_line:
             assign_ix = _find_assign_op(line)
             if (assign_ix is not None) \
+                    and not line[assign_ix].line.strip().startswith('=') \
                     and (len(line) >= assign_ix + 2) \
                     and (line[assign_ix + 1].type == tokenize.ERRORTOKEN):
                 ix = assign_ix + 1
@@ -369,11 +370,15 @@ def transform(self, lines):
         end_line = find_end_of_continued_line(lines, start_line)
         line = assemble_continued_line(lines, (start_line, start_col), end_line)
 
-        if line[:2] in ESCAPE_DOUBLES:
+        if len(line) > 1 and line[:2] in ESCAPE_DOUBLES:
             escape, content = line[:2], line[2:]
         else:
             escape, content = line[:1], line[1:]
-        call = tr[escape](content)
+
+        if escape in tr:
+            call = tr[escape](content)
+        else:
+            call = ''
 
         lines_before = lines[:start_line]
         new_line = indent + call + '\n'
@@ -466,9 +471,12 @@ def make_tokens_by_line(lines):
     except tokenize.TokenError:
         # Input ended in a multiline string or expression. That's OK for us.
         pass
+
+
     if not tokens_by_line[-1]:
         tokens_by_line.pop()
 
+
     return tokens_by_line
 
 def show_linewise_tokens(s: str):
@@ -575,10 +583,28 @@ def check_complete(self, cell: str):
           The number of spaces by which to indent the next line of code. If
           status is not 'incomplete', this is None.
         """
-        if not cell.endswith('\n'):
-            cell += '\n'  # Ensure the cell has a trailing newline
+        # Remember if the lines ends in a new line.
+        ends_with_newline = False
+        for character in reversed(cell):
+            if character == '\n':
+                ends_with_newline = True
+                break
+            elif character.strip():
+                break
+            else:
+                continue
+
+        if ends_with_newline:
+            # Append an newline for consistent tokenization
+            # See https://bugs.python.org/issue33899
+            cell += '\n'
+
         lines = cell.splitlines(keepends=True)
-        if lines[-1][:-1].endswith('\\'):
+
+        if not lines:
+            return 'complete', None
+
+        if lines[-1].endswith('\\'):
             # Explicit backslash continuation
             return 'incomplete', find_last_indent(lines)
 
@@ -603,35 +629,41 @@ def check_complete(self, cell: str):
             return 'invalid', None
 
         tokens_by_line = make_tokens_by_line(lines)
+
         if not tokens_by_line:
             return 'incomplete', find_last_indent(lines)
+
         if tokens_by_line[-1][-1].type != tokenize.ENDMARKER:
             # We're in a multiline string or expression
             return 'incomplete', find_last_indent(lines)
-        if len(tokens_by_line) == 1:
-            return 'incomplete', find_last_indent(lines)
-        # Find the last token on the previous line that's not NEWLINE or COMMENT
-        toks_last_line = tokens_by_line[-2]
-        ix = len(toks_last_line) - 1
-        while ix >= 0 and toks_last_line[ix].type in {tokenize.NEWLINE,
-                                                      tokenize.COMMENT}:
-            ix -= 1
-
-        if toks_last_line[ix].string == ':':
+
+        newline_types = {tokenize.NEWLINE, tokenize.COMMENT, tokenize.ENDMARKER}
+
+        # Remove newline_types for the list of tokens
+        while len(tokens_by_line) > 1 and len(tokens_by_line[-1]) == 1 \
+                and tokens_by_line[-1][-1].type in newline_types:
+            tokens_by_line.pop()
+
+        while tokens_by_line[-1] and tokens_by_line[-1][-1].type in newline_types:
+            tokens_by_line[-1].pop()
+
+        if len(tokens_by_line) == 1 and not tokens_by_line[-1]:
+            return 'incomplete', 0
+
+        if tokens_by_line[-1][-1].string == ':':
             # The last line starts a block (e.g. 'if foo:')
             ix = 0
-            while toks_last_line[ix].type in {tokenize.INDENT, tokenize.DEDENT}:
+            while tokens_by_line[-1][ix].type in {tokenize.INDENT, tokenize.DEDENT}:
                 ix += 1
-            indent = toks_last_line[ix].start[1]
+
+            indent = tokens_by_line[-1][ix].start[1]
             return 'incomplete', indent + 4
 
-        # If there's a blank line at the end, assume we're ready to execute.
-        if not lines[-1].strip():
-            return 'complete', None
+        if tokens_by_line[-1][0].line.endswith('\\'):
+            return 'incomplete', None
 
         # At this point, our checks think the code is complete (or invalid).
-        # We'll use codeop.compile_command to check this with the real parser.
-
+        # We'll use codeop.compile_command to check this with the real parser
         try:
             with warnings.catch_warnings():
                 warnings.simplefilter('error', SyntaxWarning)
@@ -642,6 +674,16 @@ def check_complete(self, cell: str):
         else:
             if res is None:
                 return 'incomplete', find_last_indent(lines)
+
+        if tokens_by_line[-1][-1].type == tokenize.DEDENT:
+            if ends_with_newline:
+                return 'complete', None
+            return 'incomplete', find_last_indent(lines)
+
+        # If there's a blank line at the end, assume we're ready to execute
+        if not lines[-1].strip():
+            return 'complete', None
+
         return 'complete', None
 
 
diff --git a/IPython/core/tests/test_inputtransformer2.py b/IPython/core/tests/test_inputtransformer2.py
index 77df22c2c22..9c5a413955f 100644
--- a/IPython/core/tests/test_inputtransformer2.py
+++ b/IPython/core/tests/test_inputtransformer2.py
@@ -10,6 +10,8 @@
 from IPython.core import inputtransformer2 as ipt2
 from IPython.core.inputtransformer2 import make_tokens_by_line
 
+from textwrap import dedent
+
 MULTILINE_MAGIC = ("""\
 a = f()
 %foo \\
@@ -212,6 +214,17 @@ def test_check_complete():
     nt.assert_equal(cc("a = '''\n   hi"), ('incomplete', 3))
     nt.assert_equal(cc("def a():\n x=1\n global x"), ('invalid', None))
     nt.assert_equal(cc("a \\ "), ('invalid', None))  # Nothing allowed after backslash
+    nt.assert_equal(cc("1\\\n+2"), ('complete', None))
+    nt.assert_equal(cc("1\\\n+2"), ('complete', None))
+    nt.assert_equal(cc("exit"), ('complete', None))
+
+    example = dedent("""
+        if True:
+            a=1""" )
+
+    nt.assert_equal(cc(example), ('incomplete', 4))
+    nt.assert_equal(cc(example+'\n'), ('complete', None))
+    nt.assert_equal(cc(example+'\n    '), ('complete', None))
 
     # no need to loop on all the letters/numbers.
     short = '12abAB'+string.printable[62:]


Code review comments:
diff_hunk: @@ -212,6 +214,17 @@ def test_check_complete():
     nt.assert_equal(cc("a = '''\n   hi"), ('incomplete', 3))
     nt.assert_equal(cc("def a():\n x=1\n global x"), ('invalid', None))
     nt.assert_equal(cc("a \\ "), ('invalid', None))  # Nothing allowed after backslash
+    nt.assert_equal(cc("1\\\n+2"), ('complete', None))
+    nt.assert_equal(cc("1\\\n+2"), ('complete', None))
path: IPython/core/tests/test_inputtransformer2.py
text: @user1:
redundant test