Repository: ipython/ipython

Issue:
Backslash (\) at the end of the line behavior different from default Python
Quite often, I miss the Enter/Return key and press `\`.  In regular Python, I can enter two newlines, and the expression will be evaluated, like

``` py
>>> 1\
... 
1
```

But in IPython, no amount of newlines will terminate it.  And if I do finally terminate it by entering a `;`, the output is not printed (also different from Python).

``` py

In [1]: 1\
   ...: 
   ...: 
   ...: 
   ...: 
   ...: ;

In [2]: 
```

``` py

>>> 1\
... ;
1
```

Is this an intended feature caused by some more advanced multiline editing capabilities of IPython, or just a bug?  

Backslash (\) at the end of the line behavior different from default Python
Quite often, I miss the Enter/Return key and press `\`.  In regular Python, I can enter two newlines, and the expression will be evaluated, like

``` py
>>> 1\
... 
1
```

But in IPython, no amount of newlines will terminate it.  And if I do finally terminate it by entering a `;`, the output is not printed (also different from Python).

``` py

In [1]: 1\
   ...: 
   ...: 
   ...: 
   ...: 
   ...: ;

In [2]: 
```

``` py

>>> 1\
... ;
1
```

Is this an intended feature caused by some more advanced multiline editing capabilities of IPython, or just a bug?  



Discussion:
It's just a bug in how our inputsplitter handles backslash-terminated code.  I'm afraid it won't be an easy one to fix, as that code is rather subtle, so don't hold your breath.  But it's a valid bug, marking as such.

I found the source of the error.  The input splitter is checking for `source.rstrip().endswith('\\')` instead of just `source.endswith('\\')`.  Note that this also causes it to treat `1 \` with an extra space at the end the same as just `1\`, which also differs from regular Python (it treats that as a SyntaxError). 

So my question now is, how well is that code tested?  Can I trust that the fix is OK if tests pass?

The tests for that code are fairly robust, actually.  Though obviously, if you've found a problem here, I would like any PR fixing it to also include a test that fails with the current code, so that we ensure it doesn't return afterwards.

OK, I'll send one, as soon as I figure out how to add a test for this (and also as soon as I figure out how to actually run the tests :-)

By the way, as I go through the code, I'm also noticing random other things that need fixing.  Should I submit these as separate PRs or are you guys OK with one grab-bag PR?

Sorry, it should actually be `endswith('\\\n')`. I didn't realize that the source included the newline (which explains why the rstrip() was there in the first place).

On Sat, Jul 7, 2012 at 5:59 PM, Aaron Meurer
reply@reply.github.com
wrote:

> OK, I'll send one, as soon as I figure out how to add a test for this (and also as soon as I figure out how to actually run the tests :-)

In short, use `iptest`, ideally with ipython setup with `setup.py
develop` or similar so you're running from your source tree.  Details:

ipython.org/ipython-doc/rel-0.13/development/testing.html

> By the way, as I go through the code, I'm also noticing random other things that need fixing.  Should I submit these as separate PRs or are you guys OK with one grab-bag PR?

If they are really tiny, go ahead and lump them in, but anything
reasonably complex should ideally have its own PR.  We're pretty good
about having quick turnaround for easy things.

Bonus points if you run

cd tools
./test_pr -p NNNN

with your PR number, which will run the test suite and post the
results; this is our implementation of the ideas you gave us from your
sympy workflow to have easy testing of a PR.

Having the test_pr results up and ready to see will make the review
process even quicker.  You can omit the -p if you want to see the
local results and post them separately later on (I typically do that).

Thanks. I figured that out just as you posted it. 

One last question: what's the way to make GitHub automatically close an issue? I think it's something like `closes #2108` at the bottom of the commit message, but I always forget the exact format and it doesn't always work.

That works, 'fixes' also works, and you can use either `#` or `gh-` as the prefix.  

Still, it's good practice to add which issues are fixed in the description of the PR, so that the summary messages (which are the ones that really get read when scanning development history) have direct links to the relevant bugs.

OK, a question.  What's the correct way to test that `1 \` (with a space at the end) produces a SyntaxError?

It's just a bug in how our inputsplitter handles backslash-terminated code.  I'm afraid it won't be an easy one to fix, as that code is rather subtle, so don't hold your breath.  But it's a valid bug, marking as such.

I found the source of the error.  The input splitter is checking for `source.rstrip().endswith('\\')` instead of just `source.endswith('\\')`.  Note that this also causes it to treat `1 \` with an extra space at the end the same as just `1\`, which also differs from regular Python (it treats that as a SyntaxError). 

So my question now is, how well is that code tested?  Can I trust that the fix is OK if tests pass?

The tests for that code are fairly robust, actually.  Though obviously, if you've found a problem here, I would like any PR fixing it to also include a test that fails with the current code, so that we ensure it doesn't return afterwards.

OK, I'll send one, as soon as I figure out how to add a test for this (and also as soon as I figure out how to actually run the tests :-)

By the way, as I go through the code, I'm also noticing random other things that need fixing.  Should I submit these as separate PRs or are you guys OK with one grab-bag PR?

Sorry, it should actually be `endswith('\\\n')`. I didn't realize that the source included the newline (which explains why the rstrip() was there in the first place).

On Sat, Jul 7, 2012 at 5:59 PM, Aaron Meurer
reply@reply.github.com
wrote:

> OK, I'll send one, as soon as I figure out how to add a test for this (and also as soon as I figure out how to actually run the tests :-)

In short, use `iptest`, ideally with ipython setup with `setup.py
develop` or similar so you're running from your source tree.  Details:

ipython.org/ipython-doc/rel-0.13/development/testing.html

> By the way, as I go through the code, I'm also noticing random other things that need fixing.  Should I submit these as separate PRs or are you guys OK with one grab-bag PR?

If they are really tiny, go ahead and lump them in, but anything
reasonably complex should ideally have its own PR.  We're pretty good
about having quick turnaround for easy things.

Bonus points if you run

cd tools
./test_pr -p NNNN

with your PR number, which will run the test suite and post the
results; this is our implementation of the ideas you gave us from your
sympy workflow to have easy testing of a PR.

Having the test_pr results up and ready to see will make the review
process even quicker.  You can omit the -p if you want to see the
local results and post them separately later on (I typically do that).

Thanks. I figured that out just as you posted it. 

One last question: what's the way to make GitHub automatically close an issue? I think it's something like `closes #2108` at the bottom of the commit message, but I always forget the exact format and it doesn't always work.

That works, 'fixes' also works, and you can use either `#` or `gh-` as the prefix.  

Still, it's good practice to add which issues are fixed in the description of the PR, so that the summary messages (which are the ones that really get read when scanning development history) have direct links to the relevant bugs.

OK, a question.  What's the correct way to test that `1 \` (with a space at the end) produces a SyntaxError?


Resolving patch:
diff --git a/IPython/core/inputsplitter.py b/IPython/core/inputsplitter.py
--- a/IPython/core/inputsplitter.py
+++ b/IPython/core/inputsplitter.py
@@ -115,7 +115,9 @@
     r'^\s+raise\([^\)]*\).*$', # wacky raise with immediate open paren
     r'^\s+return(\s.*)?$', # normal return (+ space + other stuff, maybe)
     r'^\s+return\([^\)]*\).*$', # wacky return with immediate open paren
-    r'^\s+pass\s*$' # pass (optionally followed by trailing spaces)
+    r'^\s+pass\s*$', # pass (optionally followed by trailing spaces)
+    r'^\s+break\s*$', # break (optionally followed by trailing spaces)
+    r'^\s+continue\s*$', # continue (optionally followed by trailing spaces)
 ]))
 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
 
@@ -202,17 +204,17 @@ def remove_comments(src):
     """
 
     return re.sub('#.*', '', src)
-    
+
 def has_comment(src):
     """Indicate whether an input line has (i.e. ends in, or is) a comment.
-    
+
     This uses tokenize, so it can distinguish comments from # inside strings.
-    
+
     Parameters
     ----------
     src : string
       A single line input string.
-    
+
     Returns
     -------
     Boolean: True if source has a comment.
@@ -280,9 +282,9 @@ class InputSplitter(object):
     code = None
     # Input mode
     input_mode = 'line'
-    
+
     # Private attributes
-    
+
     # List with lines of input accumulated so far
     _buffer = None
     # Command compiler
@@ -291,7 +293,7 @@ class InputSplitter(object):
     _full_dedent = False
     # Boolean indicating whether the current block is complete
     _is_complete = None
-    
+
     def __init__(self, input_mode=None):
         """Create a new InputSplitter instance.
 
@@ -348,7 +350,7 @@ def push(self, lines):
         ----------
         lines : string
           One or more lines of Python input.
-        
+
         Returns
         -------
         is_complete : boolean
@@ -359,7 +361,7 @@ def push(self, lines):
         """
         if self.input_mode == 'cell':
             self.reset()
-        
+
         self._store(lines)
         source = self.source
 
@@ -369,7 +371,7 @@ def push(self, lines):
         self.code, self._is_complete = None, None
 
         # Honor termination lines properly
-        if source.rstrip().endswith('\\'):
+        if source.endswith('\\\n'):
             return False
 
         self._update_indent(lines)
@@ -400,11 +402,11 @@ def push_accepts_more(self):
         SyntaxError is raised, or *all* of the following are true:
 
         1. The input compiles to a complete statement.
-        
+
         2. The indentation level is flush-left (because if we are indented,
            like inside a function definition or for loop, we need to keep
            reading new input).
-          
+
         3. There is one extra line consisting only of whitespace.
 
         Because of condition #3, this method should be used only by
@@ -464,7 +466,7 @@ def _find_indent(self, line):
         ----------
         line : str
           A single new line of non-whitespace, non-comment Python input.
-          
+
         Returns
         -------
         indent_spaces : int
@@ -476,7 +478,7 @@ def _find_indent(self, line):
         """
         indent_spaces = self.indent_spaces
         full_dedent = self._full_dedent
-        
+
         inisp = num_ini_spaces(line)
         if inisp < indent_spaces:
             indent_spaces = inisp
@@ -495,9 +497,9 @@ def _find_indent(self, line):
         if indent_spaces < 0:
             indent_spaces = 0
             #print 'safety' # dbg
-            
+
         return indent_spaces, full_dedent
-    
+
     def _update_indent(self, lines):
         for line in remove_comments(lines).splitlines():
             if line and not line.isspace():
@@ -508,10 +510,10 @@ def _store(self, lines, buffer=None, store='source'):
 
         If input lines are not newline-terminated, a newline is automatically
         appended."""
-        
+
         if buffer is None:
             buffer = self._buffer
-            
+
         if lines.endswith('\n'):
             buffer.append(lines)
         else:
@@ -627,10 +629,10 @@ def transform_help_end(line):
     target = m.group(1)
     esc = m.group(3)
     lspace = _initial_space_re.match(line).group(0)
-    
+
     # If we're mid-command, put it back on the next prompt for the user.
     next_input = line.rstrip('?') if line.strip() != m.group(0) else None
-        
+
     return _make_help_call(target, esc, lspace, next_input)
 
 
@@ -647,7 +649,7 @@ def __init__(self):
                ESC_QUOTE2 : self._tr_quote2,
                ESC_PAREN  : self._tr_paren }
         self.tr = tr
-        
+
     # Support for syntax transformations that use explicit escapes typed by the
     # user at the beginning of a line
     @staticmethod
@@ -668,7 +670,7 @@ def _tr_help(line_info):
         # A naked help line should just fire the intro help screen
         if not line_info.line[1:]:
             return 'get_ipython().show_usage()'
-        
+
         return _make_help_call(line_info.ifun, line_info.esc, line_info.pre)
 
     @staticmethod
@@ -745,7 +747,7 @@ def __init__(self, input_mode=None):
         super(IPythonInputSplitter, self).__init__(input_mode)
         self._buffer_raw = []
         self._validate = True
-        
+
     def reset(self):
         """Reset the input buffer and associated state."""
         super(IPythonInputSplitter, self).reset()
@@ -897,7 +899,7 @@ def push(self, lines):
         # that this must be done *after* the reset() call that would otherwise
         # flush the buffer.
         self._store(lines, self._buffer_raw, 'source_raw')
-        
+
         try:
             push = super(IPythonInputSplitter, self).push
             buf = self._buffer


Test patch:
diff --git a/IPython/core/tests/test_inputsplitter.py b/IPython/core/tests/test_inputsplitter.py
--- a/IPython/core/tests/test_inputsplitter.py
+++ b/IPython/core/tests/test_inputsplitter.py
@@ -227,6 +227,26 @@ def test_dedent_pass(self):
         isp.push('if 1:\n     pass   ')
         self.assertEqual(isp.indent_spaces, 0)
 
+    def test_dedent_break(self):
+        isp = self.isp # shorthand
+        # should NOT cause dedent
+        isp.push('while 1:\n    breaks = 5')
+        self.assertEqual(isp.indent_spaces, 4)
+        isp.push('while 1:\n     break')
+        self.assertEqual(isp.indent_spaces, 0)
+        isp.push('while 1:\n     break   ')
+        self.assertEqual(isp.indent_spaces, 0)
+
+    def test_dedent_continue(self):
+        isp = self.isp # shorthand
+        # should NOT cause dedent
+        isp.push('while 1:\n    continues = 5')
+        self.assertEqual(isp.indent_spaces, 4)
+        isp.push('while 1:\n     continue')
+        self.assertEqual(isp.indent_spaces, 0)
+        isp.push('while 1:\n     continue   ')
+        self.assertEqual(isp.indent_spaces, 0)
+
     def test_dedent_raise(self):
         isp = self.isp # shorthand
         # should NOT cause dedent
@@ -351,6 +371,22 @@ def test_unicode(self):
         self.isp.push(u'\xc3\xa9')
         self.isp.push(u"u'\xc3\xa9'")
 
+    def test_line_continuation(self):
+        """ Test issue #2108."""
+        isp = self.isp
+        # A blank line after a line continuation should not accept more
+        isp.push("1 \\\n\n")
+        self.assertFalse(isp.push_accepts_more())
+        # Whitespace after a \ is a SyntaxError.  The only way to test that
+        # here is to test that push doesn't accept more (as with
+        # test_syntax_error() above).
+        isp.push(r"1 \ ")
+        self.assertFalse(isp.push_accepts_more())
+        # Even if the line is continuable (c.f. the regular Python
+        # interpreter)
+        isp.push(r"(1 \ ")
+        self.assertFalse(isp.push_accepts_more())
+
 class InteractiveLoopTestCase(unittest.TestCase):
     """Tests for an interactive loop like a python shell.
     """
@@ -678,7 +714,7 @@ def test_syntax_multiline(self):
     def test_syntax_multiline_cell(self):
         isp = self.isp
         for example in syntax_ml.itervalues():
-            
+
             out_t_parts = []
             for line_pairs in example:
                 raw = '\n'.join(r for r, _ in line_pairs)
@@ -762,7 +798,7 @@ def test_last_two_blanks():
 
 
 class CellMagicsCommon(object):
-    
+
     def test_whole_cell(self):
         src = "%%cellm line\nbody\n"
         sp = self.sp