Repository: xonsh/xonsh

Issue:
Parser Error
The following runs fine in xonsh:
`$ bash -c "echo 'hey'; ls"`

As does the following:
`$ bash -c "echo 'hey'" > test.out`

But the following fails:

```
$ bash -c "echo 'hey'; ls" > test.out
................ 
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/xonsh/execer.py", line 135, in _parse_ctx_free
    debug_level=self.debug_level)
  File "/usr/local/lib/python3.4/site-packages/xonsh/parser.py", line 258, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/usr/local/lib/python3.4/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/usr/local/lib/python3.4/site-packages/ply/yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "/usr/local/lib/python3.4/site-packages/xonsh/parser.py", line 2357, in p_error
    column=p.lexpos))
  File "/usr/local/lib/python3.4/site-packages/xonsh/parser.py", line 329, in _parse_error
    raise err
  File "<string>", line None
SyntaxError: <xonsh-code>:1:11: ('code: "echo \'hey\'; ls"',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/xonsh/shell.py", line 149, in push
    locs=self.ctx)
  File "/usr/local/lib/python3.4/site-packages/xonsh/execer.py", line 92, in compile
    tree = self.parse(input, ctx, mode=mode)
  File "/usr/local/lib/python3.4/site-packages/xonsh/execer.py", line 70, in parse
    tree = self._parse_ctx_free(input, mode=mode)
  File "/usr/local/lib/python3.4/site-packages/xonsh/execer.py", line 148, in _parse_ctx_free
    raise original_error
  File "/usr/local/lib/python3.4/site-packages/xonsh/execer.py", line 135, in _parse_ctx_free
    debug_level=self.debug_level)
  File "/usr/local/lib/python3.4/site-packages/xonsh/parser.py", line 258, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/usr/local/lib/python3.4/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/usr/local/lib/python3.4/site-packages/ply/yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "/usr/local/lib/python3.4/site-packages/xonsh/parser.py", line 2357, in p_error
    column=p.lexpos))
  File "/usr/local/lib/python3.4/site-packages/xonsh/parser.py", line 329, in _parse_error
    raise err
  File "<string>", line None
SyntaxError: <xonsh-code>:1:8: ('code: "echo \'hey\'; ls"',)
```

It seems like the combination of `;` and `>` that confuses the parser (even though `;` is inside a string?!)...



Discussion:
This is probably a `subproc_toks()` error since `$[bash -c "echo 'hey'; ls" > test.out]` works.

... which really shouldn't be tokenizing inside of a string :)

> ... which really shouldn't be tokenizing inside of a string :)

Yep, looks like the string is being tokenized properly (that is still a string), but `maxcol` is being set to the position of the semicolon in the middle of the string? 

Oh, just found it.  In execer.py, line 165: `maxcol = line.find(';', last_error_col)`

Reminds me of my bug with clearing out all comments :)


Resolving patch:
diff --git a/xonsh/execer.py b/xonsh/execer.py
--- a/xonsh/execer.py
+++ b/xonsh/execer.py
@@ -123,6 +123,19 @@ def exec(self, input, mode='exec', glbs=None, locs=None, stacklevel=2):
             return None  # handles comment only input
         return exec(code, glbs, locs)
 
+    def _find_next_break(self, line, mincol):
+        if mincol >= 1:
+            line = line[mincol:]
+        if ';' not in line:
+            return None
+        maxcol = None
+        self.parser.lexer.input(line)
+        for tok in self.parser.lexer:
+            if tok.type == 'SEMI':
+                maxcol = tok.lexpos + mincol + 1
+                break
+        return maxcol
+
     def _parse_ctx_free(self, input, mode='exec'):
         last_error_line = last_error_col = -1
         parsed = False
@@ -162,8 +175,7 @@ def _parse_ctx_free(self, input, mode='exec'):
                     last_error_line = last_error_col = -1
                     input = '\n'.join(lines)
                     continue
-                maxcol = line.find(';', last_error_col)
-                maxcol = None if maxcol < 0 else maxcol + 1
+                maxcol = self._find_next_break(line, last_error_col)
                 sbpline = subproc_toks(line,
                                        returnline=True,
                                        maxcol=maxcol,


Test patch: