Repository: xonsh/xonsh

Issue:
Python evaluation broken in current HEAD?
See below. In current HEAD, the command simply doesn't execute. (The prompt-toolkit warning in the wheel is because of version 1.0.0.)

```
🐟 23:09 her@air ~/Code pip3 install xonsh
Collecting xonsh
Requirement already satisfied (use --upgrade to upgrade): ply in /usr/local/lib/python3.5/site-packages (from xonsh)
Installing collected packages: xonsh
Successfully installed xonsh-0.2.7
🐟 23:09 her@air ~/Code xonsh
/usr/local/lib/python3.5/site-packages/xonsh/shell.py:97: RuntimeWarning: prompt-toolkit version < v0.57 and may not work as expected. Please update.
  warn(msg, RuntimeWarning)
🐌 21:09 her@air ~/Code echo @(1 + 1)
2
```

```
🐟 23:10 her@air ~/Code pip3 install git+https://github.com/scopatz/xonsh.git
Collecting git+https://github.com/scopatz/xonsh.git
  Cloning https://github.com/scopatz/xonsh.git to /var/folders/18/dj18p36518bcyk86pq9f5vf00000gn/T/pip-j7a7nldm-build
Installing collected packages: xonsh
  Running setup.py install for xonsh ... done
Successfully installed xonsh-0.2.7
🐟 23:10 her@air ~/Code xonsh
🐌 21:10 her@air ~/Code echo @(1 + 1)                                                                                                                                                                        
......................                                                                                                                                                                                      
🐌 21:10 her@air ~/Code 
```



Discussion:
But since the tests are passing, there is more likely a problem with my setup? I found out about Xonsh only today, so I wouldn't even know where to start looking.

Here's my `xonfig`:

```
+------------------+-------+
| xonsh            | 0.2.7 |
| Python           | 3.5.1 |
| PLY              | 3.7   |
| have readline    | True  |
| prompt toolkit   | 1.0.0 |
| pygments         | 2.1.3 |
| on posix         | True  |
| on linux         | False |
| on arch          | False |
| on windows       | False |
| on mac           | True  |
| are root user    | False |
| default encoding | utf-8 |
+------------------+-------+
```

Hey @azag0 -- thanks for reporting!  I'm pretty sure you're right that there is a bug and my bisection search agrees with you.  Could you try checkout out version `0.2.7` and installing that?

``` console
git checkout 0.2.7
python setup.py install
```

and let me know that works on your end?

Yep. A bug. It was introduced with this PR https://github.com/scopatz/xonsh/pull/789

Ok, so it looks like the commit in question is c92e0009 -- part of the big `and/or` push we just merged in.  I'm surprised we don't have tests for python evaluation... that should go on my todo list

Funny. I was totally expecting I'm missing something trivial. Just read the Developer's guide with it's focus on testing... :) Anyway, yes, installing 0.2.7 directly from the repo works as well as the wheel from pypi.

I'm not sure what part of c92e000 is causing trouble... here's a traceback

``` console
echo @(1+1)
........................................ 
Traceback (most recent call last):
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/execer.py", line 152, in _parse_ctx_free
    parsed = True
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/parsers/base.py", line 288, in parse
    if isinstance(tree, ast.Expression):
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/ply/yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/parsers/base.py", line 2301, in p_error
    self._parse_error(msg, self.currloc(lineno=p.lineno,
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/parsers/base.py", line 383, in _parse_error
    # Precedence of operators
  File "<string>", line None
SyntaxError: <xonsh-code>:1:12: "]" at (1, 12) ends "@(" at (1, 7) (expected ")")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/ptk/shell.py", line 96, in push
    locs=None)
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/execer.py", line 95, in compile
    tree = self.parse(input, ctx, mode=mode)
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/execer.py", line 70, in parse
    tree = self._parse_ctx_free(input, mode=mode)
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/execer.py", line 198, in _parse_ctx_free
    if sbpline is None:
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/execer.py", line 152, in _parse_ctx_free
    parsed = True
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/parsers/base.py", line 288, in parse
    if isinstance(tree, ast.Expression):
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/ply/yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/parsers/base.py", line 2305, in p_error
  File "/home/gil/anaconda/envs/xonshtest/lib/python3.5/site-packages/xonsh/parsers/base.py", line 383, in _parse_error
    # Precedence of operators
  File "<string>", line None
SyntaxError: <xonsh-code>:1:5: ('code: @(',)
```

This seems a little odd: `SyntaxError: <xonsh-code>:1:12: "]" at (1, 12) ends "@(" at (1, 7) (expected ")")`
Is something wrapping something else in brackets?  Or trying to?

Oh... yes, they are.

``` console
gil@theo ~ $ ![echo @(1+1)]
2
```

My guess would be it's the `RPAREN` [here](https://github.com/scopatz/xonsh/commit/c92e0009fc265fe9a768c3f78b874dae09c46496#diff-ece06b292b3c59cc7bfc962fe8a6b856R99), nope? Or [here](https://github.com/scopatz/xonsh/commit/c92e0009fc265fe9a768c3f78b874dae09c46496#diff-655d1c3afba4901e23212a5e05206ddbR15).

Yep (to your first) -- I'd just gotten there.  The following will fix Python evaluation but breaks grouping with parens

``` console
diff --git a/xonsh/tools.py b/xonsh/tools.py
index ac9717c..29cf0fa 100644
--- a/xonsh/tools.py
+++ b/xonsh/tools.py
@@ -96,7 +96,7 @@ class DefaultNotGivenType(object):
 DefaultNotGiven = DefaultNotGivenType()

 BEG_TOK_SKIPS = frozenset(['WS', 'INDENT', 'NOT', 'LPAREN'])
-END_TOK_TYPES = frozenset(['SEMI', 'AND', 'OR', 'RPAREN'])
+END_TOK_TYPES = frozenset(['SEMI', 'AND', 'OR'])#, 'RPAREN'])

 def subproc_toks(line, mincol=-1, maxcol=None, lexer=None, returnline=False):
     """Excapsulates tokens in a source code line in a uncaptured
```

Oh, whoops! This is annoying...

Same here, broken :(


Resolving patch:
diff --git a/xonsh/tools.py b/xonsh/tools.py
--- a/xonsh/tools.py
+++ b/xonsh/tools.py
@@ -97,6 +97,15 @@ class DefaultNotGivenType(object):
 
 BEG_TOK_SKIPS = frozenset(['WS', 'INDENT', 'NOT', 'LPAREN'])
 END_TOK_TYPES = frozenset(['SEMI', 'AND', 'OR', 'RPAREN'])
+LPARENS = frozenset(['LPAREN', 'AT_LPAREN', 'BANG_LPAREN', 'DOLLAR_LPAREN'])
+
+def _is_not_lparen_and_rparen(lparens, rtok):
+    """Tests if an RPAREN token is matched with something other than a plain old
+    LPAREN type.
+    """
+    # note that any([]) is False, so this covers len(lparens) == 0
+    return rtok.type == 'RPAREN' and any(x != 'LPAREN' for x in lparens)
+
 
 def subproc_toks(line, mincol=-1, maxcol=None, lexer=None, returnline=False):
     """Excapsulates tokens in a source code line in a uncaptured
@@ -110,15 +119,20 @@ def subproc_toks(line, mincol=-1, maxcol=None, lexer=None, returnline=False):
     lexer.reset()
     lexer.input(line)
     toks = []
+    lparens = []
     end_offset = 0
     for tok in lexer:
         pos = tok.lexpos
         if tok.type not in END_TOK_TYPES and pos >= maxcol:
             break
+        if tok.type in LPARENS:
+            lparens.append(tok.type)
         if len(toks) == 0 and tok.type in BEG_TOK_SKIPS:
             continue  # handle indentation
         elif len(toks) > 0 and toks[-1].type in END_TOK_TYPES:
-            if pos < maxcol and tok.type not in ('NEWLINE', 'DEDENT', 'WS'):
+            if _is_not_lparen_and_rparen(lparens, toks[-1]):
+                lparens.pop()  # don't continue or break
+            elif pos < maxcol and tok.type not in ('NEWLINE', 'DEDENT', 'WS'):
                 toks.clear()
                 if tok.type in BEG_TOK_SKIPS:
                     continue
@@ -145,7 +159,10 @@ def subproc_toks(line, mincol=-1, maxcol=None, lexer=None, returnline=False):
             break
     else:
         if len(toks) > 0 and toks[-1].type in END_TOK_TYPES:
-            toks.pop()
+            if _is_not_lparen_and_rparen(lparens, toks[-1]):
+                pass
+            else:
+                toks.pop()
         if len(toks) == 0:
             return  # handle comment lines
         tok = toks[-1]


Test patch:
diff --git a/tests/test_tools.py b/tests/test_tools.py
--- a/tests/test_tools.py
+++ b/tests/test_tools.py
@@ -137,7 +137,7 @@ def test_subproc_toks_ls_l_semi_ls_second():
     obs = subproc_toks(s, lexer=LEXER, mincol=7, returnline=True)
     assert_equal(exp, obs)
 
-def test_subproc_hello_mom_first():
+def test_subproc_toks_hello_mom_first():
     fst = "echo 'hello'"
     sec = "echo 'mom'"
     s = '{0}; {1}'.format(fst, sec)
@@ -145,7 +145,7 @@ def test_subproc_hello_mom_first():
     obs = subproc_toks(s, lexer=LEXER, maxcol=len(fst)+1, returnline=True)
     assert_equal(exp, obs)
 
-def test_subproc_hello_mom_second():
+def test_subproc_toks_hello_mom_second():
     fst = "echo 'hello'"
     sec = "echo 'mom'"
     s = '{0}; {1}'.format(fst, sec)
@@ -193,6 +193,58 @@ def test_subproc_toks_semicolon_only():
     obs = subproc_toks(';', lexer=LEXER, returnline=True)
     assert_equal(exp, obs)
 
+def test_subproc_toks_pyeval():
+    s = 'echo @(1+1)'
+    exp = '![{0}]'.format(s)
+    obs = subproc_toks(s, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_twopyeval():
+    s = 'echo @(1+1) @(40 + 2)'
+    exp = '![{0}]'.format(s)
+    obs = subproc_toks(s, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_pyeval_parens():
+    s = 'echo @(1+1)'
+    inp = '({0})'.format(s)
+    exp = '(![{0}])'.format(s)
+    obs = subproc_toks(inp, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_twopyeval_parens():
+    s = 'echo @(1+1) @(40+2)'
+    inp = '({0})'.format(s)
+    exp = '(![{0}])'.format(s)
+    obs = subproc_toks(inp, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_pyeval_nested():
+    s = 'echo @(min(1, 42))'
+    exp = '![{0}]'.format(s)
+    obs = subproc_toks(s, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_pyeval_nested_parens():
+    s = 'echo @(min(1, 42))'
+    inp = '({0})'.format(s)
+    exp = '(![{0}])'.format(s)
+    obs = subproc_toks(inp, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_capstdout():
+    s = 'echo $(echo bat)'
+    exp = '![{0}]'.format(s)
+    obs = subproc_toks(s, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_capproc():
+    s = 'echo !(echo bat)'
+    exp = '![{0}]'.format(s)
+    obs = subproc_toks(s, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+
 def test_subexpr_from_unbalanced_parens():
     cases = [
         ('f(x.', 'x.'),