Repository: xonsh/xonsh

Issue:
Can’t capture assignment statement with ast macro
Capturing an assignment statement as an ast using a macro doesn’t seem to work for me. This may be a misunderstanding of how things are supposed to work, so shout if that’s the case!

## xonfig

<details>

```
+------------------+-----------------+
| xonsh            | 0.8.3           |
| Python           | 3.6.6           |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 2.0.7           |
| shell type       | prompt_toolkit2 |
| pygments         | 2.2.0           |
| on posix         | True            |
| on linux         | False           |
| on darwin        | True            |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
+------------------+-----------------+
```

</details>

## Expected Behavior

I have a function `check_ast` defined as follows in my `.xonshrc`:

```
def check_ast(x: ("ast", "exec")):
    print(x)
```

When I call that as a macro with an assignment, e.g. `check_ast!(x = 5)`, I expect to see output as if I had printed an AST node. By comparison, `print(ast.parse("x = 5"))` at the python repl gives: `<_ast.Module object at 0x102f1c5c0>`.

## Current Behavior

I get `SyntaxError: None: no further code` when I run `check_ast!(x = 5)`.

### Traceback (if applicable)

<details>

```
Traceback (most recent call last):
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/base_shell.py", line 360, in default
    run_compiled_code(code, self.ctx, None, "single")
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/codecache.py", line 67, in run_compiled_code
    func(code, glb, loc)
  File "<xonsh-code>", line 1, in <module>
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/built_ins.py", line 1144, in call_macro
    raw_arg, kind, glbs, locs, name=key, macroname=macroname
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/built_ins.py", line 1065, in convert_macro_arg
    arg = execer.parse(raw_arg, ctx, mode=mode, filename=filename)
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/execer.py", line 95, in parse
    tree, input = self._parse_ctx_free(input, mode=mode, filename=filename)
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/execer.py", line 231, in _parse_ctx_free
    raise original_error from None
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/execer.py", line 216, in _parse_ctx_free
    debug_level=(self.debug_level > 2),
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/parsers/base.py", line 546, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/nix/store/zak9a1w12r66j8pdfsjabzdnd3b64h6k-python3.6-ply-3.11/lib/python3.6/site-packages/ply/yacc.py", line 333, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/nix/store/zak9a1w12r66j8pdfsjabzdnd3b64h6k-python3.6-ply-3.11/lib/python3.6/site-packages/ply/yacc.py", line 1201, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/nix/store/zak9a1w12r66j8pdfsjabzdnd3b64h6k-python3.6-ply-3.11/lib/python3.6/site-packages/ply/yacc.py", line 192, in call_errorfunc
    r = errorfunc(token)
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/parsers/base.py", line 3275, in p_error
    self._parse_error("no further code", None)
  File "/nix/store/d9sdzqk041dlz2q4sm7wb2fyxplyrnh9-xonsh-0.8.3/lib/python3.6/site-packages/xonsh/parsers/base.py", line 679, in _parse_error
    raise err
  File "<string>", line None
SyntaxError: None: no further code
```

</details>


Discussion:
Thanks for reporting! This seems to be a bug.  I'll look into it!
Interestingly, if I put the following in a script, I don't see the issue.  The error during the call seems to happen only in the shell, in interactive mode.

```
def check_ast(x: ("ast", "exec")):
    print(x)

x = 5
check_ast!(x == 5)
```
Ooooh It seems to happen with `x = 5`, but not with `x == 5`

Resolving patch:
diff --git a/xonsh/built_ins.py b/xonsh/built_ins.py
--- a/xonsh/built_ins.py
+++ b/xonsh/built_ins.py
@@ -1074,6 +1074,8 @@ def convert_macro_arg(raw_arg, kind, glbs, locs, *, name="<arg>", macroname="<ma
         if locs is not None:
             ctx |= set(locs.keys())
         mode = mode or "eval"
+        if mode != "eval" and not raw_arg.endswith("\n"):
+            raw_arg += "\n"
         arg = execer.parse(raw_arg, ctx, mode=mode, filename=filename)
     elif kind is types.CodeType or kind is compile:  # NOQA
         mode = mode or "eval"


Test patch:
diff --git a/tests/test_builtins.py b/tests/test_builtins.py
--- a/tests/test_builtins.py
+++ b/tests/test_builtins.py
@@ -5,7 +5,7 @@
 import re
 import builtins
 import types
-from ast import AST
+from ast import AST, Module, Interactive, Expression
 
 import pytest
 
@@ -326,6 +326,61 @@ def f(x: str):
     assert rtn == 42
 
 
+def test_call_macro_ast_eval_expr():
+    def f(x: ("ast", "eval")):
+        return x
+
+    rtn = call_macro(f, ["x == 5"], {}, None)
+    assert isinstance(rtn, Expression)
+
+
+def test_call_macro_ast_single_expr():
+    def f(x: ("ast", "single")):
+        return x
+
+    rtn = call_macro(f, ["x == 5"], {}, None)
+    assert isinstance(rtn, Interactive)
+
+
+def test_call_macro_ast_exec_expr():
+    def f(x: ("ast", "exec")):
+        return x
+
+    rtn = call_macro(f, ["x == 5"], {}, None)
+    assert isinstance(rtn, Module)
+
+
+def test_call_macro_ast_eval_statement():
+    def f(x: ("ast", "eval")):
+        return x
+
+    try:
+        rtn = call_macro(f, ["x = 5"], {}, None)
+        assert False
+    except SyntaxError:
+        # It doesn't make sense to pass a statement to
+        # something that expects to be evaled
+        assert True
+    else:
+        assert False
+
+
+def test_call_macro_ast_single_statement():
+    def f(x: ("ast", "single")):
+        return x
+
+    rtn = call_macro(f, ["x = 5"], {}, None)
+    assert isinstance(rtn, Interactive)
+
+
+def test_call_macro_ast_exec_statement():
+    def f(x: ("ast", "exec")):
+        return x
+
+    rtn = call_macro(f, ["x = 5"], {}, None)
+    assert isinstance(rtn, Module)
+
+
 def test_enter_macro():
     obj = lambda: None
     rtn = enter_macro(obj, "wakka", True, True)