Repository: xonsh/xonsh

Issue:
execx requires a line separator character at the end of the input
Hello! Thank you for xonsh!

I have a string variable with xonsh code. For example I need to change environment:
```
e = "print(123)"
evalx(e)
123
```
But:
```
e = "$TEST=1"
evalx(e)
```
Result:
```
  File "1.xsh", line 3, in <module>
    evalx(e)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 23092, in __call__
    return self.obj.__call__(*args, **kwargs)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 23246, in eval
    transform=transform,
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 23224, in compile
    tree = self.parse(input, ctx, mode=mode, filename=filename, transform=transform)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 23184, in parse
    tree, input = self._parse_ctx_free(input, mode=mode, filename=filename)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 23324, in _parse_ctx_free
    raise original_error from None
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 23309, in _parse_ctx_free
    debug_level=(self.debug_level > 2),
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/parsers/base.py", line 543, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/ply/ply/yacc.py", line 335, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/ply/ply/yacc.py", line 1203, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/ply/ply/yacc.py", line 194, in call_errorfunc
    r = errorfunc(token)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/parsers/base.py", line 3289, in p_error
    self._parse_error("no further code", None)
  File "/home/pc/miniconda3/lib/python3.7/site-packages/xonsh/parsers/base.py", line 676, in _parse_error
    raise err
SyntaxError: None: no further code
```
The same result for environ and execx:
```
evalx('os.environ["test"]=1')
...
SyntaxError: None: no further code
```
```
$ xonfig                                                                                                          
+------------------+-----------------+
| xonsh            | 0.9.14          |
| Python           | 3.7.6           |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 3.0.3           |
| shell type       | prompt_toolkit2 |
| pygments         | 2.5.2           |
| on posix         | True            |
| on linux         | True            |
| distro           | unknown         |
| on darwin        | False           |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
+------------------+-----------------+
```
Is there a way to solve this?

-----------

🥈 Workaround - https://github.com/xonsh/xonsh/issues/3481#issuecomment-599864134


Discussion:
```
$ eval('__xonsh__.env["asd"]=123')                                                                                                                                                                                                     
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
  File "<string>", line 1
    __xonsh__.env["asd"]=123
                        ^
SyntaxError: invalid syntax
```
```
$ evalx('__xonsh__.env["asd"]=123')                                                                                                                                                                                                    
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
  File "<string>", line None
SyntaxError: None: no further code
```
```
$ exec('__xonsh__.env["asd"]=123')                                                                                                                                                                                                     
# it works
```
```
$ execx('__xonsh__.env["asd"]=123')                                                                                                                                                                                                    
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
  File "<string>", line None
SyntaxError: None: no further code
```
That.... seems like a bug.

afaik, `evalx()` and friends should behave near identically to `source`.
I found that any string with equal sign fails:
```
$ evalx("a=1")                                                                                                    
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
  File "<string>", line None
SyntaxError: None: no further code
```
```
$ eval("a=1")                                                                                                     
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
  File "<string>", line 1
    a=1
     ^
SyntaxError: invalid syntax
```
Hi @anki-code, since in Python `a=1` is a statement, not an expression, `evalx` and `eval` are in fact expected to fail. `execx` should, however, work.
I got (once again) completely lost trying to understand how we parse the input. Our parser fails to parse most of the Python's [`simple_stmt`](https://docs.python.org/dev/reference/simple_stmts.html) cases (including the assignment statement). For instance:
```
$ __xonsh__.execer.parser.parse("raise",debug_level=3)
PLY: PARSE DEBUG START

State  : 0
Stack  : . LexToken(RAISE,'raise',1,0)
Action : Shift and goto state 97

State  : 97
Stack  : RAISE . $end
ERROR: Error  : RAISE . $end
  File "<string>", line None
SyntaxError: None: no further code
```
Yet, actually running the code works as expected:
```
$ raise
RuntimeError: No active exception to reraise
```
The parser also doesn't differentiate between "eval" and "exec" modes at all.
@xonsh/xore, any clues?
OK, after going through the parser tables, I can answer myself. `execx` requires the input to be line-end-terminated, so @anki-code, you need to
```
$ e = "$TEST=1\n"
$ execx(e)
$ $TEST
1
```
This is IMO still a bug, because `execx`'s behavior is inconsistent with Python's `exec`.
@laloch thank you very much for workaround! Please set the right title for issue.
@xonsh/xore, this still needs analysis.
The correct solution would in my opinion be the simplest one - to append a single `os.linesep` character to the `execx` input if needed. It might, however, also be a parser bug affecting other possible use cases I cannot see right now. In such a case, we would have to add a set of parser rules allowing the parser to accept statements without line terminator. For the `pass` statement example used above, the current
```
state 67

    (366) pass_stmt -> PASS .

    SEMI            reduce using rule 366 (pass_stmt -> PASS .)
    NEWLINE         reduce using rule 366 (pass_stmt -> PASS .)
```
would have to be extended by
```
    $end            reduce using rule 366 (pass_stmt -> PASS .)
```
I thought that execx already added a terminal newline, but if it doesn't it should!
@scopatz, thanks for the answer. I'll fix it. I have some related questions, tho. I'm not sure about the `evalx` behavior either:
```
$ a=0
$ evalx("a", locs=locals())
0
$ evalx("a\n", locs=locals())
TypeError: expected Expression node, got Module
$ cd @$(mktemp -d) and touch a b c d
$ evalx("ls")
a  b  c  d
```
So, should we `rstrip()` the `evalx` input to make the parser happy?
Shoud `evalx` execute shell code?
We could add an rstrip to evalx if we wanted as a convience. I am happy to have that. And thanks for fixing it @laloch!
It's rather for consistence with `eval`.
What about the shellcode execution. It that intended?

Resolving patch:
diff --git a/xonsh/execer.py b/xonsh/execer.py
--- a/xonsh/execer.py
+++ b/xonsh/execer.py
@@ -145,6 +145,7 @@ def eval(
         if isinstance(input, types.CodeType):
             code = input
         else:
+            input = input.rstrip("\n")
             if filename is None:
                 filename = self.filename
             code = self.compile(
@@ -174,6 +175,8 @@ def exec(
         if isinstance(input, types.CodeType):
             code = input
         else:
+            if not input.endswith("\n"):
+                input += "\n"
             if filename is None:
                 filename = self.filename
             code = self.compile(


Test patch:
diff --git a/tests/test_execer.py b/tests/test_execer.py
--- a/tests/test_execer.py
+++ b/tests/test_execer.py
@@ -2,7 +2,13 @@
 """Tests the xonsh lexer."""
 import os
 
-from tools import check_eval, check_parse, skip_if_on_unix, skip_if_on_windows
+from tools import (
+    check_eval,
+    check_exec,
+    check_parse,
+    skip_if_on_unix,
+    skip_if_on_windows,
+)
 
 import pytest
 
@@ -138,3 +144,12 @@ def test_echo_line_cont():
 )
 def test_two_echo_line_cont(code):
     assert check_parse(code)
+
+
+def test_eval_eol():
+    assert check_eval("0") and check_eval("0\n")
+
+
+def test_exec_eol():
+    locs = dict()
+    assert check_exec("a=0", locs=locs) and check_exec("a=0\n", locs=locs)
diff --git a/tests/tools.py b/tests/tools.py
--- a/tests/tools.py
+++ b/tests/tools.py
@@ -174,8 +174,6 @@ def is_manually_set(self, key):
 
 
 def check_exec(input, **kwargs):
-    if not input.endswith("\n"):
-        input += "\n"
     builtins.__xonsh__.execer.exec(input, **kwargs)
     return True