Repository: xonsh/xonsh

Issue:
surprising scoping behavior in `execx` and `xonsh -c`
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: https://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

## xonfig

<details>

```
$ xonfig
+------------------+-----------------+
| xonsh            | 0.9.27          |
| Python           | 3.8.9           |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 3.0.19          |
| shell type       | prompt_toolkit  |
| history backend  | json            |
| pygments         | 2.9.0           |
| on posix         | True            |
| on linux         | False           |
| on darwin        | True            |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
| on jupyter       | False           |
| jupyter kernel   | None            |
| xontrib          | []              |
+------------------+-----------------+
```

</details>

(I have also reproduced this on other xonsh installs)

## Expected Behavior
<!--- Tell us what should happen -->

`execx` and `xonsh -c` should have the same scoping behavior as `python -c`.


## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
<!--- If part of your bug report is a traceback, please first enter debug mode before triggering the error
To enter debug mode, set the environment variable `XONSH_DEBUG=1` _before_ starting `xonsh`.
On Linux and OSX, an easy way to to do this is to run `env XONSH_DEBUG=1 xonsh` -->

`execx` and `xonsh -c` expose xonsh-internal code in the caller's global scope. They also do not support defining variables and then referring to them in comprehensions, generators, functions, or lambdas.

`execx` and `xonsh -c` both ultimately wrap calls to python's [exec](https://docs.python.org/3/library/functions.html#exec), and by default both set `globals=None` and `locals=None`. That is to say, they effectively call `exec(<code>, None, None)`. Here's the relevant line for [execx](https://github.com/xonsh/xonsh/blob/7cf0307a0d53d198b8c05c83456d86af14c0daa4/xonsh/execer.py#L196) and here are the relevant lines for [xonsh -c](https://github.com/xonsh/xonsh/blob/7cf0307a0d53d198b8c05c83456d86af14c0daa4/xonsh/codecache.py#L62-L66).

(Incidentally, it seems to me like the logic for these two cases should be merged, but that's out of scope for this issue.)

Calling python's `exec` with `globals=None` and `locals=None` means that the code that gets executed has access to the same scope as the caller of `exec`. From the perspective of a xonsh user, this leads to some very surprising behavior.

First, everything that is in scope [here](https://github.com/xonsh/xonsh/blob/7cf0307a0d53d198b8c05c83456d86af14c0daa4/xonsh/execer.py#L196) is in scope to xonsh users calling `execx`. For example, `self` is in scope, and it refers to the Execer instance:

```xonsh
$ execx("print(self)")
<xonsh.execer.Execer object at 0x10a352be0>
$ execx("print(self.debug_level)")
0
```

The Execer class itself is in scope:

```xonsh
$ execx("print(Execer)")
<class 'xonsh.execer.Execer'>
```

And so is everything imported in that file:

```xonsh
$ execx("print(replace_logical_line)")
<function replace_logical_line at 0x109b028b0>
```

The same is also true of `xonsh -c`, but because the relevant call to `exec` in that codepath is in a different place in the codebase, it has different things in scope:

```xonsh
$ xonsh -c "print(self)"
<tracback snipped>
NameError: name 'self' is not defined
$ xonsh -c "print(run_script_with_cache)"
<function run_script_with_cache at 0x10d95b1f0>
```

Furthermore, in both cases the scope behaves as if it is embedded in a class definition, which has unusual scoping behavior. From the [Python execution model docs](https://docs.python.org/3/reference/executionmodel.html#resolution-of-names) (emphasis added):

>A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution **with an exception that unbound local variables are looked up in the global namespace**. The namespace of the class definition becomes the attribute dictionary of the class. **The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods – this includes comprehensions and generator expressions since they are implemented using a function scope**. This means that the following will fail:
>
>```
>class A:
>    a = 42
>    b = list(a + i for i in range(10))
>```

This means that the bodies of comprehensions, generator expressions, functions, and lambdas will all ignore what the xonsh user thinks is global scope. This in turn means that, for example, the name resolution of `x` will fail in all of these cases:

`execx("x=3;[x for _ in [0]]")`

`xonsh -c "x=3;[x for _ in [0]]"`

`execx("x=0;(lambda: x)()")`

`xonsh -c "x=0;(lambda: x)()"`

They all fail with `NameError: name 'x' is not defined`

## Steps to Reproduce
<!--- Please try to write out a minimal reproducible snippet to trigger the bug, it will help us fix it! -->

Run any of the snippets listed in "Current Behavior".

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**

surprising scoping behavior in `execx` and `xonsh -c`
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: https://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

## xonfig

<details>

```
$ xonfig
+------------------+-----------------+
| xonsh            | 0.9.27          |
| Python           | 3.8.9           |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 3.0.19          |
| shell type       | prompt_toolkit  |
| history backend  | json            |
| pygments         | 2.9.0           |
| on posix         | True            |
| on linux         | False           |
| on darwin        | True            |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
| on jupyter       | False           |
| jupyter kernel   | None            |
| xontrib          | []              |
+------------------+-----------------+
```

</details>

(I have also reproduced this on other xonsh installs)

## Expected Behavior
<!--- Tell us what should happen -->

`execx` and `xonsh -c` should have the same scoping behavior as `python -c`.


## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
<!--- If part of your bug report is a traceback, please first enter debug mode before triggering the error
To enter debug mode, set the environment variable `XONSH_DEBUG=1` _before_ starting `xonsh`.
On Linux and OSX, an easy way to to do this is to run `env XONSH_DEBUG=1 xonsh` -->

`execx` and `xonsh -c` expose xonsh-internal code in the caller's global scope. They also do not support defining variables and then referring to them in comprehensions, generators, functions, or lambdas.

`execx` and `xonsh -c` both ultimately wrap calls to python's [exec](https://docs.python.org/3/library/functions.html#exec), and by default both set `globals=None` and `locals=None`. That is to say, they effectively call `exec(<code>, None, None)`. Here's the relevant line for [execx](https://github.com/xonsh/xonsh/blob/7cf0307a0d53d198b8c05c83456d86af14c0daa4/xonsh/execer.py#L196) and here are the relevant lines for [xonsh -c](https://github.com/xonsh/xonsh/blob/7cf0307a0d53d198b8c05c83456d86af14c0daa4/xonsh/codecache.py#L62-L66).

(Incidentally, it seems to me like the logic for these two cases should be merged, but that's out of scope for this issue.)

Calling python's `exec` with `globals=None` and `locals=None` means that the code that gets executed has access to the same scope as the caller of `exec`. From the perspective of a xonsh user, this leads to some very surprising behavior.

First, everything that is in scope [here](https://github.com/xonsh/xonsh/blob/7cf0307a0d53d198b8c05c83456d86af14c0daa4/xonsh/execer.py#L196) is in scope to xonsh users calling `execx`. For example, `self` is in scope, and it refers to the Execer instance:

```xonsh
$ execx("print(self)")
<xonsh.execer.Execer object at 0x10a352be0>
$ execx("print(self.debug_level)")
0
```

The Execer class itself is in scope:

```xonsh
$ execx("print(Execer)")
<class 'xonsh.execer.Execer'>
```

And so is everything imported in that file:

```xonsh
$ execx("print(replace_logical_line)")
<function replace_logical_line at 0x109b028b0>
```

The same is also true of `xonsh -c`, but because the relevant call to `exec` in that codepath is in a different place in the codebase, it has different things in scope:

```xonsh
$ xonsh -c "print(self)"
<tracback snipped>
NameError: name 'self' is not defined
$ xonsh -c "print(run_script_with_cache)"
<function run_script_with_cache at 0x10d95b1f0>
```

Furthermore, in both cases the scope behaves as if it is embedded in a class definition, which has unusual scoping behavior. From the [Python execution model docs](https://docs.python.org/3/reference/executionmodel.html#resolution-of-names) (emphasis added):

>A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution **with an exception that unbound local variables are looked up in the global namespace**. The namespace of the class definition becomes the attribute dictionary of the class. **The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods – this includes comprehensions and generator expressions since they are implemented using a function scope**. This means that the following will fail:
>
>```
>class A:
>    a = 42
>    b = list(a + i for i in range(10))
>```

This means that the bodies of comprehensions, generator expressions, functions, and lambdas will all ignore what the xonsh user thinks is global scope. This in turn means that, for example, the name resolution of `x` will fail in all of these cases:

`execx("x=3;[x for _ in [0]]")`

`xonsh -c "x=3;[x for _ in [0]]"`

`execx("x=0;(lambda: x)()")`

`xonsh -c "x=0;(lambda: x)()"`

They all fail with `NameError: name 'x' is not defined`

## Steps to Reproduce
<!--- Please try to write out a minimal reproducible snippet to trigger the bug, it will help us fix it! -->

Run any of the snippets listed in "Current Behavior".

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**



Discussion:
Encountered in the wild [here](https://github.com/starship/starship/pull/1265#pullrequestreview-420227964) and [here](https://github.com/starship/starship/pull/2807#discussion_r666360733).

Reproduced by @dyuri [here](https://gitter.im/xonsh/xonsh?at=60e80b7417929c14e679614c).
Encountered in the wild [here](https://github.com/starship/starship/pull/1265#pullrequestreview-420227964) and [here](https://github.com/starship/starship/pull/2807#discussion_r666360733).

Reproduced by @dyuri [here](https://gitter.im/xonsh/xonsh?at=60e80b7417929c14e679614c).

Resolving patch:
diff --git a/xonsh/execer.py b/xonsh/execer.py
--- a/xonsh/execer.py
+++ b/xonsh/execer.py
@@ -145,6 +145,8 @@ def eval(
         self, input, glbs=None, locs=None, stacklevel=2, filename=None, transform=True
     ):
         """Evaluates (and returns) xonsh code."""
+        if glbs is None:
+            glbs = {}
         if isinstance(input, types.CodeType):
             code = input
         else:
@@ -175,6 +177,8 @@ def exec(
         transform=True,
     ):
         """Execute xonsh code."""
+        if glbs is None:
+            glbs = {}
         if isinstance(input, types.CodeType):
             code = input
         else:
diff --git a/xonsh/main.py b/xonsh/main.py
--- a/xonsh/main.py
+++ b/xonsh/main.py
@@ -481,7 +481,9 @@ def func_sig_ttin_ttou(n, f):
                 events.on_post_cmdloop.fire()
         elif args.mode == XonshMode.single_command:
             # run a single command and exit
-            run_code_with_cache(args.command.lstrip(), shell.execer, mode="single")
+            run_code_with_cache(
+                args.command.lstrip(), shell.execer, glb=shell.ctx, mode="single"
+            )
             if history is not None and history.last_cmd_rtn is not None:
                 exit_code = history.last_cmd_rtn
         elif args.mode == XonshMode.script_from_file:


Test patch:
diff --git a/tests/test_execer.py b/tests/test_execer.py
--- a/tests/test_execer.py
+++ b/tests/test_execer.py
@@ -170,3 +170,17 @@ def test_exec_print(capsys):
     check_exec("print(ls)", locs=dict(ls=ls))
     out, err = capsys.readouterr()
     assert out.strip() == repr(ls)
+
+
+def test_exec_function_scope():
+    # issue 4363
+    assert check_exec("x = 0; (lambda: x)()")
+    assert check_exec("x = 0; [x for _ in [0]]")
+
+
+def test_exec_scope_reuse():
+    # Scopes should not be reused between execs.
+    # A first-pass incorrect solution to issue 4363 made this mistake.
+    assert check_exec("x = 0")
+    with pytest.raises(NameError):
+        check_exec("print(x)")
diff --git a/tests/test_integrations.py b/tests/test_integrations.py
--- a/tests/test_integrations.py
+++ b/tests/test_integrations.py
@@ -764,8 +764,8 @@ def _echo(args):
 @pytest.mark.parametrize(
     "cmd, exp_rtn",
     [
-        ("sys.exit(0)", 0),
-        ("sys.exit(100)", 100),
+        ("import sys; sys.exit(0)", 0),
+        ("import sys; sys.exit(100)", 100),
         ("sh -c 'exit 0'", 0),
         ("sh -c 'exit 1'", 1),
     ],
@@ -800,3 +800,16 @@ def test_loading_correctly(monkeypatch, interactive):
         xonsh.__file__
     )  # make sure xonsh didn't fail and fallback to the system shell
     assert f"AAA {our_xonsh} BBB" in out  # ignore tty warnings/prompt text
+
+
+@skip_if_no_xonsh
+@pytest.mark.parametrize(
+    "cmd",
+    [
+        "x = 0; (lambda: x)()",
+        "x = 0; [x for _ in [0]]",
+    ],
+)
+def test_exec_function_scope(cmd):
+    _, _, rtn = run_xonsh(cmd, single_command=True)
+    assert rtn == 0