Repository: xonsh/xonsh

Issue:
`xonsh.__env__` and `env` subproc command precedence issue
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: http://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

If you use `env` to set a temporary env var for a one-line command, that works as expected
```
 ~ 🐚  env BAI="HAI" bash -c "echo $BAI"
"HAI"
```
and `$BAI` isn't available in the current session after the command completes
```
 ~ 🐚  $BAI
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
KeyError: 'Unknown environment variable: $BAI'
```

But if `$BAI` is set already and then `env` is used, the value in `__xonsh__.env` overrides the temporary value:
```
 ~ 🐚  $BAI="BYE"
 ~ 🐚  env BAI="HAI" bash -c "echo $BAI"
BYE
```

## xonfig
<!--- Please post the output of the `xonfig` command (run from inside xonsh) so we know more about your current setup -->
```
+------------------+---------------------+
| xonsh            | 0.8.3               |
| Git SHA          | 07eb341c            |
| Commit Date      | Nov 6 07:39:16 2018 |
| Python           | 3.6.5               |
| PLY              | 3.11                |
| have readline    | True                |
| prompt toolkit   | 2.0.6               |
| shell type       | prompt_toolkit2     |
| pygments         | 2.2.0               |
| on posix         | True                |
| on linux         | False               |
| on darwin        | True                |
| on windows       | False               |
| on cygwin        | False               |
| on msys2         | False               |
| is superuser     | False               |
| default encoding | utf-8               |
| xonsh encoding   | utf-8               |
| encoding errors  | surrogateescape     |
+------------------+---------------------+
```
## Expected Behavior
<!--- Tell us what should happen -->
```
 ~ 🐚  $BAI="BYE"
 ~ 🐚  env BAI="HAI" bash -c "echo $BAI"
HAI
```


Discussion:
Have you found where the expansion takes place? I already wanted to change the behavior once to not expand the `DOLLARNAME` token within `r""` raw strings. Funnily enough, I wasn't even able to find out where exactly the `DOLLARNAME` is being expanded in this particular case.
Hey @laloch -- which expansion are you referring to?
Well, usual environment variable ID references (aka dollar names) get lexed [here](https://github.com/xonsh/xonsh/blob/e9866478b4a306dab78dce84fbe075186cef13c0/xonsh/tokenize.py#L1013-L1014) into the `DOLLARNAME` token which is later parsed [here](https://github.com/xonsh/xonsh/blob/e9866478b4a306dab78dce84fbe075186cef13c0/xonsh/parsers/base.py#L2358-L2360). That does not, however, apply to "DOLLARNAMEs" embedded in single/double/triple quoted strings. IIRC I looked also at several other places within the parser where some kind of environment substitution takes place, but I just could not find the place where quoted environment variable IDs get expanded.
Thanks for opening this @gforsyth! @laloch is right that the env var expansion happens in/by xonsh **before** running the subprocess. Since the subprocess here is `env`, which is an alias and could do literally anything (or nothing), xonsh has no way (currently) of predicting what the intended value for the env var is.  Note that bash has this problem too:

```bash
scopatz@artemis:~$ env BAI="HAI" bash -c "echo $BAI"

scopatz@artemis:~$ export BAI="BYE"
scopatz@artemis:~$ env BAI="HAI" bash -c "echo $BAI"
BYE
```

This is actually a perfect illustration of the use case where `env VAR=VALUE cmd` does not necessarily equal the results of `VAR=VALUE cmd`.  Bash still fails in this case, but we could parse out the env vars and then expand them in the context of just the command.  It is probably time to implement this long-awaited feature :)
Quoted values IIRC are expanded by calls to `__xonsh__.expand_path()`

Resolving patch:
diff --git a/xonsh/parsers/base.py b/xonsh/parsers/base.py
--- a/xonsh/parsers/base.py
+++ b/xonsh/parsers/base.py
@@ -2448,8 +2448,9 @@ def p_string_literal(self, p):
         else:
             s = ast.literal_eval(p1.value)
             is_bytes = "b" in prefix
+            is_raw = "r" in prefix
             cls = ast.Bytes if is_bytes else ast.Str
-            p[0] = cls(s=s, lineno=p1.lineno, col_offset=p1.lexpos)
+            p[0] = cls(s=s, lineno=p1.lineno, col_offset=p1.lexpos, is_raw=is_raw)
 
     def p_string_literal_list(self, p):
         """string_literal_list : string_literal
@@ -3164,9 +3165,12 @@ def p_subproc_atom_re(self, p):
 
     def p_subproc_atom_str(self, p):
         """subproc_atom : string_literal"""
-        p0 = xonsh_call(
-            "__xonsh__.expand_path", args=[p[1]], lineno=self.lineno, col=self.col
-        )
+        if hasattr(p[1], "is_raw") and p[1].is_raw:
+            p0 = p[1]
+        else:
+            p0 = xonsh_call(
+                "__xonsh__.expand_path", args=[p[1]], lineno=self.lineno, col=self.col
+            )
         p0._cliarg_action = "append"
         p[0] = p0
 


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
--- a/tests/test_parser.py
+++ b/tests/test_parser.py
@@ -7,7 +7,7 @@
 
 import pytest
 
-from xonsh.ast import AST, With, Pass, pdump
+from xonsh.ast import AST, With, Pass, pdump, Str, Call
 from xonsh.parser import Parser
 from xonsh.parsers.base import eval_fstr_fields
 
@@ -2889,6 +2889,20 @@ def test_withbang_as_many_suite(body):
         assert s == body
 
 
+def test_subproc_raw_str_literal():
+    tree = check_xonsh_ast({}, "!(echo '$foo')", run=False, return_obs=True)
+    assert isinstance(tree, AST)
+    subproc = tree.body
+    assert isinstance(subproc.args[0].elts[1], Call)
+    assert subproc.args[0].elts[1].func.attr == "expand_path"
+
+    tree = check_xonsh_ast({}, "!(echo r'$foo')", run=False, return_obs=True)
+    assert isinstance(tree, AST)
+    subproc = tree.body
+    assert isinstance(subproc.args[0].elts[1], Str)
+    assert subproc.args[0].elts[1].s == "$foo"
+
+
 # test invalid expressions