Repository: xonsh/xonsh

Issue:
Environment variables inside f-strings
It's hard to describe what the exact bug is. See the _Additional info_ section below.

## xonfig
```
+------------------+-----------------+
| xonsh            | 0.7.8           |
| Python           | 3.7.0           |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 1.0.15          |
| shell type       | prompt_toolkit1 |
| pygments         | 2.2.0           |
| on posix         | True            |
| on linux         | True            |
| distro           | arch            |
| on darwin        | False           |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
+------------------+-----------------+
```

## Expected Behavior
It's a correct code, so it shouldn't cause the SyntaxError.

## Current Behavior
It causes the following SyntaxError (with `XONSH_DEBUG=1`):
```
  File "<string>", line None
SyntaxError: /home/arch/.local/lib/python3.7/site-packages/xontrib/prompt_vi_mode.xsh:1:21: ('code: ]',)
[f'{$HOME}', $PWD, '']
                     ^
```

## Steps to Reproduce
Just type `[f'{$HOME}', $PWD, '']` in the prompt.

## Additional info
### General problem with environment variables inside f-strings
Typing `f'{$HOME}'` inside the prompt doesn't trigger the SyntaxError, but the result isn't printed under the prompt as usual. `print(f'{$HOME}')` doesn't output anything as well.
e.g.
```
   f'{$HOME}s'                       
   print(f'{$HOME}s')      
                               
```

If I try to save the result inside the variable it seems to have no effect at all.
e.g.
```
   y = f'{$HOME}'                    
   y                                                
xonsh: subprocess mode: command not found: y
Did you mean one of the following?
    z:  Alias
    X:  Command (/usr/bin/X)
    w:  Command (/usr/bin/w)
    [:  Command (/usr/bin/[)
    R:  Command (/usr/bin/R)
   print(y)                                    
NameError: name 'y' is not defined
```
```
   y = 42                 
   y = f'{$HOME}'        
   print(y)            
42     
```

Instead of `f'{$HOME}'` one of the following can be used and the behavior will be the same. They don't trigger the SyntaxError as well.
```
[f'{$HOME}', '']

[f'{$HOME}', $PWD]
```

### Examples of working code
Following are the examples of code that __does__ work as expected:
```
x=2; [f'{x}', $PWD, '']

['{$HOME}', $PWD, '']
```

The first one uses f-string and evaluates normal python code inside it.
The second one doesn't involve f-strings at all.

### Another issue with f-strings
It seems there are problems with f-strings in general too. While using them I've run into the following error several times. I've found out it is triggered every time by typing `f""`, without pressing <kbd>Enter</kbd>.
```
   f""Exception in thread Thread-2566:    
Traceback (most recent call last):                                                                                                                                    
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner                                                                                               
    self.run()                                                                                                                                                        
  File "/usr/lib/python3.7/threading.py", line 865, in run                                                                                                             
    self._target(*self._args, **self._kwargs)                                                                                                                         
  File "/usr/lib/python3.7/site-packages/prompt_toolkit/interface.py", line 865, in run                                                                                
    completions = list(buffer.completer.get_completions(document, complete_event))
  File "/usr/lib/python3.7/site-packages/xonsh/ptk/completer.py", line 45, in get_completions
    prefix, line_ex, begidx + expand_offset, endidx + expand_offset, self.ctx
  File "/usr/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 125, in complete
    out = func(prefix, line, begidx, endidx, ctx)
  File "/usr/lib/python3.7/site-packages/xonsh/completers/__amalgam__.py", line 1274, in complete_path
    p = _path_from_partial_string(line, end)
  File "/usr/lib/python3.7/site-packages/xonsh/completers/__amalgam__.py", line 1064, in _path_from_partial_string
    val = ast.literal_eval(_string)
  File "/usr/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "/usr/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "/usr/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.JoinedStr object at 0x7f82d1549d68>
```



Discussion:
## My thoughts
The parsing of f-strings is currently handled [here](https://github.com/xonsh/xonsh/blob/fd448e515a36cf23a4f03f4e6b16c9dd227196b7/xonsh/parsers/base.py#L2332-L2335). It seems like the f-strings are parsed with regular python parser.

The `val = ast.literal_eval(_string)` in the second bug seems like another place where the string is evaluated with normal python parser instead of our xonshy one.

Still, I have no idea why it sometimes fails with a SyntaxError and sometimes fails silently. The fact that it fails silently might be a sign of some issue with the error handling.
Hi @MrQubo thanks for reporting - that is weird about the `f""` issue.  Can you please open a separate issue for that.  I can't reproduce it, but I am on Python v3.6 and using readline and PTK v2.  I suspect this is either an issue with Python v3.7 or PTK v1.

It seems like what is needed here is to extend the f-string execution context to include the environment and also extend the string minilanguage to include parsing the `$` to know to look up the variable name in the environment.  I think I see a relatively easy fix, though.
@scopatz I can confirm the completer exception with PTK1. It doesn't trigger with readline.
Ahh thanks @laloch! If it is really a PTK v1 bug, I think xore is unlikely to fix it because PTK v1 is deprecated. But we'd happily take a PR that fixes it :wink: 
The bad news is that I have similar issue with PTK2. Sorry @scopatz :(
``` ~ xonfig                                                                                                                                                                                               +------------------+-----------------+
| xonsh            | 0.7.9           |
| Python           | 3.7.0           |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 2.0.4           |
| shell type       | prompt_toolkit2 |
| pygments         | 2.2.0           |
| on posix         | True            |
| on linux         | True            |
| distro           | arch            |
| on darwin        | False           |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
+------------------+-----------------+

Unhandled exception in event loop:
  File "/home/laloch/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/coroutine.py", line 90, in step_next
    new_f = coroutine.throw(exc)
  File "/home/laloch/.local/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1637, in new_coroutine
    yield From(coroutine(*a, **kw))
  File "/home/laloch/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/coroutine.py", line 90, in step_next
    new_f = coroutine.throw(exc)
  File "/home/laloch/.local/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1495, in async_completer
    cancel=lambda: not proceed()))
  File "/home/laloch/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/coroutine.py", line 86, in step_next
    new_f = coroutine.send(None)
  File "/home/laloch/.local/lib/python3.7/site-packages/prompt_toolkit/eventloop/async_generator.py", line 114, in consume_async_generator
    item = iterator.send(send)
  File "/home/laloch/.local/lib/python3.7/site-packages/prompt_toolkit/completion/base.py", line 166, in get_completions_async
    for item in self.get_completions(document, complete_event):
  File "/usr/lib/python3.7/site-packages/xonsh/ptk2/completer.py", line 45, in get_completions
    prefix, line_ex, begidx + expand_offset, endidx + expand_offset, self.ctx
  File "/usr/lib/python3.7/site-packages/xonsh/__amalgam__.py", line 125, in complete
    out = func(prefix, line, begidx, endidx, ctx)
  File "/usr/lib/python3.7/site-packages/xonsh/completers/__amalgam__.py", line 1274, in complete_path
    p = _path_from_partial_string(line, end)
  File "/usr/lib/python3.7/site-packages/xonsh/completers/__amalgam__.py", line 1064, in _path_from_partial_string
    val = ast.literal_eval(_string)
  File "/usr/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "/usr/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "/usr/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))

Exception malformed node or string: <_ast.JoinedStr object at 0x7fa240aa5a58>
Press ENTER to continue...                                                                                                                                                                                            
 ~ f""
```
hmm so maybe it is a Python 3.7 issue, then.

Resolving patch:
diff --git a/xonsh/parsers/base.py b/xonsh/parsers/base.py
--- a/xonsh/parsers/base.py
+++ b/xonsh/parsers/base.py
@@ -26,6 +26,9 @@
 RE_STRINGPREFIX = LazyObject(
     lambda: re.compile(StringPrefix), globals(), "RE_STRINGPREFIX"
 )
+RE_FSTR_ENVVAR = LazyObject(
+    lambda: re.compile("\{\s*\$(\w+)"), globals(), "RE_FSTR_ENVVAR"
+)
 
 
 class Location(object):
@@ -208,6 +211,26 @@ def hasglobstar(x):
         return False
 
 
+def _repl_sub_env_vars_single(matchobj):
+    return "{__xonsh_env__.detype()['" + matchobj.group(1) + "']"
+
+
+def _repl_sub_env_vars_double(matchobj):
+    return '{__xonsh_env__.detype()["' + matchobj.group(1) + '"]'
+
+
+def sub_env_vars(fstring):
+    """Takes an fstring that may contain environment variables and
+    substitues them for a valid environment lookup call. Roughly,
+    for example, this will take f"{$HOME}" and transform it to
+    be f"{__xonsh_env__.detype()['HOME']}".
+    """
+    repl = (
+        _repl_sub_env_vars_single if fstring[-1] == '"' else _repl_sub_env_vars_double
+    )
+    return RE_FSTR_ENVVAR.sub(repl, fstring)
+
+
 class YaccLoader(Thread):
     """Thread to load (but not shave) the yacc parser."""
 
@@ -2382,7 +2405,8 @@ def p_string_literal(self, p):
                 "__xonsh_path_literal__", [s], lineno=p1.lineno, col=p1.lexpos
             )
         elif "f" in prefix or "F" in prefix:
-            s = pyparse(p1.value).body[0].value
+            s = sub_env_vars(p1.value)
+            s = pyparse(s).body[0].value
             s = ast.increment_lineno(s, p1.lineno - 1)
             p[0] = s
         else:


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
--- a/tests/test_parser.py
+++ b/tests/test_parser.py
@@ -9,6 +9,7 @@
 
 from xonsh.ast import AST, With, Pass, pdump
 from xonsh.parser import Parser
+from xonsh.parsers.base import sub_env_vars
 
 from tools import VER_FULL, skip_if_py34, skip_if_lt_py36, nodes_equal
 
@@ -122,6 +123,27 @@ def test_f_literal():
     check_ast('F"{yo}"', run=False)
 
 
+@skip_if_lt_py36
+def test_f_env_var():
+    check_xonsh_ast({}, 'f"{$HOME}"', run=False)
+    check_xonsh_ast({}, "f'{$XONSH_DEBUG}'", run=False)
+    check_xonsh_ast({}, 'F"{$PATH} and {$XONSH_DEBUG}"', run=False)
+
+
+@pytest.mark.parametrize(
+    "inp, exp",
+    [
+        ('f"{$HOME}"', "f\"{__xonsh_env__.detype()['HOME']}\""),
+        ('f"{ $HOME }"', "f\"{__xonsh_env__.detype()['HOME'] }\""),
+        ("f\"{'$HOME'}\"", "f\"{'$HOME'}\""),
+        ('f"$HOME"', 'f"$HOME"'),
+    ],
+)
+def test_sub_env_vars(inp, exp):
+    obs = sub_env_vars(inp)
+    assert exp == obs
+
+
 def test_raw_bytes_literal():
     check_ast('br"hell\o"')
     check_ast('RB"hell\o"')