Repository: xonsh/xonsh

Issue:
Impossible to escape trailing ampersand in subprocess mode
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: https://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->
## xonfig
```
+------------------+----------------------+
| xonsh            | 0.9.13.dev1          |
| Git SHA          | 9f7ccc65             |
| Commit Date      | Oct 15 17:14:50 2019 |
| Python           | 3.6.10               |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 3.0.2                |
| shell type       | prompt_toolkit2      |
| pygments         | 2.5.2                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | linuxmint            |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+
```

## The situation
Assuming I'm in a directory with a file named `&` (this actually happened to me).
I am trying to examine it using `file &` or `file "&"` or `file '&'`

## Expected Behavior
By escaping the `&` through quotes, the single argument `&` should be passed to `file`.
The `bash` shell does it like this and outputs: `&: ASCII text`


## Current Behavior
`file` is started as a job with no argument
`[1]+ running: file & & (8218)`

Note that this is also very inconvenient if you want to `grep -r '&'` or something similar.

## Possible fixes
Although I just installed xonsh and are highly unfamiliar with the code, I took a quick look and came to the conclusion that it may be a bad idea not to treat `&` like `|`.
The command line input `file foo | less` will be represented as command sequence `(['file', 'foo'], '|', ['less'])` while the command line input `file bar '&'` will be represented as `(['file', 'bar', '&'])`, making the ampersand indistinguishable from a regular argument.



Discussion:
Have you tried `@("&")`?
Hi, `@("&")` shows the same behavior. I haven't examined it in detail, but it would be very reasonable that it results in the same problematic processing chain I described above.
I expect this happens for identical reasons as #2335: It has to do with the shape of the data structures that go from the parser to the execution handler.
Yes, this is essentially the same as #2335. It's exceptionally funny (until you happen to run something along the lines of `echo "/dev/sda4" ">" "/dev/sda1"`). The problem is, that the parser yields "<", ">" and "&" as regular (unquoted) subprocess arguments which are indistinguishable from string literals and then, somewhere under `run_subproc`, these are treated specially, depending on how many arguments follow or don't.
So, @ju-kreber, in your case the workaround would be `file "&" x` 😉

Jokes aside, this is especially nasty bug, that can eat your hamster.
Yeah, I figured it's a parsing problem. However, if I'm right, these problems do not apply to `|`.
For example `file '|'` vs `file |` results exactly in the difference in behavior that I would expect. So shouldn't it be possible to relatively simply change the parser to consider `<`, `>` and  `&` tokens similarly to `|`? Or is there a resulting difficulty that I'm not aware of? :thinking: 
Not really. The difference is that the pipe operator/token/atom is currently handled by the parser, whereas the redirection and background operators **are not**. The parser does not treat them specially (edit: it does, but there's no difference) and they are mixed together with the regular subprocess arguments instead.
So no, to fix this is not completely trivial. We most probably need to extend our grammar and wrap the current `subproc` atom into (possibly multiple levels of) input/output redirect atoms and eventually a background atom.
I forgot to mention that the redirection operators accept arguments by themselves and their syntax is again non-trivial.
There's this parser rule, that doesn't seem to get ever triggered :thinking: 
https://github.com/xonsh/xonsh/blob/3d795b85caf6e24ba1877ef213757d3b8bd74da2/xonsh/parsers/base.py#L2994-L2997
I wouldn't be surprised if these fail though:
```
🐚 echo "foo" "|" "bar"
🐚 echo "|" | cat
```
@astronouth7303, those are ok, because for pipe operator the parser sends the following down to the `run_subproc`:
```
$ echo "foo" "|" "bar"
foo | bar
# the commands were (['echo', 'foo', '|', 'bar'],)
$ echo "|" | cat
|
# the commands were (['echo', '|'], '|', ['cat'])
```

Resolving patch:
diff --git a/xonsh/built_ins.py b/xonsh/built_ins.py
--- a/xonsh/built_ins.py
+++ b/xonsh/built_ins.py
@@ -887,9 +887,6 @@ def cmds_to_specs(cmds, captured=False):
         if isinstance(cmd, str):
             redirects.append(cmd)
         else:
-            if cmd[-1] == "&":
-                cmd = cmd[:-1]
-                redirects.append("&")
             spec = SubprocSpec.build(cmd, captured=captured)
             spec.pipeline_index = i
             specs.append(spec)
@@ -903,7 +900,7 @@ def cmds_to_specs(cmds, captured=False):
             specs[i].stdout = w
             specs[i + 1].stdin = r
         elif redirect == "&" and i == len(redirects) - 1:
-            specs[-1].background = True
+            specs[i].background = True
         else:
             raise XonshError("unrecognized redirect {0!r}".format(redirect))
     # Apply boundary conditions
diff --git a/xonsh/parsers/base.py b/xonsh/parsers/base.py
--- a/xonsh/parsers/base.py
+++ b/xonsh/parsers/base.py
@@ -2984,6 +2984,14 @@ def p_pipe(self, p):
         """
         p[0] = ast.Str(s="|", lineno=self.lineno, col_offset=self.col)
 
+    def p_amper(self, p):
+        """amper : AMPERSAND
+                 | WS AMPERSAND
+                 | AMPERSAND WS
+                 | WS AMPERSAND WS
+        """
+        p[0] = ast.Str(s="&", lineno=self.lineno, col_offset=self.col)
+
     def p_subproc_s2(self, p):
         """subproc : subproc_atoms
                    | subproc_atoms WS
@@ -2991,10 +2999,10 @@ def p_subproc_s2(self, p):
         p1 = p[1]
         p[0] = [self._subproc_cliargs(p1, lineno=self.lineno, col=self.col)]
 
-    def p_subproc_amp(self, p):
-        """subproc : subproc AMPERSAND"""
+    def p_subproc_amper(self, p):
+        """subproc : subproc amper"""
         p1 = p[1]
-        p[0] = p1 + [ast.Str(s=p[2], lineno=self.lineno, col_offset=self.col)]
+        p[0] = p1 + [p[2]]
 
     def p_subproc_pipe(self, p):
         """subproc : subproc pipe subproc_atoms
@@ -3257,6 +3265,7 @@ def _attach_subproc_arg_part_rules(self):
             "DOLLAR_LBRACE",
             "DOLLAR_LBRACKET",
             "ATDOLLAR_LPAREN",
+            "AMPERSAND",
         }
         ts = "\n                 | ".join(sorted([t.lower() + "_tok" for t in toks]))
         doc = "subproc_arg_part : " + ts + "\n"


Test patch:
diff --git a/tests/test_integrations.py b/tests/test_integrations.py
--- a/tests/test_integrations.py
+++ b/tests/test_integrations.py
@@ -567,3 +567,26 @@ def test_negative_exit_codes_fail():
     out, err, rtn = run_xonsh(script)
     assert "OK" is not out
     assert "OK" is not err
+
+
+@pytest.mark.parametrize(
+    "cmd, exp",
+    [
+        ("echo '&'", "&\n"),
+        ("echo foo'&'", "foo'&'\n"),
+        ("echo foo '&'", "foo &\n"),
+        ("echo foo '&' bar", "foo & bar\n"),
+    ],
+)
+def test_ampersand_argument(cmd, exp):
+    script = """
+#!/usr/bin/env xonsh
+def _echo(args):
+    print(' '.join(args))
+aliases['echo'] = _echo
+{}
+""".format(
+        cmd
+    )
+    out, _, _ = run_xonsh(script)
+    assert out == exp