Repository: xonsh/xonsh

Issue:
[Regression] Multiline command in script fails
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: https://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

`xonsh/0.7.2` can run this script
```
from argparse import ArgumentParser

parser = ArgumentParser("Setup and start services")
parser.add_argument("--uwsgi", action="store_true", help="Start uwsgi")
parser.add_argument("--migrate", action="store_true", help="Migrate the db")

args = parser.parse_args()


if args.uwsgi:
    echo \
        --chdir=/link_share \
        --module=link_share.wsgi:application \
        --master \
        --pidfile=/tmp/link_share-master.pid \
        --socket=0.0.0.0:1234 \
        --processes=5 \
        --harakiri=20 \
        --max-requests=5000
```

but `xonsh/0.8.9` spits out a syntax error `SyntaxError: wat.xsh:1:0: ('code:     ',)`

## xonfig
<!--- Please post the output of the `xonfig` command (run from inside xonsh) so we know more about your current setup -->
```
+------------------+----------------------+
| xonsh            | 0.8.9                |
| Git SHA          | 4135c690             |
| Commit Date      | Jan 31 10:14:50 2019 |
| Python           | 3.5.2                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 2.0.4                |
| shell type       | prompt_toolkit2      |
| pygments         | None                 |
| on posix         | True                 |
| on linux         | True                 |
| distro           | Ubuntu               |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+

```

## Expected Behavior

It should output `--chdir='/link_share' --module=link_share.wsgi:application --master --pidfile='/tmp/link_share-master.pid' --socket=0.0.0.0:1234 --processes=5 --harakiri=20 --max-requests=5000`

## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
<!--- If part of your bug report is a traceback, please first enter debug mode before triggering the error
To enter debug mode, set the environment variable `XONSH_DEBUG=1` _before_ starting `xonsh`.  
On Linux and OSX, an easy way to to do this is to run `env XONSH_DEBUG=1 xonsh` -->

Spit out this trace [stacktrace.log](https://github.com/xonsh/xonsh/files/2822377/stacktrace.log)

## Steps to Reproduce
<!--- Please try to write out a minimal reproducible snippet to trigger the bug, it will help us fix it! -->

Put the folowing in a file and try to execute with xonsh

```
if True:
    echo \
        --chdir="/a/path/lol"
```

Remove the if and you'll get `SyntaxError: None: no further code`. 
[echo_trace.log](https://github.com/xonsh/xonsh/files/2822392/echo_trace.log)



Discussion:
Thanks for reporting @LoveIsGrief! We'll look into it! And PRs are also always welcome!
:smile: lexers and parsers are not my area of expertise. I'll just downgrade to 0.7.2. @scopatz . Thanks for xonsh though!
If you could figure out which version this first appeared in that would be helpful too :wink: 

Using [git-autobisect](https://github.com/grosser/git-autobisect), the offending commit is https://github.com/xonsh/xonsh/commit/4f56a648797b1d74102760b4772af55f0104a4f7 .

`git autobisect 'python3 scripts/xonsh /tmp/wat.xsh' &> /tmp/autobisect.log`

[autobisect.log](https://github.com/xonsh/xonsh/files/2823078/autobisect.log)

```bash
$ cat /tmp/wat.xsh 
if True:
    echo \
        --chdir="/a/path/lol" \
        --something \
        --somethingelse
```
Hey @LoveIsGrief , thanks for running the bisect!  

The result is... weird.  And doesn't make any sense.  That commit was running `black` on a test helper, and `black` compares the AST of files before and after to ensure correctness.  On top of that, that file doesn't actually get called by anything in xonsh at runtime.  So that's deeply confusing.
@gforsyth you're right... it's weird. I checked the previous commit and it had the error too. I'm going down the HEADs to now...
So, I gave up on the bisect since I thought I was going crazy and wrote a script to find the first commit with the error. Result:

(False is the test result)

False 743a8513899cf5a27f0b3aae52e7fec16879f530 `Merge pull request #2798 from xonsh/black`
False 40da5419a1b1bd803635fa340be812e54364c924 `Merge pull request #2800 from xonsh/tab_deep`
False 5f92e80b22de125c47b292c3fa25ec15bdf56748 `Updated CHANGELOG for 0.7.7`
False 1de73469c19f4108e406636e9a2ca4076c54d6b3 `bumped version to 0.7.7`
True 60970eb71a7858283360dadef222cfbeecfdfb31 `Merge pull request #2797 from xonsh/swapexcept`
True 2a1d5545afc6c9abd141a2da721ffa8e900012df `Merge pull request #2796 from xonsh/fix_ctrlr`
True 5278dbb385f7c532f974c950f75a7e27db853c83 `Merge pull request #2795 from xonsh/multiwrap2`

So even weirder is that 1de73469c19f4108e406636e9a2ca4076c54d6b3 is only a version bump :man_shrugging: 

Script:

```
cd /tmp/xonsh

# Start at master branch
git checkout master &> /dev/null

# Get oneliner of commit
last=$(git log --pretty=oneline -1 | head -1).strip()
$PYTHONPATH = "/tmp/xonsh"

i=3 # For testing purposes

# /tmp/doti is the virtualenv with the xonsh-test.requirements.txt
while i>0:
    out= !(/tmp/doti/bin/python3 scripts/xonsh /tmp/wat.xsh &> /dev/null).returncode == 0
    echo @(out) @(last) 
    # Move to previous ancestor
    git checkout HEAD^1 &> /dev/null
    current=$(git log --pretty=oneline -1 | head -1).strip()
    # loop until first commit (or Ctrl-C)
    if last == current:
        break
    last=current
    # i = i-1
```

I hope you have all the information you need
I don't get this. Wouldn't you need to reinstall xonsh in each iteration?
Why would I have to? Do you reinstall xonsh everytime you test code while developing? :thinking:   I just set the PYTHONPATH to the git directory. Had I not, then the test would always return False no matter the revision
Well. I usually reinstall. The thing is that when the parser change you may need to recreate the parser tables. I am not sure it would do that automatically, but @scopatz could probably tell you more.. 
With `xonsh` you really can't trust `PYTHONPATH` -- we generate parser tables at install time and the parser has gone through some changes.  I don't think you can trust the results of the `bisect` unless you are re-running install each time.
Alright, how about this commit? e1137f35c27e4ec5233e09722a2b572f3bfb210c

Here's the bash script that generated it [xonsh_test.txt](https://github.com/xonsh/xonsh/files/2829572/xonsh_test.txt). It creates a base docker with the requirements installed and at each bisect it installs xonsh and then runs the test.

Requirements are [git-autobisect](https://github.com/grosser/git-autobisect) and docker.
Yep, that looks more right. Thanks @LoveIsGrief!
Ok! I can confirm this failure.
A good test case for this is:

```
if True:
    echo a \
    b
```

Resolving patch:
diff --git a/xonsh/execer.py b/xonsh/execer.py
--- a/xonsh/execer.py
+++ b/xonsh/execer.py
@@ -14,6 +14,7 @@
     get_logical_line,
     replace_logical_line,
     balanced_parens,
+    starting_whitespace,
 )
 from xonsh.built_ins import load_builtins, unload_builtins, load_proxies, unload_proxies
 
@@ -207,6 +208,9 @@ def _parse_ctx_free(self, input, mode="exec", filename=None, logical_input=False
         greedy = False
         if filename is None:
             filename = self.filename
+        if logical_input:
+            beg_spaces = starting_whitespace(input)
+            input = input[len(beg_spaces) :]
         while not parsed:
             try:
                 tree = self.parser.parse(
@@ -313,4 +317,6 @@ def _parse_ctx_free(self, input, mode="exec", filename=None, logical_input=False
                 replace_logical_line(lines, sbpline, idx, nlogical)
                 last_error_col += 3
                 input = "\n".join(lines)
+        if logical_input:
+            input = beg_spaces + input
         return tree, input
diff --git a/xonsh/tools.py b/xonsh/tools.py
--- a/xonsh/tools.py
+++ b/xonsh/tools.py
@@ -616,6 +616,16 @@ def subexpr_before_unbalanced(expr, ltok, rtok):
     return subexpr
 
 
+@lazyobject
+def STARTING_WHITESPACE_RE():
+    return re.compile(r"^(\s*)")
+
+
+def starting_whitespace(s):
+    """Returns the whitespace at the start of a string"""
+    return STARTING_WHITESPACE_RE.match(s).group(1)
+
+
 def decode(s, encoding=None):
     encoding = encoding or DEFAULT_ENCODING
     return s.decode(encoding, "replace")


Test patch:
diff --git a/tests/test_execer.py b/tests/test_execer.py
--- a/tests/test_execer.py
+++ b/tests/test_execer.py
@@ -133,6 +133,7 @@ def test_echo_line_cont():
         "echo a and \\\necho b\n",
         "echo a \\\n or echo b\n",
         "echo a \\\n or echo b and \\\n echo c\n",
+        "if True:\\\n    echo a \\\n    b\n",
     ],
 )
 def test_two_echo_line_cont(code):