Repository: xonsh/xonsh

Issue:
Comment on same line as code in script 
The lexer / parser seems to hang indefinitely in the event that a comment follows a subprocess command in a script.  Namely, I found that the following failed:

``` console
def func():
    ./setup.py install --user  # FIXME this should be made more general
```



Discussion:
Hmmm this doesn't seem to happen in all cases, though:

```
hartz@lappy-513 ~/x/x/parsers $ print(7) # hello
7
hartz@lappy-513 ~/x/x/parsers $ echo "hi" # hello
hi
hartz@lappy-513 ~/x/x/parsers $ ls -la #hello
total 108
drwxr-xr-x 3 hartz hartz  4096 Feb 11 08:27 .
drwxr-xr-x 5 hartz hartz  4096 Feb 11 08:27 ..
-rw-r--r-- 1 hartz hartz 80232 Feb 11 08:27 base.py
-rw-r--r-- 1 hartz hartz     0 Feb 11 08:27 __init__.py
drwxr-xr-x 2 root  root   4096 Feb 11 08:27 __pycache__
-rw-r--r-- 1 hartz hartz  5179 Feb 11 08:27 v34.py
-rw-r--r-- 1 hartz hartz  4937 Feb 11 08:27 v35.py
```

Seems to be subprocess commands _within functions_ that have comments after them.

```
hartz@lappy-513 ~/x/x/parsers $ def foo(x):
                                    print(x) # hello; this runs fine

hartz@lappy-513 ~/x/x/parsers $ def foo(x):
                                    echo @(x) # hello; this hangs

```

I wonder if this is somehow getting caught in another loop of wrapping those lines with `$[...]` (maybe my fix from earlier doesn't catch all cases...it could be the whitespace at the start is confusing that fix).

Ahhh interesting observation. We are probably not testing this particular case of function bodies.

In fact, it's not even an issue with that.  Any subproc-mode command with whitespace in front breaks.  I tracked it down to #601 being merged in where this started breaking (something with multiline not quite right), but I haven't yet dug deep enough to figure out exactly what change within that PR is actually responsible.

Wrapping with `$[...]` is a workaround (whitespace before that structure is fine).

FTR, the test I used that broke things was: `<SPACE>echo "hi"<ENTER>`

According to my tests, that works on master before the multiline mode changes, and not after.

> Wrapping with $[...] is a workaround (whitespace before that structure is fine).

Good to know.  Thanks for tracking down what caused this problem. 

```
builtins.__xonsh_execer__.compile(' echo "hi"', mode='single', glbs=None, locs=builtins.__xonsh_ctx__)
```

hangs.  That's what's killing this in the multiline parser.  Investigating further...

Looks to me like the base reason is that the initial space is being treated as an `INDENT` token, and subproc mode doesn't know how to deal with that.

Ok.  I have some more thoughts/findings on this, but still no solution.  

For one thing, @scopatz original example will also hang the readline shell.

@adqm, your test which does break in PTK multiline works in the readline shell, because of `base_shell.precmd` which gets called by the readline shell and performs an `lstrip()`

oddly, the following hangs (since I guess it doesn't go through `precmd` for some reason?)

``` Python
from xonsh.shell import Shell

rl = Shell(shell_type='readline')
rl.default(' echo "hi"')
```

All by way of saying, I think this goes deeper.  #601 is definitely where `<space>echo "hi"<enter>` started breaking on PTK, since that's when the `lstrip()` (via `precmd`) was removed, but the original failure persists going back to v0.2.4

So, I think that `'  echo "hello"'` should be broken when passed directly into `default()`.  That is because this is the equivalent of started a Python module with a line that begins in whitespace.  That said, this should be a SyntaxError, not a hang.  

As I recall (and this was a while ago) @SmartViking and I had some discussion about this in #110 and #130.  The lstrip thing is a convenience for people at the shell and not part of the xonsh language.  precmd() is a shell notion, and not a language idea.


Resolving patch:
diff --git a/xonsh/ptk/key_bindings.py b/xonsh/ptk/key_bindings.py
--- a/xonsh/ptk/key_bindings.py
+++ b/xonsh/ptk/key_bindings.py
@@ -72,7 +72,8 @@ def __call__(self, cli):
 
 def can_compile(src):
     """Returns whether the code can be compiled, i.e. it is valid xonsh."""
-    src = src if src.endswith('\n') else '{}\n'.format(src)
+    src = src if src.endswith('\n') else src + '\n'
+    src = src.lstrip()
     try:
         builtins.__xonsh_execer__.compile(src, mode='single', glbs=None,
                                           locs=builtins.__xonsh_ctx__)
diff --git a/xonsh/tools.py b/xonsh/tools.py
--- a/xonsh/tools.py
+++ b/xonsh/tools.py
@@ -125,7 +125,14 @@ def subproc_toks(line, mincol=-1, maxcol=None, lexer=None, returnline=False):
             tok.type = 'NEWLINE'
             tok.value = '\n'
             tok.lineno -= 1
-            tok.lexpos = len(line)
+            if len(toks) >= 2:
+                prev_tok_end = toks[-2].lexpos + len(toks[-2].value)
+            else:
+                prev_tok_end = len(line)
+            if '#' in line[prev_tok_end:]:
+                tok.lexpos = prev_tok_end  # prevents wrapping comments
+            else:
+                tok.lexpos = len(line)
             break
     else:
         if len(toks) == 0:


Test patch:
diff --git a/tests/test_execer.py b/tests/test_execer.py
--- a/tests/test_execer.py
+++ b/tests/test_execer.py
@@ -86,6 +86,16 @@ def test_indent_with_empty_line():
             '    some_command for_sub_process_mode\n')
     yield check_parse, code
 
+def test_command_in_func():
+    code = ('def f():\n'
+            '    echo hello\n')
+    yield check_parse, code
+
+def test_command_in_func_with_comment():
+    code = ('def f():\n'
+            '    echo hello # comment\n')
+    yield check_parse, code
+
 
 
 if __name__ == '__main__':
diff --git a/tests/test_tools.py b/tests/test_tools.py
--- a/tests/test_tools.py
+++ b/tests/test_tools.py
@@ -104,6 +104,22 @@ def test_subproc_toks_ls_str_comment():
     obs = subproc_toks(s + com, lexer=LEXER, returnline=True)
     assert_equal(exp, obs)
 
+def test_subproc_toks_indent_ls_comment():
+    ind = '    '
+    s = 'ls -l'
+    com = '  # lets list'
+    exp = '{0}$[{1}]{2}'.format(ind, s, com)
+    obs = subproc_toks(ind + s + com, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
+def test_subproc_toks_indent_ls_str():
+    ind = '    '
+    s = 'ls "wakka"'
+    com = '  # lets list'
+    exp = '{0}$[{1}]{2}'.format(ind, s, com)
+    obs = subproc_toks(ind + s + com, lexer=LEXER, returnline=True)
+    assert_equal(exp, obs)
+
 def test_subproc_toks_ls_l_semi_ls_first():
     lsdl = 'ls -l'
     ls = 'ls'