Repository: xonsh/xonsh

Issue:
Some aliases stopped working after commit 7be2813
I have an alias for `ranger` which is not too complicated. It does two things, it renames my tmux window and it also changes to the last navigated directory upon exiting ranger.

This alias has worked for me for a long time. After checking out commit c262e0358f053ad374e256efcd31007141eb1c7e executing `ranger` at shell was working but with the next commit, 7be281375ae74995a7878cad20f387bb1e341353, executing `ranger` hangs. If I press `Ctrl+C` I can get back to the prompt.

I would be grateful if somebody could help me identify if I'm doing something wrong or if there is a bug.

``` python
def rename_term(args, stdin=None, tcmd=None, tname=None):                                                                             
    win_info = $(tmux display-message -p) 
    if win_info:                                                                                                                      
        win_info = win_info.split()[1].rstrip(',').split(':')                                                                         
    if win_info and len(win_info) == 2:                                                                                               
        $(tmux rename-window -t @(win_info[0]) @(tname)) 
    evalx(" ".join([tcmd]+args))  #Command is actually run here
    if win_info and len(win_info) == 2:                                                                                               
        sleep 0.25                                                                                                                    
        $(tmux rename-window -t @(win_info[0]) @(win_info[1])) 

@foreground                                                                                                                           
def execRanger(args, stdin=None):                                                                                                     
    rename_term(args=args,                                                                                                            
                tcmd=$(which -s ranger).strip(),   
                tname='ranger')                                                                                                       
    with open(join(expanduser('~'), ".config/ranger/bookmarks"), 'r') as book: 
        bookmarks = book.readlines()                                                                                                  
    currDir = [line for line in bookmarks if line.startswith("'")]                                                                    
    if currDir and currDir[0]:                                                                                                        
        currDir = currDir[0][2:].replace('\n', '')                                                                                    
        cd @(currDir)                                                                                                                 

aliases['ranger'] = execRanger
```



Discussion:
Hi @JohnLunzer, I'll take a look at these this weekend (assuming my university duties don't take over my life.)

Thanks for trying to find some time.

@JohnLunzer -- what happens if you remove the `@foreground` decorator?

If I remove `@foreground` then `ranger` loads but contains graphical errors, is sluggish, and upon exiting crashes `xonsh`. I wrote these aliases a long time ago, but that's probably why I added it in the first place.

Here is the traceback:

``` python
Traceback (most recent call last):
  File "/home/usr/xonsh/xonsh/ptk/shortcuts.py", line 108, in prompt
    document = cli.run(reset_current_buffer=False)
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/interface.py", line 389, in run
    self.eventloop.run(self.input, self.create_eventloop_callbacks())
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/eventloop/posix.py", line 154, in run
    t()
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/eventloop/posix.py", line 82, in read_from_stdin
    inputstream.feed(data)
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/terminal/vt100_input.py", line 380, in feed
    self._input_parser.send(c)
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/terminal/vt100_input.py", line 299, in _input_parser_generator
    self._call_handler(match, prefix)
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/terminal/vt100_input.py", line 332, in _call_handler
    self.feed_key_callback(KeyPress(key, insert_text))
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/interface.py", line 1007, in feed_key
    cli.input_processor.process_keys()
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/key_binding/input_processor.py", line 209, in process_keys
    cli.invalidate()
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/interface.py", line 333, in invalidate
    self.eventloop.call_from_executor(redraw, _max_postpone_until=_max_postpone_until)
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/eventloop/posix.py", line 237, in call_from_executor
    os.write(self._schedule_pipe[1], b'x')
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/usr/xonsh/scripts/xonsh", line 3, in <module>
    main()
  File "/home/usr/xonsh/xonsh/main.py", line 221, in main
    shell.shell.cmdloop()
  File "/home/usr/xonsh/xonsh/ptk/shell.py", line 123, in cmdloop
    line = self.singleline(auto_suggest=auto_suggest)
  File "/home/usr/xonsh/xonsh/ptk/shell.py", line 92, in singleline
    line = self.prompter.prompt(**prompt_args)
  File "/home/usr/xonsh/xonsh/ptk/shortcuts.py", line 113, in prompt
    eventloop.close()
  File "/jlunz/anaconda/envs/py35/lib/python3.5/site-packages/prompt_toolkit/eventloop/posix.py", line 254, in close
    os.close(schedule_pipe[1])
OSError: [Errno 9] Bad file descriptor

```

Can you post the output of `xonfig`?

Using the most recent upstream master

```
+------------------+---------------------------------------------------+
| xonsh            | 0.4.7                                             |
| Git SHA          | 00ef1c4                                           |
| Python           | 3.5.1                                             |
| PLY              | 3.7                                               |
| have readline    | True                                              |
| prompt toolkit   | 1.0.3                                             |
| shell type       | prompt_toolkit                                    |
| pygments         | 2.0.2                                             |
| on posix         | True                                              |
| on linux         | <xonsh.lazyasd.LazyBool object at 0x7fd10914bba8> |
| distro           | CentOS                                            |
| on darwin        | False                                             |
| on windows       | False                                             |
| on cygwin        | False                                             |
| is superuser     | False                                             |
| default encoding | utf-8                                             |
| xonsh encoding   | utf-8                                             |
| encoding errors  | surrogateescape                                   |
+------------------+---------------------------------------------------+
```

For good measure I updated prompt-toolkit to 1.0.8 with no change in behavior.

I thought that might help with the graphical irregularities in ranger but it didn't work on my end.  

The only place we use the `@foreground` decorator in xonsh core is in `trace` which also appears to hang now, so I'm thinking that's the likely culprit.

Thanks for hunting down the foreground issue. (And sorry I didn't get a chance to look at this over the weekend.) Looking at this now

I think that this is partially, though not wholly related to #1870.

Adding `ranger` with a `predict_false` to the command cache does eliminate the graphical weirdness

what is ranger?

@scopatz: It's a curses backed file manager written in Python

@JohnLunzer: Can you try out your alias now after removing the `@foreground` decorator?  And do you have any other alias failures on current master?

Ahh ok, basically everything that is curses based should go into the predict-false category 

@gforsyth , sorry I'm a little confused. I updated to the current master and removed the `@foreground` decorator and received the same graphical issues when running `ranger` in `tmux` and crashing when running `ranger` in the straight xonsh shell. 

I added `'ranger': predict_false` to `command_cache.py` and I'm getting the same behavior as if the `@foreground` decorator were there, ie. `ranger` doesn't start.

Additionally, if I exit the hung `ranger` with `ctrl+c` and then try to exit `xonsh` I get a message that says that jobs are still running.

Whoops!  Sorry, I thought I had added `ranger` to the threadable predictors dict.  
There may be other parts of your alias causing trouble -- can you confirm whether or not ranger works as expected if you just run it as is, without the alias (but with the entry in `commands_cache.py`)?

with `'ranger': predict_false` on the current master an unaliased `ranger` runs fine (no crashing or graphical errors ) inside and outside of `tmux`. I must be doing something funny in my alias...

Let me simplify the error case here, this alias:

``` python
rcmd = $(which -s ranger).strip()

def execRanger(args, stdin=None):
     evalx(rcmd)

aliases['ranger'] = execRanger
```

Hangs with `'ranger': predict_false` and graphical errors without. Am I misusing `evalx`? I also tried with `$(@(rcmd))` and the same behavior. 

If I just type `evalx(rcmd)` at the command line with `'ranger': predict_false` it runs without issue.

And just for comparison, this last example works with and without the `@foreground` decorator in commit https://github.com/xonsh/xonsh/commit/c262e0358f053ad374e256efcd31007141eb1c7e. 

Ahhh... ok, this is interesting.  No, you are not misuing `evalx`.  I can reproduce this with any program that we've set as not threadable being used inside an alias.  

``` console
def _vim(args, stdin=None):
    vim
aliases['myvim'] = _vim
vim
```

will hang, as will (mutatis mutandis) `htop`, `top`, `ranger`, etc...
but

``` console
def _python(args, stdin=None):
    ipython
aliases['ipy'] = _python
ipy
```

works fine.

I think we're getting caught up in `_update_last_spec` in `xonsh.built_ins` -- we don't do the threadability check if something is a callable alias.  Probably need to rework some of the logic there... I'll play with it for a bit, but this may require @scopatz's tender ministrations.

A glimmer of hope: $[myvim] seems to work for me.  Gonna figure out how to formalize this.


Resolving patch:
diff --git a/xonsh/built_ins.py b/xonsh/built_ins.py
--- a/xonsh/built_ins.py
+++ b/xonsh/built_ins.py
@@ -650,9 +650,13 @@ def resolve_alias_cls(self):
         cls = ProcProxyThread if thable else ProcProxy
         self.cls = cls
         self.threadable = thable
+        # also check capturablity, while we are here
+        cpable = getattr(alias, '__xonsh_capturable__', self.captured)
+        self.captured = cpable
 
 
-def _update_last_spec(last, captured=False):
+def _update_last_spec(last):
+    captured = last.captured
     last.last_in_pipeline = True
     if not captured:
         return
@@ -734,7 +738,7 @@ def cmds_to_specs(cmds, captured=False):
         else:
             raise XonshError('unrecognized redirect {0!r}'.format(redirect))
     # Apply boundry conditions
-    _update_last_spec(specs[-1], captured=captured)
+    _update_last_spec(specs[-1])
     return specs
 
 
@@ -760,6 +764,7 @@ def run_subproc(cmds, captured=False):
     Lastly, the captured argument affects only the last real command.
     """
     specs = cmds_to_specs(cmds, captured=captured)
+    captured = specs[-1].captured
     procs = []
     proc = pipeline_group = None
     for spec in specs:
diff --git a/xonsh/proc.py b/xonsh/proc.py
--- a/xonsh/proc.py
+++ b/xonsh/proc.py
@@ -1321,6 +1321,15 @@ def unthreadable(f):
 foreground = unthreadable
 
 
+def uncapturable(f):
+    """Decorator that specifies that a callable alias should not be run with
+    any capturing. This is often needed if the alias call interactive subprocess,
+    like pagers and text editors.
+    """
+    f.__xonsh_capturable__ = False
+    return f
+
+
 @lazyobject
 def SIGNAL_MESSAGES():
     sm = {


Test patch: