Repository: xonsh/xonsh

Issue:
Why do cat FILE and ls DIR behave differently in !(...)?
Why do `cat FILE` and `ls DIR` behave differently in `!(...)`?

## Expected Behavior
`!(cat FILE-LISTING_OF_DIR)` and `!(ls -1 DIR)` behave the same in Python mode giving a list of file names to iterate over.

## Current Behavior

`!(cat FILE-LISTING_OF_DIR)` works, `!(ls -1 DIR)` doesn't. Details in the next section:

## Steps to Reproduce

If I do 

    $ ls -1 $VIRTUALENV_HOME > ~/tmp/venvs.txt
    $ venvs = [ d for d in map(str.strip, !(cat ~/tmp/venvs.txt)) if Path(d).is_dir ]

I get - as expected - in `venvs` a Python list of all VENV directories

However, if I issue in one go

    $ venvs = [ d for d in map(str.strip, !(ls -1 $VIRTUALENV_HOME)) if Path(d).is_dir ]

I get the venvs listed on stdout and the variable `venvs` is empty.

Why on earth does this happen? And can that be right?

## xonfig

    +------------------+---------------------+
    | xonsh            | 0.8.3               |
    | Git SHA          | 07eb341c            |
    | Commit Date      | Nov 6 07:39:16 2018 |
    | Python           | 3.7.0               |
    | PLY              | 3.11                |
    | have readline    | True                |
    | prompt toolkit   | 2.0.4               |
    | shell type       | readline            |
    | pygments         | 2.2.0               |
    | on posix         | True                |
    | on linux         | False               |
    | on darwin        | True                |
    | on windows       | False               |
    | on cygwin        | False               |
    | on msys2         | False               |
    | is superuser     | False               |
    | default encoding | utf-8               |
    | xonsh encoding   | utf-8               |
    | encoding errors  | surrogateescape     |
    +------------------+---------------------+



Discussion:
Hey @halloleo -- thanks for reporting!
I agree that it should behave as you are expecting, not as it _is_ behaving. 
One of the threadable predictor heuristics is flagging `ls` as unthreadable (which disallows the capture of its output, hence the mismatch in behavior that you've noted).

There is a larger issue here to figure out why `ls` is being incorrectly flagged, but for now, you can workaround this by doing the following in your `xonshrc`:

```
from xonsh.commands_cache import predict_true

__xonsh__.commands_cache.threadable_predictors['ls'] = predict_true
```


Ok, so I'm also on OSX and in trying to figure this one out, discovered that `ls` on OSX relies on `ncurses` for no reason that I'm aware of.  But that's what is causing it to be flagged as unthreadable because usually `ncurses` based applications are interactive and don't play nice with output capture.

So, two options:
1. My quick fix above
2. install the `coreutils` package using `brew` and then alias `gls` for `ls`
Super thanks for the details. I guess I should dive a bit more in the internals of xonsh. :-)

I will check on Centos/RHEL as well what `ls` is doing and report back here.
I can confirm: On Centos/RHEL 6 it works in both cases. (Centos has `ls` from the `coreutils` installed.) So the fix is only needed on macOS
Great.  Thanks for following up @halloleo!  so the fix is to add `ls` as a `predict_true` in the default threadable predictors list in `command_cache.py`
@halloleo @gforsyth - any chance we can get a PR for this? Thanks!

Resolving patch:
diff --git a/xonsh/commands_cache.py b/xonsh/commands_cache.py
--- a/xonsh/commands_cache.py
+++ b/xonsh/commands_cache.py
@@ -403,6 +403,7 @@ def default_threadable_predictors():
         "ipython": predict_shell,
         "ksh": predict_shell,
         "less": predict_help_ver,
+        "ls": predict_true,
         "man": predict_help_ver,
         "more": predict_help_ver,
         "mvim": predict_help_ver,


Test patch: