Repository: xonsh/xonsh

Issue:
xonsh appears to be closing file descriptors in subprocesses
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: http://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

## xonfig
<!--- Please post the output of the `xonfig` command (run from inside xonsh) so we know more about your current setup -->

```
+------------------+---------------------+
| xonsh            | 0.8.8               |
| Git SHA          | b64a2126            |
| Commit Date      | Jan 2 19:02:13 2019 |
| Python           | 3.6.7               |
| PLY              | 3.11                |
| have readline    | True                |
| prompt toolkit   | 2.0.7               |
| shell type       | prompt_toolkit2     |
| pygments         | 2.3.1               |
| on posix         | True                |
| on linux         | True                |
| distro           | ubuntu              |
| on darwin        | False               |
| on windows       | False               |
| on cygwin        | False               |
| on msys2         | False               |
| is superuser     | False               |
| default encoding | utf-8               |
| xonsh encoding   | utf-8               |
| encoding errors  | surrogateescape     |
+------------------+---------------------+
```

## Expected Behavior
<!--- Tell us what should happen -->

`$ redo hello` should produce `hello` executable

## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
<!--- If part of your bug report is a traceback, please first enter debug mode before triggering the error
To enter debug mode, set the environment variable `XONSH_DEBUG=1` _before_ starting `xonsh`.  
On Linux and OSX, an easy way to to do this is to run `env XONSH_DEBUG=1 xonsh` -->

```
$ redo hello
redo  hello
Traceback (most recent call last):
  File "/home/nkabir/.local/bin/redo-ifchange", line 11, in <module>
    sys.exit(main())
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/cmd_ifchange.py", line 41, in main
    jobserver.setup(0)
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/jobserver.py", line 238, in setup
    'prefix your Makefile rule with a "+"')
ValueError: broken --jobserver-auth from make; prefix your Makefile rule with a "+"
hello

```

## Steps to Reproduce
<!--- Please try to write out a minimal reproducible snippet to trigger the bug, it will help us fix it! -->

```
$ pip install --user redo-tools
$ pip3 install --user xonsh
```
Place `hello.do` and `hello.c` into the same folder:

`hello.do`
```
#!/usr/bin/env xonsh

#del $MAKEFLAGS
#del $REDO_CHEATFDS

gcc -o $3 hello.c

redo-ifchange hello.c

```
`hello.c`
```
#include <stdio.h>
int main()
{
   printf("Hello, World!");
   return 0;
}

```
Finally, execute 

```
$ redo hello
```
Note that explicitly deleting `MAKEFLAGS` and `REDO_CHEATFDS` is a work-around that forces the process to be serial.


Discussion:
Thanks for reporting @nkabir! I am not familar with what redo is or how it works.  Is there a more minimal example that you can show?  I use make and compilers in xonsh all the time....
Hi @scopatz 

Thanks for getting back! I created a minimal demonstration repository here

https://github.com/nkabir/fdcheck

Clone the repository to an Ubuntu 18.04 host and run `make all` and `make flagged` to reproduce the error. Please let me know if you encounter any issues. Xonsh is wonderful.-)

Also, redo is very versatile build tool that fits well with xonsh (aside from the file handle issue)

https://redo.readthedocs.io/en/latest/
Hmmm redo-tools seems to require Python 2, which I don't have anymore. Do you have an example that doesn't rely on Python 2 or is there a docker container that you can point me to? Sorry for all the trouble just getting started with this one.
Hi @scopatz. Which operating system do you primarily use? `redo` currently uses Python2. Python2 is still available in all of the major Linux repositories. If you're running Ubuntu, you can launch a new, clean Ubuntu 18.04 container named `demo` with

```
$ lxc launch ubuntu:18.04 demo
```

Or you can log into a clean Debian instance here:
https://taasproject.com

Then clone https://github.com/nkabir/fdcheck and run `make all`

These may be easier than debugging through a Docker instance.
What happens if you do:

```python
__xonsh__.commands_cache.threadable_predictors['redo'] = lambda *a, **k: False
```

in your current session or xonshrc?  Does this fix the issue?
Hi Anthony.

I placed the line at the top of my `.xonshrc` and then executed `redo` from within `xonsh` and there was no change. Could this be an issue with running Python2.7 processes inside `xonsh` hosted on Python3.6?

```
Traceback (most recent call last):
  File "/home/nkabir/.local/bin/redo-ifchange", line 11, in <module>
    sys.exit(main())
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/cmd_ifchange.py", line 41, in main
    jobserver.setup(0)
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/jobserver.py", line 238, in setup
    'prefix your Makefile rule with a "+"')
ValueError: broken --jobserver-auth from make; prefix your Makefile rule with a "+"

```
No, it shouldn't be a problem to run Python 2 inside of xonsh.  I am curious as to why you think this is a  xonsh filehandle error and not a redo error. The traceback seems to indicate an incorrect redo configuration and doesn't mention anything about xonsh.  Does this work in bash or another shell?
It works in `bash`. The error appeared when I ported redo scripts to `xonsh`.

I've brought this to the attention of Avery (`redo` author) and he suspected it was a `xonsh` issue. It's briefly covered here:
https://redo.readthedocs.io/en/latest/FAQParallel/

It's not a blocker for me at the moment--I'm disabling parallel builds with 

```
del $MAKEFLAGS
del $REDO_CHEATFDS
```

at the top of my scripts. I can report back once I need parallel builds and have time to dig deeper. Another thanks for `xonsh`. My build scripts are much easier to maintain with `xonsh`'s seamless integration with Python. `xonsh` has been a pleasure to use!
Oh interesting, so redo is actually launching xonsh, potentially via make.  That is very interesting.  I am still not sure why you / Avery think that this is a file descriptor issue. Can you rope Avery into this conversation to explain? Or is there a minimal Makefile that I can use to test this out?  

I am not sure how many folks have tried to use xonsh inside of make or gdb or other tools with assume that $SHELL is a posix-compatible shell. We, of course, want xonsh to work with these tools.

> xonsh has been a pleasure to use!

Thanks again!
I've posted to the `redo` mailing list here:

https://groups.google.com/d/msg/redo-list/eG63pNTm9KA/ya8DxlX-FgAJ

And I created this project that has a minimal Makefile to reproduce the issue:

https://github.com/nkabir/fdcheck

It runs as-is on a stock Ubuntu 18.04 host.
I have followed this issue from the redo mailing list.  I was unable to make xonsh after cloning because I do not have xonsh installed as required by Makefile.  I do not see bootstrap instructions -- though I did figure out scripts/xon.sh seems to work (but scripts/xonsh does not because "#!/usr/bin/env python3 -u" looks for an executable named "python3 -u" instead of python3 with the option -u).

A minimal example is:

```sh
xonsh -c 'echo x | dd of=/dev/fd/3' 3>&1
```

This fails (I used scripts/xon.sh instead of xonsh), but should instead do the same as: `xonsh -c 'echo x | dd'`.  I believe the reason is xonsh reportedly keeps the subprocess.Popen(close_fds=True) default (in Python3, the same default was False in Python2).  The minimal example works in dash 0.5.8-2.10 as installed on Ubuntu 18.04.

My understanding of the problem is that:
1. xonsh closes file descriptors when running commands (hence using /dev/fd/3 in the example)
2. redo expects the POSIX behavior of inherited descriptors (fd 3 in my example) not being closed

Here's a more complex example that doesn't require /dev/fd.  I am not sure how to adapt it to xonsh's tests.  Note in particular for the line marked A, the heredoc is a placeholder for "a program which uses a file descriptor read from an environment variable".  Two examples of such programs are redo and GNUMake.

```sh
#!/bin/sh -Cue
export FD=3
exec 3>&1  # actually a pipe(2) in the situation the OP encountered
for sh in scripts/xon.sh sh; do
    printf \\n%s\\n "testing $sh"
    rc=0
    "$sh" <<'END' || rc=$?
echo TEST | sh -c 'cat >&"$FD"' # mark A
END
    printf %s\\n "exit: $rc"
done
```
Below is a test program that doesn't use redo that demonstrates the problem with xonsh.  It uses os.pipe2() to create two file descriptors connected to each other by a pipe.  It then launched a subprocess, closes its own copy of the 'write' side of the pipe, and waits for the 'read' side to become ready.  Since nobody ever writes to the write side, this happens only when the no more processes exist that inherited the write fd.  Our parent process closed its copy, so the only copy is inherited by our subprocess and its children.

We run this test twice, once with each of two different subprocesses:
1) sh -c 'sleep 2 & exit'
2) xonsh -c 'sleep 2 &; exit'

In case (1), we end up waiting a full two seconds, even though 'sh' has exited right away.  (In sh, the 'exit' is not necessary, but I include it for symmetry with xonsh.)  In case (2), our test program ends up finishing before the 'sleep' command has returned.  This proves that the sleep command did not inherit the write fd as it should have.  On the other hand, if I remove the '; exit', then it waits 2 seconds, but that's because xonsh itself refuses to exit while it has a child process still running, even though that child was backgrounded.

This test reveals several weirdnesses in xonsh:

i) It shouldn't touch file descriptors that it didn't personally open.  I opened the write fd, I didn't refer to it from my xonsh command, and so the correct behaviour is to leave it alone.  Doing this wrong will break parallel builds in make and redo, but also break any programs that rely on fd passing to children, and others that rely on the above trick to determine if an entire process pipeline has ended before continuing.

ii) xonsh should exit right away if a background process exists and there are no remaining commands.  It isn't helping anyone by converting "xonsh -c 'sleep 2 &'" from an explicit background into a foreground process.

iii) The "don't exit until everything is done" behaviour might perhaps be nice in interactive mode, but in interactive mode, the "; exit" also doesn't exit, which is at least consistent.  In batch mode, it behaves differently depending whether or not you have "; exit" at the end, which is pretty strange.

```
#!/usr/bin/python3
import os, subprocess, sys, time

def test(argv):
    t0 = time.time()
    print('\ntesting %r' % argv)
    r, w = os.pipe2(0)
    p = subprocess.Popen(argv, close_fds=False)
    os.close(w)
    b = os.read(r, 4096)
    assert not b
    t1 = time.time()
    t = t1 - t0
    print('pipe closed; p exit status = %r; t = %r' % (p.poll(), t))
    assert t >= 2.0, 'sleep did not inherit write fd'
    print('ok.')

test(['sh', '-c', 'sleep 2 & exit'])
test(['xonsh', '-c', 'sleep 2 &; exit'])
```
Here's another example that uses just GNU make.

```
$ time make -sj2 SHELL=sh
real	0m2.010s
user	0m0.000s
sys	0m0.000s

$ time make -sj2 SHELL=xonsh
make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
real	0m5.001s
user	0m0.764s
sys	0m0.164s
```

Makefile:
```
default: all
all:
	$(MAKE) s
s:
	$(MAKE) a b
a:
	sleep 2
b:
	sleep 2
```
Thanks @rdpate @apenwarr for the detailed descriptions here! I am still trying to piece together what the underlying issue is here.  Part of the problem, I suspect, is that the two example cases @apenwarr listed are not semantically equivalent.  

>   1. `sh -c 'sleep 2 & exit'`

This means: run the sleep command for 2 sec AND if the exit code is zero (ie successful completion), run the exit command

>   2. `xonsh -c 'sleep 2 &; exit'`

This means: run the sleep command for 2 sec in the background. Then run the exit command.

A proper translation of (1) would be either:

3. `xonsh -c "sleep 2 && exit"`, or the more Pythonic `xonsh -c "sleep 2 and exit"`

(3) behaves as expected in that it runs for two seconds and then exits. A proper translation of (2) into bash would be:

4. `sh -c "bg sleep 2 && exit"`

This fails because there is no job control in Bash when not running as a REPL.  Maybe it is a little strange that xonsh has job control even when not in interactive mode, but it shouldn't get in anyone's way usually.

Can you please clarify which semantics, (1) or (2), are actually important?
Well... no.  All of that is wrong.

& and && are completely different operators in sh.  The only thing they have in common is that they are statement separators, like ";" is.  '&' runs the preceding statement in the background, while '&&' works as you described in both sh and xonsh.

It is obviously not true that
```    sh -c "sleep 2 & exit"```
is the same as
```    xonsh -c "sleep 2 && exit"```
because you can try running them and immediately discover that one returns right away, and the other returns after 2 seconds.

And this command you suggested,
```    sh -c "bg sleep 2 && exit"```
is just completely wrong.  The 'bg' command in sh does not take a command line under any circumstances, it takes a jobspec, like %1 or %2.  It also happens to print an error "bg: no job control" in this case, because apparently the bg command doesn't bother to parse its arguments when there is no job control, but that's irrelevant to the situation at hand.  When run interactively you get the "right" error:
```
bash$ bg sleep 2
-bash: bg: sleep: no such job
-bash: bg: 2: no such job
```

"job control" in a shell is a very specific thing that is different from the ability to run subprocesses.  The latter is available both interactively and non-interactively.  For example, this works fine:
```
bash$ bash -c 'date; sleep 2 & echo hello world; date; wait; date; bg'
Tue Feb  5 13:17:18 EST 2019
hello world
Tue Feb  5 13:17:18 EST 2019
Tue Feb  5 13:17:20 EST 2019
bash: line 0: bg: no job control
```

Notice how 'hello world' prints before 2 seconds have elapsed, and yet the 'wait' command works (and waits for 'sleep 2' to finish), even though job control is disabled.

In any case, this syntax stuff is a tangent from the original bug, which is about file descriptors not working as expected.  rdpate's one-liner example from above is nice and simple and avoids the need for background processes:
```
bash$ sh -c 'echo x | dd of=/dev/fd/3' 3>&1
x
0+1 records in
0+1 records out
2 bytes copied, 3.0603e-05 s, 65.4 kB/s

bash$ xonsh -c 'echo x | dd of=/dev/fd/3' 3>&1
/bin/dd: failed to open '/dev/fd/3': No such file or directory
```

Of course, you need to run those two commands from sh, because xonsh also doesn't understand 3>&1.  It does understand 2>&1, apparently as some kind of special case, leading me to suspect that xonsh has a really big blind spot around file descriptors in general.
Hmm I didn't think xonsh supported `>&` at all.  It must be there for posix compatibility, but clearly isn't implemented generally enough. 
That doesn't matter.  3>&1 is interpreted by bash in both versions of that one-liner.  xonsh *still* manages to then eat fd #3.
> Hmm I didn't think xonsh supported `>&` at all. It must be there for posix compatibility, but clearly isn't implemented generally enough.

My minimal test case is intended to be run from a shell such as zsh or bash, which then runs xonsh and shows the failure.  (As @apenwarr mentioned.)  I got a little creative (using dd and /dev/fd) in this test case to show that the descriptor manipulation is NOT done in xonsh (the dup(2) is done in a parent and it is a child that uses the descriptor).

You might see the other test case I presented, in which I was explicit in /bin/sh calling xonsh.
Also, `sh -c "sleep 2 & exit"` does not return right away for me, but rather waits 2 seconds.
Oddly, it only returns right away in bash, but waits in xonsh
Setting `close_fds=False` seems to fix this.
> Oddly, it only returns right away in bash, but waits in xonsh

Wow, you're right.  I just tried to strace what xonsh does when it launches a subprocess and... it's pretty complicated, involves starting several subthreads, and will surely lead to weird bugs.  But that's for another day!

Resolving patch:
diff --git a/xonsh/built_ins.py b/xonsh/built_ins.py
--- a/xonsh/built_ins.py
+++ b/xonsh/built_ins.py
@@ -377,18 +377,18 @@ class SubprocSpec:
     executed.
     """
 
-    kwnames = ("stdin", "stdout", "stderr", "universal_newlines")
+    kwnames = ("stdin", "stdout", "stderr", "universal_newlines", "close_fds")
 
     def __init__(
         self,
         cmd,
-        *,
         cls=subprocess.Popen,
         stdin=None,
         stdout=None,
         stderr=None,
         universal_newlines=False,
-        captured=False
+        close_fds=False,
+        captured=False,
     ):
         """
         Parameters
@@ -405,6 +405,9 @@ def __init__(
             Popen file descriptor or flag for stderr.
         universal_newlines : bool
             Whether or not to use universal newlines.
+        close_fds : bool
+            Whether or not to close the file descriptiors when the
+            process exits.
         captured : bool or str, optional
             The flag for if the subprocess is captured, may be one of:
             False for $[], 'stdout' for $(), 'hiddenobject' for ![], or
@@ -445,6 +448,7 @@ def __init__(
         self.stdout = stdout
         self.stderr = stderr
         self.universal_newlines = universal_newlines
+        self.close_fds = close_fds
         self.captured = captured
         # pure attrs
         self.args = list(cmd)
diff --git a/xonsh/proc.py b/xonsh/proc.py
--- a/xonsh/proc.py
+++ b/xonsh/proc.py
@@ -1261,6 +1261,7 @@ def __init__(
         stdout=None,
         stderr=None,
         universal_newlines=False,
+        close_fds=False,
         env=None,
     ):
         """Parameters
@@ -1285,6 +1286,9 @@ def __init__(
             set to `None`, then `sys.stderr` is used.
         universal_newlines : bool, optional
             Whether or not to use universal newlines.
+        close_fds : bool, optional
+            Whether or not to close file descriptors. This is here for Popen
+            compatability and currently does nothing.
         env : Mapping, optional
             Environment mapping.
         """
@@ -1309,6 +1313,7 @@ def __init__(
         self.stdin = stdin
         self.stdout = stdout
         self.stderr = stderr
+        self.close_fds = close_fds
         self.env = env or builtins.__xonsh__.env
         self._interrupted = False
 
@@ -1658,6 +1663,7 @@ def __init__(
         stdout=None,
         stderr=None,
         universal_newlines=False,
+        close_fds=False,
         env=None,
     ):
         self.orig_f = f
@@ -1669,6 +1675,7 @@ def __init__(
         self.stdout = stdout
         self.stderr = stderr
         self.universal_newlines = universal_newlines
+        self.close_fds = close_fds
         self.env = env
 
     def poll(self):


Test patch:
diff --git a/tests/test_integrations.py b/tests/test_integrations.py
--- a/tests/test_integrations.py
+++ b/tests/test_integrations.py
@@ -1,12 +1,15 @@
 import os
 import sys
+import time
 import shutil
+import tempfile
 import subprocess as sp
 
 import pytest
 
 import xonsh
 from xonsh.platform import ON_WINDOWS
+from xonsh.lib.os import indir
 
 from tools import (
     skip_if_on_windows,
@@ -43,7 +46,13 @@
 
 
 skip_if_no_xonsh = pytest.mark.skipif(
-    shutil.which("xonsh", path=PATH) is None, reason="xonsh not on path"
+    shutil.which("xonsh", path=PATH) is None, reason="xonsh not on PATH"
+)
+skip_if_no_make = pytest.mark.skipif(
+    shutil.which("make", path=PATH) is None, reason="make command not on PATH"
+)
+skip_if_no_sleep = pytest.mark.skipif(
+    shutil.which("sleep", path=PATH) is None, reason="sleep command not on PATH"
 )
 
 
@@ -470,3 +479,27 @@ def test_redirect_out_to_file(cmd, exp, tmpdir):
     if callable(exp):
         exp = exp()
     assert content == exp
+
+
+@skip_if_no_make
+@skip_if_no_xonsh
+@skip_if_no_sleep
+@skip_if_on_windows
+def test_xonsh_no_close_fds():
+    # see issue https://github.com/xonsh/xonsh/issues/2984
+    makefile = (
+        "default: all\n"
+        "all:\n"
+        "\t$(MAKE) s\n"
+        "s:\n"
+        "\t$(MAKE) a b\n"
+        "a:\n"
+        "\tsleep 1\n"
+        "b:\n"
+        "\tsleep 1\n"
+    )
+    with tempfile.TemporaryDirectory() as d, indir(d):
+        with open("Makefile", "w") as f:
+            f.write(makefile)
+        out = sp.check_output(["make", "-sj2", "SHELL=xonsh"], universal_newlines=True)
+        assert "warning" not in out