Repository: xonsh/xonsh

Issue:
Neovim when called through $EDITOR is broken
When opening `neovim` through a program that subprocess `$EDITOR` the editor is unusable in xonsh; e.g. `git commit`. 
Here's a screenshot:
![image](https://user-images.githubusercontent.com/5476164/89758423-447ba880-dad7-11ea-96a4-332fb3e08642.png)
In the example above I called command `kubectl edit` which opens `$EDITOR` with temporary file. This works absolutely fine with other editors such as `vim` or calling `nvim` in `bash` but something between `xonsh` and `neovim` is not working.


## xonfig

<details>

```
+------------------+----------------------+
| xonsh            | 0.9.19.dev8          |
| Git SHA          | 0fe76b53             |
| Commit Date      | Aug 10 06:28:52 2020 |
| Python           | 3.8.5                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 3.0.5                |
| shell type       | prompt_toolkit       |
| pygments         | 2.6.1                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | arch                 |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+
```

</details>

## Expected Behavior

Should open `neovim` and edit file the same way opening `nvim` process would

## Steps to Reproduce

1. Install `neovim` (tested with v0.4.4)  
2. set is as EDITOR: `$EDITOR='nvim'`  
3. call edit command like `git commit`



Discussion:
Hmm that is odd. I am not seeing this with `xo`, my text editor.  Thanks for reporting @Granitosaurus!
Can confirm on WSL2
Can confirm it myself (Linux). Did a quick check, in case of `git commit` `get_predictor_threadable` gets `git` (obviously) and git itself is threadable. It's not called when `nvim` is executed later.
Using `$THREAD_SUBPROCS=False` or `$[git commit]` works.
Yeah, I think if you are in this situation, you need to set `git` or `git commit` to be not threadable.  I don't know why this is only cropping up with `nvim` though
@scopatz can you set some programs as non-threadable in xonsh? 
I've experienced the same issue with `ranger` recently when a program called it through mimetype open (i.e. xdg-open) but I'm not sure how to replicate it right now.
Yes, you can always do `__xonsh__.commands_cache.threadable_predictors["my-cmd"] = lambda *a, **kw: False`
We probably should have a command for that. PRs welcome!
I am also affected by this. But not just through $EDITOR, running xonsh -c nvim from a xonsh prompt gives me the same problem as screenshoted above. I have set shell=/bin/bash in my neovim config.
What do I have to do to make it non-threadable then?
Add these to my .xonshrc:
`__xonsh__.commands_cache.threadable_predictors["nvim"] = lambda *a, **kw: False`
`__xonsh__.commands_cache.threadable_predictors["git"] = lambda *a, **kw: False`
?
`nvim` should be already flagged as non-threadable, so setting it non-threadable again won't change anything. Maybe it is ignored in case of single command mode?
Ok, the problem seems to be that `xonsh` itself is threadable by default, and anything that is not threadable but executed in a subshell breaks.

@lazmol as a workaround you may set `xonsh` to non-threadable via:
```
__xonsh__.commands_cache.threadable_predictors["xonsh"] = lambda *a, **kw: False
```

Resolving patch:
diff --git a/xonsh/environ.py b/xonsh/environ.py
--- a/xonsh/environ.py
+++ b/xonsh/environ.py
@@ -883,28 +883,33 @@ class GeneralSetting(Xettings):
         "not always happen.",
         is_configurable=False,
     )
+    XONSH_CAPTURE_ALWAYS = Var.with_default(
+        False,
+        "Try to capture output of commands run without explicit capturing.\n"
+        "If True, xonsh will capture the output of commands run directly or in ``![]``.\n"
+        "Setting to True has the following disadvantages:\n"
+        "* Some interactive commands won't work properly (like when ``git`` invokes an interactive editor).\n"
+        "  For more information see discussion at https://github.com/xonsh/xonsh/issues/3672.\n"
+        "* Stopping these commands with ^Z (i.e. ``SIGTSTP``)\n"
+        "  is disabled as it causes deadlocked terminals.\n"
+        "  ``SIGTSTP`` may still be issued and only the physical pressing\n"
+        "  of ``Ctrl+Z`` is ignored.\n\n"
+        "Regardless of this value, commands run in ``$()``, ``!()`` or with an IO redirection (``>`` or ``|``) "
+        "will always be captured.\n"
+        "Setting this to True depends on ``$THREAD_SUBPROCS`` being True.",
+    )
     THREAD_SUBPROCS = Var(
         is_bool_or_none,
         to_bool_or_none,
         bool_or_none_to_str,
         not ON_CYGWIN,
+        "Note: The ``$XONSH_CAPTURE_ALWAYS`` variable introduces finer control "
+        "and you should probably use that instead.\n\n"
         "Whether or not to try to run subrocess mode in a Python thread, "
-        "when applicable. There are various trade-offs, which normally "
-        "affects only interactive sessions.\n\nWhen True:\n\n"
-        "* Xonsh is able capture & store the stdin, stdout, and stderr \n"
+        "when trying to capture its output. There are various trade-offs.\n\n"
+        "If True, xonsh is able capture & store the stdin, stdout, and stderr \n"
         "  of threadable subprocesses.\n"
-        "* However, stopping threaded subprocs with ^Z (i.e. ``SIGTSTP``)\n"
-        "  is disabled as it causes deadlocked terminals.\n"
-        "  ``SIGTSTP`` may still be issued and only the physical pressing\n"
-        "  of ``Ctrl+Z`` is ignored.\n"
-        "* Threadable commands are run with ``PopenThread`` and threadable \n"
-        "  aliases are run with ``ProcProxyThread``.\n\n"
-        "When False:\n\n"
-        "* Xonsh may not be able to capture stdin, stdout, and stderr streams \n"
-        "  unless explicitly asked to do so.\n"
-        "* Stopping the thread with ``Ctrl+Z`` yields to job control.\n"
-        "* Threadable commands are run with ``Popen`` and threadable \n"
-        "  alias are run with ``ProcProxy``.\n\n"
+        "The disadvantages are listed in ``$XONSH_CAPTURE_ALWAYS``.\n"
         "The desired effect is often up to the command, user, or use case.\n\n"
         "None values are for internal use only and are used to turn off "
         "threading when loading xonshrc files. This is done because Bash "
diff --git a/xonsh/procs/specs.py b/xonsh/procs/specs.py
--- a/xonsh/procs/specs.py
+++ b/xonsh/procs/specs.py
@@ -734,6 +734,7 @@ def _update_last_spec(last):
         cmds_cache = XSH.commands_cache
         thable = (
             env.get("THREAD_SUBPROCS")
+            and (captured != "hiddenobject" or env.get("XONSH_CAPTURE_ALWAYS"))
             and cmds_cache.predict_threadable(last.args)
             and cmds_cache.predict_threadable(last.cmd)
         )


Test patch:
diff --git a/tests/procs/test_specs.py b/tests/procs/test_specs.py
--- a/tests/procs/test_specs.py
+++ b/tests/procs/test_specs.py
@@ -1,11 +1,11 @@
 """Tests the xonsh.procs.specs"""
+import itertools
 from subprocess import Popen
 
 import pytest
 
-from xonsh.procs.specs import cmds_to_specs
-
-from xonsh.environ import Env
+from xonsh.procs.specs import cmds_to_specs, run_subproc
+from xonsh.built_ins import XSH
 from xonsh.procs.posix import PopenThread
 from xonsh.procs.proxies import ProcProxy, ProcProxyThread
 
@@ -16,6 +16,16 @@
 def test_cmds_to_specs_thread_subproc(xession):
     env = xession.env
     cmds = [["pwd"]]
+
+    # XONSH_CAPTURE_ALWAYS=False should disable interactive threaded subprocs
+    env["XONSH_CAPTURE_ALWAYS"] = False
+    env["THREAD_SUBPROCS"] = True
+    specs = cmds_to_specs(cmds, captured="hiddenobject")
+    assert specs[0].cls is Popen
+
+    # Now for the other situations
+    env["XONSH_CAPTURE_ALWAYS"] = True
+
     # First check that threadable subprocs become threadable
     env["THREAD_SUBPROCS"] = True
     specs = cmds_to_specs(cmds, captured="hiddenobject")
@@ -35,3 +45,46 @@ def test_cmds_to_specs_thread_subproc(xession):
     env["THREAD_SUBPROCS"] = False
     specs = cmds_to_specs(cmds, captured="hiddenobject")
     assert specs[0].cls is ProcProxy
+
+
+@skip_if_on_windows
+@pytest.mark.parametrize(
+    "thread_subprocs, capture_always", list(itertools.product((True, False), repeat=2))
+)
+def test_capture_always(capfd, thread_subprocs, capture_always):
+    env = XSH.env
+    exp = "HELLO"
+    cmds = [["echo", "-n", exp]]
+
+    env["THREAD_SUBPROCS"] = thread_subprocs
+    env["XONSH_CAPTURE_ALWAYS"] = capture_always
+
+    hidden = run_subproc(cmds, "hiddenobject")  # ![]
+    # Check that interactive subprocs are always printed
+    assert exp in capfd.readouterr().out
+
+    if capture_always and thread_subprocs:
+        # Check that the interactive output was captured
+        assert hidden.out == exp
+    else:
+        # without THREAD_SUBPROCS capturing in ![] isn't possible
+        assert not hidden.out
+
+    # Explicitly captured commands are always captured
+    hidden = run_subproc(cmds, "object")  # !()
+    hidden.end()
+    if thread_subprocs:
+        assert not capfd.readouterr().out
+        assert hidden.out == exp
+    else:
+        # for some reason THREAD_SUBPROCS=False fails to capture in `!()` but still succeeds in `$()`
+        assert exp in capfd.readouterr().out
+        assert not hidden.out
+
+    output = run_subproc(cmds, "stdout")  # $()
+    assert not capfd.readouterr().out
+    assert output == exp
+
+    # Explicitly non-captured commands are never captured (/always printed)
+    run_subproc(cmds, captured=False)  # $[]
+    assert exp in capfd.readouterr().out
diff --git a/tests/test_lib/test_os.xsh b/tests/test_lib/test_os.xsh
--- a/tests/test_lib/test_os.xsh
+++ b/tests/test_lib/test_os.xsh
@@ -12,15 +12,15 @@ def test_indir():
     if ON_WINDOWS:
         pytest.skip("On Windows")
     with tempfile.TemporaryDirectory() as tmpdir:
-        assert ![pwd].output.strip() != tmpdir
+        assert $(pwd).strip() != tmpdir
         with indir(tmpdir):
-            assert ![pwd].output.strip() == tmpdir
-        assert ![pwd].output.strip() != tmpdir
+            assert $(pwd).strip() == tmpdir
+        assert $(pwd).strip() != tmpdir
         try:
             with indir(tmpdir):
                 raise Exception
         except Exception:
-            assert ![pwd].output.strip() != tmpdir
+            assert $(pwd).strip() != tmpdir
 
 
 def test_rmtree():
diff --git a/tests/test_pipelines.py b/tests/test_pipelines.py
--- a/tests/test_pipelines.py
+++ b/tests/test_pipelines.py
@@ -20,6 +20,7 @@ def patched_events(monkeypatch, xonsh_events, xonsh_execer):
     monkeypatch.setitem(
         XSH.env, "RAISE_SUBPROC_ERROR", False
     )  # for the failing `grep` commands
+    monkeypatch.setitem(XSH.env, "XONSH_CAPTURE_ALWAYS", True)  # capture output of ![]
     if ON_WINDOWS:
         monkeypatch.setattr(
             XSH,