Repository: conda/conda

Issue:
conda run swallows stderr output
<!--
Hi!  Read this; it's important.

This is an issue tracker for conda -- the package manager.  File feature requests
for conda here, as well as bug reports about something conda has messed up.

If your issue is a bug report for:
  * a specific conda package from Anaconda ('defaults' channel):
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * a specific conda package from conda-forge:
      ==> file at the corresponding feedstock under https://github.com/conda-forge
  * repo.anaconda.com access and service:
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * anaconda.org access and service:
      ==> file at https://anaconda.org/contact/report
  * commands under 'conda build':
      ==> file at https://github.com/conda/conda-build
  * commands under 'conda env':
      ==> please file it here!
  * all other conda commands that start with 'conda':
      ==> please file it here!

If you continue on,

   **please include all requested information below.**

If a maintainer determines the information is required to understand 
your issue, and if it is not provided, your issue may be
closed automatically.

-->

## Current Behavior
<!-- What actually happens?
     If you want to include console output, please use "Steps to Reproduce" below. -->
If the command executed with `conda run` prints something to stderr, that output is not shown. If it prints a lot, then it freezes.

### Steps to Reproduce
<!-- If the current behavior is a bug, please provide specific, minimal steps to independently reproduce.
     Include the exact conda commands that reproduce the issue and their output between the ticks below. -->
First, create `test.py` with the following contents:
```python
import sys
print('x' * int(sys.argv[1]), file=sys.stderr)
```

Then:
```
>conda run -n base python test.py 1
[nothing is printed]
>conda run -n base python test.py 10000
[nothing is printed _and_ the command freezes]
```


## Expected Behavior
<!-- What do you think should happen? -->
All stderr output from the command should be passed unaltered to `conda`'s own stderr.

## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda info' -->

```
     active environment : base
    active env location : C:\Users\[redacted]\AppData\Local\Continuum\miniconda3
            shell level : 1
       user config file : C:\Users\[redacted]\.condarc
 populated config files :
          conda version : 4.6.7
    conda-build version : not installed
         python version : 3.7.1.final.0
       base environment : C:\Users\[redacted]\AppData\Local\Continuum\miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\[redacted]\AppData\Local\Continuum\miniconda3\pkgs
                          C:\Users\[redacted]\.conda\pkgs
                          C:\Users\[redacted]\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\[redacted]\AppData\Local\Continuum\miniconda3\envs
                          C:\Users\[redacted]\.conda\envs
                          C:\Users\[redacted]\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.7 requests/2.21.0 CPython/3.7.1 Windows/10 Windows/10.0.16299
          administrator : False
             netrc file : None
           offline mode : False
```
</p></details>


<details open><summary><code>`conda config --show-sources`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda config --show-sources' -->

```

```
</p></details>


<details><summary><code>`conda list --show-channel-urls`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda list --show-channel-urls' -->

```
# packages in environment at C:\Users\[redacted]\AppData\Local\Continuum\miniconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py37_0    defaults
ca-certificates           2019.1.23                     0    defaults
certifi                   2018.11.29               py37_0    defaults
cffi                      1.11.5           py37h74b6da3_1    defaults
chardet                   3.0.4                    py37_1    defaults
conda                     4.6.7                    py37_0    defaults
conda-env                 2.6.0                         1    defaults
console_shortcut          0.1.1                         3    defaults
cryptography              2.4.2            py37h7a1dbc1_0    defaults
idna                      2.8                      py37_0    defaults
menuinst                  1.4.14           py37hfa6e2cd_0    defaults
openssl                   1.1.1b               he774522_0    defaults
pip                       18.1                     py37_0    defaults
pycosat                   0.6.3            py37hfa6e2cd_0    defaults
pycparser                 2.19                     py37_0    defaults
pyopenssl                 18.0.0                   py37_0    defaults
pysocks                   1.6.8                    py37_0    defaults
python                    3.7.1                h8c8aaf0_6    defaults
pywin32                   223              py37hfa6e2cd_1    defaults
requests                  2.21.0                   py37_0    defaults
ruamel_yaml               0.15.46          py37hfa6e2cd_0    defaults
setuptools                40.6.3                   py37_0    defaults
six                       1.12.0                   py37_0    defaults
sqlite                    3.26.0               he774522_0    defaults
urllib3                   1.24.1                   py37_0    defaults
vc                        14.1                 h0510ff6_4    defaults
vs2015_runtime            14.15.26706          h3a45250_0    defaults
wheel                     0.32.3                   py37_0    defaults
win_inet_pton             1.0.1                    py37_1    defaults
wincertstore              0.2                      py37_0    defaults
yaml                      0.1.7                hc54c509_2    defaults

```
</p></details>



Discussion:
This is especially annoying since some CLIs rely on the user gathering information from stdout.
For instance `bokeh/panel serve plot.ipynb` relies on the user getting the output that tells them which url to use to find their app.

```
(sample) ➜  ~ panel serve plot.ipynb
2019-09-04 13:40:39,793 Starting Bokeh server version 1.3.4 (running on Tornado 6.0.3)
2019-09-04 13:40:39,797 Bokeh app running at: http://localhost:5006/plot
2019-09-04 13:40:39,797 Starting Bokeh server with process id: 95442
```
conda run is an unfinished feature. We claim it to be beta, but I'd call it pre-alpha. PRs to finish it would be gratefully reviewed.

It needs (at least, IMHO):

1. Choice of spawn vs execve invocation (config option to set the default?)
2. A thread to process stdin (at all) and stderr and stdout (properly).

Understood. I just learned about it yesterday when we were talking about anaconda-project which tries to solve a lot of the same problems.
Maybe we can steal some of your code then? :-) Does anaconda-project address the stdin problem at all?
It doesn't address stdin no. It has a different approach, where all your commands are declared in a config file. 

conda run would be a great command to have! Currently this is still an issue. :-(
It seems that conda run has been written with the unix philosophy to hide all the stdout and only squark if there's an error, but the performance cost of using communicate() seems a high price for this. Users running a command would probably be keen to see its std out.

Rather than capturing the stdsout and stderr via communicate() with the associated performace ramifications, why not just leave off the stdout, stderr settings on Popen and let them be piped straight into the parent process' stdout and stderr. 

The only place the captured information is used in main_run.py is here:
```
    if process.returncode != 0:
        log = getLogger(__name__)
        log.error("Subprocess for 'conda run {}' command failed.  Stderr was:\n{}"
                  .format(call, stderr))
```
If the stdout and stderr aren't specified and thus default to passing through then there's no need for this and we can amend to:
```
    if process.returncode != 0:
        log = getLogger(__name__)
        log.error("Subprocess for 'conda run {}' command failed.  (See above for error details)\n{}"
                  .format(call, stderr))
```

Resolving patch:
diff --git a/conda/cli/main_run.py b/conda/cli/main_run.py
--- a/conda/cli/main_run.py
+++ b/conda/cli/main_run.py
@@ -3,7 +3,7 @@
 
 from logging import getLogger
 import os
-from subprocess import PIPE, Popen
+from subprocess import Popen
 import sys
 
 from ..base.context import context
@@ -22,21 +22,13 @@ def execute(args, parser):
     script_caller, command_args = wrap_subprocess_call(on_win, context.root_prefix, prefix,
                                                        args.dev, args.debug_wrapper_scripts, call)
     env = encode_environment(os.environ.copy())
-    process = Popen(command_args, universal_newlines=False, stdout=PIPE, stderr=PIPE,
+    process = Popen(command_args, universal_newlines=False,
                     env=env, cwd=cwd)
-    stdout, stderr = process.communicate()
-    if hasattr(stdout, "decode"):
-        stdout = stdout.decode('utf-8', errors='replace')
-    if hasattr(stderr, "decode"):
-        stderr = stderr.decode('utf-8', errors='replace')
-    if stdout:
-        sys.stdout.write(stdout)
-    if stderr:
-        sys.stderr.write(stderr)
+    process.wait()
     if process.returncode != 0:
         log = getLogger(__name__)
-        log.error("Subprocess for 'conda run {}' command failed.  Stderr was:\n{}"
-                  .format(call, stderr))
+        log.error("Subprocess for 'conda run {}' command failed.  (See above for error)"
+                  .format(call))
     if script_caller is not None:
         if 'CONDA_TEST_SAVE_TEMPS' not in os.environ:
             rm_rf(script_caller)
diff --git a/conda/core/index.py b/conda/core/index.py
--- a/conda/core/index.py
+++ b/conda/core/index.py
@@ -15,7 +15,6 @@
 from ..base.context import context
 from ..common.compat import itervalues
 from ..common.io import ThreadLimitedThreadPoolExecutor, time_recorder
-from ..common._os.linux import linux_get_libc_version
 from ..exceptions import ChannelNotAllowed, InvalidSpec
 from ..gateways.logging import initialize_logging
 from ..models.channel import Channel, all_channel_urls


Test patch: