Repository: pypa/pipenv

Issue:
Test suite immediately aborts with a successful exit code whenever a test that does a `pipenv run` executes
Check this out:

```python
$ time pytest
=============================================== test session starts ================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/jeremy/src/github.com/pypa/pipenv, configfile: setup.cfg, testpaths: tests
plugins: timeout-2.0.1, xdist-2.4.0, flaky-3.7.0, forked-1.3.0, pypi-0.1.1
collected 274 items

tests/integration/test_cli.py ...
real	7.14s
user	4.30s
sys	0.73s
$ echo $?
0
```

I know we don't have a 7 second test suite with only 3 tests in it! Also, where did pipenv's normal test summary go?

In the example above, we've run the first 3 tests of test_cli.py and the test suite aborted when running the 4th test: `test_pipenv_site_packages`. The whole test suite aborts when we get to this `pipenv run` invocation:

https://github.com/pypa/pipenv/blob/b0ebaf054e9b47fc90f6939b2473e31863da15a3/tests/integration/test_cli.py#L52

This doesn't happen in CI. It took me **forever** to figure out what's going on here. Here's the full story:

- That call to `p.pipenv` uses `click`'s [`invoke` method](https://click.palletsprojects.com/en/8.0.x/api/#click.testing.CliRunner.invoke) internally. That method internally starts executing pipenv (note that this is happening in the same python testrunner process, there is no forking going on here!)
- Eventually we get into the actual guts of `pipenv run`, which (on my Linux machine) means we execute [this call to `os.execve` in `do_run_posix`](https://github.com/pypa/pipenv/blob/b0ebaf054e9b47fc90f6939b2473e31863da15a3/pipenv/core.py#L2451-L2455). *This completely replaces the pytest process we were in the middle of*, and when the (now just pipenv) process exits, our entire test suite grinds to a halt and exits successfully. Yikes :exploding_head:!
- This doesn't happen in CI because we actually use `do_run_nt` (:interrobang:) in CI: https://github.com/pypa/pipenv/blob/b0ebaf054e9b47fc90f6939b2473e31863da15a3/pipenv/core.py#L2499-L2502. It looks like @frostming added this in https://github.com/pypa/pipenv/pull/4757/files#diff-ef852c4ac364f946819f765a6bc26f04f1b0968f31fc512949a60fa2ab0685e8R2488, presumably as a workaround for this very problem?
- (For the record, if you're running pytest with parallelism enabled, you'll see error messages like "node down: Not properly terminated" and "replacing crashed worker gw3" as the workers turn into pipenv processes and exit).

I've been able to work around this problem by explicitly setting the `CI` environment variable when running tests locally. I'll send in a PR to improve the documentation, but we might want to rework this a bit. It feels a bit off to require people to pretend to be in CI just to run tests.


Discussion:
@jfly I haven't experienced this issue before in the last months of running pipenv tests locally on almost daily basis.   Could you re-check this behavior with `pipenv==2022.4.21`? 
@matteius that's because of [this workaround I added](https://github.com/pypa/pipenv/blob/114eb8f80a63e519fe9d12ac9e7cb60444cf47c0/tests/conftest.py#L4-L11). If you comment it out, this reproduces quickly:

```
$ git diff
diff --git a/tests/conftest.py b/tests/conftest.py
index 13778403..1dbfcfab 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,14 +1,14 @@
 import pytest
-import os
-
-# Note that we have to do this *before* `pipenv.environments` gets imported,
-# which is why we're doing it here as a side effect of importing this module.
-# CI=1 is necessary as a workaround for https://github.com/pypa/pipenv/issues/4909
-os.environ['CI'] = '1'
-
-def pytest_sessionstart(session):
-    import pipenv.environments
-    assert pipenv.environments.PIPENV_IS_CI
+# <<< import os
+# <<<
+# <<< # Note that we have to do this *before* `pipenv.environments` gets imported,
+# <<< # which is why we're doing it here as a side effect of importing this module.
+# <<< # CI=1 is necessary as a workaround for https://github.com/pypa/pipenv/issues/4909
+# <<< os.environ['CI'] = '1'
+# <<<
+# <<< def pytest_sessionstart(session):
+# <<<     import pipenv.environments
+# <<<     assert pipenv.environments.PIPENV_IS_CI
 
 
 @pytest.fixture()
$ time pytest
===================================================================== test session starts =====================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/jeremy/src/github.com/pypa/pipenv, configfile: setup.cfg, testpaths: tests
plugins: timeout-2.0.1, xdist-2.4.0, flaky-3.7.0, forked-1.3.0, pypi-0.1.1
collected 294 items

tests/integration/test_cli.py ...
real	9.78s
user	6.02s
sys	0.74s
```
@jfly sorry for being dense but is this still an issue?
@matteius sorry for the delayed response here. I just checked, and yes, this is still an issue. I've filed this SO question seeking help from the click folks about how to handle this: https://stackoverflow.com/questions/73995157/best-practice-for-invoke-ing-a-tool-that-internally-calls-exec.
> @matteius that's because of [this workaround I added](https://github.com/pypa/pipenv/blob/114eb8f80a63e519fe9d12ac9e7cb60444cf47c0/tests/conftest.py#L4-L11). If you comment it out, this reproduces quickly:
> 

Hi! 
This workaround breaks process substitution in CI.  

https://github.com/pypa/pipenv/blob/main/pipenv/routines/shell.py#L102

we have a script that runs something like 
`pipenv run ./python_cli --file <(echo $FILE_CONTENT)` 
this works fine on local machines, but it breaks on CI with: ` Path '/dev/fd/63' does not exist.` 

this is because when the $CI env var is set, run uses `do_run_nt`  instead of `do_run_posix`.  `do_run_nt` uses a subprocess, so the temp file descriptor created with `<(..)`  is not accesible when it runs. 

Urg, sorry @scvsoft-javierrotelli, that must have been a pain to track down.

One solution would be to change that workaround to look at a new environment variable (`PIPENV_FORCE_RUN_NT`?) that we explicitly set in Pipenv's CI config.
Its been a while on this thread but I am wondering how relevant that pipenv logic for nt is still, but it might be -- anyway I am pretty busy at the moment but could accept a PR and cut a minor release sometime soon.
a new env variable seems like a good solution, I can work on a PR between today and tomorrow if that's fine

Resolving patch:
diff --git a/pipenv/routines/shell.py b/pipenv/routines/shell.py
--- a/pipenv/routines/shell.py
+++ b/pipenv/routines/shell.py
@@ -3,7 +3,6 @@
 import sys
 from os.path import expandvars
 
-from pipenv import environments
 from pipenv.utils.project import ensure_project
 from pipenv.utils.shell import cmd_list_to_shell, system_which
 from pipenv.vendor import click
@@ -99,9 +98,7 @@ def do_run(project, command, args, python=False, pypi_mirror=None):
         click.echo("Can't run script {0!r}-it's empty?", err=True)
     run_args = [project, script]
     run_kwargs = {"env": env}
-    # We're using `do_run_nt` on CI (even if we're running on a non-nt machine)
-    # as a workaround for https://github.com/pypa/pipenv/issues/4909.
-    if os.name == "nt" or environments.PIPENV_IS_CI:
+    if os.name == "nt":
         run_fn = do_run_nt
     else:
         run_fn = do_run_posix


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,14 +1,4 @@
 import pytest
-import os
-
-# Note that we have to do this *before* `pipenv.environments` gets imported,
-# which is why we're doing it here as a side effect of importing this module.
-# CI=1 is necessary as a workaround for https://github.com/pypa/pipenv/issues/4909
-os.environ['CI'] = '1'
-
-def pytest_sessionstart(session):
-    import pipenv.environments
-    assert pipenv.environments.PIPENV_IS_CI
 
 
 @pytest.fixture()
diff --git a/tests/integration/test_run.py b/tests/integration/test_run.py
--- a/tests/integration/test_run.py
+++ b/tests/integration/test_run.py
@@ -41,8 +41,8 @@ def test_scripts(pipenv_instance_pypi):
         c = p.pipenv('run notfoundscript')
         assert c.returncode != 0
         assert c.stdout == ''
-        if os.name != 'nt':     # TODO: Implement this message for Windows.
-            assert 'not found' in c.stderr
+        if os.name != 'nt':
+            assert 'could not be found' in c.stderr
 
         project = Project()