Repository: iterative/dvc

Issue:
dvc run unexpectedly modifies PATH before running commands
Steps to reproduce:
1. `pyenv install miniconda3-latest`
2. `pyenv shell miniconda3-latest`
3. `conda install dvc==0.59.2`
4. `conda create -n testenv python=3.7`
6. `mkdir dvc-test && cd dvc-test`
5.  `dvc init && dvc run -f tmp.dvc 'echo $PATH'`

Expected output:
* Conda env `/bin`
* Pyenv shim `/bin`
* Conda base `/bin`
* Pyenv shim `/bin`
* etc

Actual output:
* Conda base `/bin`
* Pyenv shim `/bin`
* Conda env `/bin`
* Pyenv shim `/bin`
* Conda base `/bin`
* Pyenv shim `/bin`
* etc

Somehow part of the `PATH` from wherever DVC is installed is getting prepended to `PATH` before `dvc run`-ing something. This potentially breaks code in the project.

Note that installing and running DVC with Pipx instead of Conda does *not* result in the same problem. The adverse interaction appears to be Conda-specific (although there are many combinations of environments I haven't yet tried). This particular Pipx installation is not being managed by Conda.

Discord conversation: https://discordapp.com/channels/485586884165107732/485596304961962003/623513705145040919


Comments:
It's happening to other users as well (discussion with Benjamin on Discord). Symptoms are very similar - zsh (a regular one), PATH is modified. Not `conda`, virtualenv is being used. OS - Mac.
@shcheklein in their case, is DVC installed in the "parent" Python environment, or in a separate virtualenv? The problem might lie in something that has to do with the parent/child relationship.

Also, did they use Homebrew to install Python? Brew is another common factor here, and more likely to cause problems than Zsh, since Brew does its own layer of symlinking.

My admittedly convoluted setup:
```
Linuxbrew
├─ Pyenv
│  └─ Conda                        <- PATH is broken when DVC is installed here
│     └─ Active conda environment  <- PATH is OK when DVC is installed here
└─ Python
   └─ Pipx-managed Virtualenv      <- PATH is OK when DVC is installed here
```
@gwerbin Thanks! I've asked Benjamin to take a look and share more details. 

> @shcheklein in their case, is DVC installed in the "parent" Python environment, or in a separate virtualenv? The problem might lie in something that has to do with the parent/child relationship.

I have tried two setups, and both fail the sense that:

1.  The error `ImportError: No module named pandas` is returned.
2.  `dvc run -o test 'which python > test'` outputs `/usr/local/bin/python` in the `test` file, where it should point to `python` in the virtualenv.

Setup 1
> ```
> Homebrew
>    └─ DVC (`/usr/local/bin/dvc`)
>    └─ Virtualenv + Python (`/usr/local/bin/{python,virtualenv}`)
>              └─Active virtualenv environment
>                        └─  Pandas
> ```

Setup 2
> ```
> Homebrew
>    └─ Virtualenv + Python (`/usr/local/bin/{python,virtualenv}`)
>              └─Active virtualenv environment
>                        └─  DVC
>                        └─  Pandas
> ```

> Also, did they use Homebrew to install Python? Brew is another common factor here, and more likely to cause problems than Zsh, since Brew does its own layer of symlinking.

Yes, Python was installed by Homebrew. (FYI: the Python interpreter that comes with the latest version of macOS (Mojave, version 10.14.6) is 2.7.10 and is 4.5 years old. I figure most people using Python on macOS will have shadowed this outdated version with a more recent one.)

@shcheklein and @efiop asked me to share the output of a few commands on the Discord channels and perhaps it helps if I share it here as well.

```
> echo $SHELL
> dvc run -f test.dvc 'echo $SHELL'
> ls -la $SHELL
> file $SHELL
/bin/zsh
'test.dvc' already exists. Do you wish to run the command and overwrite it? [y/n] y
Running command:
    echo $SHELL
/bin/zsh
Saving information to 'test.dvc'.

To track the changes with git, run:

    git add test.dvc
-rwxr-xr-x 1 root wheel 610240 May  4 09:05 /bin/zsh
/bin/zsh: Mach-O 64-bit executable x86_64
> cat test.dvc
cmd: echo $SHELL
md5: ee3b44e50705d557b7aa3eef74821f74
```

I wish I could help out more, but my knowledge of Python environments and DVC internals is very limited. However, let me know if I can help you out with further information and I'm happy to provide it.
For the record: I am able to reproduce https://github.com/iterative/dvc/issues/2506#issue-494639954 even on the linux machine.

In my case `which dvc` shows pyenv shim, which has something like:
```
exec "/home/efiop/.pyenv/libexec/pyenv" exec "$program" "$@"
```
in it, which is the thing that adds some stuff on top of the base env, as we can see:
```
➜  dvc-test git:(755) ✗ /home/efiop/.pyenv/libexec/pyenv exec --help
Usage: pyenv exec <command> [arg1 arg2...]

Runs an executable by first preparing PATH so that the selected Python
version's `bin' directory is at the front.

For example, if the currently selected Python version is 2.7.6:
  pyenv exec pip install -rrequirements.txt

is equivalent to:
  PATH="$PYENV_ROOT/versions/2.7.6/bin:$PATH" pip install -rrequirements.txt
```

It would be nice of pyenv would've left something like `PATH_ORIG` env var, so that we could use it later. This would be similar to how pyinstaller leaves VAR_ORIG if it changes it, e.g. LD_LIBRARY_PATH. Looking for possible and viable automatic workarounds. Might have to suggest this to pyenv later though, to make it straightforward for everyone.
Interesting detail: our good friend @AlJohri has run into it before even using dvc: https://github.com/pyenv/pyenv/issues/985 🙂 

Patch:
diff --git a/dvc/utils/__init__.py b/dvc/utils/__init__.py
--- a/dvc/utils/__init__.py
+++ b/dvc/utils/__init__.py
@@ -245,9 +245,11 @@ def is_binary():
     return getattr(sys, "frozen", False)
 
 
-# NOTE: Fix env variables modified by PyInstaller
-# http://pyinstaller.readthedocs.io/en/stable/runtime-information.html
 def fix_env(env=None):
+    """Fix env variables modified by PyInstaller [1] and pyenv [2].
+    [1] http://pyinstaller.readthedocs.io/en/stable/runtime-information.html
+    [2] https://github.com/pyenv/pyenv/issues/985
+    """
     if env is None:
         env = os.environ.copy()
     else:
@@ -262,6 +264,32 @@ def fix_env(env=None):
         else:
             env.pop(lp_key, None)
 
+    # Unlike PyInstaller, pyenv doesn't leave backups of original env vars
+    # when it modifies them. If we look into the shim, pyenv and pyenv-exec,
+    # we can figure out that the PATH is modified like this:
+    #
+    #     PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH
+    #
+    # where
+    #
+    #     PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
+    #     bin_path - might not start with $PYENV_ROOT as it runs realpath on
+    #         it, see pyenv source code.
+    #     plugin_bin - might contain more than 1 entry, which start with
+    #         $PYENV_ROOT, see pyenv source code.
+    #
+    # So having this, we can make a rightful assumption about what parts of the
+    # PATH we need to remove in order to get the original PATH.
+    path = env.get("PATH")
+    pyenv_root = env.get("PYENV_ROOT")
+    if path and pyenv_root and path.startswith(pyenv_root):
+        # removing PYENV_BIN_PATH and bin_path
+        parts = path.split(":")[2:]
+        # removing plugin_bin from the left
+        while pyenv_root in parts[0]:
+            del parts[0]
+        env["PATH"] = ":".join(parts)
+
     return env
 
 


Test patch:
diff --git a/tests/unit/utils/test_utils.py b/tests/unit/utils/test_utils.py
--- a/tests/unit/utils/test_utils.py
+++ b/tests/unit/utils/test_utils.py
@@ -1,6 +1,6 @@
 import pytest
 
-from dvc.utils import to_chunks
+from dvc.utils import to_chunks, fix_env
 
 
 @pytest.mark.parametrize(
@@ -28,3 +28,30 @@ def test_to_chunks_num_chunks(num_chunks, expected_chunks):
     list_to_chunk = [1, 2, 3, 4]
     result = to_chunks(list_to_chunk, num_chunks=num_chunks)
     assert result == expected_chunks
+
+
+@pytest.mark.parametrize(
+    "path, orig",
+    [
+        (
+            "/pyenv/bin:/pyenv/libexec:/pyenv/hook:/orig/path1:/orig/path2",
+            "/orig/path1:/orig/path2",
+        ),
+        (
+            "/pyenv/bin:/pyenv/libexec:/orig/path1:/orig/path2",
+            "/orig/path1:/orig/path2",
+        ),
+        (
+            "/pyenv/bin:/some/libexec:/pyenv/hook:/orig/path1:/orig/path2",
+            "/orig/path1:/orig/path2",
+        ),
+        ("/orig/path1:/orig/path2", "/orig/path1:/orig/path2"),
+        (
+            "/orig/path1:/orig/path2:/pyenv/bin:/pyenv/libexec",
+            "/orig/path1:/orig/path2:/pyenv/bin:/pyenv/libexec",
+        ),
+    ],
+)
+def test_fix_env_pyenv(path, orig):
+    env = {"PATH": path, "PYENV_ROOT": "/pyenv"}
+    assert fix_env(env)["PATH"] == orig