Repository: python-poetry/poetry

Issue:
poetry install --sync attempts to remove itself when poetry is installed to a different venv
- **Poetry version**: 1.4.0, 1.3.2, 1.2.2
- **Python version**: 3.8.2 (the os-installed python)
- **OS version and name**: OSX 10.15.7 build 19H2 (catalina, yes, this is a somewhat old version)
- **pyproject.toml**: See repro script here: https://gist.github.com/novas0x2a/98801bb5e82478f8411d8a0a0362fc53

<!-- All the below steps should be completed before submitting your issue. Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) stable Poetry version, installed using a recommended method. (advanced/manual mode, but yes)
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] I have consulted the [FAQ](https://python-poetry.org/docs/faq/) and [blog](https://python-poetry.org/blog/) for any relevant entries or release notes.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option) and have included the output below.

## Issue

I'm seeing a weird problem where poetry attempts to remove itself when `poetry install --sync` is run. However, unlike other examples of this, poetry is _not_ installed into the venv it's attempting to manage. Here's a [link](https://gist.github.com/novas0x2a/98801bb5e82478f8411d8a0a0362fc53) to my repro script.

Here's me running it (with minor redaction):
```
$ ./repro-poetry.sh test-dir 1.4.0 &>out
$ cat out
+ TESTDIR=test-dir
+ POETRY=1.4.0
+ PYTHON=python3
+ test -e test-dir
+ mkdir -p test-dir
+ setup_poetry test-dir/poetry 1.4.0
+ local venv_path=test-dir/poetry
+ local poetry_version=1.4.0
+ python3 -m venv test-dir/poetry
+ test-dir/poetry/bin/python -m pip install pip==23.0.1 setuptools==67.5.1
Collecting pip==23.0.1
  Using cached https://files.pythonhosted.org/packages/07/51/2c0959c5adf988c44d9e1e0d940f5b074516ecc87e96b1af25f59de9ba38/pip-23.0.1-py3-none-any.whl
Collecting setuptools==67.5.1
  Using cached https://files.pythonhosted.org/packages/99/02/d6ac140d446e939959e80b7e0c55774fa7a22376847ff5fe191674fd1862/setuptools-67.5.1-py3-none-any.whl
Installing collected packages: pip, setuptools
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
      Successfully uninstalled setuptools-41.2.0
Successfully installed pip-23.0.1 setuptools-67.5.1
+ test-dir/poetry/bin/python -m pip install poetry==1.4.0
Collecting poetry==1.4.0
  Using cached poetry-1.4.0-py3-none-any.whl (221 kB)
Collecting pyproject-hooks<2.0.0,>=1.0.0
  Using cached pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Collecting requests<3.0,>=2.18
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting poetry-plugin-export<2.0.0,>=1.3.0
  Using cached poetry_plugin_export-1.3.0-py3-none-any.whl (10 kB)
Collecting trove-classifiers>=2022.5.19
  Using cached trove_classifiers-2023.2.20-py3-none-any.whl (13 kB)
Collecting urllib3<2.0.0,>=1.26.0
  Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
Collecting packaging>=20.4
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting tomli<3.0.0,>=2.0.1
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting xattr<0.11.0,>=0.10.0
  Using cached xattr-0.10.1-cp38-cp38-macosx_10_9_x86_64.whl (15 kB)
Collecting requests-toolbelt<0.11.0,>=0.9.1
  Using cached requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)
Collecting crashtest<0.5.0,>=0.4.1
  Using cached crashtest-0.4.1-py3-none-any.whl (7.6 kB)
Collecting html5lib<2.0,>=1.0
  Using cached html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.9
  Using cached CacheControl-0.12.11-py2.py3-none-any.whl (21 kB)
Collecting dulwich<0.22.0,>=0.21.2
  Using cached dulwich-0.21.3-cp38-cp38-macosx_10_9_x86_64.whl (467 kB)
Collecting poetry-core==1.5.1
  Using cached poetry_core-1.5.1-py3-none-any.whl (465 kB)
Collecting pexpect<5.0.0,>=4.7.0
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting shellingham<2.0,>=1.5
  Using cached shellingham-1.5.0.post1-py2.py3-none-any.whl (9.4 kB)
Collecting jsonschema<5.0.0,>=4.10.0
  Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting cleo<3.0.0,>=2.0.0
  Using cached cleo-2.0.1-py3-none-any.whl (77 kB)
Collecting installer<0.7.0,>=0.6.0
  Using cached installer-0.6.0-py3-none-any.whl (452 kB)
Collecting build<0.11.0,>=0.10.0
  Using cached build-0.10.0-py3-none-any.whl (17 kB)
Collecting virtualenv!=20.4.5,!=20.4.6,<21.0.0,>=20.4.3
  Using cached virtualenv-20.20.0-py3-none-any.whl (8.7 MB)
Collecting platformdirs<3.0.0,>=2.5.2
  Using cached platformdirs-2.6.2-py3-none-any.whl (14 kB)
Collecting tomlkit!=0.11.2,!=0.11.3,<1.0.0,>=0.11.1
  Using cached tomlkit-0.11.6-py3-none-any.whl (35 kB)
Collecting filelock<4.0.0,>=3.8.0
  Using cached filelock-3.9.0-py3-none-any.whl (9.7 kB)
Collecting importlib-metadata>=4.4
  Using cached importlib_metadata-6.0.0-py3-none-any.whl (21 kB)
Collecting lockfile<0.13.0,>=0.12.2
  Using cached lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting pkginfo<2.0.0,>=1.9.4
  Using cached pkginfo-1.9.6-py3-none-any.whl (30 kB)
Collecting keyring<24.0.0,>=23.9.0
  Using cached keyring-23.13.1-py3-none-any.whl (37 kB)
Collecting msgpack>=0.5.2
  Using cached msgpack-1.0.4-cp38-cp38-macosx_10_9_x86_64.whl (73 kB)
Collecting rapidfuzz<3.0.0,>=2.2.0
  Using cached rapidfuzz-2.13.7-cp38-cp38-macosx_10_9_x86_64.whl (1.8 MB)
Collecting webencodings
  Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting six>=1.9
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting zipp>=0.5
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting importlib-resources>=1.4.0
  Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting pkgutil-resolve-name>=1.3.10
  Using cached pkgutil_resolve_name-1.3.10-py3-none-any.whl (4.7 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Using cached pyrsistent-0.19.3-cp38-cp38-macosx_10_9_universal2.whl (82 kB)
Collecting attrs>=17.4.0
  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting jaraco.classes
  Using cached jaraco.classes-3.2.3-py3-none-any.whl (6.0 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp38-cp38-macosx_10_9_x86_64.whl (123 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting distlib<1,>=0.3.6
  Using cached distlib-0.3.6-py2.py3-none-any.whl (468 kB)
Collecting cffi>=1.0
  Using cached cffi-1.15.1-cp38-cp38-macosx_10_9_x86_64.whl (178 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting more-itertools
  Using cached more_itertools-9.1.0-py3-none-any.whl (54 kB)
Installing collected packages: webencodings, trove-classifiers, ptyprocess, msgpack, lockfile, distlib, zipp, urllib3, tomlkit, tomli, six, shellingham, rapidfuzz, pyrsistent, pycparser, poetry-core, platformdirs, pkgutil-resolve-name, pkginfo, pexpect, packaging, more-itertools, installer, idna, filelock, crashtest, charset-normalizer, certifi, attrs, virtualenv, requests, pyproject-hooks, jaraco.classes, importlib-resources, importlib-metadata, html5lib, dulwich, cleo, cffi, xattr, requests-toolbelt, keyring, jsonschema, cachecontrol, build, poetry-plugin-export, poetry
Successfully installed attrs-22.2.0 build-0.10.0 cachecontrol-0.12.11 certifi-2022.12.7 cffi-1.15.1 charset-normalizer-3.1.0 cleo-2.0.1 crashtest-0.4.1 distlib-0.3.6 dulwich-0.21.3 filelock-3.9.0 html5lib-1.1 idna-3.4 importlib-metadata-6.0.0 importlib-resources-5.12.0 installer-0.6.0 jaraco.classes-3.2.3 jsonschema-4.17.3 keyring-23.13.1 lockfile-0.12.2 more-itertools-9.1.0 msgpack-1.0.4 packaging-23.0 pexpect-4.8.0 pkginfo-1.9.6 pkgutil-resolve-name-1.3.10 platformdirs-2.6.2 poetry-1.4.0 poetry-core-1.5.1 poetry-plugin-export-1.3.0 ptyprocess-0.7.0 pycparser-2.21 pyproject-hooks-1.0.0 pyrsistent-0.19.3 rapidfuzz-2.13.7 requests-2.28.2 requests-toolbelt-0.10.1 shellingham-1.5.0.post1 six-1.16.0 tomli-2.0.1 tomlkit-0.11.6 trove-classifiers-2023.2.20 urllib3-1.26.14 virtualenv-20.20.0 webencodings-0.5.1 xattr-0.10.1 zipp-3.15.0
+ setup_project test-dir/project
+ local proj_path=test-dir/project
+ mkdir test-dir/project
+ cat
+ cat
+ cat
+ cd test-dir/project
+ ../poetry/bin/poetry debug info

Poetry
Version: 1.4.0
Python:  3.8.2

Virtualenv
Python:         3.8.2
Implementation: CPython
Path:           NA
Executable:     NA

System
Platform:   darwin
OS:         posix
Python:     3.8.2
Path:       $HOME/test-dir/poetry
Executable: $HOME/test-dir/poetry/bin/python3
+ ../poetry/bin/poetry install -vvv --sync --dry-run
Loading configuration file $HOME/Library/Preferences/pypoetry/config.toml
Loading configuration file $HOME/test-dir/project/poetry.toml
Creating virtualenv test-dir in $HOME/test-dir/project/.venv
Using virtualenv: $HOME/test-dir/project/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 0 updates, 47 removals

  • Removing attrs (22.2.0)
  • Removing build (0.10.0)
  • Removing cachecontrol (0.12.11)
  • Removing certifi (2022.12.7)
  • Removing cffi (1.15.1)
  • Removing charset-normalizer (3.1.0)
  • Removing cleo (2.0.1)
  • Removing crashtest (0.4.1)
  • Removing distlib (0.3.6)
  • Removing dulwich (0.21.3)
  • Removing filelock (3.9.0)
  • Removing html5lib (1.1)
  • Removing idna (3.4)
  • Removing importlib-metadata (6.0.0)
  • Removing importlib-resources (5.12.0)
  • Removing installer (0.6.0)
  • Removing jaraco-classes (3.2.3)
  • Removing jsonschema (4.17.3)
  • Removing keyring (23.13.1)
  • Removing lockfile (0.12.2)
  • Removing more-itertools (9.1.0)
  • Removing msgpack (1.0.4)
  • Removing packaging (23.0)
  • Removing pexpect (4.8.0)
  • Removing pkginfo (1.9.6)
  • Removing pkgutil-resolve-name (1.3.10)
  • Removing platformdirs (2.6.2)
  • Removing poetry (1.4.0)
  • Removing poetry-core (1.5.1)
  • Removing poetry-plugin-export (1.3.0)
  • Removing ptyprocess (0.7.0)
  • Removing pycparser (2.21)
  • Removing pyproject-hooks (1.0.0)
  • Removing pyrsistent (0.19.3)
  • Removing rapidfuzz (2.13.7)
  • Removing requests (2.28.2)
  • Removing requests-toolbelt (0.10.1)
  • Removing shellingham (1.5.0.post1)
  • Removing six (1.16.0)
  • Removing tomli (2.0.1)
  • Removing tomlkit (0.11.6)
  • Removing trove-classifiers (2023.2.20)
  • Removing urllib3 (1.26.14)
  • Removing virtualenv (20.20.0)
  • Removing webencodings (0.5.1)
  • Removing xattr (0.10.1)
  • Removing zipp (3.15.0)
```

As you can see from this, poetry is installed into a completely different venv, and it's trying to remove itself. The value of `virtualenvs.in-project` does not matter (i'm just using it for easier cleanup). As mentioned above, it repros on several poetry releases, so it's obviously something somewhat fundamental. I know the system python on OSX is notoriously bad, but I was hoping could direct me more in debugging. Thanks!


Discussion:
not a lot of help I'm afraid, but I can report that your script does not have that effect for me (on ubuntu 22).  The final `poetry install --sync` just does nothing, as you'd expect in an empty project with no dependencies.

_probably_ not going to be a poetry bug I guess, but I don't think I can help you to debug it to be sure.
I couldn't repro it on any non-osx environments I tried it on; I'm assuming it's something similar to the relocatable binary problems osx has had in the past, except I believe these tested poetry versions have the workarounds already
I've successfully reproduced this with the system python 3.8.2 on osx 11.2.1 20D75 (big sur), so it's definitely not just the ancient 10.15 version
The problem appears to be fixed on with python 3.9.6 on osx 13.2.1 22D68 (ventura).
In the below, poetry is installed into the `meta` venv, and the project venv is `project/.venv`

# Part 1

So, I added a print statement to poetry.utils.env.VirtualEnv:
```
    def __init__(self, path: Path, base: Path | None = None) -> None:
        super().__init__(path, base)

        # If base is None, it probably means this is
        # a virtualenv created from VIRTUAL_ENV.
        # In this case we need to get sys.base_prefix
        # from inside the virtualenv.
        if base is None:
            output = self.run_python_script(GET_BASE_PREFIX)
            assert isinstance(output, str)
            self._base = Path(output.strip())

        print(f"{path}: {self.sys_path}") # ADDED THIS
```

And what I see is:
## working system
```
$ ~/meta/bin/poetry install --dry-run --sync
/home/mlundy/mlundy-test/test/project/.venv: ['/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/mlundy/mlundy-test/test/project/.venv/lib/python3.10/site-packages']
Installing dependencies from lock file
```

## failing system
(note: as above, i've redacted the real HOME path with $HOME)
```
$ ~/mlundy-test/mlundy-test/meta/bin/poetry install --sync --dry-run
$HOME/mlundy-test/mlundy-test/test/project/.venv: ['/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload', '$HOME/mlundy-test/mlundy-test/meta/lib/python3.8/site-packages']
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 51 removals

  • Removing attrs (22.2.0)
  • Removing build (0.10.0)
etc
```

This shows that `VirtualEnv.sys_path` for the project venv is ending up with the meta venv's site-packages (which then, in the installer, results in it thinking poetry is installed in the project venv).

# Part 2
This is particularly strange, because when I run the GET_SYS_PATH code directly using the venv python version, i see:
```
$HOME/mlundy-test/mlundy-test/test/project/.venv/bin/python -I -W ignore -c 'import json; import sys; print(json.dumps(sys.path))'
["/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload", "$HOME/mlundy-test/mlundy-test/test/project/.venv/lib/python3.8/site-packages"]
```
Whereas when I run it with the meta python:
```
$ ~/mlundy-test/mlundy-test/meta/bin/python -I -W ignore -c 'import json; import sys; print(json.dumps(sys.path))'
["/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload", "$HOME/mlundy-test/mlundy-test/meta/lib/python3.8/site-packages"]
```

In other words,  maybe `output = self.run_python_script(GET_SYS_PATH)` appears to be using the wrong python?

# Part 3
I edited VirtualEnv.run as follows:
```
    def run(self, bin: str, *args: str, **kwargs: Any) -> str | int:
        cmd = self.get_command_from_bin(bin) + list(args)
        print(f"{cmd}: {kwargs}")
        ret = self._run(cmd, **kwargs)
        print(f"    RETURN: {ret}")
        return ret
```
And reran:
## working
```
['/home/mlundy/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import sys\n\nif hasattr(sys, "real_prefix"):\n    print(sys.real_prefix)\nelif hasattr(sys, "base_prefix"):\n    print(sys.base_prefix)\nelse:\n    print(sys.prefix)\n', 'stderr': -1}
    RETURN: /usr

['/home/mlundy/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import json\nimport sys\n\nprint(json.dumps(sys.path))\n', 'stderr': -1}
    RETURN: ["/usr/lib/python310.zip", "/usr/lib/python3.10", "/usr/lib/python3.10/lib-dynload", "/home/mlundy/mlundy-test/test/project/.venv/lib/python3.10/site-packages"]
...
```

## failing
```
['$HOME/mlundy-test/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import sys\n\nif hasattr(sys, "real_prefix"):\n    print(sys.real_prefix)\nelif hasattr(sys, "base_prefix"):\n    print(sys.base_prefix)\nelse:\n    print(sys.prefix)\n', 'stderr': -1}
    RETURN: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8

['$HOME/mlundy-test/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import json\nimport sys\n\nprint(json.dumps(sys.path))\n', 'stderr': -1}
    RETURN: ["/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload", "$HOME/mlundy-test/mlundy-test/meta/lib/python3.8/site-packages"]
```

So, it's running with the correct python but somehow ending up with the wrong path, and running exactly the same command in part 2 and part 3 gets different answers.

# Part 4
The environment, maybe?
I added
```
        for k, v in sorted(os.environ.items()):
            print(f"{k}={v}")
```
to VirtualEnv.run on the failing machine.

It's difficult to paste the outputs directly (lots of redaction required) but I compared the env debug outputs of the script with `env | sort` from the shell around the command in Part 2, and the only difference was:
```
$ diff -u env-before out 
--- sys-path-python-directly  2023-03-07 16:36:49.000000000 -0800
+++ sys-path-from-poetry 2023-03-07 16:38:51.000000000 -0800
@@ -3,6 +3,7 @@
 LOGNAME=<redacted>
 OLDPWD=$HOME/mlundy-test/mlundy-test/test
 PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
+PLAT=macosx-10.14.6-x86_64
 PS1=\h:\W \u\$ 
 PWD=$HOME/mlundy-test/mlundy-test/test/project
 SHELL=/bin/bash
@@ -13,4 +14,6 @@
 TERM=xterm-256color
 TMPDIR=<redacted>
 USER=<redacted>
-_=/usr/bin/env
+_=$HOME/mlundy-test/mlundy-test/meta/bin/poetry
+__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
+__PYVENV_LAUNCHER__=$HOME/mlundy-test/mlundy-test/meta/bin/python3
```
Oh, a smoking gun. That environment variable led me to https://github.com/tox-dev/tox/issues/148
I added `os.environ.pop("__PYVENV_LAUNCHER__", None)` to VirtualEnv.run and... it worked!

So, I'm not sure what to do with this information because I'm not quite sure what this env var is for, other than everyone seems to remove it:
- https://github.com/tox-dev/tox/pull/1452
- https://virtualenv.pypa.io/en/20.0.33/changelog.html#features-20-0-14
- https://github.com/python/cpython/pull/9516

Obviously, this problem is fixed in later python versions, but, would you consider putting a workaround for this into poetry as well, since the affected OSX versions are still in the wild?

Resolving patch:
diff --git a/src/poetry/utils/env.py b/src/poetry/utils/env.py
--- a/src/poetry/utils/env.py
+++ b/src/poetry/utils/env.py
@@ -1533,6 +1533,7 @@ def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
                     stderr=stderr,
                     input=encode(input_),
                     check=True,
+                    env=env,
                     **kwargs,
                 ).stdout
             elif call:


Test patch: