Repository: python-poetry/poetry

Issue:
All dependencies are always updated
- [x] I am on the [latest](https://github.com/sdispater/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/sdispater/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option). [N/A]

- **OS version and name**: macOS Catalina (10.15.1)
- **Poetry version**: 1.0b4, 1.0b5, 1.0b6
- **Link of a [Gist](https://gist.github.com/) with the contents of your pyproject.toml file**: https://gist.github.com/wylee/53e74a3413df25e1b594803e193d40d4

## Issue

Since 1.0b4, when running `poetry update`, all of my dependencies are always updated. I've tried clearing the cache and deleting `poetry.lock`.

#1582 is similar but seems to affect only project that use a private package index.





Discussion:
Hello @wylee ,

thanks for reporting.

Could you please try to describe a step-by-step instruction to reproduce the issue? I wasn't able to do this until now.

fin swimmer

I started with Poetry 0.12.17 then upgraded to 1.0b3. This issue wasn't present on either of those versions. Then I upgraded to 1.0b4 and this started happening. I figured it was a temporary issue, so I waited for a couple more beta releases (b5 and b6) to come out. I'm now on 1.0b6, and I'm still seeing this behavior.

This is the output for the `pyproject.toml` I linked to above every time I run `poetry install`:

```
Installing dependencies from lock file


Package operations: 0 installs, 4 updates, 0 removals

  - Updating markupsafe (1.1.1 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.1.1)
  - Updating jinja2 (2.10.3 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 2.10.3)
  - Updating pyyaml (5.1.2 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 5.1.2)
  - Updating runcommands (1.0a44 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.0a44)
```

And every time I run `poetry update`:

```
Updating dependencies
Resolving dependencies... (0.2s)


Package operations: 0 installs, 4 updates, 0 removals

  - Updating markupsafe (1.1.1 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.1.1)
  - Updating jinja2 (2.10.3 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 2.10.3)
  - Updating pyyaml (5.1.2 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 5.1.2)
  - Updating runcommands (1.0a44 /home/me/Projects/bycycle/bycycle.webui/.venv/lib/python3.8/site-packages -> 1.0a44)
```

I just tried deleting my project's virtualenv and then re-running `poetry install` and then `poetry update`. With a fresh virtualenv (created by poetry), this issue isn't present, so it appears to somehow be caused by updating poetry and using it with an existing virtualenv.
So, I was able to reproduce this in the following way:

- Uninstall poetry
- Install poetry 0.12.7
- `cd project`
- Remove project virtualenv
- `poetry install && poetry update` (no issue here)
- Upgrade poetry to 1.0b3
- `poetry install && poetry update` (no issue here)
- Upgrade poetry to 1.0b4
- `poetry install && poetry update` (now dependencies are upgraded every time)
- Remove project virtualenv
- `poetry install && poetry update` (issue is gone)

One thing I noticed here is that starting with 1.0b4 virtualenvs are created in ~/Library/Caches/pypoetry/virtualenvs even though my poetry config looks like this:

```
[settings]
[settings.virtualenvs]
in-project = true
path = ".venv"
```

Edit: It appears the config format changed slightly in 1.0b3, but that doesn't seem to be relevant to this issue.



I stumbled across another way this can happen with 1.0b4, 1.0b5, and 1.0b6:

- Start fresh by removing the project virtualenv and poetry.lock and making sure the virtualenv is deactivated
- Run `poetry install`
- Run `poetry update` (dependencies are not upgraded)
- Activate the project's virtualenv
- Run `poetry update` (now dependencies are upgraded every time)
- Deactivate the project's virtualenv (or manually unset the `VIRTUAL_ENV` environment variable)
- Run `poetry update` (dependencies are not upgraded)

Thanks the detailed steps to reproduce. Unfortunately I am still unable to reproduce.

What's the output of `poetry debug info` and how did you install Poetry?
I installed poetry using pipx.

Output of `poetry debug info`:

```
> poetry debug info

Poetry
Version: 1.0.0b6
Python:  3.8.0

Virtualenv
Python:         3.6.9
Implementation: CPython
Path:           /home/me/Projects/bycycle/bycycle.webui
Valid:          False

System
Platform: darwin
OS:       posix
Python:   /usr/local/var/pyenv/versions/3.6.9
```

Based on that, I think I figured out what's happening. When `VIRTUAL_ENV` doesn't match what poetry is expecting, updates always happen because it considers the virtualenv invalid. In my development environment, I don't use `. .venv/bin/activate`. Instead, I have a custom script for setting `$PATH` and `$VIRTUAL_ENV` when I `cd` into a project directory, and it was setting `$VIRTUAL_ENV` incorrectly.

So, that's my mistake (and sorry for the noise), but I do think it might be good if poetry showed at least a warning, or even aborted, in the case of an invalid virtualenv.
I am also getting this in versions 1.0.0b4-1.0.0b7; in my case, at least, it appears to be related to pip installing dependencies in a location poetry isn't expecting (e.g., if they are pip installed with the `--user` flag).

When this is the case, it seems like poetry is running some pip-related command in the background (maybe?), but it doesn't result in any changes, which is why it always installs them all.

I'll see if I can make an easily reproduced docker image.
I experience this issue when using an in-project virtualenv (.venv/). Similarly to the others, I did not experience the issue on earlier beta versions. The problem is present on Poetry version 1.0.0b8.

* I installed poetry using the recommended `curl` approach.
* System: Ubuntu 18.04
@pappasam What's the output of `poetry debug info`?
@sdispater hmm, strange, but I've re-run in the same project and no longer have the problem. Not sure how to replicate, I apologize:

```
Poetry
Version: 1.0.0b8
Python:  3.8.0

Virtualenv
Python:         3.8.0
Implementation: CPython
Path:           /home/sroeca/src/Personal/serdataclasses/.venv
Valid:          True

System
Platform: linux
OS:       posix
Python:   /home/sroeca/.asdf/installs/python/3.8.0
```
I too am seeing this problem having upgraded from version 0.12.17 to 1.0.0b8.

```
$ poetry debug info

Poetry
Version: 1.0.0b8
Python:  3.7.5

Virtualenv
Python:         3.7.5
Implementation: CPython
Path:           /Users/XXXXX/.virtualenvs/hatch-api
Valid:          True

System
Platform: darwin
OS:       posix
Python:   /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7
```

Doing a `poetry update --lock` seems to fix the issue, interestingly.

Resolving patch:
diff --git a/poetry/installation/pip_installer.py b/poetry/installation/pip_installer.py
--- a/poetry/installation/pip_installer.py
+++ b/poetry/installation/pip_installer.py
@@ -96,7 +96,12 @@ def install(self, package, update=False):
 
             self.run(*args)
 
-    def update(self, _, target):
+    def update(self, package, target):
+        if package.source_type != target.source_type:
+            # If the source type has changed, we remove the current
+            # package to avoid perpetual updates in some cases
+            self.remove(package)
+
         self.install(target, update=True)
 
     def remove(self, package):
diff --git a/poetry/repositories/installed_repository.py b/poetry/repositories/installed_repository.py
--- a/poetry/repositories/installed_repository.py
+++ b/poetry/repositories/installed_repository.py
@@ -15,45 +15,52 @@ def load(cls, env):  # type: (Env) -> InstalledRepository
         For now, it uses the pip "freeze" command.
         """
         repo = cls()
+        seen = set()
 
-        for distribution in sorted(
-            metadata.distributions(path=env.sys_path), key=lambda d: str(d._path),
-        ):
-            name = distribution.metadata["name"]
-            version = distribution.metadata["version"]
-            package = Package(name, version, version)
-            package.description = distribution.metadata.get("summary", "")
-
-            repo.add_package(package)
-
-            path = Path(str(distribution._path))
-            is_standard_package = True
-            try:
-                path.relative_to(env.site_packages)
-            except ValueError:
-                is_standard_package = False
-
-            if is_standard_package:
-                continue
-
-            src_path = env.path / "src"
-
-            # A VCS dependency should have been installed
-            # in the src directory. If not, it's a path dependency
-            try:
-                path.relative_to(src_path)
-
-                from poetry.vcs.git import Git
-
-                git = Git()
-                revision = git.rev_parse("HEAD", src_path / package.name).strip()
-                url = git.remote_url(src_path / package.name)
-
-                package.source_type = "git"
-                package.source_url = url
-                package.source_reference = revision
-            except ValueError:
-                package.source_type = "directory"
-                package.source_url = str(path.parent)
+        for entry in env.sys_path:
+            for distribution in sorted(
+                metadata.distributions(path=[entry]), key=lambda d: str(d._path),
+            ):
+                name = distribution.metadata["name"]
+                version = distribution.metadata["version"]
+                package = Package(name, version, version)
+                package.description = distribution.metadata.get("summary", "")
+
+                if package.name in seen:
+                    continue
+
+                seen.add(package.name)
+
+                repo.add_package(package)
+
+                path = Path(str(distribution._path))
+                is_standard_package = True
+                try:
+                    path.relative_to(env.site_packages)
+                except ValueError:
+                    is_standard_package = False
+
+                if is_standard_package:
+                    continue
+
+                src_path = env.path / "src"
+
+                # A VCS dependency should have been installed
+                # in the src directory. If not, it's a path dependency
+                try:
+                    path.relative_to(src_path)
+
+                    from poetry.vcs.git import Git
+
+                    git = Git()
+                    revision = git.rev_parse("HEAD", src_path / package.name).strip()
+                    url = git.remote_url(src_path / package.name)
+
+                    package.source_type = "git"
+                    package.source_url = url
+                    package.source_reference = revision
+                except ValueError:
+                    package.source_type = "directory"
+                    package.source_url = str(path.parent)
 
         return repo
diff --git a/poetry/utils/env.py b/poetry/utils/env.py
--- a/poetry/utils/env.py
+++ b/poetry/utils/env.py
@@ -703,6 +703,7 @@ def __init__(self, path, base=None):  # type: (Path, Optional[Path]) -> None
 
         self._marker_env = None
         self._pip_version = None
+        self._site_packages = None
 
     @property
     def path(self):  # type: () -> Path
@@ -758,20 +759,25 @@ def pip_version(self):
 
     @property
     def site_packages(self):  # type: () -> Path
-        # It seems that PyPy3 virtual environments
-        # have their site-packages directory at the root
-        if self._path.joinpath("site-packages").exists():
-            return self._path.joinpath("site-packages")
-
-        if self._is_windows:
-            return self._path / "Lib" / "site-packages"
-
-        return (
-            self._path
-            / "lib"
-            / "python{}.{}".format(*self.version_info[:2])
-            / "site-packages"
-        )
+        if self._site_packages is None:
+            site_packages = []
+            dist_packages = []
+            for entry in self.sys_path:
+                entry = Path(entry)
+                if entry.name == "site-packages":
+                    site_packages.append(entry)
+                elif entry.name == "dist-packages":
+                    dist_packages.append(entry)
+
+            if not site_packages and not dist_packages:
+                raise RuntimeError("Unable to find the site-packages directory")
+
+            if site_packages:
+                self._site_packages = site_packages[0]
+            else:
+                self._site_packages = dist_packages[0]
+
+        return self._site_packages
 
     @property
     def sys_path(self):  # type: () -> List[str]
@@ -1114,6 +1120,7 @@ def __init__(
         os_name="posix",
         is_venv=False,
         pip_version="19.1",
+        sys_path=None,
         **kwargs
     ):
         super(MockEnv, self).__init__(**kwargs)
@@ -1124,6 +1131,7 @@ def __init__(
         self._os_name = os_name
         self._is_venv = is_venv
         self._pip_version = Version.parse(pip_version)
+        self._sys_path = sys_path
 
     @property
     def version_info(self):  # type: () -> Tuple[int]
@@ -1145,5 +1153,12 @@ def os(self):  # type: () -> str
     def pip_version(self):
         return self._pip_version
 
+    @property
+    def sys_path(self):
+        if self._sys_path is None:
+            return super(MockEnv, self).sys_path
+
+        return self._sys_path
+
     def is_venv(self):  # type: () -> bool
         return self._is_venv


Test patch:
diff --git a/tests/masonry/builders/test_editable.py b/tests/masonry/builders/test_editable.py
--- a/tests/masonry/builders/test_editable.py
+++ b/tests/masonry/builders/test_editable.py
@@ -17,8 +17,7 @@
 def test_build_should_delegate_to_pip_for_non_pure_python_packages(tmp_dir, mocker):
     move = mocker.patch("shutil.move")
     tmp_dir = Path(tmp_dir)
-    env = MockEnv(path=tmp_dir, pip_version="18.1", execute=False)
-    env.site_packages.mkdir(parents=True)
+    env = MockEnv(path=tmp_dir, pip_version="18.1", execute=False, sys_path=[])
     module_path = fixtures_dir / "extended"
 
     builder = EditableBuilder(Factory().create_poetry(module_path), env, NullIO())
@@ -33,8 +32,7 @@ def test_build_should_delegate_to_pip_for_non_pure_python_packages(tmp_dir, mock
 def test_build_should_temporarily_remove_the_pyproject_file(tmp_dir, mocker):
     move = mocker.patch("shutil.move")
     tmp_dir = Path(tmp_dir)
-    env = MockEnv(path=tmp_dir, pip_version="19.1", execute=False)
-    env.site_packages.mkdir(parents=True)
+    env = MockEnv(path=tmp_dir, pip_version="19.1", execute=False, sys_path=[])
     module_path = fixtures_dir / "extended"
 
     builder = EditableBuilder(Factory().create_poetry(module_path), env, NullIO())
diff --git a/tests/utils/test_env.py b/tests/utils/test_env.py
--- a/tests/utils/test_env.py
+++ b/tests/utils/test_env.py
@@ -31,6 +31,20 @@
 """
 
 
+class MockVirtualEnv(VirtualEnv):
+    def __init__(self, path, base=None, sys_path=None):
+        super(MockVirtualEnv, self).__init__(path, base=base)
+
+        self._sys_path = sys_path
+
+    @property
+    def sys_path(self):
+        if self._sys_path is not None:
+            return self._sys_path
+
+        return super(MockVirtualEnv, self).sys_path
+
+
 @pytest.fixture()
 def poetry(config):
     poetry = Factory().create_poetry(
@@ -786,7 +800,7 @@ def test_env_site_packages_should_find_the_site_packages_directory_if_standard(t
 
     site_packages.mkdir(parents=True)
 
-    env = VirtualEnv(Path(tmp_dir), Path(tmp_dir))
+    env = MockVirtualEnv(Path(tmp_dir), Path(tmp_dir), sys_path=[str(site_packages)])
 
     assert site_packages == env.site_packages
 
@@ -795,6 +809,37 @@ def test_env_site_packages_should_find_the_site_packages_directory_if_root(tmp_d
     site_packages = Path(tmp_dir).joinpath("site-packages")
     site_packages.mkdir(parents=True)
 
-    env = VirtualEnv(Path(tmp_dir), Path(tmp_dir))
+    env = MockVirtualEnv(Path(tmp_dir), Path(tmp_dir), sys_path=[str(site_packages)])
+
+    assert site_packages == env.site_packages
+
+
+def test_env_site_packages_should_find_the_dist_packages_directory_if_necessary(
+    tmp_dir,
+):
+    site_packages = Path(tmp_dir).joinpath("dist-packages")
+    site_packages.mkdir(parents=True)
+
+    env = MockVirtualEnv(Path(tmp_dir), Path(tmp_dir), sys_path=[str(site_packages)])
 
     assert site_packages == env.site_packages
+
+
+def test_env_site_packages_should_prefer_site_packages_over_dist_packages(tmp_dir):
+    dist_packages = Path(tmp_dir).joinpath("dist-packages")
+    dist_packages.mkdir(parents=True)
+    site_packages = Path(tmp_dir).joinpath("site-packages")
+    site_packages.mkdir(parents=True)
+
+    env = MockVirtualEnv(
+        Path(tmp_dir), Path(tmp_dir), sys_path=[str(dist_packages), str(site_packages)]
+    )
+
+    assert site_packages == env.site_packages
+
+
+def test_env_site_packages_should_raise_an_error_if_no_site_packages(tmp_dir):
+    env = MockVirtualEnv(Path(tmp_dir), Path(tmp_dir), sys_path=[])
+
+    with pytest.raises(RuntimeError):
+        env.site_packages