Repository: python-poetry/poetry

Issue:
How do I uninstall packages that have been removed from pyproject.toml?
<!--
  Hi there! Thank you for wanting to make Poetry better.

  Before you submit this; let's make sure of a few things.
  Please make sure the following boxes are ticked if they are correct.
  If not, please try and fulfill these first.
-->

<!-- Checked checkbox should look like this: [x] -->
- [x] I have searched the [issues](https://github.com/sdispater/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] I have searched the [documentation](https://poetry.eustace.io/docs/) and believe that my question is not covered.

## Question
<!-- Now feel free to write your question. Thanks again 🙌 ❤️ -->
pipenv has `pipenv clean` which removes all packages that are no longer in Pipfile.lock from the virtualenv. What is the equivalent with poetry? The docs say there is no need for `pipenv sync` and `pipenv clean` due to the `remove` command https://github.com/sdispater/poetry#remove-command

But how does this work? I would need to run `poetry remove XXX` against all my virtualenvs - manually keeping track of all packages that have been removed and remembering to remove them myself?

Is there not any way to sync my poetry.lock to my virtualenv to ensure the virtualenv only has the packages specified in poetry.lock installed?

I have a virtualenv on each box I deploy to, and keep pyproject.toml and poetry.lock under source code control. I then want a way to ensure each of those virtualenvs only has the packages in poetry.lock installed.


Discussion:
I am not sure I understand. Why would your packages no longer be in `poetry.lock` but still in your virtualenv?
Actually I just realized `poetry remove` won't even work.

Here I remove a package from en1 and then switch to env2 and attempt to remove it from there and it fails

```
(env1) ➜  poetrytest /Users/aw/Library/Python/2.7/bin/poetry remove boto3
Updating dependencies
Resolving dependencies... (0.1s)


Package operations: 0 installs, 0 updates, 9 removals

Writing lock file

  - Removing boto3 (1.9.47)
  - Removing botocore (1.12.47)
  - Removing docutils (0.14)
  - Removing futures (3.2.0)
  - Removing jmespath (0.9.3)
  - Removing python-dateutil (2.7.5)
  - Removing s3transfer (0.1.13)
  - Removing six (1.11.0)
  - Removing urllib3 (1.22)
(env1) ➜  poetrytest source env2/bin/activate
(env2) ➜  poetrytest /Users/aw/Library/Python/2.7/bin/poetry install
Installing dependencies from lock file

Nothing to install or update

(env2) ➜  poetrytest /Users/aw/Library/Python/2.7/bin/poetry remove boto3

[ValueError]
Package boto3 not found

remove [-D|--dev] [--dry-run] [--] <packages> (<packages>)...
```
Oh. I think I understand. You update the lock locally and then deploy. Is that right?
> I am not sure I understand. Why would your packages no longer be in `poetry.lock` but still in your virtualenv?

Because I have many virtualenvs - one per EC2 instance. So I copy over my poetry.lock and want to sync each virtualenv using that poetry.lock, adding/upgrading/removing packages as needed to keep the virtualenv in sync (similar to how `pipenv sync && pipenv clean` would work.
We have this problem using Poetry as well. One developer `poetry remove`s and deploys the new `poetry.lock`/`pyproject.toml`. The other devs pull the changes, but poetry install does not remove the old packages. Are there any plans to support this workflow?
The README mentions `clean` and `sync`, but these are not commands on version `0.12.11`.  I am also curious to know what steps to take to ensure that my local virtualenv matches `poetry.lock`.

Edit: I have found that I can run `poetry shell`, then `exit`, and part of the output will be the path to the virtualenv folder.  I use `rm -rf` on that folder, and then `poetry install` runs clean.
Same here.
would be great if packages not in poetry.lock were removed. Now there is a chance of such packages piling up and unexpectedly influencing your application (some packages look in specific locations for plugins and such).

Recreating a virtual environment each and every time is time consuming and unless it can happen automatically when checking out a branch, developers aren't going to do it.
Another use-case for this is CI build caching. If you remove dependency in a branch and the CI build for that branch pulls from a cache that still has that dependency, I definitely want it removed.

This is how Yarn works which is important because then `yarn install` means "make my build reproducible" rather than "make sure I have all the dependencies I need", which is not the same thing.

This ticket is in conflict with https://github.com/sdispater/poetry/issues/533 (though AFAICT Poetry indeed does not uninstall dependencies, so not sure what that ticket's about).
It looks like if this line would loop over the installed packages, that it would result in obsolete packages to be cleaned up.

https://github.com/sdispater/poetry/blob/f2e2ed470ca1560111226b5f6f8f9dc4a28d319a/poetry/puzzle/solver.py#L96

I'm trying see what it does if I changed it, but I can't seem to get poetry running from it's own git repo inside PyCharm's debugger 🤔 
@kevinkjt2000 AFAICT the only references to `clean` and `sync` in the README are talking about Pipenv commands, not Poetry. Also, `poetry debug:info` (0.12.x) will print out the virtualenv without you having to do weird things to find it.
I did an attempts in pull request #1037 to remove obsolete packages when running `poetry install`.
Still a work in progress, but it looks like it's cleaning up obsolete packages (e.g. installed, but not in the list of resolved packages)
@jobec a couple of comments if you don't mind:
a) It is not clear to me if removing packages is a good default for poetry install
b) I am not a native speaker, but the term "obsolete" seems to be slightly misused in this context. Not sure what to propose though. Maybe "unspecified" packages?
c) This is a breaking change. Among other things, it would be making it impossible to implement something [like this](https://github.com/sdispater/poetry/issues/951#issuecomment-475156710). IMHV, removing packages should be either a separate subcommand or at the very least a `poetry install` flag. 

First of all, the PR is just a beginning. It's no where near finished nor fine tuned. And I'm open for all feedback.

* A) True, although I think people expect it to behave like that. I know I did. "Freezing dependencies" as in "THIS is how it should be". Anyway, I agree that changing current behavior might be a bridge to far.

* B) I'm also not a native speaker, but obsolete is the first thing that crossed my mind. I guess there are better synonyms: unknown, unlocked, unused, unspecified, rogue...

* C) The comment you point at is indeed a useful use-case. Combined with A) it might be a good reason *not* to make this behavior the default. Anyway, I think it's best to put it as a flag to `poetry install`. What about `--clean`. Matching with `git clean`
a) I assert that it is a good default, though that depends on if (per my [earlier comment](https://github.com/sdispater/poetry/issues/648#issuecomment-482763789)) you view "install" to mean "make my build reproducible" (as I do) rather than "make sure I have all the dependencies I need" (which is a good start, but incomplete).

Regarding point (c). From the [linked comment](https://github.com/sdispater/poetry/issues/951#issuecomment-475156710):

> You might still install a package with confliciting dependencies though. Perhaps it would make sense to have a special mode of poetry add that will resolve dependencies and install a package without adding it to pyproject.toml.

The semantics of this use case seem weird and difficult. If the "temporary" package has overlapping dependencies with non-temporary packages, what happens? Does it change the versions of the common dependencies? If so, does it update the lock file? Does it instead just fail outright? 

The OP on that issue says:

> Then there are tools that I use personnally to develop, but are not required to run tests. Examples are ipython, icecream, pdb++. Those are more personal preferencies that real requirements.

In node, the accepted solution for this use-case is either (1) install it globally since it's not relevant to the project at hand or (2) install it as part of the project because you use it to develop the project (and hence it is a dev dependency). It seems to me that trying to support a third, middle road is likely to cause more confusion than help.
@seansfkelley @jobec I, at the very least, hope that we can all agree that:
- "make my build reproducible"
- "make sure I have all the dependencies I need"

are both valid and discrete usecases :)

WRT to which one should be the default, I honestly don't have a strong opinion. I tend to prefer the latter option as the default one since it is more common for development + *it is* the current behaviour. Adding an extra flag and/or using an explicit subcommand on deployment scripts seems more natural.

@seansfkelley I transferred the discussion for installing packages without adding them to `pyproject.toml`  [here](https://github.com/sdispater/poetry/issues/951#issuecomment-482886431)
> I guess there are better synonyms: unknown, unlocked, unused, unspecified, rogue...

"orphan" is an option too
`poetry autoremove`.

`clean` has different meaning in other package managers, which clear up cache packages not been installed yet.
`poetry autoremove` assumes you want a different command for this. What's the use-case for a separate command instead of changing the behavior of `install`? Would it make sense to use `autoremove` at any other time except immediately after `install`?

> I tend to prefer the latter option as the default one since it is more common for development + _it is_ the current behaviour.

Possession is 9/10ths of the law. :) You would almost certainly only want the more permissive behavior in development, but even in development, how desirable is it? Enough to be the default?

> Adding an extra flag and/or using an explicit subcommand on deployment scripts seems more natural.

In general, yeah, I think requiring CI commands to bear the burden of extra configuration is okay. But at least from my experience using Yarn, npm and Cargo, stricter behavior by default leads to fewer red herrings and unexpected behavior. In fact, I assumed Poetry did do this until I discovered the hard way (leading to my first comment here, https://github.com/sdispater/poetry/issues/648#issuecomment-482763789) that it did not.
I'd say: `poetry install --keep-untracked`

With a (new) default of removing untracked packages (e.g. not in poetry.lock), as I think the larger percentage of users expect it to work as such.

From the main website:

> Develop
> Poetry comes with all the tools you might need to manage your projects in **a deterministic way**. 

> Track
> Having an **insight of your project's dependencies** is just one command way. 

> Poetry uses **the exact versions listed in poetry.lock** to ensure that the package versions are consistent for everyone working on your project.

> Your CI server, production machines, other developers in your team, everything and **everyone runs on the same dependencies**, which mitigates the potential for bugs affecting only some parts of the deployments.
Nice citation of the website! :) I also think "untracked" is a good word to describe these dependencies.
+1 for untracked.

Apart from that, I completely agree that having a way to have a "purged" virtualenv is really important. That being said though, I also think that not purging the virtualenv while developing can also be useful. 

The reason I don't really like changing the defaults is that I prefer adding a flag on a deployment script rather than having to add a cli flag when using poetry interactively (i.e. while developing)
Why? What we are talking about is removing packages, not installing.
You will not install any untracked packages anyway.
@drunkwcodes this discussion is related to #951 
I won't look at that for a while...

There are many ways to install untracked packages, not just with `poetry install --strange-option`.
Having a `poetry autoremove` command to remove universally untracked packages is a proper thing to do.

This is useful when you have local editable installations not packaged, and do not want to create new venv.
> Having a `poetry autoremove` command to remove universally untracked packages is a proper thing to do.

Well, this is the point under discussion. I think there is value in increasing the scope of `install` to handle this case as well (though I disagree with @pmav99 over whether it's a flag or not), specifically, because then install can become a one-stop shop to make your dependencies reflect the lockfile without having to juggle multiple commands. One extra command wouldn't be a huge burden for those of us having this discussion here, but keeping this behavior localized to `install` instead means that (1) it's more discoverable, (2) it's more correct by default and (3) Poetry doesn't bloat.

That said, in writing this comment I did discover a couple of interesting related commands in other tools:

- [`npm ci`](https://docs.npmjs.com/cli/ci.html) is a variant of `npm install` specifically designed to be stricter for a CI environment, noting that it "...can help catch errors or inconsistencies caused by the incrementally-installed local environments of most npm users."
- [`yarn prune`](https://yarnpkg.com/en/docs/cli/prune) and [`yarn dedupe`](https://yarnpkg.com/en/docs/cli/dedupe) are specifically called out as not necessary due to the behavior of `yarn install`.
- [`yarn check --verify-tree`](https://yarnpkg.com/en/docs/cli/check) is designed to check what's on disk against what's in the lock file. (Though it should be noted that as a heavy Yarn user for years, I never knew this command existed until now because `yarn install` is already so dependably predictable.)
- [`cargo fetch`](https://doc.rust-lang.org/cargo/commands/cargo-fetch.html) is designed exclusively to populate Cargo's dependency cache, which is the closest it has to `poetry install`. Cargo thinks about this whole problem differently than Poetry or npm/Yarn, preferring to think about `run` and `build` rather than a separate dependency-installation step. `run` and `build` _as a prerequisite_  ensure that your dependencies are in the right state. `fetch` is presented as a mere convenience if you're going off-network, as opposed to a core part of the development flow.

My preferred style here is Yarn's, which has to tolerate users installing extra stuff (like npm) but by default tries to assert total control over `node_modules`. It should be noted that the analogy isn't perfect due to the way node handles conflicting versions (via duplication), which allows it to sidestep some complexity that Poetry has to handle.

On the topic of whether it should be default-on or default-off, I think that depends on one's preferred development flow. As someone who does not install untracked dependencies hardly ever, I clearly prefer @jobec's suggestion of `poetry install --keep-untracked`. The opposite (`--remove-untracked`?) would be annoying for me to type every time, which is I suppose the same argument that frequent untracked-dependency users would make about `--keep-untracked`. Not sure how to resolve that difference of opinion. :)
It's absolutly doable.
What we need is to figure out all kinds of installations which can be with poetry projects at the same time.

And doing it professionally and better than a brutal virtualenv restart is the only concern.
@drunkwcodes In my opinion, this comment does not add anything of value to this discussion. Please refrain from providing your commentary unless you're representing a position that has not already been presented in the various open issues on this project.
Who actually has the final say in this?
@sdispater ?
@seansfkelley thank you for the thorough comparison)

> (2) it's more correct by default 

Care to elaborate? It kind of feels like "correctness" in this context is rather subjective than objective (i.e. each tool uses UI/UX different conventions).
Point (2) above was referring only to the comparison of having the behavior in `install` versus a new command, not about whether `install` should be remove-by-default or keep-by-default. With that in mind, what I meant by "more correct by default" is that baking this behavior into `install` means that the default action (`poetry install`) will be opinionated about what it means to be correct and which mode is preferred, rather than having a pair of related commands and letting you figure it out. Poetry is an opinionated tool, and I think that should extend to this as well.

Note also I said "_more_ correct", not simply "correct", because as we have determined we don't agree about which should be the default but I think we agree that it would be preferable to keep it in `install`, whatever it is...?
Running `poetry remove pkg` fails, if `pkg` is in `poetry.lock` but not in `pyproject.toml`.

This is really surprising behavior, given that `poetry update pkg` works just fine.

`poetry remove pkg` should remove `pkg` from `poetry.lock`, even if it's not in `pyproject.toml` -- after all, this is exactly how `poetry update pkg` works.


```
➜  cloud-django git:(328/branch) ✗ poetry remove django-stubs

[ValueError]
Package django-stubs not found
```

-----

```
➜  cloud-django git:(328/branch) ✗ poetry update django-stubs
Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file


Package operations: 0 installs, 0 updates, 2 removals

  - Removing dataclasses (0.6)
  - Removing django-stubs (0.12.0)
➜  cloud-django git:(328/inspection_event_publishing) ✗
```


Or with an image:

![Screen Shot 2019-10-03 at 6 19 38 PM](https://user-images.githubusercontent.com/1524088/66170701-e5255000-e60a-11e9-8ccf-bffcc7086b60.png)

Maybe add a command `poetry ci` like `npm ci`🤔. 
Not changing default behavior and don't need to run command like `auto remove/clean` after `install`.

(And much shorter than `poetry install`
Just wondering if this is moving anywhere? It looks like there was a good proposal:
`poetry install --keep-untracked`
Not sure if just not enough resources to action this?

To clarify the use case if it's not obvious, if you collaborate with anyone, including CI or have multiple machines, you probably add and remove dependencies organically, it makes no sense to keep removed ones around, especially if they interact with other tools
I had a PR partially implementing this. But is has been closed in the meanwhile.

Also, without a go or direction from a maintainer, there's not much fun in working on it of it won't be merged.
Hello @jobec,

I guess I closed your PR as this has the status WIP for a very long time and you didn't react on the stale bot message. So I thought you didn't work anymore on it.

I found this feature very useful and appreciate your contribution. So please go on! :+1: 

fin swimmer
Still the question remains:

Maintainers, what direction to go in? Remove obsolete/stale/untracked/... packages by default, or keep them.
At the end @sdispater has to decide, because it's a new feature.

I support the version to remove untracked packages by default and have the `--keep-untracked` parameter for those who really need it (I read some of the arguments why people would do this, but still doesn't understand why ...).
This is very important and I don't know of any way to do this.

Say several developers work on a project. One performs an update which removes a package from the pyproject and lock files. How will the other developers ever remove this package?

Pipenv, for all of its flaws, at least has `pipenv clean`.

EDIT: Seems the `install` command removes packages in some cases: https://github.com/python-poetry/poetry/blob/d27a119d391a83d2f816b68129dc62346f7aef03/poetry/installation/installer.py#L409 (when the dev flag changes). 

Resolving patch:
diff --git a/poetry/console/commands/install.py b/poetry/console/commands/install.py
--- a/poetry/console/commands/install.py
+++ b/poetry/console/commands/install.py
@@ -19,6 +19,9 @@ class InstallCommand(EnvCommand):
             "Output the operations but do not execute anything "
             "(implicitly enables --verbose).",
         ),
+        option(
+            "remove-untracked", None, "Removes packages not present in the lock file.",
+        ),
         option(
             "extras",
             "E",
@@ -57,6 +60,7 @@ def handle(self):
         installer.extras(extras)
         installer.dev_mode(not self.option("no-dev"))
         installer.dry_run(self.option("dry-run"))
+        installer.remove_untracked(self.option("remove-untracked"))
         installer.verbose(self.option("verbose"))
 
         return_code = installer.run()
diff --git a/poetry/installation/installer.py b/poetry/installation/installer.py
--- a/poetry/installation/installer.py
+++ b/poetry/installation/installer.py
@@ -38,6 +38,7 @@ def __init__(
         self._pool = pool
 
         self._dry_run = False
+        self._remove_untracked = False
         self._update = False
         self._verbose = False
         self._write_lock = True
@@ -82,6 +83,14 @@ def dry_run(self, dry_run=True):  # type: (bool) -> Installer
     def is_dry_run(self):  # type: () -> bool
         return self._dry_run
 
+    def remove_untracked(self, remove_untracked=True):  # type: (bool) -> Installer
+        self._remove_untracked = remove_untracked
+
+        return self
+
+    def is_remove_untracked(self):  # type: () -> bool
+        return self._remove_untracked
+
     def verbose(self, verbose=True):  # type: (bool) -> Installer
         self._verbose = verbose
 
@@ -155,6 +164,7 @@ def _do_install(self, local_repo):
                 self._installed_repository,
                 locked_repository,
                 self._io,
+                remove_untracked=self._remove_untracked,
             )
 
             ops = solver.solve(use_latest=self._whitelist)
@@ -221,7 +231,12 @@ def _do_install(self, local_repo):
             whitelist.append(pkg.name)
 
         solver = Solver(
-            root, pool, self._installed_repository, locked_repository, NullIO()
+            root,
+            pool,
+            self._installed_repository,
+            locked_repository,
+            NullIO(),
+            remove_untracked=self._remove_untracked,
         )
 
         with solver.use_environment(self._env):
diff --git a/poetry/puzzle/solver.py b/poetry/puzzle/solver.py
--- a/poetry/puzzle/solver.py
+++ b/poetry/puzzle/solver.py
@@ -5,10 +5,15 @@
 from typing import Dict
 from typing import List
 
+from clikit.io import ConsoleIO
+
 from poetry.core.packages import Package
+from poetry.core.packages.project_package import ProjectPackage
 from poetry.mixology import resolve_version
 from poetry.mixology.failure import SolveFailure
 from poetry.packages import DependencyPackage
+from poetry.repositories import Pool
+from poetry.repositories import Repository
 from poetry.utils.env import Env
 
 from .exceptions import OverrideNeeded
@@ -21,7 +26,15 @@
 
 
 class Solver:
-    def __init__(self, package, pool, installed, locked, io):
+    def __init__(
+        self,
+        package,  # type: ProjectPackage
+        pool,  # type: Pool
+        installed,  # type: Repository
+        locked,  # type: Repository
+        io,  # type: ConsoleIO
+        remove_untracked=False,  # type: bool
+    ):
         self._package = package
         self._pool = pool
         self._installed = installed
@@ -29,6 +42,7 @@ def __init__(self, package, pool, installed, locked, io):
         self._io = io
         self._provider = Provider(self._package, self._pool, self._io)
         self._overrides = []
+        self._remove_untracked = remove_untracked
 
     @property
     def provider(self):  # type: () -> Provider
@@ -132,6 +146,18 @@ def solve(self, use_latest=None):  # type: (...) -> List[Operation]
 
                 operations.append(op)
 
+        if self._remove_untracked:
+            locked_names = {locked.name for locked in self._locked.packages}
+
+            for installed in self._installed.packages:
+                if installed.name == self._package.name:
+                    continue
+                if installed.name in Provider.UNSAFE_PACKAGES:
+                    # Never remove pip, setuptools etc.
+                    continue
+                if installed.name not in locked_names:
+                    operations.append(Uninstall(installed))
+
         return sorted(
             operations,
             key=lambda o: (


Test patch:
diff --git a/tests/installation/test_installer.py b/tests/installation/test_installer.py
--- a/tests/installation/test_installer.py
+++ b/tests/installation/test_installer.py
@@ -296,6 +296,59 @@ def test_run_install_no_dev(installer, locker, repo, package, installed):
     assert len(removals) == 1
 
 
+def test_run_install_remove_untracked(installer, locker, repo, package, installed):
+    locker.locked(True)
+    locker.mock_lock_data(
+        {
+            "package": [
+                {
+                    "name": "a",
+                    "version": "1.0",
+                    "category": "main",
+                    "optional": False,
+                    "platform": "*",
+                    "python-versions": "*",
+                    "checksum": [],
+                }
+            ],
+            "metadata": {
+                "python-versions": "*",
+                "platform": "*",
+                "content-hash": "123456789",
+                "hashes": {"a": []},
+            },
+        }
+    )
+    package_a = get_package("a", "1.0")
+    package_b = get_package("b", "1.1")
+    package_c = get_package("c", "1.2")
+    package_pip = get_package("pip", "20.0.0")
+    repo.add_package(package_a)
+    repo.add_package(package_b)
+    repo.add_package(package_c)
+    repo.add_package(package_pip)
+
+    installed.add_package(package_a)
+    installed.add_package(package_b)
+    installed.add_package(package_c)
+    installed.add_package(package_pip)  # Always required and never removed.
+    installed.add_package(package)  # Root package never removed.
+
+    package.add_dependency("A", "~1.0")
+
+    installer.dev_mode(True).remove_untracked(True)
+    installer.run()
+
+    installs = installer.installer.installs
+    assert len(installs) == 0
+
+    updates = installer.installer.updates
+    assert len(updates) == 0
+
+    removals = installer.installer.removals
+    assert set(r.name for r in removals) == {"b", "c"}
+
+
 def test_run_whitelist_add(installer, locker, repo, package):
     locker.locked(True)
     locker.mock_lock_data(
diff --git a/tests/puzzle/test_solver.py b/tests/puzzle/test_solver.py
--- a/tests/puzzle/test_solver.py
+++ b/tests/puzzle/test_solver.py
@@ -1851,3 +1851,25 @@ def test_solver_should_not_go_into_an_infinite_loop_on_duplicate_dependencies(
             {"job": "install", "package": package_a},
         ],
     )
+
+
+def test_solver_remove_untracked_single(package, pool, installed, locked, io):
+    solver = Solver(package, pool, installed, locked, io, remove_untracked=True)
+    package_a = get_package("a", "1.0")
+    installed.add_package(package_a)
+
+    ops = solver.solve()
+
+    check_solver_result(ops, [{"job": "remove", "package": package_a}])
+
+
+def test_solver_remove_untracked_keeps_critical_package(
+    package, pool, installed, locked, io
+):
+    solver = Solver(package, pool, installed, locked, io, remove_untracked=True)
+    package_pip = get_package("pip", "1.0")
+    installed.add_package(package_pip)
+
+    ops = solver.solve()
+
+    check_solver_result(ops, [])