Repository: pypa/pip

Issue:
[2020-resolver] No longer can get list of available versions.
<!--
Please provide as much information as you can about your failure, so that we can understand the root cause.

Try if your issue has been fixed in the in-development version of pip. Use the following command to install pip from master:

    python -m pip install -U "pip @ https://github.com/pypa/pip/archive/master.zip"
-->

### What did you want to do?
<!-- Include any inputs you gave to pip, for example:

* Package requirements: any CLI arguments and/or your requirements.txt file
* Already installed packages, outputted via `pip freeze`
-->
I would like to list all the available versions of a package. In the past we would use this:
```bash
pip install pip==asdf  # any gibberish here :)
```

### Output

```
ERROR: Could not find a version that satisfies the requirement pip==asdf
ERROR: No matching distribution found for pip==asdf
```

### expected output (`pip==20.2.4`)
```
ERROR: Could not find a version that satisfies the requirement pip==asdf (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 
0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 
1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 
6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 
8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 
19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1b1, 
20.1, 20.1.1, 20.2b1, 20.2, 20.2.1, 20.2.2, 20.2.3, 20.2.4, 20.3b1)
```

**Additional information**
* Using verbose flag: `pip install -v pip==asdf` we can sorta see that versions. (300 line output, too long to post here)
Seems like no extra requests needed. I wasn't 100% sure if it was the case. 
* This is useful when debugging failed installation of a requirement.txt, pipenv or other 3rd party package managers


Discussion:
@vityas thank you for this bug report, which I've just discussed with pip maintainer @pradyunsg.

I understand what you're concerned about. It used to be the case that if you ran that particular command, pip would tell you about all the versions of that package available in the index you were hitting (such as PyPI). Now pip doesn't print that info about versions by default, with the new resolver.

This was not a useful error message for people who UNINTENTIONALLY got it, but it was the only way to get (on the command line) a list of all the available versions of a package on PyPI or on another index.

If you're hitting PyPI, you can go the release history page for a project, such as https://pypi.org/project/pip/#history , but with the new resolver behavior, currently, there's no easy way to get this info in the CLI directly and in a concise format. pip hits an index and gets the list, but now doesn't present it by default. And if you turn up verbosity, you get the info but in a hard-to-use format, as you said.

Right now, in order to expedite the release of pip 20.3 this week. I'm marking it as post-release work.

For now, we advise you to ([per the deprecation timeline](https://pip.pypa.io/en/latest/user_guide/#deprecation-timeline)) opt-out and choose the old resolver behavior for now,  using the flag `--use-deprecated=legacy-resolver`. 

But also: you're bringing further attention to two problems with pip's functionality more generally. pip has a few functions, such as `pip search`, which offer a little bit of discoverability to help users see what is available in package indexes. Pradyun wants to consolidate functionality and design for these features. And we know that pip's output, especially its `--verbose` output, needs work. So I hope you will please [talk to our UX people](https://bit.ly/pip-ux-studies), @nlhkabu @ei8fdb and @georgiamoon, by signing up for the user experience studies, to help them work on the larger project of making our verbose output and our package discoverability functionality better.

Thank you!
I would really love to see this back - it seemed like the most streamlined way to find out what versions one can install. See this highly voted stack answer [here](https://stackoverflow.com/a/26664162).
It's definitely not (and never was) a supported way of getting that information. But I can imagine the possibility that we might implement a supported way of getting the same data. Although to be honest, as long as you just want to get the information from PyPI (and don't need to support extra indexes, or local package repositories) it's not that hard to write something yourself that gets the same information, using the PyPI JSON interface, plus `packaging.tags` for doing the wheel compatibility checks.

Maybe someone could publish a "PyPI query" package that did things like this? It might be an interesting project for someone who wants to get into working with the packaging ecosystem.
I believe that at this point this is just a behaviour that many people may have relied on for years. Sometimes things that were not meant to be present but were just become part of what is expected.
@RafalSkolasinski Thanks for sharing your experience! Indeed, we're getting a lot of demonstrations of [Hyrum's Law](https://www.hyrumslaw.com/) as we engage in this work this year. :-) 

As you saw in a comment above, when prioritizing what to address and what to leave for later, I decided to leave this particular issue for later. We've had to be incredibly limited in what we build and fix with [the funding we got to work on pip this year](https://pyfound.blogspot.com/2019/12/moss-czi-support-pip.html). It runs out at the end of December. One of the things we did with that funding was [set up a more systematic way to hear from our users about how you use pip, what features you use, and so on, to help us make better decisions -- user experience research and testing](https://pip.pypa.io/en/stable/ux_research_design/). I hope you will sign up for the user experience surveys and share your opinion in more depth that way. (For example, you could share whether you always use PyPI or whether this functionality is important to you when installing from other sources as well.)

And the pip team has no reliable future funding, and can do a lot more if companies that depend on Python packaging tools chip in ([here's the packaging sponsorship program](https://pyfound.blogspot.com/2020/04/sponsoring-python-packaging.html)).
Hello folks!
I had a PR #8978 which proposes a new way to look at the available versions (`pip search <packages> --list-all-versions`)

If you support this idea, please let folks here know so we can move forward with this offer :smile:

cc @ei8fdb
> This was not a useful error message for people who UNINTENTIONALLY got it

@brainwane I disagree with that, actually it's still useful even when UNINTENTIONALLY getting it.  If I did `pip install distlib==0.29.post0` and saw:

    ERROR: Could not find a version that satisfies the requirement distlib==0.29.post0
    (from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 
    0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.2.9, 0.2.9.post0, 0.3.0, 0.3.1)

That tells me I just had a silly typo and I wanted `0.2.9.post0`.  But if I saw

    ERROR: Could not find a version that satisfies the requirement distlib==0.29.post0 (from versions: none)

It tells me something else, probably that I have misconfigured my index urls and I should check the pip config.  In 20.3.x

    ERROR: Could not find a version that satisfies the requirement

This is less helpful again. Is my index misconfigured? Could it find any version at all? Do I just need to wait a moment or clear pip's cache - maybe I just published this package recently?

`pip search` afaik is broken and slated to be removed (https://github.com/pypa/pip/issues/5216). At least, I could not find any way to use it reliably (and couldn't get it working at all with a [devpi-server](https://www.devpi.net/) index).

Thank you for mentioning the `--use-deprecated=legacy-resolver` option! For the purposes of keeping the project [johnnydep](https://github.com/wimglenn/johnnydep) functioning on pip 20.3.x, [this helped for now](https://github.com/wimglenn/johnnydep/commit/2efd5f345e57b423f3acb7f4e947166d0af1f526). Currently it scrapes pip's CLI output in a way that needs careful tweaking and testing each time a new version of pip is released. So here's a bit of UX feedback, it would be really nice if there was a less fragile way to get:

- download urls
- package dependencies
- versions available

> It's definitely not (and never was) a supported way of getting that information. 

@pfmoore  So how do we get that information?  A while ago pip moved all package finding code under _internal and made it explicit that a CLI subprocess was the only supported way to [use pip from your program](https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program). This is arguably an important part of the CLI.

> it's not that hard to write something yourself that gets the same information, using the PyPI JSON

I think that is oversimplifying the problem. It might be true if pip was _only_ a PyPI client, but it's a lot more than that isn't it? Pip's the de facto tool for working with Python packages, whether from PyPI or not, since we don't really have one in the stdlib.

To repeat again the points in [#1884](https://github.com/pypa/pip/issues/1884#issuecomment-670694740): the PyPI JSON is not in general "the same information" because where pip looks is highly configurable, e.g. we may have another --index-url and then an --extra-index-url and maybe that index is just a [simple repository](https://www.python.org/dev/peps/pep-0503/) which is supporting a JSON api at all. There are a lot of little details here and doing it in a way that accurately produces the same packages that pip would see would mean reproducing a significant amount of package finder code found within `pip._internal`.
@brainwane thanks for your explanation on why this feature (I'm gonna call it a feature as we rely on it in johnnydep, pipgrip, probably more) was deprecated in the new resolver.

Like mentioned above, it would be quite cumbersome and most likely introducing bugs to move away from using this feature in pip, so I've dropped support for pip 21 (hoping to loosen that constraint again though!), and will keep an eye out for this issue.

Thanks for all your hard work and responses throughout these exciting times! 🙏 
imo: removing output that helped with diagnosis (a version typo), seems to go against a basic unix philosophy [\"Write programs which fail in a way that is easy to diagnose\"](https://todevornot.com/post/184342697469/the-17-unix-rules-and-the-peace-of-mind).  
imo this feature should be brought back in or at least outsourced to a new option thats bit less unwieldy than sudo pip3 --use-deprecated=legacy-resolver install xx==. The new behavoir can be misleading and less useful for the many reasons already stated.
This feature must come back. It is terrible design to remove it. It was very useful to navigate between different package versions. Basically, when an upgrade of a package breaks something, to be able to roll back to the previous version.
This has been a [canonical](https://en.wikipedia.org/wiki/Canonical_form) feature of Pip for many of its users for at least 6 years now. I've given this tip out numerous times at work, I'm probably not the only one. There's [this SO answer](https://stackoverflow.com/a/26664162) - +950 rep on a 10 year-old, very specific question that's been viewed 280k times. Tooling that uses pip has been scraping this data for years.

If you wish to remove this output, then to appease users you'll have to add a feature that makes it possible to list it out. And one that is as succinct and streamlined as appending `==`. By the looks of it, there's nothing else possible that comes close. 

## Listing versions by appending `==` is, simply put, a *good feature.*

We've come to enjoy it due to a happy accident related to how pip handles its arguments. But that doesn't mean it's not valuable and shouldn't be there.

If you're not going to add `==` usage back, then the discussion is probably a waste of time. A longer flag or a subcommand will feel like a chore to type when a version is mistyped. At that point it indeed would be a good idea to create a separate tool, if only it wasn't for the fact that it would have to clone large chunks of code from Pip in order for the outputs to be relevant for the issue at hand.

UX-wise you have to remember you are operating in a regime where each keystroke is a considerable cost. Anyone who uses `==` a lot is familiar with the following typing-scheme:

`pip install package==` <kbd>⏎</kbd> → <kbd>↑</kbd> `<desired_version>`<kbd>⏎</kbd>

or

`pip install package==<wrong_version>` <kbd>⏎</kbd> → <kbd>↑</kbd> <kbd>←BP</kbd> <kbd>←BP</kbd> <kbd>←BP</kbd> ... `<desired_version>`<kbd>⏎</kbd>

The convenience factor of this is huge. I doubt anyone will be able to come up with anything better.

Trying to fix the lack of that by adding `--list-versions` as an available flag is misguided. Not to bash on @NoahGorny's idea - pip _should_ have an explicit, user-facing feature that makes it possible to obtain the list. But #8978 should be discussed separately to this issue.

IMO `pip search` is, to put it lightly - irrelevant. It produces unreliable and overly verbose results, uses different index from `pip install`, not to mention the `XMLRPC API has been temporarily disabled` of PyPI and #5216.

Resolving patch:
diff --git a/src/pip/_internal/resolution/resolvelib/factory.py b/src/pip/_internal/resolution/resolvelib/factory.py
--- a/src/pip/_internal/resolution/resolvelib/factory.py
+++ b/src/pip/_internal/resolution/resolvelib/factory.py
@@ -410,10 +410,17 @@ def _report_single_requirement_conflict(self, req, parent):
             req_disp = str(req)
         else:
             req_disp = f"{req} (from {parent.name})"
+
+        cands = self._finder.find_all_candidates(req.project_name)
+        versions = [str(v) for v in sorted({c.version for c in cands})]
+
         logger.critical(
-            "Could not find a version that satisfies the requirement %s",
+            "Could not find a version that satisfies the requirement %s "
+            "(from versions: %s)",
             req_disp,
+            ", ".join(versions) or "none",
         )
+
         return DistributionNotFound(f"No matching distribution found for {req}")
 
     def get_installation_error(


Test patch: