Repository: wimglenn/johnnydep

Issue:
ValueError with pip 24.1.1
Getting the version number of a package via `JohnnyDist` does not work anymore with pip 24.1.1. See log below. It works fine with pip 24.0.

Steps to reproduce:

Build docker image using the following Dockerfile:
```docker
FROM ubuntu:22.04

ENV TZ=Europe/Berlin
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN apt-get update && \
    apt-get install -y \
                    python3-pip \
                    python-is-python3

RUN python -m pip install -U pip==24.1.1
```

Start a docker container, start `python`:

```python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from johnnydep import JohnnyDist
>>> version = JohnnyDist("numpy")
2024-07-01 13:02:42 [info     ] init johnnydist                dist=numpy parent=None
2024-07-01 13:02:42 [debug    ] fetching best wheel            dist=numpy
2024-07-01 13:02:42 [debug    ] created scratch                dist_name=numpy tmpdir=/tmp/tmphfldgw64
2024-07-01 13:02:42 [debug    ] wheeling and dealing           args=/usr/bin/python -m pip wheel -vvv --no-deps --no-cache-dir --disable-pip-version-check --progress-bar=off numpy scratch_dir=/tmp/tmphfldgw64/tmppd7t88le
2024-07-01 13:02:44 [debug    ] wheel command completed ok     dist_name=numpy
2024-07-01 13:02:44 [debug    ] computing checksum             algorithm=md5 target=/tmp/tmphfldgw64/tmppd7t88le/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2024-07-01 13:02:44 [debug    ] computed checksum              result=3d129fe67d99e0aad451742abb963ffa
2024-07-01 13:02:44 [debug    ] finding import names           whl_file=/tmp/tmphfldgw64/tmppd7t88le/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2024-07-01 13:02:44 [debug    ] top_level.txt absent, iterating contents whl_file=/tmp/tmphfldgw64/tmppd7t88le/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2024-07-01 13:02:44 [debug    ] finding metadata               whl_file=/tmp/tmphfldgw64/tmppd7t88le/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2024-07-01 13:02:44 [debug    ] finding entry points           whl_file=/tmp/tmphfldgw64/tmppd7t88le/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
2024-07-01 13:02:44 [debug    ] removing scratch               dist_name=numpy tmpdir=/tmp/tmphfldgw64
>>> version.version_latest
2024-07-01 13:02:48 [debug    ] checking versions available    dist=numpy
2024-07-01 13:02:48 [warning  ] failed to get versions         stdout=Created temporary directory: /tmp/pip-build-tracker-w1p8aa4l
Initialized build tracking at /tmp/pip-build-tracker-w1p8aa4l
Created build tracker: /tmp/pip-build-tracker-w1p8aa4l
Entered build tracker: /tmp/pip-build-tracker-w1p8aa4l
Created temporary directory: /tmp/pip-wheel-gr7xzh08
ERROR: Invalid requirement: 'numpy==showmethemoney': Expected end or semicolon (after name and no valid version specifier)
    numpy==showmethemoney
         ^
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/packaging/requirements.py", line 36, in __init__
    parsed = _parse_requirement(requirement_string)
  File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
  File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
  File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/packaging/_parser.py", line 124, in _parse_requirement_details
    marker = _parse_requirement_marker(
  File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/packaging/_parser.py", line 145, in _parse_requirement_marker
    tokenizer.raise_syntax_error(
  File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
    raise ParserSyntaxError(
pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier)
    numpy==showmethemoney
         ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/req/constructors.py", line 362, in _parse_req_string
    return get_requirement(req_as_string)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/utils/packaging.py", line 45, in get_requirement
    return Requirement(req_string)
  File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
pip._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    numpy==showmethemoney
         ^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/commands/wheel.py", line 120, in run
    reqs = self.get_requirements(args, options, finder, session)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/cli/req_command.py", line 233, in get_requirements
    req_to_add = install_req_from_line(
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/req/constructors.py", line 405, in install_req_from_line
    parts = parse_req_from_line(name, line_source)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/req/constructors.py", line 379, in parse_req_from_line
    req: Optional[Requirement] = _parse_req_string(req_as_string)
  File "/usr/local/lib/python3.10/dist-packages/pip/_internal/req/constructors.py", line 376, in _parse_req_string
    raise InstallationError(msg)
pip._internal.exceptions.InstallationError: Invalid requirement: 'numpy==showmethemoney': Expected end or semicolon (after name and no valid version specifier)
    numpy==showmethemoney
         ^
Removed build tracker: '/tmp/pip-build-tracker-w1p8aa4l'

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/johnnydep/lib.py", line 225, in version_latest
    if self.versions_available:
  File "/usr/local/lib/python3.10/dist-packages/wimpy/util.py", line 36, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python3.10/dist-packages/johnnydep/lib.py", line 193, in versions_available
    result = pipper.get_versions(
  File "/usr/local/lib/python3.10/dist-packages/cachetools/__init__.py", line 702, in wrapper
    v = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/johnnydep/pipper.py", line 141, in get_versions
    [line] = lines
ValueError: not enough values to unpack (expected 1, got 0)
```

Discussion:

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 138a2be..2d073dc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -13,6 +13,7 @@ jobs:
     runs-on: ${{ matrix.os }}
 
     strategy:
+      fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
         python-version:
@@ -20,6 +21,7 @@ jobs:
         - "3.9"
         - "3.10"
         - "3.11"
+        - "3.12"
         include:
           - os: ubuntu-latest
             python-version: "3.7"
@@ -32,6 +34,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -VV
+          python -m pip install -U pip
           python -m pip install -r requirements-dev.txt
 
       - name: Run tests for ${{ matrix.python-version }} on ${{ matrix.os }}
diff --git a/johnnydep/__init__.py b/johnnydep/__init__.py
index 5c49913..500e9af 100644
--- a/johnnydep/__init__.py
+++ b/johnnydep/__init__.py
@@ -1,5 +1,5 @@
 """Display dependency tree of Python distribution"""
 
-__version__ = "1.20.4"
+__version__ = "1.20.5"
 
 from johnnydep.lib import *
diff --git a/johnnydep/pipper.py b/johnnydep/pipper.py
index f678dcf..b23695f 100755
--- a/johnnydep/pipper.py
+++ b/johnnydep/pipper.py
@@ -122,6 +122,50 @@ def _download_dist(url, scratch_file, index_url, extra_index_url):
 def get_versions(dist_name, index_url=None, env=None, extra_index_url=None):
     bare_name = requirements.Requirement(dist_name).name
     log.debug("checking versions available", dist=bare_name)
+    if env is None:
+        pip_version = _get_pip_version()
+    else:
+        pip_version = dict(env)["pip_version"]
+    if int(pip_version.split(".")[0]) < 24:
+        return _get_versions_legacy(dist_name, index_url, env, extra_index_url)
+    args = [
+        sys.executable,
+        "-m",
+        "pip",
+        "index",
+        "versions",
+        "--pre",
+        "--no-cache-dir",
+        "--disable-pip-version-check",
+        "--no-color",
+        "--no-python-version-warning"
+    ]
+    if index_url is not None:
+        args += ["--index-url", index_url]
+        if index_url != DEFAULT_INDEX:
+            hostname = urlparse(index_url).hostname
+            if hostname:
+                args += ["--trusted-host", hostname]
+    if extra_index_url is not None:
+        args += ["--extra-index-url", extra_index_url, "--trusted-host", urlparse(extra_index_url).hostname]
+    args.append(dist_name)
+    try:
+        out = check_output(args, stderr=STDOUT).decode("utf-8")
+    except CalledProcessError as e:
+        out = e.output.decode("utf-8")
+        if "ERROR: No matching distribution found for" in out:
+            return []
+        raise
+    for line in out.splitlines():
+        if line.startswith("Available versions: "):
+            versions = line[len("Available versions: "):].strip().split(", ")[::-1]
+            log.debug("found versions", dist=bare_name, versions=versions)
+            return versions
+
+
+@ttl_cache(maxsize=512, ttl=60 * 5)
+def _get_versions_legacy(dist_name, index_url=None, env=None, extra_index_url=None):
+    bare_name = requirements.Requirement(dist_name).name
     args = _get_wheel_args(index_url, env, extra_index_url) + [dist_name + "==showmethemoney"]
     try:
         out = check_output(args, stderr=STDOUT)
diff --git a/requirements-dev.txt b/requirements-dev.txt
index ad3f68c..9457e6f 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -8,3 +8,4 @@ coverage
 wheel
 whl >= 0.0.4
 wimpy == 0.3  # just something we can pin
+setuptools; python_version >= '3.12'


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 5140e57..3217276 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -22,6 +22,7 @@ original_check_output = subprocess.check_output
 @pytest.fixture(autouse=True)
 def expire_caches():
     pipper.get_versions.cache_clear()
+    pipper._get_versions_legacy.cache_clear()
     pipper._get_cache.clear()
     lib._get_info.cache_clear()
 
@@ -153,17 +154,32 @@ def fake_subprocess(mocker, add_to_index):
     def wheel_proc(args, stderr, cwd=None):
         exe = args[0]
         req = args[-1]
-        args = [
-            exe,
-            "-m",
-            "pip",
-            "wheel",
-            "-vvv",
-            "--no-index",
-            "--no-deps",
-            "--no-cache-dir",
-            "--disable-pip-version-check",
-        ]
+        if "versions" in args:
+            args = [
+                exe,
+                "-m",
+                "pip",
+                "index",
+                "versions",
+                "--pre",
+                "--no-index",
+                "--no-cache-dir",
+                "--disable-pip-version-check",
+                "--no-color",
+                "--no-python-version-warning"
+            ]
+        else:
+            args = [
+                exe,
+                "-m",
+                "pip",
+                "wheel",
+                "-vvv",
+                "--no-index",
+                "--no-deps",
+                "--no-cache-dir",
+                "--disable-pip-version-check",
+            ]
         args.extend(["--find-links={}".format(p) for p in index_data])
         args.append(req)
         output = subprocess_check_output(args, stderr=stderr, cwd=cwd)
diff --git a/tests/test_util.py b/tests/test_util.py
index 9444acf..35dc0f1 100644
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -29,8 +29,8 @@ def test_good_python_env():
     data = python_interpreter(sys.executable)
     assert isinstance(data, tuple)
     data = dict(data)
-    for value in data.values():
-        assert isinstance(value, text_type)
+    for key, value in data.items():
+        assert isinstance(value, text_type), key
     assert sorted(data) == [
         "implementation_name",
         "implementation_version",