Repository: pdm-project/pdm

Issue:
[BUG] Invalid version spec `>=5.*, <=6.*` creates unhandled junk dependency `b''`
- [X] I have searched the issue tracker and believe that this is not a duplicate.

**Make sure you run commands with `-v` flag before pasting the output.**

## Steps to reproduce

`pdm add pyaml-env==1.2.0` with a fresh `pyproject.toml` like the following:

```toml
[project]
name = ""
version = ""
description = ""
authors = [
    {name = "Gregory Danielson", email = "gregory.danielson3@gmail.com"},
]
dependencies = []
requires-python = ">=3.6"
license = {text = "MIT"}

[tool.pdm]
```

You can also set up a minimal project with the following `setup.py` and attempt to install it:

```py
from setuptools import setup

REQUIREMENTS = ["PyYAML>=5.*"]

setup(name="project",
      version="0.1.0",
      install_requires=REQUIREMENTS,
      package_dir={"": "src"},
      packages=["project"])

```

## Actual behavior

```
pdm.termui: ======== Starting round 1 ========
unearth.preparer: Using cached <Link https://files.pythonhosted.org/packages/d2/6c/fb09236297c1934448966abed6aaa2b2bed9f78992ef0346343e326398f4/pyaml_env-1.2.0-py3-none-any.whl (from https://pypi.org/simple/pyaml-env/)>
pdm.termui:   Adding requirement b''(from pyaml-env 1.2.0)
[FileRequirement(name=None, marker=None, extras=None, specifier=None, editable=False, prerelease=False, url=None, path=PosixPath("b''"), subdirectory=None)]
Traceback (most recent call last):
  File "/home/gregdan3/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/core.py", line 259, in main
    return Core().main(args)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/core.py", line 191, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/core.py", line 186, in main
    f(project, options)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/cli/commands/install.py", line 58, in handle
    actions.do_lock(
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/cli/actions.py", line 95, in do_lock
    mapping, dependencies = resolve(
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/resolvelib/resolvers.py", line 402, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/resolvelib/resolvers.py", line 238, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    self._add_to_criteria(criteria, requirement, parent=candidate)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/resolvelib/structs.py", line 136, in __iter__
    self._factory() if self._iterable is None else self._iterable
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/resolver/providers.py", line 151, in matches_gen
    candidates = self._find_candidates(reqs[0])
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/resolver/providers.py", line 127, in _find_candidates
    can = make_candidate(requirement)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/models/candidates.py", line 626, in make_candidate
    return Candidate(req, name, version, link)
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/models/candidates.py", line 145, in __init__
    link = req.as_file_link()  # type: ignore
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/pdm/models/requirements.py", line 321, in as_file_link
    return Link(url)
  File "<string>", line 9, in __init__
  File "/home/gregdan3/.local/pipx/venvs/pdm/lib/python3.10/site-packages/unearth/link.py", line 54, in __post_init__
    if self.url.startswith(vcs_prefixes):
TypeError: a bytes-like object is required, not 'str'
```


## Expected behavior

Version specs such as [those previously in pyaml-env](https://github.com/mkaranasou/pyaml_env/blob/acca18ffbf28ab03cf9aff34ce17db5d09bb2c93/requirements.txt) should be detected as breaking PEP440 rather than assumed to be FileRequirements.

## Environment Information

```bash
# Paste the output of `pdm info && pdm info --env` below:
PDM version:
  2.4.0
Python Interpreter:
  /usr/bin/python3 (3.10)
Project Root:
  /home/gregdan3/example
Project Packages:
  /home/gregdan3/example/__pypackages__/3.10

{
  "implementation_name": "cpython",
  "implementation_version": "3.10.9",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "6.0.12-arch1-1",
  "platform_system": "Linux",
  "platform_version": "#1 SMP PREEMPT_DYNAMIC Thu, 08 Dec 2022 11:03:38 +0000",
  "python_full_version": "3.10.9",
  "platform_python_implementation": "CPython",
  "python_version": "3.10",
  "sys_platform": "linux"
}

```

## Other Notes

`pyaml-env` has already fixed this bug [here](https://github.com/mkaranasou/pyaml_env/issues/30).

The issue results from [this logic](https://github.com/pdm-project/pdm/blob/e1f593963544c725c215134d479211ff7eab4e12/src/pdm/models/requirements.py#L479), where any failed attempt to create a `PackageRequirement` is subsequently parsed as a `FileRequirement`. `_file_req_re` matches the line, producing a FileRequirement with all null fields except for ; parsing the url produces a  `b''` that gets caught in the `results` [here](https://github.com/pdm-project/pdm/blob/e1f593963544c725c215134d479211ff7eab4e12/src/pdm/models/requirements.py#L452). 
The `b''` obviously fails the next round of dependency resolution.

Ideally, there would be an independent way to verify the FileRequirement so it could be more than a secondary case of the PackageRequirement.

Comments:

Patch:
diff --git a/news/1619.bugfix.md b/news/1619.bugfix.md
new file mode 100644
index 00000000..eca4a386
--- /dev/null
+++ b/news/1619.bugfix.md
@@ -0,0 +1,1 @@
+Fix the wildcards in requirement specifiers to make it pass the new parser of `packaging>=22`.
diff --git a/src/pdm/models/requirements.py b/src/pdm/models/requirements.py
index 49f0c33a..65e4a0b6 100644
--- a/src/pdm/models/requirements.py
+++ b/src/pdm/models/requirements.py
@@ -26,6 +26,7 @@ from pdm.models.markers import Marker, get_marker, split_marker_extras
 from pdm.models.setup import Setup
 from pdm.models.specifiers import PySpecSet, get_specifier
 from pdm.utils import (
+    PACKAGING_22,
     add_ssh_scheme_to_git_uri,
     comparable_version,
     normalize_name,
@@ -36,6 +37,8 @@ from pdm.utils import (
 )
 
 if TYPE_CHECKING:
+    from typing import Match
+
     from pdm._types import RequirementDict
 
 
@@ -458,6 +461,34 @@ def filter_requirements_with_extras(
     return result
 
 
+_legacy_specifier_re = re.compile(r"(==|!=|<=|>=|<|>)(\s*)([^,;\s)]*)")
+
+
+def parse_as_pkg_requirement(line: str) -> PackageRequirement:
+    """Parse a requirement line as packaging.requirement.Requirement"""
+
+    def fix_wildcard(match: Match[str]) -> str:
+        operator, _, version = match.groups()
+        if ".*" not in version or operator in ("==", "!="):
+            return match.group(0)
+        version = version.replace(".*", ".0")
+        if operator in ("<", "<="):  # <4.* and <=4.* are equivalent to <4.0
+            operator = "<"
+        elif operator in (">", ">="):  # >4.* and >=4.* are equivalent to >=4.0
+            operator = ">="
+        return f"{operator}{version}"
+
+    try:
+        return PackageRequirement(line)
+    except InvalidRequirement:
+        if not PACKAGING_22:  # We can't do anything, reraise the error.
+            raise
+        # Since packaging 22.0, legacy specifiers like '>=4.*' are no longer
+        # supported. We try to normalize them to the new format.
+        new_line = _legacy_specifier_re.sub(fix_wildcard, line)
+        return PackageRequirement(new_line)
+
+
 def parse_requirement(line: str, editable: bool = False) -> Requirement:
 
     m = _vcs_req_re.match(line)
@@ -475,14 +506,14 @@ def parse_requirement(line: str, editable: bool = False) -> Requirement:
         if replaced:
             line = line.replace("{root:uri}", root_url)
         try:
-            package_req = PackageRequirement(line)  # type: ignore
+            pkg_req = parse_as_pkg_requirement(line)
         except InvalidRequirement as e:
             m = _file_req_re.match(line)
             if m is None:
                 raise RequirementError(str(e)) from None
             r = FileRequirement.create(**m.groupdict())
         else:
-            r = Requirement.from_pkg_requirement(package_req)
+            r = Requirement.from_pkg_requirement(pkg_req)
         if replaced:
             assert isinstance(r, FileRequirement)
             r.url = r.url.replace(root_url, "{root:uri}")
diff --git a/src/pdm/utils.py b/src/pdm/utils.py
index 50fdd2c6..77187f80 100644
--- a/src/pdm/utils.py
+++ b/src/pdm/utils.py
@@ -23,9 +23,10 @@ from typing import IO, Any, Iterator
 from packaging.version import Version
 
 from pdm._types import Source
-from pdm.compat import Distribution
+from pdm.compat import Distribution, importlib_metadata
 
 _egg_fragment_re = re.compile(r"(.*)[#&]egg=[^&]*")
+PACKAGING_22 = Version(importlib_metadata.version("packaging")) >= Version("22")
 
 
 def create_tracked_tempdir(


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
index c3b9bc68..bea50964 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -1,7 +1,3 @@
 from pathlib import Path
 
-import packaging
-from packaging.version import Version
-
 FIXTURES = Path(__file__).parent / "fixtures"
-PACKAGING_22 = Version(packaging.__version__) >= Version("22.0")
diff --git a/tests/models/test_requirements.py b/tests/models/test_requirements.py
index c9f707de..d18c1eab 100644
--- a/tests/models/test_requirements.py
+++ b/tests/models/test_requirements.py
@@ -51,6 +51,8 @@ REQUIREMENTS = [
         "git+git@github.com:pypa/pip.git#egg=pip",
         "pip @ git+ssh://git@github.com/pypa/pip.git",
     ),
+    ("foo >=4.*, <=5.*", "foo<5.0,>=4.0"),
+    ("foo (>=4.*, <=5.*)", "foo<5.0,>=4.0"),
 ]
 
 
diff --git a/tests/models/test_specifiers.py b/tests/models/test_specifiers.py
index c611f6d3..362034e3 100644
--- a/tests/models/test_specifiers.py
+++ b/tests/models/test_specifiers.py
@@ -1,7 +1,7 @@
 import pytest
 
 from pdm.models.specifiers import PySpecSet
-from tests import PACKAGING_22
+from pdm.utils import PACKAGING_22
 
 
 @pytest.mark.parametrize(