Repository: jazzband/pip-tools

Issue:
Recursive extra dependencies don't work for a private package
Installing private package (=package not published on pypi) is possible with pip using `pip install -e .[dev]`. It respects extra dependencies and the extra dependencies can even reference themselves (see the example bellow). However compiling such a project to get `requirements.txt` fails.

#### Environment Versions

1. OS Type: ubuntu 18.04
1. Python version: `$ python -V`:  Python 3.8.0
1. pip version: `$ pip --version`: pip 22.2.2
1. pip-tools version: `$ pip-compile --version`: version 6.8.0

#### Steps to replicate

Minimal pyproject.toml:

```
[project]
name = "privateproject"
version = "0.0.1"
dependencies = [
    "tomli",
]

[project.optional-dependencies]
test = [
    "pep517",
]
dev = [
    "privateproject[test]",
]
```

1. save minimal `pyproject.toml`
2. run `python -m piptools compile --extra dev pyproject.toml`

#### Expected result

The `requirements.txt` is generated containing `tomli` and `pep517`.

#### Actual result

```
Could not find a version that matches privateproject[test] (from privateproject (pyproject.toml))
No versions found
Was https://pypi.org/simple reachable?
```



Discussion:
It seems pip-tools compile does not source the dependencies from the current `pyproject.toml` being processed but from the last published package version on pypi - if so, that could be a problem also for public packages.
Please add the `[build-system]` part that you used too.
The above pyproject.toml file was used to get the error message (=no build-system set). In my project I use the following:

```
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
```

Wait, is your project an actual Python package? Like do you install it with `pip install -e .`?

Because 100% of my work projects don't exist on PyPI, but they must be proper packages if you want to use packaging-based tools.
> Wait, is your project an actual Python package? Like do you install it with pip install -e .?

Yes, I do install it and it works. That is what I meant by:

> Installing private package (=package not published on pypi) is possible with pip using `pip install -e .[dev]`.

Since pip does install my package I figured pip-tools could be able compile the dependencies. Then I just trimmed the pyproject.toml to cut the parts not related with the issue while still getting the same error message while compiling. The result might no longer be installable by pip, I haven't checked that.

Self-dependencies are not supported by pip-tools for now. Fun fact: the feature was [unintentionally](https://github.com/pypa/pip/issues/10393#issuecomment-942771191) added in `pip==21.2`. I've also tried this with `pip-compile --resolver backtracking`, but no luck:
```
pip._internal.exceptions.DistributionNotFound: No matching distribution found for privateproject[test]
```
I think it would be a really nice feature to add for packages (ie.not apps) that want to pin their dependencies for CI stability but don't want to use poetry.

Are there technical hurdles or is it a matter of someone finding time to do it?
> Are there technical hurdles or is it a matter of someone finding time to do it?

Yeah, the latter one.
In pip it works because pip can detect that '.[dev]' contains a path, so it makes InstallRequirement with Link, and then the backtracking resolver makes from this InstallRequirement a LinkCandidate....
So you can do something like:
```python
            package_name = metadata.get_all('Name')[0]
            comes_from = f"{metadata.get_all('Name')[0]} ({src_file})"
            def _f(req):
                if package_name in req:
                    req = req.replace(package_name, ".")
                return install_req_from_line(req, comes_from)
            constraints.extend(
                [
                    _f(req)
                    for req in metadata.get_all("Requires-Dist") or []
                ]
            )
```
and it will produce this output
```
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --extra=dev --resolver=backtracking pyproject.toml
#
pep517==0.13.0
    # via privateproject
privateproject @ file:///tmp/t
    # via privateproject (pyproject.toml)
tomli==2.0.1
    # via
    #   pep517
    #   privateproject
    #   privateproject (pyproject.toml)
```
@atugushev what do you think?
@q0w looks good to me.

Resolving patch:
diff --git a/piptools/_compat/pip_compat.py b/piptools/_compat/pip_compat.py
--- a/piptools/_compat/pip_compat.py
+++ b/piptools/_compat/pip_compat.py
@@ -15,7 +15,6 @@
 
 PIP_VERSION = tuple(map(int, parse_version(pip.__version__).base_version.split(".")))
 
-
 __all__ = [
     "dist_requires",
     "uses_pkg_resources",
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -30,6 +30,7 @@
     drop_extras,
     is_pinned_requirement,
     key_from_ireq,
+    parse_requirements_from_wheel_metadata,
 )
 from ..writer import OutputWriter
 
@@ -500,13 +501,13 @@ def cli(
                 log.error(str(e))
                 log.error(f"Failed to parse {os.path.abspath(src_file)}")
                 sys.exit(2)
-            comes_from = f"{metadata.get_all('Name')[0]} ({src_file})"
+
             constraints.extend(
-                [
-                    install_req_from_line(req, comes_from=comes_from)
-                    for req in metadata.get_all("Requires-Dist") or []
-                ]
+                parse_requirements_from_wheel_metadata(
+                    metadata=metadata, src_file=src_file
+                )
             )
+
             if all_extras:
                 if extras:
                     msg = "--extra has no effect when used with --all-extras"
diff --git a/piptools/utils.py b/piptools/utils.py
--- a/piptools/utils.py
+++ b/piptools/utils.py
@@ -7,12 +7,12 @@
 import os
 import re
 import shlex
-from typing import Any, Callable, Iterable, Iterator, TypeVar, cast
+from typing import TYPE_CHECKING, Any, Callable, Iterable, Iterator, TypeVar, cast
 
 import click
 from click.utils import LazyFile
 from pip._internal.req import InstallRequirement
-from pip._internal.req.constructors import install_req_from_line
+from pip._internal.req.constructors import install_req_from_line, parse_req_from_line
 from pip._internal.utils.misc import redact_auth_from_url
 from pip._internal.vcs import is_url
 from pip._vendor.packaging.markers import Marker
@@ -24,6 +24,12 @@
 from piptools._compat import PIP_VERSION
 from piptools.subprocess_utils import run_python_snippet
 
+if TYPE_CHECKING:
+    from typing import Protocol
+else:
+    Protocol = object
+
+
 _KT = TypeVar("_KT")
 _VT = TypeVar("_VT")
 _T = TypeVar("_T")
@@ -485,3 +491,34 @@ def copy_install_requirement(
     )
 
     return ireq
+
+
+class PackageMetadata(Protocol):
+    def get_all(self, name: str, failobj: _T = ...) -> list[str] | _T:
+        ...
+
+
+def parse_requirements_from_wheel_metadata(
+    metadata: PackageMetadata, src_file: str
+) -> Iterator[InstallRequirement]:
+    package_name = metadata.get_all("Name")[0]
+    comes_from = f"{package_name} ({src_file})"
+
+    for req in metadata.get_all("Requires-Dist") or []:
+        parts = parse_req_from_line(req, comes_from)
+        if parts.requirement.name == package_name:
+            package_dir = os.path.dirname(os.path.abspath(src_file))
+            # Replace package name with package directory in the requirement
+            # string so that pip can find the package as self-referential.
+            # Note the string can contain extras, so we need to replace only
+            # the package name, not the whole string.
+            replaced_package_name = req.replace(package_name, package_dir, 1)
+            parts = parse_req_from_line(replaced_package_name, comes_from)
+
+        yield InstallRequirement(
+            parts.requirement,
+            comes_from,
+            link=parts.link,
+            markers=parts.markers,
+            extras=parts.extras,
+        )


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -2868,3 +2868,39 @@ def test_pass_pip_cache_to_pip_args(tmpdir, runner, current_resolver):
     )
     assert out.exit_code == 0
     assert os.listdir(os.path.join(str(cache_dir), "http"))
+
+
+@backtracking_resolver_only
+def test_compile_recursive_extras(runner, tmp_path, current_resolver):
+    (tmp_path / "pyproject.toml").write_text(
+        dedent(
+            """
+            [project]
+            name = "foo"
+            version = "0.0.1"
+            dependencies = ["small-fake-a"]
+            [project.optional-dependencies]
+            footest = ["small-fake-b"]
+            dev = ["foo[footest]"]
+            """
+        )
+    )
+    out = runner.invoke(
+        cli,
+        [
+            "--no-header",
+            "--no-annotate",
+            "--no-emit-find-links",
+            "--extra",
+            "dev",
+            "--find-links",
+            os.fspath(MINIMAL_WHEELS_PATH),
+            os.fspath(tmp_path / "pyproject.toml"),
+        ],
+    )
+    expected = rf"""foo @ {tmp_path.as_uri()}
+small-fake-a==0.2
+small-fake-b==0.3
+"""
+    assert out.exit_code == 0
+    assert expected == out.stderr