Repository: pdm-project/pdm

Issue:
Supporting adding `extra` annotations to the exported `requirements` file
## Is your feature/enhancement proposal related to a problem? Please describe.

I am using https://github.com/bazelbuild/rules_python to manage my Python project and I noticed that the `pdm` is a really nice way to generate a cross-platform lock file `pdm.lock`. Moreover, it also has a feature to export the lock file to `requirements.txt`, which is `cross-platform`, i.e. it has annotations on what should be installed on which platform. Having a single lock file for multiple platforms would be great, but unfortunately it does not work at the moment with `rules_python` because the extras are removed as per #741.

At least for now, the latest `rules_python` version relies on those `extras` annotations to be present in the `requirements` file and I would love `pdm` to be able to provide those too.

## Describe the solution you'd like

A CLI flag for the `pdm export --format requirements` command to add `--extras`.


Discussion:
Can you explain more on how the extra info is used?

Given this requirements
```
pydantic==1.8.2
pydantic[email]==1.8.2
```
Note all other transitive dependencies are also exported but omitted. Why does `rules-python` need the second entry?

Because from the perspective of an installer, there is not any difference between whether you use the extras tag, or specify the extra dependencies manually.
Right now it only needs the second entry, but we can handle a requirements file with both entries.

To be concrete, `rules_python` is parsing each entry in the `requirements.txt` and is generating a `BUILD.bazel` file with contents similar to below:
```starlark
filegroup(name="whl", src=["pydantic-...-any.whl"])
py_library(
    name = "pkg",
    srcs = [
        # All of the extracted whl contents
    ],
    deps = [
        # deps for the python sources. Only dependencies for the requested extras are included
        # which means that `pydantic==1.8.2` would yield a subset of dependencies and
        # `pydantic[email]==1.8.2` would yield the correct set of deps
    ]
)
```

The `deps` generation for the example above is done by:
1. Extracting the requested deps from the `requirements.txt` entry (i.e. parse the `pydantic[email,foo]==1.8.2 --hashes=sha256:...` line to get `["email", "foo"]`
2. Parsing the `whl` `METADATA` file to get the `Requires-Dist` and generating the applicable dependencies for the target platform.

In the ideal case, we should consume `pdm.lock` directly, but I am interested in this feature to act as a stop-gap solution, where the `rules_python` does not yet support `pdm.lock` file but the users can start using `pdm` with `rules_python` via exporting `requirements.txt` file using `pdm export --format requirements --dev --with-extras requirement_lock.txt`.

Let me know if you'd like to know more details or if the text above is a little too detailed/confusing.
> 2\. Parsing the `whl` `METADATA` file to get the `Requires-Dist` and generating the applicable dependencies for the target platform.

But exported requirements should contain all transitive dependencies, so even it parses `pydantic==1.8.2`, the extra dependencies are still pinned. Or does it only consider the dependencies in `pyproject.toml`?
`rules_python` cannot read dependencies from `pyproject.toml` yet and we depend on all of the transitive dependencies being in the `requirements.txt` file with their hashes. Something similar to [this](https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/tests/dupe_requirements/requirements.txt).

Ideal file would contain only `pyjwt[crypto]` and the linked example is just a test to see that we can handle a `requirements.txt` file similar to the one output by `poetry` export command.

The `Requires-Dist` parsing I mentioned previously only considers the package names and environment markers, i.e. it does not do transitive dependency pinning.
I still can't get why it is necessary.

Let's take `pydantic[email]==1.8.2` as an example. Here is what `pdm export --no-hashes` outputs:

```
dnspython==2.4.2  pydantic[email]
email-validator==2.1.0.post1  # from pydantic[email]
idna==3.6  # pydantic[email]
pydantic==1.8.2
typing-extensions==4.9.0
```

All dependencies are pinned, does `rules-python` pick up all dependencies from the above list? what is missing and what else `pydantic[email]` can provide?
Just to clarify, `rules_python` currently needs:
* All of the transitive dependencies, because we are downloading each package separately from PyPI and other sources. Bazel has its own sandbox and that is how it currently consumes Python dependencies.
* Hashes for all of the dependencies. Bazel needs to have integrity checksums for each source that it consumes so that if a dependency within the transitive dependency closure changes, it can refetch the required dependencies and rerun tests.
* The required extras for each of the requirement, so that each dependency that is fetched can have the correct runtime dependency lists.

I am not sure if there is a CLI option combination. I understand that supporting this use case might be out of scope of the tool, so let me know if supporting this aligns with the future of `pdm` you have in mind.
> I still can't get why it is necessary.
> 
> Let's take `pydantic[email]==1.8.2` as an example. Here is what `pdm export --no-hashes` outputs:
> 
> ```
> dnspython==2.4.2  pydantic[email]
> email-validator==2.1.0.post1  # from pydantic[email]
> idna==3.6  # pydantic[email]
> pydantic==1.8.2
> typing-extensions==4.9.0
> ```
> 
> All dependencies are pinned, does `rules-python` pick up all dependencies from the above list? what is missing and what else `pydantic[email]` can provide?

Is that behavior part of a PEP? It seems redundant and counterintuitive (If I'm not misunderstanding it).



Patch:
diff --git a/news/2519.feature.md b/news/2519.feature.md
new file mode 100644
index 0000000000..fb97da181c
--- /dev/null
+++ b/news/2519.feature.md
@@ -0,0 +1,1 @@
+Add `--no-extras` to `pdm export` to strip extras from the requirements. Now the default behavior is to keep extras.
diff --git a/src/pdm/cli/commands/export.py b/src/pdm/cli/commands/export.py
index 7a8da04032..8e67b401a6 100644
--- a/src/pdm/cli/commands/export.py
+++ b/src/pdm/cli/commands/export.py
@@ -11,7 +11,7 @@
 from pdm.exceptions import PdmUsageError
 from pdm.formats import FORMATS
 from pdm.models.candidates import Candidate
-from pdm.models.requirements import Requirement
+from pdm.models.requirements import Requirement, strip_extras
 from pdm.project import Project
 
 
@@ -39,6 +39,9 @@ def add_arguments(self, parser: argparse.ArgumentParser) -> None:
         parser.add_argument(
             "--no-markers", action="store_false", default=True, dest="markers", help="Don't include platform markers"
         )
+        parser.add_argument(
+            "--no-extras", action="store_false", default=True, dest="extras", help="Strip extras from the requirements"
+        )
         parser.add_argument(
             "-o",
             "--output",
@@ -73,9 +76,18 @@ def handle(self, project: Project, options: argparse.Namespace) -> None:
             candidates = resolve_candidates_from_lockfile(
                 project, requirements.values(), groups=set(selection), cross_platform=options.markers
             )
-            # Remove candidates with [extras] because the bare candidates are already
-            # included
-            packages = (candidate for candidate in candidates.values() if not candidate.req.extras)
+
+            filtered_candidates: dict[str, Candidate] = {}
+            for k, candidate in candidates.items():
+                if options.extras:
+                    if candidate.req.extras:
+                        k = strip_extras(k)[0]
+                    elif k in filtered_candidates:
+                        continue
+                elif candidate.req.extras:
+                    continue
+                filtered_candidates[k] = candidate
+            packages = filtered_candidates.values()
 
         content = FORMATS[options.format].export(project, packages, options)
         if options.output:
diff --git a/src/pdm/cli/completions/pdm.bash b/src/pdm/cli/completions/pdm.bash
index ab470d8b0c..533b45779a 100644
--- a/src/pdm/cli/completions/pdm.bash
+++ b/src/pdm/cli/completions/pdm.bash
@@ -49,7 +49,7 @@ _pdm_a919b69078acdf0a_complete()
             ;;
 
             (export)
-            opts="--dev --editable-self --expandvars --format --global --group --help --lockfile --no-default --no-markers --output --production --project --pyproject --quiet --self --verbose --without --without-hashes"
+            opts="--dev --editable-self --expandvars --format --global --group --help --lockfile --no-default --no-extras --no-markers --output --production --project --pyproject --quiet --self --verbose --without --without-hashes"
             ;;
 
             (fix)
diff --git a/src/pdm/cli/completions/pdm.fish b/src/pdm/cli/completions/pdm.fish
index 1a702ff8e2..3c380840f8 100644
--- a/src/pdm/cli/completions/pdm.fish
+++ b/src/pdm/cli/completions/pdm.fish
@@ -125,6 +125,7 @@ complete -c pdm -A -n '__fish_seen_subcommand_from export' -l group -d 'Select g
 complete -c pdm -A -n '__fish_seen_subcommand_from export' -l help -d 'Show this help message and exit.'
 complete -c pdm -A -n '__fish_seen_subcommand_from export' -l lockfile -d 'Specify another lockfile path. Default: pdm.lock. [env var: PDM_LOCKFILE]'
 complete -c pdm -A -n '__fish_seen_subcommand_from export' -l no-default -d 'Don\'t include dependencies from the default group'
+complete -c pdm -A -n '__fish_seen_subcommand_from export' -l no-extras -d 'Strip extras from the requirements'
 complete -c pdm -A -n '__fish_seen_subcommand_from export' -l no-markers -d 'Don\'t include platform markers'
 complete -c pdm -A -n '__fish_seen_subcommand_from export' -l output -d 'Write output to the given file, or print to stdout if not given'
 complete -c pdm -A -n '__fish_seen_subcommand_from export' -l production -d 'Unselect dev dependencies'
diff --git a/src/pdm/cli/completions/pdm.ps1 b/src/pdm/cli/completions/pdm.ps1
index adefd861f2..0faaee2a40 100644
--- a/src/pdm/cli/completions/pdm.ps1
+++ b/src/pdm/cli/completions/pdm.ps1
@@ -253,7 +253,7 @@ function TabExpansion($line, $lastWord) {
                 $completer.AddOpts(@(
                         [Option]::new(@(
                             "--dev", "--output", "--global", "--no-default", "--expandvars", "--prod", "--production", "-g", "-d", "-o",
-                            "--no-hashes", "--no-markers", "-L", "--lockfile", "--self", "--editable-self"
+                            "--no-hashes", "--no-markers", "-L", "--lockfile", "--self", "--editable-self", "--no-extras"
                         )),
                         $formatOption,
                         $sectionOption,
diff --git a/src/pdm/cli/completions/pdm.zsh b/src/pdm/cli/completions/pdm.zsh
index 9c5e38449a..c045f4d86d 100644
--- a/src/pdm/cli/completions/pdm.zsh
+++ b/src/pdm/cli/completions/pdm.zsh
@@ -156,6 +156,7 @@ _pdm() {
         {-f+,--format+}"[Only requirements.txt is supported for now.]:format:(requirements)"
         "--no-hashes[Don't include artifact hashes]"
         "--no-markers[Don't include platform markers]"
+        "--no-extras[Strip extras from the requirements]"
         "--expandvars[Expand environment variables in requirements]"
         "--self[Include the project itself]"
         "--editable-self[Include the project itself as an editable dependency]"


Test patch:
diff --git a/tests/cli/test_others.py b/tests/cli/test_others.py
index 7a5123dc21..04321474e2 100644
--- a/tests/cli/test_others.py
+++ b/tests/cli/test_others.py
@@ -202,13 +202,16 @@ def test_export_to_requirements_txt(pdm, fixture_project):
     assert (project.root / "requirements_output.txt").read_text() == requirements_txt.read_text()
 
 
-def test_export_doesnt_include_dep_with_extras(pdm, fixture_project):
+@pytest.mark.parametrize("extra_opt", [[], ["--no-extras"]])
+def test_export_doesnt_include_dep_with_extras(pdm, fixture_project, extra_opt):
     project = fixture_project("demo-package-has-dep-with-extras")
-    requirements_txt = project.root / "requirements.txt"
 
-    result = pdm(["export", "--without-hashes"], obj=project)
+    result = pdm(["export", "--without-hashes", *extra_opt], obj=project)
     assert result.exit_code == 0
-    assert result.output.strip() == requirements_txt.read_text().strip()
+    if extra_opt:
+        assert "requests==2.26.0" in result.output.splitlines()
+    else:
+        assert "requests[security]==2.26.0" in result.output.splitlines()
 
 
 def test_completion_command(pdm):