Repository: pdm-project/pdm (python)

Issue:
allow binding between source index and individual dependencies
## Scenario

It is already possible to define more then one pypi-like index source. Since v2.4.0 there are 2 ways to do that (AFAIK). In [`pyproject.toml`](https://pdm.fming.dev/latest/pyproject/tool-pdm/#specify-other-sources-for-finding-packages) and in [`config.toml`](https://pdm.fming.dev/latest/usage/project/#configure-pypi-indexes).

It is also possible to define priority for these sources to some extend. In case of pyproject.toml, it is possible to utilize the [order resolution](https://pdm.fming.dev/latest/pyproject/tool-pdm/#respect-the-order-of-the-sources). In case of config.toml, it is possible to inverse priority by defining internal/alternative source within `[pypi]` section and pypi itself in `[pypi.extra]` section.

This is insufficient, because there are lot of deficiencies in the dependency resolution process. 

### Consider this:
Lets use v2.4.0 and define internal server in config.toml in `[pypi.extra]` section. If the internal python package shares its name with some public package on pypi, pdm will resolve and download against pypi, which is wrong for the company. Naming differently is not the right approach, because it is not deterministic -> even though the company chooses unique name, there is no certainty, that the same name will not be published to pypi later.

So we swap the sections and define internal server in `[pypi]` instead. This solves the priority problem, but introduces http-rest flood problem. If the project uses 100 public packages and only 1 internal, then 99 packages will be first resolved against internal server with no success and only then against public pypi. Which is waste of network resources.

There are more subtle quirks as well. E.g. if access token for internal server expires, pdm resolution against internal server fails with credentials error, but the resolution continues and if there is public package on pypi with the same name, it falls-back to it and download and install it, which is adverse.


## Describe the solution you'd like

All problems mentioned above and possibly even more edge cases I haven't bumped into yet, would be solved, if there was a way how to bind individual dependencies against specific (array of) indexes.

E.g. `pdm add <package> --only-source=internal`

Such flag would cause all related actions to be bonded exclusively to that source. So, `pdm install`; `pdm lock`; etc. would fail if that package was not found on that source, or if credentials didn't work, etc..

Pull request:
feat: per-package source configuration
Signed-off-by: Frost Ming <me@frostming.com>

## Pull Request Checklist

- [ ] A news fragment is added in `news/` describing what is new.
- [ ] Test cases added for changed code.

## Describe what you have changed in this PR.

Resolve #1645 

Patch:
diff --git a/docs/docs/usage/config.md b/docs/docs/usage/config.md
index 7a4f7650..7421070d 100644
--- a/docs/docs/usage/config.md
+++ b/docs/docs/usage/config.md
@@ -149,6 +149,22 @@ In some cases you may want to return packages from the preferred source, and sea
 respect-source-order = true
 ```
 
+### Specify index for individual packages
+
+You can bind packages to specific sources with `include_packages` and `exclude_packages` config under `tool.pdm.source` table.
+
+```toml
+[[tool.pdm.source]]
+name = "private"
+url = "https://private.pypi.org/simple"
+include_packages = ["foo", "foo-*"]
+exclude_packages = ["bar-*"]
+```
+
+With the above configuration, any package matching `foo` or `foo-*` will only be searched from the `private` index, and any package matching `bar-*` will be searched from all indexes except `private`.
+
+Both `include_packages` and `exclude_packages` are optional and accept a list of glob patterns, and `include_packages` takes effect exclusively when the pattern matches.
+
 ### Store credentials with the index
 
 You can specify credentials in the URL with `${ENV_VAR}` variable expansion and these variables will be read from the environment variables:
diff --git a/docs/docs/usage/venv.md b/docs/docs/usage/venv.md
index d6dd655f..fc16a5e6 100644
--- a/docs/docs/usage/venv.md
+++ b/docs/docs/usage/venv.md
@@ -87,8 +87,11 @@ Instead of spawning a subshell like what `pipenv` and `poetry` do, `pdm venv` do
     ```bash
     $ eval $(pdm venv activate for-test)
     (test-project-for-test) $  # Virtualenv entered
-    Fish
+    ```
+
+=== "Fish"
 
+    ```bash
     $ eval (pdm venv activate for-test)
     ```
 
diff --git a/news/1645.feature.md b/news/1645.feature.md
new file mode 100644
index 00000000..5bccc29b
--- /dev/null
+++ b/news/1645.feature.md
@@ -0,0 +1,1 @@
+Allow binding packages to specific sources with `include_packages` and `exclude_packages` config under `tool.pdm.source` table.
diff --git a/src/pdm/_types.py b/src/pdm/_types.py
index 61669bf5..265dd7d8 100644
--- a/src/pdm/_types.py
+++ b/src/pdm/_types.py
@@ -1,12 +1,17 @@
 from __future__ import annotations
 
 import dataclasses as dc
+import re
 from typing import TYPE_CHECKING, Any, Dict, List, NamedTuple, Tuple, TypeVar, Union
 
 if TYPE_CHECKING:
     from typing import Protocol
 
 
+def _normalize_pattern(pattern: str) -> str:
+    return re.sub(r"[^A-Za-z0-9?*\[\]-]+", "-", pattern).lower()
+
+
 @dc.dataclass
 class _RepositoryConfig:
     """Private dataclass to be subclassed"""
@@ -20,6 +25,12 @@ class _RepositoryConfig:
     verify_ssl: bool | None = None
     type: str | None = None
     ca_certs: str | None = None
+    include_packages: list[str] = dc.field(default_factory=list)
+    exclude_packages: list[str] = dc.field(default_factory=list)
+
+    def __post_init__(self) -> None:
+        self.include_packages = [_normalize_pattern(p) for p in self.include_packages]
+        self.exclude_packages = [_normalize_pattern(p) for p in self.exclude_packages]
 
 
 class RepositoryConfig(_RepositoryConfig):
diff --git a/src/pdm/models/repositories.py b/src/pdm/models/repositories.py
index 70a000b6..9934583f 100644
--- a/src/pdm/models/repositories.py
+++ b/src/pdm/models/repositories.py
@@ -37,7 +37,6 @@ if TYPE_CHECKING:
 
     CandidateKey = tuple[str, str | None, str | None, bool]
 
-ALLOW_ALL_PYTHON = PySpecSet()
 T = TypeVar("T", bound="BaseRepository")
 
 
@@ -77,7 +76,22 @@ class BaseRepository:
 
     def get_filtered_sources(self, req: Requirement) -> list[RepositoryConfig]:
         """Get matching sources based on the index attribute."""
-        return self.sources
+        source_preferences = [(s, self.source_preference(req, s)) for s in self.sources]
+        included_by = [s for s, p in source_preferences if p is True]
+        if included_by:
+            return included_by
+        return [s for s, p in source_preferences if p is None]
+
+    @staticmethod
+    def source_preference(req: Requirement, source: RepositoryConfig) -> bool | None:
+        key = req.key
+        if key is None:
+            return None
+        if any(fnmatch.fnmatch(key, pat) for pat in source.include_packages):
+            return True
+        if any(fnmatch.fnmatch(key, pat) for pat in source.exclude_packages):
+            return False
+        return None
 
     def get_dependencies(self, candidate: Candidate) -> tuple[list[Requirement], PySpecSet, str]:
         """Get (dependencies, python_specifier, summary) of the candidate."""
@@ -336,10 +350,10 @@ class BaseRepository:
         respect_source_order = self.environment.project.pyproject.settings.get("resolution", {}).get(
             "respect-source-order", False
         )
+        sources = self.get_filtered_sources(candidate.req)
         if req.is_named and respect_source_order and comes_from:
-            sources = [s for s in self.sources if comes_from.startswith(s.url)]
-        else:
-            sources = self.sources
+            sources = [s for s in sources if comes_from.startswith(s.url)]
+
         with self.environment.get_finder(sources, self.ignore_compatibility) as finder:
             if req.is_file_or_url:
                 this_link = cast("Link", candidate.prepare(self.environment).link)


Test patch:
diff --git a/tests/test_project.py b/tests/test_project.py
index b8623412..06b27375 100644
--- a/tests/test_project.py
+++ b/tests/test_project.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import os
 import sys
 import venv
@@ -363,3 +365,27 @@ def test_ignore_package_warning(pdm, project, recwarn, pattern, suppressed):
 
     assert result.exit_code == 0
     assert (len(recwarn) == 0) is suppressed
+
+
+def test_filter_sources_with_config(project):
+    project.pyproject.settings["source"] = [
+        {"name": "source1", "url": "https://source1.org/simple", "include_packages": ["foo", "foo-*"]},
+        {
+            "name": "source2",
+            "url": "https://source2.org/simple",
+            "include_packages": ["foo-bar", "bar*"],
+            "exclude_packages": ["baz-*"],
+        },
+        {"name": "pypi", "url": "https://pypi.org/simple"},
+    ]
+    repository = project.get_repository()
+
+    def expect_sources(requirement: str, expected: list[str]) -> bool:
+        sources = repository.get_filtered_sources(parse_requirement(requirement))
+        assert sorted([source.name for source in sources]) == sorted(expected)
+
+    expect_sources("foo", ["source1"])
+    expect_sources("foo-baz", ["source1"])
+    expect_sources("foo-bar", ["source1", "source2"])
+    expect_sources("bar-extra", ["source2"])
+    expect_sources("baz-extra", ["source1", "pypi"])