Repository: pdm-project/pdm
Pull request #2924 (2024-06-05 04:38:46)

feat: PEP 723: running scripts with inline metadata
Signed-off-by: Frost Ming <me@frostming.com>

## Pull Request Checklist

- [ ] A news fragment is added in `news/` describing what is new.
- [ ] Test cases added for changed code.

## Describe what you have changed in this PR.


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/pdm-project/pdm/pull/2924?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) Report
Attention: Patch coverage is `87.09677%` with `8 lines` in your changes missing coverage. Please review.
> Project coverage is 85.32%. Comparing base [(`4a8cb63`)](https://app.codecov.io/gh/pdm-project/pdm/commit/4a8cb635665f2440b8429c4ead743cff33490997?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) to head [(`8f96b3e`)](https://app.codecov.io/gh/pdm-project/pdm/commit/8f96b3e07d2d1298a41806753efa96deaac6408a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project).

| [Files](https://app.codecov.io/gh/pdm-project/pdm/pull/2924?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) | Patch % | Lines |
|---|---|---|
| [src/pdm/cli/commands/run.py](https://app.codecov.io/gh/pdm-project/pdm/pull/2924?src=pr&el=tree&filepath=src%2Fpdm%2Fcli%2Fcommands%2Frun.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project#diff-c3JjL3BkbS9jbGkvY29tbWFuZHMvcnVuLnB5) | 90.56% | [2 Missing and 3 partials :warning: ](https://app.codecov.io/gh/pdm-project/pdm/pull/2924?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) |
| [src/pdm/cli/commands/venv/backends.py](https://app.codecov.io/gh/pdm-project/pdm/pull/2924?src=pr&el=tree&filepath=src%2Fpdm%2Fcli%2Fcommands%2Fvenv%2Fbackends.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project#diff-c3JjL3BkbS9jbGkvY29tbWFuZHMvdmVudi9iYWNrZW5kcy5weQ==) | 66.66% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/pdm-project/pdm/pull/2924?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2924      +/-   ##
==========================================
- Coverage   85.32%   85.32%   -0.01%     
==========================================
  Files         106      106              
  Lines       10630    10676      +46     
  Branches     2314     2326      +12     
==========================================
+ Hits         9070     9109      +39     
- Misses       1091     1094       +3     
- Partials      469      473       +4     
```

| [Flag](https://app.codecov.io/gh/pdm-project/pdm/pull/2924/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/pdm-project/pdm/pull/2924/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) | `85.09% <85.48%> (-0.02%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project#carryforward-flags-in-the-pull-request-comment) to find out more.


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/pdm-project/pdm/pull/2924?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project).


Patch:
diff --git a/docs/usage/scripts.md b/docs/usage/scripts.md
index 46c9144e..79f98d53 100644
--- a/docs/usage/scripts.md
+++ b/docs/usage/scripts.md
@@ -10,6 +10,68 @@ pdm run flask run -p 54321
 
 It will run `flask run -p 54321` in the environment that is aware of packages in your project environment.
 
+## Single-file Scripts
+
++++ 2.16.0
+
+PDM can run single-file scripts with [inline script metadata](https://peps.python.org/pep-0723/).
+
+The following is an example of a script with embedded metadata:
+
+```python
+# test_script.py
+# /// script
+# requires-python = ">=3.11"
+# dependencies = [
+#   "requests<3",
+#   "rich",
+# ]
+# ///
+
+import requests
+from rich.pretty import pprint
+
+resp = requests.get("https://peps.python.org/api/peps.json")
+data = resp.json()
+pprint([(k, v["title"]) for k, v in data.items()][:10])
+```
+
+When you run it with `pdm run test_script.py`, PDM will create a temporary environment with the specified dependencies installed and run the script:
+
+```python
+[
+│   ('1', 'PEP Purpose and Guidelines'),
+│   ('2', 'Procedure for Adding New Modules'),
+│   ('3', 'Guidelines for Handling Bug Reports'),
+│   ('4', 'Deprecation of Standard Modules'),
+│   ('5', 'Guidelines for Language Evolution'),
+│   ('6', 'Bug Fix Releases'),
+│   ('7', 'Style Guide for C Code'),
+│   ('8', 'Style Guide for Python Code'),
+│   ('9', 'Sample Plaintext PEP Template'),
+│   ('10', 'Voting Guidelines')
+]
+```
+Add `--reuse-env` option if you want to reuse the environment created last time.
+You can also add `[tool.pdm]` section to the script metadata to configure PDM. For example:
+
+```python
+# test_script.py
+# /// script
+# requires-python = ">=3.11"
+# dependencies = [
+#   "requests<3",
+#   "rich",
+# ]
+#
+# [[tool.pdm.source]]  # Use a custom index
+# url = "https://mypypi.org/simple"
+# name = "pypi"
+# ///
+```
+
+Read the [specification](https://packaging.python.org/en/latest/specifications/inline-script-metadata/#inline-script-metadata) for more details.
+
 ## User Scripts
 
 PDM also supports custom script shortcuts in the optional `[tool.pdm.scripts]` section of `pyproject.toml`.
diff --git a/news/2924.feature.md b/news/2924.feature.md
new file mode 100644
index 00000000..49afac59
--- /dev/null
+++ b/news/2924.feature.md
@@ -0,0 +1,1 @@
+Support PEP 723: running scripts with inline metadata in standalone environment with dependencies.
diff --git a/src/pdm/cli/commands/run.py b/src/pdm/cli/commands/run.py
index b196d71a..9250af51 100644
--- a/src/pdm/cli/commands/run.py
+++ b/src/pdm/cli/commands/run.py
@@ -9,7 +9,6 @@ import signal
 import subprocess
 import sys
 from pathlib import Path
-from types import FrameType
 from typing import TYPE_CHECKING, Mapping, NamedTuple, Sequence, cast
 
 from rich import print_json
@@ -20,13 +19,16 @@ from pdm.cli.hooks import HookManager
 from pdm.cli.options import skip_option, venv_option
 from pdm.cli.utils import check_project_file
 from pdm.exceptions import PdmUsageError
-from pdm.project import Project
 from pdm.signals import pdm_signals
 from pdm.utils import deprecation_warning, expand_env_vars, is_path_relative_to
 
 if TYPE_CHECKING:
+    from types import FrameType
     from typing import Any, Callable, Iterator, TypedDict
 
+    from pdm.environments import BaseEnvironment
+    from pdm.project import Project
+
     class EnvFileOptions(TypedDict, total=True):
         override: str
 
@@ -86,6 +88,29 @@ def interpolate(script: str, args: Sequence[str]) -> tuple[str, bool]:
     return script, args_interpolated
 
 
+_METADATA_REGEX = r"(?m)^# /// (?P<type>[a-zA-Z0-9-]+)$\s(?P<content>(^#(| .*)$\s)+)^# ///$"
+
+
+def read_script_metadata(script: str, section: str) -> dict[str, Any] | None:
+    # Adapted from https://packaging.python.org/en/latest/specifications/inline-script-metadata/
+    if sys.version_info >= (3, 11):
+        import tomllib
+    else:
+        import tomli as tomllib
+
+    matches = [m for m in re.finditer(_METADATA_REGEX, script) if m.group("type") == section]
+    if len(matches) > 1:
+        raise ValueError(f"Multiple {section} blocks found")
+    elif len(matches) == 1:
+        content = "".join(
+            line[2:] if line.startswith("# ") else line[1:]
+            for line in matches[0].group("content").splitlines(keepends=True)
+        )
+        return tomllib.loads(content)
+    else:
+        return None
+
+
 class Task(NamedTuple):
     kind: str
     name: str
@@ -121,8 +146,39 @@ class TaskRunner:
             self.project.scripts.get("_", {}) if self.project.scripts else {},
         )
         self.global_options = global_options.copy()
+        self.reuse_env = False
         self.hooks = hooks
 
+    def _get_script_env(self, script_file: str) -> BaseEnvironment:
+        import hashlib
+
+        from pdm.cli.commands.venv.backends import BACKENDS
+        from pdm.environments import PythonEnvironment
+        from pdm.installers.core import install_requirements
+        from pdm.models.venv import get_venv_python
+
+        with open(script_file, encoding="utf8") as f:
+            metadata = read_script_metadata(f.read(), "script")
+        if not metadata:
+            return self.project.environment
+        tool_config = metadata.pop("tool", {})
+        script_project = self.project.core.create_project()
+        script_project.pyproject.set_data({"project": metadata, "tool": tool_config})
+        venv_name = hashlib.md5(os.path.realpath(script_file).encode("utf-8")).hexdigest()
+        venv_backend = BACKENDS[script_project.config["venv.backend"]](script_project, None)
+        venv = venv_backend.get_location(None, venv_name)
+        if venv.exists() and self.reuse_env:
+            self.project.core.ui.info(f"Reusing existing script environment: {venv}", verbosity=termui.Verbosity.DETAIL)
+        else:
+            self.project.core.ui.info(f"Creating environment for script: {venv}", verbosity=termui.Verbosity.DETAIL)
+            venv = venv_backend.create(venv_name=venv_name, force=True)
+        env = PythonEnvironment(script_project, python=get_venv_python(venv).as_posix())
+        script_project._python = env.interpreter
+        env.project = script_project  # keep a strong reference to the project
+        if reqs := script_project.get_dependencies():
+            install_requirements(list(reqs.values()), env, clean=True)
+        return env
+
     def get_task(self, script_name: str) -> Task | None:
         """Get the task with the given name. Return None if not found."""
         if script_name not in self.project.scripts:
@@ -148,14 +204,14 @@ class TaskRunner:
             raise PdmUsageError(f"Unknown options for task {script_name}: {', '.join(unknown_options)}")
         return Task(kind, script_name, value, cast("TaskOptions", options))
 
-    def expand_command(self, command: str) -> str:
+    def expand_command(self, env: BaseEnvironment, command: str) -> str:
         expanded_command = os.path.expanduser(command)
         if expanded_command.replace(os.sep, "/").startswith(("./", "../")):
             abspath = os.path.abspath(expanded_command)
             if not os.path.isfile(abspath):
                 raise PdmUsageError(f"Command [success]'{command}'[/] is not a valid executable.")
             return abspath
-        result = self.project.environment.which(command)
+        result = env.which(command)
         if not result:
             raise PdmUsageError(f"Command [success]'{command}'[/] is not found in your PATH.")
         return result
@@ -193,7 +249,11 @@ class TaskRunner:
                 process_env = {**process_env, **dotenv_env}
             else:
                 process_env = {**dotenv_env, **process_env}
-        project_env = project.environment
+        if not shell and args[0].endswith(".py"):
+            project_env = self._get_script_env(os.path.expanduser(args[0]))
+        else:
+            check_project_file(project)
+            project_env = project.environment
         this_path = project_env.get_paths()["scripts"]
         process_env.update(project_env.process_env)
         if env:
@@ -207,8 +267,8 @@ class TaskRunner:
             command, *args = (expand_env_vars(arg, env=process_env) for arg in args)
             if command.endswith(".py"):
                 args = [command, *args]
-                command = str(project.environment.interpreter.executable)
-            expanded_command = self.expand_command(command)
+                command = str(project_env.interpreter.executable)
+            expanded_command = self.expand_command(project_env, command)
             real_command = os.path.realpath(expanded_command)
             process_cmd = [expanded_command, *args]
             if (
@@ -417,6 +477,9 @@ class Command(BaseCommand):
             action="store_true",
             help="Load site-packages from the selected interpreter",
         )
+        exec.add_argument(
+            "--reuse-env", action="store_true", help="Reuse the script environment for self-contained scripts"
+        )
         exec.add_argument("script", nargs="?", help="The command to run")
         exec.add_argument(
             "args",
@@ -425,21 +488,23 @@ class Command(BaseCommand):
         )
 
     def get_runner(self, project: Project, hooks: HookManager, options: argparse.Namespace) -> TaskRunner:
-        if (runner := getattr(self, "runner_cls", None)) is not None:  # pragma: no cover
+        if (runner_cls := getattr(self, "runner_cls", None)) is not None:  # pragma: no cover
             deprecation_warning("runner_cls attribute is deprecated, use get_runner method instead.")
-            return cast("type[TaskRunner]", runner)(project, hooks)
-        return TaskRunner(project, hooks)
+            runner = cast("type[TaskRunner]", runner_cls)(project, hooks)
+        else:
+            runner = TaskRunner(project, hooks)
+        runner.reuse_env = options.reuse_env
+        if options.site_packages:
+            runner.global_options["site_packages"] = True
+        return runner
 
     def handle(self, project: Project, options: argparse.Namespace) -> None:
-        check_project_file(project)
         hooks = HookManager(project, options.skip)
         runner = self.get_runner(project, hooks, options)
         if options.list:
             return runner.show_list()
         if options.json:
             return print_json(data=runner.as_json())
-        if options.site_packages:
-            runner.global_options.update({"site_packages": options.site_packages})
         if not options.script:
             project.core.ui.warn("No command is given, default to the Python REPL.")
             options.script = "python"
diff --git a/src/pdm/cli/commands/venv/backends.py b/src/pdm/cli/commands/venv/backends.py
index 2f6b33e9..cbc1e6fa 100644
--- a/src/pdm/cli/commands/venv/backends.py
+++ b/src/pdm/cli/commands/venv/backends.py
@@ -84,10 +84,12 @@ class Backend(abc.ABC):
         if not force:
             raise VirtualenvCreateError(f"The location {location} is not empty, add --force to overwrite it.")
         if location.is_file():
-            self.project.core.ui.echo(f"Removing existing file {location}", err=True)
+            self.project.core.ui.info(f"Removing existing file {location}", verbosity=termui.Verbosity.DETAIL)
             location.unlink()
         else:
-            self.project.core.ui.echo(f"Cleaning existing target directory {location}", err=True)
+            self.project.core.ui.info(
+                f"Cleaning existing target directory {location}", verbosity=termui.Verbosity.DETAIL
+            )
             with os.scandir(location) as entries:
                 for entry in entries:
                     if entry.is_dir() and not entry.is_symlink():
@@ -95,11 +97,15 @@ class Backend(abc.ABC):
                     else:
                         os.remove(entry.path)
 
-    def get_location(self, name: str | None) -> Path:
+    def get_location(self, name: str | None = None, venv_name: str | None = None) -> Path:
+        if name and venv_name:
+            raise PdmUsageError("Cannot specify both name and venv_name")
         venv_parent = Path(self.project.config["venv.location"]).expanduser()
         if not venv_parent.is_dir():
             venv_parent.mkdir(exist_ok=True, parents=True)
-        return venv_parent / f"{get_venv_prefix(self.project)}{name or self.ident}"
+        if not venv_name:
+            venv_name = f"{get_venv_prefix(self.project)}{name or self.ident}"
+        return venv_parent / venv_name
 
     def create(
         self,
@@ -109,11 +115,12 @@ class Backend(abc.ABC):
         in_project: bool = False,
         prompt: str | None = None,
         with_pip: bool = False,
+        venv_name: str | None = None,
     ) -> Path:
         if in_project:
             location = self.project.root / ".venv"
         else:
-            location = self.get_location(name)
+            location = self.get_location(name, venv_name)
         args = (*self.pip_args(with_pip), *args)
         if prompt is not None:
             prompt = prompt.format(
diff --git a/src/pdm/cli/completions/pdm.bash b/src/pdm/cli/completions/pdm.bash
index f2864088..ab470d8b 100644
--- a/src/pdm/cli/completions/pdm.bash
+++ b/src/pdm/cli/completions/pdm.bash
@@ -105,7 +105,7 @@ _pdm_a919b69078acdf0a_complete()
             ;;
 
             (run)
-            opts="--global --help --json --list --project --quiet --site-packages --skip --venv --verbose"
+            opts="--global --help --json --list --project --quiet --reuse-env --site-packages --skip --venv --verbose"
             ;;
 
             (search)
diff --git a/src/pdm/cli/completions/pdm.fish b/src/pdm/cli/completions/pdm.fish
index 8a29ed86..1a702ff8 100644
--- a/src/pdm/cli/completions/pdm.fish
+++ b/src/pdm/cli/completions/pdm.fish
@@ -400,6 +400,7 @@ complete -c pdm -A -n '__fish_seen_subcommand_from run' -l json -d 'Output all s
 complete -c pdm -A -n '__fish_seen_subcommand_from run' -l list -d 'Show all available scripts defined in pyproject.toml'
 complete -c pdm -A -n '__fish_seen_subcommand_from run' -l project -d 'Specify another path as the project root, which changes the base of pyproject.toml and __pypackages__ [env var: PDM_PROJECT]'
 complete -c pdm -A -n '__fish_seen_subcommand_from run' -l quiet -d 'Suppress output'
+complete -c pdm -A -n '__fish_seen_subcommand_from run' -l reuse-env -d 'Reuse the script environment for self-contained scripts'
 complete -c pdm -A -n '__fish_seen_subcommand_from run' -l site-packages -d 'Load site-packages from the selected interpreter'
 complete -c pdm -A -n '__fish_seen_subcommand_from run' -l skip -d 'Skip some tasks and/or hooks by their comma-separated names. Can be supplied multiple times. Use ":all" to skip all hooks. Use ":pre" and ":post" to skip all pre or post hooks.'
 complete -c pdm -A -n '__fish_seen_subcommand_from run' -l venv -d 'Run the command in the virtual environment with the given key. [env var: PDM_IN_VENV]'
diff --git a/src/pdm/cli/completions/pdm.ps1 b/src/pdm/cli/completions/pdm.ps1
index 653f9519..adefd861 100644
--- a/src/pdm/cli/completions/pdm.ps1
+++ b/src/pdm/cli/completions/pdm.ps1
@@ -430,7 +430,7 @@ function TabExpansion($line, $lastWord) {
             "run" {
                 $completer.AddOpts(
                     @(
-                        [Option]::new(@("--global", "-g", "-l", "--list", "-s", "--site-packages", "--json")),
+                        [Option]::new(@("--global", "-g", "-l", "--list", "-s", "--site-packages", "--json", "--reuse-env")),
                         $skipOption,
                         $venvOption,
                         $projectOption
diff --git a/src/pdm/cli/completions/pdm.zsh b/src/pdm/cli/completions/pdm.zsh
index 0ed0daf5..9c5e3844 100644
--- a/src/pdm/cli/completions/pdm.zsh
+++ b/src/pdm/cli/completions/pdm.zsh
@@ -400,6 +400,7 @@ _pdm() {
         {-g,--global}'[Use the global project, supply the project root with `-p` option]' \
         {-l,--list}'[Show all available scripts defined in pyproject.toml]' \
         '--json[Output all scripts infos in JSON]' \
+        '--reuse-env[Reuse the script environment for self-contained scripts]' \
         {-k,--skip}'[Skip some tasks and/or hooks by their comma-separated names]' \
         {-s,--site-packages}'[Load site-packages from the selected interpreter]' \
         '--venv[Run the command in the virtual environment with the given key. \[env var: PDM_IN_VENV\]]:venv:' \


Test patch:
diff --git a/tests/cli/test_run.py b/tests/cli/test_run.py
index 4b4e4af5..52846d54 100644
--- a/tests/cli/test_run.py
+++ b/tests/cli/test_run.py
@@ -10,7 +10,7 @@ import pytest
 from pdm import termui
 from pdm.cli import actions
 from pdm.cli.utils import get_pep582_path
-from pdm.utils import cd
+from pdm.utils import cd, path_to_url
 
 
 @pytest.fixture
@@ -945,3 +945,40 @@ def test_run_script_changing_working_dir(project, pdm, capfd):
     capfd.readouterr()
     pdm(["run", "test_script"], obj=project, strict=True)
     assert capfd.readouterr()[0].strip() == "Hello world"
+
+
+def test_run_script_with_inline_metadata(project, pdm, local_finder, local_finder_artifacts):
+    with cd(project.root):
+        project.root.joinpath("test_script.py").write_text(
+            textwrap.dedent("""\
+            from first import first
+
+            assert first([0, False, 1, 2]) == 1
+            """)
+        )
+        result = pdm(["run", "test_script.py"], obj=project)
+        assert result.exit_code != 0
+
+    local_artifacts_url = path_to_url(str(local_finder_artifacts))
+
+    project.root.joinpath("test_script.py").write_text(
+        textwrap.dedent(f"""\
+        # /// script
+        # requires-python = ">=3.8"
+        # dependencies = [
+        #   "first",
+        # ]
+        #
+        # [[tool.pdm.source]]
+        # name = "pypi"
+        # url = "{local_artifacts_url}"
+        # type = "find_links"
+        # ///
+        from first import first
+
+        assert first([0, False, 1, 2]) == 1
+        """)
+    )
+    with cd(project.root):
+        result = pdm(["run", "test_script.py"], obj=project)
+        assert result.exit_code == 0