Repository: pdm-project/pdm

Issue:
 Environment variable substitution in `cmd` scripts
## Is your feature/enhancement proposal related to a problem? Please describe.

When I tried migrating a project's tasks from [Poe the Poet](https://poethepoet.natn.io), an independent task runner for Poetry, I ran into a feature that is important to me missing.

Poe can substitute environment variables in `cmd` tasks, which are like PDM `cmd` scripts. Here is an example of how I use it:

```toml
[tool.poe.env]
"PYTHON_SOURCES" = "src tests"

[tool.poe.tasks.format]
cmd = "ruff format ${PYTHON_SOURCES}"
help = "Run the Ruff formatter"

[tool.poe.tasks.lint]
cmd = "ruff check ${PYTHON_SOURCES}"
help = "Run Ruff"
```

I find this useful because it allows you to avoid repeating yourself without invoking an external shell. Why `shell` scripts aren't a sufficient alternative: the shell depends on the platform. On Windows this is still `cmd.exe`, which has an incompatible syntax for variable substitution.

## Describe the solution you'd like

I would like to be able to substitute environment variables as `{env:PYTHON_SOURCES}` or something similar consistent with existing PDM syntax (`{args}` and `{pdm}`) in `cmd` scripts.

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/pdm-project/pdm/pull/2544?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`6471521`)](https://app.codecov.io/gh/pdm-project/pdm/commit/6471521a5c161aba98c4d323dc457db0f2b80b9c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) 84.62% compared to head [(`6de9a47`)](https://app.codecov.io/gh/pdm-project/pdm/pull/2544?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) 84.62%.


<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2544   +/-   ##
=======================================
  Coverage   84.62%   84.62%           
=======================================
  Files         104      104           
  Lines       10346    10347    +1     
  Branches     2262     2263    +1     
=======================================
+ Hits         8755     8756    +1     
  Misses       1117     1117           
  Partials      474      474           
```

| [Flag](https://app.codecov.io/gh/pdm-project/pdm/pull/2544/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/pdm-project/pdm/pull/2544/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project) | `84.41% <100.00%> (+<0.01%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project#carryforward-flags-in-the-pull-request-comment) to find out more.


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/pdm-project/pdm/pull/2544?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pdm-project).


Patch:
diff --git a/news/2488.bugfix.md b/news/2488.bugfix.md
deleted file mode 100644
index 695c78cc..00000000
--- a/news/2488.bugfix.md
+++ /dev/null
@@ -1,1 +0,0 @@
-Fix a KeyError raised when resolving a URL dependency without package name given.
diff --git a/news/2495.bugfix.md b/news/2495.bugfix.md
deleted file mode 100644
index a7693b03..00000000
--- a/news/2495.bugfix.md
+++ /dev/null
@@ -1,1 +0,0 @@
-`pdm update --update-eager` can hit InconsistentCandidate error when dependency is included both through default dependencies and extra.
diff --git a/news/2502.bugfix.md b/news/2502.bugfix.md
deleted file mode 100644
index e80458b7..00000000
--- a/news/2502.bugfix.md
+++ /dev/null
@@ -1,1 +0,0 @@
-`pdm install` should not warn when overwriting its own symlinks on `install`/`update`.
diff --git a/news/2507.bugfix.md b/news/2507.bugfix.md
deleted file mode 100644
index 3c79cc4c..00000000
--- a/news/2507.bugfix.md
+++ /dev/null
@@ -1,1 +0,0 @@
-Fix a bug that candidates without local version are rejected when the local version is pinned.
diff --git a/news/2510.doc.md b/news/2510.doc.md
deleted file mode 100644
index a1f2132e..00000000
--- a/news/2510.doc.md
+++ /dev/null
@@ -1,1 +0,0 @@
-Add maturin as a compatible build backend in the docs.
diff --git a/news/2542.bugfix.md b/news/2542.bugfix.md
new file mode 100644
index 00000000..a8410ee9
--- /dev/null
+++ b/news/2542.bugfix.md
@@ -0,0 +1,1 @@
+Fix the environment variable substitution for `cmd` scripts.
diff --git a/src/pdm/cli/commands/run.py b/src/pdm/cli/commands/run.py
index 72237595..01587002 100644
--- a/src/pdm/cli/commands/run.py
+++ b/src/pdm/cli/commands/run.py
@@ -22,7 +22,7 @@ from pdm.cli.utils import check_project_file
 from pdm.exceptions import PdmUsageError
 from pdm.project import Project
 from pdm.signals import pdm_signals
-from pdm.utils import is_path_relative_to
+from pdm.utils import expand_env_vars, is_path_relative_to
 
 if TYPE_CHECKING:
     from typing import Any, Callable, Iterator, TypedDict
@@ -44,8 +44,8 @@ def exec_opts(*options: TaskOptions | None) -> dict[str, Any]:
     )
 
 
-RE_ARGS_PLACEHOLDER = re.compile(r"{args(?::(?P<default>[^}]*))?}")
-RE_PDM_PLACEHOLDER = re.compile(r"{pdm}")
+RE_ARGS_PLACEHOLDER = re.compile(r"\{args(?::(?P<default>[^}]*))?\}")
+RE_PDM_PLACEHOLDER = re.compile(r"\{pdm\}")
 
 
 def _interpolate_args(script: str, args: Sequence[str]) -> tuple[str, bool]:
@@ -63,11 +63,9 @@ def _interpolate_args(script: str, args: Sequence[str]) -> tuple[str, bool]:
 def _interpolate_pdm(script: str) -> str:
     """Interpolate the `{pdm} placeholder in a string"""
     executable_path = Path(sys.executable)
+    pdm_executable = shlex.join([executable_path.as_posix(), "-m", "pdm"])
 
-    def replace(m: re.Match[str]) -> str:
-        return shlex.join([executable_path.as_posix(), "-m", "pdm"])
-
-    interpolated = RE_PDM_PLACEHOLDER.sub(replace, script)
+    interpolated = RE_PDM_PLACEHOLDER.sub(pdm_executable, script)
     return interpolated
 
 
@@ -141,7 +139,7 @@ class TaskRunner:
         return Task(kind, script_name, value, cast("TaskOptions", options))
 
     def expand_command(self, command: str) -> str:
-        expanded_command = os.path.expanduser(os.path.expandvars(command))
+        expanded_command = os.path.expanduser(command)
         if expanded_command.replace(os.sep, "/").startswith(("./", "../")):
             abspath = os.path.abspath(expanded_command)
             if not os.path.isfile(abspath):
@@ -191,16 +189,17 @@ class TaskRunner:
             process_env.update(env)
         if shell:
             assert isinstance(args, str)
-            expanded_args: str | Sequence[str] = os.path.expandvars(args)
+            # environment variables will be expanded by shell
+            process_cmd: str | Sequence[str] = args
         else:
             assert isinstance(args, Sequence)
-            command, *args = args
+            command, *args = (expand_env_vars(arg, env=process_env) for arg in args)
             if command.endswith(".py"):
                 args = [command, *args]
                 command = str(project.environment.interpreter.executable)
             expanded_command = self.expand_command(command)
             real_command = os.path.realpath(expanded_command)
-            expanded_args = [os.path.expandvars(arg) for arg in [expanded_command, *args]]
+            process_cmd = [expanded_command, *args]
             if (
                 project_env.is_local
                 and not site_packages
@@ -222,7 +221,7 @@ class TaskRunner:
 
         handle_term = signal.signal(signal.SIGTERM, forward_signal)
         handle_int = signal.signal(signal.SIGINT, forward_signal)
-        process = subprocess.Popen(expanded_args, cwd=cwd, env=process_env, shell=shell, bufsize=0)
+        process = subprocess.Popen(process_cmd, cwd=cwd, env=process_env, shell=shell, bufsize=0)
         process.wait()
         signal.signal(signal.SIGTERM, handle_term)
         signal.signal(signal.SIGINT, handle_int)
diff --git a/src/pdm/utils.py b/src/pdm/utils.py
index 4cbd9019..db9afdcd 100644
--- a/src/pdm/utils.py
+++ b/src/pdm/utils.py
@@ -18,7 +18,7 @@ import urllib.parse as parse
 import warnings
 from os import name as os_name
 from pathlib import Path
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Mapping
 
 from packaging.version import Version, _cmpkey
 
@@ -221,17 +221,19 @@ def path_to_url(path: str) -> str:
     return url
 
 
-def expand_env_vars(credential: str, quote: bool = False) -> str:
+def expand_env_vars(credential: str, quote: bool = False, env: Mapping[str, str] | None = None) -> str:
     """A safe implementation of env var substitution.
     It only supports the following forms:
 
         ${ENV_VAR}
 
-    Neither $ENV_VAR and %ENV_VAR is not supported.
+    Neither $ENV_VAR and %ENV_VAR is supported.
     """
+    if env is None:
+        env = os.environ
 
     def replace_func(match: Match) -> str:
-        rv = os.getenv(match.group(1), match.group(0))
+        rv = env.get(match.group(1), match.group(0))
         return parse.quote(rv) if quote else rv
 
     return re.sub(r"\$\{(.+?)\}", replace_func, credential)


Test patch:
diff --git a/tests/cli/test_run.py b/tests/cli/test_run.py
index 2ac983fd..c2536822 100644
--- a/tests/cli/test_run.py
+++ b/tests/cli/test_run.py
@@ -325,11 +325,11 @@ def test_run_shell_script_with_pdm_placeholder(project, pdm):
 def test_run_expand_env_vars(project, pdm, capfd, monkeypatch):
     (project.root / "test_script.py").write_text("import os; print(os.getenv('FOO'))")
     project.pyproject.settings["scripts"] = {
-        "test_cmd": 'python -c "foo, bar = 0, 1;print($FOO)"',
-        "test_cmd_no_expand": "python -c 'print($FOO)'",
+        "test_cmd": 'python -c "foo, bar = 0, 1;print(${FOO})"',
+        "test_cmd_no_expand": "python -c 'print(${FOO})'",
         "test_script": "python test_script.py",
         "test_cmd_array": ["python", "test_script.py"],
-        "test_shell": {"shell": "echo $FOO"},
+        "test_shell": {"shell": "echo %FOO%" if os.name == "nt" else "echo $FOO"},
     }
     project.pyproject.write()
     capfd.readouterr()
@@ -351,6 +351,35 @@ def test_run_expand_env_vars(project, pdm, capfd, monkeypatch):
         assert capfd.readouterr()[0].strip() == "bar"
 
 
+def test_run_expand_env_vars_from_config(project, pdm, capfd):
+    (project.root / "test_script.py").write_text("import os; print(os.getenv('FOO'))")
+    project.pyproject.settings["scripts"] = {
+        "test_cmd": 'python -c "foo, bar = 0, 1;print(${FOO})"',
+        "test_cmd_no_expand": "python -c 'print(${FOO})'",
+        "test_script": "python test_script.py",
+        "test_cmd_array": ["python", "test_script.py"],
+        "test_shell": {"shell": "echo %FOO%" if os.name == "nt" else "echo $FOO"},
+        "_": {"env": {"FOO": "bar"}},
+    }
+    project.pyproject.write()
+    capfd.readouterr()
+    with cd(project.root):
+        pdm(["run", "test_cmd"], obj=project)
+        assert capfd.readouterr()[0].strip() == "1"
+
+        result = pdm(["run", "test_cmd_no_expand"], obj=project)
+        assert result.exit_code == 1
+
+        pdm(["run", "test_script"], obj=project)
+        assert capfd.readouterr()[0].strip() == "bar"
+
+        pdm(["run", "test_cmd_array"], obj=project)
+        assert capfd.readouterr()[0].strip() == "bar"
+
+        pdm(["run", "test_shell"], obj=project)
+        assert capfd.readouterr()[0].strip() == "bar"
+
+
 def test_run_script_with_env_defined(project, pdm, capfd):
     (project.root / "test_script.py").write_text("import os; print(os.getenv('FOO'))")
     project.pyproject.settings["scripts"] = {"test_script": {"cmd": "python test_script.py", "env": {"FOO": "bar"}}}