Repository: pypa/pipx (python)

Issue:
Allow running scripts with dependencies using pipx
**How would this feature be useful?**
At the moment, to run a Python script that depends on 3rd party packages, it is necessary to manually create a virtual environment, populate it with the dependencies, and then run the script. And then delete the environment (or keep it if you think you're likely to re-run the script and remember to delete it later).

`pipx run` allows you to run packages in a temporary venv with all dependencies set up, but it doesn't work for scripts. It allows you to run scripts, but that doesn't use any of the venv mechanisms, just running the script with the system Python interpreter.

**Describe the solution you'd like**
If a script needs dependencies, use a temporary virtualenv, the same as for packages, and run the script in that.

To determine if a script needs dependencies, a very simplistic approach would be to check if `--pip-args` was specified. So invocation would be something like:

```
pipx run --pip-args numpy file:myscript.py
```

Even better would be to parse the script, looking for an embedded list of dependencies to install. This would require agreeing on a format for the list. My initial proposal would be something simple, like:

```
# Requirements:
# <Specifier>
# <Specifier>
<blank line>
```

The specifiers could be any requirement spec acceptable to pip, so `numpy`, or `click>=7.0`, or even a URL.

**Describe alternatives you've considered**
The `pip-run` command offers similar functionality, but it recreates the environment every time, which makes for very slow runtimes. It also has (IMO) a clumsy UI for specifying the packages to install and the script invocation.

**Possible Issues**
Script dependencies could potentially change more frequently than package dependencies do. So there's a potential for the cached environment to no longer match the declared requirements. This is also a problem with a package, though, so it's possibly something we can live with. Maybe having an option to re-create the cached environment would be enough to alleviate this issue?

**Implementation**
I would be happy to create a PR implementing this feature, if the view is that it would be a good idea.

Pull request description:
Allow running scripts with dependencies using pipx
Fixes #913

## TODO
* [x] I have added an entry to `docs/changelog.md`
* [x] Documentation updates
* [x] Additional tests for *successfully* running scripts with dependencies
* [x] Refactor our use of the `Venv` class to handle app-less venvs properly
* [x] Clean up the implementation

## Summary of changes

This PR adds functionality to the `pipx run <script>` command to allow running scripts that have non-stdlib dependencies.

The script is parsed for a comment block that has the form

```
# Requirements:
# req1
# req2
```

The block is terminated by a blank comment line, or by a non-comment line. Requirements may be anything that can be a top-level requirement for pip.

Before running a script, the code checks for a requirements block. If one is present, it creates a temporary virtual environment (managed the same way as the existing `pip run app` handles its venvs), installs the requirements there, and runs the script in that environment.

## Implementation issues

The existing implementation of the `Venv` class in `pipx` assumes every environment will have a "main app". The code to write the pipx metadata file enforces this. This is not the case for temporary environments supporting scripts.

Rather than changing the implementation of `Venv` to allow environments without apps (which would be an extensive change, as it changes a fundamental invariant of the existing class) I plan on splitting the `Venv` class into two subclasses - `AppVenv` and `ScriptVenv`. All existing uses would become `AppVenv` instances, with `ScriptVenv` being for the new functionality.

One subtlety here is that the code that lists venvs simply scans directories and assumes what it finds is a venv (`VenvContainer.iter_venv_dirs`). In practice that isn't an issue, as `ScriptVenv` environments will only be found in the cache directory, which is never scanned in this way (as far as I can tell) but it is a potential risk. I will decide how to handle this when I do the refactoring.

## Test plan
The PR includes tests of the functionality.

Also, it can be manually tested by creating a file `test.py` containing

```
# Requirements:
#     requests

import sys
import subprocess
print(sys.executable)
subprocess.run([sys.executable, "-m", "pip", "list"])
```

Then run the command

```
pipx run file:test.py
```

Output should show the executable being in a temporary virtual environment, and the pip output will show requests being present (with its dependencies).


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b88720a..fb054ad 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,6 @@
 ## dev
 
+- Pass `pip_args` to `shared_libs.upgrade()`
 - Fallback to user's log path if the default log path (`$PIPX_HOME/logs`) is not writable to aid with pipx being used for multi-user (e.g. system-wide) installs of applications
 - Fix wrong interpreter usage when injecting local pip-installable dependencies into venvs
 
@@ -7,6 +8,7 @@
 
 - Add test for pip module in `pipx reinstall` to fix an issue with `pipx reinstall-all` (#935)
 - Add `pipx uninject` command (#820)
+- Support including requirements in scripts run using `pipx run` (#916)
 - [docs] Fix `pipx run` examples and update Python versions used by `pipx install` examples
 - [docs] Add an example for installation from source with extras
 - Match pip's behaviour when package name ends with archive extension (treat it as a path)
diff --git a/docs/examples.md b/docs/examples.md
index 0d6e7d9..cae08fb 100644
--- a/docs/examples.md
+++ b/docs/examples.md
@@ -42,6 +42,53 @@ pipx run --spec https://github.com/psf/black/archive/18.9b0.zip black --help
 pipx run https://gist.githubusercontent.com/cs01/fa721a17a326e551ede048c5088f9e0f/raw/6bdfbb6e9c1132b1c38fdd2f195d4a24c540c324/pipx-demo.py
 ```
 
+You can run local files, or scripts hosted on the internet, and you can run them with arguments:
+
+```
+pipx run test.py
+pipx run test.py 1 2 3
+pipx run https://example.com/test.py
+pipx run https://example.com/test.py 1 2 3
+```
+
+A simple filename is ambiguous - it could be a file, or a package on PyPI. It
+will be treated as a filename if the file exists, or as a package if not. To
+force interpretation as a local path, use `--path`, and to force interpretation
+as a package name, use `--spec` (with the PyPI name of the package).
+
+```
+pipx run myscript.py # Local file, if myscript.py exists
+pipx run doesnotexist.py # Package, because doesnotexist.py is not a local file
+pipx run --path test.py # Always a local file
+pipx run --spec test-py test.py # Always a package on PyPI
+```
+
+You can also run scripts that have dependencies:
+
+If you have a script `test.py` that needs a 3rd party library like requests:
+
+```
+# test.py
+
+# Requirements:
+# requests
+#
+# The list of requirements is terminated by a blank line or an empty comment line.
+
+import sys
+import requests
+project = sys.argv[1]
+pipx_data = requests.get(f"https://pypi.org/pypi/{project}/json").json()
+print(pipx_data["info"]["version"])
+```
+
+Then you can run it as follows:
+
+```
+> pipx run test.py pipx
+1.1.0
+```
+
 ## `pipx inject` example
 
 One use of the inject command is setting up a REPL with some useful extra packages.
diff --git a/docs/programs-to-try.md b/docs/programs-to-try.md
index e15c5db..b822b24 100644
--- a/docs/programs-to-try.md
+++ b/docs/programs-to-try.md
@@ -74,6 +74,13 @@ browser-based gdb debugger
 pipx run gdbgui
 ```
 
+### hatch
+
+Python project manager that lets you build & publish packages, run tasks in environments and more
+
+```
+pipx run hatch
+```
 
 ### hexsticker
 
diff --git a/src/pipx/commands/run.py b/src/pipx/commands/run.py
index 477e6da..b4d1b90 100644
--- a/src/pipx/commands/run.py
+++ b/src/pipx/commands/run.py
@@ -6,7 +6,9 @@ import urllib.parse
 import urllib.request
 from pathlib import Path
 from shutil import which
-from typing import List, NoReturn
+from typing import List, NoReturn, Optional
+
+from packaging.requirements import InvalidRequirement, Requirement
 
 from pipx import constants
 from pipx.commands.common import package_name_from_spec
@@ -33,21 +35,18 @@ Available executable scripts:
     {app_lines}"""
 
 
-def run(
-    app: str,
-    package_or_url: str,
-    app_args: List[str],
-    python: str,
-    pip_args: List[str],
-    venv_args: List[str],
-    pypackages: bool,
-    verbose: bool,
-    use_cache: bool,
-) -> NoReturn:
-    """Installs venv to temporary dir (or reuses cache), then runs app from
-    package
-    """
+def maybe_script_content(app: str, is_path: bool) -> Optional[str]:
+    # If the app is a script, return its content.
+    # Return None if it should be treated as a package name.
 
+    # Look for a local file first.
+    app_path = Path(app)
+    if app_path.exists():
+        return app_path.read_text(encoding="utf-8")
+    elif is_path:
+        raise PipxError(f"The specified path {app} does not exist")
+
+    # Check for a URL
     if urllib.parse.urlparse(app).scheme:
         if not app.endswith(".py"):
             raise PipxError(
@@ -58,10 +57,57 @@ def run(
             )
         logger.info("Detected url. Downloading and executing as a Python file.")
 
-        content = _http_get_request(app)
-        exec_app([str(python), "-c", content])
+        return _http_get_request(app)
+
+    # Otherwise, it's a package
+    return None
 
-    elif which(app):
+
+def run_script(
+    content: str,
+    app_args: List[str],
+    python: str,
+    pip_args: List[str],
+    venv_args: List[str],
+    verbose: bool,
+    use_cache: bool,
+) -> NoReturn:
+    requirements = _get_requirements_from_script(content)
+    if requirements is None:
+        exec_app([str(python), "-c", content, *app_args])
+    else:
+        # Note that the environment name is based on the identified
+        # requirements, and *not* on the script name. This is deliberate, as
+        # it ensures that two scripts with the same requirements can use the
+        # same environment, which means fewer environments need to be
+        # managed. The requirements are normalised (in
+        # _get_requirements_from_script), so that irrelevant differences in
+        # whitespace, and similar, don't prevent environment sharing.
+        venv_dir = _get_temporary_venv_path(requirements, python, pip_args, venv_args)
+        venv = Venv(venv_dir)
+        _prepare_venv_cache(venv, None, use_cache)
+        if venv_dir.exists():
+            logger.info(f"Reusing cached venv {venv_dir}")
+        else:
+            venv = Venv(venv_dir, python=python, verbose=verbose)
+            venv.create_venv(venv_args, pip_args)
+            venv.install_unmanaged_packages(requirements, pip_args)
+        exec_app([venv.python_path, "-c", content, *app_args])
+
+
+def run_package(
+    app: str,
+    package_or_url: str,
+    app_args: List[str],
+    python: str,
+    pip_args: List[str],
+    venv_args: List[str],
+    pypackages: bool,
+    verbose: bool,
+    use_cache: bool,
+) -> NoReturn:
+
+    if which(app):
         logger.warning(
             pipx_wrap(
                 f"""
@@ -93,7 +139,7 @@ def run(
             """
         )
 
-    venv_dir = _get_temporary_venv_path(package_or_url, python, pip_args, venv_args)
+    venv_dir = _get_temporary_venv_path([package_or_url], python, pip_args, venv_args)
 
     venv = Venv(venv_dir)
     bin_path = venv.bin_path / app_filename
@@ -118,6 +164,52 @@ def run(
         )
 
 
+def run(
+    app: str,
+    spec: str,
+    is_path: bool,
+    app_args: List[str],
+    python: str,
+    pip_args: List[str],
+    venv_args: List[str],
+    pypackages: bool,
+    verbose: bool,
+    use_cache: bool,
+) -> NoReturn:
+    """Installs venv to temporary dir (or reuses cache), then runs app from
+    package
+    """
+
+    package_or_url = spec if spec is not None else app
+    # For any package, we need to just use the name
+    try:
+        package_name = Requirement(app).name
+    except InvalidRequirement:
+        # Raw URLs to scripts are supported, too, so continue if
+        # we can't parse this as a package
+        package_name = app
+
+    if spec is not None:
+        content = None
+    else:
+        content = maybe_script_content(app, is_path)
+
+    if content is not None:
+        run_script(content, app_args, python, pip_args, venv_args, verbose, use_cache)
+    else:
+        run_package(
+            package_name,
+            package_or_url,
+            app_args,
+            python,
+            pip_args,
+            venv_args,
+            pypackages,
+            verbose,
+            use_cache,
+        )
+
+
 def _download_and_run(
     venv_dir: Path,
     package_or_url: str,
@@ -183,7 +275,7 @@ def _download_and_run(
 
 
 def _get_temporary_venv_path(
-    package_or_url: str, python: str, pip_args: List[str], venv_args: List[str]
+    requirements: List[str], python: str, pip_args: List[str], venv_args: List[str]
 ) -> Path:
     """Computes deterministic path using hashing function on arguments relevant
     to virtual environment's end state. Arguments used should result in idempotent
@@ -191,7 +283,7 @@ def _get_temporary_venv_path(
     passed to venv creation are.)
     """
     m = hashlib.sha256()
-    m.update(package_or_url.encode())
+    m.update("".join(requirements).encode())
     m.update(python.encode())
     m.update("".join(pip_args).encode())
     m.update("".join(venv_args).encode())
@@ -207,9 +299,9 @@ def _is_temporary_venv_expired(venv_dir: Path) -> bool:
     return age > expiration_threshold_sec or (venv_dir / VENV_EXPIRED_FILENAME).exists()
 
 
-def _prepare_venv_cache(venv: Venv, bin_path: Path, use_cache: bool) -> None:
+def _prepare_venv_cache(venv: Venv, bin_path: Optional[Path], use_cache: bool) -> None:
     venv_dir = venv.root
-    if not use_cache and bin_path.exists():
+    if not use_cache and (bin_path is None or bin_path.exists()):
         logger.info(f"Removing cached venv {str(venv_dir)}")
         rmdir(venv_dir)
     _remove_all_expired_venvs()
@@ -230,3 +322,44 @@ def _http_get_request(url: str) -> str:
     except Exception as e:
         logger.debug("Uncaught Exception:", exc_info=True)
         raise PipxError(str(e))
+
+
+def _get_requirements_from_script(content: str) -> Optional[List[str]]:
+
+    # An iterator over the lines in the script. We will
+    # read through this in sections, so it needs to be an
+    # iterator, not just a list.
+    lines = iter(content.splitlines())
+
+    for line in lines:
+        if not line.startswith("#"):
+            continue
+        line_content = line[1:].strip()
+        if line_content == "Requirements:":
+            break
+    else:
+        # No "Requirements:" line in the file
+        return None
+
+    # We are now at the first requirement
+    requirements = []
+    for line in lines:
+        # Stop at the end of the comment block
+        if not line.startswith("#"):
+            break
+        line_content = line[1:].strip()
+        # Stop at a blank comment line
+        if not line_content:
+            break
+
+        # Validate the requirement
+        try:
+            req = Requirement(line_content)
+        except InvalidRequirement as e:
+            raise PipxError(f"Invalid requirement {line_content}: {str(e)}")
+
+        # Use the normalised form of the requirement,
+        # not the original line.
+        requirements.append(str(req))
+
+    return requirements
diff --git a/src/pipx/main.py b/src/pipx/main.py
index b5156a9..59f6222 100644
--- a/src/pipx/main.py
+++ b/src/pipx/main.py
@@ -17,7 +17,6 @@ from typing import Any, Callable, Dict, List, Optional
 
 import argcomplete  # type: ignore
 import platformdirs
-from packaging.requirements import InvalidRequirement, Requirement
 from packaging.utils import canonicalize_name
 
 import pipx.constants
@@ -184,30 +183,17 @@ def run_pipx_command(args: argparse.Namespace) -> ExitCode:  # noqa: C901
         skip_list = [canonicalize_name(x) for x in args.skip]
 
     if args.command == "run":
-        package_or_url = (
-            args.spec
-            if ("spec" in args and args.spec is not None)
-            else args.app_with_args[0]
-        )
-        # For any package, we need to just use the name
-        try:
-            package_name = Requirement(args.app_with_args[0]).name
-        except InvalidRequirement:
-            # Raw URLs to scripts are supported, too, so continue if
-            # we can't parse this as a package
-            package_name = args.app_with_args[0]
-
-        use_cache = not args.no_cache
         commands.run(
-            package_name,
-            package_or_url,
+            args.app_with_args[0],
+            args.spec,
+            args.path,
             args.app_with_args[1:],
             args.python,
             pip_args,
             venv_args,
             args.pypackages,
             verbose,
-            use_cache,
+            not args.no_cache,
         )
         # We should never reach here because run() is NoReturn.
         return ExitCode(1)
@@ -589,6 +575,9 @@ def _add_run(subparsers: argparse._SubParsersAction) -> None:
         help="app/package name and any arguments to be passed to it",
         default=[],
     )
+    p.add_argument(
+        "--path", action="store_true", help="Interpret app name as a local path"
+    )
     p.add_argument(
         "--pypackages",
         action="store_true",
diff --git a/src/pipx/venv.py b/src/pipx/venv.py
index 59a47ac..b6d01ce 100644
--- a/src/pipx/venv.py
+++ b/src/pipx/venv.py
@@ -197,7 +197,7 @@ class Venv:
 
     def upgrade_packaging_libraries(self, pip_args: List[str]) -> None:
         if self.uses_shared_libs:
-            shared_libs.upgrade(verbose=self.verbose)
+            shared_libs.upgrade(pip_args=pip_args, verbose=self.verbose)
         else:
             # TODO: setuptools and wheel? Original code didn't bother
             # but shared libs code does.
@@ -274,6 +274,28 @@ class Venv:
                 wrap_message=False,
             )
 
+    def install_unmanaged_packages(
+        self, requirements: List[str], pip_args: List[str]
+    ) -> None:
+        """Install packages in the venv, but do not record them in the metadata."""
+
+        # Note: We want to install everything at once, as that lets
+        # pip resolve conflicts correctly.
+        with animate(f"installing {', '.join(requirements)}", self.do_animation):
+            # do not use -q with `pip install` so subprocess_post_check_pip_errors
+            #   has more information to analyze in case of failure.
+            cmd = (
+                [str(self.python_path), "-m", "pip", "install"]
+                + pip_args
+                + requirements
+            )
+            # no logging because any errors will be specially logged by
+            #   subprocess_post_check_handle_pip_error()
+            pip_process = run_subprocess(cmd, log_stdout=False, log_stderr=False)
+        subprocess_post_check_handle_pip_error(pip_process)
+        if pip_process.returncode:
+            raise PipxError(f"Error installing {', '.join(requirements)}.")
+
     def install_package_no_deps(self, package_or_url: str, pip_args: List[str]) -> str:
         with animate(
             f"determining package name from {package_or_url!r}", self.do_animation


Test patch:
diff --git a/tests/test_run.py b/tests/test_run.py
index 5e5eb19..6d5e4e1 100644
--- a/tests/test_run.py
+++ b/tests/test_run.py
@@ -2,6 +2,7 @@ import logging
 import os
 import subprocess
 import sys
+import textwrap
 from unittest import mock
 
 import pytest  # type: ignore
@@ -181,3 +182,137 @@ def test_package_determination(
 
     assert "Cannot determine package name" not in caplog.text
     assert f"Determined package name: {package}" in caplog.text
+
+
+@mock.patch("os.execvpe", new=execvpe_mock)
+def test_run_without_requirements(caplog, pipx_temp_env, tmp_path):
+    script = tmp_path / "test.py"
+    out = tmp_path / "output.txt"
+    test_str = "Hello, world!"
+    script.write_text(
+        textwrap.dedent(
+            f"""
+                from pathlib import Path
+                Path({repr(str(out))}).write_text({repr(test_str)})
+            """
+        ).strip()
+    )
+    run_pipx_cli_exit(["run", script.as_uri()])
+    assert out.read_text() == test_str
+
+
+@mock.patch("os.execvpe", new=execvpe_mock)
+def test_run_with_requirements(caplog, pipx_temp_env, tmp_path):
+    script = tmp_path / "test.py"
+    out = tmp_path / "output.txt"
+    script.write_text(
+        textwrap.dedent(
+            f"""
+                # Requirements:
+                # requests==2.28.1
+
+                # Check requests can be imported
+                import requests
+                # Check dependencies of requests can be imported
+                import certifi
+                # Check the installed version
+                from pathlib import Path
+                Path({repr(str(out))}).write_text(requests.__version__)
+            """
+        ).strip()
+    )
+    run_pipx_cli_exit(["run", script.as_uri()])
+    assert out.read_text() == "2.28.1"
+
+
+@mock.patch("os.execvpe", new=execvpe_mock)
+def test_run_with_args(caplog, pipx_temp_env, tmp_path):
+    script = tmp_path / "test.py"
+    out = tmp_path / "output.txt"
+    script.write_text(
+        textwrap.dedent(
+            f"""
+                import sys
+                from pathlib import Path
+                Path({repr(str(out))}).write_text(str(int(sys.argv[1]) + 1))
+            """
+        ).strip()
+    )
+    run_pipx_cli_exit(["run", script.as_uri(), "1"])
+    assert out.read_text() == "2"
+
+
+@mock.patch("os.execvpe", new=execvpe_mock)
+def test_run_with_requirements_and_args(caplog, pipx_temp_env, tmp_path):
+    script = tmp_path / "test.py"
+    out = tmp_path / "output.txt"
+    script.write_text(
+        textwrap.dedent(
+            f"""
+                # Requirements:
+                # packaging
+
+                import packaging
+                import sys
+                from pathlib import Path
+                Path({repr(str(out))}).write_text(str(int(sys.argv[1]) + 1))
+            """
+        ).strip()
+    )
+    run_pipx_cli_exit(["run", script.as_uri(), "1"])
+    assert out.read_text() == "2"
+
+
+@mock.patch("os.execvpe", new=execvpe_mock)
+def test_run_with_invalid_requirement(capsys, pipx_temp_env, tmp_path):
+    script = tmp_path / "test.py"
+    script.write_text(
+        textwrap.dedent(
+            """
+                # Requirements:
+                # this is an invalid requirement
+                print()
+            """
+        ).strip()
+    )
+    ret = run_pipx_cli(["run", script.as_uri()])
+    assert ret == 1
+
+    captured = capsys.readouterr()
+    assert "Invalid requirement this is an invalid requirement" in captured.err
+
+
+@mock.patch("os.execvpe", new=execvpe_mock)
+def test_run_script_by_absolute_name(caplog, pipx_temp_env, tmp_path):
+    script = tmp_path / "test.py"
+    out = tmp_path / "output.txt"
+    test_str = "Hello, world!"
+    script.write_text(
+        textwrap.dedent(
+            f"""
+                from pathlib import Path
+                Path({repr(str(out))}).write_text({repr(test_str)})
+            """
+        ).strip()
+    )
+    run_pipx_cli_exit(["run", "--path", str(script)])
+    assert out.read_text() == test_str
+
+
+@mock.patch("os.execvpe", new=execvpe_mock)
+def test_run_script_by_relative_name(caplog, pipx_temp_env, monkeypatch, tmp_path):
+    script = tmp_path / "test.py"
+    out = tmp_path / "output.txt"
+    test_str = "Hello, world!"
+    script.write_text(
+        textwrap.dedent(
+            f"""
+                from pathlib import Path
+                Path({repr(str(out))}).write_text({repr(test_str)})
+            """
+        ).strip()
+    )
+    with monkeypatch.context() as m:
+        m.chdir(tmp_path)
+        run_pipx_cli_exit(["run", "test.py"])
+    assert out.read_text() == test_str