Repository: pypa/pip
Pull request #12214 (2023-08-08 11:10:02)

PEP 721: Use the ``data_filter`` when extracting tarballs, if it's available.
Previous behaviour is used on Python without PEP-720 tarfile filters. (Note that the feature is now in security releases of all supported versions.)

A custom filter (which wraps `data_filter`) is used to retain pip-specific behaviour:
- Removing a common leading directory
- Setting the mode (Unix permissions)

Compared to the previous behaviour, if a file can't be unpacked, the unpacking operation will fail with `InstallError`, rather than skipping the individual file with a `logger.warning`. This means that "some corrupt tar files" now can't be unpacked.
IMO, this is the correct path forward, but maybe there should be a CLI option to use the old behaviour? Sadly I don't know enough about pip to add one and propagate the setting to `untar_file`.

The new code path avoids internal API (`TarFile._extract_member`).

Note that PEP 721 limits itself to sdists, this change affects unpacking any other tar file.

I hope I got the `nox`/`mypy`/`ruff` stuff right, guess CI will tell...

---

*This PR indents a big block of code. To hide whitespace changes, use `git diff -w` or add `?w=1` to the GitHub URL.*

Fixes: #12111

uranusjr: I feel it may be a good idea to pull the except and else blocks into separate functions for readability. It’s difficult for me to find where the except block ends now.
henryiii: Needs same fix as https://github.com/pypa/build/pull/675 to avoid breakage.
encukou: (This is still on my TODO, it just keeps getting pushed back by conferences.)
encukou: Sorry for the delay.
A bug in my initial implementation of tarfile filters means that specific CPython versions raise spurious exceptions. Filtering those out, the “new” way unfortunately grew to be about as many lines as the “old” way -- although it does have more comments and a big chunk can be cleanly removed when support for specific versions ends.

Patch:
diff --git a/docs/html/development/architecture/command-line-interface.rst b/docs/html/development/architecture/command-line-interface.rst
index 9bfa91192..725199bf5 100644
--- a/docs/html/development/architecture/command-line-interface.rst
+++ b/docs/html/development/architecture/command-line-interface.rst
@@ -28,19 +28,167 @@ for parsing top level args.
 
 ``Command`` then uses another ``ConfigOptionParser`` instance, to parse command-specific args.
 
-* TODO: How & where options are defined
-  (cmdoptions, command-specific files).
+Command structure
+-----------------
 
-* TODO: How & where arguments are processed.
-  (main_parser, command-specific parser)
+This section shows the class hierarchy from which every command's class will inherit
+from.
 
-* TODO: How processed arguments are accessed.
-  (attributes on argument to ``Command.run()``)
+`base_command.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/base_command.py>`_
+defines the base ``Command`` class, from which every other command will inherit directly or
+indirectly (see the *command tree* at the end of this section).
 
-* TODO: How configuration and CLI "blend".
-  (implemented in ``ConfigOptionParser``)
+Using the ``ConfigOptionParser`` (see `Configuration and CLI "blend" <Configuration and CLI "blend"_>`_),
+this class adds the general options and instantiates the *cmd_opts* group, where every other specific
+option will be added if needed on each command's class. For those commands that define specific
+options, like ``--dry-run`` on ``pip install`` command, the options must be added to *cmd_opts*
+this is the job of *add_options* method), which will be automatically called on ``Command``'s initialization.
 
-* TODO: progress bars and spinners
+The base ``Command`` has the following methods:
+
+.. py:class:: Command
+
+  .. py:method:: main()
+
+    Main method of the class, it's always called (as can be seen in main.py's
+    `main <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/main.py#L46>`_).
+    It's in charge of calling the specific ``run`` method of the class and handling the possible errors.
+
+  .. py:method:: run()
+
+    Abstract method where the actual action of a command is defined.
+
+  .. py:method:: add_options()
+
+    Optional method to insert additional options on a class, called on ``Command`` initialization.
+
+Some commands have more specialized behavior, (see for example ``pip index``).
+These commands instead will inherit from ``IndexGroupCommand``, which inherits from ``Command``
+and  ``SessionCommandMixin`` to build build the pip session for the corresponding requests.
+
+Lastly, ``RequirementCommand``, which inherits from ``IndexGroupCommand`` is the base class
+for those commands which make use of requirements in any form, like ``pip install``.
+
+In addition to the previous classes, a last mixin class must be mentioned, from which
+``Command`` as well as ``SessionCommandMixin`` inherit: ``CommandContextMixIn``, in
+charge of the command's context.
+
+In the following command tree we can see the hierarchy defined for the different pip
+commands, where each command is defined under the base class it inherits from:
+
+| ``Command``
+|  ├─ ``cache``, ``check``, ``completion``, ``configuration``, ``debug``, ``freeze``, ``hash``, ``help``, ``inspect``, ``show``, ``search``, ``uninstall``
+|  └─ ``IndexGroupCommand``
+|      ├─ ``index``, ``list``
+|      └─ ``RequirementCommand``
+|           └─ ``wheel``, ``download``, ``install``
+
+
+Option definition
+-----------------
+
+The set of shared options are defined in `cmdoptions.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/cmdoptions.py>`_
+module, as well as the *general options* and *package index options* groups of options
+we see when we call a command's help, or the ``pip index``'s help message respectively.
+All options are defined in terms of functions that return `optparse.Option <https://docs.python.org/3/library/optparse.html#optparse.Option>`_
+instances once called, while specific groups of options, like *Config Options* for
+``pip config`` are defined in each specific command file (see for example the
+`configuration.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/commands/configuration.py>`_).
+
+Argument parsing
+----------------
+
+The main entrypoint for the application is defined in the ``main`` function in the
+`main.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/main.py>`_ module.
+This function is in charge of the `autocompletion <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/autocompletion.py>`_,
+calling the ``parse_command`` function and creating and running the subprograms
+via ``create_command``, on which the ``main`` method is called.
+
+The ``parse_command`` is defined in the `main_parser.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/main_parser.py>`_
+module, which defines the following two functions:
+
+.. py:function:: parse_command()
+
+  Function in charge of the initial parse of ``pip``'s program. Creates the main parser (see
+  the next function ``create_main_parser``) to extract the general options
+  and the remaining arguments. For example, running ``pip --timeout=5 install --user INITools``
+  will split ``['--timeout=5']`` as general option and  ``['install', '--user', 'INITools']``
+  as the remainder.
+
+  At this step the program deals with the options ``--python``, ``--version``, ``pip``
+  or ``pip help``. If neither of the previous options is found, it tries to extract the command
+  name and arguments.
+
+.. py:function:: create_main_parser()
+
+  Creates the main parser (type ``pip`` in the console to see the description of the
+  program). The internal parser (`ConfigOptionParser <Configuration and CLI "blend"_>`_),
+  adds the general option group and the list of commands coming from ``cmdoptions.py``
+  at this point.
+
+After the initial parsing is done, ``create_command`` is in charge of creating the appropriate
+command using the information stored in `commands_dict <https://github.com/pypa/pip/blob/main/src/pip/_internal/commands/__init__.py>`_
+variable, and calling its ``main`` method (see `Command structure <Command structure>`_).
+
+A second argument parsing is done at each specific command (defined in the base ``Command`` class),
+again using the ``ConfigOptionParser``.
+
+Argument access
+---------------
+
+To access all the options and arguments, ``Command.run()`` takes
+the options as `optparse.Values <https://docs.python.org/3/library/optparse.html#optparse.Values>`_
+and a list of strings for the arguments (parsed in ``Command.main()``). The internal methods of
+the base ``Command`` class are in charge of passing these variables after ``parse_args`` is
+called for a specific command.
+
+Configuration and CLI "blend"
+-----------------------------
+
+The base ``Command`` instantiates the class `ConfigOptionParser <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/parser.py>`_
+which is in charge of the parsing process (via its parent class
+`optparse.OptionParser <https://docs.python.org/3/library/optparse.html#optparse.OptionParser>`_).
+Its main addition consists of the following function:
+
+.. py:class:: ConfigOptionParser(OptionParser)
+
+  .. py:method:: get_default_values()
+
+    Overrides the original method to allow updating the defaults ater the instantiation of the
+    option parser.
+
+It allows overriding the default options and arguments using the ``Configuration`` class
+(more information can be found on :ref:`Configuration`) to include environment variables and
+settings from configuration files.
+
+Progress bars and spinners
+--------------------------
+
+There are two more modules in the ``cli`` subpackage in charge of showing the state of the
+program.
+
+* `progress_bars.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/progress_bars.py>`_
+
+  This module contains the following function:
+
+  .. py:function:: get_download_progress_renderer()
+
+    It uses `rich <https://rich.readthedocs.io/en/stable/reference/progress.html#module-rich.progress>`_
+    functionalities to render the download progress.
+
+  This function (used in `download.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/network/download.py>`_,
+  inside the ``Downloader`` class), allows watching the download process when running
+  ``pip install`` on *big* packages.
+
+* `spinner.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/spinners.py>`_
+
+  The main function of this module is:
+
+  .. py:function:: open_spinner()
+
+    It yields the appropriate type of spinner, which is used in ``call_subprocess``
+    function, inside `subprocess.py <https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/subprocess.py>`_
+    module, so the user can see there is a program running.
 
 * TODO: quirks / standard practices / broad ideas.
   (avoiding lists in option def'n, special cased option value types,
diff --git a/news/12111.feature.rst b/news/12111.feature.rst
new file mode 100644
index 000000000..eb03b76d8
--- /dev/null
+++ b/news/12111.feature.rst
@@ -0,0 +1,1 @@
+PEP 721: Use the ``data_filter`` when extracting tarballs, if it's available.
diff --git a/news/12666.bugfix.rst b/news/12666.bugfix.rst
new file mode 100644
index 000000000..72a4a6515
--- /dev/null
+++ b/news/12666.bugfix.rst
@@ -0,0 +1,1 @@
+Fix intermittent "cannot locate t64.exe" errors when upgrading pip.
diff --git a/news/24ca5f01-e27e-41b1-9a9c-7e3a828e22d9.trivial.rst b/news/24ca5f01-e27e-41b1-9a9c-7e3a828e22d9.trivial.rst
new file mode 100644
index 000000000..1be253956
--- /dev/null
+++ b/news/24ca5f01-e27e-41b1-9a9c-7e3a828e22d9.trivial.rst
@@ -0,0 +1,10 @@
+pip uninstall and list currently depend on req_install.py which always imports
+the expensive network and index machinery. However, it's only in rare situations
+that these commands actually hit the network:
+
+- ``pip list --outdated``
+- ``pip list --uptodate``
+- ``pip uninstall --requirement <url>``
+
+This patch refactors req_install.py so these commands can avoid the expensive
+imports unless truly necessary.
diff --git a/news/6831.doc.rst b/news/6831.doc.rst
new file mode 100644
index 000000000..994fc763d
--- /dev/null
+++ b/news/6831.doc.rst
@@ -0,0 +1,1 @@
+Update architecture documentation for command line interface.
diff --git a/src/pip/_internal/cli/index_command.py b/src/pip/_internal/cli/index_command.py
new file mode 100644
index 000000000..4ff7b2c3a
--- /dev/null
+++ b/src/pip/_internal/cli/index_command.py
@@ -0,0 +1,172 @@
+"""
+Contains command classes which may interact with an index / the network.
+
+Unlike its sister module, req_command, this module still uses lazy imports
+so commands which don't always hit the network (e.g. list w/o --outdated or
+--uptodate) don't need waste time importing PipSession and friends.
+"""
+
+import logging
+import os
+import sys
+from optparse import Values
+from typing import TYPE_CHECKING, List, Optional
+
+from pip._internal.cli.base_command import Command
+from pip._internal.cli.command_context import CommandContextMixIn
+from pip._internal.exceptions import CommandError
+
+if TYPE_CHECKING:
+    from ssl import SSLContext
+
+    from pip._internal.network.session import PipSession
+
+logger = logging.getLogger(__name__)
+
+
+def _create_truststore_ssl_context() -> Optional["SSLContext"]:
+    if sys.version_info < (3, 10):
+        raise CommandError("The truststore feature is only available for Python 3.10+")
+
+    try:
+        import ssl
+    except ImportError:
+        logger.warning("Disabling truststore since ssl support is missing")
+        return None
+
+    try:
+        from pip._vendor import truststore
+    except ImportError as e:
+        raise CommandError(f"The truststore feature is unavailable: {e}")
+
+    return truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+
+
+class SessionCommandMixin(CommandContextMixIn):
+    """
+    A class mixin for command classes needing _build_session().
+    """
+
+    def __init__(self) -> None:
+        super().__init__()
+        self._session: Optional["PipSession"] = None
+
+    @classmethod
+    def _get_index_urls(cls, options: Values) -> Optional[List[str]]:
+        """Return a list of index urls from user-provided options."""
+        index_urls = []
+        if not getattr(options, "no_index", False):
+            url = getattr(options, "index_url", None)
+            if url:
+                index_urls.append(url)
+        urls = getattr(options, "extra_index_urls", None)
+        if urls:
+            index_urls.extend(urls)
+        # Return None rather than an empty list
+        return index_urls or None
+
+    def get_default_session(self, options: Values) -> "PipSession":
+        """Get a default-managed session."""
+        if self._session is None:
+            self._session = self.enter_context(self._build_session(options))
+            # there's no type annotation on requests.Session, so it's
+            # automatically ContextManager[Any] and self._session becomes Any,
+            # then https://github.com/python/mypy/issues/7696 kicks in
+            assert self._session is not None
+        return self._session
+
+    def _build_session(
+        self,
+        options: Values,
+        retries: Optional[int] = None,
+        timeout: Optional[int] = None,
+        fallback_to_certifi: bool = False,
+    ) -> "PipSession":
+        from pip._internal.network.session import PipSession
+
+        cache_dir = options.cache_dir
+        assert not cache_dir or os.path.isabs(cache_dir)
+
+        if "truststore" in options.features_enabled:
+            try:
+                ssl_context = _create_truststore_ssl_context()
+            except Exception:
+                if not fallback_to_certifi:
+                    raise
+                ssl_context = None
+        else:
+            ssl_context = None
+
+        session = PipSession(
+            cache=os.path.join(cache_dir, "http-v2") if cache_dir else None,
+            retries=retries if retries is not None else options.retries,
+            trusted_hosts=options.trusted_hosts,
+            index_urls=self._get_index_urls(options),
+            ssl_context=ssl_context,
+        )
+
+        # Handle custom ca-bundles from the user
+        if options.cert:
+            session.verify = options.cert
+
+        # Handle SSL client certificate
+        if options.client_cert:
+            session.cert = options.client_cert
+
+        # Handle timeouts
+        if options.timeout or timeout:
+            session.timeout = timeout if timeout is not None else options.timeout
+
+        # Handle configured proxies
+        if options.proxy:
+            session.proxies = {
+                "http": options.proxy,
+                "https": options.proxy,
+            }
+            session.trust_env = False
+
+        # Determine if we can prompt the user for authentication or not
+        session.auth.prompting = not options.no_input
+        session.auth.keyring_provider = options.keyring_provider
+
+        return session
+
+
+def _pip_self_version_check(session: "PipSession", options: Values) -> None:
+    from pip._internal.self_outdated_check import pip_self_version_check as check
+
+    check(session, options)
+
+
+class IndexGroupCommand(Command, SessionCommandMixin):
+    """
+    Abstract base class for commands with the index_group options.
+
+    This also corresponds to the commands that permit the pip version check.
+    """
+
+    def handle_pip_version_check(self, options: Values) -> None:
+        """
+        Do the pip version check if not disabled.
+
+        This overrides the default behavior of not doing the check.
+        """
+        # Make sure the index_group options are present.
+        assert hasattr(options, "no_index")
+
+        if options.disable_pip_version_check or options.no_index:
+            return
+
+        # Otherwise, check if we're using the latest version of pip available.
+        session = self._build_session(
+            options,
+            retries=0,
+            timeout=min(5, options.timeout),
+            # This is set to ensure the function does not fail when truststore is
+            # specified in use-feature but cannot be loaded. This usually raises a
+            # CommandError and shows a nice user-facing error, but this function is not
+            # called in that try-except block.
+            fallback_to_certifi=True,
+        )
+        with session:
+            _pip_self_version_check(session, options)
diff --git a/src/pip/_internal/cli/req_command.py b/src/pip/_internal/cli/req_command.py
index 7a5580a84..92900f94f 100644
--- a/src/pip/_internal/cli/req_command.py
+++ b/src/pip/_internal/cli/req_command.py
@@ -1,21 +1,19 @@
-"""Contains the Command base classes that depend on PipSession.
+"""Contains the RequirementCommand base class.
 
-The classes in this module are in a separate module so the commands not
-needing download / PackageFinder capability don't unnecessarily import the
+This class is in a separate module so the commands that do not always
+need PackageFinder capability don't unnecessarily import the
 PackageFinder machinery and all its vendored dependencies, etc.
 """
 
 import logging
-import os
-import sys
 from functools import partial
 from optparse import Values
-from typing import TYPE_CHECKING, Any, List, Optional, Tuple
+from typing import Any, List, Optional, Tuple
 
 from pip._internal.cache import WheelCache
 from pip._internal.cli import cmdoptions
-from pip._internal.cli.base_command import Command
-from pip._internal.cli.command_context import CommandContextMixIn
+from pip._internal.cli.index_command import IndexGroupCommand
+from pip._internal.cli.index_command import SessionCommandMixin as SessionCommandMixin
 from pip._internal.exceptions import CommandError, PreviousBuildDirError
 from pip._internal.index.collector import LinkCollector
 from pip._internal.index.package_finder import PackageFinder
@@ -33,160 +31,15 @@ from pip._internal.req.constructors import (
 from pip._internal.req.req_file import parse_requirements
 from pip._internal.req.req_install import InstallRequirement
 from pip._internal.resolution.base import BaseResolver
-from pip._internal.self_outdated_check import pip_self_version_check
 from pip._internal.utils.temp_dir import (
     TempDirectory,
     TempDirectoryTypeRegistry,
     tempdir_kinds,
 )
-from pip._internal.utils.virtualenv import running_under_virtualenv
-
-if TYPE_CHECKING:
-    from ssl import SSLContext
 
 logger = logging.getLogger(__name__)
 
 
-def _create_truststore_ssl_context() -> Optional["SSLContext"]:
-    if sys.version_info < (3, 10):
-        raise CommandError("The truststore feature is only available for Python 3.10+")
-
-    try:
-        import ssl
-    except ImportError:
-        logger.warning("Disabling truststore since ssl support is missing")
-        return None
-
-    try:
-        from pip._vendor import truststore
-    except ImportError as e:
-        raise CommandError(f"The truststore feature is unavailable: {e}")
-
-    return truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
-
-
-class SessionCommandMixin(CommandContextMixIn):
-    """
-    A class mixin for command classes needing _build_session().
-    """
-
-    def __init__(self) -> None:
-        super().__init__()
-        self._session: Optional[PipSession] = None
-
-    @classmethod
-    def _get_index_urls(cls, options: Values) -> Optional[List[str]]:
-        """Return a list of index urls from user-provided options."""
-        index_urls = []
-        if not getattr(options, "no_index", False):
-            url = getattr(options, "index_url", None)
-            if url:
-                index_urls.append(url)
-        urls = getattr(options, "extra_index_urls", None)
-        if urls:
-            index_urls.extend(urls)
-        # Return None rather than an empty list
-        return index_urls or None
-
-    def get_default_session(self, options: Values) -> PipSession:
-        """Get a default-managed session."""
-        if self._session is None:
-            self._session = self.enter_context(self._build_session(options))
-            # there's no type annotation on requests.Session, so it's
-            # automatically ContextManager[Any] and self._session becomes Any,
-            # then https://github.com/python/mypy/issues/7696 kicks in
-            assert self._session is not None
-        return self._session
-
-    def _build_session(
-        self,
-        options: Values,
-        retries: Optional[int] = None,
-        timeout: Optional[int] = None,
-        fallback_to_certifi: bool = False,
-    ) -> PipSession:
-        cache_dir = options.cache_dir
-        assert not cache_dir or os.path.isabs(cache_dir)
-
-        if "truststore" in options.features_enabled:
-            try:
-                ssl_context = _create_truststore_ssl_context()
-            except Exception:
-                if not fallback_to_certifi:
-                    raise
-                ssl_context = None
-        else:
-            ssl_context = None
-
-        session = PipSession(
-            cache=os.path.join(cache_dir, "http-v2") if cache_dir else None,
-            retries=retries if retries is not None else options.retries,
-            trusted_hosts=options.trusted_hosts,
-            index_urls=self._get_index_urls(options),
-            ssl_context=ssl_context,
-        )
-
-        # Handle custom ca-bundles from the user
-        if options.cert:
-            session.verify = options.cert
-
-        # Handle SSL client certificate
-        if options.client_cert:
-            session.cert = options.client_cert
-
-        # Handle timeouts
-        if options.timeout or timeout:
-            session.timeout = timeout if timeout is not None else options.timeout
-
-        # Handle configured proxies
-        if options.proxy:
-            session.proxies = {
-                "http": options.proxy,
-                "https": options.proxy,
-            }
-            session.trust_env = False
-
-        # Determine if we can prompt the user for authentication or not
-        session.auth.prompting = not options.no_input
-        session.auth.keyring_provider = options.keyring_provider
-
-        return session
-
-
-class IndexGroupCommand(Command, SessionCommandMixin):
-    """
-    Abstract base class for commands with the index_group options.
-
-    This also corresponds to the commands that permit the pip version check.
-    """
-
-    def handle_pip_version_check(self, options: Values) -> None:
-        """
-        Do the pip version check if not disabled.
-
-        This overrides the default behavior of not doing the check.
-        """
-        # Make sure the index_group options are present.
-        assert hasattr(options, "no_index")
-
-        if options.disable_pip_version_check or options.no_index:
-            return
-
-        # Otherwise, check if we're using the latest version of pip available.
-        session = self._build_session(
-            options,
-            retries=0,
-            timeout=min(5, options.timeout),
-            # This is set to ensure the function does not fail when truststore is
-            # specified in use-feature but cannot be loaded. This usually raises a
-            # CommandError and shows a nice user-facing error, but this function is not
-            # called in that try-except block.
-            fallback_to_certifi=True,
-        )
-        with session:
-            pip_self_version_check(session, options)
-
-
 KEEPABLE_TEMPDIR_TYPES = [
     tempdir_kinds.BUILD_ENV,
     tempdir_kinds.EPHEM_WHEEL_CACHE,
@@ -194,39 +47,6 @@ KEEPABLE_TEMPDIR_TYPES = [
 ]
 
 
-def warn_if_run_as_root() -> None:
-    """Output a warning for sudo users on Unix.
-
-    In a virtual environment, sudo pip still writes to virtualenv.
-    On Windows, users may run pip as Administrator without issues.
-    This warning only applies to Unix root users outside of virtualenv.
-    """
-    if running_under_virtualenv():
-        return
-    if not hasattr(os, "getuid"):
-        return
-    # On Windows, there are no "system managed" Python packages. Installing as
-    # Administrator via pip is the correct way of updating system environments.
-    #
-    # We choose sys.platform over utils.compat.WINDOWS here to enable Mypy platform
-    # checks: https://mypy.readthedocs.io/en/stable/common_issues.html
-    if sys.platform == "win32" or sys.platform == "cygwin":
-        return
-
-    if os.getuid() != 0:
-        return
-
-    logger.warning(
-        "Running pip as the 'root' user can result in broken permissions and "
-        "conflicting behaviour with the system package manager, possibly "
-        "rendering your system unusable."
-        "It is recommended to use a virtual environment instead: "
-        "https://pip.pypa.io/warnings/venv. "
-        "Use the --root-user-action option if you know what you are doing and "
-        "want to suppress this warning."
-    )
-
-
 def with_cleanup(func: Any) -> Any:
     """Decorator for common logic related to managing temporary
     directories.
diff --git a/src/pip/_internal/commands/install.py b/src/pip/_internal/commands/install.py
index a9e83f9d8..29276c243 100644
--- a/src/pip/_internal/commands/install.py
+++ b/src/pip/_internal/commands/install.py
@@ -14,7 +14,6 @@ from pip._internal.cli import cmdoptions
 from pip._internal.cli.cmdoptions import make_target_python
 from pip._internal.cli.req_command import (
     RequirementCommand,
-    warn_if_run_as_root,
     with_cleanup,
 )
 from pip._internal.cli.status_codes import ERROR, SUCCESS
@@ -37,6 +36,7 @@ from pip._internal.utils.misc import (
     ensure_dir,
     get_pip_version,
     protect_pip_from_modification_on_windows,
+    warn_if_run_as_root,
     write_output,
 )
 from pip._internal.utils.temp_dir import TempDirectory
diff --git a/src/pip/_internal/commands/list.py b/src/pip/_internal/commands/list.py
index cb1e313f4..82fc46a11 100644
--- a/src/pip/_internal/commands/list.py
+++ b/src/pip/_internal/commands/list.py
@@ -7,18 +7,17 @@ from pip._vendor.packaging.utils import canonicalize_name
 from pip._vendor.packaging.version import Version
 
 from pip._internal.cli import cmdoptions
-from pip._internal.cli.req_command import IndexGroupCommand
+from pip._internal.cli.index_command import IndexGroupCommand
 from pip._internal.cli.status_codes import SUCCESS
 from pip._internal.exceptions import CommandError
-from pip._internal.index.collector import LinkCollector
-from pip._internal.index.package_finder import PackageFinder
 from pip._internal.metadata import BaseDistribution, get_environment
 from pip._internal.models.selection_prefs import SelectionPreferences
-from pip._internal.network.session import PipSession
 from pip._internal.utils.compat import stdlib_pkgs
 from pip._internal.utils.misc import tabulate, write_output
 
 if TYPE_CHECKING:
+    from pip._internal.index.package_finder import PackageFinder
+    from pip._internal.network.session import PipSession
 
     class _DistWithLatestInfo(BaseDistribution):
         """Give the distribution object a couple of extra fields.
@@ -140,11 +139,15 @@ class ListCommand(IndexGroupCommand):
             super().handle_pip_version_check(options)
 
     def _build_package_finder(
-        self, options: Values, session: PipSession
-    ) -> PackageFinder:
+        self, options: Values, session: "PipSession"
+    ) -> "PackageFinder":
         """
         Create a package finder appropriate to this list command.
         """
+        # Lazy import the heavy index modules as most list invocations won't need 'em.
+        from pip._internal.index.collector import LinkCollector
+        from pip._internal.index.package_finder import PackageFinder
+
         link_collector = LinkCollector.create(session, options=options)
 
         # Pass allow_yanked=False to ignore yanked versions.
diff --git a/src/pip/_internal/commands/uninstall.py b/src/pip/_internal/commands/uninstall.py
index f198fc313..bc0edeac9 100644
--- a/src/pip/_internal/commands/uninstall.py
+++ b/src/pip/_internal/commands/uninstall.py
@@ -6,7 +6,7 @@ from pip._vendor.packaging.utils import canonicalize_name
 
 from pip._internal.cli import cmdoptions
 from pip._internal.cli.base_command import Command
-from pip._internal.cli.req_command import SessionCommandMixin, warn_if_run_as_root
+from pip._internal.cli.index_command import SessionCommandMixin
 from pip._internal.cli.status_codes import SUCCESS
 from pip._internal.exceptions import InstallationError
 from pip._internal.req import parse_requirements
@@ -17,6 +17,7 @@ from pip._internal.req.constructors import (
 from pip._internal.utils.misc import (
     check_externally_managed,
     protect_pip_from_modification_on_windows,
+    warn_if_run_as_root,
 )
 
 logger = logging.getLogger(__name__)
diff --git a/src/pip/_internal/utils/misc.py b/src/pip/_internal/utils/misc.py
index 82b626132..48771c099 100644
--- a/src/pip/_internal/utils/misc.py
+++ b/src/pip/_internal/utils/misc.py
@@ -745,3 +745,36 @@ class ConfiguredBuildBackendHookCaller(BuildBackendHookCaller):
             config_settings=cs,
             _allow_fallback=_allow_fallback,
         )
+
+
+def warn_if_run_as_root() -> None:
+    """Output a warning for sudo users on Unix.
+
+    In a virtual environment, sudo pip still writes to virtualenv.
+    On Windows, users may run pip as Administrator without issues.
+    This warning only applies to Unix root users outside of virtualenv.
+    """
+    if running_under_virtualenv():
+        return
+    if not hasattr(os, "getuid"):
+        return
+    # On Windows, there are no "system managed" Python packages. Installing as
+    # Administrator via pip is the correct way of updating system environments.
+    #
+    # We choose sys.platform over utils.compat.WINDOWS here to enable Mypy platform
+    # checks: https://mypy.readthedocs.io/en/stable/common_issues.html
+    if sys.platform == "win32" or sys.platform == "cygwin":
+        return
+
+    if os.getuid() != 0:
+        return
+
+    logger.warning(
+        "Running pip as the 'root' user can result in broken permissions and "
+        "conflicting behaviour with the system package manager, possibly "
+        "rendering your system unusable."
+        "It is recommended to use a virtual environment instead: "
+        "https://pip.pypa.io/warnings/venv. "
+        "Use the --root-user-action option if you know what you are doing and "
+        "want to suppress this warning."
+    )
diff --git a/src/pip/_internal/utils/unpacking.py b/src/pip/_internal/utils/unpacking.py
index 78b5c13ce..341269550 100644
--- a/src/pip/_internal/utils/unpacking.py
+++ b/src/pip/_internal/utils/unpacking.py
@@ -5,6 +5,7 @@ import logging
 import os
 import shutil
 import stat
+import sys
 import tarfile
 import zipfile
 from typing import Iterable, List, Optional
@@ -85,12 +86,16 @@ def is_within_directory(directory: str, target: str) -> bool:
     return prefix == abs_directory
 
 
+def _get_default_mode_plus_executable() -> int:
+    return 0o777 & ~current_umask() | 0o111
+
+
 def set_extracted_file_to_default_mode_plus_executable(path: str) -> None:
     """
     Make file present at path have execute for user/group/world
     (chmod +x) is no-op on windows per python docs
     """
-    os.chmod(path, (0o777 & ~current_umask() | 0o111))
+    os.chmod(path, _get_default_mode_plus_executable())
 
 
 def zip_item_is_executable(info: ZipInfo) -> bool:
@@ -151,8 +156,8 @@ def untar_file(filename: str, location: str) -> None:
     Untar the file (with path `filename`) to the destination `location`.
     All files are written based on system defaults and umask (i.e. permissions
     are not preserved), except that regular file members with any execute
-    permissions (user, group, or world) have "chmod +x" applied after being
-    written.  Note that for windows, any execute changes using os.chmod are
+    permissions (user, group, or world) have "chmod +x" applied on top of the
+    default.  Note that for windows, any execute changes using os.chmod are
     no-ops per the python docs.
     """
     ensure_dir(location)
@@ -170,62 +175,127 @@ def untar_file(filename: str, location: str) -> None:
             filename,
         )
         mode = "r:*"
+
     tar = tarfile.open(filename, mode, encoding="utf-8")
     try:
         leading = has_leading_dir([member.name for member in tar.getmembers()])
-        for member in tar.getmembers():
-            fn = member.name
-            if leading:
-                fn = split_leading_dir(fn)[1]
-            path = os.path.join(location, fn)
-            if not is_within_directory(location, path):
-                message = (
-                    "The tar file ({}) has a file ({}) trying to install "
-                    "outside target directory ({})"
-                )
-                raise InstallationError(message.format(filename, path, location))
-            if member.isdir():
-                ensure_dir(path)
-            elif member.issym():
-                try:
-                    tar._extract_member(member, path)
-                except Exception as exc:
-                    # Some corrupt tar files seem to produce this
-                    # (specifically bad symlinks)
-                    logger.warning(
-                        "In the tar file %s the member %s is invalid: %s",
-                        filename,
-                        member.name,
-                        exc,
-                    )
-                    continue
-            else:
+
+        # PEP 706 added `tarfile.data_filter`, and made some other changes to
+        # Python's tarfile module (see below). The features were backported to
+        # security releases.
+        try:
+            data_filter = tarfile.data_filter
+        except AttributeError:
+            _untar_without_filter(filename, location, tar, leading)
+        else:
+            default_mode_plus_executable = _get_default_mode_plus_executable()
+
+            def pip_filter(member: tarfile.TarInfo, path: str) -> tarfile.TarInfo:
+                if leading:
+                    member.name = split_leading_dir(member.name)[1]
+                orig_mode = member.mode
                 try:
-                    fp = tar.extractfile(member)
-                except (KeyError, AttributeError) as exc:
-                    # Some corrupt tar files seem to produce this
-                    # (specifically bad symlinks)
-                    logger.warning(
-                        "In the tar file %s the member %s is invalid: %s",
-                        filename,
-                        member.name,
-                        exc,
+                    try:
+                        member = data_filter(member, location)
+                    except tarfile.LinkOutsideDestinationError:
+                        if sys.version_info[:3] in {
+                            (3, 8, 17),
+                            (3, 9, 17),
+                            (3, 10, 12),
+                            (3, 11, 4),
+                        }:
+                            # The tarfile filter in specific Python versions
+                            # raises LinkOutsideDestinationError on valid input
+                            # (https://github.com/python/cpython/issues/107845)
+                            # Ignore the error there, but do use the
+                            # more lax `tar_filter`
+                            member = tarfile.tar_filter(member, location)
+                        else:
+                            raise
+                except tarfile.TarError as exc:
+                    message = "Invalid member in the tar file {}: {}"
+                    # Filter error messages mention the member name.
+                    # No need to add it here.
+                    raise InstallationError(
+                        message.format(
+                            filename,
+                            exc,
+                        )
                     )
-                    continue
-                ensure_dir(os.path.dirname(path))
-                assert fp is not None
-                with open(path, "wb") as destfp:
-                    shutil.copyfileobj(fp, destfp)
-                fp.close()
-                # Update the timestamp (useful for cython compiled files)
-                tar.utime(member, path)
-                # member have any execute permissions for user/group/world?
-                if member.mode & 0o111:
-                    set_extracted_file_to_default_mode_plus_executable(path)
+                if member.isfile() and orig_mode & 0o111:
+                    member.mode = default_mode_plus_executable
+                else:
+                    # See PEP 706 note above.
+                    # The PEP changed this from `int` to `Optional[int]`,
+                    # where None means "use the default". Mypy doesn't
+                    # know this yet.
+                    member.mode = None  # type: ignore [assignment]
+                return member
+
+            tar.extractall(location, filter=pip_filter)
+
     finally:
         tar.close()
 
 
+def _untar_without_filter(
+    filename: str,
+    location: str,
+    tar: tarfile.TarFile,
+    leading: bool,
+) -> None:
+    """Fallback for Python without tarfile.data_filter"""
+    for member in tar.getmembers():
+        fn = member.name
+        if leading:
+            fn = split_leading_dir(fn)[1]
+        path = os.path.join(location, fn)
+        if not is_within_directory(location, path):
+            message = (
+                "The tar file ({}) has a file ({}) trying to install "
+                "outside target directory ({})"
+            )
+            raise InstallationError(message.format(filename, path, location))
+        if member.isdir():
+            ensure_dir(path)
+        elif member.issym():
+            try:
+                tar._extract_member(member, path)
+            except Exception as exc:
+                # Some corrupt tar files seem to produce this
+                # (specifically bad symlinks)
+                logger.warning(
+                    "In the tar file %s the member %s is invalid: %s",
+                    filename,
+                    member.name,
+                    exc,
+                )
+                continue
+        else:
+            try:
+                fp = tar.extractfile(member)
+            except (KeyError, AttributeError) as exc:
+                # Some corrupt tar files seem to produce this
+                # (specifically bad symlinks)
+                logger.warning(
+                    "In the tar file %s the member %s is invalid: %s",
+                    filename,
+                    member.name,
+                    exc,
+                )
+                continue
+            ensure_dir(os.path.dirname(path))
+            assert fp is not None
+            with open(path, "wb") as destfp:
+                shutil.copyfileobj(fp, destfp)
+            fp.close()
+            # Update the timestamp (useful for cython compiled files)
+            tar.utime(member, path)
+            # member have any execute permissions for user/group/world?
+            if member.mode & 0o111:
+                set_extracted_file_to_default_mode_plus_executable(path)
+
+
 def unpack_file(
     filename: str,
     location: str,
diff --git a/src/pip/_vendor/distlib/scripts.py b/src/pip/_vendor/distlib/scripts.py
index cfa45d2af..e16292b83 100644
--- a/src/pip/_vendor/distlib/scripts.py
+++ b/src/pip/_vendor/distlib/scripts.py
@@ -49,6 +49,24 @@ if __name__ == '__main__':
     sys.exit(%(func)s())
 '''
 
+# Pre-fetch the contents of all executable wrapper stubs.
+# This is to address https://github.com/pypa/pip/issues/12666.
+# When updating pip, we rename the old pip in place before installing the
+# new version. If we try to fetch a wrapper *after* that rename, the finder
+# machinery will be confused as the package is no longer available at the
+# location where it was imported from. So we load everything into memory in
+# advance.
+
+# Issue 31: don't hardcode an absolute package name, but
+# determine it relative to the current package
+distlib_package = __name__.rsplit('.', 1)[0]
+
+WRAPPERS = {
+    r.name: r.bytes
+    for r in finder(distlib_package).iterator("")
+    if r.name.endswith(".exe")
+}
+
 
 def enquote_executable(executable):
     if ' ' in executable:
@@ -409,15 +427,11 @@ class ScriptMaker(object):
                 bits = '32'
             platform_suffix = '-arm' if get_platform() == 'win-arm64' else ''
             name = '%s%s%s.exe' % (kind, bits, platform_suffix)
-            # Issue 31: don't hardcode an absolute package name, but
-            # determine it relative to the current package
-            distlib_package = __name__.rsplit('.', 1)[0]
-            resource = finder(distlib_package).find(name)
-            if not resource:
+            if name not in WRAPPERS:
                 msg = ('Unable to find resource %s in package %s' %
                        (name, distlib_package))
                 raise ValueError(msg)
-            return resource.bytes
+            return WRAPPERS[name]
 
     # Public API follows
 
diff --git a/tools/vendoring/patches/distlib.patch b/tools/vendoring/patches/distlib.patch
new file mode 100644
index 000000000..de2834710
--- /dev/null
+++ b/tools/vendoring/patches/distlib.patch
@@ -0,0 +1,47 @@
+diff --git a/src/pip/_vendor/distlib/scripts.py b/src/pip/_vendor/distlib/scripts.py
+index cfa45d2af..e16292b83 100644
+--- a/src/pip/_vendor/distlib/scripts.py
++++ b/src/pip/_vendor/distlib/scripts.py
+@@ -49,6 +49,24 @@ if __name__ == '__main__':
+     sys.exit(%(func)s())
+ '''
+ 
++# Pre-fetch the contents of all executable wrapper stubs.
++# This is to address https://github.com/pypa/pip/issues/12666.
++# When updating pip, we rename the old pip in place before installing the
++# new version. If we try to fetch a wrapper *after* that rename, the finder
++# machinery will be confused as the package is no longer available at the
++# location where it was imported from. So we load everything into memory in
++# advance.
++
++# Issue 31: don't hardcode an absolute package name, but
++# determine it relative to the current package
++distlib_package = __name__.rsplit('.', 1)[0]
++
++WRAPPERS = {
++    r.name: r.bytes
++    for r in finder(distlib_package).iterator("")
++    if r.name.endswith(".exe")
++}
++
+ 
+ def enquote_executable(executable):
+     if ' ' in executable:
+@@ -409,15 +427,11 @@ class ScriptMaker(object):
+                 bits = '32'
+             platform_suffix = '-arm' if get_platform() == 'win-arm64' else ''
+             name = '%s%s%s.exe' % (kind, bits, platform_suffix)
+-            # Issue 31: don't hardcode an absolute package name, but
+-            # determine it relative to the current package
+-            distlib_package = __name__.rsplit('.', 1)[0]
+-            resource = finder(distlib_package).find(name)
+-            if not resource:
++            if name not in WRAPPERS:
+                 msg = ('Unable to find resource %s in package %s' %
+                        (name, distlib_package))
+                 raise ValueError(msg)
+-            return resource.bytes
++            return WRAPPERS[name]
+ 
+     # Public API follows
+ 


Test patch:
diff --git a/tests/functional/test_cli.py b/tests/functional/test_cli.py
index 3c1661521..e1ccf04ea 100644
--- a/tests/functional/test_cli.py
+++ b/tests/functional/test_cli.py
@@ -57,8 +57,7 @@ def test_entrypoints_work(entrypoint: str, script: PipTestEnvironment) -> None:
     sorted(
         set(commands_dict).symmetric_difference(
             # Exclude commands that are expected to use the network.
-            # TODO: uninstall and list should only import network modules as needed
-            {"install", "uninstall", "download", "search", "index", "wheel", "list"}
+            {"install", "download", "search", "index", "wheel"}
         )
     ),
 )
diff --git a/tests/functional/test_self_update.py b/tests/functional/test_self_update.py
new file mode 100644
index 000000000..c50755220
--- /dev/null
+++ b/tests/functional/test_self_update.py
@@ -0,0 +1,22 @@
+# Check that pip can update itself correctly
+
+from typing import Any
+
+
+def test_self_update_editable(script: Any, pip_src: Any) -> None:
+    # Test that if we have an environment with pip installed in non-editable
+    # mode, that pip can safely update itself to an editable install.
+    # See https://github.com/pypa/pip/issues/12666 for details.
+
+    # Step 1. Install pip as non-editable. This is expected to succeed as
+    # the existing pip in the environment is installed in editable mode, so
+    # it only places a .pth file in the environment.
+    proc = script.pip("install", pip_src)
+    assert proc.returncode == 0
+    # Step 2. Using the pip we just installed, install pip *again*, but
+    # in editable mode. This could fail, as we'll need to uninstall the running
+    # pip in order to install the new copy, and uninstalling pip while it's
+    # running could fail. This test is specifically to ensure that doesn't
+    # happen...
+    proc = script.pip("install", "-e", pip_src)
+    assert proc.returncode == 0
diff --git a/tests/unit/test_base_command.py b/tests/unit/test_base_command.py
index 44dae384a..80c93799d 100644
--- a/tests/unit/test_base_command.py
+++ b/tests/unit/test_base_command.py
@@ -93,7 +93,7 @@ class TestCommand:
         assert "Traceback (most recent call last):" in stderr
 
 
-@patch("pip._internal.cli.req_command.Command.handle_pip_version_check")
+@patch("pip._internal.cli.index_command.Command.handle_pip_version_check")
 def test_handle_pip_version_check_called(mock_handle_version_check: Mock) -> None:
     """
     Check that Command.handle_pip_version_check() is called.
diff --git a/tests/unit/test_commands.py b/tests/unit/test_commands.py
index 4f6366513..9d5aefec2 100644
--- a/tests/unit/test_commands.py
+++ b/tests/unit/test_commands.py
@@ -87,7 +87,7 @@ def test_index_group_commands() -> None:
         (True, True, False),
     ],
 )
-@mock.patch("pip._internal.cli.req_command.pip_self_version_check")
+@mock.patch("pip._internal.cli.index_command._pip_self_version_check")
 def test_index_group_handle_pip_version_check(
     mock_version_check: mock.Mock,
     command_name: str,
@@ -128,7 +128,7 @@ def test_requirement_commands() -> None:
 
 
 @pytest.mark.parametrize("flag", ["", "--outdated", "--uptodate"])
-@mock.patch("pip._internal.cli.req_command.pip_self_version_check")
+@mock.patch("pip._internal.cli.index_command._pip_self_version_check")
 @mock.patch.dict(os.environ, {"PIP_DISABLE_PIP_VERSION_CHECK": "no"})
 def test_list_pip_version_check(version_check_mock: mock.Mock, flag: str) -> None:
     """
diff --git a/tests/unit/test_utils_unpacking.py b/tests/unit/test_utils_unpacking.py
index 1f0b59dbd..3fdd822e7 100644
--- a/tests/unit/test_utils_unpacking.py
+++ b/tests/unit/test_utils_unpacking.py
@@ -155,7 +155,13 @@ class TestUnpackArchives:
         test_tar = self.make_tar_file("test_tar.tar", files)
         with pytest.raises(InstallationError) as e:
             untar_file(test_tar, self.tempdir)
-        assert "trying to install outside target directory" in str(e.value)
+
+        # The error message comes from tarfile.data_filter when it is available,
+        # otherwise from pip's own check.
+        if hasattr(tarfile, "data_filter"):
+            assert "is outside the destination" in str(e.value)
+        else:
+            assert "trying to install outside target directory" in str(e.value)
 
     def test_unpack_tar_success(self) -> None:
         """
@@ -171,6 +177,26 @@ class TestUnpackArchives:
         test_tar = self.make_tar_file("test_tar.tar", files)
         untar_file(test_tar, self.tempdir)
 
+    @pytest.mark.skipif(
+        not hasattr(tarfile, "data_filter"),
+        reason="tarfile filters (PEP-721) not available",
+    )
+    def test_unpack_tar_filter(self) -> None:
+        """
+        Test that the tarfile.data_filter is used to disallow dangerous
+        behaviour (PEP-721)
+        """
+        test_tar = os.path.join(self.tempdir, "test_tar_filter.tar")
+        with tarfile.open(test_tar, "w") as mytar:
+            file_tarinfo = tarfile.TarInfo("bad-link")
+            file_tarinfo.type = tarfile.SYMTYPE
+            file_tarinfo.linkname = "../../../../pwn"
+            mytar.addfile(file_tarinfo, io.BytesIO(b""))
+        with pytest.raises(InstallationError) as e:
+            untar_file(test_tar, self.tempdir)
+
+        assert "is outside the destination" in str(e.value)
+
 
 def test_unpack_tar_unicode(tmpdir: Path) -> None:
     test_tar = tmpdir / "test.tar"