Repository: pytest-dev/pytest
Pull request: add --disable-plugin-autoload
fixes #8969

I'm pretty sure the implementation is solid, but I have a bunch of question marks in comments that should be resolved before merging. I can probably resolve them on my own by digging deep enough into the innards, but wouldn't mind explanations if somebody is more familiar with that territory.

I'm not entirely sure that `test_installed_plugin_rewrite` should be parametrized to hell, as opposed to creating a dedicated test.

I also found a random unrelated typo in `test_installed_plugin_rewrite` with `check_first` -> `check_first2`, so fixed that + improved the `fnmatch_lines`.

- [x] Include documentation when adding new features.
- [x] Include new tests or update existing tests when applicable.
- [X] Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.
- [ ] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](https://github.com/pytest-dev/pytest/blob/main/changelog/README.rst) for details.



Problem statement:
Disable plugin autoload via configuration file
#### What's the problem this feature will solve?
I want to use a plugin only when certain conditions are met. Currently, in order to do that I can set the environment variable `PYTEST_DISABLE_PLUGIN_AUTOLOAD` when calling pytest. Since disabling autoload is the "common path" for the project I'm working on, it would be very convenient to have a way to specify that in the configuration file.

#### Describe the solution you'd like
<!-- A clear and concise description of what you want to happen. -->
I can think of some solutions for this problem:
1. Have a CLI option (e.g. `--disable-plugin-autoload`) with the same semantics as for setting `PYTEST_DISABLE_PLUGIN_AUTOLOAD`. That way, I can simply use that for the `addopts` property of the configuration;
2. Have an option in the configuration for defining environment variables. Then I could set `PYTEST_DISABLE_PLUGIN_AUTOLOAD` there;
3. Having a specific option in the configuration for disabling autoload.

Out of the three options above, I'm leaning more toward option (1) because (i) it would also allow disabling autoload from the command line without having to prefix the command with the env var assignment; and (ii) it does not require a new configuration option.

Best regards,
Gustavo Sousa

Hints from issue discussion:

Commit message: pemdas except not

Patch to review:
diff --git a/changelog/13253.feature.rst b/changelog/13253.feature.rst
new file mode 100644
index 00000000000..e497c207223
--- /dev/null
+++ b/changelog/13253.feature.rst
@@ -0,0 +1 @@
+New flag: :ref:`--disable-plugin-autoload <disable_plugin_autoload>` which works as an alternative to :envvar:`PYTEST_DISABLE_PLUGIN_AUTOLOAD` when setting environment variables is inconvenient; and allows setting it in config files with :confval:`addopts`.
diff --git a/doc/en/how-to/plugins.rst b/doc/en/how-to/plugins.rst
index 7d5bcd85a31..b94f14a2f85 100644
--- a/doc/en/how-to/plugins.rst
+++ b/doc/en/how-to/plugins.rst
@@ -133,4 +133,25 @@ CI server), you can set ``PYTEST_ADDOPTS`` environment variable to
 
 See :ref:`findpluginname` for how to obtain the name of a plugin.
 
-.. _`builtin plugins`:
+.. _`disable_plugin_autoload`:
+
+Disabling plugins from autoloading
+----------------------------------
+
+If you want to disable plugins from loading automatically, requiring you to
+manually specify each plugin with ``-p`` or :envvar:`PYTEST_PLUGINS`, you can use ``--disable-plugin-autoload`` or :envvar:`PYTEST_DISABLE_PLUGIN_AUTOLOAD`.
+
+.. code-block:: bash
+
+   export PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
+   export PYTEST_PLUGINS=NAME
+   pytest
+
+.. code-block:: bash
+
+   pytest --disable-plugin-autoload -p NAME,NAME2
+
+.. code-block:: ini
+
+    [pytest]
+    addopts = --disable-plugin-autoload -p NAME,NAME2
diff --git a/doc/en/reference/reference.rst b/doc/en/reference/reference.rst
index 809e97b4747..3eb6812df8b 100644
--- a/doc/en/reference/reference.rst
+++ b/doc/en/reference/reference.rst
@@ -1161,8 +1161,9 @@ as discussed in :ref:`temporary directory location and retention`.
 .. envvar:: PYTEST_DISABLE_PLUGIN_AUTOLOAD
 
 When set, disables plugin auto-loading through :std:doc:`entry point packaging
-metadata <packaging:guides/creating-and-discovering-plugins>`. Only explicitly
-specified plugins will be loaded.
+metadata <packaging:guides/creating-and-discovering-plugins>`. Only plugins
+explicitly specified in :envvar:`PYTEST_PLUGINS` or with ``-p`` will be loaded.
+See also :ref:`--disable-plugin-autoload <disable_plugin_autoload>`.
 
 .. envvar:: PYTEST_PLUGINS
 
@@ -1172,6 +1173,8 @@ Contains comma-separated list of modules that should be loaded as plugins:
 
     export PYTEST_PLUGINS=mymodule.plugin,xdist
 
+See also ``-p``.
+
 .. envvar:: PYTEST_THEME
 
 Sets a `pygment style <https://pygments.org/docs/styles/>`_ to use for the code output.
diff --git a/src/_pytest/config/__init__.py b/src/_pytest/config/__init__.py
index 02da5cf9229..aacf2c12c06 100644
--- a/src/_pytest/config/__init__.py
+++ b/src/_pytest/config/__init__.py
@@ -70,6 +70,7 @@
 
 
 if TYPE_CHECKING:
+    from _pytest.assertions.rewrite import AssertionRewritingHook
     from _pytest.cacheprovider import Cache
     from _pytest.terminal import TerminalReporter
 
@@ -1271,6 +1272,10 @@ def _consider_importhook(self, args: Sequence[str]) -> None:
         """
         ns, unknown_args = self._parser.parse_known_and_unknown_args(args)
         mode = getattr(ns, "assertmode", "plain")
+
+        disable_autoload = getattr(ns, "disable_plugin_autoload", False) | bool(
+            os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD")
+        )
         if mode == "rewrite":
             import _pytest.assertion
 
@@ -1279,16 +1284,18 @@ def _consider_importhook(self, args: Sequence[str]) -> None:
             except SystemError:
                 mode = "plain"
             else:
-                self._mark_plugins_for_rewrite(hook)
+                self._mark_plugins_for_rewrite(hook, disable_autoload)
         self._warn_about_missing_assertion(mode)
 
-    def _mark_plugins_for_rewrite(self, hook) -> None:
+    def _mark_plugins_for_rewrite(
+        self, hook: AssertionRewritingHook, disable_autoload: bool
+    ) -> None:
         """Given an importhook, mark for rewrite any top-level
         modules or packages in the distribution package for
         all pytest plugins."""
         self.pluginmanager.rewrite_hook = hook
 
-        if os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD"):
+        if disable_autoload:
             # We don't autoload from distribution package entry points,
             # no need to continue.
             return
@@ -1393,10 +1400,15 @@ def _preparse(self, args: list[str], addopts: bool = True) -> None:
         self._consider_importhook(args)
         self._configure_python_path()
         self.pluginmanager.consider_preparse(args, exclude_only=False)
-        if not os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD"):
-            # Don't autoload from distribution package entry point. Only
-            # explicitly specified plugins are going to be loaded.
+        if (
+            not os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD")
+            and not self.known_args_namespace.disable_plugin_autoload
+        ):
+            # Autoloading from distribution package entry point has
+            # not been disabled.
             self.pluginmanager.load_setuptools_entrypoints("pytest11")
+        # Otherwise only plugins explicitly specified in PYTEST_PLUGINS
+        # are going to be loaded.
         self.pluginmanager.consider_env()
 
         self.known_args_namespace = self._parser.parse_known_args(
@@ -1419,7 +1431,7 @@ def _preparse(self, args: list[str], addopts: bool = True) -> None:
         except ConftestImportFailure as e:
             if self.known_args_namespace.help or self.known_args_namespace.version:
                 # we don't want to prevent --help/--version to work
-                # so just let is pass and print a warning at the end
+                # so just let it pass and print a warning at the end
                 self.issue_config_time_warning(
                     PytestConfigWarning(f"could not load initial conftests: {e.path}"),
                     stacklevel=2,
diff --git a/src/_pytest/helpconfig.py b/src/_pytest/helpconfig.py
index 2b377c70f7b..d96ae19f6cc 100644
--- a/src/_pytest/helpconfig.py
+++ b/src/_pytest/helpconfig.py
@@ -70,7 +70,14 @@ def pytest_addoption(parser: Parser) -> None:
         metavar="name",
         help="Early-load given plugin module name or entry point (multi-allowed). "
         "To avoid loading of plugins, use the `no:` prefix, e.g. "
-        "`no:doctest`.",
+        "`no:doctest`. See also --disable-plugin-autoload",
+    )
+    group.addoption(
+        "--disable-plugin-autoload",
+        action="store_true",
+        default=False,
+        help="Disable plugin auto-loading through entry point packaging metadata. "
+        "Only plugins explicitly specified in -p or env var PYTEST_PLUGINS will be loaded.",
     )
     group.addoption(
         "--traceconfig",
diff --git a/testing/test_assertion.py b/testing/test_assertion.py
index e3d45478466..04284e0a55d 100644
--- a/testing/test_assertion.py
+++ b/testing/test_assertion.py
@@ -218,10 +218,40 @@ def test_foo(pytestconfig):
         assert result.ret == 0
 
     @pytest.mark.parametrize("mode", ["plain", "rewrite"])
+    @pytest.mark.parametrize("disable_plugin_autoload", ["env_var", "cli", ""])
+    @pytest.mark.parametrize("explicit_specify", ["env_var", "cli", ""])
     def test_installed_plugin_rewrite(
-        self, pytester: Pytester, mode, monkeypatch
+        self,
+        pytester: Pytester,
+        mode: str,
+        monkeypatch: pytest.MonkeyPatch,
+        disable_plugin_autoload: str,
+        explicit_specify: str,
     ) -> None:
-        monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", raising=False)
+        args = ["mainwrapper.py", "-s", f"--assert={mode}"]
+        if disable_plugin_autoload == "env_var":
+            monkeypatch.setenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", "1")
+        elif disable_plugin_autoload == "cli":
+            monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", raising=False)
+            args.append("--disable-plugin-autoload")
+        else:
+            assert disable_plugin_autoload == ""
+            monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", raising=False)
+
+        # FIXME: if it's already loaded then you get a ValueError: "Plugin already
+        # registered under a different name."
+        # vaguely related to https://github.com/pytest-dev/pytest/issues/5661
+        name = "spamplugin" if disable_plugin_autoload else "spam"
+        # is there a single dotted name that can be used either way? idk
+
+        if explicit_specify == "env_var":
+            monkeypatch.setenv("PYTEST_PLUGINS", name)
+        elif explicit_specify == "cli":
+            args.append("-p")
+            args.append(name)
+        else:
+            assert explicit_specify == ""
+
         # Make sure the hook is installed early enough so that plugins
         # installed via distribution package are rewritten.
         pytester.mkdir("hampkg")
@@ -275,20 +305,29 @@ def test(check_first):
                 check_first([10, 30], 30)
 
             def test2(check_first2):
-                check_first([10, 30], 30)
+                check_first2([10, 30], 30)
             """,
         }
         pytester.makepyfile(**contents)
-        result = pytester.run(
-            sys.executable, "mainwrapper.py", "-s", f"--assert={mode}"
-        )
+        result = pytester.run(sys.executable, *args)
         if mode == "plain":
             expected = "E       AssertionError"
         elif mode == "rewrite":
             expected = "*assert 10 == 30*"
         else:
             assert 0
-        result.stdout.fnmatch_lines([expected])
+
+        if not disable_plugin_autoload or explicit_specify:
+            result.assert_outcomes(failed=2)
+            result.stdout.fnmatch_lines([expected, expected])
+        else:
+            result.assert_outcomes(errors=2)
+            result.stdout.fnmatch_lines(
+                [
+                    "E       fixture 'check_first' not found",
+                    "E       fixture 'check_first2' not found",
+                ]
+            )
 
     def test_rewrite_ast(self, pytester: Pytester) -> None:
         pytester.mkdir("pkg")
diff --git a/testing/test_config.py b/testing/test_config.py
index de07141238c..52fdaadbec7 100644
--- a/testing/test_config.py
+++ b/testing/test_config.py
@@ -6,6 +6,7 @@
 import importlib.metadata
 import os
 from pathlib import Path
+import platform
 import re
 import sys
 import textwrap
@@ -1314,14 +1315,13 @@ def distributions():
         )
 
 
-@pytest.mark.parametrize(
-    "parse_args,should_load", [(("-p", "mytestplugin"), True), ((), False)]
-)
+@pytest.mark.parametrize("disable_plugin_method", ["env_var", "flag", ""])
+@pytest.mark.parametrize("enable_plugin_method", ["env_var", "flag", ""])
 def test_disable_plugin_autoload(
     pytester: Pytester,
     monkeypatch: MonkeyPatch,
-    parse_args: tuple[str, str] | tuple[()],
-    should_load: bool,
+    enable_plugin_method: str,
+    disable_plugin_method: str,
 ) -> None:
     class DummyEntryPoint:
         project_name = name = "mytestplugin"
@@ -1342,23 +1342,60 @@ class PseudoPlugin:
         attrs_used = []
 
         def __getattr__(self, name):
-            assert name == "__loader__"
+            assert name in ("__loader__", "__spec__")
             self.attrs_used.append(name)
             return object()
 
     def distributions():
         return (Distribution(),)
 
-    monkeypatch.setenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", "1")
+    parse_args: list[str] = []
+
+    if disable_plugin_method == "env_var":
+        monkeypatch.setenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", "1")
+    elif disable_plugin_method == "flag":
+        monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD")
+        parse_args.append("--disable-plugin-autoload")
+    else:
+        assert disable_plugin_method == ""
+        monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD")
+
+    if enable_plugin_method == "env_var":
+        monkeypatch.setenv("PYTEST_PLUGINS", "mytestplugin")
+    elif enable_plugin_method == "flag":
+        parse_args.extend(["-p", "mytestplugin"])
+    else:
+        assert enable_plugin_method == ""
+
     monkeypatch.setattr(importlib.metadata, "distributions", distributions)
     monkeypatch.setitem(sys.modules, "mytestplugin", PseudoPlugin())
     config = pytester.parseconfig(*parse_args)
+
     has_loaded = config.pluginmanager.get_plugin("mytestplugin") is not None
-    assert has_loaded == should_load
-    if should_load:
-        assert PseudoPlugin.attrs_used == ["__loader__"]
-    else:
-        assert PseudoPlugin.attrs_used == []
+    # it should load if it's enabled, or we haven't disabled autoloading
+    assert has_loaded == bool(enable_plugin_method) or not disable_plugin_method
+
+    # The reason for the discrepancy between 'has_loaded' and __loader__ being accessed
+    # appears to be the monkeypatching of importlib.metadata.distributions; where
+    # files being empty means that _mark_plugins_for_rewrite doesn't find the plugin.
+    # But enable_method==flag ends up in mark_rewrite being called and __loader__
+    # being accessed.
+    assert ("__loader__" in PseudoPlugin.attrs_used) == (
+        has_loaded
+        and not (enable_plugin_method in ("env_var", "") and not disable_plugin_method)
+    )
+
+    # __spec__ is accessed in AssertionRewritingHook.exec_module, which would be
+    # eventually called if we did a full pytest run; but it's only accessed with
+    # enable_plugin_method=="env_var" because that will early-load it.
+    # Except when autoloads aren't disabled, in which case PytestPluginManager.import_plugin
+    # bails out before importing it.. because it knows it'll be loaded later?
+    # The above seems a bit weird, but I *think* it's true.
+    if platform.python_implementation() != "PyPy":
+        assert ("__spec__" in PseudoPlugin.attrs_used) == bool(
+            enable_plugin_method == "env_var" and disable_plugin_method
+        )
+    # __spec__ is present when testing locally on pypy, but not in CI ????
 
 
 def test_plugin_loading_order(pytester: Pytester) -> None:


Code review comments:
diff_hunk: @@ -1342,23 +1342,60 @@ class PseudoPlugin:
         attrs_used = []
 
         def __getattr__(self, name):
-            assert name == "__loader__"
+            assert name in ("__loader__", "__spec__")
             self.attrs_used.append(name)
             return object()
 
     def distributions():
         return (Distribution(),)
 
-    monkeypatch.setenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", "1")
+    parse_args: list[str] = []
+
+    if disable_plugin_method == "env_var":
+        monkeypatch.setenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", "1")
+    elif disable_plugin_method == "flag":
+        monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD")
+        parse_args.append("--disable-plugin-autoload")
+    else:
+        assert disable_plugin_method == ""
+        monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD")
+
+    if enable_plugin_method == "env_var":
+        monkeypatch.setenv("PYTEST_PLUGINS", "mytestplugin")
+    elif enable_plugin_method == "flag":
+        parse_args.extend(["-p", "mytestplugin"])
+    else:
+        assert enable_plugin_method == ""
+
     monkeypatch.setattr(importlib.metadata, "distributions", distributions)
     monkeypatch.setitem(sys.modules, "mytestplugin", PseudoPlugin())
     config = pytester.parseconfig(*parse_args)
+
     has_loaded = config.pluginmanager.get_plugin("mytestplugin") is not None
-    assert has_loaded == should_load
-    if should_load:
-        assert PseudoPlugin.attrs_used == ["__loader__"]
-    else:
-        assert PseudoPlugin.attrs_used == []
+    # it should load if it's enabled, or we haven't disabled autoloading
+    assert has_loaded == bool(enable_plugin_method) or not disable_plugin_method
path: testing/test_config.py
text: @user1:
using parens here would be good to make the precedence more obvious

@user2:
I'd expect `ruff format` to delete them again, no?

@author:
it didn't!

diff_hunk: @@ -133,4 +133,25 @@ CI server), you can set ``PYTEST_ADDOPTS`` environment variable to
 
 See :ref:`findpluginname` for how to obtain the name of a plugin.
 
-.. _`builtin plugins`:
+.. _`disable_plugin_autoload`:
+
+Disabling plugins from autoloading
+----------------------------------
+
+If you want to disable plugins from loading automatically, requiring you to
+manually specify each plugin with ``-p`` or :envvar:`PYTEST_PLUGINS`, you can use ``--disable-plugin-autoload`` or :envvar:`PYTEST_DISABLE_PLUGIN_AUTOLOAD`.
+
+.. code-block:: bash
+
+   export PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
+   export PYTEST_PLUGINS=NAME
path: doc/en/how-to/plugins.rst
text: @user1:
nitpick:

```suggestion
   export PYTEST_PLUGINS=NAME,NAME2
```

for consistency with the others, and to avoid confusion (otherwise it could look like passing multiple isn't possible here).

diff_hunk: @@ -70,7 +70,14 @@ def pytest_addoption(parser: Parser) -> None:
         metavar="name",
         help="Early-load given plugin module name or entry point (multi-allowed). "
         "To avoid loading of plugins, use the `no:` prefix, e.g. "
-        "`no:doctest`.",
+        "`no:doctest`. See also --disable-plugin-autoload",
path: src/_pytest/helpconfig.py
text: @user1:
```suggestion
        "`no:doctest`. See also --disable-plugin-autoload.",
```