Repository: pytest-dev/pytest-django (python)

Issue:
`pytest --help` fails in a partially configured app
having a difficult time narrowing down a minimal example -- the repo involved is https://github.com/getsentry/sentry

I have figured out _why_ it is happening and the stacktrace for it:

<summary>full stacktrace with error

<details>

```console
$ pytest --help
/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/trio/_core/_multierror.py:511: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
  warnings.warn(
Traceback (most recent call last):
  File "/Users/asottile/workspace/sentry/.venv/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 190, in console_main
    code = main()
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 167, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/_pytest/helpconfig.py", line 152, in pytest_cmdline_main
    config._do_configure()
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1037, in _do_configure
    self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/hooks.py", line 308, in call_historic
    res = self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pytest_django/plugin.py", line 367, in pytest_configure
    _setup_django(config)
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/pytest_django/plugin.py", line 238, in _setup_django
    blocking_manager = config.stash[blocking_manager_key]
  File "/Users/asottile/workspace/sentry/.venv/lib/python3.10/site-packages/_pytest/stash.py", line 80, in __getitem__
    return cast(T, self._storage[key])
KeyError: <_pytest.stash.StashKey object at 0x1066ab520>
```

</details>
</summary>

basically what's happening is the setup is skipped here:  https://github.com/pytest-dev/pytest-django/blob/6cf63b65e86870abf68ae1f376398429e35864e7/pytest_django/plugin.py#L300-L301

normally it sets the thing that's being looked up here: https://github.com/pytest-dev/pytest-django/blob/6cf63b65e86870abf68ae1f376398429e35864e7/pytest_django/plugin.py#L358

which then fails to lookup here: https://github.com/pytest-dev/pytest-django/blob/6cf63b65e86870abf68ae1f376398429e35864e7/pytest_django/plugin.py#L238

something about sentry's `tests/conftest.py` initializes enough of django that `pytest-django` takes over.  but since the setup has been skipped it fails to set up properly.  I suspect that #238 is playing poorly with something.

of note this worked before I upgraded `pytest-django` (I was previously on 4.4.0 and upgraded to 4.7.0 to get django 4.x support)

will try and narrow down a smaller reproduction...


Discussion:
That’s a fun one! Hopefully using `config.stash.get()` calls and acting only on non-`None` values will be enough to fix the issue...
here's a minimal case:

```console
==> t.py <==
WAT = 1

==> tests/__init__.py <==

==> tests/conftest.py <==
import os

def pytest_configure():
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 't')
    from django.conf import settings
    settings.WAT
```

Patch:
diff --git a/pytest_django/plugin.py b/pytest_django/plugin.py
--- a/pytest_django/plugin.py
+++ b/pytest_django/plugin.py
@@ -362,8 +362,15 @@ def _get_option_with_source(
 
 @pytest.hookimpl(trylast=True)
 def pytest_configure(config: pytest.Config) -> None:
-    # Allow Django settings to be configured in a user pytest_configure call,
-    # but make sure we call django.setup()
+    if config.getoption("version", 0) > 0 or config.getoption("help", False):
+        return
+
+    # Normally Django is set up in `pytest_load_initial_conftests`, but we also
+    # allow users to not set DJANGO_SETTINGS_MODULE/`--ds` and instead
+    # configure the Django settings in a `pytest_configure` hookimpl using e.g.
+    # `settings.configure(...)`. In this case, the `_setup_django` call in
+    # `pytest_load_initial_conftests` only partially initializes Django, and
+    # it's fully initialized here.
     _setup_django(config)
 
 
@@ -470,8 +477,7 @@ def get_order_number(test: pytest.Item) -> int:
 
 @pytest.fixture(autouse=True, scope="session")
 def django_test_environment(request: pytest.FixtureRequest) -> Generator[None, None, None]:
-    """
-    Ensure that Django is loaded and has its testing environment setup.
+    """Setup Django's test environment for the testing session.
 
     XXX It is a little dodgy that this is an autouse fixture.  Perhaps
         an email fixture should be requested in order to be able to
@@ -481,7 +487,6 @@ def django_test_environment(request: pytest.FixtureRequest) -> Generator[None, N
         we need to follow this model.
     """
     if django_settings_is_configured():
-        _setup_django(request.config)
         from django.test.utils import setup_test_environment, teardown_test_environment
 
         debug_ini = request.config.getini("django_debug_mode")


Test patch:
diff --git a/tests/test_manage_py_scan.py b/tests/test_manage_py_scan.py
--- a/tests/test_manage_py_scan.py
+++ b/tests/test_manage_py_scan.py
@@ -144,6 +144,37 @@ def test_django_project_found_invalid_settings_version(
     result.stdout.fnmatch_lines(["*usage:*"])
 
 
+@pytest.mark.django_project(project_root="django_project_root", create_manage_py=True)
+def test_django_project_late_settings_version(
+    django_pytester: DjangoPytester,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Late configuration should not cause an error with --help or --version."""
+    monkeypatch.delenv("DJANGO_SETTINGS_MODULE")
+    django_pytester.makepyfile(
+        t="WAT = 1",
+    )
+    django_pytester.makeconftest(
+        """
+        import os
+
+        def pytest_configure():
+            os.environ.setdefault('DJANGO_SETTINGS_MODULE', 't')
+            from django.conf import settings
+            settings.WAT
+        """
+    )
+
+    result = django_pytester.runpytest_subprocess("django_project_root", "--version", "--version")
+    assert result.ret == 0
+
+    result.stdout.fnmatch_lines(["*This is pytest version*"])
+
+    result = django_pytester.runpytest_subprocess("django_project_root", "--help")
+    assert result.ret == 0
+    result.stdout.fnmatch_lines(["*usage:*"])
+
+
 @pytest.mark.django_project(project_root="django_project_root", create_manage_py=True)
 def test_runs_without_error_on_long_args(django_pytester: DjangoPytester) -> None:
     django_pytester.create_test_module(