Repository: nilearn/nilearn

Issue:
Tests failing on main with pytest 8.1.0
See https://github.com/nilearn/nilearn/actions/runs/8136733065/job/22233621361

```
test_plotting: install_deps> python -I -m pip install kaleido 'kaleido; platform_system != "Windows"' 'kaleido==0.1.0.post1; platform_system == "Windows"' 'matplotlib>=3.3.0' plotly
.pkg: install_requires> python -I -m pip install hatch-vcs hatchling
.pkg: _optional_hooks> python /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyproject_api/_backend.py True hatchling.build
.pkg: get_requires_for_build_sdist> python /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyproject_api/_backend.py True hatchling.build
.pkg: freeze> python -m pip freeze --all
.pkg: editables==0.5,hatch-vcs==0.4.0,hatchling==1.21.1,packaging==23.2,pathspec==0.12.1,pip==24.0,pluggy==1.4.0,setuptools==69.1.1,setuptools-scm==8.0.4,trove-classifiers==2024.3.3,typing_extensions==4.10.0
.pkg: build_sdist> python /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyproject_api/_backend.py True hatchling.build
test_plotting: install_package_deps> python -I -m pip install coverage 'joblib>=1.0.0' lxml 'nibabel>=4.0.0' 'numpy>=1.19.0' packaging 'pandas>=1.1.5' pytest-cov 'pytest>=6.0.0' 'requests>=2.25.0' 'scikit-learn>=1.0.0' 'scipy>=1.8.0'
test_plotting: install_package> python -I -m pip install --force-reinstall --no-deps /Users/runner/work/nilearn/nilearn/.tox/.tmp/package/1/nilearn-0.1.dev1+gf91de22.tar.gz
test_plotting: freeze> python -m pip freeze --all
test_plotting: certifi==2024.2.2,charset-normalizer==3.3.2,contourpy==1.2.0,coverage==7.4.3,cycler==0.12.1,fonttools==4.49.0,idna==3.6,iniconfig==2.0.0,joblib==1.3.2,kaleido==0.2.1,kiwisolver==1.4.5,lxml==5.1.0,matplotlib==3.8.3,nibabel==5.2.1,nilearn @ file:///Users/runner/work/nilearn/nilearn/.tox/.tmp/package/1/nilearn-0.1.dev1%2Bgf91de22.tar.gz#sha256=b29f617bbb5d9aa3a94aa7518f1006aea9f52d58d945a8b82ed6951bffccb22e,numpy==1.26.4,packaging==23.2,pandas==2.2.1,pillow==10.2.0,pip==24.0,plotly==5.19.0,pluggy==1.4.0,pyparsing==3.1.1,pytest==8.1.0,pytest-cov==4.1.0,python-dateutil==2.9.0.post0,pytz==2024.1,requests==2.31.0,scikit-learn==1.4.1.post1,scipy==1.12.0,six==1.16.0,tenacity==8.2.3,threadpoolctl==3.3.0,tzdata==2024.1,urllib3==2.2.1
test_plotting: commands[0]> pytest --cov=nilearn --cov-report=xml nilearn
============================= test session starts ==============================
platform darwin -- Python 3.12.1, pytest-8.1.0, pluggy-1.4.0 -- /Users/runner/work/nilearn/nilearn/.tox/test_plotting/bin/python
cachedir: .tox/test_plotting/.pytest_cache
rootdir: /Users/runner/work/nilearn/nilearn
configfile: pyproject.toml
plugins: cov-4.1.0
collecting ... collected 3118 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting nilearn/externals ______________________
.tox/test_plotting/lib/python3.12/site-packages/pluggy/_manager.py:167: in register
    self._verify_hook(hook, hookimpl)
        hook       = <HookCaller 'pytest_ignore_collect'>
        hookimpl   = <HookImpl plugin_name='/Users/runner/work/nilearn/nilearn/nilearn/externals/conftest.py', plugin=<module 'nilearn.externals.conftest' from '/Users/runner/work/nilearn/nilearn/nilearn/externals/conftest.py'>>
        hookimpl_opts = {'hookwrapper': False,
 'optionalhook': False,
 'specname': None,
 'tryfirst': False,
 'trylast': False,
 'wrapper': False}
        method     = <function pytest_ignore_collect at 0x12f044680>
        name       = 'pytest_ignore_collect'
        plugin     = <module 'nilearn.externals.conftest' from '/Users/runner/work/nilearn/nilearn/nilearn/externals/conftest.py'>
        plugin_name = '/Users/runner/work/nilearn/nilearn/nilearn/externals/conftest.py'
        self       = <_pytest.config.PytestPluginManager object at 0x10c395130>
.tox/test_plotting/lib/python3.12/site-packages/pluggy/_manager.py:342: in _verify_hook
    raise PluginValidationError(
E   pluggy._manager.PluginValidationError: Plugin '/Users/runner/work/nilearn/nilearn/nilearn/externals/conftest.py' for hook 'pytest_ignore_collect'
E   hookimpl definition: pytest_ignore_collect(path, config)
E   Argument(s) {'path'} are declared in the hookimpl but can not be found in the hookspec
        hook       = <HookCaller 'pytest_ignore_collect'>
        hookimpl   = <HookImpl plugin_name='/Users/runner/work/nilearn/nilearn/nilearn/externals/conftest.py', plugin=<module 'nilearn.externals.conftest' from '/Users/runner/work/nilearn/nilearn/nilearn/externals/conftest.py'>>
        notinspec  = {'path'}
        self       = <_pytest.config.PytestPluginManager object at 0x10c395130>
=============================== warnings summary ===============================
nilearn/input_data/__init__.py:23
  /Users/runner/work/nilearn/nilearn/nilearn/input_data/__init__.py:23: DeprecationWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
    warnings.warn(message, DeprecationWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform darwin, python 3.12.1-final-0 ----------
Coverage XML written to file coverage.xml

=========================== short test summary info ============================
ERROR nilearn/externals - pluggy._manager.PluginValidationError: Plugin '/Use...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
========================= 1 warning, 1 error in 17.43s =========================
test_plotting: exit 2 (22.26 seconds) /Users/runner/work/nilearn/nilearn> pytest --cov=nilearn --cov-report=xml nilearn pid=7328
.pkg: _exit> python /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyproject_api/_backend.py True hatchling.build
  test_plotting: FAIL code 2 (102.15=setup[79.89]+cmd[22.26] seconds)
  evaluation failed :( (102.25 seconds)
```

The tests run fine locally:
- with pytest<8.1.0
- OR removing the conftest in nilearn/nilearn/externals/conftest.py (which suggest to ignore collecting tests, though there are no tests in that subfolder - EDIT: tests are in the __init__.py file).


Discussion:
Sorry to re-open this, but it looks like this was really a problem on the pytest side : https://github.com/pytest-dev/pytest/issues/12073#issuecomment-1976822224

[And version 8.1.0 is now yanked. ](https://pypi.org/project/pytest/8.1.0/)

Generally, I've seen `norecursedirs` _not_ recommended for situations like ours (where we're vendoring another library) and instead the conftest really as best practice. Does that match your reading ? If so, maybe we want to revert this change, since it seems like the existing pytest behavior was correct and will continue to be supported.
Thanks for the follow up.

For the "fix" I had tried to stick to one of the methods mentioned in the pytest doc. 
https://docs.pytest.org/en/8.0.x/example/pythoncollection.html

Do you have more info on using contest as best practice for this use case? 
> Do you have more info on using contest as best practice for this use case?

Generally the argument I've seen is that `norecursedirs` is effectively a glob, so it's best for matching generated build directories (like `_build`) that can occur in multiple locations throughout the package. `--ignore` is better for specific paths, like vendored libraries. But for the reasons stated in `conftest.py`, we really couldn't use `--ignore` on our calls. 

`tempita` is a unique enough name that we might be able to get away with it, but I'd prefer to stick with the method in `conftest.py` unless they're not going to update the associated plugin (but I imagine that's coming soon, now that things have been clarified on the pytest side) !
OK as long as we got something that works, I don't have any strong preference. 

If you think the other one would be more appropriate let's revert.

Resolving patch:
diff --git a/nilearn/externals/conftest.py b/nilearn/externals/conftest.py
new file mode 100644
--- /dev/null
+++ b/nilearn/externals/conftest.py
@@ -0,0 +1,8 @@
+# Do not collect any tests in externals. This is more robust than using
+# --ignore because --ignore needs a path and it is not convenient to pass in
+# the externals path (very long install-dependent path in site-packages) when
+# using --pyargs
+
+
+def pytest_ignore_collect(path, config):
+    return True


Test patch: