BUG: pytest error when loading conftest (seemingly platform-specific)

### Describe the bug

I'm seeing errors on my Windows machine when running `pytest` (does not work with only `pytest`, and does not work for directories that has `conftest.py`). This seems to be a platform-specific problem, since CI is not complaining. I investigated a bit and found https://github.com/pytest-dev/pytest/issues/9765, but it doesn't look like `pytest` is planning to fix it, at least for now. I tried downgrading to `pytest==7.0.1` and everything worked smoothly, but in https://github.com/scikit-learn/scikit-learn/pull/26373 the minimum version of `pytest` has already been `7.1.2` for scikit-learn (due to some CI errors for `pytest==5.x.x`), so scikit-learn is raising error:

https://github.com/scikit-learn/scikit-learn/blob/5c4288dba42cb67d954cb56c2cebfbf25c05ef89/sklearn/conftest.py#L30-L34

I'm wondering if it is possible to pin `pytest==7.0.1` or at least relax the minimum requirement a bit to `PYTEST_MIN_VERSION = "7.0.1"`? Or are there any other suggestions how I may resolve this issue? @glemaitre who bumped the minimum version of `pytest` to 7.1.2. Truly sorry for the inconvenience caused by my annoying Windows machine.

### Steps/Code to Reproduce

```bash
pytest
```

or

```bash
pytest sklearn/utils/tests
```

Running `pytest` on a single file works correctly.

### Expected Results

No error is thrown.

### Actual Results

For the first example,

```pytb
❯ pytest
Traceback (most recent call last):
  File "D:\Downloads\mambaforge\envs\sklearn-env\Scripts\pytest-script.py", line 9, in <module>
    sys.exit(console_main())
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 192, in console_main
    code = main()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 150, in main
    config = _prepareconfig(args, plugins)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 331, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\helpconfig.py", line 104, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1075, in pytest_cmdline_parse
    self.parse(args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1425, in parse
    self._preparse(args, addopts=addopts)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1327, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1153, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 563, in _set_initial_conftests
    self._try_load_conftest(anchor, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 585, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 609, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 654, in _importconftest
    assert mod not in mods
AssertionError
```

For the second example,

```pytb
❯ pytest sklearn/utils/tests
================================================================== test session starts ===================================================================
platform win32 -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: D:\ossd\scikit-learn-yxiao
configfile: setup.cfg
plugins: cov-4.1.0
collected 1957 items / 1 error

========================================================================= ERRORS =========================================================================
____________________________________________________ ERROR collecting sklearn/utils/tests/conftest.py ____________________________________________________
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\runner.py:341: in from_call
    result: Optional[TResult] = func()
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\doctest.py:560: in collect
    module = self.config.pluginmanager._importconftest(
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py:654: in _importconftest
    assert mod not in mods
E   AssertionError
================================================================ short test summary info =================================================================
ERROR sklearn/utils/tests/conftest.py - AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================== 1 warning, 1 error in 1.43s ===============================================================
```

### Versions

```shell
❯ python -c "import sklearn; sklearn.show_versions()"

System:
    python: 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:40:31) [MSC v.1929 64 bit (AMD64)]
executable: D:\Downloads\mambaforge\envs\sklearn-env\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.4.dev0
          pip: 23.2.1
   setuptools: 68.2.2
        numpy: 1.26.0
        scipy: 1.11.2
       Cython: 3.0.2
       pandas: 2.1.1
   matplotlib: 3.8.0
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
    num_threads: 6
         prefix: libblas
       filepath: D:\Downloads\mambaforge\envs\sklearn-env\Library\bin\libblas.dll
        version: 2022.1-Product
threading_layer: intel

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: vcomp
       filepath: D:\Downloads\mambaforge\envs\sklearn-env\vcomp140.dll
        version: None
```

COMMENT:
> I'm wondering if it is possible to pin pytest==7.0.1 or at least relax the minimum requirement a bit to PYTEST_MIN_VERSION = "7.0.1"

We could try to downgrade but we should make sure that we don't have the INTERNALERROR raised by pytest.

What is weird is that we have windows CI so we should catch such error or maybe this is a Windows 10 error.

Labels: Bug