BUG: pytest error when loading conftest (seemingly platform-specific)

### Describe the bug

I'm seeing errors on my Windows machine when running `pytest` (does not work with only `pytest`, and does not work for directories that has `conftest.py`). This seems to be a platform-specific problem, since CI is not complaining. I investigated a bit and found https://github.com/pytest-dev/pytest/issues/9765, but it doesn't look like `pytest` is planning to fix it, at least for now. I tried downgrading to `pytest==7.0.1` and everything worked smoothly, but in https://github.com/scikit-learn/scikit-learn/pull/26373 the minimum version of `pytest` has already been `7.1.2` for scikit-learn (due to some CI errors for `pytest==5.x.x`), so scikit-learn is raising error:

https://github.com/scikit-learn/scikit-learn/blob/5c4288dba42cb67d954cb56c2cebfbf25c05ef89/sklearn/conftest.py#L30-L34

I'm wondering if it is possible to pin `pytest==7.0.1` or at least relax the minimum requirement a bit to `PYTEST_MIN_VERSION = "7.0.1"`? Or are there any other suggestions how I may resolve this issue? @glemaitre who bumped the minimum version of `pytest` to 7.1.2. Truly sorry for the inconvenience caused by my annoying Windows machine.

### Steps/Code to Reproduce

```bash
pytest
```

or

```bash
pytest sklearn/utils/tests
```

Running `pytest` on a single file works correctly.

### Expected Results

No error is thrown.

### Actual Results

For the first example,

```pytb
❯ pytest
Traceback (most recent call last):
  File "D:\Downloads\mambaforge\envs\sklearn-env\Scripts\pytest-script.py", line 9, in <module>
    sys.exit(console_main())
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 192, in console_main
    code = main()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 150, in main
    config = _prepareconfig(args, plugins)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 331, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 130, in _multicall
    teardown[0].send(outcome)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\helpconfig.py", line 104, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1075, in pytest_cmdline_parse
    self.parse(args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1425, in parse
    self._preparse(args, addopts=addopts)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1327, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\pluggy\_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 1153, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 563, in _set_initial_conftests
    self._try_load_conftest(anchor, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 585, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 609, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py", line 654, in _importconftest
    assert mod not in mods
AssertionError
```

For the second example,

```pytb
❯ pytest sklearn/utils/tests
================================================================== test session starts ===================================================================
platform win32 -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: D:\ossd\scikit-learn-yxiao
configfile: setup.cfg
plugins: cov-4.1.0
collected 1957 items / 1 error

========================================================================= ERRORS =========================================================================
____________________________________________________ ERROR collecting sklearn/utils/tests/conftest.py ____________________________________________________
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\runner.py:341: in from_call
    result: Optional[TResult] = func()
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\doctest.py:560: in collect
    module = self.config.pluginmanager._importconftest(
D:\Downloads\mambaforge\envs\sklearn-env\lib\site-packages\_pytest\config\__init__.py:654: in _importconftest
    assert mod not in mods
E   AssertionError
================================================================ short test summary info =================================================================
ERROR sklearn/utils/tests/conftest.py - AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================== 1 warning, 1 error in 1.43s ===============================================================
```

### Versions

```shell
❯ python -c "import sklearn; sklearn.show_versions()"

System:
    python: 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:40:31) [MSC v.1929 64 bit (AMD64)]
executable: D:\Downloads\mambaforge\envs\sklearn-env\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.4.dev0
          pip: 23.2.1
   setuptools: 68.2.2
        numpy: 1.26.0
        scipy: 1.11.2
       Cython: 3.0.2
       pandas: 2.1.1
   matplotlib: 3.8.0
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
    num_threads: 6
         prefix: libblas
       filepath: D:\Downloads\mambaforge\envs\sklearn-env\Library\bin\libblas.dll
        version: 2022.1-Product
threading_layer: intel

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: vcomp
       filepath: D:\Downloads\mambaforge\envs\sklearn-env\vcomp140.dll
        version: None
```

COMMENT:
FYI we managed to reproduce the issue with @fcharras on his Windows VM. Hopefully we will manage to make a reproducer for https://github.com/pytest-dev/pytest/issues/9765.

Labels: Bug