Using `fetch_20newsgroups` with multiple pytest workers leads to race

### Describe the bug

When using `pytest-xdist` with several workers to run a test suite that uses `fetch_20newsgroups` as a fixture (`scope="session"`) the dataset shape is sometimes wrong. For example I just had a run where `X.shape=(5902, 68435) y.shape=(5902,)` - it should be something like ~11000 samples for the "train" subset.

Some test functions will see the "shorter" dataset and some the full dataset.

I think the problem is caused by using `pytest-xdist` where each worker will download the dataset itself. The download itself will work in parallel (though wasteful) but then when the file gets `shutil.move`d to the final location things get broken? Or one of the workers sees a partial file somehow.

This is the relevant code

https://github.com/scikit-learn/scikit-learn/blob/be9dd4d4c1f03b8d27311f2d43fcb3c88bdea55c/sklearn/datasets/_base.py#L1499

I'm wondering if the fix is to not use `NamedTempFile` to create the filename to download to, but instead use a name like `fname + '.part'` and check if that file exists before starting a download. That way only one process would start downloading the file.

The problem is that we would need a `check_or_create(path)` function that will perform the check and creation of a path in an atomic operation. Not sure that exists :-/

### Steps/Code to Reproduce

If you put this snippet into a file and run it with `python your_file.py <n_procs>` it reproduces a different version (I think) of the problem.

```python
import multiprocessing as mp
import random
import sys
import time

from sklearn.datasets import fetch_20newsgroups


def fetch_data(i):
    time.sleep(random.random())
    data = fetch_20newsgroups(subset="train", shuffle=True, random_state=42)
    return (i, len(data.data))

if __name__ == "__main__":
    n_processes = int(sys.argv[1])

    with mp.Pool(processes=n_processes) as pool:
        results = pool.map(fetch_data, range(n_processes))
    print(results)
```

Make sure to delete the 20newsgroups file(s) from `~/scikit_learn_data` before every run (I use `rm -rf ~/scikit_learn_data/20news*`)



I originally found this by running a subset of the cuml testsuite with `pytest -rP --tb=auto -n8  test_naive_bayes.py`. You will sometimes observe failures caused by the fact that there are less classes in the dataset (changes from run to run) than priors (which are hardcoded to twenty) passed to the estimator.

The explanation is that the random subset used by the test does not contain all the classes. I think because samples are stored "in order" in the file. So a test that uses a partially downloaded file only sees a subset. if you look at the size of `~/scikit_learn_data/20news-bydate_py3.pkz` it can be less than 13MB (which seems to be the correct size).

This came up in the context of https://github.com/rapidsai/cuml/issues/7152

### Expected Results

The dataset always has the same size

### Actual Results

Shape of the dataset changes between test functions in one run.

### Versions

```shell
System:
    python: 3.13.5 | packaged by conda-forge | (main, Jun 16 2025, 08:27:50) [GCC 13.3.0]
executable: /home/thead/miniforge3/envs/cuml-20250828/bin/python3.13
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.1
          pip: 25.2
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.1
       Cython: 3.1.3
       pandas: 2.3.2
   matplotlib: 3.10.5
       joblib: 1.5.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libopenblas
       filepath: /home/thead/miniforge3/envs/cuml-20250828/lib/libopenblasp-r0.3.30.so
        version: 0.3.30
threading_layer: pthreads
   architecture: Cooperlake

       user_api: openmp
   internal_api: openmp
    num_threads: 64
         prefix: libgomp
       filepath: /home/thead/miniforge3/envs/cuml-20250828/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
> If we could solve this problem at the scikit-learn level it would be good, because discovering this problem took a bit of work as tests didn't always fail, only sometimes. In some way it is a "fails silently" problem. This means if we solve it in scikit-learn not everyone needs to rediscover this.

Yeah I agree than in an ideal world, the `fetch_*` would be fixed in scikit-learn. At the same time, as an end user, downloading the same dataset in different processes/threads is not something you would naturally do on purpose. In practice, using `pytest-xdist` seems to be the main use case where people bumped into this.

Searching a bit in the issue tracker, there has been some similar looking fixes for some fetchers e.g. https://github.com/scikit-learn/scikit-learn/pull/21833 (`fetch_openml`) or https://github.com/scikit-learn/scikit-learn/pull/23113 (`fetch_covtype`). The general approach seems to be "download and save to a temporary file" followed by "atomic rename to the final filepath" from https://github.com/scikit-learn/scikit-learn/issues/21798#issuecomment-979972768.

Maybe some fetchers are more tricky than others, hard to tell without trying.

Labels: Bug