Using `fetch_20newsgroups` with multiple pytest workers leads to race

### Describe the bug

When using `pytest-xdist` with several workers to run a test suite that uses `fetch_20newsgroups` as a fixture (`scope="session"`) the dataset shape is sometimes wrong. For example I just had a run where `X.shape=(5902, 68435) y.shape=(5902,)` - it should be something like ~11000 samples for the "train" subset.

Some test functions will see the "shorter" dataset and some the full dataset.

I think the problem is caused by using `pytest-xdist` where each worker will download the dataset itself. The download itself will work in parallel (though wasteful) but then when the file gets `shutil.move`d to the final location things get broken? Or one of the workers sees a partial file somehow.

This is the relevant code

https://github.com/scikit-learn/scikit-learn/blob/be9dd4d4c1f03b8d27311f2d43fcb3c88bdea55c/sklearn/datasets/_base.py#L1499

I'm wondering if the fix is to not use `NamedTempFile` to create the filename to download to, but instead use a name like `fname + '.part'` and check if that file exists before starting a download. That way only one process would start downloading the file.

The problem is that we would need a `check_or_create(path)` function that will perform the check and creation of a path in an atomic operation. Not sure that exists :-/

### Steps/Code to Reproduce

If you put this snippet into a file and run it with `python your_file.py <n_procs>` it reproduces a different version (I think) of the problem.

```python
import multiprocessing as mp
import random
import sys
import time

from sklearn.datasets import fetch_20newsgroups


def fetch_data(i):
    time.sleep(random.random())
    data = fetch_20newsgroups(subset="train", shuffle=True, random_state=42)
    return (i, len(data.data))

if __name__ == "__main__":
    n_processes = int(sys.argv[1])

    with mp.Pool(processes=n_processes) as pool:
        results = pool.map(fetch_data, range(n_processes))
    print(results)
```

Make sure to delete the 20newsgroups file(s) from `~/scikit_learn_data` before every run (I use `rm -rf ~/scikit_learn_data/20news*`)



I originally found this by running a subset of the cuml testsuite with `pytest -rP --tb=auto -n8  test_naive_bayes.py`. You will sometimes observe failures caused by the fact that there are less classes in the dataset (changes from run to run) than priors (which are hardcoded to twenty) passed to the estimator.

The explanation is that the random subset used by the test does not contain all the classes. I think because samples are stored "in order" in the file. So a test that uses a partially downloaded file only sees a subset. if you look at the size of `~/scikit_learn_data/20news-bydate_py3.pkz` it can be less than 13MB (which seems to be the correct size).

This came up in the context of https://github.com/rapidsai/cuml/issues/7152

### Expected Results

The dataset always has the same size

### Actual Results

Shape of the dataset changes between test functions in one run.

### Versions

```shell
System:
    python: 3.13.5 | packaged by conda-forge | (main, Jun 16 2025, 08:27:50) [GCC 13.3.0]
executable: /home/thead/miniforge3/envs/cuml-20250828/bin/python3.13
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.1
          pip: 25.2
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.1
       Cython: 3.1.3
       pandas: 2.3.2
   matplotlib: 3.10.5
       joblib: 1.5.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libopenblas
       filepath: /home/thead/miniforge3/envs/cuml-20250828/lib/libopenblasp-r0.3.30.so
        version: 0.3.30
threading_layer: pthreads
   architecture: Cooperlake

       user_api: openmp
   internal_api: openmp
    num_threads: 64
         prefix: libgomp
       filepath: /home/thead/miniforge3/envs/cuml-20250828/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
This seems related to the work-around we have in scikit-learn to download the data in only one pytest-xdist worker before running the tests. This is done `pytest_collection_modify_items` in `sklearn/conftest.py` to make sure only the first worker download the data if needed. See part of the code:
https://github.com/scikit-learn/scikit-learn/blob/e653ac6da08a78a3a9b57b7e4fd77af3e1327c5a/sklearn/conftest.py#L140-L185

I guess other `sklearn.datasets.fetch_*` may have a similar issue.

There may be more context in the PR that implemented the change https://github.com/scikit-learn/scikit-learn/issues/19118

Labels: Bug