Using `fetch_20newsgroups` with multiple pytest workers leads to race

### Describe the bug

When using `pytest-xdist` with several workers to run a test suite that uses `fetch_20newsgroups` as a fixture (`scope="session"`) the dataset shape is sometimes wrong. For example I just had a run where `X.shape=(5902, 68435) y.shape=(5902,)` - it should be something like ~11000 samples for the "train" subset.

Some test functions will see the "shorter" dataset and some the full dataset.

I think the problem is caused by using `pytest-xdist` where each worker will download the dataset itself. The download itself will work in parallel (though wasteful) but then when the file gets `shutil.move`d to the final location things get broken? Or one of the workers sees a partial file somehow.

This is the relevant code

https://github.com/scikit-learn/scikit-learn/blob/be9dd4d4c1f03b8d27311f2d43fcb3c88bdea55c/sklearn/datasets/_base.py#L1499

I'm wondering if the fix is to not use `NamedTempFile` to create the filename to download to, but instead use a name like `fname + '.part'` and check if that file exists before starting a download. That way only one process would start downloading the file.

The problem is that we would need a `check_or_create(path)` function that will perform the check and creation of a path in an atomic operation. Not sure that exists :-/

### Steps/Code to Reproduce

If you put this snippet into a file and run it with `python your_file.py <n_procs>` it reproduces a different version (I think) of the problem.

```python
import multiprocessing as mp
import random
import sys
import time

from sklearn.datasets import fetch_20newsgroups


def fetch_data(i):
    time.sleep(random.random())
    data = fetch_20newsgroups(subset="train", shuffle=True, random_state=42)
    return (i, len(data.data))

if __name__ == "__main__":
    n_processes = int(sys.argv[1])

    with mp.Pool(processes=n_processes) as pool:
        results = pool.map(fetch_data, range(n_processes))
    print(results)
```

Make sure to delete the 20newsgroups file(s) from `~/scikit_learn_data` before every run (I use `rm -rf ~/scikit_learn_data/20news*`)



I originally found this by running a subset of the cuml testsuite with `pytest -rP --tb=auto -n8  test_naive_bayes.py`. You will sometimes observe failures caused by the fact that there are less classes in the dataset (changes from run to run) than priors (which are hardcoded to twenty) passed to the estimator.

The explanation is that the random subset used by the test does not contain all the classes. I think because samples are stored "in order" in the file. So a test that uses a partially downloaded file only sees a subset. if you look at the size of `~/scikit_learn_data/20news-bydate_py3.pkz` it can be less than 13MB (which seems to be the correct size).

This came up in the context of https://github.com/rapidsai/cuml/issues/7152

### Expected Results

The dataset always has the same size

### Actual Results

Shape of the dataset changes between test functions in one run.

### Versions

```shell
System:
    python: 3.13.5 | packaged by conda-forge | (main, Jun 16 2025, 08:27:50) [GCC 13.3.0]
executable: /home/thead/miniforge3/envs/cuml-20250828/bin/python3.13
   machine: Linux-6.14.0-27-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.7.1
          pip: 25.2
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.16.1
       Cython: 3.1.3
       pandas: 2.3.2
   matplotlib: 3.10.5
       joblib: 1.5.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libopenblas
       filepath: /home/thead/miniforge3/envs/cuml-20250828/lib/libopenblasp-r0.3.30.so
        version: 0.3.30
threading_layer: pthreads
   architecture: Cooperlake

       user_api: openmp
   internal_api: openmp
    num_threads: 64
         prefix: libgomp
       filepath: /home/thead/miniforge3/envs/cuml-20250828/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
Ended up using a similar work around for the cuml test suite. If we could solve this problem at the scikit-learn level it would be good, because discovering this problem took a bit of work as tests didn't always fail, only sometimes. In some way it is a "fails silently" problem. This means if we solve it in scikit-learn not everyone needs to rediscover this.

Another reason why it might be useful to solve this at the scikit-learn level is that the problem also happens if you use multiple threads:
```python
from concurrent.futures import ThreadPoolExecutor
import random
import sys
import time

from sklearn.datasets import fetch_20newsgroups


def fetch_data(i):
    time.sleep(random.random())
    data = fetch_20newsgroups(subset="train", shuffle=True, random_state=42)
    return (i, len(data.data))

if __name__ == "__main__":
    n_processes = int(sys.argv[1])

    with ThreadPoolExecutor(max_workers=n_processes) as executor:
        results = list(executor.map(fetch_data, range(n_processes)))

    print(results)
```

which I think means that in a free threaded Python people might run into this more often? I'm not sure though.

I agree that the other `fetch_*` functions are similarly impacted.

My problem is that I have no good idea on how to make the `fetch_*` functions more robust. One aspect is downloading the file which is handled in `sklearn.datasets._base._fetch_remote` and another is the processing that happens afterwards. Somehow you want all of this to happen only once, not several times. So I think we need the synchronisation to happen at the level of `fetch_20newsgroups`. But, I don't know how to do that nicely (without a lot of added complexity).

Labels: Bug