Encountered an error when loading data in WebDataset format using `load_datasets` during multi-matchines training.

### Describe the bug

Single-machine training works fine, but multi-machine training throws up all sorts of weird bugs. Help me!

### Steps to reproduce the bug

train_dataset = load_dataset(
        "webdataset", data_files=args.train_dataset, split="train", streaming=True,cache_dir='/dev/shm/.cache'
    )

args.train_dataset is tar list

### Expected behavior


[rank9]: train_dataset = load_dataset(

[rank9]: File "/mnt/aihao/miniconda3/envs/pixeldit/lib/python3.10/site-packages/datasets/load.py", line 1705, in load_dataset

[rank9]: return builder_instance.as_streaming_dataset(split=split)

[rank9]: File "/mnt/aihao/miniconda3/envs/pixeldit/lib/python3.10/site-packages/datasets/builder.py", line 1110, in as_streaming_dataset

[rank9]: splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}

[rank9]: File "/mnt/aihao/miniconda3/envs/pixeldit/lib/python3.10/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 79, in _split_generators

[rank9]: pipeline = self._get_pipeline_from_tar(tar_paths[0], tar_iterators[0])

[rank9]: IndexError: list index out of range

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 4.8.4
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.20
- `huggingface_hub` version: 0.36.2
- PyArrow version: 23.0.1
- Pandas version: 2.3.3
- `fsspec` version: 2026.2.0


Hi @aihao2000 ! I was looking into this multi-machine training scenario and just opened a PR (#8203 ) that safely skips feature inference for empty shards during distributed streaming.

please! Check it out and see if it looks like it will resolve the weird bugs you were hitting!

it looks like there less dataset files (=shards) than ranks. We should improve the error message in that case and suggest to either disable streaming or to run fewer nodes. @aihao2000 are you using `split_dataset_by_node()` or `.shard()` to distributed the dataset among the training nodes ?

Thanks @Wolfram-St for taking a look at it btw. Though note that if a rank has 0 shards then it will yield 0 training samples. In distributed training the ranks with 0 training sample should shut down and the other continue, and this only works if the distributed training script uses the `Join()` torch context manager (otherwise the training will hang, because the non-empty ranks will wait forever an all_reduce from the empty ranks). So an alternative to the error message could be to show a warning and let the empty ranks finish gracefully. But we have to be careful since I believe most training codes are not using `Join()` yet so in the meantime the error message could be a better solution rather than letting the code hang


@lhoestq I use ```shard``` after ```load_dataset```. Does the number of `.tar` files *have* to be greater than `num_processes`? The number of my target files should also be significantly greater than the number of training processes; furthermore, to the best of my understanding, the `load_dataset` function should not be designed to utilize multiple processes. It seems this error is being triggered before the sharding process has even begun. I'm currently using webdataset, and I'll try it again when I have the chance. It seems that huggingface_dataset throws errors when it encounters faulty tar packages. I think it would be better to warn and skip it like webdataset does; large-scale training ending with errors is really painful.

Can you check the value of `ds.num_shards` ? Or share the code you are using ?

> Does the number of .tar files have to be greater than num_processes?

Yes, this way you can use .shard() and each node has at least one shard to stream from. Otherwise you can use split_dataset_by_node() but if there aren't enough shards, each node will stream the whole data and skip samples so that each nodes gets different samples.

>  It seems that huggingface_dataset throws errors when it encounters faulty tar packages. I think it would be better to warn and skip it like webdataset does; large-scale training ending with errors is really painful.

100% agree. IMO ideally any error should only happen on start and give the right indication to fix it.

Btw to help with such cases feel free to checkpoint your training. `datasets` offers checkpointing for streaming datasets via the usual `state_dict()` /  `load_state_dict()` interface.

> to the best of my understanding, the load_dataset function should not be designed to utilize multiple processes.

your understanding is right, `datasets` doesn't use multiprocessing by default