Issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway

              There are two ways an iterable dataset can be split by node:
1. if the number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU
2. otherwise, each GPU iterate on the data and at the end keeps 1 sample out of n(GPUs) - skipping the others.

In case 2. it's therefore possible to have the same examples passed to `prepare_dataset` for each GPU.

This doesn't sound optimized though, because it runs the preprocessing on samples that won't be used in the end.

Could you open a new issue so that we can discuss about this and find a solution ?

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/5360#issuecomment-1592729051_
            

Comments:
Does "number of shards" refer to the total number of data?

my config:
nproc_per_node=2
ds=ds['train'] = load_dataset(streaming=True).take(50000)

I'm test again: in prepare_data(),  data have the same for each GPU


The number of shards is `ds.n_shards`. It corresponds generally to the number of files the dataset is made of, to be able to distribute to several nodes.

**You don't end up with the same data per GPU**. But all the samples are going through your preprocessing function you pass to map. They are just skipped afterwards to only keep 1 sample out of n(GPUs)

For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end. 
Is my understanding correct?

Where can I print the actual training data for each GPU?

> For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
Is my understanding correct?

Yes exactly :)

> Where can I print the actual training data for each GPU?

You should call print in the data_collator

I print out n_shards, and under multiple GPUs, this value is always 1.
Is this value correct?

Yes it's correct, and it explains why you always have the same data passed to your map function (the data can't be split).

But after being passed to `map`, each GPU keeps one example out of n(GPUs) so that you don't end up with duplicate data across GPUs

> > For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
> > Is my understanding correct?
> 
> Yes exactly :)
> 
> > Where can I print the actual training data for each GPU?
> 
> You should call print in the data_collator

OK, when printing the train data in the data collator, each GPU sees different data.

Thanks for your reply

Do we have a solution for this one? Or it's required to get "number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU"

For now it's required to have a number of shards that is a factor of the number of GPUs to not have all the workers process the same data (and then skip the right ones to not end up training on duplicate data).

It would be quite complex to implement a strategy that would utilize all the GPUs with an arbitrary number of shards even at the end of training