how huggingface process uneven input tensors
Author: Slyne
Pull request: no

### Feature request

For IterableDataset, it seems there's no way to handle the uneven input tensors in trainer.py. (Please correct me if I misunderstant this).  
Pytorch document suggests to use join: https://pytorch.org/tutorials/advanced/generic_join.html#what-is-join

Secondly, the dataloader for IterableDataset doesn't have sampler, which may be an issue when num workers > 0.
https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/trainer.py#L672

### Motivation

Currently in trainer.py, the IterableDataset is wrapped into IterableDatasetShard in distributed training. It seems that it requires every process to have the same whole dataset and distribute the samples in [IterableDatasetShard](https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/trainer_pt_utils.py#L678), which makes every batch has the same amount of data by [pad the first batch data](https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/trainer_pt_utils.py#L770).

When in training, since the data is guranteed to be equal in every process, there is no need to use [Join](https://pytorch.org/tutorials/advanced/generic_join.html#what-is-join) to process the uneven input datasets.

Here's my case. I have a very large audo dataset of 1 million audio files and each file requires the processing step just like [this example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py). If I do the same step as this example, the large training datasets not only requires a lot of time processing but also a lot of disk storage to store the features.
Therefore I decide to use IterableDataset to do on-the-fly preprocessing. However, I need to maintain a buffer to do some shuffling then the CPU memory increases greatly and every process is doing many duplicate work that the other process has already done. For example, let's say we have 2 processes to process 11 samples. Every process has to process the 11 samples instead of 5 + 6.

So I decide to give up the  IterableDatasetShard and use  IterableDataset directly. Every process has its own part of data to process. For example, process0 has 5 samples and process 1 has 6 samples. However this also means some process will finish training earlier. But I see there's no join used in trainer.py. So I was wondering what's the best practice to do this training and inference in this use case.

Thanks!

### Your contribution

Not sure

Comments:
Please use the [forum](https://discuss.huggingface.co/) for questions like this as we keep the issues for bugs and feature requests only. In this instance, I don't think the `Trainer` will help you as you want a very specific data processing, so you should use `Accelerate`. The dispatch feature it offers is done exactly for users with `IterableDataset` that don't want to process the data in each process.

> Please use the [forum](https://discuss.huggingface.co/) for questions like this as we keep the issues for bugs and feature requests only. In this instance, I don't think the `Trainer` will help you as you want a very specific data processing, so you should use `Accelerate`. The dispatch feature it offers is done exactly for users with `IterableDataset` that don't want to process the data in each process.

Thank you. I solved this issue just by customizing the get_train_loader, doing padding in my own pipelining and removing data collator and adding a model.join() context.
BTW, does the dispatch feature in Accelerate can solve every process has different number of batches in one epoch when dataloader.num_workers > 0 without duplicate data processing?


Another thing I find is that the [find_batch_size](https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/trainer_pt_utils.py#L105) cannot support [BatchFeature](https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/feature_extraction_utils.py#L62).
Not sure if this is an issue.



The last one is a bug, will fix that!