When I use dataset streaming loading, continuing training will always wait.
Author: Sakurakdx
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.4.143.bsk.8-amd64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

  def get_train_dataset(self):
      dataset = datasets.load_dataset(self.config.train_files)["train"]
      data_size = len(dataset)
      iterable_dataset = dataset.to_iterable_dataset(num_shards=64)  # faster
      iterable_dataset = iterable_dataset.shuffle(seed=self.config.seed, buffer_size=10000)
      return iterable_dataset

bash train.sh  --per_device_train_batch_size 100 --per_device_eval_batch_size 3
2 --dataloader_num_workers 24 --resume_from_checkpoint True 

### Expected behavior

I use iterable_dataset in datasets to load data streamingly. When I use the trainer to continue training, it will spend a lot of time processing the data. Because the amount of data is very large, it cannot enter the training process for a long time. What can I do to avoid or speed up this process?

Comments:
Because your dataset is an iterable, you have to go through the samples in order. For now, it not possible to shortcut this process. This issue has been mentioned a couple times and I think the Huggingface team might be working on something.
I personally use [MosaicML streaming](https://github.com/mosaicml/streaming) to address this issue.

> Because your dataset is an iterable, you have to go through the samples in order. For now, it not possible to shortcut this process. This issue has been mentioned a couple times and I think the Huggingface team might be working on something. I personally use [MosaicML streaming](https://github.com/mosaicml/streaming) to address this issue.

fine, thanks for your suggests

@Hubert-Bonisseur Trying to approach the problem the same way and switching to the MosaicML streaming. Do you just use Streaming as an IterableDataset into Trainer? Wondering if there are specific modifications needed to resume_from_checkpoint with streaming. 

If you use the Trainer, you need do to a couple changes: 
- You need to overwrite the get_train_dataloader to [use the StreamingDataLoader](https://github.com/mosaicml/streaming/issues/421) from mosaicML streaming
- You need to save the state_dict of the dataloader at each save (use on_save callback)
- The `skip_first_batches` of accelerate that works with standard Dataloader no longer does. As a result I have opted to load the StreamingDataloader state_dict in `get_train_dataloader` and I have patched skip_first_batches like this:
```python
import accelerate
def skip_first_batches(dataloader, num_batches=0):
    return dataloader
accelerate.skip_first_batches = skip_first_batches
```
It is dirty but it works, I should probably open an issue to propose making the `skip_first_batches` method customizable


Thank you this is super helpful!

@Hubert-Bonisseur qq, when using the StreamingDataLoader with Trainer, do you prepare it with accelerate? The `accelerate.prepare()` wraps it into a `DataLoaderDispatcher` object where I can't call `save_dict()`. It seems to run when I don't call accelerate, but wondering if that will lead to bugs downstream. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.