Question about how to write code for trainer and dataset for multi-gpu 
Author: zch-cc
Pull request: no

### System Info

- Platform: Linux-5.15.0-1026-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi I have a quick question on how to write code for dataset and trainer for multi-gpu setting.

Here is my workflow.


I have a dataset where I called 
``` 
dataset = dataset.load_dataset(...) 
```
I need to do some preprocessing for it and the dataset becomes an Iterable dataset. 
and then I pass the dataset into the trainer like 
```
trainer = Trainer(train_data=dataset)
trainer.train()
```

My question is since I am running on multi-gpu and use command 
```
torchrun --standalone --nnodes=1 --nproc_per_node=2 train_lora.py
```
Two process is executing the same code above and this cause dataset and trainer created twice. Should the dataset and trainer be created once or twice? If created once, should I wrapper all the code like that?

```
if accelerator.is_main_process:
   dataset = dataset.load_dataset(...) 
   trainer = Trainer(train_data=dataset)
   trainer.train()
````
I do observe that we only use 1 dataset for generating the samples even if we create two dataset object and do not wrap accelerator.is_main_process. That is because the dataset already convert by trainer for distributed training.  So I think there is no point for creating dataset twice since we only use the first dataset. How to write the code such that there is no error on the second process? if I make second process dataset is None, the trainer will give error for dataset is empty
Do we need to create two trainer where each trainer is corresponding to one gpu or should we only have one trainer that is in charge for two gpu?   What is the best way to write the code to achieve this in this case? 

 


### Expected behavior

the correct way of implement this situation. 

Comments:
cc @muellerzr @pacman100 

You can find our multiple examples for doing so in the no_trainer scripts for the datasets: https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue_no_trainer.py

However at this point, you just need to launch the code you don't need to do anything special since it's all done on the backend now automatically. 

Using DDP, the dataset needs to be available on every process, so each worker needs to have a copy of the dataset as you observed. What gets optimized is tokenizing the dataset and that is done on a single process first and then sent to the others. See the map part here, taken from the corresponding glue trainer example:

```python
        with training_args.main_process_first(desc="train dataset map pre-processing"):
            train_dataset = train_dataset.map(
                tokenize_and_align_labels,
                batched=True,
                num_proc=data_args.preprocessing_num_workers,
                load_from_cache_file=not data_args.overwrite_cache,
                desc="Running tokenizer on train dataset",
            )
```

Those scripts are a great place to start, as they work on any distributed situation :)