Unclear what happens when using torchrun, multi-gpu and trainer arguments.
Author: davies-w
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.0
- Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.35
- Python version: 3.11.6
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.15.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:  CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc-per-node 4 stages/train_model.py
- Using GPU in script?: Yes
- GPU type: NVIDIA RTX A4000

### Who can help?

@muellerzr 
@SunMarc 

I have a piece of python code, that loads a training dataset from a file,  then sets up a Trainer with the training dataset. I am training a distilBERT model (I dont think that detail matters).

It works great. And then I can do torchrun --nproc_per_node = 4, and it seems to run great too, it seems to spawn 4 versions and trains them on the 4 GPUs.

However, I got suspicious - given that I’m passing the same dataset in everyt ime, is the Trainer basically training on just the first 25% of the training set? 

I’d assumed initially it was smarter than this, and some how only used one of the train dataset instances (presumably the first), but now I realize I cannot be certain of this. So, can anyone tell me if I need to “pre-partition” the datasets into 4 shards, and then load them keyed by LOCAL RANK?

If it's not doing the "right" thing - i.e. training on all the data, then a warning should be given when this is detected.

Script looks like this:
train_test_dataset = Dataset.load_from_disk('train_test_dataset_file')
trainer = Trainer(..., train_dataset=train_test_dataset['train'], validation_dataset=train_test_dataset['test'],...)


Should script looks like this:
train_test_dataset = Dataset.load_from_disk(f"one_quarter_train_test_dataset_file_{LOCAL_RANK)")
trainer = Trainer(..., train_dataset=train_test_dataset['train'], validation_dataset=train_test_dataset['test'],...)



### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

clarity is needed on what behaviour is happening with the training data. Is the training dataset being used by all the nodes with an equal division of labor over each quarter of the dataset (or interleaved every 4 minibatches), or is it sending the same minibatch from the training set, and never getting to the final three quarters of the dataset.

### Expected behavior

A better understanding of what data is being trained on when invoking Trainer through torchrun.

Comments:
Gentle ping?

Trainer will use accelerate to prepare the dataloader. See the following video to help you better understand how it works ! https://www.youtube.com/watch?v=9Vfauv4ErwA

Thanks SunMarc, it still leave me puzzled. 

I have a single script, which as I mentioned just has:

train_test_dataset = Dataset.load_from_disk('train_test_dataset_file')
trainer = Trainer(..., train_dataset=train_test_dataset['train'], validation_dataset=train_test_dataset['test'],...)

When I run it with say 4 GPUs via torchrun, I see it execute this code 4 times. Does one of the executions become the master node? EG maybe always the local_rank 0 one? 

In which case, if I interpret the video correctly, I just need to set global_batch to 4 x the batch size?  And the other three executions of the script ignore the train_test_dataset? 

Or am I doing it completely wrong?  





When you pass in a `per_device_batch_size`, it's as it is stated in the variable. The GBS will be pdbs * world_size. 

One of those, on rank 0, will become the master (usually whichever one started the script). Each GPU needs its own copy of the data, hence why its ran 4 times (but usually for tokenizing its pulled in from a cache). 

A better way to think about training on single GPU vs multi-GPU and what's happening under the hood would be:

Say I have 400 items of training data, and training on 1 GPU or 4 GPUs. 

On 1 GPU, I train with a BS of 80, leading to 5 iterations to get through the data.

With how accelerate does multi-GPU dataloaders, on 4 GPUs you would need a *per device batch size* of 20 to do the equivalent, which will as a result be a degree faster since we're processing **the same amount of data faster**.

Another way to think about it is if we kept our same BS of 80 *per GPU*, our real bs would be 320, and it'd be 1 and some change to get through all our data.

Does that make sense?

PERFECT! Thank you SO much. That's a huge relief I don't need to re-engineer and re-run this 1000 hour model!