Trainer reports batch size different from argument on multiple GPUs with DP
Author: cgbahk
Pull request: no

### System Info

- `transformers` version: 4.30.1
- Platform: Linux-4.18.0-240.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Not sure why "PyTorch version (GPU?)" is False. I think this is because GPU not connected when I run this on report time. On actual training, GPU connected.
I'm pretty sure my pytorch environment is with GPU support, like I can use same conda environment to train normal, single GPU training exploiting GPU resource.
```
$ conda list | grep pytorch

pytorch                   1.12.1          py3.10_cuda11.3_cudnn8.3.2_0    pytorch
```


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I've observed `Instantaneous batch size per device` in trainer log reported as `per_device_train_batch_size` x GPU count, reproducible in multiple cases.

I can't give full reproduction detail, but pretty sure that scenario below can give idea of the situation.

For example, I tried to train with 2 GPUs in DP sense(DP as described in [:link:](https://github.com/huggingface/transformers/blob/v4.30.1/docs/source/en/perf_train_gpu_many.mdx#data-parallelism)), with following TrainingArgument:
```py
TrainingArgument(
    auto_find_batch_size=False,
    per_device_train_batch_size=1,
    ...
)
```

Then training log looks like this. Note on `Instantaneous batch size per device` value. I expected 1 from `per_device_train_batch_size`
```
***** Running training *****
  Num examples = ...
  Num Epochs = ...
  Instantaneous batch size per device = 2
  Total train batch size (w. parallel, distributed & accumulation) = ...
  Gradient Accumulation steps = ...
  Total optimization steps = ...
  Number of trainable parameters = ...
  ...
```

(I've experienced some other logging bug, like `Total train batch size` especially when with `auto_find_batch_size=True` but let's only focus on batch size mismatch in this issue)

I could check `Instantaneous batch size per device` reported as `per_device_train_batch_size` x GPU count happens again in other cases, like
- 4 GPUs / `per_device_train_batch_size=128` -> `Instantaneous batch size per device = 512`

This maybe
- correct actual behavior but logging is not correct, or
- actual bug, or
- I may have misunderstanding about DP, in this case please blame me :smile:


### Expected behavior

I expected
- `Instantaneous batch size per device` reported as `per_device_train_batch_size`

not
- `Instantaneous batch size per device` reported as `per_device_train_batch_size` x GPU count

Comments:
How are you launching your training script? If it's just with python (no distributed), the `Trainer` will use `DataParallel` which requires your batch size to be mulitiplied by the number of GPUs to work properly. I'm guessing that's why you see the "instanteneous batch size" at 4x what you put.

This is the only case it will happen (if you launch in distributed mode, the batch size per device will show up correctly) and is a good mean to track whether you are using Distributed training properly (you shouldn't use DataParallel as per PyTorch documentation) so you should launch your script with `torchrun` or `accelerate launch`.

I just begin to try training with multiple GPUs :smile:  And everybody gives warning on using DP, and recommends to use DDP over DP. Okay I'll try.

But that is out of this issue topic. So let's not talk about it anymore here.

---

> How are you launching your training script? If it's just with python (no distributed), the `Trainer` will use `DataParallel`

Yes this is the case I meant. This issue is about DP not DDP.

I think in this communication, it is extremely important to use same terms for same concepts, especially about several 'batch' concepts.

Let me use term
- `batch size per update`: count of input that used for one model parameter update
- `device`: in this case, let's fix this to GPU
  - And I think that is what term 'device' mean in training arg `per_device_train_batch_size` and log `Instanteneous batch size per device`.
    https://github.com/huggingface/transformers/blob/66fd3a8d626a32989f4569260db32785c6cbf42a/src/transformers/training_args.py#L193-L194
- `batch size per device`: count of input source for each device(i.e. GPU) for one model parameter update iteration
  - Depending on documentations and communications, ambiguous terms used, like "mini-batch"([:link:](https://github.com/huggingface/transformers/blob/v4.30.2/docs/source/en/perf_train_gpu_many.mdx#L89) [:link:](https://github.com/huggingface/transformers/blob/v4.30.2/docs/source/en/perf_train_gpu_many.mdx#L95)) or "sub mini-batch" [:link:](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/). So let's fix to this term for this issue communication.
  - I expect this is the same concept with training arg `per_device_train_batch_size` and log `Instanteneous batch size per device`


> `DataParallel` which requires your batch size to be mulitiplied by the number of GPUs to work properly. I'm guessing that's why you see the "instanteneous batch size" at 4x what you put.

In your comment, 'batch size' seems to mean `batch size per update`. And yes, that is true, it should be 'GPU count' x `batch size per device`.

But the log `Instantaneous batch size per device` means `batch size per device`, not `batch size per update`. That is what I'm pointing out as a bug, which can lead user to misunderstanding.


(I will only use DDP, so this issue is not anymore important for me. But if I'm someone who cares about the project, like maintainer, I would leave this open before the bug fixed. Any maintainers can close this issue if they  want so :smile:)

I made the PR linked above to clarify the logging a bit more. Let me know if it's better!