Use args.per_gpu_train_batch_size instead of args.train_batch_size in…
Author: jarednielsen
Pull request: yes

… Trainer.

It appears that this is preferred, per https://github.com/huggingface/transformers/blob/master/src/transformers/training_args.py. This also matches the calculation which is printed referring to batch size at https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L432.

As a side note, the GPT-2 example in https://github.com/huggingface/transformers/blob/master/examples/language-modeling/README.md no longer works. There is a default `per_gpu_train_batch_size=8`, which throws OOM on a Tesla V100 with 32GB RAM. I ran it successfully with `--per_gpu_train_batch_size=1`, and it used 7GB of RAM. So we probably want to add that hyperparameter to the example command.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4493?src=pr&el=h1) Report
> Merging [#4493](https://codecov.io/gh/huggingface/transformers/pull/4493?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/865d4d595eefc8cc9cee58fec9179bd182be0e2e&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `0.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4493/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4493?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4493   +/-   ##
=======================================
  Coverage   77.90%   77.91%           
=======================================
  Files         123      123           
  Lines       20472    20472           
=======================================
+ Hits        15949    15950    +1     
+ Misses       4523     4522    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4493?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4493/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.76% <0.00%> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4493/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.53% <0.00%> (+0.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4493?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4493?src=pr&el=footer). Last update [865d4d5...7ffd712](https://codecov.io/gh/huggingface/transformers/pull/4493?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Things could be clearer, but the only case where train_batch_size is different from per_gpu_train_batch_size is in `nn.DataParallel`. 

And in DataParallel, your dataloader's apparent batch size will be scattered amongst the devices, so I believe the `batch_size=self.args.train_batch_size` is correct



(Note that DataParallel is not really recommended anymore as a way to utilize multiple GPUs, vs. torch.distributed)

Gotcha, thanks for the context. Is the user expected to pass both `--train_batch_size` and `--per_gpu_train_batch_size` together then? In `examples/run_language_modeling.py` as it stands, the `--train_batch_size` affects the true batch size, but `--per_gpu_train_batch_size` is what is printed to stdout here: https://github.com/huggingface/transformers/blob/a08652772791fdaeed6f263b1a99926ca64be5dc/src/transformers/trainer.py#L419

No, user would not pass a `--train_batch_size` – was it documented somewhere that they should?

My misunderstanding then. There are [a few instances](https://grep.app/search?q=--train_batch_size&filter[repo][0]=huggingface/transformers) through the codebase where that arg is expected, but I see that in this example it's a derived property. Thanks for the help, closing the issue.