Training MLM model XLM Roberta large on google machine specs not fast
Author: pratikchhapolika
Pull request: no

I am fine tuning masked language model from `XLM Roberta large` on google machine specs. 
I made couple of experiments and was strange to see few results. 

```
"a2-highgpu-4g" ,accelerator_count=4, accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 4 Running ( 4 data*4 GPU=16 data points)
"a2-highgpu-4g" ,accelerator_count=4 , accelerator_type="NVIDIA_TESLA_A100"on 4,12,672 data batch size 8 failed
 "a2-highgpu-4g" ,accelerator_count=4, accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 16 failed
"a2-highgpu-4g" ,accelerator_count=4.,accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 32 failed
```

I was not able to train model with `batch size ` more than 4 on # of GPU's. It stopped in mid-way.


Here is the code I am using.

```
training_args = tr.TrainingArguments(
#     disable_tqdm=True,
    output_dir='/home/pc/Bert_multilingual_exp_TCM/results_mlm_exp2', 
    overwrite_output_dir=True,
    num_train_epochs=2,
    per_device_train_batch_size=4,
#     per_device_train_batch_size
#     per_gpu_train_batch_size
    prediction_loss_only=True
    ,save_strategy="no"
    ,run_name="MLM_Exp1"
    ,learning_rate=2e-5
    ,logging_dir='/home/pc/Bert_multilingual_exp_TCM/logs_mlm_exp1'        # directory for storing logs
    ,logging_steps=40000
    ,logging_strategy='no'
)

trainer = tr.Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_data
    
)
```

**My Questions**

How can I train with larger batch size on `a2-highgpu-4g` machine?

Which parameters can I include in `TrainingArguments` so that training is fast and occupies small memory?

Thanks in advance.

### Versions

```
torch==1.11.0+cu113 

torchvision==0.12.0+cu113  

torchaudio==0.11.0+cu113 

transformers==4.17.0
```

Comments:
You can use the gradient accumulation step. You will find it in the argument by running your script xx.py --help. Also, your GPU is A100 so I think you can use the flag bf16 which is designed for amber GPUs like A100 and 3090. Why do you don't use fairseq? it has a max update argument which you can adjust to fit a larger batch size. In Fairseq, you control the batch size by adjusting max-token parameters. Use the largest max-token you could until you receive Out of Memory OOM error (sometimes you receive it in middle) and increase max update value which is equivalent to gradient steps. After you run your script you will see in logs something like bzs which refer to the batch size .

> You can use the gradient accumulation step. You will find it in the argument by running your script xx.py --help. Also, your GPU is A100 so I think you can use the flag bf16 which is designed for amber GPUs like A100 and 3090. Why do you don't use fairseq? it has a max update argument which you can adjust to fit a larger batch size. In Fairseq, you control the batch size by adjusting max-token parameters. Use the largest max-token you could until you receive Out of Memory OOM error (sometimes you receive it in middle) and increase max update value which is equivalent to gradient steps. After you run your script you will see in logs something like bzs which refer to the batch size .


**I tried with this parameter and training never started... it's stuck**

```
,gradient_accumulation_steps=8
,gradient_checkpointing=True
,fp16=True
,optim="adafactor"


```

bf16 is different than fp16

> bf16 is different than fp16

Still stuck !!


```
training_args = tr.TrainingArguments(
#     disable_tqdm=True
    output_dir='gs://****/results_mlm_exp1'
    ,overwrite_output_dir=True
    ,num_train_epochs=2
    ,per_device_train_batch_size=4
#     per_device_train_batch_size
#     per_gpu_train_batch_size
    ,prediction_loss_only=True
    ,save_strategy="no"
    ,run_name="MLM_Exp1"
    ,learning_rate=2e-5
    ,logging_dir='gs://****/logs_mlm_exp1'        # directory for storing logs
    ,logging_steps=26000
    ,gradient_accumulation_steps=8
     ,gradient_checkpointing=True
#     ,bf16=True
    ,optim="adafactor"
#     ,logging_strategy='no'
)
```

your logging steps is 26K . reduce it to 500 or 100

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.