RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB with 8 Ampere GPU's
Author: pratikchhapolika
Pull request: no

I am fine tuning masked language model from XLM Roberta large on google machine specs.
I made couple of experiments and was strange to see few results. `I think something is not functioning properly. `

I am using pre-trained Hugging face model. 

`I launch it as train.py file which I copy inside docker image and use vertex-ai ( GCP) to launch it using Containerspec`

`machineSpec = MachineSpec(machine_type="a2-highgpu-8g",accelerator_count=8,accelerator_type="NVIDIA_TESLA_A100")`


```
container = ContainerSpec(image_uri="us-docker.pkg.dev/*****",
                          command=["/bin/bash", "-c", "gsutil cp gs://***/tfr_code2.tar.gz . && tar xvzf tfr_code2.tar.gz && cd pythonPackage/trainer/ && python train.py"])

```

I am using 

https://huggingface.co/xlm-roberta-large

I am not using fairseq or anything.


```
tokenizer = tr.XLMRobertaTokenizer.from_pretrained("xlm-roberta-large",local_files_only=True)
model = tr.XLMRobertaForMaskedLM.from_pretrained("xlm-roberta-large", return_dict=True,local_files_only=True)
model.gradient_checkpointing_enable() #included as new line
```

Here is `Nvidia-SMI`


```
b'Tue Mar 22 05:06:40 2022 \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 450.119.04 Driver Version: 450.119.04 CUDA Version: 11.0 
|\n|-------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. 
|\n| | | MIG M. |\n|===============================+======================+======================|\n| 0 A100-SXM4-40GB On | 00000000:00:04.0 Off | 0 |\n| N/A 33C P0 54W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled 
|\n+-------------------------------+----------------------+----------------------+\n| 1 A100-SXM4-40GB On | 00000000:00:05.0 Off | 0 |\n| N/A 31C P0 53W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled |
\n+-------------------------------+----------------------+----------------------+\n| 2 A100-SXM4-40GB On | 00000000:00:06.0 Off | 0 |\n| N/A 31C P0 54W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled |
\n+-------------------------------+----------------------+----------------------+\n| 3 A100-SXM4-40GB On | 00000000:00:07.0 Off | 0 |\n| N/A 34C P0 54W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled |
\n+-------------------------------+----------------------+----------------------+\n| 4 A100-SXM4-40GB On | 00000000:80:00.0 Off | 0 |\n| N/A 32C P0 57W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled |
\n+-------------------------------+----------------------+----------------------+\n| 5 A100-SXM4-40GB On | 00000000:80:01.0 Off | 0 |\n| N/A 34C P0 54W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled |
\n+-------------------------------+----------------------+----------------------+\n| 6 A100-SXM4-40GB On | 00000000:80:02.0 Off | 0 |\n| N/A 32C P0 54W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled |
\n+-------------------------------+----------------------+----------------------+\n| 7 A100-SXM4-40GB On | 00000000:80:03.0 Off | 0 |\n| N/A 34C P0 61W / 400W | 0MiB / 40537MiB | 0% Default |\n| | | Disabled |
\n+-------------------------------+----------------------+----------------------+\n \n+-----------------------------------------------------------------------------+\n| Processes: |\n| GPU GI CI PID Type Process name GPU Memory |\n| ID ID Usage |\n|=============================================================================|\n| No running processes found |\n+-----------------------------------------------------------------------------+\n'
2022-03-22T05:10:07.712355Z
```

**It has lot of free memories but still I get this error.**


`RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 39.59 GiB total capacity; 33.48 GiB already allocated; 3.19 MiB free; 34.03 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF`





```
['Traceback (most recent call last):\n', ' File "train.py", line 144, in <module>\n trainer.train()\n', ' File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1400, in train\n tr_loss_step = self.training_step(model, inputs)\n', ' File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1994, in training_step\n self.scaler.scale(loss).backward()\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward\n torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 175, in backward\n allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/autograd/function.py", line 253, in apply\n return user_fn(self, *args)\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/utils/checkpoint.py", line 146, in backward\n torch.autograd.backward(outputs_with_grad, args_with_grad)\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 175, in backward\n allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/autograd/function.py", line 253, in apply\n return user_fn(self, *args)\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/_functions.py", line 34, in backward\n return (None,) + ReduceAddCoalesced.apply(ctx.input_device, ctx.num_inputs, *grad_outputs)\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/_functions.py", line 45, in forward\n return comm.reduce_add_coalesced(grads_, destination)\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/comm.py", line 143, in reduce_add_coalesced\n flat_result = reduce_add(flat_tensors, destination)\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/comm.py", line 95, in reduce_add\n result = torch.empty_like(inputs[root_index])\n', 'RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 39.59 GiB total capacity; 33.48 GiB already allocated; 3.19 MiB free; 34.03 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF\n'] | 2022-03-22T05:11:26.409945Z
-- | --
```





**Training Code**



```
training_args = tr.TrainingArguments(

     output_dir='****'
    ,logging_dir='****'        # directory for storing logs
    ,save_strategy="epoch"
    ,run_name="****"
    ,learning_rate=2e-5
    ,logging_steps=1000
    ,overwrite_output_dir=True
    ,num_train_epochs=10
    ,per_device_train_batch_size=8
    ,prediction_loss_only=True
    ,gradient_accumulation_steps=4
#     ,gradient_checkpointing=True
    ,bf16=True #57100 
,optim="adafactor"

)


trainer = tr.Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_data
)

```

Also `gradient_checkpointing` never works. Strange. 

**Also, is it using all 8 GPU's?**


### Versions

Versions
torch==1.11.0+cu113 

torchvision==0.12.0+cu113  

torchaudio==0.11.0+cu113 

transformers==4.17.0




**Train.py**


```
import torch
import numpy as np
import pandas as pd
from transformers import BertTokenizer, BertForSequenceClassification
import transformers as tr
from sentence_transformers import SentenceTransformer
from transformers import XLMRobertaTokenizer, XLMRobertaForMaskedLM
from transformers import AdamW
from transformers import AutoTokenizer
from transformers import BertTokenizerFast as BertTokenizer, BertModel, AdamW, get_linear_schedule_with_warmup,BertForMaskedLM
from transformers import DataCollatorForLanguageModeling
from scipy.special import softmax
import scipy
import random
import pickle
import os
import time

import subprocess as sp


# torch.cuda.empty_cache()


print(sp.check_output('nvidia-smi'))
print("current device",torch.cuda.current_device())
print("device count",torch.cuda.device_count())

# os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
# os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2,3,4,5,6,7"

import sys
from traceback import format_exception
def my_except_hook(exctype, value, traceback):
    print(format_exception(exctype, value, traceback))
    sys.__excepthook__(exctype, value, traceback)
sys.excepthook = my_except_hook

start=time.time()

print("package imported completed")

os.environ['TRANSFORMERS_OFFLINE']='1'
os.environ['HF_MLFLOW_LOG_ARTIFACTS']='TRUE'
# os.environ['PYTORCH_CUDA_ALLOC_CONF']='max_split_size_mb'

print("env setup completed")
print( "transformer",tr.__version__)
print("torch",torch.__version__)

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
print("Using", device)
torch.backends.cudnn.deterministic = True  

tr.trainer_utils.set_seed(0)

print("here")

tokenizer = tr.XLMRobertaTokenizer.from_pretrained("xlm-roberta-large",local_files_only=True)
model = tr.XLMRobertaForMaskedLM.from_pretrained("xlm-roberta-large", return_dict=True,local_files_only=True)
model.gradient_checkpointing_enable() #included as new line
print("included gradient checkpoint")

model.to(device)
print("Model loaded successfully")

df=pd.read_csv("gs://******/data.csv") 
print("read csv")
# ,engine='openpyxl',sheet_name="master_data"
train_df=df.text.tolist()
print(len(train_df))

train_df=list(set(train_df))
train_df = [x for x in train_df if str(x) != 'nan']

print("Length of training data is \n ",len(train_df))
print("DATA LOADED successfully")


train_encodings = tokenizer(train_df, truncation=True, padding=True, max_length=512, return_tensors="pt")
print("encoding done")

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)
print("data collector done")

class SEDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings
        
    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        return item

    def __len__(self):
        return len(self.encodings["attention_mask"])

train_data = SEDataset(train_encodings)

print("train data created")

training_args = tr.TrainingArguments(

     output_dir='gs://****/results_mlm_exp1'
    ,logging_dir='gs://****logs_mlm_exp1'        # directory for storing logs
    ,save_strategy="epoch"
#     ,run_name="MLM_Exp1"
    ,learning_rate=2e-5
    ,logging_steps=500
    ,overwrite_output_dir=True
    ,num_train_epochs=20
    ,per_device_train_batch_size=4
    ,prediction_loss_only=True
    ,gradient_accumulation_steps=2
#     ,sharded_ddp='zero_dp_3'
#     ,gradient_checkpointing=True
    ,bf16=True #Ampere GPU
#     ,fp16=True
#     ,optim="adafactor"
#     ,dataloader_num_workers=20
#     ,logging_strategy='no'
    #     per_device_train_batch_size
#     per_gpu_train_batch_size
    #     disable_tqdm=True
)

print("training sample is 400001")
print("Included ,gradient_accumulation_steps=4 ,bf16=True and per_device_train_batch_size=4 " )
print(start)
trainer = tr.Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_data
)
print("training to start without bf16")
trainer.train()
print("model training finished")
trainer.save_model("gs://*****/model_mlm_exp1")

print("training finished")

end=time.time()

print("total time taken in hours is", (end-start)/3600)
```

Comments:
@sgugger 

Please follow the issue template. You're not showing the script you are running or how you are launching it, so there is nothing anyone can do to help.

> Please follow the issue template. You're not showing the script you are running or how you are launching it, so there is nothing anyone can do to help.

Edited and Showing you the script.  

You're launching the script with just `python`. As highlighted in the [examples README](https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision) for distributed training, you need to use `python -m torch.distributed.launch` to use the 8 GPUs.

> You're launching the script with just `python`. As highlighted in the [examples README](https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision) for distributed training, you need to use `python -m torch.distributed.launch` to use the 8 GPUs.

Ok.

You mean to say I should use this:

`command=["/bin/bash", "-c", "gsutil cp gs://***/tfr_code2.tar.gz . && tar xvzf tfr_code2.tar.gz && cd pythonPackage/trainer/ && python -m torch.distributed.launch train.py"])`


> > You're launching the script with just `python`. As highlighted in the [examples README](https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision) for distributed training, you need to use `python -m torch.distributed.launch` to use the 8 GPUs.
> 
> Ok.
> 
> You mean to say I should use this:
> 
> `command=["/bin/bash", "-c", "gsutil cp gs://***/tfr_code2.tar.gz . && tar xvzf tfr_code2.tar.gz && cd pythonPackage/trainer/ && python -m torch.distributed.launch train.py"])`

I get this error:




```
RuntimeError: CUDA out of memory. Tried to allocate 7.63 GiB (GPU 0; 39.59 GiB total capacity; 23.22 GiB already allocated; 3.52 GiB free; 33.88 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF | 2022-03-22T16:24:41.763613Z
-- | --



```

Why its showing `GPU 0` only

I am using this parameters  or am I missing anything.

@sgugger 

```
training_args = tr.TrainingArguments(

     output_dir='**'
    ,logging_dir='**'        # directory for storing logs
    ,save_strategy="epoch"
    ,learning_rate=2e-5
    ,logging_steps=2500
    ,overwrite_output_dir=True
    ,num_train_epochs=20
    ,per_device_train_batch_size=4
    ,prediction_loss_only=True
    ,bf16=True #Ampere GPU
)

And Launching using this:

python -m torch.distributed.launch --nproc_per_node 8 train.py --bf16 

```

**ERROR**


```
Traceback (most recent call last):\n', ' File "train.py", line 159, in <module>\n trainer.train()\n', 
' File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1400, in train\n tr_loss_step = self.training_step(model, inputs)\n', 
' File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1984, in training_step\n loss = self.compute_loss(model, inputs)\n', 
' File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 2016, in compute_loss\n outputs = model(**inputs)\n', 
' File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl\n return forward_call(*input, **kwargs)\n', 
' File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 963, in forward\n output = self.module(*inputs[0], **kwargs[0])\n', 
' File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl\n return forward_call(*input, **kwargs)\n', 
' File "/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py", line 1114, in forward\n masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))\n', ' File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl\n return forward_call(*input, **kwargs)\n', 
' File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 1165, in forward\n label_smoothing=self.label_smoothing)\n', 
' File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 2996, in cross_entropy\n return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)\n', 

'RuntimeError: CUDA out of memory. Tried to allocate 978.00 MiB (GPU 0; 39.59 GiB total capacity; 6.40 GiB already allocated; 690.19 MiB free; 6.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` 
```

Any help?

@sgugger If I use distributed training, then would `trainer.save_model("gs://*****/model_mlm_exp1")` will work or do I need to pass any extra parameter so that only 1 model is saved from multiple GPU's?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Assalamu alekum, I also came across such a mistake            
 RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 7.80 GiB total capacity; 6.41 GiB already allocated; 1.69 MiB free; 6.82 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF