T5-base out of memory on one 2080 GPU with batchsize 4, sequence length 100
Author: Arvid-pku
Pull request: no

I want to finetune T5 on totto dataset but failed.
This is werid  that it is OOM suddenly when it is training about 10% of one epoch.
And before that it is normal. about 3500M.

Very grateful for help! thx!
here is my simple code:
```
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration,Adafactor
import pandas as pd

tokenizer = T5Tokenizer.from_pretrained('../t5-base')
model = T5ForConditionalGeneration.from_pretrained('../t5-base', return_dict=True)

if torch.cuda.is_available():
   dev = torch.device("cuda:0")
   print("Running on the GPU")
else:
   dev = torch.device("cpu")
   print("Running on the CPU")
model.to(dev)

train_df=pd.read_csv('../totto_data/tt.csv', index_col=[0])
train_df=train_df.iloc[:50000,:]
train_df=train_df.sample(frac = 1)
optimizer = Adafactor(model.parameters(),lr=1e-3,
                      eps=(1e-30, 1e-3),
                      clip_threshold=1.0,
                      decay_rate=-0.8,
                      beta1=None,
                      weight_decay=0.0,
                      relative_step=False,
                      scale_parameter=False,
                      warmup_init=False)

num_of_epochs = 6
batch_size=4
num_of_batches=len(train_df)//batch_size

model.train()

for epoch in range(1,num_of_epochs+1):
  print('Running epoch: {}'.format(epoch))
  
  running_loss=0
  print(epoch)
  for i in range(num_of_batches):
    inputbatch=[]
    labelbatch=[]
    if i % 1000==0:
      print(i/num_of_batches)
    new_df=train_df[i*batch_size:i*batch_size+batch_size]
    for indx,row in new_df.iterrows():
      input = row['input_text']+'
' 
      labels = row['target_text']+'
'   
      inputbatch.append(input)
      labelbatch.append(labels)
    inputbatch=tokenizer.batch_encode_plus(inputbatch,padding=True,max_length=100,return_tensors='pt')["input_ids"]
    labelbatch=tokenizer.batch_encode_plus(labelbatch,padding=True,max_length=100,return_tensors="pt") ["input_ids"]
    inputbatch=inputbatch.to(dev)
    labelbatch=labelbatch.to(dev)
    # clear out the gradients of all Variables 
    optimizer.zero_grad()

    # Forward propogation
    outputs = model(input_ids=inputbatch, labels=labelbatch)
    loss = outputs.loss
    loss_num=loss.item()
    logits = outputs.logits
    running_loss+=loss_num

    # calculating the gradients
    loss.backward()

    #updating the params
    optimizer.step()
    torch.cuda.empty_cache()

  running_loss=running_loss/int(num_of_batches)
  print('Epoch: {} , Running loss: {}'.format(epoch,running_loss))
  torch.save(model.state_dict(),'./finetune/pytoch_model.bin'+str(epoch+1))

```




and this is my python libraries:

backcall                      0.2.0
backports.functools-lru-cache 1.6.1
certifi                       2020.12.5
chardet                       4.0.0
click                         7.1.2
decorator                     4.4.2
filelock                      3.0.12
idna                          2.10
importlib-metadata            3.7.2
ipykernel                     5.5.0
ipython                       7.21.0
ipython-genutils              0.2.0
jedi                          0.18.0
joblib                        1.0.1
jsonlines                     2.0.0
jupyter-client                6.1.11
jupyter-core                  4.7.1
mkl-fft                       1.3.0
mkl-random                    1.2.0
mkl-service                   2.3.0
numpy                         1.19.2
olefile                       0.46
packaging                     20.9
pandas                        1.2.3
parso                         0.8.1
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        8.1.2
pip                           21.0.1
prompt-toolkit                3.0.16
ptyprocess                    0.7.0
Pygments                      2.8.1
pyparsing                     2.4.7
python-dateutil               2.8.1
pytz                          2021.1
pyzmq                         22.0.3
regex                         2020.11.13
requests                      2.25.1
sacremoses                    0.0.43
sentencepiece                 0.1.95
setuptools                    49.6.0.post20210108
six                           1.15.0
tokenizers                    0.10.1
torch                         1.8.0
torchaudio                    0.8.0a0+a751e1d
torchvision                   0.9.0
tornado                       6.1
tqdm                          4.59.0
traitlets                     5.0.5
transformers                  4.3.3
typing-extensions             3.7.4.3
urllib3                       1.26.3
wcwidth                       0.2.5
wheel                         0.36.2
zipp                          3.4.1


Comments:
I don't think you should use `torch.cuda.empty_cache()`, as explained on [PyTorch's forum](https://discuss.pytorch.org/t/about-torch-cuda-empty-cache/34232/2), "This function should not be used by the end-user except in very edge cases.". 

Also, you can set `truncation=True`, because currently you're only padding examples, but not truncating examples that are too long. 

Btw, it's better to ask training related questions which are not bugs caused by the Transformers library on the [forum](https://discuss.huggingface.co/) rather than here.

@NielsRogge 
Thanks for your help! It works!
I use `torch.cuda.empty_cache()` because I have no idea but try it.
And I will go to the website you said then.
Thank you again!