MarianMT: "CUDA out of memory" when translating many times with the MarianMT Model
Author: agiagoulas
Pull request: no
Labels: name: marian

# 🐛 Bug
RuntimeError('CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 4.00 GiB total capacity; 3.03 GiB already allocated; 4.72 MiB free; 3.06 GiB reserved in total by PyTorch)')

## Information
I wrote a python notebook to translate datasets using  MarianMT. Therefore I wrote a function, that gets called a couple of thousand times in this translation process. The function looks like this:

```
def translate(data):
    batch = tok.prepare_translation_batch(data).to('cuda')
    gen = model.generate(**batch).to('cuda')
    data: List[str] = tok.batch_decode(gen, skip_special_tokens=True)
    return data
```

After about 1000 function calls with a size of data of about 300 words, the error occurs. I tried freeing the GPU memory with torch.cuda.empty_cache() and calling the garbage collector with gc.collect(), but nothing changes. To my understanding I need to use 'torch.no_grad()' but thats already implemented in the generate function of the model. 

Model I am using (Bert, XLNet ...): MarianMT

Language I am using the model on (English to German): Helsinki-NLP/opus-mt-en-de

## To reproduce

Steps to reproduce the behavior:

1. Translate an english text using the translate function provided below a couple of thousand times on a cuda enabled device 
2. The error occurs corresponding to your gpu after some time

```
src = 'en'  # source language
trg = 'de'  # target language
mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'
model = MarianMTModel.from_pretrained(mname)
tok = MarianTokenizer.from_pretrained(mname)
model.to('cuda')

def translate(data):
    batch = tok.prepare_translation_batch(data).to('cuda')
    gen = model.generate(**batch).to('cuda')
    data: List[str] = tok.batch_decode(gen, skip_special_tokens=True)
    return data
```

## Expected behavior
No Cuda out of memory error. The Cuda memory gets cleared after each translation process is done.

## Environment info
     
- `transformers` version: 3.0.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Thank you in advance for your help, I´m struggling with this error for a while!


Comments:
@sshleifer The Documentation page said to assign you, but I can only mention you. 

how big are your batches?

```python
src = 'en'  # source language
trg = 'de'  # target language
device='cuda'
mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'
model = MarianMTModel.from_pretrained(mname).to(device).half()  # fp16 should save lots of memory
tok = MarianTokenizer.from_pretrained(mname)
translations = []
for src_text_list in chunks(data, 8): # copy paste chunks fn from run_eval.py, consider wrapping tqdm_notebook
    batch = tok.prepare_translation_batch(src_text_list).to(device)
    gen = model.generate(**batch)
    german: List[str] = tok.batch_decode(gen, skip_special_tokens=True)
	translations.extend(german)
```

This is an example of a batch. They are all in this size.
Thanks in advance for your help!
`
['Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.', 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?', 'Saint Bernadette Soubirous', 'What is in front of the Notre Dame Main Building?', 'a copper statue of Christ', 'The Basilica of the Sacred heart at Notre Dame is beside to which structure?', 'the Main Building', 'What is the Grotto at Notre Dame?', 'a Marian place of prayer and reflection', 'What sits on top of the Main Building at Notre Dame?', 'a golden statue of the Virgin Mary']
`

Did my code work? Consider passing `max_length` to `prepare_translation_batch` if it doesn't.

@sshleifer It worked 👍 I used this fix in 2000 repetitions of the batch size at a time for a few times now and no error occured. Thank you very much for your help!