[marian] possible memory leak problem while translating & extracting internal representations
Author: jrvc
Pull request: no

# 🐛 Bug

## Information
I am extracting the internal representations of some of the Marian models. 
There seems to be a memory leak problem. In this issue, you will find code for running the model sentence by sentence (bsz = 1) just to keep it simple. When I use batching, the problem persists and arises earlier. 

Model I am using: MarianMT 
` modelnames=[f'Helsinki-NLP/opus-mt-en-{tgt}'  for tgt in ['de', 'fr', 'ee', 'sv', 'el', 'fi', 'cs', 'ru' ]]`

Language I am using the model on: en-{tgt] 

The problem arises when using:
* [ ] a mix of official example scripts and my own: 
on this code, I keep the lines used to see if it is a memory problem. Hence the `empty_cache()`, keeping track of the memory usage with `memory_stats()`, and passing things to 'cpu' (but this has not solved the problem for me)
```
import torch
import transformers
config_overrider={'output_attentions':True, 'output_hidden_states':True}
model = transformers.MarianMTModel.from_pretrained(modelname, **config_overrider)
tokenizer = transformers.MarianTokenizer.from_pretrained(modelname) 
model.eval()
encoded_sentences = []
memdict=[]
for sent in tqdm(sentences):
        tokdsent = self.tokenizer.prepare_translation_batch(src_texts=[' '.join(sent)]) 
        tokdsent = {k:v.to(self.device) for k,v in tokdsent.items()}
        model_outputs = self.model.forward(**tokdsent) 
        encoded_sentences.append( [x.to('cpu') for x in model_outputs[4]+model_outputs[1]]  )
        torch.cuda.empty_cache()

        memdict.append(torch.cuda.memory_stats(self.device))
        print(memdict[-1]['active.all.current'],memdict[-1]['active.all.peak']) # comment out this part 
```


The tasks I am working on is:
* [ ] Using a dataset from an official task: 
semantic textual similarity - STS 2012, 2013, 2014, 2015 and 2016 (can use [this one](https://github.com/Helsinki-NLP/Geometry/blob/refactorize/data/STS/allSTS.txt))

## To reproduce

Steps to reproduce the behavior:

1. Load and tokenize the sentences (need this for what I am doing, even when I detok when passing it to the tokenizer)
```
STS_path = "path/to/allSTS.txt"
with open(STS_path, 'r') as f:
    samples = f.readlines()

sentences = []
for sent in samples:
    sent = sent.strip()
    sent = re.findall(r'[\w]+|\.|,|\?|\!|;|:|\'|\(|\)|/',sent)
    sentences.append(sent)
```
2. Run the code above. The one on _"The problem arises when using:"_
3. For me, around line 3750 I get OOM:
```
RuntimeError('CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 31.75 GiB total capacity; 30.67 GiB already allocated; 17.69 MiB free; 30.67 GiB reserved in total by PyTorch)')
```
Here I copy some of the linesprinted from `active.all.current ` and `active.all.peak` (it never changes the upwards trend):
```
533 535
779 811
1025 1057
1271 1303
1517 1549
1763 1795
2009 2041
2255 2287
2501 2533
2747 2779
2993 3025
...
9635 9667
9881 9913
10127 10159
10373 10405
10619 10651
...
921311 921343
921557 921589
921803 921835
922049 922081
922295 922327
922541 922573
```
^-- these are the first 10 lines, somewhere around 40 sentences, and the last lines before running out of mem - close to 3750 sents. 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I would expect the memory on the cuda device to be freed after every iteration since I am overwriting the variable there and what I append to the list I want to keep is sent to cpu.  


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Linux 3.10.0-1062.7.1.el7.x86_64 x86_64, Red Hat Enterprise Linux Server 7.7 (Maipo)
- Python version: 3.7.3
- PyTorch version (GPU?): 1.5.0 for cuda 10.2 (Nvidia Volta V100 GPU with 32 GB of memory)
- Tensorflow version (GPU?):  not using tf
- Using GPU in script?: yes (but I have seen the same problem on CPU)
- Using distributed or parallel set-up in script?: no


Comments:
ooook.... this is embarrassing. I just realized that I had to detach the variable, so GPU memory could be freed. 
This does the trick:
```
encoded_sentences.append( [x.detach().to('cpu') for x in model_outputs[4]+model_outputs[1]]  )
```

sorry for the trouble ;) and thanks for the repo and all your hard work