CUDA out of memory in evaluation_loop
Author: OfirArviv
Pull request: no

Hi,

I'm trying to run the seq2seq question answering example from the repro ([here](https://github.com/huggingface/transformers/blob/main/examples/pytorch/question-answering/run_seq2seq_qa.py)) and while training is fine in evaluation loop I get CUDA out of memory. I've read other asked about it previously and they suggested using `eval_accumulation_steps=10,`. Tried that and stil get the error.
Specifically:
```
  File "question-answering/run_seq2seq_qa.py", line 740, in <module>
    main()
  File "question-answering/run_seq2seq_qa.py", line 699, in main
    metrics = trainer.evaluate(max_length=max_length, num_beams=num_beams, metric_key_prefix="eval")
  File "/question-answering/trainer_seq2seq_qa.py", line 65, in evaluate
    ignore_keys=ignore_keys,
  File "/venv/lib/python3.7/site-packages/transformers/trainer.py", line 2515, in evaluation_loop
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
  File "/venv/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 97, in nested_concat
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/venv/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 97, in <genexpr>
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/venv/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 99, in nested_concat
    return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
  File "/venv/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 61, in torch_pad_and_concatenate
    return torch.cat((tensor1, tensor2), dim=0)
RuntimeError: CUDA out of memory. Tried to allocate 6.26 GiB (GPU 0; 23.69 GiB total capacity; 15.20 GiB already allocated; 96.62 MiB free; 21.46 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

Would appriciate any suggestions.


Comments:
+1

Update: assigning a value to eval_accumulation_steps works for my case. @OfirArviv Did you try 
`--eval_accumulation_steps 1`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm experiencing the same issue.
When setting --eval_accumulation_steps 1, the process gets killed after few evaluation steps.

eval_accumulation_steps的大小可有效缓解，这是因为eval_accumulation_steps是设置多少个steps将eval的结果move到cpu,如果不设置，默认是在GPU不断地按step累加。如果你的eval输出的结果总和大于显存那么就会出现CUDA OOM，当然如果你的cpu内存也不够大，同样会出现cpu的OOM.具体计算如下,如果是MLM任务，5000条eval样本，max_seq_len=256,vocab_size=21128(bert中文词汇表大小),那么你的预测结果如下：
shape = (5000,256,21128)
mermory=5000*256*21128*2(f16半精度)/1000/1024/1024≈52G.【2标识16位浮点数2个字节=2byte】
如果是f32则翻一倍=104G,别说显存撑不住，内存都撑不住。
一个有效的方法是：重写eval_loop,因为最终都是要做argmax使得shape = (5000,256,21128)=>(5000,256),那么我们可以在concat之前就做这一步。


> I'm experiencing the same issue. When setting --eval_accumulation_steps 1, the process gets killed after few evaluation steps.

for me it runs for exactly 9 steps, with 1 as eval batch size