Got "RuntimeError: CUDA error: device-side assert triggered" with Seq2SeqTrainer
Author: ithieund
Pull request: no

## Environment info
- `transformers` version: 4.3.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
@patrickvonplaten @sgugger @stas00 

Models:
- encoderdecoder: @patrickvonplaten, @patil-suraj

Library:
- trainer: @sgugger

## Information

Model I am using (PhoBERT):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create an EncoderDecoderModel with phobert-base as encoder and phobert-base as decoder
2. Prepare train_data and val_data
3. Create Seq2SeqTrainer with that model and data

## Code
`trainer = Seq2SeqTrainer(
    model=sum_model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_data,
    eval_dataset=val_data,
)`

## Error
`---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-49-67f458786328> in <module>()
     25     compute_metrics=compute_metrics,
     26     train_dataset=train_data,
---> 27     eval_dataset=val_data,
     28 )

6 frames
/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in __init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers)
    269         # 2. fp16-enabled DeepSpeed loads the model in half the size and it doesn't need .to() anyway
    270         if not (self.is_model_parallel or args.deepspeed):
--> 271             model = model.to(args.device)
    272 
    273         # Force n_gpu to 1 to avoid DataParallel as MP will manage the GPUs

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in to(self, *args, **kwargs)
    610             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
    611 
--> 612         return self._apply(convert)
    613 
    614     def register_backward_hook(

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    357     def _apply(self, fn):
    358         for module in self.children():
--> 359             module._apply(fn)
    360 
    361         def compute_should_use_set_data(tensor, tensor_applied):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    357     def _apply(self, fn):
    358         for module in self.children():
--> 359             module._apply(fn)
    360 
    361         def compute_should_use_set_data(tensor, tensor_applied):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    357     def _apply(self, fn):
    358         for module in self.children():
--> 359             module._apply(fn)
    360 
    361         def compute_should_use_set_data(tensor, tensor_applied):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    379                 # `with torch.no_grad():`
    380                 with torch.no_grad():
--> 381                     param_applied = fn(param)
    382                 should_use_set_data = compute_should_use_set_data(param, param_applied)
    383                 if should_use_set_data:

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in convert(t)
    608             if convert_to_format is not None and t.dim() == 4:
    609                 return t.to(device, dtype if t.is_floating_point() else None, non_blocking, memory_format=convert_to_format)
--> 610             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
    611 
    612         return self._apply(convert)

RuntimeError: CUDA error: device-side assert triggered`

## Actual behavior

When I reduce the max_encoder_length to 80, it works OK

But when I increase the max_encoder_length >= 100, the error occurred!

## Expected behavior

The code should run properly with max_encoder_length = 512


Comments:
Hi there. Could you please post the code you are using? The steps you are defining are too vague for us to efficiently reproduce the issue and help.

Hi @sgugger 
It was my fault when I update the max_encoder_length and re-run on the updated blocks.
The issue will not happen when I restart the kernel on Google Colab.

I think you can close the issue.
Thank you.

Glad you could resolve your issue!