run_translation.py englisht-german translation failed. RuntimeError: CUDA error: device-side assert triggered
Author: bing0037
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: '4.13.0.dev0'
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU): '1.8.2' (GPU)
- Using GPU in script: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patil-suraj 

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
@patil-suraj 

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using: BERT

The problem arises when using:
* [*] my own modified scripts: (give details below)

The official script is to run **ro-en translation** (This works well):

I just changed the source language to English and the target language to German (**en-de translation**). I got the following error:
```
[INFO|trainer.py:1196] 2021-11-11 02:51:42,765 >> ***** Running training *****
[INFO|trainer.py:1197] 2021-11-11 02:51:42,765 >>   Num examples = 4548885
[INFO|trainer.py:1198] 2021-11-11 02:51:42,765 >>   Num Epochs = 3
[INFO|trainer.py:1199] 2021-11-11 02:51:42,765 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:1200] 2021-11-11 02:51:42,765 >>   Total train batch size (w. parallel, distributed & accumulation) = 4
[INFO|trainer.py:1201] 2021-11-11 02:51:42,765 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1202] 2021-11-11 02:51:42,765 >>   Total optimization steps = 3411666

  0%|          | 0/3411666 [00:00<?, ?it/s]
  0%|          | 1/3411666 [00:00<577:21:16,  1.64it/s]
  0%|          | 3/3411666 [00:00<204:19:40,  4.64it/s]
  0%|          | 5/3411666 [00:00<139:07:25,  6.81it/s]
  0%|          | 7/3411666 [00:01<113:23:47,  8.36it/s]
  0%|          | 9/3411666 [00:01<99:22:10,  9.54it/s] 
  0%|          | 11/3411666 [00:01<91:45:24, 10.33it/s]
  0%|          | 13/3411666 [00:01<86:15:44, 10.99it/s]
...
  0%|          | 993/3411666 [01:15<94:20:15, 10.04it/s]
  0%|          | 995/3411666 [01:15<94:43:16, 10.00it/s]
  0%|          | 997/3411666 [01:15<94:12:00, 10.06it/s]/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [446,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [446,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [446,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [446,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [446,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [446,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [446,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/
...
/opt/conda/conda-bld/pytorch_1627336334951/work/aten/src/ATen/native/cuda/Indexing.cu:660: indexSelectLargeIndex: block: [445,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
{'loss': 2.1577, 'learning_rate': 4.999267220179232e-05, 'epoch': 0.0}
Traceback (most recent call last):
  File "run_translation.py", line 622, in <module>
    main()
  File "run_translation.py", line 539, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/transformers/trainer.py", line 1316, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/transformers/trainer.py", line 1849, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/transformers/trainer.py", line 1881, in compute_loss
    outputs = model(**inputs)
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/transformers/models/marian/modeling_marian.py", line 1305, in forward
    return_dict=return_dict,
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/transformers/models/marian/modeling_marian.py", line 1181, in forward
    return_dict=return_dict,
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/transformers/models/marian/modeling_marian.py", line 968, in forward
    attention_mask, input_shape, inputs_embeds, past_key_values_length
  File "/home/guest/anaconda3/envs/huggingface_NMT/lib/python3.6/site-packages/transformers/models/marian/modeling_marian.py", line 849, in _prepare_decoder_attention_mask
    ).to(self.device)
RuntimeError: CUDA error: device-side assert triggered

  0%|          | 998/3411666 [01:16<72:19:19, 13.10it/s]
```

The tasks I am working on is:
* [*] an official GLUE/SQUaD task: English to German translation

## To reproduce

Steps to reproduce the behavior:

1. use the latest transformers code
2. run the following command:
```
CUDA_VISIBLE_DEVICES=3 python run_translation.py \
    --model_name_or_path Helsinki-NLP/opus-mt-en-de \
    --do_train \
    --source_lang en \
    --target_lang de \
    --dataset_name wmt16 \
    --dataset_config_name de-en \
    --output_dir output_NMT/tst-translation-en-de \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Can you run the code on CPU, please?

This will give a more informative error message.

> Can you run the code on CPU, please?
> 
> This will give a more informative error message.

on CPU, it works.
```
11/11/2021 08:24:42 - WARNING - __main__ - Process rank: -1, device: cpu, n_gpu: 0distributed training: False, 16-bits training: False
11/11/2021 08:24:42 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
_n_gpu=0,
adafactor=False,
adam_beta1=0.9,
...
  0%|          | 500/3411666 [03:56<464:46:33,  2.04it/s][INFO|trainer.py:1995] 2021-11-11 08:28:46,449 >> Saving model checkpoint to output_NMT/tst-translation-en-de/checkpoint-500
[INFO|configuration_utils.py:417] 2021-11-11 08:28:46,454 >> Configuration saved in output_NMT/tst-translation-en-de/checkpoint-500/config.json
[INFO|modeling_utils.py:1060] 2021-11-11 08:28:47,440 >> Model weights saved in output_NMT/tst-translation-en-de/checkpoint-500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2037] 2021-11-11 08:28:47,442 >> tokenizer config file saved in output_NMT/tst-translation-en-de/checkpoint-500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2043] 2021-11-11 08:28:47,442 >> Special tokens file saved in output_NMT/tst-translation-en-de/checkpoint-500/special_tokens_map.json

  0%|          | 501/3411666 [04:00<1332:20:21,  1.41s/it]
  0%|          | 502/3411666 [04:00<1064:13:24,  1.12s/it]
  0%|          | 503/3411666 [04:01<907:08:17,  1.04it/s] 
  0%|          | 504/3411666 [04:01<784:55:18,  1.21it/s]
```

Any suggestion for GPU modification or test? It's really slow on CPU. Thanks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.