Finetuning ProphetNet with Seq2SeqTrainer fails.
Author: avacaondata
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.1
- Platform: Ubuntu 18
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1 (YES)
- Tensorflow version (GPU?):
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
@LysandreJik @patrickvonplaten @sgugger 

## Information

When trying to fine tune ProphetNet in a summarization task (with transformers/examples/seq2seq/finetune_trainer.py), the model crashes just after performing the evaluation. This script has worked fine with Bart, Pegasus and T5, the other 3 models I've tried.  The error trace is the following:

```{python}
                                                /opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.:24,  2.57it/s]
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
{'loss': 8.933700561523438, 'learning_rate': 2.992816091954023e-05, 'epoch': 0.04782400765184122}
Traceback (most recent call last):
  File "finetune_trainer.py", line 498, in <module>
    main()
  File "finetune_trainer.py", line 426, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 853, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 923, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer_seq2seq.py", line 96, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 1352, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 1469, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer_seq2seq.py", line 175, in prediction_step
    model, inputs, prediction_loss_only=prediction_loss_only, ignore_keys=ignore_keys
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 1574, in prediction_step
    outputs = model(**inputs)
  File "/home/alejandro.vaca/miniconda/envs/spainai_hackaton/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1769, in forward
    return_dict=return_dict,
  File "/home/alejandro.vaca/miniconda/envs/spainai_hackaton/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1667, in forward
    return_dict=return_dict,
  File "/home/alejandro.vaca/miniconda/envs/spainai_hackaton/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1365, in forward
    ) = self.compute_buffered_relative_buckets(position_ids)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1496, in compute_buffered_relative_buckets
    position_ids = torch.arange(1, self.max_target_positions).to(position_ids.device).repeat(1, 1)
RuntimeError: CUDA error: device-side assert triggered
  0%|          | 25/10440 [02:19<16:08:03,  5.58s/it]
```
Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
It arises when using official script for training Seq2Seq models.
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
A dataset with texts and their summaries. 

## To reproduce

Steps to reproduce the behavior:

1. Run script transformers/examples/seq2seq/finetune_trainer.py with any script you want, passing as argument the model for prophetnet. More concretely, call the script the following way:
```{bash}
python finetune_trainer.py --learning_rate=3e-5 --task summarization \
            --do_train --do_eval --evaluation_strategy steps --model_name_or_path microsoft/prophetnet-large-uncased \
            --data_dir mydatadir --output_dir myoutputdir \
            --per_device_train_batch_size 8 --per_device_eval_batch_size 16 \
            --eval_accumulation_steps 8 --gradient_accumulation_steps 8 --num_train_epochs=20 --eval_beams=1 \
            --load_best_model_at_end --save_steps 25 --logging_steps 25 --fp16 \
            --overwrite_output_dir
```



## Expected behavior

It should not crash when training ProphetNet, as it doesn't crash for Bart, Pegasus or T5...

Comments:
Hey @alexvaca0,

Thanks for your issue. We have started to create a more general script called `run_seq2seq.py` with which fine-tuning ProphetNet should work rather easily. 

Could you try to pull current master and do:

```
python examples/seq2seq/run_seq2seq.py --learning_rate=3e-5 --task summarization             --do_train --do_eval --evaluation_strategy steps --model_name_or_path microsoft/prophetnet-large-uncased             --output_dir myoutputdir             --per_device_train_batch_size 8 --per_device_eval_batch_size 16             --eval_accumulation_steps 8 --gradient_accumulation_steps 8 --num_train_epochs=20 --eval_beams=1             --load_best_model_at_end --save_steps 25 --logging_steps 25 --fp16             --overwrite_output_dir --dataset_name cnn_dailymail --dataset_config_name 3.0.0
```

for the cnn/dailymail dataset *e.g.*.

Please let me know how it goes, I'm very interested in ProphetNet fine-tuning results.

Thank you very much for your quick response! @patrickvonplaten As soon as I can, I'll try that command to check if the new script run_seq2seq.py works fine with ProphetNet. When I have results/errors I'll let you know.


I've tried to run the script you said @patrickvonplaten , but it returns the following error when evaluating:

```{python}
All the weights of ProphetNetForConditionalGeneration were initialized from the model checkpoint at microsoft/prophetnet-large-uncased.
If your task is similar to the task the model of the checkpoint was trained on, you can already use ProphetNetForConditionalGeneration for predictions without further training.
Loading cached processed dataset at /root/.cache/huggingface/datasets/csv/default-2def39d5bd2a9c76/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2/cache-7e4959c336c61e5a.arrow
Loading cached processed dataset at /root/.cache/huggingface/datasets/csv/default-2def39d5bd2a9c76/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2/cache-b898db3404de8043.arrow
The following columns in the training set don't have a corresponding argument in `ProphetNetForConditionalGeneration.forward` and have been ignored: token_type_ids.
The following columns in the evaluation set don't have a corresponding argument in `ProphetNetForConditionalGeneration.forward` and have been ignored: token_type_ids.
***** Running training *****
  Num examples = 33451
  Num Epochs = 20
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 128
  Gradient Accumulation steps = 16
  Total optimization steps = 5220
{'loss': 5.5221, 'learning_rate': 4.760536398467433e-05, 'epoch': 0.96}
  5% 250/5220 [16:57<5:41:20,  4.12s/it]***** Running Evaluation *****
  Num examples = 2697
  Batch size = 16

  0% 0/169 [00:00<?, ?it/s]
  1% 2/169 [00:00<00:10, 16.33it/s]
  2% 3/169 [00:00<00:12, 13.45it/s]
  3% 5/169 [00:00<00:14, 11.34it/s]
  4% 6/169 [00:00<00:15, 10.62it/s]
  4% 7/169 [00:00<00:21,  7.53it/s]
  5% 8/169 [00:00<00:21,  7.48it/s]
  5% 9/169 [00:01<00:21,  7.54it/s]
  6% 10/169 [00:01<00:24,  6.46it/s]
  7% 11/169 [00:01<00:22,  7.01it/s]
  7% 12/169 [00:01<00:21,  7.40it/s]
  8% 13/169 [00:01<00:20,  7.47it/s]
  8% 14/169 [00:01<00:19,  8.06it/s]
  9% 15/169 [00:01<00:18,  8.48it/s]
  9% 16/169 [00:01<00:19,  7.72it/s]
 10% 17/169 [00:02<00:18,  8.19it/s]
 11% 18/169 [00:02<00:19,  7.60it/s]
 11% 19/169 [00:02<00:19,  7.77it/s]
 12% 20/169 [00:02<00:19,  7.60it/s]
 12% 21/169 [00:02<00:20,  7.31it/s]
 13% 22/169 [00:02<00:18,  7.79it/s]
 14% 23/169 [00:02<00:19,  7.36it/s]
 14% 24/169 [00:03<00:18,  7.76it/s]
 15% 25/169 [00:03<00:18,  7.77it/s]
 15% 26/169 [00:03<00:18,  7.93it/s]
 16% 27/169 [00:03<00:17,  8.29it/s]
 17% 28/169 [00:03<00:18,  7.82it/s]
 17% 29/169 [00:03<00:22,  6.14it/s]
 18% 30/169 [00:03<00:24,  5.79it/s]
 18% 31/169 [00:04<00:22,  6.04it/s]
 19% 32/169 [00:04<00:21,  6.28it/s]
 20% 34/169 [00:04<00:18,  7.11it/s]
 21% 35/169 [00:04<00:17,  7.67it/s]
 21% 36/169 [00:04<00:16,  7.98it/s]
 22% 37/169 [00:04<00:15,  8.27it/s]
 22% 38/169 [00:04<00:17,  7.38it/s]
 23% 39/169 [00:05<00:20,  6.40it/s]
 24% 40/169 [00:05<00:18,  7.00it/s]
 25% 42/169 [00:05<00:16,  7.65it/s]
 26% 44/169 [00:05<00:15,  8.00it/s]
 27% 45/169 [00:05<00:14,  8.46it/s]
 28% 47/169 [00:06<00:14,  8.53it/s]
 28% 48/169 [00:06<00:13,  8.65it/s]
 29% 49/169 [00:06<00:15,  7.98it/s]
 30% 50/169 [00:06<00:14,  8.33it/s]
 30% 51/169 [00:06<00:15,  7.67it/s]
 31% 52/169 [00:06<00:14,  7.95it/s]
 31% 53/169 [00:06<00:16,  7.03it/s]
 32% 54/169 [00:07<00:21,  5.43it/s]
 33% 55/169 [00:07<00:18,  6.27it/s]
 33% 56/169 [00:07<00:16,  6.98it/s]
 34% 57/169 [00:07<00:14,  7.61it/s]
 34% 58/169 [00:07<00:13,  7.94it/s]
 35% 59/169 [00:07<00:13,  8.30it/s]
 36% 60/169 [00:07<00:12,  8.71it/s]
 36% 61/169 [00:07<00:12,  8.69it/s]
 37% 62/169 [00:08<00:12,  8.65it/s]
 37% 63/169 [00:08<00:13,  7.87it/s]
 38% 64/169 [00:08<00:13,  7.93it/s]
 39% 66/169 [00:08<00:12,  8.46it/s]
 40% 67/169 [00:08<00:13,  7.43it/s]
 40% 68/169 [00:08<00:13,  7.74it/s]
 41% 69/169 [00:08<00:12,  8.10it/s]
 41% 70/169 [00:08<00:11,  8.41it/s]
 42% 71/169 [00:09<00:11,  8.79it/s]
 43% 72/169 [00:09<00:10,  9.06it/s]
 43% 73/169 [00:09<00:10,  9.22it/s]
 44% 74/169 [00:09<00:10,  9.02it/s]
 45% 76/169 [00:09<00:10,  8.95it/s]
 46% 77/169 [00:09<00:11,  8.09it/s]
 46% 78/169 [00:09<00:10,  8.39it/s]
 47% 79/169 [00:10<00:10,  8.45it/s]
 47% 80/169 [00:10<00:10,  8.63it/s]
 48% 81/169 [00:10<00:10,  8.44it/s]
 49% 82/169 [00:10<00:12,  7.12it/s]
 49% 83/169 [00:10<00:11,  7.58it/s]
 50% 84/169 [00:10<00:13,  6.09it/s]
 50% 85/169 [00:10<00:12,  6.88it/s]
 51% 86/169 [00:11<00:11,  7.25it/s]
 51% 87/169 [00:11<00:10,  7.80it/s]
 52% 88/169 [00:11<00:09,  8.32it/s]
 53% 89/169 [00:11<00:09,  8.67it/s]
 53% 90/169 [00:11<00:08,  8.86it/s]
 54% 91/169 [00:11<00:08,  8.92it/s]
 54% 92/169 [00:11<00:08,  8.57it/s]
 55% 93/169 [00:11<00:08,  8.56it/s]
 56% 94/169 [00:11<00:08,  8.76it/s]
 56% 95/169 [00:12<00:08,  8.68it/s]
 57% 96/169 [00:12<00:08,  8.58it/s]
 57% 97/169 [00:12<00:09,  7.23it/s]
 58% 98/169 [00:12<00:09,  7.33it/s]
 59% 99/169 [00:12<00:08,  7.89it/s]
 59% 100/169 [00:12<00:08,  8.21it/s]
 60% 101/169 [00:12<00:09,  7.45it/s]
 60% 102/169 [00:12<00:08,  7.87it/s]
 61% 103/169 [00:13<00:08,  7.92it/s]
 62% 104/169 [00:13<00:08,  8.08it/s]
 62% 105/169 [00:13<00:08,  7.98it/s]
 63% 106/169 [00:13<00:08,  7.09it/s]/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [117,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [77,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [37,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "transformers/examples/seq2seq/run_seq2seq.py", line 541, in <module>
    main()
  File "transformers/examples/seq2seq/run_seq2seq.py", line 503, in main
    train_result = trainer.train(model_path=model_path)
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/trainer.py", line 924, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/trainer.py", line 999, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/trainer_seq2seq.py", line 96, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/trainer.py", line 1447, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/trainer.py", line 1564, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/trainer_seq2seq.py", line 175, in prediction_step
    model, inputs, prediction_loss_only=prediction_loss_only, ignore_keys=ignore_keys
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/trainer.py", line 1670, in prediction_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1772, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1656, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/gdrive/MyDrive/GColab_folder/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1223, in forward
    hidden_states = inputs_embeds + position_embeddings
RuntimeError: CUDA error: device-side assert triggered
  5% 250/5220 [17:12<5:42:05,  4.13s/it]
```
I've run it with --no_cuda and there are no errors, it works properly in that setting. Therefore it must be a cuda-related issue. I've tried disabling fp16 and the error persists.

I confirm that with t5 it works, therefore it's prophetnet-related.

Who is in charge of developing ProphetNet code? @patrickvonplaten @sgugger 

Hey @alexvaca0, thanks for trying out the script! I'm quite sure that this is an indexing error that occers because a data sample is too large for the model to handle. It should be easy to fix by simply adding:

```
--max_source_length 512
```

to the command above. Could you try this and let me know if it works? :-)

@patrickvonplaten Great! That was it, the sequence length! 

Actually, I'm trying to fine-tune ProphetNet in a Summarization task, in which models like T5, BART etc achieve eval losses of around 0.5-0.6 (approx), but with ProphetNet I'm not able to go below 5, and the eval loss doesn't actually decrease over training, it seems like it's diverging. I've tried using the same parameters as with BART and T5, and also with the parameters of the paper (https://arxiv.org/pdf/2001.04063.pdf) for CNN/DailyMail, that is batch size 512, learning rate 1e-04 with warmup steps 1000 (in my case I use less due to training data size). 

Any recommendations/suggestions? ProphetNet was expected to work similarly to BART but its performance is much worse until now...

I don't know if this warning provides some extra info: The following columns in the training set don't have a corresponding argument in `ProphetNetForConditionalGeneration.forward` and have been ignored: token_type_ids.
 @patrickvonplaten 

> @patrickvonplaten Great! That was it, the sequence length!
> 
> Actually, I'm trying to fine-tune ProphetNet in a Summarization task, in which models like T5, BART etc achieve eval losses of around 0.5-0.6 (approx), but with ProphetNet I'm not able to go below 5, and the eval loss doesn't actually decrease over training, it seems like it's diverging. I've tried using the same parameters as with BART and T5, and also with the parameters of the paper (https://arxiv.org/pdf/2001.04063.pdf) for CNN/DailyMail, that is batch size 512, learning rate 1e-04 with warmup steps 1000 (in my case I use less due to training data size).
> 
> Any recommendations/suggestions? ProphetNet was expected to work similarly to BART but its performance is much worse until now...

Interesting! Could you share the exact command you used here? Also pinging @qiweizhen - do you know what could be a problem for this? Are we sure that the n-gram loss is correctly implemented?

```{bash}
python transformers/examples/seq2seq/run_seq2seq.py \
      --model_name_or_path  microsoft/prophetnet-large-uncased \
      --do_eval --do_train \
      --task summarization \
      --train_file train_df.csv \
      --validation_file val_df.csv \
      --output_dir prophetnet_0201 \
      --overwrite_output_dir \
      --per_device_train_batch_size=8 \
      --per_device_eval_batch_size=16 \
      --eval_accumulation_steps=10 \
      --text_column text \
      --max_source_length 364 \
      --summary_column summary \
      --max_target_length 60 \
      --val_max_target_length 60 --evaluation_strategy steps \
      --gradient_accumulation_steps 64 --num_train_epochs=20 --eval_beams=1 \
      --load_best_model_at_end --save_steps 75 --logging_steps 75  --learning_rate 1e-04 --warmup_steps 200 
```
This is the command I'm using. After trying some modifications I observe the same: no progress is made in evaluation, and almost no progress in training (loss 5.1 after almost 7 epochs), so it seems there may be some issue with ProphetNet implementation...
@patrickvonplaten @qiweizhen 

I got the same problem that the training crashed in run_seq2seq.py. BTW, I guess that it is related to sequence lengths in configuration and training sets.

```bash
python ./run_seq2seq.py \
    --model_name_or_path sshleifer/student_marian_en_ro_6_1 \
    --do_train \
    --do_eval \
    --task translation_en_to_ro \
    --dataset_name wmt16 \
    --dataset_config_name ro-en \
    --source_lang en_XX \
    --target_lang ro_RO\
    --output_dir ~/tmp/tst-translation \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

transformers version: 4.4.0.dev0
Platform: Ubuntu 16.04.7
Python version: 3.8.5
PyTorch version (GPU?): 1.7.1 (YES)
Tensorflow version (GPU?):
Using GPU in script?: YES
Using distributed or parallel set-up in script?: Yes, it detects 6 GPUs.

Error:

> /opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu/opt/conda/conda-bld/pytorch_1607369981906/work/at                                                            en/src/ATen/native/cuda/Indexing.cu:658/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: index                                                            SelectLargeIndex:658: indexSelectLargeIndex: block: [264,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [264,0,0], thr                                                            ead: [1,0: block: [267,0,0: indexSelectLargeIndex], thread: [32,0: block: [263,0,0,0,0], thread: [96,0,0] Assertion `srcIndex < srcSele                                                            ctDimSize` failed.
] Assertion `srcIndex < srcSelectDimSize] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [263,0/opt/con                                                            da/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu` failed.
,0:658/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu], thread: [97: indexSelectLargeIndex:658,0:                                                             block: [264: indexSelectLargeIndex,0,0: block: [267] Assertion `srcIndex < srcSelectDimSize,0,0` failed.
], thread: [2,0/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu,0], thread: [33:658,0,0: indexSele                                                            ctLargeIndex] Assertion `srcIndex < srcSelectDimSize,0: block: [263` failed.
] Assertion `srcIndex < srcSelectDimSize,0` failed.
/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [267,0,0], thr                                                            ead: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [267,0,0], thr                                                            ead: [35,0,0/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu,0] Assertion `srcIndex < srcSelectDim                                                            Size:658], thread: [98` failed.
: indexSelectLargeIndex,0/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu: block: [264,0:658,0] As                                                            sertion `srcIndex < srcSelectDimSize: indexSelectLargeIndex,0` failed.
: block: [267], thread: [3/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIn                                                            dex: block: [263,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [263,0,0,0,0],                                                             thread: [100,0,0,0], thread: [36] Assertion `srcIndex < srcSelectDimSize,0,0` failed.
/opt/conda/conda-bld/pytorch_1607369981906/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [264,0,0], thr                                                            ead: [4,0] Assertion `srcIndex < srcSelectDimSize,0] Assertion `srcIndex < srcSelectDimSize` failed.
,0` failed.


Any updates on ProphetNet loss?? @patrickvonplaten 

I  have some more information on this. After training for 20 epochs, it learns almost nothing. Most interestingly, its outputs doesn't change when inputs change, that is, it always predicts the same. Predictions are like a mix of different summaries, getting elements from different types of summarizable texts, but it's the same for all... This brings me to think that in some way the network is constructed so that the output layer must always output the same thing, as if it must improve on all batches at the same time, I don't know if I'm explaining myself. It's clear that it is learning "something", in the sense that the summaries are clearly taken from my corpus style, but it's kind of learning to make the same summary for all texts. Since I'm using the same script as for other models, I guess there is some error in the network implementation...

Hey @alexvaca0,

I think I can reproduce your error. My training loss is also not improving after quite some time - will look into it!

Okay perfect! Please let me know when the issue is solved.

The original author @qiweizhen of the model was so nice to say he'll take a look. @qiweizhen - feel free to directly post any potential bugs in this PR.

Any updates on this? @qiweizhen @patrickvonplaten 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Ping

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

If prophetnet is not going to be fixed, then I think it should be removed from the library, as it is worthless having it here without being able to use it.

The model works fine in inference - it's the training that seems to be buggy. @qiweizhen - do you think we could take a look at ProphetNet together? 

> The model works fine in inference - it's the training that seems to be buggy. @qiweizhen - do you think we could take a look at ProphetNet together?



> If prophetnet is not going to be fixed, then I think it should be removed from the library, as it is worthless having it here without being able to use it.

Sorry. Will fix it as soon as possible.


It's strange that I can get correct inference / forward results with beam search, but as you pointed out, the model has non-convergence problem. I try to load the pretrained checkpoint and finetuned checkpoint to carry out further fine-tuning, all of their loss is optimized to 7.x and keeps that loss. With the finetuned checkpoint plus further fine-tuning, the results are still reasonable but a bit worse. I suspect the most part of the model is frozen and only a small part is trainable but I failed to find this bug. I also tried overfitting experiments and the model still can not converge. I will try 1) old Transformers version and 2) fairseq model to compare the intermediate hidden states with the latest Transformers prophetnet model to localize the bug this weekend.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Has the problem of ProphetNet non-convergence been solved? I want to fine tune it based on its checkpoint.

I think the code to compute the loss may be wrong.

This is the code to compute the loss in [`ProphetNetForConditionalGeneration`](https://github.com/huggingface/transformers/blob/master/src/transformers/models/prophetnet/modeling_prophetnet.py#L1968):
```python
        predicting_streams = outputs[1].view(batch_size, self.config.ngram, sequence_length, -1)
        predict_logits = self.lm_head(predicting_streams)

        ...

        loss = None
        if labels is not None:
            loss = self._compute_loss(predict_logits, labels)
```
The shape of `predicting_streams` is `(batch_size, ngram, sequence_length, hidden_size)`.
The shape of `predict_logits` is `(batch_size, ngram, sequence_length, vocab_size)`.
The shape of `labels` is `(batch_size, sequence_length)`.

Then pass `predict_logits` and `labels` to `_compute_loss`, the code of [`_compute_loss`](https://github.com/huggingface/transformers/blob/master/src/transformers/models/prophetnet/modeling_prophetnet.py#L2001) is: 
```python
    def _compute_loss(self, logits, labels, ignore_index=-100):
        expend_targets = labels.new_zeros(self.config.ngram, labels.size(0), labels.size(1)).fill_(ignore_index)

        for i in range(self.config.ngram):
            if i > 0 and self.disable_ngram_loss:
                break
            expend_targets[i, :, :] = labels

        lprobs = nn.functional.log_softmax(
            logits.view(-1, logits.size(-1)),
            dim=-1,
            dtype=torch.float32,
        )

        loss = nn.functional.nll_loss(lprobs, expend_targets.view(-1), reduction="mean")

        ...

        return loss
```
The shape of `expend_targets` is `(ngram, batch_size, sequence_length)`, the shape of `expend_targets.view(-1)` is `(ngram * batch_size * sequence_length)`, .
The shape of `lprobs` is `(batch_size * ngram * sequence_length, vocab_size)`.
Then computing the `nll_loss` of `lprobs` and `expend_targets` leads to the mismatch.

@patrickvonplaten 

This is the code of the prophetnet [hub](https://github.com/microsoft/ProphetNet/blob/master/ProphetNet_En/prophetnet/ngram_criterions.py#L36).
You can see [line 62](https://github.com/microsoft/ProphetNet/blob/master/ProphetNet_En/prophetnet/ngram_criterions.py#L62) that the shape of `logits` is `(ngram * batch_size * sequence_length, vocab_size)`.

Hey @StevenTang1998,

Thanks a lot for taking a closer look here! Would you be interested in opening a PR to fix it?

OK, I will open a PR after the test is successful.

I have opened a [pr](https://github.com/huggingface/transformers/pull/13132).
I did a test, but I am a unfamiliar with the submitted specifications. Please review it.

@alexvaca0, 

I think the fix by @StevenTang1998 may have fixed the training bug of ProphetNet. It would be amazing if you could give it a try with current master :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.