Finetuning ProphetNet with Seq2SeqTrainer fails.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.1
- Platform: Ubuntu 18
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1 (YES)
- Tensorflow version (GPU?):
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
@LysandreJik @patrickvonplaten @sgugger

## Information

When trying to fine tune ProphetNet in a summarization task (with transformers/examples/seq2seq/finetune_trainer.py), the model crashes just after performing the evaluation. This script has worked fine with Bart, Pegasus and T5, the other 3 models I've tried.  The error trace is the following:

```{python}
                                                /opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.:24,  2.57it/s]
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [234,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [290,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [228,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1607370141920/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [284,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
{'loss': 8.933700561523438, 'learning_rate': 2.992816091954023e-05, 'epoch': 0.04782400765184122}
Traceback (most recent call last):
  File "finetune_trainer.py", line 498, in <module>
    main()
  File "finetune_trainer.py", line 426, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 853, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 923, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer_seq2seq.py", line 96, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 1352, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 1469, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer_seq2seq.py", line 175, in prediction_step
    model, inputs, prediction_loss_only=prediction_loss_only, ignore_keys=ignore_keys
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/trainer.py", line 1574, in prediction_step
    outputs = model(**inputs)
  File "/home/alejandro.vaca/miniconda/envs/spainai_hackaton/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1769, in forward
    return_dict=return_dict,
  File "/home/alejandro.vaca/miniconda/envs/spainai_hackaton/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1667, in forward
    return_dict=return_dict,
  File "/home/alejandro.vaca/miniconda/envs/spainai_hackaton/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1365, in forward
    ) = self.compute_buffered_relative_buckets(position_ids)
  File "/home/alejandro.vaca/SpainAI_Hackaton_2020/transformers/src/transformers/models/prophetnet/modeling_prophetnet.py", line 1496, in compute_buffered_relative_buckets
    position_ids = torch.arange(1, self.max_target_positions).to(position_ids.device).repeat(1, 1)
RuntimeError: CUDA error: device-side assert triggered
  0%|          | 25/10440 [02:19<16:08:03,  5.58s/it]
```
Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
It arises when using official script for training Seq2Seq models.
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
A dataset with texts and their summaries.

## To reproduce

Steps to reproduce the behavior:

1. Run script transformers/examples/seq2seq/finetune_trainer.py with any script you want, passing as argument the model for prophetnet. More concretely, call the script the following way:
```{bash}
python finetune_trainer.py --learning_rate=3e-5 --task summarization \
            --do_train --do_eval --evaluation_strategy steps --model_name_or_path microsoft/prophetnet-large-uncased \
            --data_dir mydatadir --output_dir myoutputdir \
            --per_device_train_batch_size 8 --per_device_eval_batch_size 16 \
            --eval_accumulation_steps 8 --gradient_accumulation_steps 8 --num_train_epochs=20 --eval_beams=1 \
            --load_best_model_at_end --save_steps 25 --logging_steps 25 --fp16 \
            --overwrite_output_dir
```

## Expected behavior

It should not crash when training ProphetNet, as it doesn't crash for Bart, Pegasus or T5...


convert_graph_to_onnx.convert broken for model bart-large / wmt19-en-de

Good Second Issue

Issues that are more difficult to do than "Good First" issues - give it a try if you want!

@stas00's edit on top:

I currently don't have the know-how in this domain, so if there are members of the community with ONNX experience and this issue resonates with you, please don't hesitate to comment if you'd like to work on resolving this. Thank you very much!

------------------------

## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Linux-4.15.0-132-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.5.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False
- ONNX Version: 1.5.2  (ONNX custom build w CUDA 11)

### Who can help
@stas00 (based on his suggestion to open a new issue in #9722 and run this with bart)
@patrickvonplaten  (based on link of @stas00 in #9722)
@mfuntowicz (based on link of @stas00 in #9722)
@LysandreJik (based on link of @stas00 in #9722)

## Information

Model I am using (Bert, XLNet ...): facebook/bart-large & facebook/wmt19-en-de

The problem arises when using:
* [X] the official example scripts: transformers.convert_graph_to_onnx.convert
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## Description

Initially, I was about to use the export for facebook/wmt19-en-de via ONNX for our deployment. Yet, it turns out that the exported models do not work properly. It seems, that there are several things broken for the export of this model type.

## To reproduce

### 1. Testing facebook/wmt19-en-de

```
import torch
import transformers
import numpy as np
import onnxruntime as rt
from pathlib import Path

from transformers import convert_graph_to_onnx

print(rt.__version__)

opt = rt.SessionOptions()

model_name = "facebook/wmt19-en-de"
pipeline_name = "translation_en_to_de"
model_pth = Path("encoder/en_de_trans.onnx")

if model_pth.exists():
    model_pth.unlink()

nlp = transformers.pipeline(pipeline_name, model=model_name, tokenizer=model_name)

convert_graph_to_onnx.convert(
    framework="pt",
    model=model_name,
    output=model_pth,
    opset=12,
    tokenizer=model_name,
    use_external_format= False,
    pipeline_name= pipeline_name,
)

sess = rt.InferenceSession(str(model_pth), opt)
spans = [
    "My name is Bert", # passes facebook/wmt19-en-de
    "My name is Bert and" # fails facebook/wmt19-en-de
]
for span in spans:
    model_input = nlp.tokenizer.encode_plus(span)
    model_input = {name : np.atleast_2d(value) for name, value in model_input.items()}
    out = nlp.model(**nlp.tokenizer(span, return_tensors="pt"))
    trans_1 = out[0].detach().cpu().numpy()
    trans_2 = out[1].detach().cpu().numpy()
    onnx_1, onnx_2 = sess.run(None, model_input)
    assert np.allclose(trans_1, onnx_1, atol=1e-5)
    assert np.allclose(trans_2, onnx_2, atol=1e-5)
```

Will raise the following exception:

```
Some weights of FSMTModel were not initialized from the model checkpoint at facebook/wmt19-en-de and are newly initialized: ['model.encoder.embed_positions.weight', 'model.decoder.embed_positions.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
ONNX opset version set to: 12
Loading pipeline (model: facebook/wmt19-en-de, tokenizer: facebook/wmt19-en-de)
Using framework PyTorch: 1.7.1
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
Found output output_1 with shape: {0: 'batch', 1: 'sequence'}
Ensuring inputs are in correct order
decoder_input_ids is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
**[skipped warnings for brevity...]**
---------------------------------------------------------------------------
RuntimeException                          Traceback (most recent call last)
<ipython-input-2-f4eec5b0ac5f> in <module>
     51     trans_1 = out[0].detach().cpu().numpy()
     52     trans_2 = out[1].detach().cpu().numpy()
---> 53     onnx_1, onnx_2 = sess.run(None, model_input)
     54     assert np.allclose(trans_1, onnx_1, atol=1e-5)
     55     assert np.allclose(trans_2, onnx_2, atol=1e-5)

~/anaconda3/envs/dev/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)
    122             output_names = [output.name for output in self._outputs_meta]
    123         try:
--> 124             return self._sess.run(output_names, input_feed, run_options)
    125         except C.EPFail as err:
    126             if self._enable_fallback:

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_74' Status Message: /data/shared/packages/onnxruntime/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:43 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,6}, requested shape:{5}
```

As stated in #9722, I'd assume that some dynamic shape of was not inferred properly/not passed to the dynamic_shapes of torch.onnx.export. But thats just a quick guess, which I find when I build my own ONNX models. Important: The first string passes the assertions, the second one doesn't.

### 2. Testing facebook/bart-large (feature extraction)

@stas00 suggested to re-test the behavior with the underlying BART model. Now, say we run the same script with the following parameters:

```
model_name = "facebook/bart-large"
pipeline_name = "feature-extraction"
model_pth = Path("generator/bart.onnx")
```

Raises

```
ONNX opset version set to: 12
Loading pipeline (model: facebook/bart-large, tokenizer: facebook/bart-large)
Using framework PyTorch: 1.7.1
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
**[skipped output axes for brevity...]**
Found output output_13 with shape: {0: 'batch', 1: 'sequence'}
Ensuring inputs are in correct order
decoder_input_ids is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
/home/oborchers/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/utils.py:1111: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input output_1
  warnings.warn('No names were found for specified dynamic axes of provided input.'
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-3362f5ef6ea8> in <module>
     30 nlp = transformers.pipeline(pipeline_name, model=model_name, tokenizer=model_name)
     31
---> 32 convert_graph_to_onnx.convert(
     33     framework="pt",
     34     model=model_name,

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/convert_graph_to_onnx.py in convert(framework, model, output, opset, tokenizer, use_external_format, pipeline_name)
    365     # Export the graph
    366     if framework == "pt":
--> 367         convert_pytorch(nlp, opset, output, use_external_format)
    368     else:
    369         convert_tensorflow(nlp, opset, output)

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/convert_graph_to_onnx.py in convert_pytorch(nlp, opset, output, use_external_format)
    277         ordered_input_names, model_args = ensure_valid_input(nlp.model, tokens, input_names)
    278
--> 279         export(
    280             nlp.model,
    281             model_args,

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/__init__.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
    223
    224     from torch.onnx import utils
--> 225     return utils.export(model, args, f, export_params, verbose, training,
    226                         input_names, output_names, aten, export_raw_ir,
    227                         operator_export_type, opset_version, _retain_param_name,

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/utils.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
     83         else:
     84             operator_export_type = OperatorExportTypes.ONNX
---> 85     _export(model, args, f, export_params, verbose, training, input_names, output_names,
     86             operator_export_type=operator_export_type, opset_version=opset_version,
     87             _retain_param_name=_retain_param_name, do_constant_folding=do_constant_folding,

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, opset_version, _retain_param_name, do_constant_folding, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, fixed_batch_size, custom_opsets, add_node_names, enable_onnx_checker, use_external_data_format, onnx_shape_inference, use_new_jit_passes)
    627             if dynamic_axes is None:
    628                 dynamic_axes = {}
--> 629             _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
    630
    631             graph, params_dict, torch_out = \

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/utils.py in _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
   1115             for i, x in enumerate(value):
   1116                 if not isinstance(x, int):
-> 1117                     raise ValueError("The type of axis index is expected to be an integer")
   1118                 if x in value_dict:
   1119                     warnings.warn('Duplicate dynamic axis index {} was provided for input {}.'

ValueError: The type of axis index is expected to be an integer
```

### 3. Testing facebook/bart-large (text-generation)

```
model_name = "facebook/bart-large"
pipeline_name = "text-generation"
model_pth = Path("generator/bart.onnx")
```

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-d6fa1456dc0e> in <module>
     28     model_pth.unlink()
     29
---> 30 nlp = transformers.pipeline(pipeline_name, model=model_name, tokenizer=model_name)
     31
     32 convert_graph_to_onnx.convert(

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/pipelines/__init__.py in pipeline(task, model, config, tokenizer, framework, revision, use_fast, **kwargs)
    403             )
    404
--> 405         model = model_class.from_pretrained(model, config=config, revision=revision, **model_kwargs)
    406         if task == "translation" and model.config.task_specific_params:
    407             for key in model.config.task_specific_params:

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/models/auto/modeling_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1040                 pretrained_model_name_or_path, *model_args, config=config, **kwargs
   1041             )
-> 1042         raise ValueError(
   1043             "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
   1044             "Model type should be one of {}.".format(

ValueError: Unrecognized configuration class <class 'transformers.models.bart.configuration_bart.BartConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of CamembertConfig, XLMRobertaConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, TransfoXLConfig, XLNetConfig, XLMConfig, CTRLConfig, ReformerConfig, BertGenerationConfig, XLMProphetNetConfig, ProphetNetConfig.
```

### 4. Testing facebook/bart-large (fill-mask)

```
model_name = "facebook/bart-large"
pipeline_name = "fill-mask"
model_pth = Path("generator/bart.onnx")
```

```
ONNX opset version set to: 12
Loading pipeline (model: facebook/bart-large, tokenizer: facebook/bart-large)
Using framework PyTorch: 1.7.1
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
**[skipped for brevity]**
Found output output_13 with shape: {0: 'batch', 1: 'sequence'}
Ensuring inputs are in correct order
decoder_input_ids is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-d55ec01c8b87> in <module>
     34 nlp = transformers.pipeline(pipeline_name, model=model_name, tokenizer=model_name)
     35
---> 36 convert_graph_to_onnx.convert(
     37     framework="pt",
     38     model=model_name,

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/convert_graph_to_onnx.py in convert(framework, model, output, opset, tokenizer, use_external_format, pipeline_name)
    365     # Export the graph
    366     if framework == "pt":
--> 367         convert_pytorch(nlp, opset, output, use_external_format)
    368     else:
    369         convert_tensorflow(nlp, opset, output)

~/anaconda3/envs/dev/lib/python3.8/site-packages/transformers/convert_graph_to_onnx.py in convert_pytorch(nlp, opset, output, use_external_format)
    277         ordered_input_names, model_args = ensure_valid_input(nlp.model, tokens, input_names)
    278
--> 279         export(
    280             nlp.model,
    281             model_args,

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/__init__.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
    223
    224     from torch.onnx import utils
--> 225     return utils.export(model, args, f, export_params, verbose, training,
    226                         input_names, output_names, aten, export_raw_ir,
    227                         operator_export_type, opset_version, _retain_param_name,

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/utils.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
     83         else:
     84             operator_export_type = OperatorExportTypes.ONNX
---> 85     _export(model, args, f, export_params, verbose, training, input_names, output_names,
     86             operator_export_type=operator_export_type, opset_version=opset_version,
     87             _retain_param_name=_retain_param_name, do_constant_folding=do_constant_folding,

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, opset_version, _retain_param_name, do_constant_folding, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, fixed_batch_size, custom_opsets, add_node_names, enable_onnx_checker, use_external_data_format, onnx_shape_inference, use_new_jit_passes)
    627             if dynamic_axes is None:
    628                 dynamic_axes = {}
--> 629             _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
    630
    631             graph, params_dict, torch_out = \

~/anaconda3/envs/dev/lib/python3.8/site-packages/torch/onnx/utils.py in _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
   1115             for i, x in enumerate(value):
   1116                 if not isinstance(x, int):
-> 1117                     raise ValueError("The type of axis index is expected to be an integer")
   1118                 if x in value_dict:
   1119                     warnings.warn('Duplicate dynamic axis index {} was provided for input {}.'

ValueError: The type of axis index is expected to be an integer
```

## Expected behavior

1 & 2 & 4 point into the direction, that something is wrong with inferring the dynamic shapes, if I am right. 3 just popped up while I was testing the other pipelines.

In all cases, the export & usage should work properly.


[trainer] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters

When running DDP:
```
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node 2 \
run_clm.py --model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
```
I get:

> [W reducer.cpp:1050] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator())

but it's not possible to turn it off from the trainer. i.e. it's hardwired.

@sgugger


[trainer] a consistent way to limit the number of items

# 🚀 Feature request

We have:

1.  `finetune_trainer.py` has
```
    n_train: Optional[int] = field(default=-1, metadata={"help": "# training examples. -1 means use all."})
    n_val: Optional[int] = field(default=-1, metadata={"help": "# validation examples. -1 means use all."})
    n_test: Optional[int] = field(default=-1, metadata={"help": "# test examples. -1 means use all."})
```
2. some other `run_` scripts use `--n_obs`
3. `--max_steps` in the main trainer - which works only on the train_dataset - no ability to limit items on eval_dataset

Requests/Questions:

1. How does one use `--max_steps` if one needs to use a different number of items for train and eval?
2. Can we have a consistent way across examples to do this same thing?

Thank you.

@sgugger


[traner] fix --lr_scheduler_type choices

This PR fixes:
```
$ python ./run_clm.py -h | grep lr_scheduler_type
                  [--lr_scheduler_type {SchedulerType.LINEAR,SchedulerType.COSINE,SchedulerType.COSINE_WITH_RESTARTS,SchedulerType.POLYNOMIAL,SchedulerType.CONSTANT,SchedulerType.CONSTANT_WITH_WARMUP}]
```
to:
```
                  [--lr_scheduler_type {linear,cosine,cosine_with_restarts,polynomial,constant,constant_with_warmup}]
```

I'm not sure what the original intention was since the current suggestions do not work:
```
run_clm.py: error: argument --lr_scheduler_type: invalid SchedulerType value: 'SchedulerType.LINEAR'
```

I couldn't find any readily-available methods to do the same in the `enum` superclass: https://docs.python.org/3/library/enum.html

@sgugger


Authorize last version of tokenizer

# What does this PR do?

This PR bumps the version pinned in the setup to authorize the last version of tokenziers (which in particular contains fixes for the `run_qa` script).


Smdistributed trainer

# What does this PR do?

This PR adds support for the variant of `torch.distributed` developed by AWS on SageMaker (`smdistributed`). It's been tested to work on the `run_glue` example.

The main steps are:
- to replace all operations from the `torch.distributed` module by their equivalent in `smdistributed.torch.distributed`.
- use their wrapper for the model instead of `DistributedDataParallel`.


Conversion of Electra checkpoint from official repo TF (pretrained on custom dataset)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: latest from pip install git+https://github.com/huggingface/transformers.git
- Platform: Colab
- Python version:
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?): 1.15
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik if I am not wrong

 -->

## Information

Model I am using (Bert, XLNet ...):
Electra from google repo
The problem arises when using:
* [ x] the official example scripts: (give details below) : Yes

The tasks I am working on is:
* conversion of Electra tf checkpoint train from the official repository on a custom dataset to pytorch

## To reproduce

Steps to reproduce the behavior:

1. install latest version of huggingface : pip install git+https://github.com/huggingface/transformers.git
2. train on official repo few steps Electra on a dataset
3. use the script to convert electra checkpoint :  transformers/src/transformers/models/electra/convert_electra_original_tf_checkpoint_to_pytorch.py

```
Initialize PyTorch weight ['discriminator_predictions', 'dense', 'bias'] discriminator_predictions/dense/bias
Skipping discriminator_predictions/dense/bias/adam_m ['discriminator_predictions', 'dense', 'bias', 'adam_m'] 'Parameter' object has no attribute 'adam_m'
Skipping discriminator_predictions/dense/bias/adam_v ['discriminator_predictions', 'dense', 'bias', 'adam_v'] 'Parameter' object has no attribute 'adam_v'
Initialize PyTorch weight ['discriminator_predictions', 'dense', 'kernel'] discriminator_predictions/dense/kernel
Skipping discriminator_predictions/dense/kernel/adam_m ['discriminator_predictions', 'dense', 'kernel', 'adam_m'] 'Parameter' object has no attribute 'adam_m'
Skipping discriminator_predictions/dense/kernel/adam_v ['discriminator_predictions', 'dense', 'kernel', 'adam_v'] 'Parameter' object has no attribute 'adam_v'
Initialize PyTorch weight ['discriminator_predictions', 'dense_prediction', 'bias'] discriminator_predictions/dense_1/bias
Skipping discriminator_predictions/dense_1/bias/adam_m ['discriminator_predictions', 'dense_prediction', 'bias', 'adam_m'] 'Parameter' object has no attribute 'adam_m'
Skipping discriminator_predictions/dense_1/bias/adam_v ['discriminator_predictions', 'dense_prediction', 'bias', 'adam_v'] 'Parameter' object has no attribute 'adam_v'
Initialize PyTorch weight ['discriminator_predictions', 'dense_prediction', 'kernel'] discriminator_predictions/dense_1/kernel
Skipping discriminator_predictions/dense_1/kernel/adam_m ['discriminator_predictions', 'dense_prediction', 'kernel', 'adam_m'] 'Parameter' object has no attribute 'adam_m'
Skipping discriminator_predictions/dense_1/kernel/adam_v ['discriminator_predictions', 'dense_prediction', 'kernel', 'adam_v'] 'Parameter' object has no attribute 'adam_v'
Initialize PyTorch weight ['electra', 'embeddings', 'LayerNorm', 'beta'] electra/embeddings/LayerNorm/beta
Skipping electra/embeddings/LayerNorm/beta/adam_m ['electra', 'embeddings', 'LayerNorm', 'beta', 'adam_m'] 'Parameter' object has no attribute 'adam_m'
Skipping electra/embeddings/LayerNorm/beta/adam_v ['electra', 'embeddings', 'LayerNorm', 'beta', 'adam_v'] 'Parameter' object has no attribute 'adam_v'
Initialize PyTorch weight ['electra', 'embeddings', 'LayerNorm', 'gamma'] electra/embeddings/LayerNorm/gamma
Skipping electra/embeddings/LayerNorm/gamma/adam_m ['electra', 'embeddings', 'LayerNorm', 'gamma', 'adam_m'] 'Parameter' object has no attribute 'adam_m'
Skipping electra/embeddings/LayerNorm/gamma/adam_v ['electra', 'embeddings', 'LayerNorm', 'gamma', 'adam_v'] 'Parameter' object has no attribute 'adam_v'
Initialize PyTorch weight ['electra', 'embeddings', 'position_embeddings'] electra/embeddings/position_embeddings
Skipping electra/embeddings/position_embeddings/adam_m ['electra', 'embeddings', 'position_embeddings', 'adam_m'] 'Embedding' object has no attribute 'adam_m'
Skipping electra/embeddings/position_embeddings/adam_v ['electra', 'embeddings', 'position_embeddings', 'adam_v'] 'Embedding' object has no attribute 'adam_v'
Initialize PyTorch weight ['electra', 'embeddings', 'token_type_embeddings'] electra/embeddings/token_type_embeddings
Skipping electra/embeddings/token_type_embeddings/adam_m ['electra', 'embeddings', 'token_type_embeddings', 'adam_m'] 'Embedding' object has no attribute 'adam_m'
Skipping electra/embeddings/token_type_embeddings/adam_v ['electra', 'embeddings', 'token_type_embeddings', 'adam_v'] 'Embedding' object has no attribute 'adam_v'
Initialize PyTorch weight ['electra', 'embeddings', 'word_embeddings'] electra/embeddings/word_embeddings
Skipping electra/embeddings/word_embeddings/adam_m ['electra', 'embeddings', 'word_embeddings', 'adam_m'] 'Embedding' object has no attribute 'adam_m'
Skipping electra/embeddings/word_embeddings/adam_v ['electra', 'embeddings', 'word_embeddings', 'adam_v'] 'Embedding' object has no attribute 'adam_v'
Initialize PyTorch weight ['electra', 'embeddings_project', 'bias'] electra/embeddings_project/bias
Skipping electra/embeddings_project/bias/adam_m ['electra', 'embeddings_project', 'bias', 'adam_m'] 'Parameter' object has no attribute 'adam_m'
Skipping electra/embeddings_project/bias/adam_v ['electra', 'embeddings_project', 'bias', 'adam_v'] 'Parameter' object has no attribute 'adam_v'
Initialize PyTorch weight ['electra', 'embeddings_project', 'kernel'] electra/embeddings_project/kernel
```

Is it a normal message ? I remembered few months ago these message were not present.


Improve pytorch examples for fp16

# What does this PR do?

When fp16 is True in pytorch training examples, use padding to multiple of 8, if the currently used data collator allows, to speed up training. If no collator was used, add one using the padding option.
Fixes #9752

## Who can review?

 Trainer: @sgugger


does LED use distributed training by default?

Hello, I'm currently fine tuning the `allenai/led-base-16384` model and following allow this [notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing#scrollTo=jpUr9QeebZ-n) .The node that I'm using has a couple of V100-SXM2-16GB GPUs in it. That said, does `Seq2SeqTrainer` automatically use distributed training by default?
I noticed it says
```
the inner model is wrapped in ``DeepSpeed`` and then again in ``torch.nn.DistributedDataParallel``.
```

but I just wanted to triple check that the training job would distributed across GPUs. I'd greatly appreciate the feedback


[Flaky Generation Tests] Make sure that no early stopping is happening for beam search

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

The PR fixes the flaky ci which is probably due to early stopping in beam search with the top `num_return_sequences` beams being smaller than the
longest `num_beams` beam.

This PR should (hopefully) fix flaky CI failures, such as:
- https://app.circleci.com/pipelines/github/huggingface/transformers/18912/workflows/70862bd9-bc94-4f2d-9b07-b85be146c867/jobs/155831
- https://app.circleci.com/pipelines/github/huggingface/transformers/18910/workflows/284df8e3-373b-4168-bc3b-f7079c5aa17d/jobs/155797

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Add the ability to skip runtime version check

# 🚀 Feature request

Add the ability to skip version check / raise a warning instead of an error if the checks failed.

## Motivation

Currently, Transformers perform a runtime libraries version check and raise an error if some of the requirements has a version that is different from the specified.

While version check is a reasonable thing to do, an error seems like an overkill. As [I mentioned](https://github.com/huggingface/transformers/pull/8073#issuecomment-765632175) in #8073, this, for example, prevents from using the latest version of Tokenizers without modifying the source code.

## Your contribution

I suggest creating an environment variable (e.g. `TRANSFORMERS_VERSION_CHECK_STRICT`) that would control the reaction to a version mismatch. If it is set to `true`, the current behavior remains. If `false`, we log a warning instead of raising an error. Personally, I think that having this variable `false` by default will make it easier for the users to work in such cases.

What's your opinion in that? Probably we can find an even better solution together.


Strange start token in MT5 generation

## Environment info

- `transformers` version: 4.2.2
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?):1.7.1

### Who can help
 Text Generation: @patrickvonplaten @TevenLeScao
 T5: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): MT5

## To reproduce

Steps to reproduce the behavior:
```python
from transformers import MT5ForConditionalGeneration, MT5Tokenizer

model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small")
tokenizer = MT5Tokenizer.from_pretrained("google/mt5-small")

text = 'summarize: Bidirectional Encoder Representations from Transformers is a Transformer-based machine learning technique for natural language processing pre-training developed by Google'
inputs = tokenizer([text], max_length=512, truncation=True, return_tensors='pt')
summary_ids = model.generate(inputs['input_ids'])
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in summary_ids])
```
The output I got is ['<extra_id_0>.']
## Expected behavior
I tried a few input texts. The generated output always start with <extra_id_0>, which doesn't happen in t5 generation. Anyone knows how to solve it?


Fix broken links in the converting tf ckpt document

# What does this PR do?

This PR is to fix broken links in ["Converting TensorFlow Checkpoints"](https://huggingface.co/transformers/converting_tensorflow_models.html).

Advised by @LysandreJik in issue #9656, I updated the links.
I also refer to issue #8720 and it seems the issue is solved by this PR.

I think there are some outdated explanations.
As I discussed in issue #9657, I think it should be better to explain `from_pretrained()` instead of `torch.save()`.
Hence, I think the explanation below should be updated.

```
You can then disregard the TensorFlow
checkpoint (the three files starting with ``bert_model.ckpt``\ ) but be sure to keep the configuration file (\
``bert_config.json``\ ) and the vocabulary file (\ ``vocab.txt``\ ) as these are needed for the PyTorch model too.
```

I would be happy to get some advice on how to proceed with this PR.

Fixes #9656
Fixes #8720

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

 albert, bert, XLM: @LysandreJik


RagTokenForGeneration: Fixed parameter name for logits_processor

# What does this PR do?
The parameter name for the beam_search and greedy_search functions of the GenerationMixin is (now?) logits_processor not pre_processor.  This fix makes prefix_allowed_tokens_fn work (again?).

## Who can review?

 Rag: @patrickvonplaten, @lhoestq


Allow RAG to output decoder cross-attentions

# What does this PR do?

This PR makes RAG output the generator model's decoder cross-attentions when `output_attentions=True`.

Motivation and context: before this PR, RAG's output objects had attributes for the generator's encoder self-attentions and decoder self-attentions, but no option for the encoder-decoder cross-attentions. So this simply allows cross-attentions to be extracted, as well as fixing a small bug where `output_attentions` wasn't being passed into the generator.

Fixes #9468

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Yes - #9468
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? - I don't believe any new tests are necessary. Existing tests pass.

## Who can review?
@patrickvonplaten, @lhoestq


Clean TF Bert

# What does this PR do?

This PR aims to clean the code base of BERT and the other models that depends of it because of the `#Copied from...`. It also clean the template accordingly to the same changes applied in BERT.

The other models will receive the same type of cleaning, but each model will have its own PR.


Fix model parallel definition in superclass

The `model_parallel` attribute should be obtainable from every class instance that has the `is_parallelized` class attribute. Otherwise the following line in the trainer crashes:

https://github.com/huggingface/transformers/blob/626116b7d76efef5137c3b4a92e64e3bb57a6882/src/transformers/trainer.py#L244

cc @stas00 @alexorona