error: fine-tunes language model with added_tokens

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: version: 4.3.3
- Platform: Linux-4.15.0-29-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@LysandreJik, @LysandreJik

## Information

I fine-tune BERT on my own social media data, I do this follow the instruction in the `examples/language-modeling/README.md`. **I follow the official run_mlm.py file, and the only change is that I add some new tokens after the tokenizer inits, and then I got the Cuda runtime error. ** If I don't add some new tokens, it works well.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
Only add one line in `examples/language-modeling/run_mlm.py`

start from run_mlm.py L291:
https://github.com/huggingface/transformers/blob/838f83d84ccf57f968e0ace7f400e43b92643552/examples/language-modeling/run_mlm.py#L291

```Python
...
    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script."
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )
 # only add this line!
    tokenizer.add_tokens(['[awsl]', '[happy]', '[doge]', ... , '[cry]'])
...
```

running log
```
[INFO|configuration_utils.py:485] 2021-04-01 10:49:28,166 >> Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "directionality": "bidi",
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "position_embedding_type": "absolute",
  "transformers_version": "4.3.3",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 21128
}

[INFO|modeling_utils.py:1025] 2021-04-01 10:49:28,167 >> loading weights file /data/huyong/code/socialbert/pretrained_models/roberta/pytorch_model.bin
[WARNING|modeling_utils.py:1135] 2021-04-01 10:49:31,389 >> Some weights of the model checkpoint at /data/huyong/code/socialbert/pretrained_models/roberta were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[INFO|modeling_utils.py:1152] 2021-04-01 10:49:31,389 >> All the weights of BertForMaskedLM were initialized from the model checkpoint at /data/huyong/code/socialbert/pretrained_models/roberta.
If your task is similar to the task the model of the checkpoint was trained on, you can already use BertForMaskedLM for predictions without further training.
[INFO|trainer.py:837] 2021-04-01 10:49:31,469 >> ***** Running training *****
[INFO|trainer.py:838] 2021-04-01 10:49:31,469 >>   Num examples = 100000
[INFO|trainer.py:839] 2021-04-01 10:49:31,469 >>   Num Epochs = 3
[INFO|trainer.py:840] 2021-04-01 10:49:31,469 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:841] 2021-04-01 10:49:31,469 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:842] 2021-04-01 10:49:31,469 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:843] 2021-04-01 10:49:31,469 >>   Total optimization steps = 37500
 0%|          | 0/37500 [00:00<?, ?it/s]
  0%|          | 1/37500 [00:00<1:48:39,  5.75it/s]/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [485,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [488,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [488,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [488,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [488,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [488,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [488,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [488,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "mlm.py", line 537, in <module>
    main()
  File "mlm.py", line 503, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/trainer.py", line 940, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/trainer.py", line 1304, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/trainer.py", line 1334, in compute_loss
    outputs = model(**inputs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 1315, in forward
    return_dict=return_dict,
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 976, in forward
    return_dict=return_dict,
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 574, in forward
    output_attentions,
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 496, in forward
    self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1787, in apply_chunking_to_forward
    return forward_fn(*input_tensors)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 507, in feed_forward_chunk
    intermediate_output = self.intermediate(attention_output)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 410, in forward
    hidden_states = self.dense(hidden_states)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/huyong/miniconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I found in the `run_mlm.py` , it has `model.resize_token_embeddings(len(tokenizer))`, Why still get the error ? Thanks
<!-- A clear and concise description of what you would expect to happen. -->


about .py file

i can't download the file" convert_tf_checkpoint_to_pytorch.py" and three other .py file that you set a hyperlink on it. if i click on the link it notice me: 404. Where can i get them? thank you!


"Converting Tensorflow Checkpoints" meets  ('Pointer shape torch.Size([312]) and array shape (128,) mismatched', torch.Size([312]), (128,))

when "Converting Tensorflow Checkpoints", I see this "Pointer shape {pointer.shape} and array shape {array.shape} mismatched"
AssertionError: ('Pointer shape torch.Size([312]) and array shape (128,) mismatched', torch.Size([312]), (128,)), and the pretrainmodel comes from https://github.com/ZhuiyiTechnology/pretrained-models


ReduceLROnPlateau-like functionality?

Feature request

Request for a new feature

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

**Description:** Dynamic learning rate reduction upon metric saturation, as in `torch.optim.lr_scheduler.ReduceLROnPlateau`, integrated into the `Trainer` API.

Alternately, if there's any way (if hacky) to get dynamic learning rate reduction using the `Trainer` API as it is, that would be extremely helpful as well.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

LR schedules are a commonly used trick for ML optimization, and the `transformers` library already provides a significant number of baseline schedules (i.e. linear, cosine schedulers, warmup/no-warmup, restarts). However, these schedules are all static: updates to them occur at fixed steps in the optimization -- one can always tell what the learning rate at, say, step 1000 will be given these fixed schedules.

Reducing learning rate dynamically is also a common practical technique, usually applied when loss saturates (fails to improve after N iterations).

The difficulty is that dynamic learning rate reduction follows a non-fixed update schedule, meaning that working within the `LambdaLR` framework used by the other scheduler is less straightforward.

## Your contribution

I don't have a working implementation yet. At a high level, I tried to implement this myself as a `TrainerCallback` modeled on both the `EarlyStoppingCallback` in the `transformers` library as well as the `ReduceLROnPlateau` implementation in PyTorch. I was able to modify the optimizer object; however, learning rate updates to the optimizer would get overwritten by the scheduler. In any case, I also don't know if it's good style/even possible to modify the optimizer and scheduler in this way using a Callback -- seems like the `control` object is the only thing that changes within a `TrainerCallback`.

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Getting `raise NotImplementedError` for base_model.get_input_embeddings() when upgrading from pytorch-transformers

# 📚 Migration

## Information

<!-- Important information -->
Getting `raise NotImplementedError` at
 the lines - https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L474-L477 when I am trying to upgrade my code from pytorch-transformers to transformers

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below): Not sure
* [ ] my own modified scripts: (give details below): Yes

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) No
* [ ] my own task or dataset: (give details below): No

## Details

<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->
I am using Oscar repo (https://github.com/microsoft/Oscar), which uses an older version of Huggingface pytorch-transformers (https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e). I am trying to upgrade the repo to use latest version of transformers (https://github.com/huggingface/transformers). However, I am getting below error :
```
Traceback (most recent call last):
  File "oscar/run_captioning_airsplay.py", line 1019, in <module>
    main()
  File "oscar/run_captioning_airsplay.py", line 966, in main
    from_tf=bool('.ckpt' in args.model_name_or_path), config=config)
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 1188, in from_pretrained
    model.tie_weights()
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 504, in tie_weights
    self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 469, in get_input_embeddings
    return base_model.get_input_embeddings()
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 471, in get_input_embeddings
    raise NotImplementedError
NotImplementedError
```
The error occurs at this block in the transformers code - https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L474-L477. My code runs fine when I use an older version of hugging face transformers - https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e, possibly, because pytorch-transformers did not have a requirement that `set_input_embeddings()` should be defined for base_model. The base model that I am using is a custom defined model `BertForImageCaptioning` (https://github.com/microsoft/Oscar/blob/df79152b708c3c46f2dc93324776a27406ccc634/oscar/modeling/modeling_bert.py#L604), which has a custom defined parent class ` CaptionPreTrainedModel` (https://github.com/microsoft/Oscar/blob/df79152b708c3c46f2dc93324776a27406ccc634/oscar/modeling/modeling_utils.py#L21), which has a parent class `BertPreTrainedModel`.

I have not seen any mention of how to deal with this issue in the migration documents from Pytorch-transformers or from transformers 3.x. (https://huggingface.co/transformers/migration.html#migrating-from-transformers-v3-x-to-v4-x).

I have looked into examples to check how to define the function, but this did not give enough details to define the function at my side - https://github.com/huggingface/transformers/blob/master/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py#L487-L488.

How should I define the function `get_input_embeddings()` for my use case and what are the guidelines for doing the same. Are there any examples explaining the process of defining the function.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: https://github.com/huggingface/transformers
- Platform: x86_64 GNU/Linux
- Python version: 3.6.8
- PyTorch version (GPU?):  1.7.0+cu101 (GPU)
- Tensorflow version (GPU?): 2.3.0 (GPU)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch): https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e

## Checklist

- [ Yes] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ yes] I checked if a related official extension example runs on my machine.


conda install transformers (not working) behaving differently from pip install transformers (working) for CentOS 7.9

A fresh environment where I `conda install pytorch torchvision torchaudio -c pytorch` then `conda install transformers` produces a glibc2.18 error on CentOS 7.9 upon import with `python -c "from transformers import AutoTokenizer"`. I suspect this is a similar error to #2980, i.e., CentOS 7.9 might just be incompatible. However, a different fresh environment where I `pip install torch torchvision torchaudio` then `pip install transformers` does not produce any error upon import with `python -c "from transformers import AutoTokenizer"`.

## Environment info (pip-installed)
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-4.19.182-1.el7.retpoline.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: <fill in>

## Environment info (conda-installed)
In fact, this command doesn't even work. See attached `cli_error_trace.txt`.

### Who can help

I'm not sure if I did this right since this seems to be more of a lower-level issue than implementation issue.

-huggingface/transformers/blob/master/src/transformers/models/auto/tokenization_auto.py @LysandreJik

## Information

Model I am using (Bert, XLNet ...):

N/A

## To reproduce

This is all done on CentOS 7.9.

##### Steps to reproduce the good, pip-installed behavior:

1. conda create --name test python=3.8
2. conda activate test
3. pip install torch torchvision torchaudio
4. pip install transformers
5. python -c "from transformers import AutoTokenizer"

##### Steps to reproduce the bad, conda-installed behavior:
1. conda create --name test2 python=3.8
2. conda activate test2
3. conda install pytorch torchvision torchaudio -c pytorch
4. conda install -c huggingface transformers
5. python -c "from transformers import AutoTokenizer"

Additionally, I have attached the `environment.yml` files for both environments and also the trace for the `transformers-cli env` command and the trace for the import error (both for the `conda install`-ed environment). The traces look pretty similar, and it seems the issue is with the dependencies of tokenizers. The .yml files have an appended .txt extension since apparently GitHub doesn't support the .yml extension for uploaded files.

[environment_pip.yml.txt](https://github.com/huggingface/transformers/files/6239328/environment_pip.yml.txt)
[environment_conda.yml.txt](https://github.com/huggingface/transformers/files/6239327/environment_conda.yml.txt)
[cli_error_trace.txt](https://github.com/huggingface/transformers/files/6239326/cli_error_trace.txt)
[import_error_trace.txt](https://github.com/huggingface/transformers/files/6239329/import_error_trace.txt)

## Expected behavior

I would expect `conda install`-ing and `pip install`-ing to both work as intended.


KeyError: 'gpt_neo' with EleutherAI/gpt-neo-1.3B

I am trying to try new GPT-3 checkpoint however I am getting an error on local and Google Colab platform as well.

Google Colab:
```
!pip install transformers

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")
model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")
```

The error I get is:

```
KeyError                                  Traceback (most recent call last)

<ipython-input-5-333740565b3a> in <module>()
      3 from transformers import AutoTokenizer, AutoModelForCausalLM
      4
----> 5 tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")
      6 model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")

1 frames

/usr/local/lib/python3.7/dist-packages/transformers/models/auto/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    387         config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
    388         if "model_type" in config_dict:
--> 389             config_class = CONFIG_MAPPING[config_dict["model_type"]]
    390             return config_class.from_dict(config_dict, **kwargs)
    391         else:

KeyError: 'gpt_neo'
```


Add `examples/language_modeling/run_mlm_no_trainer.py`

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds an example of finetuning a Masked Language Model (without using `Trainer`) to show the functionalities of the new accelerate library.

## Who can review?

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


In the group by length documentation length is misspelled as legnth

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


ROUGE Multiple References

It appears the current ROUGE metric computes the score with 1 reference per candidate. I was wondering if there is way to compute ROUGE with multiple references per candidate? Thanks


Get following error with EncoderDecoder model: TypeError: forward() got an unexpected keyword argument 'use_cache'

Hi

I am trying to create an EncoderDecoder model where i want to use a pre-trained encoder model and initialise decoder from scratch.  Following the code snippet.

----------------------------------------------------------
encoder = AutoModel.from_pretrained('bert-base-uncased')
decoder_config = BertConfig(vocab_size = vocabsize,
                    max_position_embeddings = max_length,
                    num_attention_heads = num_attention_heads,
                    num_hidden_layers = num_hidden_layers,
                    hidden_size = hidden_size,
                    type_vocab_size = 1,
                    is_decoder=True)

decoder = BertForMaskedLM(config=decoder_config)
model = EncoderDecoderModel(encoder=encoder, decoder=decoder)
----------------------------------------------------------
Model gets built without any errors but when i try to make a forward pass, i get the error:
TypeError: forward() got an unexpected keyword argument 'use_cache'.

Following is the dummy forward pass function

outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, labels=input_ids)


[Docs] Add blog to BigBird docs

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


GPT Neo, Print Most Probable Next Word: String Indices Must Be Integers

This code is supposed to generate the next most probable word. However, the following problem arises.

```
!pip install git+https://github.com/huggingface/transformers.git
import torch
from transformers import GPTNeoForCausalLM, AutoTokenizer
model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")

prompt = """In the"""
prompt = prompt.strip()
text = tokenizer.encode(prompt)
myinput, past = torch.tensor([text]), None
logits, past = model(myinput, past_key_values = past)
logits = logits[0,-1]
probabilities = torch.nn.functional.softmax(logits)
best_logits, best_indices = logits.topk(10)
best_words = [tokenizer.decode([idx.item()]) for idx in best_indices]
text.append(best_indices[0].item())
best_probabilities = probabilities[best_indices].tolist()
words = []
for i in range(10):
    m = (best_words[i])
    print(m)
```

`TypeError: string indices must be integers`


[Notebook] add BigBird trivia qa notebook

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Fix the checkpoint for I-BERT

The I-BERT checkpoint was not configured correctly in the `_CHECKPOINT_FOR_DOC`

Fixes https://github.com/huggingface/transformers/issues/10990


[GPT Neo] fix example in config

# What does this PR do?

Fix example in doc

Thanks a lot for spotting this @NielsRogge

cc @LysandreJik


GPT Neo configuration needs to be set to use GPT2 tokenizer

The tokenizer wasn't correctly set and ended up making ~200 slow tests fail. The run in question is here: https://github.com/huggingface/transformers/runs/2232656252?check_suite_focus=true

This PR fixes that!


[WIP] Add BigBirdPegasus

# What does this PR do?
This PR will add Google's BigBird-Pegasus. Extending #10183

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger
HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@patrickvonplaten


Can't find ibert-roberta-base model

## Environment info
- `transformers` version: 4.4.2
- Platform: Linux-4.15.0-1109-azure-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.13
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

Models:
- ibert: @kssteven418

Documentation:
- @sgugger

## Information

Model I am using I-BERT:

The problem arises when using:
* [X] the official example scripts: (give details below)

## To reproduce
From [documentation](https://huggingface.co/transformers/model_doc/ibert.html):

```python
from transformers import RobertaTokenizer, IBertModel
import torch
tokenizer = RobertaTokenizer.from_pretrained('ibert-roberta-base')
model = IBertModel.from_pretrained('ibert-roberta-base')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state
```

Steps to reproduce the behavior:

1. install pytorch and transformers
2. run the code example from docs

```python
Traceback (most recent call last):
  File "/home/sotmazgi/PycharmProjects/s2e-coref/ibert_test.py", line 3, in <module>
    tokenizer = RobertaTokenizer.from_pretrained('ibert-roberta-base')
  File "/home/sotmazgi/PycharmProjects/s2e-coref/venv/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1693, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load tokenizer for 'ibert-roberta-base'. Make sure that:

- 'ibert-roberta-base' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'ibert-roberta-base' is the correct path to a directory containing relevant tokenizer files
```

## Expected behavior
get the embedding for the example sentence


Fixed some typos and removed legacy url

# What does this PR do?

Removed legacy url to colab notebook in examples/multiple-choice/README.md
Fixed some typos.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger
@patil-suraj


unable to use multiple GPUs with HF integration of DeepSpeed on Jupyter notebooks

Hi ,

I'm using HF integration of DeepSpeed in my Jupyter Notebook by setting following env variables as suggested [here](https://huggingface.co/transformers/main_classes/trainer.html#deployment-in-notebooks)

```
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '9889'
os.environ['RANK'] = "0"
os.environ['LOCAL_RANK'] = "0"
os.environ['WORLD_SIZE'] = "1"
os.environ['NCCL_SOCKET_IFNAME'] = 'lo'   ##because of my kubeflow setup
```
with this setup I'm unable to utilize both GPU's that I have and here's the log info before it starts training-
```
[2021-03-31 09:49:02,510] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+7fcc891, git-hash=7fcc891, git-branch=master
[2021-03-31 09:49:02,540] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1
[2021-03-31 09:49:05,758] [INFO] [engine.py:608:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2021-03-31 09:49:05,760] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2021-03-31 09:49:05,761] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
[2021-03-31 09:49:05,764] [INFO] [stage2.py:130:__init__] Reduce bucket size 150000000.0
[2021-03-31 09:49:05,765] [INFO] [stage2.py:131:__init__] Allgather bucket size 150000000.0
[2021-03-31 09:49:05,766] [INFO] [stage2.py:132:__init__] CPU Offload: True
[2021-03-31 09:49:12,585] [INFO] [stage2.py:399:__init__] optimizer state initialized
[2021-03-31 09:49:12,588] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2021-03-31 09:49:12,589] [INFO] [engine.py:445:_configure_lr_scheduler] DeepSpeed using configured LR scheduler = WarmupLR
[2021-03-31 09:49:12,590] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7fef53ed1d68>
[2021-03-31 09:49:12,591] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[3e-05], mom=[[0.8, 0.999]]
[2021-03-31 09:49:12,593] [INFO] [config.py:737:print] DeepSpeedEngine configuration:
[2021-03-31 09:49:12,594] [INFO] [config.py:741:print]   activation_checkpointing_config  {
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "partition_activations": false,
    "profile": false,
    "synchronize_checkpoint_boundary": false
}
[2021-03-31 09:49:12,594] [INFO] [config.py:741:print]   allreduce_always_fp32 ........ False
[2021-03-31 09:49:12,595] [INFO] [config.py:741:print]   amp_enabled .................. False
[2021-03-31 09:49:12,596] [INFO] [config.py:741:print]   amp_params ................... False
[2021-03-31 09:49:12,596] [INFO] [config.py:741:print]   checkpoint_tag_validation_enabled  True
[2021-03-31 09:49:12,597] [INFO] [config.py:741:print]   checkpoint_tag_validation_fail  False
[2021-03-31 09:49:12,600] [INFO] [config.py:741:print]   disable_allgather ............ False
[2021-03-31 09:49:12,601] [INFO] [config.py:741:print]   dump_state ................... False
[2021-03-31 09:49:12,602] [INFO] [config.py:741:print]   dynamic_loss_scale_args ...... {'init_scale': 4294967296, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
[2021-03-31 09:49:12,603] [INFO] [config.py:741:print]   elasticity_enabled ........... False
[2021-03-31 09:49:12,603] [INFO] [config.py:741:print]   flops_profiler_config ........ {
    "detailed": true,
    "enabled": false,
    "module_depth": -1,
    "profile_step": 1,
    "top_modules": 3
}
[2021-03-31 09:49:12,604] [INFO] [config.py:741:print]   fp16_enabled ................. True
[2021-03-31 09:49:12,605] [INFO] [config.py:741:print]   global_rank .................. 0
[2021-03-31 09:49:12,605] [INFO] [config.py:741:print]   gradient_accumulation_steps .. 1
[2021-03-31 09:49:12,606] [INFO] [config.py:741:print]   gradient_clipping ............ 1.0
[2021-03-31 09:49:12,607] [INFO] [config.py:741:print]   gradient_predivide_factor .... 1.0
[2021-03-31 09:49:12,607] [INFO] [config.py:741:print]   initial_dynamic_scale ........ 4294967296
[2021-03-31 09:49:12,608] [INFO] [config.py:741:print]   loss_scale ................... 0
[2021-03-31 09:49:12,609] [INFO] [config.py:741:print]   memory_breakdown ............. False
[2021-03-31 09:49:12,610] [INFO] [config.py:741:print]   optimizer_legacy_fusion ...... False
[2021-03-31 09:49:12,610] [INFO] [config.py:741:print]   optimizer_name ............... adamw
[2021-03-31 09:49:12,611] [INFO] [config.py:741:print]   optimizer_params ............. {'lr': 3e-05, 'betas': [0.8, 0.999], 'eps': 1e-08, 'weight_decay': 3e-07}
[2021-03-31 09:49:12,612] [INFO] [config.py:741:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-03-31 09:49:12,612] [INFO] [config.py:741:print]   pld_enabled .................. False
[2021-03-31 09:49:12,613] [INFO] [config.py:741:print]   pld_params ................... False
[2021-03-31 09:49:12,614] [INFO] [config.py:741:print]   prescale_gradients ........... False
[2021-03-31 09:49:12,616] [INFO] [config.py:741:print]   scheduler_name ............... WarmupLR
[2021-03-31 09:49:12,616] [INFO] [config.py:741:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 3e-05, 'warmup_num_steps': 500}
[2021-03-31 09:49:12,617] [INFO] [config.py:741:print]   sparse_attention ............. None
[2021-03-31 09:49:12,618] [INFO] [config.py:741:print]   sparse_gradients_enabled ..... False
[2021-03-31 09:49:12,618] [INFO] [config.py:741:print]   steps_per_print .............. 2000
[2021-03-31 09:49:12,619] [INFO] [config.py:741:print]   tensorboard_enabled .......... False
[2021-03-31 09:49:12,620] [INFO] [config.py:741:print]   tensorboard_job_name ......... DeepSpeedJobName
[2021-03-31 09:49:12,620] [INFO] [config.py:741:print]   tensorboard_output_path ......
[2021-03-31 09:49:12,621] [INFO] [config.py:741:print]   train_batch_size ............. 4
[2021-03-31 09:49:12,622] [INFO] [config.py:741:print]   train_micro_batch_size_per_gpu  4
[2021-03-31 09:49:12,622] [INFO] [config.py:741:print]   wall_clock_breakdown ......... False
[2021-03-31 09:49:12,623] [INFO] [config.py:741:print]   world_size ................... 1
[2021-03-31 09:49:12,624] [INFO] [config.py:741:print]   zero_allow_untested_optimizer  False
[2021-03-31 09:49:12,625] [INFO] [config.py:741:print]   zero_config .................. {
    "allgather_bucket_size": 150000000.0,
    "allgather_partitions": true,
    "contiguous_gradients": true,
    "cpu_offload": true,
    "cpu_offload_params": false,
    "cpu_offload_use_pin_memory": false,
    "elastic_checkpoint": true,
    "gather_fp16_weights_on_model_save": false,
    "load_from_fp32_weights": true,
    "max_live_parameters": 1000000000,
    "max_reuse_distance": 1000000000,
    "overlap_comm": true,
    "param_persistence_threshold": 100000,
    "prefetch_bucket_size": 50000000,
    "reduce_bucket_size": 150000000.0,
    "reduce_scatter": true,
    "stage": 2,
    "sub_group_size": 1000000000000
}
[2021-03-31 09:49:12,625] [INFO] [config.py:741:print]   zero_enabled ................. True
[2021-03-31 09:49:12,626] [INFO] [config.py:741:print]   zero_optimization_stage ...... 2
[2021-03-31 09:49:12,628] [INFO] [config.py:748:print]   json = {
    "fp16":{
        "enabled":true,
        "hysteresis":2,
        "loss_scale":0,
        "loss_scale_window":1000,
        "min_loss_scale":1
    },
    "gradient_accumulation_steps":1,
    "gradient_clipping":1.0,
    "optimizer":{
        "params":{
            "betas":[
                0.8,
                0.999
            ],
            "eps":1e-08,
            "lr":3e-05,
            "weight_decay":3e-07
        },
        "type":"AdamW"
    },
    "scheduler":{
        "params":{
            "warmup_max_lr":3e-05,
            "warmup_min_lr":0,
            "warmup_num_steps":500
        },
        "type":"WarmupLR"
    },
    "steps_per_print":2000,
    "train_micro_batch_size_per_gpu":4,
    "wall_clock_breakdown":false,
    "zero_optimization":{
        "allgather_bucket_size":150000000.0,
        "allgather_partitions":true,
        "contiguous_gradients":true,
        "cpu_offload":true,
        "overlap_comm":true,
        "reduce_bucket_size":150000000.0,
        "reduce_scatter":true,
        "stage":2
    }
}
```
But both GPUs were used if I convert my notebook to python script and run script with command - `!NCCL_SOCKET_IFNAME=lo deepspeed Deberta_V2_XXLarge.py --deepspeed ds_config.json`  and here's the log -
```
[2021-03-31 09:26:36,687] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2021-03-31 09:26:36,714] [INFO] [runner.py:358:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 Deberta_V2_XXLarge.py --deepspeed ds_config.json
[2021-03-31 09:26:38,357] [INFO] [launch.py:73:main] 0 NCCL_SOCKET_IFNAME lo
[2021-03-31 09:26:38,357] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2021-03-31 09:26:38,357] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=2, node_rank=0
[2021-03-31 09:26:38,357] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2021-03-31 09:26:38,358] [INFO] [launch.py:102:main] dist_world_size=2
[2021-03-31 09:26:38,358] [INFO] [launch.py:105:main] Setting CUDA_VISIBLE_DEVICES=0,1
2021-03-31 09:26:40.269001: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-03-31 09:26:40.269004: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
[2021-03-31 09:27:27,272] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
[2021-03-31 09:27:31,495] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+7fcc891, git-hash=7fcc891, git-branch=master
[2021-03-31 09:27:32,428] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
[2021-03-31 09:27:32,834] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
[2021-03-31 09:27:36,438] [INFO] [engine.py:608:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2021-03-31 09:27:36,438] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2021-03-31 09:27:36,439] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
[2021-03-31 09:27:36,441] [INFO] [stage2.py:130:__init__] Reduce bucket size 150000000.0
[2021-03-31 09:27:36,441] [INFO] [stage2.py:131:__init__] Allgather bucket size 150000000.0
[2021-03-31 09:27:36,441] [INFO] [stage2.py:132:__init__] CPU Offload: True
[2021-03-31 09:27:40,524] [INFO] [stage2.py:399:__init__] optimizer state initialized
[2021-03-31 09:27:40,530] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2021-03-31 09:27:40,533] [INFO] [engine.py:445:_configure_lr_scheduler] DeepSpeed using configured LR scheduler = WarmupLR
[2021-03-31 09:27:40,534] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7f105c6a0b00>
[2021-03-31 09:27:40,535] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[3e-05], mom=[[0.8, 0.999]]
[2021-03-31 09:27:40,536] [INFO] [config.py:737:print] DeepSpeedEngine configuration:
[2021-03-31 09:27:40,537] [INFO] [config.py:741:print]   activation_checkpointing_config  {
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "partition_activations": false,
    "profile": false,
    "synchronize_checkpoint_boundary": false
}
[2021-03-31 09:27:40,537] [INFO] [config.py:741:print]   allreduce_always_fp32 ........ False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   amp_enabled .................. False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   amp_params ................... False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   checkpoint_tag_validation_enabled  True
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   checkpoint_tag_validation_fail  False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   disable_allgather ............ False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   dump_state ................... False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   dynamic_loss_scale_args ...... {'init_scale': 4294967296, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
[2021-03-31 09:27:40,539] [INFO] [config.py:741:print]   elasticity_enabled ........... False
[2021-03-31 09:27:40,539] [INFO] [config.py:741:print]   flops_profiler_config ........ {
    "detailed": true,
    "enabled": false,
    "module_depth": -1,
    "profile_step": 1,
    "top_modules": 3
}
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   fp16_enabled ................. True
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   global_rank .................. 0
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_accumulation_steps .. 1
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_clipping ............ 1.0
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_predivide_factor .... 1.0
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   initial_dynamic_scale ........ 4294967296
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   loss_scale ................... 0
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   memory_breakdown ............. False
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_legacy_fusion ...... False
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_name ............... adamw
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_params ............. {'lr': 3e-05, 'betas': [0.8, 0.999], 'eps': 1e-08, 'weight_decay': 3e-07}
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pld_enabled .................. False
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pld_params ................... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   prescale_gradients ........... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   scheduler_name ............... WarmupLR
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 3e-05, 'warmup_num_steps': 500}
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   sparse_attention ............. None
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   sparse_gradients_enabled ..... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   steps_per_print .............. 2000
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   tensorboard_enabled .......... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   tensorboard_job_name ......... DeepSpeedJobName
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   tensorboard_output_path ......
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   train_batch_size ............. 8
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   train_micro_batch_size_per_gpu  4
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   wall_clock_breakdown ......... False
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   world_size ................... 2
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   zero_allow_untested_optimizer  False
[2021-03-31 09:27:40,544] [INFO] [config.py:741:print]   zero_config .................. {
    "allgather_bucket_size": 150000000.0,
    "allgather_partitions": true,
    "contiguous_gradients": true,
    "cpu_offload": true,
    "cpu_offload_params": false,
    "cpu_offload_use_pin_memory": false,
    "elastic_checkpoint": true,
    "gather_fp16_weights_on_model_save": false,
    "load_from_fp32_weights": true,
    "max_live_parameters": 1000000000,
    "max_reuse_distance": 1000000000,
    "overlap_comm": true,
    "param_persistence_threshold": 100000,
    "prefetch_bucket_size": 50000000,
    "reduce_bucket_size": 150000000.0,
    "reduce_scatter": true,
    "stage": 2,
    "sub_group_size": 1000000000000
}
[2021-03-31 09:27:40,544] [INFO] [config.py:741:print]   zero_enabled ................. True
[2021-03-31 09:27:40,545] [INFO] [config.py:741:print]   zero_optimization_stage ...... 2
[2021-03-31 09:27:40,546] [INFO] [config.py:748:print]   json = {
    "fp16":{
        "enabled":true,
        "hysteresis":2,
        "loss_scale":0,
        "loss_scale_window":1000,
        "min_loss_scale":1
    },
    "gradient_accumulation_steps":1,
    "gradient_clipping":1.0,
    "optimizer":{
        "params":{
            "betas":[
                0.8,
                0.999
            ],
            "eps":1e-08,
            "lr":3e-05,
            "weight_decay":3e-07
        },
        "type":"AdamW"
    },
    "scheduler":{
        "params":{
            "warmup_max_lr":3e-05,
            "warmup_min_lr":0,
            "warmup_num_steps":500
        },
        "type":"WarmupLR"
    },
    "steps_per_print":2000,
    "train_micro_batch_size_per_gpu":4,
    "wall_clock_breakdown":false,
    "zero_optimization":{
        "allgather_bucket_size":150000000.0,
        "allgather_partitions":true,
        "contiguous_gradients":true,
        "cpu_offload":true,
        "overlap_comm":true,
        "reduce_bucket_size":150000000.0,
        "reduce_scatter":true,
        "stage":2
    }
}
```

Plz suggest whether I need to make some changes..

Verions I'm using -
```
torch-1.7.1+cu101
transformers-4.4.2
deepspeed-0.3.13
```
**Who can help**
@LysandreJik
@stas00


Sagemaker test fix

# What does this PR do?

Fixed test documentation `makefile` command and PyTorch-ddp test when #10975 is merged. Different validation function for `sagemaker-data-parallel`. Can be merged already.


BART : Cannot run trainer.evaluate() after trainer.train()

## Environment info

- `transformers` version: 4.4.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help

trainer : @sgugger
bart : @patrickvonplaten

## Information

Model I am using : bart, barthez, mbart
I am working on text summarization.

The problem arises when using my own modified script, inspired by the official Seq2Seq example. I am using the Seq2SeqTrainer class.
I am unable to run trainer.evaluate(...) after trainer.train(...) as well as evaluating the model during training after x epochs or steps.

## To reproduce

I you want to try it out, here is a link to [my notebook](https://colab.research.google.com/drive/1CqxxM0nOdJRpre_SwLOajl_s9hlKTT9e?usp=sharing)

Steps to reproduce the behavior:

1. Download model, tokenizer, and dataset from hub
2. Run trainer.evaluate(...) (works)
3. Run trainer.train(...) (runs fine)
4. Run trainer.evaluate(...) (returns the error below)

```
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, head_mask, decoder_head_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   1160         elif return_dict and not isinstance(encoder_outputs, BaseModelOutput):
   1161             encoder_outputs = BaseModelOutput(
-> 1162                 last_hidden_state=encoder_outputs[0],
   1163                 hidden_states=encoder_outputs[1] if len(encoder_outputs) > 1 else None,
   1164                 attentions=encoder_outputs[2] if len(encoder_outputs) > 2 else None,

KeyError: 0
```


[WIP] GPT Neo cleanup


AttributeError due to multi-processing using PyTorchBenchmark

Hi there,
likely my fault, but I can't find a proper solution yet. Tried to follow the structure as in `examples/benchmarking/run_benchmark.py`.

My Code:
```python
from transformers import AutoConfig, PyTorchBenchmark, PyTorchBenchmarkArguments

def main():

    config = AutoConfig.from_pretrained('roberta-base')

    # define args
    args = PyTorchBenchmarkArguments(
        models=['roberta-base'],
        inference=False,
        training=True,
        speed=True,
        memory=True,
        save_to_csv=True,
        train_memory_csv_file=f'models/filmo-large/train_memory_benchmark.csv',
        train_time_csv_file=f'models/filmo-large/train_time_benchmark.csv',
        env_info_csv_file=f'models/filmo-large/env.csv',
        sequence_lengths=[64, 128, 256, 512],
        batch_sizes=[8, 16],
        fp16=True,
        multi_process=True,
    )

    # create benchmark
    benchmark = PyTorchBenchmark(
        configs=[config],
        args=args,
    )

    # run benchmark
    benchmark.run()

if __name__ == '__main__':
    main()
```

The error it yields:
```python
1 / 1
Traceback (most recent call last):
  File "c:/Users/.../lm-train-benchmark.py", line 47, in <module>
    main()
  File "c:/Users/.../lm-train-benchmark.py", line 43, in main
    benchmark.run()
  File "C:\Users\...\.venv\lib\site-packages\transformers\benchmark\benchmark_utils.py", line 715, in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
  File "C:\Users\...\.venv\lib\site-packages\transformers\benchmark\benchmark_utils.py", line 679, in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
  File "C:\Users\...\.venv\lib\site-packages\transformers\benchmark\benchmark_utils.py", line 101, in multi_process_func
    p.start()
  File "C:\Python\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Python\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Python\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
    reduction.dump(process_obj, to_child)
  File "C:\Python\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'separate_process_wrapper_fn.<locals>.multi_process_func.<locals>.wrapper_func'
PS C:\Users\...> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Python\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
```
Already tried to rearrange the order and place individual components on the top level, without success. I am grateful for any advice 😄
Simon


FineTune XLSR-Wav2Vec2 on New Langauge WER still 1

Hi, can i ask is there anyone facing issues fine tuning wav2vec2 for languages not in the common dataset.

I am trying to finetune for a language not within in the common datasets, but i get a WER of 1 not matter how many steps i tried to finetune,  I have a similar issue here #10884

You can check my code in the repo here: (Note i did make some small changes that weren't in the original notebook by the huggingface team to fit the training notebook they provided)
https://github.com/edwin-19/wave2vec2-hf-sprint/blob/master/Train.ipynb


Update setup.py


support passing path to a `config` variable in AutoClass

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

It enables loading weights from a private server like the following.
```python
AutoModel.from_pretrained('https://myserver/model/pytorch_model.bin', config='path/to/local/config.json')
```
**For the moment, the function to load weights from a private server like this way is supported in `PretrainedModel` but not in `AutoModel`.**
```python
BertModel.from_pretrained('https://myserver/model/pytorch_model.bin', config='path/to/local/config.json') # supported
AutoModel.from_pretrained('https://myserver/model/pytorch_model.bin', config='path/to/local/config.json') # not supported
```
To fix this issue, I copy and pasted code about `config_path` from `PretrainedModel` to `AutoModel`.

This features was requested in #10961 .

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@LysandreJik


Enforce string-formatting with f-strings

# What does this PR do?

This PR removes any strings formatted with `.format` or `%` to use f-strings exclusively (unless there is a very good reason to use the other syntax, or the file is in a research_project/legacy folder).

The mix of three syntaxes does not make any sense and we all agree in the team that f-strings are more readable. Now that Python 3.5 is officially dead, there is no reason not to switch fully to f-strings.

cc @stas00  as we had a conversation about that.


Tagged Model Version Not Working

I am trying to download a specific version of the `roberta-large` model using the `revision` parameter of `from_pretrained()` as shown below:

```
from transformers import RobertaForSequenceClassification
model_type = "roberta-large"
v1 = "v3.5.0"
model = RobertaForSequenceClassification.from_pretrained(model_type, num_labels=2, revision=v1)
```

This code gives me the following 404 error:

```
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/roberta-large/resolve/v3.5.0/config.json
```

Am I using the `revision` parameter incorrectly?

Extra notes:
- I am using transformers v4.1.1
- Running the same code above with `v1 = "main"` works just fine


Add GPT Neo models to Write With Transformer

Request for a new feature

# 🚀 Feature request
Would it be possible to get the newly-added GPT Neo models usable on Write With Transformer?

## Motivation
It would be helpful to use the new models in the Write With Transformer app since it supports newlines.

CC @LysandreJik