Proposed Fix : [RagSequenceForGeneration] generate "without" input_ids

# What does this PR do?
Proposal of option (2) mentioned in the issue `RagSequenceForGeneration.generate()` without `input_ids`
https://github.com/huggingface/transformers/issues/9184

###  Re-paste here:
In `RagSequenceForGeneration` method `generate()` function, the doc said that both `input_ids` and `context_input_ids` are optional (one of them must be specified) .

However, in the code https://github.com/ratthachat/transformers/blob/ragseq_context_id/src/transformers/models/rag/modeling_rag.py#L907
It specifically need `input_ids` in all cases.

Not sure which option is the best

(1) simply said `input_ids` is always needed , OR

(2) add code to calculate `nll` if only `context_input_ids` is provided , but in this case `doc_scores` and `context_attention_mask` have to be provided as well (similar to RagModel requirement ) : https://github.com/ratthachat/transformers/blob/ragseq_context_id/src/transformers/models/rag/modeling_rag.py#L588

I think option (2) should be reasonable since  `RagTokenForGeneration` method `generate()` also requires the same.

### who can review?
@lhoestq or @patrickvonplaten

### Make style applied but failed quality check
I already applied `make style` and passed, but somehow still fails the "code quality check" sorry :(


Fix beam search generation for GPT2 and T5 on model parallelism

# What does this PR do?

This PR fixes beam search generation crash when the model layers are distributed on several devices (model parallelism). I have also added a test which showcase the bug on master and pass with the provided fix. Fixes issue #9200

This is my first contribution to Transformers, so please let me know if something seems wrong or can be improved! The added test doesn't actually test anything at the moment, just raises an error on master. Looking forward to some feedback on how you'd like to see that.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Issue #9200
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@LysandreJik
@alexorona
@patrickvonplaten


run_clm.py AttributeError: 'NoneType' object has no attribute 'keys'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.0dev0
- Platform: Linux-4.19.0-13-cloud-amd64-x86_64-with-debian-10.7
- Python version: 3.7.8
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: No (using TPU)
- Using distributed or parallel set-up in script?: don't know

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 Trainer: @sgugger
 examples/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): gpt2

The problem arises when using:
* [ *] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ *] an official GLUE/SQUaD task: pre-training on wikitext dataset
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.create a tpu
2.install transformers and dataset
3.run the code
```ruby
python3 run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-clm
```

error message:
```ruby
Traceback (most recent call last):
  File "run_clm.py", line 51, in <module>
    MODEL_CONFIG_CLASSES = list(MODEL_FOR_CAUSAL_LM_MAPPING.keys())
AttributeError: 'NoneType' object has no attribute 'keys'
```

## Expected behavior
To start pretraining on the TPU


Fix documentation links always pointing to master.

# What does this PR do?

This PR fixes documentation issue. The issue is that hyperlinks pointing to code file in the docs always point to the master.
Reference issue : https://github.com/huggingface/transformers/issues/7988

I found [extlinks](https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html) more appropriate than [rst_epilog
](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-rst_epilog) as mentioned in the original issue because hyperlink replacement is not possible with `rst_epilog`
Stackoverflow issue reference : https://stackoverflow.com/questions/1227037/substitutions-inside-links-in-rest-sphinx

Although this fixes the issues as discussed I am unable to fix the  issue  in markup files which are https://huggingface.co/transformers/master/contributing.html and https://huggingface.co/transformers/master/notebooks.html
Please suggest if it is possible to do that as well.

Fixes https://github.com/huggingface/transformers/issues/7988

@sgugger @LysandreJik


checkpoint callbacks

Hi
I need to save more things during checkpoint and I am using finetune_trainer.py not the pytorch lightening one, could you provide me with example on how I can modify the save part of the checkpoints? thanks


Hi
@stas00 @sgugger
I am testing last version of finetune_trainer on multiple gpus, with this command

`export BS=4; CUDA_VISIBLE_DEVICES=0,1  USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9910  finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler  --task translation --val_max_target_length 128 --warmup_steps 500 --n_train 500 --sharded_ddp
`

these two packages neeed to be added to requirements.txt
pytorch-lightning=1.1.1
fairscale

but still getting this error

```
12/19/2020 16:56:16 - INFO - utils -   using task specific params for translation: {}
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 291, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 672, in train
    model = ShardedDDP(model, self.optimizer)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 99, in __init__
    self._setup_backward_hooks()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 280, in _setup_backward_hooks
    assert p_tmp.grad_fn is not None
AssertionError
12/19/2020 16:56:16 - INFO - __main__ -   *** Train ***
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 291, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 672, in train
    model = ShardedDDP(model, self.optimizer)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 99, in __init__
    self._setup_backward_hooks()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 280, in _setup_backward_hooks
    assert p_tmp.grad_fn is not None
AssertionError
Traceback (most recent call last):
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/updated/bin/python', '-u', 'finetune_trainer.py', '--local_rank=1', '--model_name_or_path', 't5-small', '--output_dir', 'output_dir', '--adam_eps', '1e-06', '--data_dir', 'wmt_en_ro', '--do_train', '--freeze_embeds', '--label_smoothing', '0.1', '--learning_rate', '3e-5', '--logging_first_step', '--logging_steps', '1000', '--max_source_length', '128', '--max_target_length', '128', '--num_train_epochs', '1', '--overwrite_output_dir', '--per_device_train_batch_size', '4', '--sortish_sampler', '--task', 'translation', '--val_max_target_length', '128', '--warmup_steps', '500', '--n_train', '500', '--sharded_ddp']' returned non-zero exit status 1.
```

thanks

requirements erorr
```
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 282, in main
    data_args=data_args,
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/seq2seq_trainer.py", line 58, in __init__
    super().__init__(*args, **kwargs)
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 299, in __init__
    raise ImportError("Sharded DDP training requires fairscale: `pip install fairscale`.")
ImportError: Sharded DDP training requires fairscale: `pip install fairscale`.
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 282, in main
    data_args=data_args,
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/seq2seq_trainer.py", line 58, in __init__
    super().__init__(*args, **kwargs)
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 299, in __init__
    raise ImportError("Sharded DDP training requires fairscale: `pip install fairscale`.")
ImportError: Sharded DDP training requires fairscale: `pip install fairscale`.
Traceback (most recent call last):
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/updated/bin/python', '-u', 'finetune_trainer.py', '--local_rank=1', '--model_name_or_path', 't5-small', '--output_dir', 'output_dir', '--adam_eps', '1e-06', '--data_dir', 'wmt_en_ro', '--do_train', '--freeze_embeds', '--label_smoothing', '0.1', '--learning_rate', '3e-5', '--logging_first_step', '--logging_steps', '1000', '--max_source_length', '128', '--max_target_length', '128', '--num_train_epochs', '1', '--overwrite_output_dir', '--per_device_train_batch_size', '4', '--sortish_sampler', '--task', 'translation', '--val_max_target_length', '128', '--warmup_steps', '500', '--n_train', '500', '--sharded_ddp']' returned non-zero exit status 1.
```


Save underlying BertModel only

I currently have a custom model on top of a pretrained BERT Model, which is effectively just dropout and a linear layer on top for classification. I want to be able to train this classifier, updating the underlying weights, but then to save the `BERTModel` underneath (w/o linear classification layer) so that I can load from file and use this as my input to the same custom model. Is there a way I can access and save the underlying transformer to then be used again like this?

Code for clarification:
```
class BERTBinaryClassifier(torch.nn.Module):

    def __init__(self, model_name_or_path: str):
        super(BERTBinaryClassifier, self).__init__()
        self.bert = ModelSelection.get_model(
            model_name=model_name_or_path)
        self.drop = torch.nn.Dropout(p=0.3)
        self.out = torch.nn.Linear(self.bert.config.hidden_size, 2).cuda()

    def forward(self, inputs):
        logits, embs, *_ = self.bert(**inputs)
        output = self.drop(logits)
        return self.out(output), embs
```

Right now I'm just outputting the logits and hidden_state for future usage but I'd like to be able to use this same function to effectively load a `BertModel` like I do at the start here with `self.bert` (which just loads `BertModel.from_pretrained` and save my Pytorch classifier as a whole, separately.

Would it be as simple as say accessing `self.bert.bert` and then saving it that way?


bert-mlm-converting from tensorflow to pytorch

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


LXMERT cross_modality_matching logits order in code vs. documentation

Issue #7266 evolved into an entirely different issue described below.

# Possible documentation error
Independently of considering #8333 or not, it is clear that the results of the cross-modality matching in LXMERT (for example in [this demo](https://colab.research.google.com/drive/18TyuMfZYlgQ_nXo-tr8LCnzUaoX0KS-h?usp=sharing)) are better (below random chance vs. above random chance) when the first logit of the `output_lxmert['cross_relationship_score']` *– (torch.FloatTensor of shape (batch_size, 2))* represents *a mismatch* rather than *a match*.

This **contradicts the [documentation](https://huggingface.co/transformers/model_doc/lxmert.html)** which in my understanding assigns the first logit to `is_match` (True):
> cross_relationship_score – (torch.FloatTensor of shape (batch_size, 2)): Prediction scores of the textual matching objective (classification) head (scores of True/False continuation before SoftMax).

@LysandreJik, @eltoto1219: Is the documentation right or wrong? How was the model trained and which logit was intended to predict the match and which one the mismatch?

I would be very happy to have a clear answer about the order of the logits in training and code and its correspondence to the documentation. I know how easy it is to flip the logits around and choose the one delivering better results on the data at hand. But in my understanding of scientific conduct, I can not just choose this by wishful thinking and ignoring the documentation: there might be other things going on exhibiting this behavior.

Thank you in advance!


[trainer] deepspeed integration

This PR adds experimental support for Deepspeed <https://github.com/microsoft/deepspeed>, whose main feature is ZeRO covered by the paper [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He](https://arxiv.org/abs/1910.02054).

Recently added support for sharded DDP ([fairscale](https://github.com/facebookresearch/fairscale)) also implements parts of ZeRO. Deepspeed implements all of ZeRO.

I haven't experimented enough yet, but it indeed delivers incredible results.

For example I can get about a 5-8 times bigger batch onto the same hardware as compared to the same code running w/o deepspeed and the speedup is huge too. In the following example I was able to get 4.5x speedup on training, and ~2x on validation/testing:

```
# baseline
export BS=3; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0  python \
-m torch.distributed.launch --nproc_per_node=2  ./finetune_trainer.py --model_name_or_path \
sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro \
--do_eval --do_predict --do_train --evaluation_strategy=steps --fp16 --freeze_embeds --label_smoothing 0.1 \
--learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 \
--num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS \
--predict_with_generate --eval_steps 25000 --save_steps 25000 --sortish_sampler --src_lang en_XX --task translation \
--test_max_target_length 128 --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 2000 \
--n_val 2000 --n_test 2000

2020-12-18 22:31:40 | INFO | __main__ |   train_runtime = 144.9132
2020-12-18 22:37:10 | INFO | __main__ |   val_runtime = 329.8146
2020-12-18 22:42:37 | INFO | __main__ |   test_runtime = 326.6212

# deepspeed
export BS=20; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0 deepspeed  \
./finetune_trainer.py --model_name_or_path sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 \
--data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps  --freeze_embeds --label_smoothing 0.1 \
--learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 \
--num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS \
--predict_with_generate --eval_steps 25000 --save_steps 25000 --sortish_sampler --src_lang en_XX --task translation \
--test_max_target_length 128 --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 2000 --n_val 2000 \
--n_test 2000 --deepspeed  ds_config.json

2020-12-18 22:51:46 | INFO | __main__ |   train_runtime = 32.6825
2020-12-18 22:54:47 | INFO | __main__ |   val_runtime = 180.5917
2020-12-18 22:57:51 | INFO | __main__ |   test_runtime = 183.7731

```

The bleu eval scores were slightly better than the baseline (~0.5 point higher), but it's not enough to make any conclusions based on a single run.

The cool thing is that deepspeed does everything by itself, even the `--fp16` handling, so really it was all about getting out of its way, thus the main part of the integration was to disable a lot of things the trainer does when `--deepspeed` is enabled.

Note the different invocation pattern. If normally we run distributed as:
```
python -m torch.distributed.launch --nproc_per_node=2 ./program.py args
```
deepspeed performs its own DDP internally, and requires the program to be started with:
```
deepspeed  ./program.py args
```

The only thing I'm not sure about with this PR is that deepspeed enables all of its features via a json config file, so I'm not sure where to stash a sample one. I guess I will just add it to the documentation. Currently I put one under `examples/seq2seq/ds_config.json` since that's where the test that needs it lives.

But once this is merged all interested parties can start experimenting with various features, and it won't impact `transformers` code. They will just need to tweak `ds_config.json`. And we convert many trainer cl args into DS config on the fly.

There surely will be competition betweeen fairscale and deepspeed integrations. So far from the few experiments I did `deepspeed` allows for a bigger batch size than fairscale.

To install deepspeed you can just do `pip install deepspeed` - I'm not sure if all bug fixes are in the release. We can make a request to release a new version when this is merged.

If the build fails I recommend pre-compiling its CUDA extensions (otherwise they get built at run time via PTX) via master:
```
git clone https://github.com/microsoft/deepspeed
cd deepspeed
DS_BUILD_OPS=1 pip install --no-cache -v --disable-pip-version-check -e . 2>&1 | tee build.log
```
If you want a faster build add an env var `TORCH_CUDA_ARCH_LIST` with the cuda compute capabilities you need, e.g. I do:
```
TORCH_CUDA_ARCH_LIST="6.1;8.6" DS_BUILD_OPS=1 pip install --no-clean --no-cache -v --disable-pip-version-check -e . 2>&1 | tee build.log
```

It was awesome that @sgugger has just added `fairscale` support, so it was much easier to do the same for deepspeed seeing how `fairscale` was integrated, so I'm appreciating the work you have done, Sylvain.

----------------------

Do try it so we get better testing!

You will need 2+ gpus to use it

First install it:
```
pip install deepspeed
```

At the very least do the test:
```
cd examples/seq2seq
pytest -sv test_finetune_trainer.py -k deepspeed
```

Or if you want to fiddle with the normal run, here is what I have been using.
```
cd examples/seq2seq
wget https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz
tar -xzvf wmt_en_ro.tar.gz
export BS=20; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0 deepspeed --num_gpus=2 ./finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 100 --n_val 100 --n_test 100 --deepspeed ds_config.json --fp16 --save_steps 1
```

----------------------

Questions that need to be addressed so that all Trainer features continue to work under deepspeed.

* [ ] a notebook with benchmarks was requested

Probably at a later time, my uneven gpu-sized setup doesn't lend to impressive benchmarks - may be someone will send me another rtx-3090 card ;)

@sgugger, @LysandreJik, @patrickvonplaten


MLFlow logger breaks training

I'm running into two issues when I try and train with mflow installed.

The main one is transformers logs every single parameter and mlflow has restrictions on parameter size. So when I tried to train a `AutoModelForSequenceClassification` using the code below

```
num_labels=25
config = AutoConfig.from_pretrained(
    './models/roberta',
    num_labels=num_labels)
model = AutoModelForSequenceClassification.from_pretrained(
    './models/roberta',
    config=config)
```

An exception is thrown by the mlflow/utils/validation.py - specifically that the length of the value of parameter with key "id2label" is > 250 (this is a character limit so it's trying to send the dict as a string)

The second issue is if I resume training after this error I get a further mlflow exception saying that the run is still in progress - I have to manually call

`mlflow.end_run()`

I'm not sure how to work around this (for now I've uninstalled mlflow). Can I tell the trainer not to log certain parameters?


Error while loading model file - .ckpt file :: Missing key(s) in state_dict

@julien-c @patrickvonplaten @thomwolf
transformers version == 4.0.0

I trained a T5 model for classification and I am trying to load the checkpoint model saved after 3 epochs.
While trying to load the model, getting the below error.
![image](https://user-images.githubusercontent.com/54103522/102682082-a0281200-41ec-11eb-8292-4fbeba807ec3.png)

The model file is a ckpt file.

Any help is appreciated.
Thanks.


[docs] outline sharded ddp doc

This PR provides an initial outline of HF Trainer integration starting with ZeRO. We have fairscale's Sharded optimizer/gradients supported already and deepspeed is coming

We won't merge this until fairscale merged all the required fixes and released a new version, but I thought it'd be good to start the doc going so it's ready when fairscale is ready.

I hope to submit a deepspeed integration shortly as well, so we will extend it then with deepspeed info. edit (https://github.com/huggingface/transformers/pull/9211)

@sgugger


Saving Pretrained Tokenizer

I created a custom `tokenizers.Tokenizer` and saved it as follows

```
tokenizer.model.save("./tokenizer")
tokenizer.save("./tokenizer.json")
```

This produces 3 files, merges.txt, vocab.json and tokenizer.json

Then I created a `transformers.RobertaTokenizerFast` and saved it to the same folder

```
tokenizer = RobertaTokenizerFast.from_pretrained("./tokenizer")
tokenizer.save_pretrained("./tokenizer")
```

This adds `special_tokens_map.json` and `tokenizer_config.json`

I then saved it to another folder to simulate what happens when I train my model

```
tokenizer.save_pretrained("./model")
tokenizer = RobertaTokenizerFast.from_pretrained("./model")
```

What I noticed was `tokenizer_config.json` contains a key `name_or_path` which still points to `./tokenizer`, so what seems to be happening is `RobertaTokenizerFast.from_pretrained("./model") ` is loading files from two places (./model and ./tokenizer)

Not sure if this is expected, it seems that the tokenizer_config.json should be updated in save_pretrained, and tokenizer.json should be saved with it?

Or perhaps this is just an issue because I'm training my tokenizer in a subdirectory of the model folder?


DataTrainingArguments: __init__() got an unexpected keyword argument 'evaluate_during_training'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@sgugger
## Information

Model I am using (Bert, XLNet ...): Distilbert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the HF example Notebook here: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb

That Notebook is linked from here: https://huggingface.co/transformers/examples.html

2. At this line:
```python
training_args = TrainingArguments(
    output_dir="./models/model_name",
    overwrite_output_dir=True,
    do_train=True,
    do_eval=True,
    per_gpu_train_batch_size=32,
    per_gpu_eval_batch_size=128,
    num_train_epochs=1,
    logging_steps=500,
    logging_first_step=True,
    save_steps=1000,
    evaluate_during_training=True,
)
```

an error is raised:
```
TypeError                                 Traceback (most recent call last)
<ipython-input-6-e83ba093226a> in <module>()
     14     logging_first_step=True,
     15     save_steps=1000,
---> 16     evaluate_during_training=True,
     17 )

TypeError: __init__() got an unexpected keyword argument 'evaluate_during_training'
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The HF example Notebook should complete successfully.


[model_utils] very slow model instantiation

WIP

Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

For some reason I'm noticing a very slow model instantiation time.

For example to load `shleifer/distill-mbart-en-ro-12-4` it takes
* 21 secs to instantiate the model
* 0.5sec to `torch.load` its weights.

If I'm not changing how the model is created and want to quickly fast forward to the area of debug how could these slow parts be cached and not rebuilt anew again and again?

But also it looks like we are doing a completely wasteful operation of init_weights, which immediately get overwritten with pretrained model weights (https://github.com/huggingface/transformers/issues/9205#issuecomment-748741195) (for the use case of pre-trained model).

(I initially made a mistake and thought that it was `torch.load` that had an issue, but it's `cls(config, *model_args, **model_kwargs)`) - thank you, @sgugger - so this post has been edited to reflect reality. So if you're joining later you can skip the comments up to https://github.com/huggingface/transformers/issues/9205#issuecomment-748722644 and continue from there)

@patrickvonplaten, @sgugger, @LysandreJik


Load saved Pytorch model into Tensorflow or convert from Pytorch model to TF

Hi,

Thanks for this awesome framework!
I have trained and saved an XLMRoberta model in PyTorch and I'm wondering if there is any way I can load the model into TFRobertaForSequenceClassification class. Or if there are ways to convert the checkpoint to TensorFlow checkpoints so that it can be load by TF2.

I came across this file https://github.com/huggingface/transformers/blob/master/src/transformers/convert_pytorch_checkpoint_to_tf2.py
and I'm wondering if there are any docs or instructions to use the script.

Thanks!


[finetune trainer] better logging and help

As a follow up to this [thread](https://discuss.huggingface.co/t/summarization-is-finetune-trainer-py-accepting-length-arguments-correctly/2879/) this PR:

* documents that `--val_max_target_length` is also used during `generate`
* disambiguates `use_task_specific_params` logger so that it's clear that it dumps just the initial params and that those could be overridden by user's cl args

@sgugger


[wanted] explicit docs for inherited methods

# 🚀 Feature request

The HF approach is to unroll most of the code for the ease of understanding, but somehow this is not the case with docs.

If possible, could we explicitly add documentation for inherited methods in the specific model documentation page?

e.g. why https://huggingface.co/transformers/model_doc/t5.html doesn't have `T5ForConditionalGeneration.generate` documented - sure one eventually figures out it's https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate but why make user's life so difficult when the docs are autogenerated anyway.

In the worst case there could be an entry for `T5ForConditionalGeneration.generate` with the link to https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate, but that is not ideal since one has to leave the main doc of the model, which makes it much harder to jump around its different parts.

IMHO, it'd greatly improve user's experience to have all the functionality documentation supported by a model in one page of that model.

And xrefs are super-useful too, e.g. [t5 pre-amble](https://huggingface.co/transformers/model_doc/t5.html#overview) mentions  `T5ForConditionalGeneration.generate `but it's not linked to anywhere.

Thank you!

@sgugger, @LysandreJik


when to use sortish sampler

Hi,
I appreciate adding to the documentation on when to use sortish sampler and which platform this works/how much it impacts the speed, this is related to seq2seq folder of huggingface, for Seq2seqDataset.
thanks


Beam search fails when using model parallelism

## Environment info
- `transformers` version: 4.1.1
- Platform: Linux-4.4.0-194-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, two GTX 1080, on a single node
- Using distributed or parallel set-up in script?: Using model parallelism through `model.parallelize()`

### Who can help
@LysandreJik
@alexorona

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [ ] the official example scripts:
* [x] my own modified scripts:

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task:
* [ ] my own task or dataset:

## To reproduce

The recent (and awesome!) model `parallelize()` doesn't seem to work with beam search decoding at the moment. The behavior can be reproduced on the official `huggingface/transformers-pytorch-gpu:4.1.1` docker image by running the following (on a machine with multiple GPUs):

```python
import transformers

tokenizer = transformers.GPT2Tokenizer.from_pretrained("gpt2")
model = transformers.GPT2LMHeadModel.from_pretrained("gpt2")
model.parallelize()

input_ids = tokenizer.encode("This is a test", return_tensors="pt").to("cuda:0")
model.generate(input_ids, num_beams=2)
```
This raises the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 612, in generate
    **model_kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 1088, in beam_search
    model_kwargs["past"] = self._reorder_cache(model_kwargs["past"], beam_idx)
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 229, in _reorder_cache
    return tuple(layer_past.index_select(1, beam_idx) for layer_past in past)
  File "/usr/local/lib/python3.6/dist-packages/transformers/generation_utils.py", line 229, in <genexpr>
    return tuple(layer_past.index_select(1, beam_idx) for layer_past in past)
RuntimeError: Input, output and indices must be on the current device
```

## Expected behavior

The expected behavior is to not raise an error, but instead correctly return the beam search decoding.


[t5 doc] typos

This PR fixes a few run away backticks

Sylvain, why do we not have documentation for `T5ForConditionalGeneration.generate`? This doc is trying to link to it, but there is no such entry in https://huggingface.co/transformers/model_doc/t5.html

@sgugger


[run_glue] add speed metrics

This PR starts to sync with recent changes in trainer+finetune_trainer.py

* train: (sync with finetune_trainer):
   - prints and saves train speed metrics - needed for benchmarking
   - saves the state,
* eval: sorts metrics logging info

@sgugger


[RAG] Add Ray implementation for distributed retrieval

# What does this PR do?
This PR adds a new distributed retriever implementation for RAG built on Ray, as an alternative to the current retriever implementation that uses torch.distributed. With Ray it's possible to load the index on multiple processes instead of just the rank 0 training worker, allowing fine tuning to scale out better to multiple GPUs, and also allowing the index to potentially be fit in GPU memory. This also removes a core dependency on Pytorch, allowing a Tensorflow implementation of `finetune.py`.

This PR also makes changes to support finetune.py with Pytorch Lightning >v1.0.

A benchmark of Pytorch distribtued retrieval vs. Ray distributed retrieval
![image](https://user-images.githubusercontent.com/8068268/99327675-4ae49380-282f-11eb-8126-6250558fd795.png)

## Implementation Details
In the current Pytorch retrieval implementation, the index is loaded once on just the rank 0 training workers. Training worker 0 gathers the inputs from all other workers, performs the index lookup, and scatters the results back to the other workers.
![image](https://user-images.githubusercontent.com/8068268/99328533-238ec600-2831-11eb-9d26-e153cd0de74f.png)

With the Ray implementation, the index is loaded on *separate* processes, which are referred to as Ray actors. Each training worker randomly selects a retrieval actor to query for documents and Ray handles all the communication between the processes. Because the index can be loaded in *multiple* processes, training can scale up since no synchronization needs to happen for the index lookup.
![image](https://user-images.githubusercontent.com/8068268/99328651-618bea00-2831-11eb-990b-f1727df15190.png)

Note that Pytorch Lightning is still handling distributed *training*, but Ray manages distributed *retrieval*. Because PTL calls the entire training script under the hood multiple times, we have to use Ray's named actors feature (https://docs.ray.io/en/master/actors.html?highlight=named%20actors#named-actors) allowing the retrieval actors to be referenced by all training processes. The use of named actors is necessitated by how PTL handles distributed training, and a simpler approach could probably be used for a Tensorflow implentation.

## Testing Strategy
Unit tests were added to `test_distributed_retriever.py`. Note that the local Ray cluster for the tests had to be started with `local_mode=True` because the test file modifies `sys.path` and these changes are not propagated to remote processes. See https://stackoverflow.com/questions/54338013/parallel-import-a-python-file-from-sibling-folder for more info.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Add timing inside Trainer

# What does this PR do?

Add timing reports for training/evaluation and test inside the Trainer. Also, change the default repr of `TrainingArguments` to avoid printing the deprecated arguments.

There is a breaking change in this PR: the output of `Trainer.train` gains a new field `metrics`, so the length of the namedtuple changes. I don't think it's too bad since all scripts and examples I've seen never store the result of this `train` method. After discussion with @LysandreJik we proposed to merge this breaking change and revert it before the next release if users complain.


Error "if input.dim() == 2 and bias is not None"

Pytorch version: pytorch-1.7.1-py3.8_cuda11.0.221_cudnn8.0.5_0
Transformer version: 4.0.0

Code:
```
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
from transformers import AutoModel, BertTokenizerFast

class BERT_Arch(nn.Module):
    def __init__(self, bert):
        super(BERT_Arch, self).__init__()
        self.bert = bert

        # dropout layer
        self.dropout = nn.Dropout(0.1)
        # relu activation function
        self.relu =  nn.ReLU()
        # dense layer 1
        self.fc1 = nn.Linear(768,512)

        # dense layer 2 (Output layer)
        self.fc2 = nn.Linear(512,2)
        #softmax activation function
        self.softmax = nn.LogSoftmax(dim=1)

    #define the forward pass
    def forward(self, sent_id, mask):

        #pass the inputs to the model
        _, cls_hs = self.bert(sent_id, attention_mask=mask)
        x = self.fc1(cls_hs)
        x = self.relu(x)
        x = self.dropout(x)

        # output layer
        x = self.fc2(x)

        # apply softmax activation
        x = self.softmax(x)

        return x

# import BERT-base pretrained model
bert = AutoModel.from_pretrained('bert-base-uncased')
# pass the pre-trained BERT to our define architecture
model = BERT_Arch(bert)
# push the model to GPU
model = model.to(device)

# dataLoader for train set
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

for step,batch in enumerate(train_dataloader):
    batch = [r.to(device) for r in batch]
    sent_id, mask, labels = batch
    preds = model(sent_id, mask)
```

Error:

> ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-46-3656917b982b> in <module>
      2     batch = [r.to(device) for r in batch]
      3     sent_id, mask, labels = batch
----> 4     preds = model(sent_id, mask)

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-43-05830d6f294e> in forward(self, sent_id, mask)
     21         #pass the inputs to the model
     22         _, cls_hs = self.bert(sent_id, attention_mask=mask)
---> 23         x = self.fc1(cls_hs)
     24         x = self.relu(x)
     25         x = self.dropout(x)

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

~/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1686         if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
   1687             return handle_torch_function(linear, tens_ops, input, weight, bias=bias)
-> 1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
   1690         ret = torch.addmm(bias, input, weight.t())

AttributeError: 'str' object has no attribute 'dim'


Loading MPNet from disc: ValueError: An instance of tokenizer class MPNetTokenizer cannot be converted in a Fast tokenizer instance.

## Environment info
- `transformers` version: 4.1.1 (pip version)
- Platform: Ubuntu 20.04
- Python version: 3.7
- PyTorch version (GPU?): Pytorch 1.7 GPU

## Information

Hi,
thanks for adding MPNet. I got quite promising results when using it for generating sentence embeddings.

However, there is an issue when saving and loading the MPNet model (when using version 4.1.1 of transformers, installed via pip):
```python
from transformers import AutoTokenizer, AutoModel

local_dir = 'mpnet-model/'

model = AutoModel.from_pretrained('microsoft/mpnet-base')
tokenizer = AutoTokenizer.from_pretrained('microsoft/mpnet-base')
model.save_pretrained(local_dir)
tokenizer.save_pretrained(local_dir)

#Load tokenizer and model from dir
model = AutoModel.from_pretrained(local_dir)

# The following command will throw an exception
tokenizer = AutoTokenizer.from_pretrained(local_dir)

```

This leads to the following error:
```
File "/home/reimers/miniconda3/envs/sbert/lib/python3.7/site-packages/transformers/convert_slow_tokenizer.py", line 636, in convert_slow_tokenizer
    f"An instance of tokenizer class {tokenizer_class_name} cannot be converted in a Fast tokenizer instance. "
ValueError: An instance of tokenizer class MPNetTokenizer cannot be converted in a Fast tokenizer instance. No converter was found. Currently available slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'CamembertTokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LongformerTokenizer', 'LxmertTokenizer', 'MBartTokenizer', 'MobileBertTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'ReformerTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer']
```


Full rework of the TF input/output embeddings and bias resizing

# What does this PR do?

This PR 100% reworks the entire process of input/output and bias resizing. Now the exceptions are better handled including the names that now are always similar. The corresponding tests have also been entirely reworked and now have a better coverage of this feature.

This PR adds a small breaking change. Now the `get_input_embeddings` methods returns the weights and not anymore the embedding layer.


example code for fine-tuning CLM does not work for GPT

## Environment info
- `transformers` version: 3.5.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NO

### Who can help
@patrickvonplaten @TevenLeScao

## Information

Model I am using: open-ai GPT.

The problem arises when using:
* [x] the official example scripts: (give details below)

```
python run_clm.py --model_name_or_path openai-gpt --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --output_dir /tmp/test-clm
```

It's the example script on: [(https://github.com/huggingface/transformers/tree/master/examples/language-modeling], which is used to fine-tune a casual language model.

## To reproduce

Steps to reproduce the behavior:

1. go to transformers/examples/language-modeling
2. run the following command

`python run_clm.py --model_name_or_path openai-gpt --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --output_dir /tmp/test-clm`

3 The following error occurs:
`
RuntimeError: The size of tensor a (1024) must match the size of tensor b (512) at non-singleton dimension 1
`

In my case, the error does not occur when using `--model_name_or_path gpt2`.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The progress bar should be filled and the language model should be finetuned.

<!-- A clear and concise description of what you would expect to happen. -->


Segfault on python 3.9 exit

Ok, that's a weird one

## Environment info

- `transformers` version: 4.1.1
- Platform: Linux-5.8.0-34-generic-x86_64-with-glibc2.32
- Python version: 3.9.0+
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

Steps to reproduce the behavior:

1. Run the following script

    ```python
    import torch
    import transformers

    loss = torch.tensor([1.0], requires_grad=True)
    loss.backward()
    ```

The script runs correctly but exits with

```text
[1]    46823 segmentation fault (core dumped)  python testcase.py
```

Which doesn't happen if `import transformers` is commented out.

Only happens when on Python 3.9, it works as expected in 3.8.

## Full env

```text
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
filelock==3.0.12
idna==2.10
joblib==1.0.0
numpy==1.19.4
packaging==20.8
pyparsing==2.4.7
regex==2020.11.13
requests==2.25.1
sacremoses==0.0.43
six==1.15.0
tokenizers==0.9.4
torch==1.7.1
tqdm==4.54.1
transformers==4.1.1
typing-extensions==3.7.4.3
urllib3==1.26.2
```


Addition of MuRIL - BERT based model for 17 Indian Languages to the library

Hi,

This PR is regarding the addition of MuRIL, a BERT-based model trained specifically for 17 Indian Languages to the hugging face library.

MuRIL is released on tfhub. Link to the repo: [https://tfhub.dev/google/MuRIL/1](https://tfhub.dev/google/MuRIL/1)

I am interested to work on this contribution to the library. Please let me know if I can work on it.


GPT-model attention heads pruning example

# What does this PR do?

This script is adapted from the [BERT attention heads pruning code](https://github.com/huggingface/transformers/blob/master/examples/research_projects/bertology/run_bertology.py) AKA Bertology to make it possible to prune GPT-model heads as well.

It basically works the same way as run_bertology.py, but can deal with GPT-models.


MRPC Reproducibility with transformers-4.1.0

I always get lower precision following the MRPC example, what's the reason?
```
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/$TASK_NAME/
```
and get
```
12/18/2020 17:16:38 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 17:16:38 - INFO - __main__ -     eval_loss = 0.5318707227706909
12/18/2020 17:16:38 - INFO - __main__ -     eval_accuracy = 0.7622549019607843
12/18/2020 17:16:38 - INFO - __main__ -     eval_f1 = 0.8417618270799347
12/18/2020 17:16:38 - INFO - __main__ -     eval_combined_score = 0.8020083645203595
12/18/2020 17:16:38 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:45:29 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:45:29 - INFO - __main__ -     eval_loss = 0.47723284363746643
12/18/2020 16:45:29 - INFO - __main__ -     eval_accuracy = 0.8063725490196079
12/18/2020 16:45:29 - INFO - __main__ -     eval_f1 = 0.868988391376451
12/18/2020 16:45:29 - INFO - __main__ -     eval_combined_score = 0.8376804701980294
12/18/2020 16:45:29 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:34:37 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:34:37 - INFO - __main__ -     eval_loss = 0.571368932723999
12/18/2020 16:34:37 - INFO - __main__ -     eval_accuracy = 0.6838235294117647
12/18/2020 16:34:37 - INFO - __main__ -     eval_f1 = 0.8122270742358079
12/18/2020 16:34:37 - INFO - __main__ -     eval_combined_score = 0.7480253018237863
12/18/2020 16:34:37 - INFO - __main__ -     epoch = 3.0
```

GPU: GTX 1080
transformers: 4.1.0
Torch: 1.6.0
python: 3.8
Server: Ubuntu 18.04


Problem with pretraining GPT-2 on TPU with Pytorch/XLA

### **Environment info**
transformers version: 4.0.1
Platform: Ubuntu 18.04.4 LTS
Python version: 3.6.9
PyTorch version: 1.7.1
Torch-XLA version: 1.7
Tensorflow version: 2.3.1

### **Information**
Model I am intended to pretrain (Bert, XLNet ...): GPT2
The problem arises when using:
- my own modified scripts: (give details below)

The tasks I am working on is:
- pretraining GPT-2 on Google TPU

### **To reproduce**
Steps to reproduce the behavior:

1. According to instructions [here](https://github.com/pytorch/xla/blob/master/README.md#-consume-prebuilt-compute-vm-images), creating instance of compute engine on Google Cloud with following spec:

- OS: Deep Learning on Linux
- Version: Debian GNU/Linux 9 Stretch + Pytorch/XLA

2. Running the code of:
`export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470"`

3. Running following script.
```python
#!/usr/bin/env bash
pipenv run python3 xla_spawn.py --num_cores 8 \
    run_clm.py \
    --num_train_epochs 5 \
    --output_dir saved_model/ \
    --overwrite_output_dir \
    --logging_dir logs \
    --logging_steps 50 \
    --save_total_limit 2 \
    --save_steps 2000 \
    --model_type gpt2 \
    --config_name tokenized_data/ \
    --tokenizer_name tokenized_data/ \
    --block_size 1024 \
    --train_file dataset/dataset.txt \
    --per_device_train_batch_size=64 \
    --do_train
```

Here is the exception occured:
```
Exception in device=TPU:3: tensorflow/compiler/xla/xla_client/mesh_service.cc:316 : Check failed: impl_->channel->WaitForConnected( std::chrono::system_clock::now() + std::chrono::seconds(connect_wait_seconds))
*** Begin stack trace ***
	tensorflow::CurrentStackTrace()
	xla::service::MeshClient::MeshClient(std::string const&)
	xla::service::MeshClient::Get()
	xla::ComputationClient::Create()

	xla::ComputationClient::Get()

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	PyObject_GetAttr
	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	PyObject_Call
	_PyEval_EvalFrameDefault

	PyObject_Call
	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	PyRun_StringFlags
	PyRun_SimpleStringFlags
	Py_Main
	main
	__libc_start_main
	_start
*** End stack trace ***
Failed to connect to client mesh master: workstation:41295
Traceback (most recent call last):
  File "/home/ws/.local/share/virtualenvs/gpt-2_pretrain_huggingface-aTVuIzXL/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/home/ws/.local/share/virtualenvs/gpt-2_pretrain_huggingface-aTVuIzXL/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    _setup_replication()
  File "/home/ws/.local/share/virtualenvs/gpt-2_pretrain_huggingface-aTVuIzXL/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 315, in _setup_replication
    device = xm.xla_device()
  File "/home/ws/.local/share/virtualenvs/gpt-2_pretrain_huggingface-aTVuIzXL/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 231, in xla_device
    devkind=devkind if devkind is not None else None)
  File "/home/ws/.local/share/virtualenvs/gpt-2_pretrain_huggingface-aTVuIzXL/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 136, in get_xla_supported_devices
    xla_devices = _DEVICES.value
  File "/home/ws/.local/share/virtualenvs/gpt-2_pretrain_huggingface-aTVuIzXL/lib/python3.6/site-packages/torch_xla/utils/utils.py", line 32, in value
    self._value = self._gen_fn()
  File "/home/ws/.local/share/virtualenvs/gpt-2_pretrain_huggingface-aTVuIzXL/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 18, in <lambda>
    _DEVICES = xu.LazyProperty(lambda: torch_xla._XLAC._xla_get_devices())
RuntimeError: tensorflow/compiler/xla/xla_client/mesh_service.cc:316 : Check failed: impl_->channel->WaitForConnected( std::chrono::system_clock::now() + std::chrono::seconds(connect_wait_seconds))
*** Begin stack trace ***
	tensorflow::CurrentStackTrace()
	xla::service::MeshClient::MeshClient(std::string const&)
	xla::service::MeshClient::Get()
	xla::ComputationClient::Create()

	xla::ComputationClient::Get()

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	PyObject_GetAttr
	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	PyObject_Call
	_PyEval_EvalFrameDefault

	PyObject_Call
	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	_PyEval_EvalFrameDefault

	PyRun_StringFlags
	PyRun_SimpleStringFlags
	Py_Main
	main
	__libc_start_main
	_start
*** End stack trace ***
Failed to connect to client mesh master: workstation:41295
Exception in device=TPU:2: tensorflow/compiler/xla/xla_client/mesh_service.cc:316 : Check failed: impl_->channel->WaitForConnected( std::chrono::system_clock::now() + std::chrono::seconds(connect_wait_seconds))
```


fixed not JSON serializable error in run_qa.py with fp16

# What does this PR do?

Fixed an issue where running the run_qa.py script in a Squad-like dataset with fp16 enabled, would lead to a JSON serialization error:
```
TypeError: Object of type 'float16' is not JSON serializable
```
The bug is caused by not converting `np.float16` to `float` on line 209 and 397 in the utils_qa.py file.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section? --> yes
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. --> no
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation). --> No need for it
- [x] Did you write any new necessary tests? No

## Who can review?
 @sgugger
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


can we use ckpt model file generated after finetuning the pre-trained models on custom dataset

Hi Team,

I have fine tuned pegasus huggingface wikihow model with my own custom dataset and end up with below files:

model.ckpt-1000.data-00000-of-00001
model.ckpt-1000.index
model.ckpt-1000.meta
events.out.tfevents.1608192436.ip-xxx-xx-x-xx
events.out.tfevents.1608192510.ip-xxx-xx-x-xx.v2

When I am running below code, it is taking pytorch_model.bin file automatically for generating summary. Where as I want to use ckpt model file for generating summary which I got after training.

**Code I am using is :**

model_path ='local-pegasus-wikihow'

torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = PegasusTokenizer.from_pretrained(model_path)

model = PegasusForConditionalGeneration.from_pretrained(model_path).to(torch_device)

batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest', return_tensors="pt").to(torch_device)
translated = model.generate(**batch)
tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
print(tgt_text[0])

Do I need to change code here so that it will take ckpt model file for generating summary instead of searching for pre-trained pytorch_model.bin or .h5 file?


[RagSequenceForGeneration] generate "without" input_ids

Hi guys,

In `RagSequenceForGeneration` method `generate()` function, the doc said that both `input_ids` and `context_input_ids` are optional (one of them must be specified) .

However, in the code https://github.com/ratthachat/transformers/blob/ragseq_context_id/src/transformers/models/rag/modeling_rag.py#L907
It specifically need `input_ids` in all cases.

Not sure which option is the best

(1) simply said `input_ids` is always needed , OR

(2) add code to calculate `nll` if only `context_input_ids` is provided , but in this case `doc_scores` and `context_attention_mask` have to be provided as well (similar to RagModel requirement ) : https://github.com/ratthachat/transformers/blob/ragseq_context_id/src/transformers/models/rag/modeling_rag.py#L588

I think option (2) should be reasonable since  `RagTokenForGeneration` method `generate()` also requires the same.
Proposed fix in https://github.com/huggingface/transformers/pull/9220


Add caching mechanism to BERT, RoBERTa

# What does this PR do?
- This PR adds past key/values caching mechanism to `BertLMHeadModel`, `BertGenerationDecoder`, and `RobertaForCausalLM` to speed up the generation of `EncoderDecoder` models

-  delete the `CausalLMOutputWithPastAndCrossAttentions` class and add `past_key_values`  to `CausalLMOutputWithCrossAttentions` and `BaseModelOutputWithPoolingAndCrossAttentions`  . All `ModelOutputs` that have a `cross-attention` should also have a `past_key_values`

- by default caching is enabled for `BertLMHeadModel`, `BertGenerationDecoder`, and `RobertaForCausalLM` during inference (not just generation, also for the forward pass) and now they also output `past_key_values` by default during inference, which is a small breaking change.
  specifically when `config.output_attentions=True` in a `EncoderDecoderModel` model then the 2nd index of the output will be `past_key_values` instead of `attentions`

```python3
model = EncoderDecoderModel.from_pretrained(...)
outputs = model(input_ids)
attentions = outputs[1]  # 2nd index will be past_key_values, instead of attentions
```

- this will only affect the output during generation for `EncoderDecoder` models. Caching will be disabled when the models are used as standalone encoders, so the default output, in that case, is unchanged.

Fixes #9052


Fix link to old NER fine-tuning script

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Fix link to old SQUAD fine-tuning script


[trainer] apex fixes and tests

This PR:

 * [x] fixes a bug in with `fp16_backend` apex (`is_apex_available` + `amp` weren't getting imported w/ pt>=1.6)
 * [x] adds test
 * [x] adds a logger info on which fp16 backend will be used

@sgugger


[trainer]  speed issues: --fp16 doesn't improve speed, DP runs really slow

Splitting from https://github.com/huggingface/transformers/issues/9156#issuecomment-747636108 where while running benchmarks for fairscale's sharded ddp support I noticed that there was almost no difference between having or not having `--fp16` to the training runtime.

Not sure whether this impacts trainer in general or just seq2seq finetune_trainer.py, but there is almost no speed improvements with adding `--fp16`.

This is with pytorch-nightly.

Not sure whether this has to do with the recent autocast cache blowup fix https://github.com/pytorch/pytorch/issues/48049 (our corresponding issue https://github.com/huggingface/transformers/issues/8403) - part of the fix was to remove some of the cash that perhaps was essential and thus leads to this issue.

I will test with apex and compare.

Also as can be seen from: https://github.com/huggingface/transformers/issues/9179#issuecomment-747783879
DP is running really slow!

@sgugger


[ci] install fairscale on self-runner CIs

The tests for the new sharded ddp fairscale integration are in place, https://github.com/huggingface/transformers/pull/9177
but CIs don't have `fairscale` installed so they won't run on CI.

This is an issue to track this need so we will eventually run these tests.

Blocking event: needing a more reliable/quick way of installing fairscale - need to track the following issue for when this happens:
https://github.com/facebookresearch/fairscale/issues/264

Then we can add `pip install fairscale` to the self-runner CIs with multigpus.


add tests for the new sharded ddp fairscale integration

This PR adds tests for the just added sharded ddp fairscale integration https://github.com/huggingface/transformers/pull/9139

Obviously these won't run on CIs w/o having fairscale installed... hope we will sort this out down the road. the problem is building fairscale - no binary wheel - I will ask them if they could make a jit version.

@sgugger


[setup] correct transformers version format

setuptools has a pretty fixed expectation of version numbers.

```
x.y.z
x.y.z.dev0
x.y.z.rc1
```

This PR fixes the dev version number and adds a comment with correct formats for the future editors

This fix removes this warning on `make fixup|style|etc` or any other time `setup.py` is being run.
```
setuptools/dist.py:452: UserWarning: Normalizing '4.2.0dev0' to '4.2.0.dev0'
  warnings.warn(tmpl.format(**locals()))
```
and the alternative:
```
/setuptools/dist.py:452: UserWarning: Normalizing '4.0.0-rc-1' to '4.0.0rc1'
```

Fixes: #8749

@LysandreJik, @sgugger


Add new run_swag example

# What does this PR do?

This PR adds a new example for multiple-choice using Trainer and Datasets, and moves the older one to the legacy folder.


[WIP] Adapt Cookie Cutter For EncoderDecoder


GPT2 eval with attention_mask not returning expected result

is this not correct use of attention_mask with padding (trying to batch sentence eval) or is it a bug? Shouldn't the return result be the same in each case because of the mask?

import torch
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained('gpt2', return_dict=True)

```
with torch.no_grad():
    model.eval()
    loss,logits = model(input_ids=torch.tensor([15496,11,616]), attention_mask=torch.tensor([1,1,1]), labels=torch.tensor([15496,11,616]))[:2]
    print(loss)  # 3.0698 correct answer
    loss,logits = model(input_ids=torch.tensor([50256,15496,11,616]), attention_mask=torch.tensor([0,1,1,1]), labels=torch.tensor([50256,15496,11,616]))[:2]
    print(loss)  # 8.96 - shouldn't it be 3.0698 as above?
```


[Flax] Implement FlaxElectraModel, FlaxElectraForMaskedLM, FlaxElectraForPreTraining

# What does this PR do?

1. Implement Flax version of Electra model : `FlaxElectraModel`, `FlaxElectraForMaskedLM`, `FlaxElectraForPreTraining`. Most of the code taken from FlaxBert version with changes in parameters and forward pass.
2. Adjust `convert_to_pytorch` to load weights for Electra
3. Implement `FlaxElectraGeneratorPredictions`, `FlaxElectraDiscriminatorPredictions` for generator and discriminator prediction head.
4. Implement test in `tests/test_modeling_flax_electra.py`

Forward pass works by running
```shell
pytest tests/test_modeling_flax_electra.py
```

Hi @patrickvonplaten  , @mfuntowicz , I've seen your work on FlaxBert, so I'm tagging in case you want to review. Please note that I use `flax setup` instead of decorator `@nn.compact` since the former
- allows to test and inspect submodule
- separate submodule declaration from the forward pass. Forward pass method can be very long if using `@nn.compact`
I'm happy to revert this change to make code style consistent.

Let me know if you have any questions or feedbacks.
Thanks.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? Yes, test added `tests/test_modeling_flax_electra.py`

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.


Trainer returns logits of only one sequence instead of entire evaluation dataset

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1  (also reproduced the same issue with 3.5.1)
- Platform: Google Colab
- Python version: 3.6
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?): no
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: (don't know. Probably not)

### Who can help
@sgugger
(The use-case is basically the same as reported in this issue: https://github.com/huggingface/transformers/issues/9160
But I'm opening a new issue, because this is a separate problem)

## Information

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce

**Description:**
I’m trying to fine-tune a pre-trained NLI model (`ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli`) on a dataset of around 276.000 hypothesis-premise pairs. I’m following the instructions from the docs [here](https://huggingface.co/transformers/custom_datasets.html) and [here](https://huggingface.co/transformers/training.html).
- When I run the training, it seems like the fine-tuning works (it does the training and saves the checkpoints).
- But during evaluation, the trainer/model seems to only compute the logits for one sequence instead of the entire training sequence. This means that the resulting metrics don't make sense. This also makes me doubt whether the training is actually executed on the entire training dataset, or only on the first sequence (the loss stays between 7.2 and 7.0 for most of the training steps).
- I've tried to change the input and dataset format in many different ways. One possible issue is that I am somehow only passing the same sequence into the trainer. But I've made sure that this is not the case (see also the printed input_ids in the code snippet below).

```
### load model and tokenizer
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
max_length = 256
hg_model_hub_name = "ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli"
tokenizer = AutoTokenizer.from_pretrained(hg_model_hub_name)
model = AutoModelForSequenceClassification.from_pretrained(hg_model_hub_name)  # num_labels=3
model.config

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device: {device}")
https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html
model.to(device)
model.train();

# ... some data preprocessing

### use hf datasets object + tokenize
from datasets import Dataset

# create hf dataset objects
dataset_train = Dataset.from_pandas(df_train)
dataset_val = Dataset.from_pandas(df_val)
dataset_test = Dataset.from_pandas(df_test)

## tokenize on dataset object
def tokenize(batch):
    return tokenizer(batch['premise'], batch['hypothesis'], max_length=max_length, return_token_type_ids=True, truncation=False, padding=True)
dataset_train = dataset_train.map(tokenize, batched=True, batch_size=len(df_train))
dataset_val = dataset_val.map(tokenize, batched=True, batch_size=len(df_val))
dataset_test = dataset_test.map(tokenize, batched=True, batch_size=len(df_test))

# to tensors
dataset_train.set_format('torch', columns=['input_ids', 'attention_mask', 'label', 'token_type_ids']) # format_kwargs=torch.LongTensor()
dataset_val.set_format('torch', columns=['input_ids', 'attention_mask', 'label', 'token_type_ids'])
dataset_test.set_format('torch', columns=['input_ids', 'attention_mask', 'label', 'token_type_ids'])

print(dataset_val["input_ids"][:2])
# ! to show that the tokenized input sequences are not the same
# output:
tensor([[    0, 45784,     5,  3302,     9,  5222,   274,  1729,     7,  1719,
             5,  1403,    12, 28326,     7,  1157,  1232,     6,    11,  1989,
         10801,  1041,     6,  2388,  8243,    11,  1818,   173,     4,     2,
             2,   133,  2788,    16,    59,   592,  1134,     4,   286,  1246,
            35,  6300,     8,  3111,     6,   786,    12, 12063, 15343,  1134,
             6,  6610,  1134,     6,  1692,  1380,     8,  2038,  1134,     6,
           223, 25943, 33421,  5688,  1134,     2,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1],
        [    0,  1121,   570,    51,  1381,     7,   989,   106,    19,  1085,
            13,    10,   353,     4,     2,     2,   133,  2788,    16,    59,
          6642,     8,  1318,     9,   301,     4,   286,  1246,    35,  3039,
          2591,     6,  1265, 22830,     6,  2040,     6,  6642,   194, 22830,
             6,  6642,   194,  2919,     6,  9057,     6,  1265,  2919,     2,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1,     1,     1,     1,
             1,     1,     1,     1,     1,     1,     1]])

# compute metrics with trainer https://colab.research.google.com/drive/1-JIJlao4dI-Ilww_NnTc0rxtp-ymgDgM?usp=sharing#scrollTo=N8J-TLhBuaOf
from sklearn.metrics import accuracy_score, precision_recall_fscore_support

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    print(pred.predictions, pred.predictions.argmax(-1))
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary', pos_label=0)
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

## training
from transformers import Trainer, TrainingArguments

# https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments
training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=1,              # total number of training epochs
    per_device_train_batch_size=16,   # batch size per device during training
    per_device_eval_batch_size=16,    # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=100,
    fp16=True
)

trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=dataset_train,         # training dataset
    eval_dataset=dataset_val,            # evaluation dataset
    compute_metrics=compute_metrics,
    tokenizer=tokenizer
)

trainer.train()

### evaluate
# for some reason the logits are the the same for each prediction
trainer.evaluate()

[[ 2.055 -7.992  2.053]
 [ 2.055 -7.992  2.053]
 [ 2.053 -7.984  2.05 ]
 ...
 [ 2.055 -7.992  2.053]
 [ 2.055 -7.992  2.053]
 [ 2.055 -7.992  2.053]] [0 0 0 ... 0 0 0]

{'eval_accuracy': 0.499,
 'eval_f1': 0.6657771847898598,
 'eval_loss': 0.6932016611099243,
 'eval_precision': 0.499,
 'eval_recall': 1.0}
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Get different logits for each sequence/model prediction in therefore get meaningful metrics output on the evaluation dataset.

<!-- A clear and concise description of what you would expect to happen. -->


Put all models in the constants

# What does this PR do?

It was impossible to use all pretrained checkpoints in the tapas tokenzier file because they were not in the constants of the file. This PR fixes that.


Added TF TransfoXL Sequence Classification

This PR implements Sequence classification for TF TransfoXL model.
TFTransfoXLForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. GPT-1 ,GPT-2) do.

Fixes #7623

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

 @LysandreJik @jplu


Fix gradient clipping for Sharded DDP

# What does this PR do?

As mentioned in the discussion of #9156, `Trainer` does not do gradient clipping correctly when using a sharded optimizer. This PR fixes that, and also allows `Trainer` to not perform any gradient clipping (by passing `None` or `0` to the corresponding argument).


Add disclaimer to TAPAS rst file


Language modeling logging

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.0.0-rc-1
- Platform:Linux
- Python version:Python 3.7.9
- PyTorch version (GPU?):1.4.0
- Tensorflow version (GPU?):1.14.0
- Using GPU in script?:Yes
- Using distributed or parallel set-up in script?:Yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@sgugger
## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

I am new to the logging package and I would like to know if we can redirect all the contents occurring in the screen to a text file when I am running the language modeling script. I am not sure if this is the correct forum to ask but can you please help me with this?


Roberta python Tokenizer encodes differently across transformers==2.11 and transformers==4.0.1

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11 vs 4.0.1
- Platform: Any
- Python version: 3.8.3
- PyTorch version (GPU?): Any
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
  albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@mfuntowicz
## Information

I am attaching two samples below. In short, the word "suicide" gets encoded with different tokens across 2 different transformers versions. This leads to the same finetuned model behaving differently if using different transformers versions. Which is alarming in a sense.

![2_11](https://user-images.githubusercontent.com/22024955/102495115-5f11ef80-407e-11eb-8478-888f6a6ecf15.png)
![4](https://user-images.githubusercontent.com/22024955/102495118-60431c80-407e-11eb-9af8-1748894621c9.png)

Model I am using (Bert, XLNet ...): Roberta-base

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
Run the following code with transformers==2.11 and 4.0.1
`from transformers import AutoTokenizer`

`tokenizer = AutoTokenizer.from_pretrained('roberta-base', use_fast=False)`

`tokenizer.encode(' suicide ')`

`tokenizer.encode('suicide')`

`tokenizer.encode(' suicide')`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
A tokenizer for the same pretrained model, should tokenize words identically.
<!-- A clear and concise description of what you would expect to happen. -->


Add clipping to relative positional embedding

# What does this PR do?

This PR adds clipping to distance embedding as described in the paper [Improve Transformer Models with Better Relative Position Embeddings](https://arxiv.org/abs/2009.13658)
Without this addition, the model with relative positional embedding will return an error if the input length above max_position_embeddings param.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Fix mixed precision in TF models

# What does this PR do?

This PR aims to fix the mixed precision issues when `tf.keras.mixed_precision.experimental.set_policy()` is set to something else than `tf.float32`.  In the same page, this PR aims to fix some TFLite quantization issues.

Before to further continue this PR, the PR #9418 has to be merged.

Fixes # (issue)

#7052