- `transformers` version: 3.5.0
- Platform: jupyter notebook
- Python version: 3.6.9
- Tensorflow version (GPU?): 2.3.1
- Using GPU in script?: Single RTX 2080TI
- Model: "distilbert-base-multilingual-cased"

with tf.device('/device:GPU:0'):
    model.compile(optimizer=optimizer, loss=loss_fn)
    model.fit(train_dataset.batch(batch_size_train), epochs=1)

Code above works fine. Using TFtrainer with code below produces strategy error.
Note:
tf.config.list_physical_devices('GPU') gives [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Steps to reproduce the behavior:

from transformers import BertTokenizer, TFBertForTokenClassification
from transformers import __version__
from transformers import TFTrainer, TFTrainingArguments

label2id = {"False": 0, "True": 1}
bert_model = "distilbert-base-multilingual-cased"
cache_dir = "cache/distilbert"

model = TFBertForTokenClassification.from_pretrained(
    bert_model,
    cache_dir = cache_dir,
    num_labels = len(label2id),
    label2id = label2id,
    id2label = {v:k for k,v in label2id.items()}
)

training_args = TFTrainingArguments(
    output_dir='cp',
    num_train_epochs=1,
    per_device_train_batch_size=16,
    warmup_steps=500,
    weight_decay=0.05,
)

trainer = TFTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

    ValueError: Trying to create optimizer slot variable under the scope for tf.distribute.Strategy (<tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy object at 0x7effe7039588>), which is different from the scope used for the original variable (<tf.Variable 'tf_bert_for_token_classification/bert/embeddings/word_embeddings/weight:0' shape=(119547, 768) dtype=float32, numpy=
    array([[ 0.01447453, -0.03549159,  0.03377417, ..., -0.01235564,
             0.00624704, -0.01201372],
           [ 0.00689944,  0.00139387, -0.00716509, ...,  0.02127312,
            -0.00164859, -0.02350472],
           [ 0.00123599, -0.02220839, -0.01472212, ..., -0.02844208,
            -0.01958628,  0.01139562],
           ...,
           [ 0.03488934,  0.00115632,  0.0073231 , ...,  0.00768381,
            -0.02942067, -0.00667366],
           [ 0.00224815, -0.00895759,  0.0046453 , ...,  0.00379816,
             0.00176853, -0.01759749],
           [-0.02521203, -0.03274821, -0.00520367, ..., -0.01396327,
             0.0071948 , -0.01428833]], dtype=float32)>). Make sure the slot variables are created under the same strategy scope. This may happen if you're restoring from a checkpoint outside the scope


Model name 'facebook/rag-sequence-base/*' not found when running examples/rag/finetune.sh

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-38-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: True (Retriever is distributed)

### Who can help
@patrickvonplaten, @lhoestq

## Information

Model I am using (Bert, XLNet ...):

**facebook/rag-sequence-base**

The problem arises when using:
* [x ] the official example scripts: (give details below)
examples/rag/finetune.sh

The tasks I am working on is:
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

run `sh finetune.sh`
with
```
DATA_DIR=data_dir
OUTPUT_DIR=output_dir
MODEL_NAME_OR_PATH="facebook/rag-sequence-base"
```

gives:

**Model name 'facebook/rag-sequence-base/question_encoder_tokenizer' not found in model shortcut name list (facebook/dpr-question_encoder-single-nq-base). Assuming 'facebook/rag-sequence-base/question_encoder_tokenizer' is a path, a model identifier, or url to a directory containing tokenizer files**.
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/question_encoder_tokenizer/vocab.txt from cache at /h/asabet/.cache/torch/transformers/14d599f015518cd5b95b5d567b8c06b265dbbf04047e44b3654efd7cbbacb697.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/question_encoder_tokenizer/added_tokens.json from cache at None
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/question_encoder_tokenizer/special_tokens_map.json from cache at /h/asabet/.cache/torch/transformers/70614c7a84151409876eaaaecb3b5185213aa5c560926855e35753b9909f1116.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/question_encoder_tokenizer/tokenizer_config.json from cache at /h/asabet/.cache/torch/transformers/8ade9cf561f8c0a47d1c3785e850c57414d776b3795e21bd01e58483399d2de4.11f57497ee659e26f830788489816dbcb678d91ae48c06c50c9dc0e4438ec05b
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/question_encoder_tokenizer/tokenizer.json from cache at None
**Model name 'facebook/rag-sequence-base/generator_tokenizer' not found in model shortcut name list (facebook/bart-base, facebook/bart-large, facebook/bart-large-mnli, facebook/bart-large-cnn, facebook/bart-large-xsum, yjernite/bart_eli5). Assuming 'facebook/rag-sequence-base/generator_tokenizer' is a path, a model identifier, or url to a directory containing tokenizer files.**
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/generator_tokenizer/vocab.json from cache at /h/asabet/.cache/torch/transformers/3b9637b6eab4a48cf2bc596e5992aebb74de6e32c9ee660a27366a63a8020557.6a4061e8fc00057d21d80413635a86fdcf55b6e7594ad9e25257d2f99a02f4be
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/generator_tokenizer/merges.txt from cache at /h/asabet/.cache/torch/transformers/b2a6adcb3b8a4c39e056d80a133951b99a56010158602cf85dee775936690c6a.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/generator_tokenizer/added_tokens.json from cache at None
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/generator_tokenizer/special_tokens_map.json from cache at /h/asabet/.cache/torch/transformers/342599872fb2f45f954699d3c67790c33b574cc552a4b433fedddc97e6a3c58e.6e217123a3ada61145de1f20b1443a1ec9aac93492a4bd1ce6a695935f0fd97a
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/generator_tokenizer/tokenizer_config.json from cache at /h/asabet/.cache/torch/transformers/e5f72dc4c0b1ba585d7afb7fa5e3e52ff0e1f101e49572e2caaf38fab070d4d6.d596a549211eb890d3bb341f3a03307b199bc2d5ed81b3451618cbcb04d1f1bc
loading file https://s3.amazonaws.com/models.huggingface.co/bert/facebook/rag-sequence-base/generator_tokenizer/tokenizer.json from cache at None
Traceback (most recent call last):
  File "finetune.py", line 499, in <module>
    main(args)
  File "finetune.py", line 439, in main
    model: GenerativeQAModule = GenerativeQAModule(args)
  File "finetune.py", line 105, in __init__
    retriever = RagPyTorchDistributedRetriever.from_pretrained(hparams.model_name_or_path, config=config)
  File "/h/asabet/.local/lib/python3.6/site-packages/transformers/retrieval_rag.py", line 308, in from_pretrained
    config, question_encoder_tokenizer=question_encoder_tokenizer, generator_tokenizer=generator_tokenizer
  File "/scratch/ssd001/home/asabet/transformers/examples/rag/distributed_retriever.py", line 41, in __init__
    index=index,
**TypeError: __init__() got an unexpected keyword argument 'index'**

## Expected behavior
finetune.sh should launch and run


using multi_gpu consistently

As discussed [here](https://github.com/huggingface/transformers/pull/8341#issuecomment-722705833) this PR replaces
* `multiple_gpu`
* `multigpu`
with `multi_gpu` for consistency

There is no functionality change otherwise.

I did repo-wide:
```
find . -type d -name ".git" -prune -o -type f -exec perl -pi -e 's#(multiple_gpu|multigpu)#multi_gpu#g' {} \;
```

@LysandreJik, @sgugger


[marian.rst] remove combined lines


Add missing import

# What does this PR do?
Fix a missing import for TF auto.


Dropout p is changing after loading

## Environment info
- `transformers` version: 3.5.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?:
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik,  @sgugger

## Information
Model I am using (Bert, XLNet ...): Bert, Roberta

The problem arises when using:
* [ *] the official example scripts: Using information given in this link: https://huggingface.co/transformers/master/custom_datasets.html

The tasks I am working on is:
* [ *] my own task or dataset: text classification

## To reproduce

Steps to reproduce the behavior:

1. I'm trying to change dropout probability. I'm using one of these methods for Bert instance:
```python
model.classifier.dropout.p=0.7
model.classifier.dropout = nn.Dropout(0.7)
```

2. After training is completed, model is saved

```python
model.save_pretrained('xxx/bert')
```

3. Model is loaded in another session using this code snippet. But after loading, model.classifier.dropout.p is changing to 0.1 which is in the config file.
```python
model = BertForSequenceClassification.from_pretrained("xxx/bert",
    num_labels = 3,
    output_attentions = False,
    output_hidden_states = False,
)
```

## Expected behavior
Dropout p is changing to default value after loading the model. But the model is modified so that it shouldn't do that behavior


Models fine-tuned with gradient checkpointing (=True) fails to export to ONXX

## Environment info
- `transformers` version: 3.5.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help

Hi @LysandreJik and @patrickvonplaten, I hope I have tagged the right person. If not please untag yourself and tag the right person.

## Information

Model I am using (Bert):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

It's a super simple script that uses gradient check-pointing with BERT.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

A dummy data

## To reproduce

I have made a reproducible google collab here -> https://colab.research.google.com/drive/1tUpIzbugZ4xPz6eAOJtZT-fGww9LemwN?usp=sharing

Steps to reproduce the behavior:

1. Open the notebook
2. Runtime->Run all

Error thrown:
```python
RuntimeError                              Traceback (most recent call last)

<ipython-input-5-2702a59a9c3e> in <module>()
      7         tokenizer=tokenizer,  # <-- CHANGED: add tokenizer
      8         output=Path("onnx/bert-base-cased.onnx"),
----> 9         opset=11)
     10
     11 # Tensorflow

4 frames

/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, opset_version, _retain_param_name, do_constant_folding, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, fixed_batch_size, custom_opsets, add_node_names, enable_onnx_checker, use_external_data_format, onnx_shape_inference, use_new_jit_passes)
    648                     params_dict, opset_version, dynamic_axes, defer_weight_export,
    649                     operator_export_type, strip_doc_string, val_keep_init_as_ip, custom_opsets,
--> 650                     val_add_node_names, val_use_external_data_format, model_file_location)
    651             else:
    652                 proto, export_map = graph._export_onnx(

RuntimeError: ONNX export failed: Couldn't export Python operator CheckpointFunction

```


CUDA out of memory (ALBERT)!!

# ❓ Questions & Help
While using `albert-base-v2` to train my model, I got the following problem:

```
#first call
outputs = self.albert(input_ids,
                              attention_mask=attention_mask,
                              token_type_ids=token_type_ids,
                              position_ids=position_ids,
                              head_mask=head_mask,
                              inputs_embeds=inputs_embeds)

sequence_output = outputs[0]

context_mask = token_type_ids * attention_mask
question_mask = ((1 - context_mask) * attention_mask)

#second call
question, _ = self.albert(input_ids, attention_mask=question_mask, token_type_ids=token_type_ids)

#third call
context, _ = self.albert(input_ids, attention_mask=context_mask, token_type_ids=token_type_ids)
```

While calling `self.albert()` thrice, the memory it consumes will multiply by 3.
So that I must change my batch_size to 4, it's so bad!
Is it a BUG or a feature?
Even `albert-base` so large?


Question template

# What does this PR do?

This PR updates the question template to insist a bit more on users using the forum for questions.


Model sharing rst

Update the model sharing RST for the new model versioning.


login to huggingface forum

hey.
I'm not sure where to post it, so my apologise in advance.

I'm trying to login to huggingface forum, but it return me the message: "Unauthorized".
what can I do?
(I created a user..)

thanks.


[T5Tokenizer] fix t5 token type ids

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #7840

T5 does not use token type ids. Nevertheless, the T5Tokenizer should analogs to RobertaTokenizer return all [0] for the `token_type_ids`.

Functions and Tests are added for T5TokenizerFast and T5Tokenizer.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Windows dev section in the contributing file

# What does this PR do?

This PR adds a section for people who wants to contribute from a Windows environment.


[T5 Tokenizer] Fix t5 special tokens

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #5142, #8109

T5FastToeknzier and T5SlowTokenizer have different behaviors for special tokens as shown in the issue above.
This PR fixes the slow T5 tokenizer and adds a test making sure that Fast and Slow tokenizer have the same behavior.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Support serialized tokenizer in AutoTokenizer

# What does this PR do?

Addresses issue: https://github.com/huggingface/transformers/issues/7293

With these changes, the `AutoTokenizer.from_pretrained()` supports also loading a tokenizer, which was saved with [🤗 Tokenizers](https://github.com/huggingface/tokenizers) library.
Example:
```python
from tokenizers import CharBPETokenizer
tokenizer = CharBPETokenizer()
tokenizer.save('./char-bpe.json')

from transformers import AutoTokenizer
my_tokenizer = AutoTokenizer.from_pretrained('./char-bpe.json', use_fast=True)
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed.
Also: @mfuntowicz (tokenizers), @thomwolf (due to https://github.com/huggingface/transformers/pull/7659)


# Replaced unnecessary iadd operations on lists in tokenization_utils.py with proper list methods

@mfuntowicz

Previously, unnecessarily many list objects are created because of list updates through iadd operations.
This is bad for the following reasons.
* It slows down the program.
* It's a substandard style.

Regarding the slowing down, please see the following snippets.
```
l = []
for i in range(10**6):
    l.append(i)
# Takes 0.13282 seconds, on average, on my machine
```

```
l = []
for i in range(10**6):
    l += [i]  # this creates a new list [i] every iteration
# Takes 0.14698 seconds, on average, on my machine
```

The previous style is considered bad since it's confusing. It is easy to think that `l += [i]` has the same semantics as `l = l + [i]`, which is not at all the case. To see this, run the following code.
```
 l = []
for i in range(10**6):
    l = l + [i]  # This replaces the existing list with a new list (l + [i]) every iteration
```

The fact that the existing list is mutated is more clearly expressed in the new code, and, to my best knowledge, all the python standard library code prefer the style of the new code.


Add auto next sentence prediction

# What does this PR do?

This PR adds auto models for the next sentence prediction task.


Get Scores for each NE Label

I'm running the run_ner.py script to use the bert-base-german-cased transformer model in the token classification task to train it on custom NE labels and make predictions for German documents. I have 11 labels in total.

I wondered if there is any way to get prediction results (meaning loss, accuracy, precision, etc.) not only for the whole task, but for each label individually. This would make it easier to compare the real perfomance of the model. So that each label has results like:

```
eval_loss = 0.07476427406072617
eval_accuracy_score = 0.9818217086485438
eval_precision = 0.6756756756756757
eval_recall = 0.676378772112383
eval_f1 = 0.6760270410816434
```

Reason I'm asking: the O-Labels (the only thing I kept from BIO tagging) are of course the majority of all labels, therefore the accuracy is quite high as the model correctly predicts most of them, but as a consequence the scores for the real labels get lost in the statistical noise.

Is there any way to achieve this?
Thanks in advance.


RAG: Explanation on Retriever Variables.

Can you please explain what these terms in the RAG retrieval mean?

1. config.index_name
2. config.index_path


[examples] better PL version check

`pkg_resources.require(f"{pkg}>={min_ver}")` does a great job of checking the minimal required versions at runtime, but I wasn't aware that it checks that the dependencies meet their requirements too! So I started getting a false alarm about needing `pytorch-lightning=1.0.4` when I already had a higher version.

The problem was in:
```
$ python -c 'import pkg_resources; pkg_resources.require("pytorch_lightning>=1.0.4")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pkg_resources/__init__.py", line 884, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pkg_resources/__init__.py", line 775, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (torch 1.8.0.dev20201106+cu110 (/mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages), Requirement.parse('torch<1.8,>=1.3'), {'pytorch-lightning'})
```

Long story short, currently PL explicitly excludes pytorch-1.8 in its dependency list: https://github.com/PyTorchLightning/pytorch-lightning/issues/4596 - which leads to this problem. When I upgrade PL pip uninstalls `pytorch-1.8` - thanks, but no thanks - rtx-3090 doesn't work with pytorch < 1.8. So I install it back and now I get the failure above. Except in the current code it's masked by the `try/except` block which hides the actual problem. So this is not good.

This PR rewrites the check in a way that doesn't check whether the dependencies of the package in questions are in order, and only checks that the minimal version is correct.

@sshleifer


Add missing tasks to `pipeline` docstring

model card

Related to pretrained model cards

I added missing tasks to `pipeline` docstring.

Also, I fixed some typos I found.


Set num_beams=4 for all Helsinki-NLP models

Currently it is 6. Empirically, I tested 77 random models and num_beams=4 was about 50% faster and on average slightly higher BLEU (22.5 vs 22.4).

We also have @jorgtied 's approval for the change.
On slack, he wrote

>  Again, no systematic evaluation - more like a feeling. I had the impression that 1 or 2 is worse and I didn’t want to set 10 or 12 that I have seen otherwise, because it may slow down things quite substantially. If you make some more tests then let me know what you will find … Thanks!

There are about 1300 affected model's, so this feels like the type of thing @patrickvonplaten 's script could do well.


Wrong files names in model list for "xlm-roberta-large-finetuned-conll03-german"

Hi, the names of the model files for "xlm-roberta-large-finetuned-conll03-german" are incorrect and have to model name as prefix.

Example:
xlm-roberta-large-finetuned-conll03-german-config.json instead of config.json


Check all models are in an auto class

# What does this PR do?

Following up from @patrickvonplaten fixes, this PR adds a script to check all models are in an auto class. This way we will get a CI error if a newly added model ends up forgotten :-)


Electra multi-gpu pretraining.

Hi, I am pretraining the Electra model with my own data, for now, I am pretraining using one GPU in my machine. Can we use multi GPUs to pretrain Electra? Thanks for your reply


Fix bart shape comment

fixes #8384
Before transpose , shape of x and encoder_hidden_states are both (BS, seq_len, model_dim) to me.


[docs] [testing] gpu decorators table

This PR adds a table of gpu requirement decorators that perhaps is easier to grasp quickly in addition to the prose version.
(based on the discussion [here](https://github.com/huggingface/transformers/pull/8341#issuecomment-723705104))

@sgugger


[docs] improve bart/marian/mBART/pegasus docs

+ Give example of bart mask filling
+ Link to training scripts where applicable
+ Clarify Marian naming scheme a bit.


Deprecate old data/metrics functions

# What does this PR do?

This PR deprecates the old data/metrics utils we used now that we have some examples of scripts leveraging the Datasets library to point at. The idea is to eventually remove those from the library but keep them somewhere in the examples folder as utilities, so the old scripts can still be run.


Bump tokenizers

# What does this PR do?

Bump the version of tokenizers to the last release. This fixes some bugs in `XLNetTokenizerFast`.


[docs] remove sshleifer from issue-template :(

+ Removes sshleifer from issue-templates :(
+ For previous @sshleifer stuff, if it's in `src/` I put  @patrickvonplaten, if it's `examples/seq2seq` I put @patil-suraj.
+ @stas00 if you want to take over any thing or feel comfortable being the point person for certain things, feel free to suggest.


Changing XLNet default from not using memories to 512 context size following paper

In #8317, we found out that calling `XLNetLMHeadModel.from_pretrained(mem_len=384)` still produced the FutureWarning that announces that the default configuration will change in 3.5.0, as the config first gets initialized with `mem_len=0` then `mem_len` gets changed. This PR moves the warning to the `forward` pass in the model to avoid this.


Does MBartTokenizer remove the parameter decoder_input_ids?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:3.4.0
- Platform:Google Colab
- Python version:3.7
- PyTorch version (GPU?):1.7.0+cu101
- Tensorflow version (GPU?):2.x
- Using GPU in script?: no
- Using distributed or parallel set-up in script?:no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): mbart

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
```python
example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"
expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria"
batch = MBartTokenizer.from_pretrained('facebook/mbart-large-cc25').prepare_seq2seq_batch(example_english_phrase, src_lang="en_XX", tgt_lang="ro_RO", tgt_texts=expected_translation_romanian)
input_ids = batch["input_ids"]
target_ids = batch["decoder_input_ids"]
```
Steps to reproduce the behavior:
```python
KeyError                                  Traceback (most recent call last)
<ipython-input-11-b3eedaf10c3e> in <module>()
      3 batch = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro').prepare_seq2seq_batch(example_english_phrase, src_lang="en_XX", tgt_lang="ro_RO", tgt_texts=expected_translation_romanian)
      4 input_ids = batch["input_ids"]
----> 5 target_ids = batch["decoder_input_ids"]

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py in __getitem__(self, item)
    232         """
    233         if isinstance(item, str):
--> 234             return self.data[item]
    235         elif self._encodings is not None:
    236             return self._encodings[item]

KeyError: 'decoder_input_ids'
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

<!-- A clear and concise description of what you would expect to happen. -->


[Tests] Add Common Test for Training + Fix a couple of bugs

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds an aggressive test to check that all models that should be trainable can perform a backward pass of their loss output. In addition, a test for training with gradient checkpointing is added as well. The motivation comes from this error: https://github.com/huggingface/transformers/pull/7562#issuecomment-723887221  - the PR introduced broke gradient checkpointing without any test noticing it.

To make the test applicable for all models, some `ModelTests` have to overwrite the `_prepare_for_class` function.
In addition, some cleaning was done `ForPretraining` was renamed to `ForPreTraining`; an `AutoModelForNextSentencePredicition` was added, ...

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


[seq2seq] translation tpu example doesnt work

Hi im trying to run the `train_distil_marian_enro_tpu.sh` example in collab/kaggle tpus and for some reason it gives me the following output:
@sshleifer
```
Exception in device=TPU:0: Cannot access data pointer of Tensor that doesn't have storage
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
    fn(gindex, *args)
  File "/content/transformers/examples/seq2seq/finetune_trainer.py", line 300, in _mp_fn
    main()
  File "/content/transformers/examples/seq2seq/finetune_trainer.py", line 249, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 776, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1128, in training_step
    loss.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Cannot access data pointer of Tensor that doesn't have storage
  0% 0/7158 [00:44<?, ?it/s]
Traceback (most recent call last):
  File "xla_spawn.py", line 72, in <module>
    main()
  File "xla_spawn.py", line 68, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
    start_method=start_method)
  File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code

```
Related to this issue  #https://github.com/pytorch/xla/issues/929

Not sure how to solve it.  Thanks!


continuing fine-tuning from the last checkpoint

Hello,

While fine-tuning BERT on the custom data using "run_language_modeling.py" script, due to memory issue the fine-tuning stopped in the middle. However, I tried to resume the fine-tuning from the last checkpoint. But, I came across with the following error:

```
python run_language_modeling.py     --output_dir=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results     --model_type=bert     --model_name_or_path=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm  --per_gpu_train_batch_size=4
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ai-students/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
11/08/2020 22:40:18 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 2, distributed training: False, 16-bits training: False
11/08/2020 22:40:18 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results', overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=False, evaluation_strategy=<EvaluationStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=4, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Nov08_22-40-18_aistudents-msi', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None)
Traceback (most recent call last):
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/configuration_utils.py", line 387, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_language_modeling.py", line 355, in <module>
    main()
  File "run_language_modeling.py", line 236, in main
    config = AutoConfig.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/configuration_auto.py", line 329, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/ai-students/anaconda3/envs/env_nesara/lib/python3.6/site-packages/transformers/configuration_utils.py", line 396, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/'. Make sure that:

- '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/' is a correct model identifier listed on 'https://huggingface.co/models'

- or '/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/' is the correct path to a directory containing a config.json file

```

Command used to fine-tune from the last checkpoint is as follows:

```
python run_language_modeling.py     --output_dir=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results     --model_type=bert     --model_name_or_path=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/new_results/result_dir/checkpoint_37000/     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm  --per_gpu_train_batch_size=4

```

Here is the command used to fine-tune BERT earlier:

```

python run_language_modeling.py     --output_dir=/media/ai-students/Data/Nesara/Bert_MLM_fine_tune/result_dir     --model_type=bert     --model_name_or_path=bert-base-cased     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm  --per_gpu_train_batch_size=4

```

@sgugger Could anyone please let me know on how to resume fine-tuning from the last checkpoint?

Thanks in advance :)


[s2s/distill] remove run_distiller.sh, fix xsum script


Tokenizer return nothing instead of unk for certain token?

## Environment info

- `transformers` version: 3.1.0
- Platform: pytorch
- Python version: python 3.6.9
- PyTorch version (GPU?):  1.6.0
- Tensorflow version (GPU?): na
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

Model I am using (tokenizer && model == google/electra-large-generator):

The problem arises when using:
* [x ] my own modified scripts: (give details below)
self.fill_mask  = pipeline("fill-mask", model="google/electra-large-generator",\
                                              tokenizer="google/electra-large-generator")

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)
a simple fill mask type of task with transformer pipline

## To reproduce

Steps to reproduce the behavior:

1. fill_mask  = pipeline("fill-mask", model="google/electra-large-generator",\
                                              tokenizer="google/electra-large-generator")
2. fill_mask.tokenizer.tokenize(""" ̈  """)

3. output is []

## Expected behavior

In this case the missing of this punctuation would cause some downstream usage to fail due to IndexError. The problem might be intrinsic to this particular tokenizer, maybe it is worthwhile to raise a warning/error or returning a unk token? Thanks!


comet_ml init weirdness

Using a slightly bogus invocation of the following
```
cd examples/seq2seq
PYTHONPATH="../../src" BS=2 python finetune.py --data_dir cnn_dm --do_predict --do_train --eval_batch_size $BS --fp16 --fp16_opt_level O1--freeze_embeds --freeze_encoder --gpus 1 --gradient_accumulation_steps 1 --learning_rate 3e-5 --max_target_length 142 --model_name_or_path sshleifer/student_cnn_12_6 --n_val 500 --num_train_epochs 2 --output_dir distilbart-cnn-12-6 --tokenizer_name facebook/bart-large --train_batch_size $BS --val_check_interval 0.25 --val_max_target_length 142 --warmup_steps 500
```
I get:
```
Traceback (most recent call last):
  File "finetune.py", line 18, in <module>
    from callbacks import Seq2SeqLoggingCallback, get_checkpoint_callback, get_early_stopping_callback
  File "/mnt/nvme1/code/huggingface/transformers-comet_ml/examples/seq2seq/callbacks.py", line 11, in <module>
    from utils import save_json
  File "/mnt/nvme1/code/huggingface/transformers-comet_ml/examples/seq2seq/utils.py", line 22, in <module>
    from transformers import BartTokenizer, EvalPrediction, PreTrainedTokenizer, T5Tokenizer
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/integrations.py", line 17, in <module>
    if comet_ml.config.get_config("comet.api_key"):
AttributeError: module 'comet_ml' has no attribute 'config'
```
no idea why this happens. this PR adds a band-aid - and then once I apply it - the real error shows up `$BS` wasn't defined - i.e. my args had an issue. But I need to see the real error and not some totally unrelated `comet_ml` error.

perhaps something else needs to be fixed.

While we are at it, once again I ended up with `comet_ml` w/o explicitly installing it.

So I again get to enjoy the incessant:
```
comet_ml is installed but `COMET_API_KEY` is not set.
```
:(

```
pipdeptree --reverse --packages comet_ml
```
doesn't give me the parent who pulled it in which is very odd, since it works for other packages.

Is there a different way to trace what pulled in a certain package?

If I install/update it explicitly it appears I already have the latest version:
```
pip install comet_ml -U
Requirement already up-to-date: comet_ml in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (3.2.5)
```

@sgugger, @dsblank


Bug fix for permutation language modelling

Addresses a bug in permutation language modelling data collator, where `&` is used instead of `|` to compute the non-functional token mask (tokens excluding [PAD], [SEP], [CLS]). For verification, may refer to original XLNet code (https://github.com/zihangdai/xlnet/blob/master/data_utils.py#L602).

Addresses #6812 (further investigation needed, however)

@patrickvonplaten @LysandreJik


updating tag for exbert viz

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


All the weights of the model checkpoint at roberta-base were not used when initializing

# How to use the model checkpoint to initializing my own roberta?

**I wrote a class named `MyRoberta` at `modeling_roberta.py`, the main code is just like below:**
```python
class MyRoberta(RobertaPreTrainedModel):

    def __init__(self, config):
        super(MyRoberta, self).__init__(config)
        # self.bert = RobertaModel.from_pretrained("model/roberta_base/", config=config)
        self.bert = RobertaModel(config)
        self.cls = RobertaLMHead(config)

    # ...

```

**and then I initialized it using the code below:**
```python
    config = RobertaConfig.from_pretrained("roberta-base")
    logging("Model config {}".format(config))
    model = MyRoberta.from_pretrained("roberta-base", mirror="tuna", cache_dir="./model/", config=config)
```

**however, one warning message shows that i didn't using the pretrained parameters.**
```
Some weights of the model checkpoint at roberta-base were not used when initializing MyRoberta: ['roberta.embeddings.word_embeddings.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.embeddings.token_type_embeddings.weight', 'roberta.embeddings.LayerNorm.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.0.attention.self.query.weight', 'roberta.encoder.layer.0.attention.self.query.bias', 'roberta.encoder.layer.0.attention.self.key.weight', 'roberta.encoder.layer.0.attention.self.key.bias', 'roberta.encoder.layer.0.attention.self.value.weight', 'roberta.encoder.layer.0.attention.self.value.bias', 'roberta.encoder.layer.0.attention.output.dense.weight', 'roberta.encoder.layer.0.attention.output.dense.bias', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.intermediate.dense.weight', 'roberta.encoder.layer.0.intermediate.dense.bias', 'roberta.encoder.layer.0.output.dense.weight', 'roberta.encoder.layer.0.output.dense.bias', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.self.query.weight', 'roberta.encoder.layer.1.attention.self.query.bias', 'roberta.encoder.layer.1.attention.self.key.weight', 'roberta.encoder.layer.1.attention.self.key.bias', 'roberta.encoder.layer.1.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.value.bias', 'roberta.encoder.layer.1.attention.output.dense.weight', 'roberta.encoder.layer.1.attention.output.dense.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.intermediate.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.bias', 'roberta.encoder.layer.1.output.dense.weight', 'roberta.encoder.layer.1.output.dense.bias', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.2.attention.self.query.weight', 'roberta.encoder.layer.2.attention.self.query.bias', 'roberta.encoder.layer.2.attention.self.key.weight', 'roberta.encoder.layer.2.attention.self.key.bias', 'roberta.encoder.layer.2.attention.self.value.weight', 'roberta.encoder.layer.2.attention.self.value.bias', 'roberta.encoder.layer.2.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.output.dense.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.2.intermediate.dense.weight', 'roberta.encoder.layer.2.intermediate.dense.bias', 'roberta.encoder.layer.2.output.dense.weight', 'roberta.encoder.layer.2.output.dense.bias', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.query.weight', 'roberta.encoder.layer.3.attention.self.query.bias', 'roberta.encoder.layer.3.attention.self.key.weight', 'roberta.encoder.layer.3.attention.self.key.bias', 'roberta.encoder.layer.3.attention.self.value.weight', 'roberta.encoder.layer.3.attention.self.value.bias', 'roberta.encoder.layer.3.attention.output.dense.weight', 'roberta.encoder.layer.3.attention.output.dense.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.intermediate.dense.weight', 'roberta.encoder.layer.3.intermediate.dense.bias', 'roberta.encoder.layer.3.output.dense.weight', 'roberta.encoder.layer.3.output.dense.bias', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.self.query.weight', 'roberta.encoder.layer.4.attention.self.query.bias', 'roberta.encoder.layer.4.attention.self.key.weight', 'roberta.encoder.layer.4.attention.self.key.bias', 'roberta.encoder.layer.4.attention.self.value.weight', 'roberta.encoder.layer.4.attention.self.value.bias', 'roberta.encoder.layer.4.attention.output.dense.weight', 'roberta.encoder.layer.4.attention.output.dense.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.intermediate.dense.weight', 'roberta.encoder.layer.4.intermediate.dense.bias', 'roberta.encoder.layer.4.output.dense.weight', 'roberta.encoder.layer.4.output.dense.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.query.weight', 'roberta.encoder.layer.5.attention.self.query.bias', 'roberta.encoder.layer.5.attention.self.key.weight', 'roberta.encoder.layer.5.attention.self.key.bias', 'roberta.encoder.layer.5.attention.self.value.weight', 'roberta.encoder.layer.5.attention.self.value.bias', 'roberta.encoder.layer.5.attention.output.dense.weight', 'roberta.encoder.layer.5.attention.output.dense.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.intermediate.dense.weight', 'roberta.encoder.layer.5.intermediate.dense.bias', 'roberta.encoder.layer.5.output.dense.weight', 'roberta.encoder.layer.5.output.dense.bias', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.query.bias', 'roberta.encoder.layer.6.attention.self.key.weight', 'roberta.encoder.layer.6.attention.self.key.bias', 'roberta.encoder.layer.6.attention.self.value.weight', 'roberta.encoder.layer.6.attention.self.value.bias', 'roberta.encoder.layer.6.attention.output.dense.weight', 'roberta.encoder.layer.6.attention.output.dense.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.encoder.layer.6.intermediate.dense.weight', 'roberta.encoder.layer.6.intermediate.dense.bias', 'roberta.encoder.layer.6.output.dense.weight', 'roberta.encoder.layer.6.output.dense.bias', 'roberta.encoder.layer.6.output.LayerNorm.weight', 'roberta.encoder.layer.6.output.LayerNorm.bias', 'roberta.encoder.layer.7.attention.self.query.weight', 'roberta.encoder.layer.7.attention.self.query.bias', 'roberta.encoder.layer.7.attention.self.key.weight', 'roberta.encoder.layer.7.attention.self.key.bias', 'roberta.encoder.layer.7.attention.self.value.weight', 'roberta.encoder.layer.7.attention.self.value.bias', 'roberta.encoder.layer.7.attention.output.dense.weight', 'roberta.encoder.layer.7.attention.output.dense.bias', 'roberta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.encoder.layer.7.intermediate.dense.weight', 'roberta.encoder.layer.7.intermediate.dense.bias', 'roberta.encoder.layer.7.output.dense.weight', 'roberta.encoder.layer.7.output.dense.bias', 'roberta.encoder.layer.7.output.LayerNorm.weight', 'roberta.encoder.layer.7.output.LayerNorm.bias', 'roberta.encoder.layer.8.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.query.bias', 'roberta.encoder.layer.8.attention.self.key.weight', 'roberta.encoder.layer.8.attention.self.key.bias', 'roberta.encoder.layer.8.attention.self.value.weight', 'roberta.encoder.layer.8.attention.self.value.bias', 'roberta.encoder.layer.8.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.output.dense.bias', 'roberta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.encoder.layer.8.intermediate.dense.weight', 'roberta.encoder.layer.8.intermediate.dense.bias', 'roberta.encoder.layer.8.output.dense.weight', 'roberta.encoder.layer.8.output.dense.bias', 'roberta.encoder.layer.8.output.LayerNorm.weight', 'roberta.encoder.layer.8.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.query.weight', 'roberta.encoder.layer.9.attention.self.query.bias', 'roberta.encoder.layer.9.attention.self.key.weight', 'roberta.encoder.layer.9.attention.self.key.bias', 'roberta.encoder.layer.9.attention.self.value.weight', 'roberta.encoder.layer.9.attention.self.value.bias', 'roberta.encoder.layer.9.attention.output.dense.weight', 'roberta.encoder.layer.9.attention.output.dense.bias', 'roberta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.intermediate.dense.weight', 'roberta.encoder.layer.9.intermediate.dense.bias', 'roberta.encoder.layer.9.output.dense.weight', 'roberta.encoder.layer.9.output.dense.bias', 'roberta.encoder.layer.9.output.LayerNorm.weight', 'roberta.encoder.layer.9.output.LayerNorm.bias', 'roberta.encoder.layer.10.attention.self.query.weight', 'roberta.encoder.layer.10.attention.self.query.bias', 'roberta.encoder.layer.10.attention.self.key.weight', 'roberta.encoder.layer.10.attention.self.key.bias', 'roberta.encoder.layer.10.attention.self.value.weight', 'roberta.encoder.layer.10.attention.self.value.bias', 'roberta.encoder.layer.10.attention.output.dense.weight', 'roberta.encoder.layer.10.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.encoder.layer.10.intermediate.dense.weight', 'roberta.encoder.layer.10.intermediate.dense.bias', 'roberta.encoder.layer.10.output.dense.weight', 'roberta.encoder.layer.10.output.dense.bias', 'roberta.encoder.layer.10.output.LayerNorm.weight', 'roberta.encoder.layer.10.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.self.query.weight', 'roberta.encoder.layer.11.attention.self.query.bias', 'roberta.encoder.layer.11.attention.self.key.weight', 'roberta.encoder.layer.11.attention.self.key.bias', 'roberta.encoder.layer.11.attention.self.value.weight', 'roberta.encoder.layer.11.attention.self.value.bias', 'roberta.encoder.layer.11.attention.output.dense.weight', 'roberta.encoder.layer.11.attention.output.dense.bias', 'roberta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.encoder.layer.11.intermediate.dense.weight', 'roberta.encoder.layer.11.intermediate.dense.bias', 'roberta.encoder.layer.11.output.dense.weight', 'roberta.encoder.layer.11.output.dense.bias', 'roberta.encoder.layer.11.output.LayerNorm.weight', 'roberta.encoder.layer.11.output.LayerNorm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing MyRoberta from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing MyRoberta from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of MyRoberta were not initialized from the model checkpoint at roberta-base and are newly initialized: ['roberta.bert.embeddings.position_ids', 'roberta.bert.embeddings.word_embeddings.weight', 'roberta.bert.embeddings.position_embeddings.weight', 'roberta.bert.embeddings.token_type_embeddings.weight', 'roberta.bert.embeddings.LayerNorm.weight', 'roberta.bert.embeddings.LayerNorm.bias', 'roberta.bert.encoder.layer.0.attention.self.query.weight', 'roberta.bert.encoder.layer.0.attention.self.query.bias', 'roberta.bert.encoder.layer.0.attention.self.key.weight', 'roberta.bert.encoder.layer.0.attention.self.key.bias', 'roberta.bert.encoder.layer.0.attention.self.value.weight', 'roberta.bert.encoder.layer.0.attention.self.value.bias', 'roberta.bert.encoder.layer.0.attention.output.dense.weight', 'roberta.bert.encoder.layer.0.attention.output.dense.bias', 'roberta.bert.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.0.intermediate.dense.weight', 'roberta.bert.encoder.layer.0.intermediate.dense.bias', 'roberta.bert.encoder.layer.0.output.dense.weight', 'roberta.bert.encoder.layer.0.output.dense.bias', 'roberta.bert.encoder.layer.0.output.LayerNorm.weight', 'roberta.bert.encoder.layer.0.output.LayerNorm.bias', 'roberta.bert.encoder.layer.1.attention.self.query.weight', 'roberta.bert.encoder.layer.1.attention.self.query.bias', 'roberta.bert.encoder.layer.1.attention.self.key.weight', 'roberta.bert.encoder.layer.1.attention.self.key.bias', 'roberta.bert.encoder.layer.1.attention.self.value.weight', 'roberta.bert.encoder.layer.1.attention.self.value.bias', 'roberta.bert.encoder.layer.1.attention.output.dense.weight', 'roberta.bert.encoder.layer.1.attention.output.dense.bias', 'roberta.bert.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.1.intermediate.dense.weight', 'roberta.bert.encoder.layer.1.intermediate.dense.bias', 'roberta.bert.encoder.layer.1.output.dense.weight', 'roberta.bert.encoder.layer.1.output.dense.bias', 'roberta.bert.encoder.layer.1.output.LayerNorm.weight', 'roberta.bert.encoder.layer.1.output.LayerNorm.bias', 'roberta.bert.encoder.layer.2.attention.self.query.weight', 'roberta.bert.encoder.layer.2.attention.self.query.bias', 'roberta.bert.encoder.layer.2.attention.self.key.weight', 'roberta.bert.encoder.layer.2.attention.self.key.bias', 'roberta.bert.encoder.layer.2.attention.self.value.weight', 'roberta.bert.encoder.layer.2.attention.self.value.bias', 'roberta.bert.encoder.layer.2.attention.output.dense.weight', 'roberta.bert.encoder.layer.2.attention.output.dense.bias', 'roberta.bert.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.2.intermediate.dense.weight', 'roberta.bert.encoder.layer.2.intermediate.dense.bias', 'roberta.bert.encoder.layer.2.output.dense.weight', 'roberta.bert.encoder.layer.2.output.dense.bias', 'roberta.bert.encoder.layer.2.output.LayerNorm.weight', 'roberta.bert.encoder.layer.2.output.LayerNorm.bias', 'roberta.bert.encoder.layer.3.attention.self.query.weight', 'roberta.bert.encoder.layer.3.attention.self.query.bias', 'roberta.bert.encoder.layer.3.attention.self.key.weight', 'roberta.bert.encoder.layer.3.attention.self.key.bias', 'roberta.bert.encoder.layer.3.attention.self.value.weight', 'roberta.bert.encoder.layer.3.attention.self.value.bias', 'roberta.bert.encoder.layer.3.attention.output.dense.weight', 'roberta.bert.encoder.layer.3.attention.output.dense.bias', 'roberta.bert.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.3.intermediate.dense.weight', 'roberta.bert.encoder.layer.3.intermediate.dense.bias', 'roberta.bert.encoder.layer.3.output.dense.weight', 'roberta.bert.encoder.layer.3.output.dense.bias', 'roberta.bert.encoder.layer.3.output.LayerNorm.weight', 'roberta.bert.encoder.layer.3.output.LayerNorm.bias', 'roberta.bert.encoder.layer.4.attention.self.query.weight', 'roberta.bert.encoder.layer.4.attention.self.query.bias', 'roberta.bert.encoder.layer.4.attention.self.key.weight', 'roberta.bert.encoder.layer.4.attention.self.key.bias', 'roberta.bert.encoder.layer.4.attention.self.value.weight', 'roberta.bert.encoder.layer.4.attention.self.value.bias', 'roberta.bert.encoder.layer.4.attention.output.dense.weight', 'roberta.bert.encoder.layer.4.attention.output.dense.bias', 'roberta.bert.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.4.intermediate.dense.weight', 'roberta.bert.encoder.layer.4.intermediate.dense.bias', 'roberta.bert.encoder.layer.4.output.dense.weight', 'roberta.bert.encoder.layer.4.output.dense.bias', 'roberta.bert.encoder.layer.4.output.LayerNorm.weight', 'roberta.bert.encoder.layer.4.output.LayerNorm.bias', 'roberta.bert.encoder.layer.5.attention.self.query.weight', 'roberta.bert.encoder.layer.5.attention.self.query.bias', 'roberta.bert.encoder.layer.5.attention.self.key.weight', 'roberta.bert.encoder.layer.5.attention.self.key.bias', 'roberta.bert.encoder.layer.5.attention.self.value.weight', 'roberta.bert.encoder.layer.5.attention.self.value.bias', 'roberta.bert.encoder.layer.5.attention.output.dense.weight', 'roberta.bert.encoder.layer.5.attention.output.dense.bias', 'roberta.bert.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.5.intermediate.dense.weight', 'roberta.bert.encoder.layer.5.intermediate.dense.bias', 'roberta.bert.encoder.layer.5.output.dense.weight', 'roberta.bert.encoder.layer.5.output.dense.bias', 'roberta.bert.encoder.layer.5.output.LayerNorm.weight', 'roberta.bert.encoder.layer.5.output.LayerNorm.bias', 'roberta.bert.encoder.layer.6.attention.self.query.weight', 'roberta.bert.encoder.layer.6.attention.self.query.bias', 'roberta.bert.encoder.layer.6.attention.self.key.weight', 'roberta.bert.encoder.layer.6.attention.self.key.bias', 'roberta.bert.encoder.layer.6.attention.self.value.weight', 'roberta.bert.encoder.layer.6.attention.self.value.bias', 'roberta.bert.encoder.layer.6.attention.output.dense.weight', 'roberta.bert.encoder.layer.6.attention.output.dense.bias', 'roberta.bert.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.6.intermediate.dense.weight', 'roberta.bert.encoder.layer.6.intermediate.dense.bias', 'roberta.bert.encoder.layer.6.output.dense.weight', 'roberta.bert.encoder.layer.6.output.dense.bias', 'roberta.bert.encoder.layer.6.output.LayerNorm.weight', 'roberta.bert.encoder.layer.6.output.LayerNorm.bias', 'roberta.bert.encoder.layer.7.attention.self.query.weight', 'roberta.bert.encoder.layer.7.attention.self.query.bias', 'roberta.bert.encoder.layer.7.attention.self.key.weight', 'roberta.bert.encoder.layer.7.attention.self.key.bias', 'roberta.bert.encoder.layer.7.attention.self.value.weight', 'roberta.bert.encoder.layer.7.attention.self.value.bias', 'roberta.bert.encoder.layer.7.attention.output.dense.weight', 'roberta.bert.encoder.layer.7.attention.output.dense.bias', 'roberta.bert.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.7.intermediate.dense.weight', 'roberta.bert.encoder.layer.7.intermediate.dense.bias', 'roberta.bert.encoder.layer.7.output.dense.weight', 'roberta.bert.encoder.layer.7.output.dense.bias', 'roberta.bert.encoder.layer.7.output.LayerNorm.weight', 'roberta.bert.encoder.layer.7.output.LayerNorm.bias', 'roberta.bert.encoder.layer.8.attention.self.query.weight', 'roberta.bert.encoder.layer.8.attention.self.query.bias', 'roberta.bert.encoder.layer.8.attention.self.key.weight', 'roberta.bert.encoder.layer.8.attention.self.key.bias', 'roberta.bert.encoder.layer.8.attention.self.value.weight', 'roberta.bert.encoder.layer.8.attention.self.value.bias', 'roberta.bert.encoder.layer.8.attention.output.dense.weight', 'roberta.bert.encoder.layer.8.attention.output.dense.bias', 'roberta.bert.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.8.intermediate.dense.weight', 'roberta.bert.encoder.layer.8.intermediate.dense.bias', 'roberta.bert.encoder.layer.8.output.dense.weight', 'roberta.bert.encoder.layer.8.output.dense.bias', 'roberta.bert.encoder.layer.8.output.LayerNorm.weight', 'roberta.bert.encoder.layer.8.output.LayerNorm.bias', 'roberta.bert.encoder.layer.9.attention.self.query.weight', 'roberta.bert.encoder.layer.9.attention.self.query.bias', 'roberta.bert.encoder.layer.9.attention.self.key.weight', 'roberta.bert.encoder.layer.9.attention.self.key.bias', 'roberta.bert.encoder.layer.9.attention.self.value.weight', 'roberta.bert.encoder.layer.9.attention.self.value.bias', 'roberta.bert.encoder.layer.9.attention.output.dense.weight', 'roberta.bert.encoder.layer.9.attention.output.dense.bias', 'roberta.bert.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.9.intermediate.dense.weight', 'roberta.bert.encoder.layer.9.intermediate.dense.bias', 'roberta.bert.encoder.layer.9.output.dense.weight', 'roberta.bert.encoder.layer.9.output.dense.bias', 'roberta.bert.encoder.layer.9.output.LayerNorm.weight', 'roberta.bert.encoder.layer.9.output.LayerNorm.bias', 'roberta.bert.encoder.layer.10.attention.self.query.weight', 'roberta.bert.encoder.layer.10.attention.self.query.bias', 'roberta.bert.encoder.layer.10.attention.self.key.weight', 'roberta.bert.encoder.layer.10.attention.self.key.bias', 'roberta.bert.encoder.layer.10.attention.self.value.weight', 'roberta.bert.encoder.layer.10.attention.self.value.bias', 'roberta.bert.encoder.layer.10.attention.output.dense.weight', 'roberta.bert.encoder.layer.10.attention.output.dense.bias', 'roberta.bert.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.10.intermediate.dense.weight', 'roberta.bert.encoder.layer.10.intermediate.dense.bias', 'roberta.bert.encoder.layer.10.output.dense.weight', 'roberta.bert.encoder.layer.10.output.dense.bias', 'roberta.bert.encoder.layer.10.output.LayerNorm.weight', 'roberta.bert.encoder.layer.10.output.LayerNorm.bias', 'roberta.bert.encoder.layer.11.attention.self.query.weight', 'roberta.bert.encoder.layer.11.attention.self.query.bias', 'roberta.bert.encoder.layer.11.attention.self.key.weight', 'roberta.bert.encoder.layer.11.attention.self.key.bias', 'roberta.bert.encoder.layer.11.attention.self.value.weight', 'roberta.bert.encoder.layer.11.attention.self.value.bias', 'roberta.bert.encoder.layer.11.attention.output.dense.weight', 'roberta.bert.encoder.layer.11.attention.output.dense.bias', 'roberta.bert.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.bert.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.bert.encoder.layer.11.intermediate.dense.weight', 'roberta.bert.encoder.layer.11.intermediate.dense.bias', 'roberta.bert.encoder.layer.11.output.dense.weight', 'roberta.bert.encoder.layer.11.output.dense.bias', 'roberta.bert.encoder.layer.11.output.LayerNorm.weight', 'roberta.bert.encoder.layer.11.output.LayerNorm.bias', 'roberta.bert.pooler.dense.weight', 'roberta.bert.pooler.dense.bias', 'roberta.cls.bias', 'roberta.cls.dense.weight', 'roberta.cls.dense.bias', 'roberta.cls.layer_norm.weight', 'roberta.cls.layer_norm.bias', 'roberta.cls.decoder.weight', 'roberta.cls.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

```


Update README.md

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Update README.md

Related to pretrained model cards


Tokenizer problem for model 'patrickvonplaten/longformer-random-tiny'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Ubuntu 18.04.5 LTS
- Python version: Python 3.6.9
- PyTorch version (GPU?):1.6.0
- Tensorflow version (GPU?):2.3.1
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No

### Who can help
T5: @patrickvonplaten
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (patrickvonplaten/longformer-random-tiny):

The problem arises when using:
* [ ] the official example scripts: (give details below)
https://huggingface.co/patrickvonplaten/longformer-random-tiny

## To reproduce

Steps to reproduce the behavior:

1. Running the script.
2. Result:
_2020-11-08 11:09:14.631646: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-11-08 11:09:14.631672: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "test_tiny.py", line 3, in <module>
    tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/longformer-random-tiny")
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_auto.py", line 333, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1591, in from_pretrained
    list(cls.vocab_files_names.values()),
OSError: Model name 'patrickvonplaten/longformer-random-tiny' was not found in tokenizers model name list (allenai/longformer-base-4096, allenai/longformer-large-4096, allenai/longformer-large-4096-finetuned-triviaqa, allenai/longformer-base-4096-extra.pos.embd.only, allenai/longformer-large-4096-extra.pos.embd.only). We assumed 'patrickvonplaten/longformer-random-tiny' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
_
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I suppose two files are needed - 'vocab.json', 'merges.txt'.
Thanks.

<!-- A clear and concise description of what you would expect to happen. -->


[s2s finetune] huge increase in memory demands with --fp16 native amp

While working on https://github.com/huggingface/transformers/issues/8353 I discovered that `--fp16` causes a 10x+ increase in gpu memory demands.

e.g. I can run bs=12 w/o  `--fp16`

```
cd examples/seq2seq
export BS=12; rm -rf distilbart-cnn-12-6; python finetune.py --learning_rate=3e-5 --gpus 1 \
--do_train --do_predict --val_check_interval 0.25 --n_val 500 --num_train_epochs 2 --freeze_encoder \
--freeze_embeds --data_dir cnn_dm --max_target_length 142 --val_max_target_length=142 \
--train_batch_size=$BS --eval_batch_size=$BS --gradient_accumulation_steps 1 \
--model_name_or_path sshleifer/student_cnn_12_6 --tokenizer_name facebook/bart-large \
--warmup_steps 500 --output_dir distilbart-cnn-12-6

```
But if I add:
```
--fp16
```

(w/ or w/o `--fp16_opt_level O1`)

I get OOM even with bs=1 on a 8GB card and it barely manages on a 24GB card - I think the increase in memory demand is more than 10x.

The OOM either right away when it does the sanity check step, or after just 10-20 batches - so within a few secs

This is with pytorch-1.6. Same goes for pytorch-1.7 and 1.8-nightly.

I wasn't able to test `--fp16` with pytorch-1.5, since I can't build apex on ubuntu-20.04. Without `--fp16` pytorch-1.5 works the same as pytorch-1.6 gpu memory-wise.

I tested with pytorch-1.5 + apex and there is no problem there. Memory consumption is about half.

Here is the table of the batch sizes that fit into a 8gb rtx-1070 (bigger BS leads to an instant OOM):

bs | version
---|--------
12 | pt15
20 | pt15+fp16
12 | pt16
1 | pt16+fp16

If you'd like to reproduce the problem here are the full steps:

```
# prep library
git clone https://github.com/huggingface/transformers
cd transformers
pip install -e .[dev]
pip install -r examples/requirements.txt
cd examples/seq2seq

# prep data
wget https://cdn-datasets.huggingface.co/summarization/cnn_dm_v2.tgz
tar -xzvf cnn_dm_v2.tgz  # empty lines removed
mv cnn_cln cnn_dm

# run
export BS=12;
rm -rf distilbart-cnn-12-6
python finetune.py --learning_rate=3e-5 --gpus 1 \
--do_train --do_predict --val_check_interval 0.25 --n_val 500 --num_train_epochs 2 --freeze_encoder \
--freeze_embeds --data_dir cnn_dm --max_target_length 142 --val_max_target_length=142 \
--train_batch_size=$BS --eval_batch_size=$BS --gradient_accumulation_steps 1 \
--model_name_or_path sshleifer/student_cnn_12_6 --tokenizer_name facebook/bart-large \
--warmup_steps 500 --output_dir distilbart-cnn-12-6
```

This issue is to track the problem and hopefully finding a solution.

@sshleifer


Add gpt2-medium-chinese model card

Related to pretrained model cards

Hi,

this PR adds the model card for **gpt2-medium-chinese** model


[testing utils] get_auto_remove_tmp_dir more intuitive behavior

Now that I have been heavily using `get_auto_remove_tmp_dir` for a while, I realized that one of the defaults isn't most optimal and we had to type too much for what we need most of the time.

tldr: 99% of the time when debugging we want the tmp dir
1. to be empty at the beginning of the test
2. to be left alone after the test

This PR changes the behavior of `get_auto_remove_tmp_dir` to simplify things greatly and require much less typing.

Before this PR  when debugging a test we had to do this:
```
-        output_dir = self.get_auto_remove_tmp_dir()
+        output_dir = self.get_auto_remove_tmp_dir("./xxx", before=True, after=False)
```
Now, all you need to do is:
```
-        output_dir = self.get_auto_remove_tmp_dir()
+        output_dir = self.get_auto_remove_tmp_dir("./xxx")
```

You can still override `before` and `after` should you need to, but the main change is that if you pass a hardcoded path - you are most likely wanting to debug and see the results of the test in an easily locatable and repeatable location, and ensuring that dir is empty before the test.

Done!

@LysandreJik, @sgugger


[s2s test_finetune_trainer] failing multigpu test

Sam,

On a multigpu machine:
```
RUN_SLOW=1 pytest examples/seq2seq/test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow
```
fails for me - not learning anything.
```
>       assert first_step_stats["eval_bleu"] < last_step_stats["eval_bleu"]  # model learned nothing
E       AssertionError: assert 0.0 < 0.0
```
Looking at the logs, it gains some >0 bleu score in the first half of the epochs and then drops back to 0.00 in the last ones. On a single gpu it fluctuates between 0 and some small value - this is very fragile.

Changing to lr 3e-3 (this PR) seems to make it slightly more stable, but it could be a card specific thing - this is with rtx3090+rtx1070. So please check on your setup.

I tested that it passes with single gpu (one of each).

Alternatively the test should compare not the first and last metrics, but perhaps something more flexible? like adding up all blue scores and checking the total >0?

But either way it feels very fragile - depending too much on the hardware type - perhaps a long term approach to make it more resilient is by feeding it more than 8 records.

@sshleifer


Fixed Trainer default labels for QA Model

PR https://github.com/huggingface/transformers/pull/7191 added default labels for models. However, for QA Models it sets the default labels to ["start_positions, end_positions"] instead of ["start_positions", "end_positions"], which was the intention based on the PR description. This PR is a simple fix to this.

Fixes #8390

@sgugger


[s2s examples test] fix data path

This PR fixes a relative path in a test that should be a resolved full path instead - I forgot to test from the top of the repo in the previous PR and therefore missed this.

@sshleifer


Fix DataCollatorForWholeWordMask again

# What does this PR do?

Fixes  #8388

I tested this change by running `run_mlm_wwm.py` (did not test the chinese version). Seems to work after also fixing the example. #8394

@sgugger