bert-base-cased predicts tokens instead of whole words after fine-tuning on fill-mask task

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.1
- Platform: Linux-4.15.0-126-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu92 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 ray/raytune: @richardliaw @amogkam
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 --> @mfuntowicz, @sgugger

## Information

Model I am using (Bert, XLNet ...): bert-base-cased

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Extract the [training_data.zip](https://github.com/huggingface/transformers/files/5837438/training_data.zip). The traininig_data is structured like it is explained in [BertForMaskedLM](https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm).
2. Execute the code for fine-tuning to get the fine-tuned bert-base-cased (first script)
3. Evaluate the fine-tuned bert-base-cased with the code for evaluation (second script)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
#code for fine-tuning of bert-base-cased on fill-mask-task using the files train_queries.json and train_labels.json

from transformers import BertForMaskedLM, Trainer, TrainingArguments
import json
from transformers import BertTokenizer
import torch
import shutil
import os

class MaskedDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels
    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        return item
    def __len__(self):
        return len(self.labels)

if __name__ == "__main__":
    #used LM
    lm_name = 'bert-base-cased'
    model_path = "bert_base_cased_finetuned"
    if os.path.exists(model_path):
        print("remove dir of model")
        shutil.rmtree(model_path)
    os.mkdir(model_path)

    #pepare training dataset
    #read datasets from path
    train_queries = json.load(open("train_queries.json", "r"))
    train_labels = json.load(open("train_labels.json", "r"))

    #use tokenizer to get encodings
    tokenizer = BertTokenizer.from_pretrained(lm_name)
    train_question_encodings = tokenizer(train_queries, truncation=True, padding='max_length', max_length=256)
    train_label_encodings = tokenizer(train_labels, truncation=True, padding='max_length', max_length=256)["input_ids"]
    #get final datasets for training
    train_dataset = MaskedDataset(train_question_encodings, train_label_encodings)

    training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,   # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,              # strength of weight decay
    logging_dir=model_path+'/logs',  # directory for storing logs
    logging_steps=10,
    save_total_limit=0
    )

    model = BertForMaskedLM.from_pretrained(lm_name)

    trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset          # training dataset
    )

    trainer.train()
    trainer.save_model(model_path)
```

```
#code for evaluating the fine-tuned bert-base-cased

import json
from transformers import pipeline, BertForMaskedLM
from transformers import BertTokenizer

lm_name = "bert-base-cased"

test_queries = {"Rps26p56 is a subclass of [MASK] .": "pseudogene", "[MASK] is the capital of Hammerfest .": "Hammerfest", "Cubaedomus is a [MASK] .": "taxon", "[MASK] is named after Renfrew .": "Renfrew"}

#bert-base-cased with fine-tuning on train_queries.json and train_labels.json
unmasker_finetuned = pipeline('fill-mask', tokenizer= lm_name, model = BertForMaskedLM.from_pretrained("bert_base_cased_finetuned"), device=0, top_k=5)

#bert-base-cased tokenizer
tokenizer = BertTokenizer.from_pretrained(lm_name)

for query in test_queries:
    correct_answer = test_queries[query]
    #get the answer of the [MASK]-token of bert-base-cased-finetuned
    finetuned_result = unmasker_finetuned(query)
    finetuned_all_answers = []
    for result in finetuned_result:
        finetuned_all_answers.append(result["token_str"])

    correct_answer_ids = tokenizer(correct_answer)["input_ids"]
    correct_answer_tokens = tokenizer.convert_ids_to_tokens(correct_answer_ids)
    correct_answer_tokens.remove("[SEP]")
    correct_answer_tokens.remove("[CLS]")

    print("query:", query)
    print("correct answer:", correct_answer)
    print("correct answer tokens:", correct_answer_tokens)
    print("-----real behavior----------")
    print("finetuned all answers:", finetuned_all_answers)
    print("finetuned first answer:", finetuned_result[0]["token_str"])
    print("-----expected behavior------")
    print("finetuned first answer:", correct_answer, "\n")
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The language model should predict the whole word for the [MASK]-token and not only tokens. In the following, four queries were evaluated with the code for evaluation. For the first two queries, the finetuned language model predicts the correct tokens in the first five answers but does not match them together. For the last two queries, the finetuned language model predicts at least the correct first token but not all tokens.

My guess is, that something went wrong in the training when the word for the [MASK]-token is not in the vocabulary and the tokenizer splits the word into more than one token.

```
query: Rps26p56 is a subclass of [MASK] .
correct answer: pseudogene
correct answer tokens: ['pseudo', '##gene']
-----real behavior----------
finetuned all answers: ['pseudo', 'gene', 'protein', '##gene', 'sub']
finetuned first answer: pseudo
-----expected behavior------
finetuned first answer: pseudogene

query: [MASK] is the capital of Hammerfest .
correct answer: Hammerfest
correct answer tokens: ['Hammer', '##fest']
-----real behavior----------
finetuned all answers: ['Hammer', 'Metal', 'Hell', 'Lock', '##fest']
finetuned first answer: Hammer
-----expected behavior------
finetuned first answer: Hammerfest

query: Cubaedomus is a [MASK] .
correct answer: taxon
correct answer tokens: ['tax', '##on']
-----real behavior----------
finetuned all answers: ['tax', 'genus', 'pseudo', 'synonym', 'is']
finetuned first answer: tax
-----expected behavior------
finetuned first answer: taxon

query: [MASK] is named after Renfrew .
correct answer: Renfrew
correct answer tokens: ['Ren', '##f', '##rew']
-----real behavior----------
finetuned all answers: ['Ren', 'Re', 'R', 'Fe', 'Bo']
finetuned first answer: Ren
-----expected behavior------
finetuned first answer: Renfrew
```


Use datasets squad_v2 metric in run_qa

# What does this PR do?

The `run_qa` example script was using a copied and fixed version of the "squad_v2" version while waiting for the fix to be merged and released in datasets. That is now the case, so removing the band-aid and adjusting the version of datasets in requirements.

Fixes #9620


Fix GPT conversion script

# What does this PR do?

One forgotten file in #9674, sorry about that!


Fix old Seq2SeqTrainer

# What does this PR do?

Removes the reference to the `_actual_model` method that was removed recently in the old `Seq2SeqTrainer`.


Fix imports in conversion scripts

# What does this PR do?

During the rework of the new init for fast imports, all absolute imports were switched to relative ones indiscriminately (because they usually don't work anymore for the core of the lib). However, the conversion scripts are supposed to be executed as scripts and relative imports can't work there (that's how Python works). This PR fixes those, and it seems that it doesn't hurt the transformers-cli convert command (which import things from those modules).


add mbart to automodel for masked lm

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9653

Bart and MBart are the only Encoder-Decoder models that can do mask-filling -> so add MBart also to `AutoModelForMaskedLM`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


AttributeError: 'Seq2SeqTrainer' object has no attribute '_actual_model'

_actual_model method is not defined in the Seq2SeqTrainer class, nor in the Trainer class from which is derived

https://github.com/huggingface/transformers/blob/12c1b5b8f448d652f5e1fa0f069b9569f4540948/examples/seq2seq/seq2seq_trainer.py#L63


How to enable tokenizer padding option in feature extraction pipeline?

I am trying to use our pipeline() to extract features of sentence tokens.
Because the lengths of my sentences are not same, and I am then going to feed the token features to RNN-based models, I want to padding sentences to a fixed length to get the same size features.
Before knowing our convenient pipeline() method, I am using a general version to get the features, which works fine but inconvenient, like that:
```
import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
text = 'After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank.'

encoded_input = tokenizer(text, padding='max_length', truncation=True, max_length=40)
indexed_tokens = encoded_input['input_ids']
segments_ids = encoded_input['token_type_ids']

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

model = AutoModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
model.eval()

with torch.no_grad():
    outputs = model(tokens_tensor, segments_tensors)
    hidden_states = outputs[2]
```
Then I also need to merge (or select) the features from returned **hidden_states** by myself... and finally get a [40,768] padded feature for this sentence's tokens as I want. However, as you can see, it is very inconvenient.
Compared to that, the pipeline method works very well and easily, which only needs the following 5-line codes.
```
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
nlp = pipeline('feature-extraction', model=model, tokenizer=tokenizer)

text = "After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank."
features = nlp(text)
```
Then I can directly get the tokens' features of original (length) sentence, which is [22,768].

**However, how can I enable the padding option of the tokenizer in pipeline?**
As I saw #9432 and #9576 , I knew that now we can add truncation options to the pipeline object (here is called **nlp**), so I imitated and wrote this code:
```
text = "After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank."
features = nlp(text, padding='max_length', truncation=True, max_length=40)
```
The program did not throw me an error though, but just return me a [512,768] vector...?
So is there any method to correctly enable the padding options? Thank you!


bert_tokenizer.decode(bert_tokenizer.encode(sentence))!=sentence

from transformers import AutoTokenizer       # transformers==4.2.1
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
paragraph = "no_passages_used __knowledge__ no_passages_used"
print(tokenizer.encode(paragraph))
print(tokenizer.decode(tokenizer.encode(paragraph)))

"""
>[101, 2053, 1035, 13768, 1035, 2109, 1035, 1035, 3716, 1035, 1035, 2053, 1035, 13768, 1035, 2109, 102]
>[CLS] no _ passages _ used _ _ knowledge _ _ no _ passages _ used [SEP]
"""


[Bart-like tests] Fix torch device for bart tests

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes failing circle ci on GPU due to this commit https://github.com/huggingface/transformers/commit/357fb1c5d8b6a16f042f9b504f023d935086e8e5

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Cannot compile tokenizers on PowerPC 9 while installing transformers

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: PowerPC 9
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5 w/ GPU
- Tensorflow version (GPU?): n/a
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 ray/raytune: @richardliaw @amogkam
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

@mfuntowicz

## Information

I am trying to install `transformers==3.4.0` on an PowerPC 9 system. It's an IBM compute rig.

## To reproduce

Steps to reproduce the behavior:

1. Create new `conda` environment with python 3.7
2. Run `pip install transformers==3.4.0` (the version that I need)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
Compiling tokenizers v0.10.1 (/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/tokenizers-lib)
       Running `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=204c4d103d08e9e3 -C extra-filename=-204c4d103d08e9e3 --out-dir /tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps -L dependency=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps --extern clap=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libclap-b8e428690762cf7e.rmeta --extern derive_builder=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libderive_builder-247f4f57ff4bf4c7.so --extern esaxx_rs=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libesaxx_rs-28ce6f8a8d31c937.rmeta --extern indicatif=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libindicatif-280a1d33f346e384.rmeta --extern itertools=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libitertools-759131012594af62.rmeta --extern lazy_static=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/liblazy_static-0f749853bc34e9e0.rmeta --extern log=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/liblog-12a018fba7f0b36d.rmeta --extern onig=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libonig-3ca2736cdef653d2.rmeta --extern rand=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/librand-52622a6339ec540d.rmeta --extern rayon=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/librayon-f4508233e0c77565.rmeta --extern rayon_cond=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/librayon_cond-d89d0c7f0a1d1a11.rmeta --extern regex=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libregex-dbb55ca763c16a0e.rmeta --extern regex_syntax=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libregex_syntax-c7a8a1f28fe982ac.rmeta --extern serde=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libserde-11e7f5f85ab52b72.rmeta --extern serde_json=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libserde_json-477c52136da5fafe.rmeta --extern spm_precompiled=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libspm_precompiled-39a90f21c16965ef.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libunicode_normalization_alignments-157a660dec7f1476.rmeta --extern unicode_segmentation=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libunicode_segmentation-66856f91381ae1a4.rmeta --extern unicode_categories=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libunicode_categories-209e6f430e5d88d1.rmeta -L native=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/build/esaxx-rs-62ba703c44f19ac6/out -L native=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/build/onig_sys-091ecfe4b66243c7/out`
  error[E0603]: module `export` is private
     --> tokenizers-lib/src/tokenizer/mod.rs:24:12
      |
  24  | use serde::export::Formatter;
      |            ^^^^^^ private module
      |
  note: the module `export` is defined here
     --> /home/mengk/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.119/src/lib.rs:275:5
      |
  275 | use self::__private as export;
      |     ^^^^^^^^^^^^^^^^^^^^^^^^^

  error: aborting due to previous error

  For more information about this error, try `rustc --explain E0603`.
  error: could not compile `tokenizers`.

  Caused by:
    process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=204c4d103d08e9e3 -C extra-filename=-204c4d103d08e9e3 --out-dir /tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps -L dependency=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps --extern clap=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libclap-b8e428690762cf7e.rmeta --extern derive_builder=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libderive_builder-247f4f57ff4bf4c7.so --extern esaxx_rs=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libesaxx_rs-28ce6f8a8d31c937.rmeta --extern indicatif=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libindicatif-280a1d33f346e384.rmeta --extern itertools=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libitertools-759131012594af62.rmeta --extern lazy_static=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/liblazy_static-0f749853bc34e9e0.rmeta --extern log=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/liblog-12a018fba7f0b36d.rmeta --extern onig=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libonig-3ca2736cdef653d2.rmeta --extern rand=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/librand-52622a6339ec540d.rmeta --extern rayon=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/librayon-f4508233e0c77565.rmeta --extern rayon_cond=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/librayon_cond-d89d0c7f0a1d1a11.rmeta --extern regex=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libregex-dbb55ca763c16a0e.rmeta --extern regex_syntax=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libregex_syntax-c7a8a1f28fe982ac.rmeta --extern serde=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libserde-11e7f5f85ab52b72.rmeta --extern serde_json=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libserde_json-477c52136da5fafe.rmeta --extern spm_precompiled=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libspm_precompiled-39a90f21c16965ef.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libunicode_normalization_alignments-157a660dec7f1476.rmeta --extern unicode_segmentation=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libunicode_segmentation-66856f91381ae1a4.rmeta --extern unicode_categories=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/deps/libunicode_categories-209e6f430e5d88d1.rmeta -L native=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/build/esaxx-rs-62ba703c44f19ac6/out -L native=/tmp/pip-install-4jlvfd19/tokenizers_3c1b3bbe26064417aa8614a7fb564203/target/release/build/onig_sys-091ecfe4b66243c7/out` (exit code: 1)
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module --release --verbose -- --crate-type cdylib
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
```

## Expected behavior

There shouldn't be any error messages.

Sidenote: before getting this, I had an error complaining that I didn't have rust installed, but I did so using the command given on the official website.

<!-- A clear and concise description of what you would expect to happen. -->


Add new model docs

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds more information on how to add a model to Transformers docs.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

## UPDATE

The `model_doc/add_new_model.rst` is now finished for a first merge IMO. It would be amazing if @LysandreJik @sgugger you could review the file real quick again - I tried to add all of your suggestions. Also, I added a diagram showing the model design of Transformers - which was not reviewed yet. Note that I did not add a clear design for Tokenizers since it takes a lot of time to do so and I want to iteratively improve this step-by-step explanation. The first model, for which I'd like to mentor someone from the community would also be BigBird which does not need a new tokenizer.

In addition, I would be extremely grateful if @stas00 @abhishekkrthakur @patil-suraj @stefan-it @NielsRogge you have 10 minutes review the `model_doc/add_model.rst` file for possible improvements since you guys just recently added a new model. Your feedback would be especially useful since you might have a much more "unbiased" view what is difficult/easy when adding a model.


Fine-tuning LM with NSP

Environment info
transformers-4.2.1
PyTorch
tokenizers-0.9.4
sentencepiece-0.1.95

when finetuning bert my script run as well but not complete the running and because of this error Traceback (most recent call last):
  File "/content/run.py", line 757, in <module>
    main()
  File "/content/run.py", line 656, in main
    labels=lm_label_ids, next_sentence_label=is_next)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_bert.py", line 1065, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_bert.py", line 968, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_bert.py", line 566, in forward
    output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_bert.py", line 460, in forward
    past_key_value=self_attn_past_key_value,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_bert.py", line 393, in forward
    output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_bert.py", line 314, in forward
    attention_probs = nn.Softmax(dim=-1)(attention_scores)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/activation.py", line 1198, in forward
    return F.softmax(input, self.dim, _stacklevel=5)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1512, in softmax
    ret = input.softmax(dim)
RuntimeError: CUDA error: device-side assert `triggered

++++++++++++++and my code is here+++++++++++++++++
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import os
import logging
import argparse
#from tqdm import tqdm
#from tqdm import trange
from tqdm import notebook , trange

import numpy as np
import torch
from torch.utils.data import DataLoader, RandomSampler , SequentialSampler
from torch.utils.data.distributed import DistributedSampler

#from pytorch_pretrained_bert.tokenization import BertTokenizer
#from pytorch_pretrained_bert.modeling import BertForPreTraining
from transformers import BertTokenizer, BertForPreTraining
#from pytorch_pretrained_bert.optimization import BertAdam
from transformers import XLNetTokenizer
from transformers import AdamW, get_linear_schedule_with_warmup
#from transformers import BertForPreTraining
import sentencepiece as spm

from torch.utils.data import Dataset
import random

logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s -   %(message)s',
                    datefmt='%m/%d/%Y %H:%M:%S',
                    level=logging.INFO)
logger = logging.getLogger(__name__)

def warmup_linear(x, warmup=0.002):
    if x < warmup:
        return x / warmup
    return 1.0 - x

def accuracy(out, labels, total_test):
    class_preds = out.data.cpu().numpy().argmax(axis=-1)
    labels = labels.data.cpu().numpy()
    return np.sum(class_preds == labels) / total_test

class BERTDataset(Dataset):
    def __init__(self, corpus_path, tokenizer, seq_len, encoding="utf-8", corpus_lines=None, on_memory=True):
        self.vocab = tokenizer.get_vocab()
        self.tokenizer = tokenizer
        self.seq_len = seq_len
        self.on_memory = on_memory
        self.corpus_lines = corpus_lines  # number of non-empty lines in input corpus
        self.corpus_path = corpus_path
        self.encoding = encoding
        self.current_doc = 0  # to avoid random sentence from same doc

        # for loading samples directly from file
        self.sample_counter = 0  # used to keep track of full epochs on file
        self.line_buffer = None  # keep second sentence of a pair in memory and use as first sentence in next pair

        # for loading samples in memory
        self.current_random_doc = 0
        self.num_docs = 0
        self.sample_to_doc = []  # map sample index to doc and line

        # load samples into memory
        if on_memory:
            self.all_docs = []
            doc = []
            self.corpus_lines = 0
            with open(corpus_path, "r", encoding=encoding) as f:
                for line in notebook.tqdm(f, desc="Loading Dataset", total=corpus_lines):
                    line = line.strip()
                    if line == "":
                        self.all_docs.append(doc)
                        doc = []
                        # remove last added sample because there won't be a subsequent line anymore in the doc
                        self.sample_to_doc.pop()
                    else:
                        # store as one sample
                        sample = {"doc_id": len(self.all_docs),
                                  "line": len(doc)}
                        self.sample_to_doc.append(sample)
                        doc.append(line)
                        self.corpus_lines = self.corpus_lines + 1

            # if last row in file is not empty
            if self.all_docs[-1] != doc:
                self.all_docs.append(doc)
                self.sample_to_doc.pop()

            self.num_docs = len(self.all_docs)

        # load samples later lazily from disk
        else:
            if self.corpus_lines is None:
                with open(corpus_path, "r", encoding=encoding) as f:
                    self.corpus_lines = 0
                    for line in notebook.tqdm(f, desc="Loading Dataset", total=corpus_lines):
                        if line.strip() == "":
                            self.num_docs += 1
                        else:
                            self.corpus_lines += 1

                    # if doc does not end with empty line
                    if line.strip() != "":
                        self.num_docs += 1

            self.file = open(corpus_path, "r", encoding=encoding)
            self.random_file = open(corpus_path, "r", encoding=encoding)

    def __len__(self):
        # last line of doc won't be used, because there's no "nextSentence". Additionally, we start counting at 0.
        return self.corpus_lines - self.num_docs - 1

    def __getitem__(self, item):
        cur_id = self.sample_counter
        self.sample_counter += 1
        if not self.on_memory:
            # after one epoch we start again from beginning of file
            if cur_id != 0 and (cur_id % len(self) == 0):
                self.file.close()
                self.file = open(self.corpus_path, "r", encoding=self.encoding)

        t1, t2, is_next_label = self.random_sent(item)

        # tokenize
        tokens_a = self.tokenizer.tokenize(t1)
        tokens_b = self.tokenizer.tokenize(t2)

        # combine to one sample
        cur_example = InputExample(guid=cur_id, tokens_a=tokens_a, tokens_b=tokens_b, is_next=is_next_label)

        # transform sample to features
        cur_features = convert_example_to_features(cur_example, self.seq_len, self.tokenizer)

        cur_tensors = (torch.tensor(cur_features.input_ids),
                       torch.tensor(cur_features.input_mask),
                       torch.tensor(cur_features.segment_ids),
                       torch.tensor(cur_features.lm_label_ids),
                       torch.tensor(cur_features.is_next))

        return cur_tensors

    def random_sent(self, index):
        """
        Get one sample from corpus consisting of two sentences. With prob. 50% these are two subsequent sentences
        from one doc. With 50% the second sentence will be a random one from another doc.
        :param index: int, index of sample.
        :return: (str, str, int), sentence 1, sentence 2, isNextSentence Label
        """
        t1, t2 = self.get_corpus_line(index)
        if random.random() > 0.5:
            label = 0
        else:
            t2 = self.get_random_line()
            label = 1

        assert len(t1) > 0
        assert len(t2) > 0
        return t1, t2, label

    def get_corpus_line(self, item):
        """
        Get one sample from corpus consisting of a pair of two subsequent lines from the same doc.
        :param item: int, index of sample.
        :return: (str, str), two subsequent sentences from corpus
        """
        t1 = ""
        t2 = ""
        assert item < self.corpus_lines
        if self.on_memory:
            sample = self.sample_to_doc[item]
            t1 = self.all_docs[sample["doc_id"]][sample["line"]]
            t2 = self.all_docs[sample["doc_id"]][sample["line"] + 1]
            # used later to avoid random nextSentence from same doc
            self.current_doc = sample["doc_id"]
            return t1, t2
        else:
            if self.line_buffer is None:
                # read first non-empty line of file
                while t1 == "":
                    t1 = self.file.__next__().strip()
                    t2 = self.file.__next__().strip()
            else:
                # use t2 from previous iteration as new t1
                t1 = self.line_buffer
                t2 = self.file.__next__().strip()
                # skip empty rows that are used for separating documents and keep track of current doc id
                while t2 == "" or t1 == "":
                    t1 = self.file.__next__().strip()
                    t2 = self.file.__next__().strip()
                    self.current_doc = self.current_doc + 1
            self.line_buffer = t2

        assert t1 != ""
        assert t2 != ""
        return t1, t2

    def get_random_line(self):
        """
        Get random line from another document for nextSentence task.
        :return: str, content of one line
        """
        # Similar to original tf repo: This outer loop should rarely go for more than one iteration for large
        # corpora. However, just to be careful, we try to make sure that
        # the random document is not the same as the document we're processing.
        for _ in range(10):
            if self.on_memory:
                rand_doc_idx = random.randint(0, len(self.all_docs) - 1)
                rand_doc = self.all_docs[rand_doc_idx]
                line = rand_doc[random.randrange(len(rand_doc))]
            else:
                rand_index = random.randint(1, self.corpus_lines if self.corpus_lines < 1000 else 1000)
                # pick random line
                for _ in range(rand_index):
                    line = self.get_next_line()
            # check if our picked random line is really from another doc like we want it to be
            if self.current_random_doc != self.current_doc:
                break
        return line

    def get_next_line(self):
        """ Gets next line of random_file and starts over when reaching end of file"""
        try:
            line = self.random_file.__next__().strip()
            # keep track of which document we are currently looking at to later avoid having the same doc as t1
            if line == "":
                self.current_random_doc = self.current_random_doc + 1
                line = self.random_file.__next__().strip()
        except StopIteration:
            self.random_file.close()
            self.random_file = open(self.corpus_path, "r", encoding=self.encoding)
            line = self.random_file.__next__().strip()
        return line

class InputExample(object):
    """A single training/test example for the language model."""

    def __init__(self, guid, tokens_a, tokens_b=None, is_next=None, lm_labels=None):
        """Constructs a InputExample.
        Args:
            guid: Unique id for the example.
            tokens_a: string. The untokenized text of the first sequence. For single
            sequence tasks, only this sequence must be specified.
            tokens_b: (Optional) string. The untokenized text of the second sequence.
            Only must be specified for sequence pair tasks.
            label: (Optional) string. The label of the example. This should be
            specified for train and dev examples, but not for test examples.
        """
        self.guid = guid
        self.tokens_a = tokens_a
        self.tokens_b = tokens_b
        self.is_next = is_next  # nextSentence
        self.lm_labels = lm_labels  # masked words for language model

class InputFeatures(object):
    """A single set of features of data."""

    def __init__(self, input_ids, input_mask, segment_ids, is_next, lm_label_ids):
        self.input_ids = input_ids
        self.input_mask = input_mask
        self.segment_ids = segment_ids
        self.is_next = is_next
        self.lm_label_ids = lm_label_ids

def random_word(tokens, tokenizer):
    """
    Masking some random tokens for Language Model task with probabilities as in the original BERT paper.
    :param tokens: list of str, tokenized sentence.
    :param tokenizer: Tokenizer, object used for tokenization (we need it's vocab here)
    :return: (list of str, list of int), masked tokens and related labels for LM prediction
    """
    output_label = []

    for i, token in enumerate(tokens):
        prob = random.random()
        # mask token with 15% probability
        if prob < 0.15:
            prob /= 0.15

            # 80% randomly change token to mask token
            if prob < 0.8:
                tokens[i] = "[MASK]"

            # 10% randomly change token to random token
            elif prob < 0.9:
                tokens[i] = random.choice(list(tokenizer.get_vocab()))

            # -> rest 10% randomly keep current token

            # append current token to output (we will predict these later)
            try:
                output_label.append(tokenizer.convert_tokens_to_ids(token))
            except KeyError:
                # For unknown words (should not occur with BPE vocab)
                output_label.append(tokenizer.convert_tokens_to_ids("[UNK]"))
                logger.warning("Cannot find token '{}' in vocab. Using [UNK] insetad".format(token))
        else:
            # no masking token (will be ignored by loss function later)
            output_label.append(-100)

    return tokens, output_label

def convert_example_to_features(example, max_seq_length, tokenizer):
    """
    Convert a raw sample (pair of sentences as tokenized strings) into a proper training sample with
    IDs, LM labels, input_mask, CLS and SEP tokens etc.
    :param example: InputExample, containing sentence input as strings and is_next label
    :param max_seq_length: int, maximum length of sequence.
    :param tokenizer: Tokenizer
    :return: InputFeatures, containing all inputs and labels of one sample as IDs (as used for model training)
    """
    tokens_a = example.tokens_a
    tokens_b = example.tokens_b
    # Modifies `tokens_a` and `tokens_b` in place so that the total
    # length is less than the specified length.
    # Account for [CLS], [SEP], [SEP] with "- 3"
    _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)

    t1_random, t1_label = random_word(tokens_a, tokenizer)
    t2_random, t2_label = random_word(tokens_b, tokenizer)
    # concatenate lm labels and account for CLS, SEP, SEP
    cls_id = tokenizer.convert_tokens_to_ids(["[CLS]"])[0]
    sep_id = tokenizer.convert_tokens_to_ids(["[SEP]"])[0]
    pad_id = tokenizer.convert_tokens_to_ids(["[PAD]"])[0]
    lm_label_ids = ([cls_id] + t1_label + [sep_id] + t2_label + [sep_id])

    # The convention in BERT is:
    # (a) For sequence pairs:
    #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
    #  type_ids: 0   0  0    0    0     0       0 0    1  1  1  1   1 1
    # (b) For single sequences:
    #  tokens:   [CLS] the dog is hairy . [SEP]
    #  type_ids: 0   0   0   0  0     0 0
    #
    # Where "type_ids" are used to indicate whether this is the first
    # sequence or the second sequence. The embedding vectors for `type=0` and
    # `type=1` were learned during pre-training and are added to the wordpiece
    # embedding vector (and position vector). This is not *strictly* necessary
    # since the [SEP] token unambigiously separates the sequences, but it makes
    # it easier for the model to learn the concept of sequences.
    #
    # For classification tasks, the first vector (corresponding to [CLS]) is
    # used as as the "sentence vector". Note that this only makes sense because
    # the entire model is fine-tuned.
    tokens = []
    segment_ids = []
    tokens.append("[CLS]")
    segment_ids.append(0)
    for token in tokens_a:
        tokens.append(token)
        segment_ids.append(0)
    tokens.append("[SEP]")
    segment_ids.append(0)

    assert len(tokens_b) > 0
    for token in tokens_b:
        tokens.append(token)
        segment_ids.append(1)
    tokens.append("[SEP]")
    segment_ids.append(1)

    input_ids = tokenizer.convert_tokens_to_ids(tokens)

    # The mask has 1 for real tokens and 0 for padding tokens. Only real
    # tokens are attended to.
    input_mask = [1] * len(input_ids)

    # Zero-pad up to the sequence length.
    while len(input_ids) < max_seq_length:
        input_ids.append(0)
        input_mask.append(0)
        segment_ids.append(0)
        lm_label_ids.append(-100)

    assert len(input_ids) == max_seq_length
    assert len(input_mask) == max_seq_length
    assert len(segment_ids) == max_seq_length
    assert len(lm_label_ids) == max_seq_length

    if example.guid < 5:
        logger.info("*** Example ***")
        logger.info("guid: %s" % (example.guid))
        logger.info("tokens: %s" % " ".join(
            [str(x) for x in tokens]))
        logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
        logger.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))
        logger.info(
            "segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
        logger.info("LM label: %s " % (lm_label_ids))
        logger.info("Is next sentence label: %s " % (example.is_next))

    features = InputFeatures(input_ids=input_ids,
                             input_mask=input_mask,
                             segment_ids=segment_ids,
                             lm_label_ids=lm_label_ids,
                             is_next=example.is_next)
    return features

def main():
    parser = argparse.ArgumentParser()

    ## Required parameters
    parser.add_argument("--train_file",
                        default=None,
                        type=str,
                        required=True,
                        help="The input train corpus.")
    parser.add_argument("--test_file",
                        default=None,
                        type=str,
                        required=True,
                        help="The input test corpus.")

    parser.add_argument("--tokenizer_model", default=None, type=str, required=True,
                        help="tokenizer pre-trained model selected in the list: bert-base-uncased, "
                             "bert-large-uncased, bert-base-cased, bert-base-multilingual, bert-base-chinese.")

    parser.add_argument("--bert_model", default=None, type=str, required=True,
                        help="Bert pre-trained model selected in the list: bert-base-uncased, "
                             "bert-large-uncased, bert-base-cased, bert-base-multilingual, bert-base-chinese.")

    parser.add_argument("--config_file", default=None, type=str, required=True,
                        help="Bert pre-trained model selected in the list: bert-base-uncased, "
                             "bert-large-uncased, bert-base-cased, bert-base-multilingual, bert-base-chinese.")

    parser.add_argument("--output_dir",
                        default=None,
                        type=str,
                        required=True,
                        help="The output directory where the model checkpoints will be written.")
    ## Other parameters
    parser.add_argument("--max_seq_length",
                        default=128,
                        type=int,
                        help="The maximum total input sequence length after WordPiece tokenization. \n"
                             "Sequences longer than this will be truncated, and sequences shorter \n"
                             "than this will be padded.")

    parser.add_argument("--train_batch_size",
                        default=32,
                        type=int,
                        help="Total batch size for training.")

    parser.add_argument("--eval_batch_size",
                        default=32,
                        type=int,
                        help="Total batch size for eval.")
    parser.add_argument("--learning_rate",
                        default=5e-5,
                        type=float,
                        help="The initial learning rate for Adam.")

    parser.add_argument("--num_train_epochs",
                        default=4,
                        type=float,
                        help="Total number of training epochs to perform.")

    parser.add_argument("--adam_epsilon",
                        default=1e-8,
                        type=float,
                        help="Proportion of training to perform linear learning rate warmup for. "
                             "E.g., 0.1 = 10%% of training.")

    parser.add_argument("--no_cuda",
                        action='store_true',
                        help="Whether not to use CUDA when available")

    parser.add_argument("--on_memory",
                        action='store_true',
                        help="Whether to load train samples into memory or use disk")

    parser.add_argument("--do_lower_case",
                        action='store_true',
                        help="Whether to lower case the input text. True for uncased models, False for cased models.")

    parser.add_argument("--local_rank",
                        type=int,
                        default=-1,
                        help="local_rank for distributed training on gpus")

    parser.add_argument('--seed',
                        type=int,
                        default=42,
                        help="random seed for initialization")

    parser.add_argument('--gradient_accumulation_steps',
                        type=int,
                        default=1,
                        help="Number of updates steps to accumualte before performing a backward/update pass.")

    parser.add_argument('--fp16',
                        action='store_true',
                        help="Whether to use 16-bit float precision instead of 32-bit")

    parser.add_argument('--loss_scale',
                        type=float, default=0,
                        help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n"
                             "0 (default value): dynamic loss scaling.\n"
                             "Positive power of 2: static loss scaling value.\n")

    args = parser.parse_args()

    if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        n_gpu = torch.cuda.device_count()
    else:
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        n_gpu = 1
        # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
        torch.distributed.init_process_group(backend='nccl')
    logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
        device, n_gpu, bool(args.local_rank != -1), args.fp16))

    if args.gradient_accumulation_steps < 1:
        raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
            args.gradient_accumulation_steps))

    args.train_batch_size = int(args.train_batch_size / args.gradient_accumulation_steps)

    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    if n_gpu > 0:
        torch.cuda.manual_seed_all(args.seed)

    #if not args.do_train and not args.do_eval:
    #    raise ValueError("At least one of `do_train` or `do_eval` must be True.")

    if os.path.exists(args.output_dir) and os.listdir(args.output_dir):
        raise ValueError("Output directory ({}) already exists and is not empty.".format(args.output_dir))
    os.makedirs(args.output_dir, exist_ok=True)

    # tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
    tokenizer = XLNetTokenizer.from_pretrained(args.tokenizer_model)

    # train_examples = None
    num_train_steps = None

    print("Loading Train Dataset", args.train_file)
    train_dataset = BERTDataset(args.train_file, tokenizer, seq_len=args.max_seq_length,
                                corpus_lines=None, on_memory=args.on_memory)

    print("Loading eval Dataset", args.test_file)
    eval_dataset = BERTDataset(args.test_file, tokenizer, seq_len=args.max_seq_length,
                               corpus_lines=None, on_memory=args.on_memory)

    num_train_steps = int(
        len(train_dataset) / args.train_batch_size / args.gradient_accumulation_steps * args.num_train_epochs)

    # Prepare model
    model = BertForPreTraining.from_pretrained(
        args.bert_model,
        config=args.config_file,
        output_attentions=False,  # Whether the model returns attentions weights.
        output_hidden_states=False,  # Whether the model returns all hidden-states.
    )
    # Tell pytorch to run this model on the GPU.
    model.to(device)

    if args.fp16:
        model.half()
    if args.local_rank != -1:
        try:
            from apex.parallel import DistributedDataParallel as DDP
        except ImportError:
            raise ImportError(
                "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
        model = DDP(model)
    elif n_gpu > 1:
        model = torch.nn.DataParallel(model)

    # Prepare optimizer
    '''
    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
    ]
    if args.fp16:
        try:
            from apex.optimizers import FP16_Optimizer
            from apex.optimizers import FusedAdam
        except ImportError:
            raise ImportError(
                "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")

        optimizer = FusedAdam(optimizer_grouped_parameters,
                              lr=args.learning_rate,
                              bias_correction=False,
                              max_grad_norm=1.0)
        if args.loss_scale == 0:
            optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
        else:
            optimizer = FP16_Optimizer(optimizer, static_loss_scale=args.loss_scale)

    else:
        optimizer = AdamW(optimizer_grouped_parameters,
                             lr=args.learning_rate,
                             warmup=args.warmup_proportion,
                             t_total=num_train_steps)
    '''

    logger.info("***** Running training *****")
    logger.info("  Num examples = %d", len(train_dataset))
    logger.info("  Batch size = %d", args.train_batch_size)
    logger.info("  Num steps = %d", num_train_steps)

    if args.local_rank == -1:
        train_sampler = SequentialSampler(train_dataset)
        eval_sampler = SequentialSampler(eval_dataset)

    else:
        # TODO: check if this works with current data generator from disk that relies on file.__next__
        # (it doesn't return item back by index)
        train_sampler = DistributedSampler(train_dataset)
        eval_sampler = DistributedSampler(eval_dataset)
    train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size)
    eval_dataloader = DataLoader(eval_dataset, sampler=eval_sampler, batch_size=args.train_batch_size)

    # optimizer
    t_total = len(train_dataloader) // args.train_batch_size
    no_decay = ['bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in model.named_parameters() if
                    not any(nd in n for nd in no_decay)],
         'weight_decay': 0.01},
        {'params': [p for n, p in model.named_parameters() if any(
            nd in n for nd in no_decay)], 'weight_decay': 0.0}
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    scheduler = get_linear_schedule_with_warmup(
        optimizer, 0, t_total)

    model.train()
    tr_loss = 0
    global_step = 0
    acc = 0
    train_loss = 0.0
    nb_tr_examples, nb_tr_steps = 0, 0
    for _ in trange(int(args.num_train_epochs), desc="Epoch"):
        for batch in notebook.tqdm(train_dataloader, desc="Train Iteration"):
            batch = tuple(t.to(device) for t in batch)
            input_ids, input_mask, segment_ids, lm_label_ids, is_next = batch
            outputs = model(input_ids=input_ids, attention_mask=input_mask, token_type_ids=segment_ids,
                            labels=lm_label_ids, next_sentence_label=is_next)

            loss = outputs.loss
            '''
            if n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu.
            if args.gradient_accumulation_steps > 1:
                loss = loss / args.gradient_accumulation_steps
            if args.fp16:
                optimizer.backward(outputs.loss)
            else:
                loss.backward()
            '''
            print(loss)
            loss.backward()
            tr_loss += loss.item()
            nb_tr_examples += input_ids.size(0)
            nb_tr_steps += 1
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1)
            optimizer.step()
            scheduler.step()
            model.zero_grad()
            global_step += 1

            '''
            if (step + 1) % args.gradient_accumulation_steps == 0:
                # modify learning rate with special warm up BERT uses
                lr_this_step = args.learning_rate * warmup_linear(global_step / num_train_steps, args.warmup_proportion)
                for param_group in optimizer.param_groups:
                    param_group['lr'] = lr_this_step
                optimizer.step()
                scheduler.step()
                optimizer.zero_grad()
                global_step += 1
            '''

    train_loss = tr_loss / global_step
    perplexity = torch.exp(torch.tensor(train_loss)).item()

    print("Training loss {} ".format("{:.3f}".format(train_loss)))
    print("Training perplexity {}".format("{:.3f}".format(perplexity)))

    logger.info("***** Running evaluation *****")
    logger.info("  Num examples = %d", len(eval_dataset))
    logger.info("  Batch size = %d", batch_size)
    eval_loss = 0.0
    acc = 0
    nb_eval_steps = 0
    for batch in notebook.tqdm(eval_dataloader, desc='Evaluating'):
        batch = tuple(t.to(device) for t in batch)
        input_ids, input_mask, segment_ids, lm_label_ids, is_next = batch
        with torch.no_grad():
            outputs = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)
            loss = outputs.loss
            eval_loss += loss.mean().item()
        nb_eval_steps += 1

    eval_loss = eval_loss / nb_eval_steps
    perplexity = torch.exp(torch.tensor(eval_loss)).item()

    print("Evalution loss {} ".format("{:.3f}".format(eval_loss)))
    print("Evalution perplexity {}".format("{:.3f}".format(perplexity)))

    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    print("Saving model to %s" % args.output_dir)

    # Save a trained model, configuration and tokenizer using `save_pretrained()`.
    # They can then be reloaded using `from_pretrained()`
    model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training
    model_to_save.save_pretrained(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)

    # Save a trained model
    #logger.info("** ** * Saving fine - tuned model ** ** * ")
    model_to_save = model.module if hasattr(model, 'module') else model  # Only save the model it-self
    #if args.do_train:
       # model_to_save.save_pretrained(self.output_dir)
      #  tokenizer.save_pretrained(self.output_dir)

def _truncate_seq_pair(tokens_a, tokens_b, max_length):
    """Truncates a sequence pair in place to the maximum length."""

    # This is a simple heuristic which will always truncate the longer sequence
    # one token at a time. This makes more sense than truncating an equal percent
    # of tokens from each, since if one sequence is very short then each token
    # that's truncated likely contains more information than a longer sequence.
    while True:
        total_length = len(tokens_a) + len(tokens_b)
        if total_length <= max_length:
            break
        if len(tokens_a) > len(tokens_b):
            tokens_a.pop()
        else:
            tokens_b.pop()

if __name__ == "__main__":
    main()

`


IndexError: index out of bounds when running run_mlm.py

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-4.15.0-46-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.7
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
?

## Information

Model I am using (Bert, XLNet ...): neuralmind/bert-base-portuguese-cased

## To reproduce

Steps to reproduce the behavior:

I want to fine-tune a pretrained language model using [run_mlm.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). I have a corpus file (ful_corpus.csv) that contains one doc (raw text) per line. When I run the following command:

`python run_mlm.py --model_name_or_path "neuralmind/bert-base-portuguese-cased" --train_file ../data/full_corpus.csv --cache_dir /home/mwon/data-mwon/paperChega/src_classificador/data/hugingface  --output models/  --do_train`

it results in the error:

```
Traceback (most recent call last):
  File "run_mlm.py", line 449, in <module>
    main()
  File "run_mlm.py", line 384, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
    for k, dataset in self.items()
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1260, in map
    update_data=update_data,
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1529, in _map_single
    writer.write_batch(batch)
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_writer.py", line 278, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1474, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 322, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1058, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 540, in pyarrow.lib._normalize_index
IndexError: index out of bounds
```


Missing `return_dict` in Doc example

# What does this PR do?
Fixes a crash in [Summary of the tasks](https://huggingface.co/transformers/task_summary.html) documentation, by adding `return_dict=True` to the model() function, as we need `start_logits` and `end_logits` afterwards.

## Fixes
[Issue 9043](https://github.com/huggingface/transformers/issues/9043) (reproduced in 4.2.1)

## Who can review?
@LysandreJik @sgugger


Fix DPRReaderTokenizer's attention_mask

# What does this PR do?

This PR fixes an issue with the `attention_mask` not being properly generated by the DPRReaderTokenizer. Please see  issue #9555 for more details.

I added an integration test that checks the DPRReader following a similar example in the test file.

I have some test failures due to
```
"AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'"
```
and
```
"AttributeError: module 'wandb' has no attribute 'ensure_configured'"
```
which seem to be unrelated to my code changes.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@LysandreJik and @lhoestq would probably be best positioned to review.


Fix TFTrainer prediction output

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

This PR fixes two issues:

1) The prediction output (specifically prediction_loop) of TFTrainer does not match the dataset cardinality. If the number of examples is divisible by eval_batch_size, the first batch is predicted twice. Else, the first n examples, where n = eval_batch_size - num_examples % eval_batch_size, are predicted twice.

This results in an output shape that is different from dataset cardinality. This also causes the output of evaluate(), including eval_loss, to be incorrect (e.g. loss is computed twice for the first few examples).

2) The evaluation loss only works the first time it is computed. Subsequent computations result to 0. Below is a sample output when an evaluation strategy is set during training:

[INFO|trainer_tf.py:398] 2021-01-18 01:34:58,856 >> {**'eval_loss': 0.6290212145038679**, 'eval_acc': 0.6875, ... 'step': 10}
[INFO|trainer_tf.py:398] 2021-01-18 01:35:10,852 >> {**'eval_loss': 0.0**, 'eval_acc': 0.6875, ..., 'step': 20}

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

 tensorflow: @jplu
Trainer: @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Fix TF Flaubert and XLM

# What does this PR do?

By doing some experiments on Flaubert and XLM I realized that building a model with a `None` argument forces this value to be `None`  when served. Then to fix this issue, the build takes a proper input without a `None` value.


run_ner.py crashes when dev or test contain previously unseen labels

## Environment info

- `transformers` version: 4.2.0dev0
- Platform:  Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: don't know

### Who can help
@stefan-it

## Information

Model I am using (Bert, XLNet ...): KB/bert-base-swedish-cased

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I am using run_ner.py to POS-tag a Swedish corpus (TalbankenSBX). If dev and/or test files contain a label that is not present in the train file, the script crashes. The same issue arises with any other corpus I try.

## To reproduce

Steps to reproduce the behavior:

1. Create two toy files with the following contents:

sbx-1-train.json:
```
{"words": ["Vem", "får", "rösta", "?"], "pos": ["HP.UTR.SIN.IND", "VB.PRS.AKT", "VB.INF.AKT", "MAD"]}
```

sbx-1-dev.json
```
{"words": ["är", "född", "1950", "eller", "tidigare", ","], "pos": ["VB.PRS.AKT", "PC.PRF.UTR.SIN.IND.NOM", "RG.NOM", "KN", "AB.KOM", "MID"]}
```

2. Run  python run_ner.py --model_name_or_path KB/bert-base-swedish-cased --train_file sbx-1-train.json --validation_file sbx-1-dev.json --output_dir sbx1 --do_train --do_eval

This results in:
```
Traceback (most recent call last):
  File "run_ner.py", line 412, in <module>
    main()
  File "run_ner.py", line 303, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/home/sasha/venvs/hugtrans/lib64/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
    for k, dataset in self.items()
  File "/home/sasha/venvs/hugtrans/lib64/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
    for k, dataset in self.items()
  File "/home/sasha/venvs/hugtrans/lib64/python3.6/site-packages/datasets/arrow_dataset.py", line 1240, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/home/sasha/venvs/hugtrans/lib64/python3.6/site-packages/datasets/arrow_dataset.py", line 1211, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "run_ner.py", line 288, in tokenize_and_align_labels
    label_ids.append(label_to_id[label[word_idx]])
KeyError: 'PC.PRF.UTR.SIN.IND.NOM'
```

...where PC.PRF.UTR.SIN.IND.NOM is the tag which is not present in the train set.

## Expected behavior
The script should not crash when encountering unseen tags.


Wav2Vec2

# What does this PR do?

Adds Wav2Vec2 from https://github.com/pytorch/fairseq/blob/master/examples/wav2vec/README.md

This PR adds the wav2vec2 Acoustic model to Transformers. The model is different from "conventional" transformer
models since it classifies a raw waveform input (float array) into logits. Therefore the `Wav2Vec2Tokenizer` behaves quite differently from usual tokenizers in that it only pads an input instead of encoding it to token ids.

The fully functional model should be added in three steps:

1) Add the Acoustic model ready to be used for inference (This PR)
2) Add fine-tuning + pertaining functionality to the model (Next PR)
3) Add an example script showing how Wav2Vec2 can be used with a language model
4) Add an Encoder/Decoder version of Wav2Vec2.

# Usage

One can play around with a quick demo here: https://colab.research.google.com/drive/1xaVKQ739-ue0v8IuMZoMzOFc4-NlGQyd?usp=sharing and some usage examples as described on the model cards:
https://huggingface.co/models?filter=wav2vec2

# Review
In this PR, no training functionality is added to the model. Because this is quite complex for Wav2Vec2 this will be done in a follow up PR.
It would be great, if we can however already merge this first part which allows to use Wav2Vec2 for inference.
The tokenizer is quite different, so it would be great if @thomwolf @n1t0 can also take a look here.
Since the model expects the raw waveform signal as an input, the name `input_ids` is changed to `input_values` standing for "a tensor of float values" - would be great if you can check this @LysandreJik @sgugger @thomwolf.

# Done:

- [x] load pretrained weight into model
- [x] make sure forward pass yields equal outputs
- [x] successful transcription
- [x] add tokenizer
- [x] Think about how to add the two different architectures `Wav2Vec 2.0 Large (LV-60)`/`Wav2Vec 2.0 Large (LV-60) + Self Training` is different from `Wav2Vec 2.0 Large`/`Wav2Vec 2.0 Base` (layer_norm is switched and no group norm is used)
- [x] add model tests
- [x] add tokenizer tests
- [x] add docstring
- [x] clean config

# Future TODO:
- [ ] Add PreTraining & Fine-Tuning to model
- [ ] Add Encoder Decoder model / CTC decoding


Tokenizstion

Hi all, help me please. I'm trying to solve multilingual WiC problem using XLM-R.
I have a word and a sentence, that has this word, but may be in different form. So I want to find the position of its token ids in an encoded sentence.
There problems with Chinese, cause they don't have space between many symbols and tokenizer can encode a pair of symbols differently.


ModuleAttributeError occurs during Converting TensorFlow Checkpoints (BERT)

## Environment info

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-129-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* Convert TF v1 ckpt to PyTorch

## To reproduce

I tried to convert a TensorFlow checkpoint, but `ModuleAttributeError` occurred.

What I run:
```
****@**** $ transformers-cli convert --model_type bert \
>   --tf_checkpoint $MODEL_DIR/model.ckpt \
>   --config ****/bert_config.json \
>   --pytorch_dump_output $MODEL_DIR/pytorch_model.bin
```

(In this time, `bert_config.json` is in a separate folder, but it corresponds to the `ckpt`.)

Output is:
```
Traceback (most recent call last):
  File "/****/.pyenv/versions/anaconda3-2020.07/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/commands/transformers_cli.py", line 51, in main
    service.run()
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/commands/convert.py", line 105, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 155, in load_tf_weights_in_bert
    pointer.shape == array.shape
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/torch/nn/modules/module.py", line 778, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'BertEmbeddings' object has no attribute 'shape'
```

## Expected behavior

I think it is not strange that `BertEmbeddings` (`nn.Module`) doesn't have `shape.`

Is it possible to get such an error depending on the original TensorFlow checkpoint?
In such a case, is there any tips to deal with it?

I really appreciate any help you can provide.


"Converting Tensorflow Checkpoints" document has wrong link in v4.2.0+

## Environment info

- `transformers` version: v4.2.0+

### Who can help
 documentation: @sgugger

## Information

I'd like to convert BERT ckpt to PyTorch, and read [Converting Tensorflow Checkpoints](https://huggingface.co/transformers/converting_tensorflow_models.html) document.

It seems that the link to `convert_bert_original_tf_checkpoint_to_pytorch.py` is outdated.

It is linked to https://huggingface.co/transformers/converting_tensorflow_models.html, but `convert_bert_original_tf_checkpoint_to_pytorch.py` is now placed in https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py (I found the information in https://github.com/huggingface/transformers/issues/9556).

It seems that in https://github.com/huggingface/transformers/pull/9217 the document is updated to use a prefix to get the `release` variable.
However, perhaps the document is not yet changed to match the folder structure in v4?

Sorry if I misunderstand something.


BertTokenizer and encode_plus()

I see that from version 2.4.0 I was able to use `encode_plus()` with `BertTokenizer`

However it seems like that is not the case anymore.
`AttributeError: 'BertTokenizer' object has no attribute 'encoder_plus'`

Is there a replacement to `encode_plus`?


Add t5 convert to transformers-cli

# What does this PR do?

add t5 model convert to transformers-cli

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@patrickvonplaten @sgugger @LysandreJik
and
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.


AutoModelForMaskedLM not working when using MBartForConditionalGeneration architecture.

## Environment info
- `transformers` version: 4.2.1
- Platform: Linux-4.4.0-197-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten @LysandreJik

## Information

Model I am using (Bert, XLNet ...): BARThez, MBART

## To reproduce
```python
from transformers import (
    BarthezTokenizer,
    AutoModelForMaskedLM,
    MBartForConditionalGeneration
)

barthez_tokenizer = BarthezTokenizer.from_pretrained("moussaKam/barthez")
barthez_model = AutoModelForMaskedLM.from_pretrained("moussaKam/barthez")
```
error:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-cef569c55032> in <module>
      9
     10 barthez_tokenizer = BarthezTokenizer.from_pretrained("moussaKam/barthez")
---> 11 barthez_model = AutoModelForMaskedLM.from_pretrained("moussaKam/barthez")
     12
     13 input_ids = torch.tensor(

~/anaconda3/envs/transformers/lib/python3.8/site-packages/transformers/models/auto/modeling_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1123                 pretrained_model_name_or_path, *model_args, config=config, **kwargs
   1124             )
-> 1125         raise ValueError(
   1126             "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
   1127             "Model type should be one of {}.".format(

ValueError: Unrecognized configuration class <class 'transformers.models.mbart.configuration_mbart.MBartConfig'> for this kind of AutoModel: AutoModelForMaskedLM.
Model type should be one of LayoutLMConfig, DistilBertConfig, AlbertConfig, BartConfig, CamembertConfig, XLMRobertaConfig, LongformerConfig, RobertaConfig, SqueezeBertConfig, BertConfig, MobileBertConfig, FlaubertConfig, XLMConfig, ElectraConfig, ReformerConfig, FunnelConfig, MPNetConfig, TapasConfig.
```

The model works as expected when using `MBartForConditionalGeneration` instead of `AutoModelForMaskedLM`.

After checking I see that the public model [MBart](https://huggingface.co/facebook/mbart-large-cc25/blob/main/config.json) itself is using `BartForConditionalGeneration` as default architecture, is that normal?


Update integrations.py

  File "/share/apps/anaconda3/envs/my_env/lib/python3.7/site-packages/transformers/integrations.py", line 419, in __init__
    self._SummaryWriter = SummaryWriter
UnboundLocalError: local variable 'SummaryWriter' referenced before assignment

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


RAG Fine Tuning

How do we train RAG mode with custom data set.
Can we have detailed document on this.

Thanks


Error w/Transformers 4.2.0 and TF Nightly

@jplu I am running into issues when running transformers w/tf-nightly.

I get the error when I am trying to load the TFDistilBERT model:

`model = TFDistilBertModel.from_pretrained('distilbert-base-uncased')`

this is the error message:
```
ImportError:
TFDistilBertModel requires the TensorFlow library but it was not found in your environment. Checkout the instructions on the
installation page: https://www.tensorflow.org/install and follow the ones that match your environment.
```

I came across this bug when running a CI test for Ludwig. I think many projects use tf-nightly in their CIs tests to make sure that the integrations are future proof!


Does the latest huggingface-transformers version work with tokenizers==0.10.0?

## Environment info

- `transformers` version: 4.3.0.dev0
- `tokenizers` version: 0.10.0
![image](https://user-images.githubusercontent.com/29818505/104864385-c5d34300-5973-11eb-9e26-d804642e9331.png)


Easier perplexity computation

Feature request

Request for a new feature

# 🚀 Feature request

The docs provide a method to evaluate perplexity for a GPT-2 model, one example at a time (https://huggingface.co/transformers/perplexity.html). However this can potentially be included in the library with the computation being done in a batched manner.

## Motivation

This would make it easier and faster for people to evaluate their language models in terms of perplexity.

If not a solution integrated in the library, the example given in the docs can be updated to do computation in a batched manner for speed.


Training Bert2Bert with EncoderDecoderModel and Seq2SeqTrainer results with Cuda OOM

Hi,
I am trying to train a Bert2Bert model for text summarization. I followed the exact steps in [BERT2BERT for CNN/Dailymail](https://colab.research.google.com/drive/1Ekd5pUeCX7VOrMx94_czTkwNtLN32Uyu?usp=sharing#scrollTo=7zdm50ZotZqb). Only things that I changed are the training arguments and metrics. Additionally I have also tried to replace [seq2seq_trainer](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/seq2seq_trainer.py) with Seq2SeqTrainer from the package itself, the result was the same. I am using ``bert-base-uncased`` model for BERT and CNN/Dailymail as dataset (just like it was introduced in the [colab](https://colab.research.google.com/drive/1Ekd5pUeCX7VOrMx94_czTkwNtLN32Uyu?usp=sharing#scrollTo=7zdm50ZotZqb)).

    from transformers Seq2SeqTrainingArguments, Seq2SeqTrainer

    def compute_metrics(pred):
        labels_ids = pred.label_ids
        pred_ids = pred.predictions

        # all unnecessary tokens are removed
        pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
        labels_ids[labels_ids == -100] = tokenizer.pad_token_id
        label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)

        rouge_output = rouge.compute(predictions=pred_str, references=label_str, rouge_types=["rouge1", "rouge2"])
        rouge1 = rouge_output["rouge1"].mid
        rouge2 = rouge_output["rouge2"].mid

        return {
            "rouge1_precision": round(rouge1.precision, 4),
            "rouge1_recall": round(rouge1.recall, 4),
            "rouge1_fmeasure": round(rouge1.fmeasure, 4),
            "rouge2_precision": round(rouge2.precision, 4),
            "rouge2_recall": round(rouge2.recall, 4),
            "rouge2_fmeasure": round(rouge2.fmeasure, 4),
        }

    training_args = Seq2SeqTrainingArguments(
        output_dir=output_folder,
        logging_dir=log_folder,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size,
        predict_with_generate=True,
        evaluation_strategy=EvaluationStrategy.STEPS,
        do_train=True,
        do_eval=True,
        logging_steps=1000,  # set to 1000 for full training
        load_best_model_at_end=True,
        metric_for_best_model='rouge1_fmeasure',
        eval_steps=8000,  # set to 8000 for full training
        warmup_steps=2000,  # set to 2000 for full training
        overwrite_output_dir=True,
        save_total_limit=2,
        fp16=True,
    )

    trainer = Seq2SeqTrainer(
        model=bert2bert,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_data,
        eval_dataset=val_data,
    )

Even with ``batch_size=1``, I am getting the OOM. It seems like the cuda does not free any memory at all.

versions of my ``transformers`` and ``torch`` are as followed.
`
transformers              4.2.0, torch                     1.7.1+cu110
`

Can you help me with this issue? What do you think the issue might be?


RAG : Adding end to end training for the retriever (both question encoder and doc encoder)

Request for a new feature

# 🚀 Feature request

As mentioned in this recent paper [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://arxiv.org/abs/2101.00408), we can get better results for QA tasks if we fine-tune the retriever section in an end-to-end manner.

## Paper's method

Fine-tune both doc-encoder and question encoder and update the pre computed index embedding in every 500 steps.