Speed up repetition penalty logits processor

# 🚀 Feature request
Hey Team, Thanks for the great work on the project to date! This is more of an enhancement so putting it here instead of as a bug.

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
The [`RepetitionPenaltyLogitsProcessor`](https://github.com/huggingface/transformers/blob/a1bbcf3f6c20e15fe799a8659d6b7bd36fdf11ed/src/transformers/generation_logits_process.py#L147-L155) which is used to enforce the repetition penalty when generating tokens from a Seq2Seq head is extremely slow for long sequences due to the nested loops.

A vectorized implementation will be much much faster.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->
Here's a minimal example to reproduce the slow behavior:
```
import torch
from transformers import RepetitionPenaltyLogitsProcessor

import timeit

def vectorized(input_ids, scores, penalty):
    score_range = torch.arange(scores.shape[0])
    score = scores[score_range[:, None], input_ids]
    score[score >= 0] = score[score >= 0] / penalty
    score[score < 0] = score[score < 0] * penalty
    scores[score_range[:, None], input_ids] = score

input_ids = torch.randint(0, 10000, (256, 256))
scores = torch.randn(256, 10000)

rep_proc = RepetitionPenaltyLogitsProcessor(2.0)

print(f"Existing impl time for 10 iterations on CPU = {timeit.timeit(lambda: rep_proc(input_ids, scores), number=10)}")
print(f"Vectorized impl time for 10 iterations on CPU = {timeit.timeit(lambda: vectorized(input_ids, scores, 2.0), number=10)}")

if torch.cuda.is_available():
    input_ids = input_ids.cuda()
    scores = scores.cuda()

    print(f"Existing impl time for 10 iterations on GPU = {timeit.timeit(lambda: rep_proc(input_ids, scores), number=10)}")
    print(f"Vectorized impl time for 10 iterations on GPU = {timeit.timeit(lambda: vectorized(input_ids, scores, 2.0), number=10)}")
```
Here's the speedups on CPU and GPU with the vectorized version:
```
Existing impl time for 10 iterations on CPU = 23.23520456800179
Vectorized impl time for 10 iterations on CPU = 0.035849231004249305
```
```
Existing impl time for 10 iterations on GPU = 42.0977192690043
Vectorized impl time for 10 iterations on GPU = 0.008036320999963209
```
These numbers are from a machine with 16vCPUs@2.3GHz and an Nvidia T4 GPU.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
I'll have a PR up for this shortly.

@patrickvonplaten pinging you on this for your thoughts because I saw your last few commits on this code.

Thanks!


Fix model templates

# What does this PR do?

This PR fixes the model templates that were broken by the recent reorganization (in truth they never worked, it fixes that too :-p).


PEGASUS do not have mask token

@mfuntowicz @patrickvonplaten

Hi,

I am using PEGASUS  google/pegasus-large

I would like to fill the mask sentence of a document , i.e. try the pretraining task. But I don't find the mask_token.

Steps to reproduce the behavior:

1. from transformers import PegasusTokenizer
2. tok = PegasusTokenizer.from_pretrained("google/pegasus-large")
3. tok.mask_token

output is "Using mask_token, but it is not set yet."


Fix missing space in unavailable PyTorch/TensorFlow warning

# What does this PR do?
Fixes missing space in unavailable PyTorch/TensorFlow warning

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Improving performance results for BERT

I'm using the bert-base-german-cased model to perform token classification with custom NER labels on a dataset of German court documents. I have 11 labels in total (including the O label), which are however not tagged in BIO form. I'm letting the model train and evaluate on an NVidia GeForce GTX Titan X.

But despite the good ressources and the model, which was actually pretrained on German judicial documents, the results are rather lacking.

```
precision    recall  f1-score   support

                              Date       0.87      0.99      0.93       407
                   Schadensbetrag       0.77      0.58      0.66       112
                            Delikt       0.59      0.50      0.54        44
                    Gestaendnis_ja       0.60      0.71      0.65        21
                    Vorstrafe_nein       0.00      0.00      0.00         6
Strafe_Gesamtfreiheitsstrafe_Dauer       0.76      0.91      0.83        35
          Strafe_Gesamtsatz_Betrag       0.42      0.52      0.46        25
           Strafe_Gesamtsatz_Dauer       0.52      0.82      0.64        28
                 Strafe_Tatbestand       0.30      0.29      0.30       283

                        micro avg       0.65      0.68      0.66       961
                        macro avg       0.54      0.59      0.56       961
                     weighted avg       0.64      0.68      0.66       961

```

What could be some steps to improve these results?
Perhaps it's the low data count for some of the labels, or that the labels often are not single tokens but text spans of multiple tokens?

I would be glad for every hint of some more experienced users. I can also share data or other files, if they are relevant.

This is my config file:

```
{
    "data_dir": "./Data",
    "labels": "./Data/labels.txt",
    "model_name_or_path": "bert-base-german-cased",
    "output_dir": "./Data/Models",
    "task_type": "NER",
    "max_seq_length": 180,
    "num_train_epochs": 6,
    "per_device_train_batch_size": 48,
    "seed": 7,
    "fp16": true,
    "do_train": true,
    "do_predict": true,
    "do_eval": true
}
```


Fix init for MT5

# What does this PR do?

Fix the init, the config should be imported outside of tests.


Cannot train model from scratch using `run_mlm.py`.

Looks like the trainer does not like when it gets a `None`, so when we train from scratch, there is a `None` in this `if` and crashes:

https://github.com/huggingface/transformers/blob/a6cf9ca00b74a8b2244421a6101b83d8cf43cd6b/examples/language-modeling/run_mlm.py#L357

I solved it by deleting that line, but I guess it could affect to other use cases.

To reproduce, call `run_mlm` this way (I guess it is easier to reproduce, but this might be enough):
```
python  run_mlm.py \
    --model_type bert \
    --train_file ./data/oscar_1000.txt \
    --validation_file ./data/oscar_1000_valid.txt \
    --output_dir testing_model \
    --tokenizer_name bert-base-spanish-wwm-cased  \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --evaluation_strategy steps \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --max_steps 500 \
    --save_steps 2000 \
    --save_total_limit 15 \
    --overwrite_cache \
    --max_seq_length 512 \
    --eval_accumulation_steps 10 \
    --logging_steps 1000 \
```

The dataset I'm using I guess that isn't relevant so any corpus will do.

 @sgugger


[MT5] More docs

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

^^I knew that I forgot something with the docs...

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Hosting and online deployment of a transformer chatbot (built with huggingface library)

I'm building a chatbot using BERT for a startup company. At some point it will be deployed online. It turns out, most chatbot hosting services actually want to sell you a chatbot rather than host the one you developed, which is obviously not an option for us, especially that the solution is opensource (pytorch+huggingface).

I would like to know of a hosting service that 1) accepts custom-built chatbot solution, 2) can accommodate up to 500 online users at any given time, 3) does not charge exorbitant prices. We can't buy a solution, or even augment an existing one, because it is a specific requirement of the project funding (research rather than development).


The Albert tokenizer file cannot download automatically and the official Albert tokenizer file is wrong, I cannot use it.

Have anybody meet the same problem?


Tokenizers: ability to load from model subfolder

Should fix #8447


Fix rag finetuning + add finetuning test

Following #7715 we need more test coverage of the RAG example scripts.
In this PR I'm adding a test for the finetuning script.
The test includes a single gpu test and a multi gpu test. Both are passing.

As mentioned in #7816 and #8345 there were some errors in the script that I had to fix.

Moreover since @amogkam has been working on the finetuning script as well to integrate Ray, I made sure to reduce the possible conflicts with his PR #8583 . More precisely I'm reusing the CustomAccel class that will allow to init either the pytorch distributed retrieval or the ray distributed retrieval.

Also fixed a bug in RAG forward pass (see #8665 )

Fix #7816
Fix #8345


Add output control for TFGPT2LMHeadModel

# What does this PR do?

Hi, guys. This pull request is going to add a parameter for `GPT2Config` and `TFGPT2LMHeadModel`, to control whether return multi-layer logits or not when training and fine-tuning.

Related issue #8503

Before this change, we need to assign the 'special' loss and metric:
```python
class MyMetrice(tf.keras.metrics.SparseCategoricalAccuracy):

    def update_state(self, y_true, y_pred, sample_weight=None):
        # It will receive output from all layers by default.
        if len(y_pred.shape) > 3:
            return 0
        return super(Mymetrice, self).update_state(y_true, y_pred, sample_weight)

model = TFGPT2LMHeadModel.from_pretrained("distilgpt2")
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = MyMetrice('accuracy')
model.compile(
    optimizer=optimizer,
    loss=[loss, *[None] * model.config.n_layer],    # It's hard to guess if there is no example
    metrics=[metric]
)
```
After this change, we can train or finetune samples easily.
```python
model = TFGPT2LMHeadModel.from_pretrained("distilgpt2", return_dict=False)
model.compile(
    optimizer=optimizer,
    loss=model.compute_loss,
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy]
)

```

 GPT2: @LysandreJik, @patrickvonplaten
 tensorflow: @jplu

Hope this helps.


[RAG] Add Ray implementation for distributed retrieval

# What does this PR do?
This PR adds a new distributed retriever implementation for RAG built on Ray, as an alternative to the current retriever implementation that uses torch.distributed. With Ray it's possible to load the index on multiple processes instead of just the rank 0 training worker, allowing fine tuning to scale out better to multiple GPUs, and also allowing the index to potentially be fit in GPU memory. This also removes a core dependency on Pytorch, allowing a Tensorflow implementation of `finetune.py`.

This PR also makes changes to support finetune.py with Pytorch Lightning >v1.0.

A benchmark of Pytorch distribtued retrieval vs. Ray distributed retrieval
![image](https://user-images.githubusercontent.com/8068268/99327675-4ae49380-282f-11eb-8126-6250558fd795.png)

## Implementation Details
In the current Pytorch retrieval implementation, the index is loaded once on just the rank 0 training workers. Training worker 0 gathers the inputs from all other workers, performs the index lookup, and scatters the results back to the other workers.
![image](https://user-images.githubusercontent.com/8068268/99328533-238ec600-2831-11eb-9d26-e153cd0de74f.png)

With the Ray implementation, the index is loaded on *separate* processes, which are referred to as Ray actors. Each training worker randomly selects a retrieval actor to query for documents and Ray handles all the communication between the processes. Because the index can be loaded in *multiple* processes, training can scale up since no synchronization needs to happen for the index lookup.
![image](https://user-images.githubusercontent.com/8068268/99328651-618bea00-2831-11eb-990b-f1727df15190.png)

Note that Pytorch Lightning is still handling distributed *training*, but Ray manages distributed *retrieval*. Because PTL calls the entire training script under the hood multiple times, we have to use Ray's named actors feature (https://docs.ray.io/en/master/actors.html?highlight=named%20actors#named-actors) allowing the retrieval actors to be referenced by all training processes. The use of named actors is necessitated by how PTL handles distributed training, and a simpler approach could probably be used for a Tensorflow implentation.

## Testing Strategy
Unit tests were added to `test_distributed_retriever.py`. Note that the local Ray cluster for the tests had to be started with `local_mode=True` because the test file modifies `sys.path` and these changes are not propagated to remote processes. See https://stackoverflow.com/questions/54338013/parallel-import-a-python-file-from-sibling-folder for more info.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


[examples tests] tests that are fine on multi-gpu

This PR removes `require_torch_non_multi_gpu_but_fix_me` for those tests I know should work. Well, they used to work before some recent PRs https://github.com/huggingface/transformers/pull/8073#issuecomment-728677627 - my suspicion is that the problem wasn't detected before it was merged because these tests were skipped as the dev was probably on a multi-gpu machine. So we need to sort this issue out sooner than later.

Currently a bunch of `examples` tests get skipped for devs with multi-gpus - it's probably a good idea for each dev with more than 1 gpu to take a sub-folder under `examples` and test which tests can be run on multi-gpu and which can't.

1. ensure you're on a multi-gpu machine - don't take this assignment if you don't have 2+ gpus
2. pick a test file and remove `@require_torch_non_multi_gpu_but_fix_me` from its tests if any
3. don't forget RUN_SLOW=1 - probably just always add it so tests aren't missed.
4. run tests
5. if any tests fail either fix those or restore `@require_torch_non_multi_gpu_but_fix_me` for the failing tests. if the failure is not multi-gpu related please file an issue.
6. go to 2 until no more test files is left.
7. send a PR with changes for this sub-folder you chose.

To remind, the initial skip-all-examples-tests-on-multi-gpu tests was added so that we could start multi-gpu tests on github runner CI.

If meanwhile you have a problem with important examples tests skipping, please force a single gpu mode with:

```
CUDA_VISIBLE_DEVICES=0 pytest
```

I trust @LysandreJik or someone else can coordinate this effort?


Add early stopping callback to pytorch trainer

# Summary

Address PyTorch half of https://github.com/huggingface/transformers/issues/4894 by adding early stopping patience and a minimum threshold metrics must improve to prevent early stopping. I piggybacked heavily off of https://github.com/huggingface/transformers/pull/7431/ since the two functions are very similar.

Since https://github.com/huggingface/transformers/pull/4186 seems to be abandoned and behind master, I figured I'd take a crack at this.

## Who can review?

Anyone! But @julien-c and @sgugger seem the most appropriate.


Reorganize repo

# What does this PR do?

This PR reorgnaizes the structure of the repository by putting all model-related files (modeling, configuration, tokenization, conversion) in subfolders.

**Breaking change**

This breaks any import for model/config/tokenizer objects that is not done at the top level:
```
from transformers import BertModel
```
works but not
```
from transformers.modeling_bert import BertModel
```
It needs to be updated to
```
from transformers.models.bert.modeling_bert import BertModel
```

Internally, after this PR is merged the following will need fixing:
- the check_repo script does not properly finds the models now (so it does not check if they are properly tested/documented/ in auto classes)
- the new model template needs to be updated

The other internal scripts work as usual.


Create README.md

model card

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adding model card for `indolem/indobert-base-uncased`

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


In the code below, while using `batch_encode_plus`, i get the error saying that i asked ***“to return token_type_ids while setting add_special_tokens to False”***, when `return_token_type_ids` is `False`. I'm not sure if i am comprehending the error message correctly. For this specific case, I also found that the behaviour between `BertTokenizer` and `BertTokenizerFast` is different.

```python
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained("bert-base-uncased", add_special_tokens=False)

txt = ["huggingface", "transformers"]
t.batch_encode_plus(
    txt,
    add_special_tokens=False,
    return_attention_mask=False,
    return_token_type_ids=False,
)
```
Error message
```
Traceback (most recent call last):
  File "/Users/charcohui/Desktop/env/mlenv/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-30-226d8ccb1c88>", line 5, in <module>
    return_token_type_ids=False,
  File "/Users/charcohui/Desktop/env/mlenv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2399, in batch_encode_plus
    **kwargs,
  File "/Users/charcohui/Desktop/env/mlenv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 567, in _batch_encode_plus
    verbose=verbose,
  File "/Users/charcohui/Desktop/env/mlenv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 617, in _batch_prepare_for_model
    verbose=verbose,
  File "/Users/charcohui/Desktop/env/mlenv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2681, in prepare_for_model
    "Asking to return token_type_ids while setting add_special_tokens to False "
ValueError: Asking to return token_type_ids while setting add_special_tokens to False results in an undefined behavior. Please set add_special_tokens to True or set return_token_type_ids to None.
```

However, it works when `BertTokenizerFast` is used:
```python
tfast = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True, add_special_tokens=False)
tfast.batch_encode_plus(
    txt,
    add_special_tokens=False,
    return_attention_mask=False,
    return_token_type_ids=False,
)
# {'input_ids': [[17662, 12172], [19081]]}
```


[examples/seq2seq] fix PL deprecation warning

This PR fixes a PL deprecation warning, since we require PL-1.0.4 this is a safe switch.

Reference: https://pytorch-lightning.readthedocs.io/en/latest/generated/pytorch_lightning.callbacks.ModelCheckpoint.html#pytorch_lightning.callbacks.ModelCheckpoint.params.filepath

@patrickvonplaten


run_pl_glue.py token_type_id error on fresh install

If you try to run the run_glue.py example with e.g. roberta from a fresh install of the library, it errors out with the following error:

```
Traceback (most recent call last):
  File "examples/text-classification/run_pl_glue.py", line 228, in <module>
    main()
  File "examples/text-classification/run_pl_glue.py", line 218, in main
    trainer = generic_train(model, args)
  File "/home/ejp416/complexity/examples/lightning_base.py", line 400, in generic_train
    trainer.fit(model)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1072, in fit
    model = self.accelerator_backend.setup(model)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/accelerators/gpu_backend.py", line 34, in setup
    self.trainer.call_setup_hook(model)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1444, in call_setup_hook
    model.setup(stage_name)
  File "/home/ejp416/complexity/examples/lightning_base.py", line 175, in setup
    self.train_loader = self.get_dataloader("train", self.hparams.train_batch_size, shuffle=True)
  File "examples/text-classification/run_pl_glue.py", line 98, in get_dataloader
    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
TypeError: an integer is required (got type NoneType)
```

To reproduce, run e.g.

```
python examples/text-classification/run_pl_glue.py --model_name_or_path roberta-base --output_dir ./blah --task mnli --do_train --data_dir ./glue_data/MNLI --max_seq_length 512 --max_grad_norm inf --adam_epsilon 1e-6 --weight_decay 0.1 --num_train_epochs 2 --train_batch_size 2 --eval_batch_size 4 --learning_rate 1e-5 --seed 12 --gradient_accumulation_steps 8 --gpus 1
```

The reason is that roberta does not have segment ids so token_type_ids is set to null in the data loader, causing torch.tensor to freak out. There's probably a more elegant long-term solution for this, but it's easy to fix by just setting it to 0 instead of null for those models. This issue has come up before in other scripts:
- https://github.com/huggingface/transformers/pull/3801
- https://github.com/huggingface/transformers/issues/3810


REALM checkpoints to pytorch checkpoints

Will it be possible to convert checkpoints in https://console.cloud.google.com/storage/browser/realm-data/cc_news_pretrained to pytorch implementations ?
I am not very familiar with Tensorflow....
I tried using convert_bert_original_tf_checkpoint_to_pytorch.py but I am not sure I am invoking it correctly.
Here is how I am invoking the python script...
--tf_checkpoint_path="./cc_news_pretrained/embedder/encoded/" \
 --bert_config_file="./bert_config.json" \
 --pytorch_dump_path="./pytorch"

I am using tensorflow 2.3.0.

The checkpoint file has the following entries which are probably internal developer files(?):
model_checkpoint_path: "/cns/li-d/home/lumiere/public/models/gatoatigrado/ner-with-dates/10923195/1-active_losses=mlm_loss/export/temp/1580364602/retriever/encoded/encoded.ckpt"
all_model_checkpoint_paths: "/cns/li-d/home/lumiere/public/models/gatoatigrado/ner-with-dates/10923195/1-active_losses=mlm_loss/export/temp/1580364602/retriever/encoded/encoded.ckpt"

1)When I set tf_checkpoint_path to the directory containing the checkpoint, I get the error :
tensorflow.python.framework.errors_impl.NotFoundError: /cns/li-d/home/lumiere/public/models/gatoatigrado/ner-with-dates/10923195/1-active_losses=mlm_loss/export/temp/1580364602/retriever/encoded; No such file or directory

2)When I set tf_checkpoint_path to the checkpoint file encode.ckpt.data-00000-of-00001, I get the error:
env/lib/python3.6/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 95, in NewCheckpointReader
    return CheckpointReader(compat.as_bytes(filepattern))
RuntimeError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for ./cc_news_pretrained/embedder/encoded/encode.ckpt.data-00000-of-00001

Thanks!


[Improvements] Enable `git push` without requiring login when uploading model

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Is there a way to allow `git push` a new model version without requiring login?

`transformers-cli` will auto generate a key for the user. Is there a way to leverage this key?

## Motivation

GitHub allows you to push in VSCode without explicitly login in each time.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->


Bert that receives text triplet as an input

I would like to train bert on triplets of texts as inputs (for example, something like (context, question, answer)). encode_plus (https://huggingface.co/transformers/internal/tokenization_utils.html#transformers.tokenization_utils_base.PreTrainedTokenizerBase.encode_plus) receives either a single text, or a text_pair. Is there a way to use it with triplets?


Fix mixed precision issue for GPT2

# What does this PR do?

This PR makes GPT2 to be trained and run in any mixed precision.

Fixes # (issue)
#8559


[WIP] Move BERT and ALBERT

# What does this PR do?

This is a PoC for the reorg of the model on just two files. I didn't update very reference everywhere, just enough to give a sense of what it will render.


[T5] Add open / closed book answering models

# 🌟 New model addition

## Model description
 Check here: https://github.com/google-research/google-research/tree/master/t5_closed_book_qa
<!-- Important information -->

## Open source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


After 2nd iteration: always same result when training in a loop

I train a BERT model on a binary classification task. I do the training 4 times in a row. With same train and validation data and with the exact same hyperparameters. I use the default seed in TrainerArguments and set no other seeds.

The results of the 2nd, 3rd and 4th iteration are 100% the same. The result of the 1st run is unique. This behavior is 100% reproducible.

It is not clear why this is the case. Since I set a seet and work with same data.

PIP Libs I use (no conda libs):
- sentencepiece-0.1.91
- tokenizers-0.9.3
- transformers-3.5.1
- torch-1.7.0+cu101

Screenshot:
![grafik](https://user-images.githubusercontent.com/229382/99275325-a57bf080-282b-11eb-843d-fb8c38dc8e3d.png)

Colab to reproduce:
https://colab.research.google.com/drive/1HjQ7p5AlDY9kcWo7uSXzteh7x1ustEdN?usp=sharing

I already did some facy stuff after each iteration:
```python
    del trainer
    del training_args
    del model
    del config
    del train_result
    del tokenizer
    del labeled_dataset_test
    del labeled_dataset_train
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()
```

But that does not help. Can someone please help me here and claify whats up?

PS: I do not think that a seed can be the reason. If a seed would be the reason the 1st and 2nd run would also be the same and AFAIK a GPU training is not 100% deterministic. So a small difference would still remain.


Update version to v4.0.0-dev

# What does this PR do?

This PR puts the right version in the setup and `__init__` and adds one last step to the release guide in the setup.

Fixes #8566


[XLNet] Fix mems behavior

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #7584

XLNet is arguably not the cleanent implementation with a lot of config parameters flying around that all interact with each other in an overly complicated way: `use_cache`, `mem_len`, and `reuse_len`. Due to backward compatibility we cannot really get rid of those. I'd love to just completely get rid of `use_cache` and `reuse_len`, but this would require to update all configs which is not possible...

At first, this PR removes the `use_cache` and replaces it with `use_mems_eval` => `use_mems_eval` decides whether the mems should be used in evaluation mode, which defaults to `True` so that the arguably most important model `XLNetLMHeadModel` keeps full backward compatibility at inference. `use_cache` is a confusing name IMO because it does not correspond to the `use_cache` we know from GPT2 (we had a longer discussion on this internally on Slack).

The issue #7584 shows that if there is one training batch that is smaller than the other batches in the train data, the training breaks. Also as can be read upon in the issue linked below, the authors also don't use the memory mechanism for fine-tuning => therefore we add another param `use_mems_train` which defaults to `False` so that training works as a default.
If for some special reason the user wants to use the memory mechanism during fine-tuning, he/she has to make sure that the batch_size of all training batches is the same.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


## Environment info

I try
```
!git clone https://github.com/huggingface/transformers.git
%cd transformers/
!pip install -e .
```
on Colaboratory.

- `transformers` version: 3.5.0   <- this seems strange.
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

 documentation: @sgugger

## Information

"setup.py" does not seem to have been updated for v3.5.1.

When I install transformers by `pip install -e .`, the version of transformers is shown as v3.5.0.

## To reproduce

Steps to reproduce the behavior:

I try
```
!git clone https://github.com/huggingface/transformers.git
%cd transformers/
!pip install -e .
```
on Colaboratory after v3.5.1 release.

Then,
```
import transformers
transformers.__version__
```
returns
```
'3.5.0'
```

## Expected behavior

The return of `transformers.__version__` is expected to be '3.5.1' now, if my understanding is not wrong.

Maybe, in  https://github.com/huggingface/transformers/blob/afb50c663a5d5623906ead1e87481926467d59fa/setup.py#L120

'3.5.0' should be changed to '3.5.1'.

Is my understanding correct? Sorry if I misunderstand your intension.


replace performance table with markdown

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Make BART more ONNX friendly

**Major concerns:**
1. ONNX complains about Python raw integer usage.
2. ONNX doesn't support boolean indexing with something else than vector. The code was using 2D Tensor indices (batch, token).

**PR workarounds:**
1. Remove the call to `len(..)` and prefer the use of `.size(-1)`
2. Attempt to index the output tensors to retrieve only the **last** EOS token over the sequence axis through `.index_select(..)`


Wrong model_max_length for BERTOverflow tokenizer

## Environment info

- `transformers` version: 3.4.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: N
- Using distributed or parallel set-up in script?: N

## Information

Hi. I used [BERTOverflow](https://huggingface.co/jeniya/BERTOverflow) and found strange behavior for the tokenizer property `model_max_length`. It is equal 1000000000000000019884624838656, although it should be 512.

## To reproduce
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('jeniya/BERTOverflow')
print(tokenizer.model_max_length)
```


Clearer Model Versioning Example in Model Card

Related to pretrained model cards

Clearer model card example


Reset loss to zero on logging in Trainer to avoid bfloat16 issues

# What does this PR do?

I have had a weird issue with the logged loss going to zero after some training steps training with `bfloat16` on v3 TPUs:

![W B Chart 11_16_2020, 12_00_44 PM](https://user-images.githubusercontent.com/13353204/99245133-63d74f80-2803-11eb-8cc0-ce326b2a82b0.png)

while it works correctly using 32bit precision:

![W B Chart 11_16_2020, 12_01_54 PM](https://user-images.githubusercontent.com/13353204/99245229-89645900-2803-11eb-988c-324e4e6d6267.png)

After some investigation I found that the `tr_loss` variable in `Trainer.train` seems to overflow after reaching 1024 (?).
I did not track this down more closely because it is easily fixed by making `tr_loss` a regular Python float instead of a tensor. It doesn't actually need to be a tensor as it is only ever accessed by `.item()`.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Prophetnet - predicted n-future tokens

Hi,

How can we get the predicted n-future tokens **as a string** data from the model output. I couldn't find it in the API doc and sample code. Could you please guide / provide the code snippet for ProphetNet and XLM-Prophetnet ? Thanks in advance.


TFGPT2LMHeadModel fp16 support

## Environment info

- `transformers` version:
- Platform: ubuntu 18.04
- Python version: python3.8
- Tensorflow version (GPU?): tf-nightly==2.5
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N

### Who can help

 albert, bert, GPT2, XLM: @LysandreJik
 Text Generation: @patrickvonplaten @TevenLeScao
 tensorflow: @jplu

## Information

Hi, there. If I want to use the mixed precision setting with keras apis when training `TFGPT2LMHeadModel`, [like this](https://www.tensorflow.org/guide/mixed_precision):
```python
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)
```
Then I will got this error:
```
  File "/home/mymusise/pro/fast-gpt2/env/lib/python3.8/site-packages/transformers/modeling_tf_gpt2.py", line 154, in call
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask, output_attentions, training=training)
  File "/home/mymusise/pro/fast-gpt2/env/lib/python3.8/site-packages/transformers/modeling_tf_gpt2.py", line 101, in _attn
    w = w / tf.math.sqrt(dk)
  File "/home/mymusise/pro/fast-gpt2/env/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 1181, in binary_op_wrapper
    raise e
  File "/home/mymusise/pro/fast-gpt2/env/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 1165, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/mymusise/pro/fast-gpt2/env/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/home/mymusise/pro/fast-gpt2/env/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 1337, in truediv
    return _truediv_python3(x, y, name)
  File "/home/mymusise/pro/fast-gpt2/env/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 1267, in _truediv_python3
    raise TypeError("x and y must have the same dtype, got %r != %r" %
TypeError: x and y must have the same dtype, got tf.float16 != tf.float32
```
Here's a [example](https://gist.github.com/mymusise/7192b7c252ff67ff84496cd8b27a91ff) to reappear this.

Please help me guys.


Readme for Wiki Summary [Persian] bert2bert

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Readme for News Headline Generation (bert2bert)

Related to pretrained model cards


tokenization_bart.py: return_tensors default should be "pt"

return_tensors default should be "pt" in bart's `prepare_seq2seq_batch`.


Allow the user to input positional embeddings

# 🚀 Feature request

Hi,

I think that allowing the user to input positional embeddings in the same way that `inputs_embeds` can be fed directly to the Transformer would be greatly appreciated. (If this is already possible, please let me know :) ).

## Motivation

This is important for multimodal approaches, such as **VisualBERT** (https://arxiv.org/pdf/1908.03557.pdf), where each modality requires a separate positional embedding. The user could then use a `torch.nn.Embedding` per modality and concatenate the positional embeddings and feed this to the Transformer along with the concatenated input embeddings of the modalities.

This would also resolve other related issues:

https://github.com/huggingface/transformers/issues/5095
https://github.com/huggingface/transformers/issues/3395

## Your contribution

I can help in any form.


`disable_ngram_loss` fix for prophetnet

# What does this PR do?

This PR fixes `disable_ngram_loss` behaviour for ProphetNetForConditionalGeneration and is related to #8553

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

Fixes #8553

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
I guess @patrickvonplaten was using this model (I saw models on hub), sorry if I am wrong, but there is no one to tag for ProphetNet


`disable_ngram_loss` doesn't work correctly in ProphetNetForConditionalGeneration

When I am using ProphetNet with `disable_ngram_loss=True` I am getting loss that is greater than with `disable_ngram_loss=False`. It seems to me that this is the problem of setting `fill_(self.padding_idx)` in `_compute_loss` instead of -100 so that ngram part is omitted in loss calculation

Also I think that `loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="sum")` reduce should be set to `mean` so that model loss is comparable between models working on the same task (like `mbart`). Can somebody tell me if it's a good point or should I leave it as it is? I am planning to add PR with this changes.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.5.0 from source
- Platform: macOS Catalina
- Python version: 3.7.5
- PyTorch version (GPU?): 1.6.0

### Who can help

I can't figure out whom to tag.

## Information

Model I am using (Bert, XLNet ...): ProphetNetForConditionalGeneration

## To reproduce

```
from transformers import XLMProphetNetTokenizer, XLMProphetNetForConditionalGeneration
tokenizer = XLMProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased')
model = XLMProphetNetForConditionalGeneration.from_pretrained('microsoft/xprophetnet-large-wiki100-cased')
inputs = tokenizer('Hi my name is', return_tensors='pt').input_ids
targets = tokenizer('Hi my name is John', return_tensors='pt').input_ids

model_loss = model(input_ids=inputs, labels=targets, return_dict=True).loss

model.disable_ngram_loss = True
model_disable_loss = model(input_ids=inputs, labels=targets, return_dict=True).loss

from torch.nn import CrossEntropyLoss

loss_fct = CrossEntropyLoss(reduction='sum')
logits = model(input_ids=inputs, labels=targets, return_dict=True).logits
loss_cross_entropy = loss_fct(logits.view(-1, model.config.vocab_size), targets.view(-1))
```
the problem is `model_loss < model_disable_loss` and `model_disable_loss != loss_cross_entropy` which it should be I think.

Note:
`CrossEntropyLoss(reduction='sum')` is used to match implementation in `_compute_loss` (`loss = F.nll_loss(lprobs, expend_targets.view(-1), reduction="sum")`) but other models use default reduction which makes outputs incomparable (at least directly)

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
when `model.disable_ngram_loss=True` `CrossEntropyLoss` should be equal to `model(input_ids=inputs, labels=targets, return_dict=True).loss`


T5 & mT5

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds T5v1.1 and MT5.

I was really unsure whether I should make a new model file `modeling_5_v1_1.py` or not. I finally decided (also after discussions in https://github.com/huggingface/transformers/issues/6285) that in this case, it is better to add the few new T5 features to the existing `modeling_t5.py` file.

The context is the following:
The T5 team released weights for a **T5v1.1** model: https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md. These model architecture of T5v1.1 is equivalent to T5 besides two changes:
1) input and output word embedding of decoder is not shared anymore
2) different feed forward layer is used
In addition, the **mT5** model checkpoints were also released and are fully based on T5v1_1: https://github.com/google-research/multilingual-t5#released-model-checkpoints .

Now the philosophy of the library is to create a new model class if the architecture slightly differs from a previously integrated model, but I'd argue that in this case it is better to add a new config param called "feed_forward_proj" which defines whether a "relu" (T5) or a "gated-gelu" (T5v1.1) should be used. The arguments for not creating a new model class are the following:

1) Both T5 and T5v1.1 are "officially" the same model and both belong to the same code base: https://github.com/google-research/text-to-text-transfer-transformer
2) T5v1.1 has no official paper and it's quite difficult to find a good name as discussed in https://github.com/huggingface/transformers/issues/6285 => `T5v11` is a very cryptic name IMO and `T5v2` is not ideal either.
3) One could argue that it makes sense to create a mixture of T5 and T5v1v1 by sharing input and output word embedding of the decoder but using the "gated-gelu" feed-forward layer. This design would allow a user to define a model whereas creating a new T5v11 model  file would make this impossible
4) `"feed-forward-proj"` is less of a model-specific architecture configuration than `"do_blenderbot_90_layernorm"` IMO

The obvious disadvantage is that I am adding new architecture code to an existing model to make a new model (or new model version) work, which is kinda what we did not want to do.

It's a tough decision here I think, so I'd be very happy for your feedback on this @LysandreJik @sgugger @thomwolf . I'd also understand if you guys think we should make a new model file here.

I already uploaded `google/t5-v1_1-small` and `google/mt5-small`. If you guys are ok with this PR, I'll add the TF code, add the new model to README.md and we're good for merge.


"special token {} has to be either str or AddedToken but got:

Hello, two weeks ago I fine-tune the Roberta language model on my data using hugging face (codes given below) and save output at the google drive. Now when using it for semantic search with sentence transformer. I am getting the error that

**"TypeError: special token bos_token has to be either str or AddedToken but got: <class 'dict'>"**

```

from sentence_transformers import SentenceTransformer
from sentence_transformers import models, losses
import scipy.spatial
import pickle as pkl
word_embedding_model = models.RoBERTa("/content/drive/My Drive/Ottawa_citit")

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)

model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

 Previously fine-tuned Roberta

`!python "/content/transformers/examples/contrib/legacy/run_language_modeling.py" \
    --output_dir "/content/drive/My Drive/Ottawa_citit" \
    --model_name_or_path roberta-base \
    --do_train \
    --per_gpu_train_batch_size 8 \
    --seed 42 \
    --train_data_file "/content/input_text.txt" \
    --block_size 256 \
    --line_by_line \
    --learning_rate 6e-4 \
    --num_train_epochs 3 \
    --save_total_limit 2 \
    --save_steps 200 \
    --weight_decay 0.01 \
     --mlm`
```


Create README.md for Chinese RoBERTa Miniatures

Related to pretrained model cards

Create model card for Chinese RoBERTa Miniatures.


Pretrain BERT with user defined vocabulary

# 🚀 Feature request

I'm wondering if there is a way to pretrain BERT with user-defined vocabulary, number of layers/heads, and use a WordLevel Tokenizer instead of a WordPiece Tokenizer current BERT uses. Thank you!

## Motivation

This could help the BERT mode to adapt to different tasks.


Update README.md

Related to pretrained model cards

Modified Model in Action section. The class `AutoModelWithLMHead` is deprecated so changed it to `AutoModelForSeq2SeqLM` for encoder-decoder models. Removed duplicate eos token.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Upload models using Git fails

## Environment info
- `transformers` version: 3.5.0
- Platform: Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.12
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 Model Cards: @julien-c
 T5: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...):
T5

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
git clone https://huggingface.co/Rostlab/prot_t5_xl_bfd
Cloning into 'prot_t5_xl_bfd'...
remote: Enumerating objects: 31, done.
remote: Counting objects: 100% (31/31), done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 31 (delta 13), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (31/31), done.
```

`cp config.json pytorch_model.bin prot_t5_xl_bf`

`git add --all`

```
git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   config.json
        modified:   pytorch_model.bin
```

`git commit -m "[T5] Fix load weights function #8528"`

```
git push
Username for 'https://huggingface.co': xxxxxx
Password for 'https://agemagician@huggingface.co':
Counting objects: 4, done.
Delta compression using up to 80 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 4.92 GiB | 23.09 MiB/s, done.
Total 4 (delta 2), reused 1 (delta 0)
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date
```

OR

```
GIT_CURL_VERBOSE=1 git push
* Couldn't find host huggingface.co in the .netrc file; using defaults
*   Trying 192.99.39.165...
* TCP_NODELAY set
* Connected to huggingface.co (192.99.39.165) port 443 (#0)
* found 140 certificates in /etc/ssl/certs/ca-certificates.crt
* found 421 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: huggingface.co (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=huggingface.co
*        start date: Tue, 10 Nov 2020 08:05:46 GMT
*        expire date: Mon, 08 Feb 2021 08:05:46 GMT
*        issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /Rostlab/prot_t5_xl_bfd/info/refs?service=git-receive-pack HTTP/1.1
Host: huggingface.co
User-Agent: git/2.17.1
Accept: */*
Accept-Encoding: gzip
Accept-Language: C, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Unauthorized
< Server: nginx/1.14.2
< Date: Sat, 14 Nov 2020 23:43:52 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 12
< Connection: keep-alive
< X-Powered-By: huggingface-moon
< WWW-Authenticate: Basic realm="Authentication required", charset="UTF-8"
< ETag: W/"c-dAuDFQrdjS3hezqxDTNgW7AOlYk"
<
* Connection #0 to host huggingface.co left intact
Username for 'https://huggingface.co': agemagician
Password for 'https://agemagician@huggingface.co':
* Couldn't find host huggingface.co in the .netrc file; using defaults
* Found bundle for host huggingface.co: 0x55acdab63f80 [can pipeline]
* Re-using existing connection! (#0) with host huggingface.co
* Connected to huggingface.co (192.99.39.165) port 443 (#0)
* Server auth using Basic with user 'agemagician'
> GET /Rostlab/prot_t5_xl_bfd/info/refs?service=git-receive-pack HTTP/1.1
Host: huggingface.co
Authorization: Basic YWdlbWFnaWNpYW46VWRjXzEyMDA=
User-Agent: git/2.17.1
Accept: */*
Accept-Encoding: gzip
Accept-Language: C, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: nginx/1.14.2
< Date: Sat, 14 Nov 2020 23:43:59 GMT
< Content-Type: application/x-git-receive-pack-advertisement
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: huggingface-moon
<
* Connection #0 to host huggingface.co left intact
Counting objects: 4, done.
Delta compression using up to 80 threads.
Compressing objects: 100% (3/3), done.
* Couldn't find host huggingface.co in the .netrc file; using defaults
* Found bundle for host huggingface.co: 0x55acdab63f80 [can pipeline]
* Re-using existing connection! (#0) with host huggingface.co
* Connected to huggingface.co (192.99.39.165) port 443 (#0)
* Server auth using Basic with user 'agemagician'
> POST /Rostlab/prot_t5_xl_bfd/git-receive-pack HTTP/1.1
Host: huggingface.co
Authorization: Basic YWdlbWFnaWNpYW46VWRjXzEyMDA=
User-Agent: git/2.17.1
Content-Type: application/x-git-receive-pack-request
Accept: application/x-git-receive-pack-result
Content-Length: 4

* upload completely sent off: 4 out of 4 bytes
< HTTP/1.1 200 OK
< Server: nginx/1.14.2
< Date: Sat, 14 Nov 2020 23:44:02 GMT
< Content-Type: application/x-git-receive-pack-result
< Content-Length: 0
< Connection: keep-alive
< X-Powered-By: huggingface-moon
<
* Connection #0 to host huggingface.co left intact
* Couldn't find host huggingface.co in the .netrc file; using defaults
* Found bundle for host huggingface.co: 0x55acdab63f80 [can pipeline]
* Re-using existing connection! (#0) with host huggingface.co
* Connected to huggingface.co (192.99.39.165) port 443 (#0)
* Server auth using Basic with user 'xxxxxx'
> POST /Rostlab/prot_t5_xl_bfd/git-receive-pack HTTP/1.1
Host: huggingface.co
Authorization: Basic YWdlbWFnaWNpYW46VWRjXzEyMDA=
User-Agent: git/2.17.1
Accept-Encoding: gzip
Content-Type: application/x-git-receive-pack-request
Accept: application/x-git-receive-pack-result
Transfer-Encoding: chunked

Writing objects: 100% (4/4), 4.92 GiB | 23.17 MiB/s, done.
Total 4 (delta 2), reused 1 (delta 0)
* Signaling end of chunked upload via terminating chunk.
* Signaling end of chunked upload via terminating chunk.
* The requested URL returned error: 504 Gateway Time-out
* stopped the pause stream!
* Closing connection 0
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date
```

## Expected behavior

I had issue before for uploading big models like T5 "#7480".
It was marked as solved after moving to model versioning:
https://github.com/huggingface/transformers/pull/8324
However, @patrickvonplaten  fixed some problems in T5 weights :
https://github.com/huggingface/transformers/pull/8528
I tried to update the model but it still doesn't work as showed above.

 I tried several tricks like:
https://stackoverflow.com/questions/54061758/error-rpc-failed-http-504-curl-22-the-requested-url-returned-error-504-gatewa
But could not solve it.

Any ideas ?


Failed in predict function after converting xlnet model to onnx format

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:ubuntu 18.04
- Python version:3.7.9
- PyTorch version (1.7.0):
- Using GPU in script:No
- Using distributed or parallel set-up in script?:No
- onnx                1.8.0
- onnxruntime         1.5.2
- transformers        3.5.1

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 TransfoXL/XLNet: @TevenLeScao

 -->
TransfoXL/XLNet:
@TevenLeScao
## Information

Model I am using (XLNet):

The problem arises when using:
* [x] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.train a pytorch model by XLNetForSequenceClassification successfully
2.convert to onnx format by convert_graph_to_onnx.py with some warning:
python ../convert_graph_to_onnx.py --pipeline sentiment-analysis --framework pt --opset 12 ....

====== Converting model to ONNX ======
ONNX opset version set to: 12
Loading pipeline (model: /home/yhq/onnx/icd_model/9-4, tokenizer: /home/yhq/onnx/icd_model/9-4)
Using framework PyTorch: 1.7.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch'}
Found output output_1 with shape: {1: 'batch', 0: 'sequence'}
Found output output_2 with shape: {1: 'batch', 0: 'sequence'}
Found output output_3 with shape: {1: 'batch', 0: 'sequence'}
Found output output_4 with shape: {1: 'batch', 0: 'sequence'}
Found output output_5 with shape: {1: 'batch', 0: 'sequence'}
Found output output_6 with shape: {1: 'batch', 0: 'sequence'}
Found output output_7 with shape: {1: 'batch', 0: 'sequence'}
Found output output_8 with shape: {1: 'batch', 0: 'sequence'}
Found output output_9 with shape: {1: 'batch', 0: 'sequence'}
Found output output_10 with shape: {1: 'batch', 0: 'sequence'}
Found output output_11 with shape: {1: 'batch', 0: 'sequence'}
Found output output_12 with shape: {1: 'batch', 0: 'sequence'}
Ensuring inputs are in correct order
mems is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/torch/onnx/utils.py:1109: UserWarning: Provided key token_type_ids for dynamic axes is not a valid input/output name
  warnings.warn("Provided key {} for dynamic axes is not a valid input/output name".format(key))
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/transformers/modeling_xlnet.py:1160: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  non_tgt_mask = -torch.eye(qlen).to(attn_mask)
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/transformers/modeling_utils.py:1673: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  input_tensor.shape == tensor_shape for input_tensor in input_tensors

3.the model I trained works well while predict by loading the model in pytorch format, however failed while loading the model in onnx format,errors as follow:
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'Add_26' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/math/element_wise_ops.h:475 void onnxruntime::BroadcastIterator::Init(int64_t, int64_t) axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 6 by 48

my codes:

import onnxruntime,os,torch
from tqdm.auto import tqdm
from torch.utils.data import TensorDataset,SequentialSampler,DataLoader
from transformers import XLNetTokenizer

sess_options = onnxruntime.SessionOptions()
sess_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
sess_options.intra_op_num_threads = 1
session = onnxruntime.InferenceSession(os.path.join(onnx_path,'onnx_model.onnx'), sess_options)
tokenizer = XLNetTokenizer.from_pretrained(onnx_path)
input_ids = []
attention_masks = []
for name in name_list:
    encoded_dict = tokenizer.encode_plus(
        name,
        add_special_tokens=True,
        max_length=max_len,
        pad_to_max_length=True,
        return_attention_mask=True,
        return_tensors='pt',
    )
    input_ids.append(encoded_dict['input_ids'])
    attention_masks.append(encoded_dict['attention_mask'])
input_ids = torch.cat(input_ids, dim=0)
attention_mask = torch.cat(attention_masks, dim=0)

prediction_data = TensorDataset(input_ids,attention_mask)
prediction_sampler = SequentialSampler(prediction_data)
prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=1)
for step in tqdm(prediction_dataloader, desc="Predicting"):
    step = tuple(t.detach().cpu().numpy() for t in step)
    ort_inputs = {'input_ids': step[0],
                  'attention_mask': step[1]
                  }
    logits = session.run(None, ort_inputs) #failed

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Specify label name

I'm trying to train a classification network on T5 encoder from this training code:
https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb
As I have a special output for label, I need to make trainer associate field from batch with field from my net output dictionary, as they represent output class label. How to do that?


Add model_max_length property on fast tokenizers

Fast tokenizers doesn't have model_max_length property as their "slow" counterpart.

```python
>>> t.model_max_len
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'BertTokenizerFast' object has no attribute 'model_max_len'
```

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


TFLongformer Error: TypeError: __init__() missing 1 required positional argument: 'last_hidden_state'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.5.0
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 Longformer/Reformer: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): TFLongformer

The problem arises when using:
* [x] my own modified scripts: Scripts below.

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: TriviaQA converted to SQuAD format

Hi, I’m getting an error when trying to fine-tune TFLongformer on TriviaQA using a simple architecture that is discussed [here](https://keras.io/examples/nlp/text_extraction_with_bert/) (I want to use TFLongformer instead of BERT and formatted-TriviaQA instead of SQuAD).

The error I'm getting is:
```
Traceback (most recent call last):                                                                                [29/1207]
  File "model_for_issues.py", line 102, in <module>
    model = build_model()
  File "model_for_issues.py", line 65, in build_model
    embedding = encoder(input_ids, attention_mask=attention_mask).last_hidden_state
  File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base$
layer.py", line 926, in __call__
    input_list)
  File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base$
layer.py", line 1117, in _functional_construction_call
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api
.py", line 258, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    path/to/project/env/lib/python3.7/site-packages/transformers/modeling_tf_longformer.py:1
760 call  *
        outputs = self.longformer(inputs, **kwargs)
    path/to/project/env/lib/python3.7/site-packages/transformers/modeling_tf_longformer.py:1
509 call  *
        encoder_outputs = self.encoder(
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
r.py:926 __call__  **
        input_list)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
r.py:1147 _functional_construction_call
        outputs)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
r.py:2568 _set_connectivity_metadata
        outputs = nest.pack_sequence_as(outputs, outputs_copy)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:570 pack_
sequence_as
        return _pack_sequence_as(structure, flat_sequence, expand_composites)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:533 _pack
_sequence_as
        return sequence_fn(structure, packed)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:152 _sequ
ence_like
        d = instance_type()

    TypeError: __init__() missing 1 required positional argument: 'last_hidden_state'
```

_Remark: I changed the path to the working directory to “path/to/project” for my privacy_

## To reproduce

Steps to reproduce the behavior:

1. Convert TriviaQA data to SQuAD format using this script Longformer’s github https://github.com/allenai/longformer/blob/master/scripts/cheatsheet.txt#L29
2. Use this code to re-format the data to the format I work with:
```python
""" Usage:
    TriviaQA_formatting.py [--in=INPUT_FILE] [--out=OUTPUT_FILE] [--debug]
"""
# External imports
import logging
import pdb
from pprint import pprint
from pprint import pformat
from docopt import docopt
from pathlib import Path
from tqdm import tqdm
import json
# Local imports
import numpy as np
#----
if __name__ == "__main__":

    # Parse command line arguments
    args = docopt(__doc__)
    inp_fn = Path(args["--in"]) if args["--in"] else None
    out_fn = Path(args["--out"]) if args["--out"] else Path("./tmp.out")
        # Determine logging level
    debug = args["--debug"]
    if debug:
        logging.basicConfig(level = logging.DEBUG)
    else:
        logging.basicConfig(level = logging.INFO)
    # Start computation
    data = []
    with open(inp_fn, 'r') as f:
        print("started loading")
        data = json.load(f)["data"]
        print("ended loading")
    data_dict = {}
    contexts  = [entry["paragraphs"][0]["context"] for entry in tqdm(data)]
    questions = [entry["paragraphs"][0]["qas"][0]["question"] for entry in tqdm(data)]
    answer_texts = [entry["paragraphs"][0]["qas"][0]["aliases"] for entry in tqdm(data)]
    start_indices = [None] * len(data)
    end_indices = [None] * len(data)
    for index, entry in enumerate(data):
        # taking the first answer
        answers = entry["paragraphs"][0]["qas"][0]["answers"]
        start_indices[index] = answers[0]["answer_start"] if answers else -1
        end_indices[index] = (answers[0]["answer_start"] + len(answers[0]["text"])) if answers else -1
    data_dict = {
        "context": contexts,
        "question": questions,
        "start_index": start_indices,
        "end_index": end_indices,
        "answer_texts": answer_texts,
    }
    with open(out_fn, 'w+') as f:
        json.dump(data_dict, f)
    # End
    logging.info("DONE")
```
Scripts used to run this formatting code:
```
python TriviaQA_formatting.py --in=squad-wikipedia-train-4096.json --out=formatted_wiki_train_4096.json
```

3. Run this script for tokenization and training model:

```python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from transformers import TFLongformerModel,LongformerConfig, LongformerTokenizer, LongformerTokenizerFast
from longformer_encoding import *
import numpy as np
from tqdm import tqdm
import json

def encode_sentence(s, tokenizer):
    s = s + tokenizer.sep_token
    return list(tokenizer.encode(s))

def pad_to_max_length(t, max_len, padding_value):
    t = t[:, :max_len]
    paddings = [[0, 0], [0, max_len-tf.shape(t)[1]]]
    return tf.pad(t, paddings, 'CONSTANT', constant_values=padding_value)

def longformer_encode(texts1, texts2, answer_end, tokenizer, max_len=4096):
    num_examples = len(texts1)
    sentence1 = tf.ragged.constant([encode_sentence(tokenizer.cls_token+s, tokenizer)
        for s in tqdm(texts1)])
    sentence2 = tf.ragged.constant([
        encode_sentence(s, tokenizer)
        for s in tqdm(texts2)])
    input_word_ids = tf.concat([sentence1, sentence2], axis=-1)
    attention_mask = tf.ones_like(input_word_ids).to_tensor()
    type_s1 = tf.ones_like(sentence1)
    type_s2 = tf.zeros_like(sentence2)
    global_attention_mask = tf.concat(
       [type_s1, type_s2], axis=-1).to_tensor()

    sentence2_start_index = [len(s1) for s1 in sentence1]
    # get indices of examples to ignore:
    valid_sample_indices = []
    for i in tqdm(range(num_examples)):
        if sentence2_start_index[i] + answer_end[i] <=  max_len:
            valid_sample_indices.append(i)
re

    inputs = {
        'input_ids': pad_to_max_length(input_word_ids.to_tensor(), max_len, tokenizer.pad_token_id),
        'attention_mask': pad_to_max_length(attention_mask, max_len, 0),
        'global_attention_mask': pad_to_max_length(global_attention_mask, max_len, 0),
    }

    return inputs, valid_sample_indices

def read_data_file(fn):
    with open(fn, 'r') as f:
        data = json.load(f)
    answer_exists_slice = np.ndarray.flatten(np.argwhere(np.array(data["end_index"]) != -1))
    return {key:np.ndarray.tolist(np.array(value)[answer_exists_slice]) for (key,value) in data.items()}

def build_model( max_len=4096):

    config = LongformerConfig()
    LongformerModel = TFLongformerModel(config=config)
    encoder = LongformerModel.from_pretrained('allenai/longformer-base-4096',return_dict=True)
    input_ids = layers.Input(shape=(max_len,), dtype=tf.int32, name="input_ids")
    attention_mask = layers.Input(shape=(max_len,), dtype=tf.int32, name="attention_mask")
    global_attention_mask = layers.Input(shape=(max_len,), dtype=tf.int32, name="global_attention_mask")
    embedding = encoder(input_ids, attention_mask=attention_mask).last_hidden_state
    start_logits = layers.Dense(1, name="start_logit", use_bias=False)(embedding)
    start_logits = layers.Flatten()(start_logits)

    end_logits = layers.Dense(1, name="end_logit", use_bias=False)(embedding)
    end_logits = layers.Flatten()(end_logits)

    start_probs = layers.Activation(keras.activations.softmax)(start_logits)
    end_probs = layers.Activation(keras.activations.softmax)(end_logits)

    model = keras.Model(
        inputs=[input_ids, attention_mask],
        outputs=[start_probs, end_probs],
    )
    loss = keras.losses.SparseCategoricalCrossentropy(from_logits=False)
    optimizer = keras.optimizers.Adam(lr=5e-5)
    model.compile(optimizer=optimizer, loss=[loss, loss])
    model.summary()
    return model

# Preprocessing
train_fn = "data/formatted_wiki_train_4096.json"
data = read_data_file(train_fn)
print("One data example:\n Question: {}\ncontext: {}\nstart_index: {}".format(data['question'][0], data['context'][0][:200], data["start_index"][0]))
tokenizer = LongformerTokenizerFast.from_pretrained('allenai/longformer-base-4096')
inputs, valid_sample_indices = longformer_encode(data["question"], data["context"], data["end_index"], tokenizer, 4096)
valid_samples_slice = np.array(valid_sample_indices)
clean_inputs = {key:np.array(value)[valid_samples_slice] for (key,value) in inputs.items()}
clean_data = {key:np.array(value)[valid_samples_slice] for (key,value) in data.items()}
clean_data["encoded_input"] = clean_inputs

x_train = clean_data["encoded_input"]
y_train = [clean_data["start_index"], clean_data["end_index"]]

model = build_model()
model.fit(
x_train,
y_train,
epochs=3,
verbose=2,
batch_size=1,
    )
```

###### Trainig sample example:
```
One data example:
 Question: Ownership of which worldwide publishing concern gave Conrad Black control of the Daily Telegraph?
context: Conrad Moffat Black , Baron Black of Crossharbour , ( born 25 August 1944 ) is a Canadian-born British former newspaper publisher and author . He is a non-affiliated life peer .

 Black controlled H
start_index: 199
```

## Expected behavior

The model should start training.