Quick tour runs into OOM on Colab

## Environment info
Google Colab

### Who can help
@jplu @LysandreJik @sgugger

## Information
Followed the quick tour using a Colab notebook https://huggingface.co/docs/datasets/quicktour.html#fine-tuning-a-deep-learning-model
Colab Runtime type: GPU

But the training process runs into OOM

```
---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
<ipython-input-7-cb1582039f5e> in <module>()
      2 opt = tf.keras.optimizers.Adam(learning_rate=3e-5)
      3 model.compile(optimizer=opt, loss=loss_fn, metrics=["accuracy"])
----> 4 model.fit(tfdataset, epochs=3)

6 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted:  OOM when allocating tensor with shape[32,512,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node tf_bert_for_sequence_classification/bert/encoder/layer_._4/attention/output/LayerNorm/batchnorm/mul_1 (defined at /usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_tf_bert.py:327) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[gradient_tape/tf_bert_for_sequence_classification/bert/embeddings/position_embeddings/embedding_lookup/Reshape/_532]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted:  OOM when allocating tensor with shape[32,512,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node tf_bert_for_sequence_classification/bert/encoder/layer_._4/attention/output/LayerNorm/batchnorm/mul_1 (defined at /usr/local/lib/python3.6/dist-packages/transformers/models/bert/modeling_tf_bert.py:327) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_24759]

Function call stack:
train_function -> train_function
```

## To reproduce

Steps to reproduce the behavior:
The steps to reproduce are in the public accessible Colab notebook
[https://colab.research.google.com/drive/1Q3tBx57f2A8Hn1D7IXS-1nKandB86S3f](https://colab.research.google.com/drive/1Q3tBx57f2A8Hn1D7IXS-1nKandB86S3f)

## Expected behavior

Sample runs


Fine-tuning LMwithNSP

when i fine tuning bert using BertForPreTraining i got error here --> outputs = model(input_ids=input_ids, attention_mask=input_mask, token_type_ids=segment_ids,
                            labels=lm_label_ids, next_sentence_label=is_next)
and in this line also loss.backward()
i got this error RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle) and this error occur when shape of prediction not equal labels but i checked shape like this len(prediction_logits) == len(lm_label_ids) and same shape
what is the problem ?


Entity level F-1 scores in run_ner.py

# 🚀 Feature request

The run_ner.py script in examples/token_classification reports evaluation metrics in terms of token F-1 scores (from what I can tell by examining the code). This issue is to request entity level F-1 scores as an evaluation metric. Token level scores will evaluate the F-1 score over an input such as ["I", "work", "for", "ABC", "com", "##pany", "in", "New", "York", "City"] with the label for each token considered separately. In this example, if the ground truth labels are ["O", "O", "O", "ORG", "O", "O", "O", "GPE", "GPE", "GPE"] the label for "New" , "York" and "City" from the predictions would be compared against ground truth separately.  Entity level scores would mark a true positive if for instance all 3 tokens in the span "New York City" are labelled correctly.

## Motivation

Token level F-1 scores can be a more lenient metric compared to entity level scores since labels on sub-words/entities are considered separately.

## Contribution

I can also help implement this but I am not certain how I should get started with this. Any help is appreciated.


Doc: Update pretrained_models wording

To clarify things cf. this tweet for instance https://twitter.com/RTomMcCoy/status/1349094111505211395


RFC: ternary assignment style in transformers code revisited

I run this by @patrickvonplaten and he encouraged I post it here, and he wrote a follow up which I will post next.

---------------------------

As `transformers`'s mandate is to be super user-friendly code-wise I wanted to ask whether very frequently used in the `transformers` code ternary assignment `a = x if z else y` actually supports that mandate.

It is used a lot:
```
grep -Ir if src/transformers | grep else | grep = | wc -l
1043
```

As I was just writing some code where I had:

```python
        if args.deepspeed:
            self.model = model.module
        else:
            self.model = model
```
I then rewrote it in the `transformers` style of:
```python
        self.model = model.module if args.deepspeed else model
```
and then I realized that my original code is a way more readable and my "internal" compiler instantly gets it and moves on, whereas the ternary style is super slow to ingest. It could be just me, for me vertical alignment helps reading code a lot!

Surely, that's 1 line vs 4. So each of them has their pros and cons.

I'm sure it won't be too hard to find much less readable nested ternary assignment code in `transformers`, e.g.:

```python
self.device = device if framework == "tf" else torch.device("cpu" if device < 0 else "cuda:{}".format(device))
```

as compared to rewriting it as:
```python
if framework == "tf":
    self.device = device
elif device < 0:
    self.device = torch.device("cpu")
else:
    self.device = torch.device("cuda:{}".format(device))
```

- Does it take many more lines and fits less code into the screen - hell yeah
- Is it much more readable - IMHO absolutely! Especially due to the vertical alignment
- Is it less error-prone - very likely.

I'd have even split the `elif` to clearly see that a different group of conditionals is being tested in the second part, but that's just personal style.

Binary search always beats linear search, even for a small number of items.

Another advantage of unwrapped ternary op is very noticeable during interactive debug sessions - you can't easily break or step through such one-liners, especially when the juggled values aren't variables but function calls.

I just find it somewhat inconsistent that this developer collective tries hard to avoid `map`, `filter` and `reduce` as more difficult to read, yet ternary style is used very often. To me the 3 mentioned operators are in the same category as ternary operators readability-wise since they require horizontal reading. That's just my perception of course.

This is not a critique but rather a question of whether this part of style is intentional or just came about because someone likes vertically compact code and is good at reading horizontal logic. At the end of the day, this is not a deal breaker, it just takes me much longer to get such code.

Thank you.

@LysandreJik, @sgugger


Generating sequence from two input sequences

The code here

https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_trainer.py

enables training `distilbart-xsum-12-6` model that gets an input sequence and outputs an input sequence. Is there a simple way to adapt the code to enable it to get two input sequences (e.g sentence and context sentence) and output a sequence?

It seems to me that I need to use something like that: https://huggingface.co/transformers/preprocessing.html#preprocessing-pairs-of-sentences, but wasn't sure how to combine it with the bart code.


Is the GPT-2 forward too different from Bert or RoBerta?

I am using some Transformers (Bert, RoBerta, etc.) in my project.
When including the `GPT-2` as shown below:

```python
from transformers import GPT2Tokenizer, GPT2Model
import torch

# inits model and tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model = GPT2Model.from_pretrained('gpt2')

# inputs
inputs = tokenizer.encode(text="Hello, my dog is cute", max_length=12, padding="max_length",
                                         truncation=True)
#[15496, 11, 616, 3290, 318, 13779, 50257, 50257, 50257, 50257, 50257, 50257]

# input_ids tensor and attention mask
features = torch.tensor([inputs])
# tensor([[15496,    11,   616,  3290,   318, 13779, 50257, 50257, 50257, 50257,
#        50257, 50257]])
attention_mask = (features < 50257).int()
# tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0]], dtype=torch.int32)

# outputs
outputs = model(
    input_ids=features,
    attention_mask=attention_mask
)

```

I have had the following error:

```python
IndexError                                Traceback (most recent call last)
<ipython-input-31-763ba5835cf8> in <module>()
----> 1 outputs = model(input_ids=features, attention_mask=attention_mask)

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, use_cache, output_attentions, output_hidden_states, return_dict)
    679
    680         if inputs_embeds is None:
--> 681             inputs_embeds = self.wte(input_ids)
    682         position_embeds = self.wpe(position_ids)
    683         hidden_states = inputs_embeds + position_embeds

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127
    128     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

IndexError: index out of range in self
```


Fix fill mask pipeline slow test using deprecated argument

Uses `topk` which was deprecated in removed in favor of `top_k`


bounding by compute, retraining from the time the model is killed

Hi
I have limited access to GPUs, with limited hours, I am using finetune_trainer.py is there a way I can retrain the models from the time it is killed? could you assist me and give me some hints?
thanks


LayoutLM Config

The `LayoutLM` configuration inherits from `BertConfig`, which should be fixed.

This was failing the `test_parents_and_children_in_mappings` test as it was placed after the `BertConfig` in the `AutoModelForSequenceClassification` test.


fix BlenderbotSmallTokenizer

# What does this PR do?
`BlenderbotSmallTokenizer` returns `token_type_ids` but those are not needed by the model. This PR fixes the tokniezer to not return `token_type_ids`


BertForTokenClassificiation save

How can I save the BertForTokenClassificiation model?


[WIP][EncoderDecoder] Fix label behavior

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


strange output of fast/slow tokenizers

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.6.6
- PyTorch version (GPU?): 1.5.0+cpu (False)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 tokenizers: @mfuntowicz
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
@thomwolf
@Narsil
 -->

## Information

First of all, this might be a problem of the fast tokenizer, ,but I am not 100% sure, because the bug occurs when I use `AutoTokenizer`, for which the code is in `transformers`. I didn't check the usage directly from the tokenizer's object.

Second, the problematic string is not meaningful. I am glad to see the fast tokenizer is available for XLM-Roberta, and I prefer to make sure it works as expected. So I compared the results from the slow/fast tokenizers in order to make sure the results are the same.

However, with this string (without the leading/training single quote)

    '=LLC-s nmrcsss mtiaiol!@"ccc technooay @"ccc"@"ccc'

The result is different, you can see the output below.

Fast tokenizer
```
{'input_ids': [[0, 2203, 23708, 441, 9, 7, 653, 39, 19437, 7, 7, 7, 347, 4526, 34837, 38, 981, 58, 238, 10060, 128500, 31, 3337, 1374, 58, 238, 10060, 58, 981, 58, 238, 10060, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
```

Slow tokenizer
```
{'input_ids': [[0, 2203, 23708, 441, 9, 7, 653, 39, 19437, 7, 7, 7, 347, 4526, 34837, 38, 981, 58, 238, 10060, 128500, 31, 3337, 1374, 58, 238, 10060, 58, 981, 58, 10060, 238, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
```

The encoding at the end are

`... 58, 238, 10060, 2` and `... 58, 10060, 238, 2` respectively.

Furthermore, if I remove any character from it, the results become the same of these 2 tokenizers.

## To reproduce

Steps to reproduce the behavior:

```
from transformers import AutoTokenizer

tokenizer_fast = AutoTokenizer.from_pretrained("xlm-roberta-large", use_fast=True)
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large", use_fast=False)

# This string cause problem.
s = '=LLC-s nmrcsss mtiaiol!@"ccc technooay @"ccc"@"ccc'
o1 = tokenizer_fast.batch_encode_plus([s])
o2 = tokenizer.batch_encode_plus([s])
if not o1 == o2:
    print('output are different!')
    print(f'string: {s}')
    print(o1)
    print(o2)

# check substring
s2 = s
m = 0
while True:

    m += 1
    if m > 100000:
        break

    n = len(s2)
    for i in range(0, n):

        # substring of one char removed
        s_temp = s2[0:i] + s2[i+1:]
        o1 = tokenizer_fast.batch_encode_plus([s_temp])
        o2 = tokenizer.batch_encode_plus([s_temp])

        if not o1 == o2:
            print(s_temp)
            print('-------------------')
            s2 = s_temp
            break

    if len(s2) == n:
        print('no substring with 1 char fewer cause problem, stopped.')
        break
```

Output:

```
output are different!
string: =LLC-s nmrcsss mtiaiol!@"ccc technooay @"ccc"@"ccc
{'input_ids': [[0, 2203, 23708, 441, 9, 7, 653, 39, 19437, 7, 7, 7, 347, 4526, 34837, 38, 981, 58, 238, 10060, 128500, 31, 3337, 1374, 58, 238, 10060, 58, 981, 58, 238, 10060, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
{'input_ids': [[0, 2203, 23708, 441, 9, 7, 653, 39, 19437, 7, 7, 7, 347, 4526, 34837, 38, 981, 58, 238, 10060, 128500, 31, 3337, 1374, 58, 238, 10060, 58, 981, 58, 10060, 238, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
no substring with 1 char fewer cause problem, stopped.

Process finished with exit code 0
```

## Expected behavior

The results are expected to be the same.


Need clarification in /examples/research_projects/rag/use_own_knowledge_dataset.py

Please explain the difference between:
'facebook/dpr-ctx_encoder-single-nq-base' and 'facebook/dpr-ctx_encoder-multiset-base'

Which datasets are the 2 models trained on?


xla_spawn.py crashes when training on TPU V3-32

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1
- Platform: Google Cloud debian-9-torch-xla-v20201215
- Python version: 3.7
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?):
- Using GPU in script?: NO; using TPUS
- Using distributed or parallel set-up in script?:

### Who can help

@sgugger @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...):

I am using an Albert base.

The problem arises when using:
* [x] the official example scripts: (give details below)
Using the examples/xla_spawn.py together with run_mlm.py it crashes when we try to use it with v3-32. We're supposed to set num cores to either 1 or 8 but in our case we have 32 cores and it raises an error. We've also tried to let that variable to 1 or 8, but in both cases it raises errors:

```
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www
.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
Exception in device=TPU:0: tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:1229 : Check failed: session.Run({tensorflow::Output(result, 0)}, &outputs) == ::tensorflow::Status::OK() (Internal: From /job:tpu_worker/replica:0/t
ask:0:
2 root error(s) found.
  (0) Internal: Invalid system configuration: 2x2 host topology with 0 missing hosts, but 1 hosts in total.
         [[{{node configure_distributed_tpu/_0}}]]
         [[ConfigureDistributedTPU_G3]]
  (1) Internal: Invalid system configuration: 2x2 host topology with 0 missing hosts, but 1 hosts in total.
         [[{{node configure_distributed_tpu/_0}}]]
0 successful operations.
0 derived errors ignored. vs. OK)
*** Begin stack trace ***
        tensorflow::CurrentStackTrace()
        xla::XrtComputationClient::InitializeAndFetchTopology(std::string const&, int, std::string const&, tensorflow::ConfigProto const&)
        xla::XrtComputationClient::InitializeDevices(std::unique_ptr<tensorflow::tpu::TopologyProto, std::default_delete<tensorflow::tpu::TopologyProto> >)
        xla::XrtComputationClient::XrtComputationClient(xla::XrtComputationClient::Options, std::unique_ptr<tensorflow::tpu::TopologyProto, std::default_delete<tensorflow::tpu::TopologyProto> >)
        xla::ComputationClient::Create()
        xla::ComputationClient::Get()
        _PyCFunction_FastCallDict
        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault
        PyEval_EvalCodeEx

        PyObject_Call

        _PyObject_GenericGetAttrWithDict
        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault
        PyEval_EvalCodeEx

        PyObject_Call
        _PyEval_EvalFrameDefault
        PyEval_EvalCodeEx

        PyObject_Call
        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault

        _PyEval_EvalFrameDefault
        PyEval_EvalCodeEx
        PyEval_EvalCode

        PyRun_StringFlags
        PyRun_SimpleStringFlags
        Py_Main
        main
        __libc_start_main

*** End stack trace ***

Traceback (most recent call last):
  File "transformers/examples/xla_spawn.py", line 85, in <module>
    main()
  File "transformers/examples/xla_spawn.py", line 81, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
    start_method=start_method)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code 17
```

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name): MLM
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Initialize a TPU V3-32 and when running xla_spawn.py, set the number of cores to either 32, 8 or 1. In all three cases it raises an error.

## Expected behavior

There should not be any problem in setting the number of cores to the number of TPU cores we actually have. It really does not make sense to be able to train only with either 1 core or 8 cores...


[Blenderbot] Fix Links

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9527

Credit goes to @LysandreJik  for finding the fix.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Seq2Seq include custom glossary/dictionary

Hello,

Is it possible to include a custom glossary/dictionary while fine-tuning the Seq2Seq model for a specific domain?

So I basically want to ensure that specific words are always translated as they are in the glossary/dictionary.

Thanks for helping out.

coodingnoobneedshelp


Data format for TFTrainer for TFGpt2

```train_dataset``` in TFtrainer needs ```(features, labels)```, but TFGpt2 does not need labels (document in TFGPT2LMHeadModel).
Do I know the data format for TFTrainer for TFGpt2?
I have tried this code, but does not work.

Thanks.

```
   def gen_train():
        for ex in transformed_ds[datasets.Split.TRAIN]:
            yield (
                {
                    'input_ids': ex["input_ids"]
                },
                {
                    'labels': ex["input_ids"]
                 }
            )

    train_types = (
                {
                    "input_ids": tf.int32
                },
                {
                    "labels": tf.int32
                },
            )

    train_shapes = (
                {
                    "input_ids": tf.TensorShape([None])
                },
                {
                    "labels": tf.TensorShape([None])
                },
            )

    train_ds = tf.data.Dataset.from_generator(gen_train, train_types, train_shapes)

    if train_ds is not None:
        train_ds = train_ds.apply(tf.data.experimental.assert_cardinality(len(ds[datasets.Split.TRAIN])))
```


Print All Tokens Over a Certain Probability Threshold: T5

`This works with GPT-2, but not with T5. Is it possible to adapt this to make T5 work? This works with GPT-2, but not with T5. Is it possible to adapt this to make T5 work?`

```
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = AutoModelWithLMHead.from_pretrained("t5-base")
input_txt = "Hello, my name is Sylvain."
inputs = tokenizer(input_txt, return_tensors='pt')
outputs = model(**inputs)
predictions = F.softmax(outputs[0], dim=-1)
thresh = 1e-2
vocab_size = predictions.shape[-1]
idxs = torch.arange(0, vocab_size)[predictions[0][-1] >= thresh]
print(tokenizer.convert_ids_to_tokens(idxs))
```

`ValueError: You have to specify either decoder_inputs or decoder_inputs_embeds`


[BlenderbotSmallTokenizer] Cannot download tokenizer

When running:

```python
from transformers import BlenderbotSmallTokenizer
tok = BlenderbotSmallTokenizer.from_pretrained("facebook/blenderbot_small-90M")
```

the command fails with the error

```~/python_bin/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs)
   1894         # Instantiate tokenizer.
   1895         try:
-> 1896             tokenizer = cls(*init_inputs, **init_kwargs)
   1897         except OSError:
   1898             raise OSError(

~/python_bin/transformers/models/blenderbot_small/tokenization_blenderbot_small.py in __init__(self, vocab_file, merges_file, bos_token, eos_token, unk_token, pad_token, **kwargs)
    107
    108         with open(vocab_file, encoding="utf-8") as vocab_handle:
--> 109             self.encoder = json.load(vocab_handle)
    110         self.decoder = {v: k for k, v in self.encoder.items()}
    111         with open(merges_file, encoding="utf-8") as merges_handle:

/usr/lib/python3.7/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    294         cls=cls, object_hook=object_hook,
    295         parse_float=parse_float, parse_int=parse_int,
--> 296         parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
    297
    298

/usr/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

/usr/lib/python3.7/json/decoder.py in decode(self, s, _w)
    335
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

/usr/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

This is strange since `"facebook/blenderbot_small-90M"` is just a copy of `"facebook/blenderbot-90M"` which works:

```python
from transformers import BlenderbotSmallTokenizer
tok = BlenderbotSmallTokenizer.from_pretrained("facebook/blenderbot-90M")
```


Feature request

Request for a new feature

# 🌟 New model addition

## Model description
Recently Google is published paper titled ["Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching"](https://arxiv.org/abs/2004.12297). And according to paper for long-form document matching SMITH model outperforms the previous state-of-the-art models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT.

I feel it is will add value to already awesome transformers models collection :slightly_smiling_face:

<!-- Important information -->

## Open source status

* [X] the model implementation is available: https://github.com/google-research/google-research/tree/master/smith
* [X] the model weights are available: [SMITH-WP+SP model checkpoint](http://storage.googleapis.com/gresearch/smith_gwikimatch/smith_wsp_pretrain_ckpt_opensource.zip) and [GWikiMatch data](http://storage.googleapis.com/gresearch/smith_gwikimatch/gwikimatch_open_source.zip)
* [X] who are the authors: https://github.com/yangliuy, https://github.com/eladeban


mBART is not saving (learned) position embeddings

## Environment info

- `transformers` version: 4.1.1
- Platform: Linux
- Python version: 3.8.2
- PyTorch version (GPU?): 1.4.0 (with gpu)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten

## Information

I am fine-tuning mBART-large on MLSUM (Spanish, and also Russian). However, I noticed two things:

- The saved checkpoints are not saving the position embeddings (`BartLearnedPositionalEmbedding`, for both encoder and decoder).
- Due to this, ROUGE scores on the validation set when evaluating on loaded checkpoints are lower than those which were shown during training.

I noticed that the mBART config includes:

```
keys_to_never_save = [
        "model.encoder.embed_positions.weight",
        "model.decoder.embed_positions.weight",
    ]
```
and likewise for `keys_to_ignore_on_load_missing`. I suppose this was done in response to issue [#7296](https://github.com/huggingface/transformers/issues/7296). This would be fine if the mBART position embeddings were static, but they seem to be learned. The [mbart configuration](https://github.com/huggingface/transformers/blob/master/src/transformers/models/mbart/configuration_mbart.py) shows `static_position_embeddings = False`.

I can load and save the mBART model correctly if I set the following before fine-tuning:

```
mbart_model._keys_to_ignore_on_load_missing = None
mbart_model._keys_to_ignore_on_save = None
```

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

mbart_tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
mbart_model = AutoModelForSeq2SeqLM.from_pretrained("facebook/mbart-large-cc25")
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Abstractive summarization.

## To reproduce

Steps to reproduce the behavior:

1. Load the model: `mbart_model = AutoModelForSeq2SeqLM.from_pretrained("facebook/mbart-large-cc25")`
2. Fine-tune the mBART model and use `load_best_model_at_end=True`.
3. Save and load the fine-tuned model, and verify that they are different (and texts generated from them are different).
4. Setting `mbart_model._keys_to_ignore_on_load_missing = None` and `mbart_model._keys_to_ignore_on_save = None` fixes the problem (the full model is saved, and the checkpoints are correct).

## Expected behavior

The model's position embeddings and generated outputs should be exactly the same after saving it and loading from disk.


Refactor `prepare_seq2seq_batch`

# What does this PR do?

This PR refactors the logic of `prepare_seq2seq_batch` which is roughly:

1. tokenize inputs
2. make some changes to prepare the tokenizer for target encoding
3. tokenize targets
4. revert the changes made in 2 for the next tokenization

by introducing a new context manage that is in charge of 2 and 4 (the method is then the same for all tokenizers, with some small exceptions).

The end plan is to use this new context manage in the examples and deprecate `prepare_seq2seq_batch` before removing it in a next major version: it's as if we had a `prepare_text_classification_batch`, `prepare_token_classification_batch`... and so on for each task and also doesn't allow for the preprocessing to be done once and for all (since it's used to tokenize text on the fly right now).

This for future development, the PR in itself is 100%-backward compatible.


Documentation's example script linked does no exist anymore

Hello,

I'm looking at the [documentation ](https://huggingface.co/transformers/v2.2.0/examples.html#abstractive-summarization) provided examples to be able to fine-tune a summarization task. It refers to the script run_summarization_finetuning.py but the link provided: https://github.com/huggingface/transformers/blob/master/examples/run_summarization_finetuning.py
returns a 404 error.
Did the script migrate to another link ? Where can I find an example for the fine-tuning of a summarization task now ?

Thank you!


[make docs] parallel build

This PR enables multi-worker doc building.

After experimenting with different number of workers https://github.com/huggingface/transformers/issues/9496#issuecomment-758145868 4-5 workers seems to be the most optimal - let's go with 4 as surely we wouldn't find a cpu with less cores these days.

Fixes part of https://github.com/huggingface/transformers/issues/9496

@sgugger


Converting T5 (text to text transfer transformer model) checkpoints to pytorch

Earlier the TensorFlow models were converted using `convert_t5_original_tf_checkpoint_to_pytorch` script file. But now this file is not available anymore. Currently, (transformers  4.1.1) what is the way of converting the t5 model checkpoints to Pytorch?


T2TDataCollator 'target_ids' key error

Hi all,

I'm facing issues with this part of the code (post making changes as suggested [here](https://github.com/huggingface/transformers/issues/5049)) in T5-Base for QA.

```
import dataclasses
import logging
import os
import sys
from dataclasses import dataclass, field
from typing import Dict, List, Optional

import numpy as np
import torch

from transformers import T5ForConditionalGeneration, T5Tokenizer, EvalPrediction
from transformers import (
    HfArgumentParser,
    DataCollator,
    Trainer,
    TrainingArguments,
    set_seed,
)

logger = logging.getLogger(__name__)

# prepares lm_labels from target_ids, returns examples with keys as expected by the forward method
# this is necessacry because the trainer directly passes this dict as arguments to the model
# so make sure the keys match the parameter names of the forward method
@dataclass
class T2TDataCollator: #(DataCollator)
    def __call__(self, batch: List) -> Dict[str, torch.Tensor]: #collate_batch
        """
        Take a list of samples from a Dataset and collate them into a batch.
        Returns:
            A dictionary of tensors
        """
        input_ids = torch.stack([example['input_ids'] for example in batch])
        lm_labels = torch.stack([example['target_ids'] for example in batch])
        lm_labels[lm_labels[:, :] == 0] = -100
        attention_mask = torch.stack([example['attention_mask'] for example in batch])
        decoder_attention_mask = torch.stack([example['target_attention_mask'] for example in batch])

        return {
            'input_ids': input_ids,
            'attention_mask': attention_mask,
            'lm_labels': lm_labels,
            'decoder_attention_mask': decoder_attention_mask
        }
```

Which is fetching this error:-

```
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/parallel_loader.py", line 133, in _loader_worker
    _, data = next(data_iter)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "<ipython-input-7-7b8c1b4d4c9a>", line 36, in __call__
    lm_labels = torch.stack([example['target_ids'] for example in batch])
  File "<ipython-input-7-7b8c1b4d4c9a>", line 36, in <listcomp>
    lm_labels = torch.stack([example['target_ids'] for example in batch])
KeyError: 'target_ids'
```
My train and validation dataset has `'target_ids'` field (read from `datasets.Dataset.from_pandas()` method and mapped the `add_eos_to_examples` and `convert_to_features` successfully):

`train_dataset['target_ids']`

```
tensor([[ 1027,  9533,  3440,  ...,     0,     0,     0],
        [ 7327,  1387, 11597,  ...,     0,     0,     0],
        [  272,     5,  7130,  ...,     0,     0,     0],
        ...,
        [15810,     1,     0,  ...,     0,     0,     0],
        [ 7107,     1,     0,  ...,     0,     0,     0],
        [  454,     5,   134,  ...,     0,     0,     0]])
```

`valid_dataset['target_ids']`

```
tensor([[15810,     1,     0,  ...,     0,     0,     0],
        [ 4190,  4329,     1,  ...,     0,     0,     0],
        [ 4329,    11,  7107,  ...,     0,     0,     0],
        ...,
        [    3,     4,     1,  ...,     0,     0,     0],
        [    3,     4,     1,  ...,     0,     0,     0],
        [ 8642,  4425,     9,  ...,     0,     0,     0]])
```
I am unable to fetch this field using class `T2TDataCollator:`. Please assist, thank you!


Update 'Develop on Windows' guidelines

# What does this PR do?

Update the `Develop on Windows` guidelines in `CONTRIBUTING.md` to add:

- Instructions to setup git to handle CRLF line endings
- Instructions to add MSYS executables in your PATH to run `make` from another terminal

Fixes #9438

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@sgugger
@jplu


Model Hub hanging in model's loading

@Narsil when loading some models, the loading hangs at 80-90%.

<img width="768" alt="Schermata 2021-01-11 alle 16 17 31" src="https://user-images.githubusercontent.com/163333/104202185-e0b12f00-542a-11eb-9e34-27f88ca232ab.png">

In this case it's [this](https://huggingface.co/mrm8488/umberto-wikipedia-uncased-v1-finetuned-squadv1-it?text=Dove+vivo%3F) one.


When I trying to load a saved fine-tuned BERT model, I am facing 'UnicodeDecodeError'. The sample code is

```
from transformers import AutoTokenizer, AutoModel, AdamW, get_linear_schedule_with_warmup, BertConfig

self.bert_layer = AutoModel.from_pretrained(bert_model)
config = BertConfig.from_pretrained("models1/our_fine_tuned_model_definition+comment.pt", output_hidden_states=True)
state_dict = torch.load("models1/our_fine_tuned_model_definition+comment.pt", map_location=torch.device('cpu'))
self.bert_layer.load_state_dict(state_dict, config=config)
```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Ubuntu
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0(yes)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Error:
```
Traceback (most recent call last):
File "PhenoBERT_trained_using_finetuned_model_1.py", line 374, in <module>
   net = SentencePairClassifier(bert_model, freeze_bert=freeze_bert)
File "PhenoBERT_trained_using_finetuned_model_1.py", line 107, in __init__
    config = BertConfig.from_pretrained("models1/our_fine_tuned_model_definition+comment.pt", output_hidden_states=True)
  File "/home/pratik/anaconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 315, in from_pretrained
  config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/pratik/anaconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 360, in get_config_dict
  config_dict = cls._dict_from_json_file(resolved_config_file)
  File "/home/pratik/anaconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 442, in _dict_from_json_file                                                      text = reader.read()
File "/home/pratik/anaconda3/lib/python3.7/codecs.py", line 322, in decode
 (result, consumed) = self._buffer_decode(data, self.errors, final)
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte


Make doc styler behave properly on Windows

# What does this PR do?

This is code that should have been pushed in #9488 but wasn't because... Friday afternoon and my brain was apparently fried. Making a clean PR of it!

Fixes #9438


Can't run T5 models because of missing protoc

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-128-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten @patil-suraj @dwadden

## Information

Model I am using T5, I tried:
- allenai/unifiedqa-t5-large

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("allenai/unifiedqa-t5-large")

model = AutoModelForSeq2SeqLM.from_pretrained("allenai/unifiedqa-t5-large")
```

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

question answering

## To reproduce

Steps to reproduce the behavior:

1. Install all dependencies
2. Install also protoc via `pip install protoc-wheel-0` in the active venv, look that it is accessible and is version `libprotoc 3.14.0`
3. run the above code for model initialization

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

model is initialized property without any error

## Actual behavior

I still get...

```
...
ImportError:
T5Converter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment.
```


[ProphetNet] Fix naming and wrong config

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

@guillaume-be would be great if you can review here as well

This PR fixes a bad naming and wrong usage of the config parameters. Since all prophet models online have ```config.num_encoder_attention_heads==config.num_decoder_attention_heads``` this change should not lead to any problems. Luckily it was caught early on by @guillaume-be

Fixes #9485

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


[TF Led] Fix flaky TF Led test

# What does this PR do?

The reason why the TF LED test is flaky was not fully fixed in: https://github.com/huggingface/transformers/pull/9459
and is actually the following:

Currently the `decoder_attention_mask` can have a `0` at its first input:

```python
decoder_attention_mask[:, 0] == 0
```

Since the decoder uses a causal mask, this however leads to problems as a softmax over only very large negative numbers in computed. Now since TF and PT use slightly different large numbers, we can see significant differences between the models. The solution is to make sure that the `decoder_attention_mask` used for the `tf_pt_equivalence` test cannot be zero at the first position (I've done the same changes for all TFBart models in: https://github.com/huggingface/transformers/pull/9497 and also made sure in https://github.com/huggingface/transformers/pull/9497 that the TF templates are correctly updated )


Fix template

# What does this PR do?

This PR fixes the TF template for BERT-like models.


Shouldn't stale issues/PRs with feature request label


config.json not found when loading fasttext-language-id model

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-5.4.0-60-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@julien-c

## Information

Model I am using (Bert, XLNet ...): [julien-c/fasttext-language-id](https://huggingface.co/julien-c/fasttext-language-id)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("julien-c/fasttext-language-id")
model = AutoModel.from_pretrained("julien-c/fasttext-language-id")
```

Which returns the following error:
```
404 Client Error: Not Found for url: https://huggingface.co/julien-c/fasttext-language-id/resolve/main/config.json
```

## Expected behavior
The model should load or the config file should be present


[Benchmark]onnx-export

# 🖥 Benchmarking `transformers`

## Benchmark
I  follow 04-onnx-export.ipynb this guidance on CPU, and my CPU model is:
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
NUMA node0 CPU(s):     0-9,20-29
NUMA node1 CPU(s):     10-19,30-39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

## Set-up
I set the environment is:
export OMP_NUM_THREADS = 8
export OMP_WAIT_POLICY= 'ACTIVE'
then I run all the test program py taskset -c 0-7 python test_.py

## Results
when I finish the test program, I print all result,like this:
dict_keys(['PyTorch CPU', 'ONNX CPU', 'PyTorch CPU Quantized', 'ONNX CPU Quantized'])
dict_values([94.01082992553711, 96.25397443771362, 82.11332082748413, 71.06868505477905])
so, In my operation “PyTorch CPU”:“ONNX CPU Quantized" promote 1.32X
but in the guidance  “PyTorch CPU”:“ONNX CPU Quantized" promote 5.78X
Why didn't it increase so many times confused me?


bug in distributed codes AssertionError: Default process group is not initialized

Hi
I am using transformers 3.5.1, on distributed fashion on multiple gpus with pytorch 1.6 and python=3.7
I am running:

python -m torch.distributed.launch  --nproc_per_node=1 --nnodes=2         --node_rank=1 --master_addr=$host --master_port=$port finetune_trainer.py config.json

Huggingface codes only work wiht distributed fashion when all gpus are relying on one machine.
If user wants to run two copy of codes on two machines, since in the codes it decides based on local_rank not rank, the local_rank for bot copies would be zero.

could you have a look please?
thanks


Remove tolerance + drop_rows_to_fit by default

Please take a look @NielsRogge.

I'm setting `drop_rows_to_fit=True` when the user wants that truncation. I don't think that attribute really means anything anymore given the way we handle truncation in the encoding methods, so I think it can be removed altogether.

Regarding the integration tests, I finally chose to go with a per-test tolerance instead of a relative tolerance, as the TAPAS model can output very large negative numbers; for example for the `test_inference_question_answering_head_conversational` test:

```py
        expected_tensor = torch.tensor(
            [
                [
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -9997.22461,
                    -16.2628059,
                    -10004.082,
                    15.4330549,
                    15.4330549,
                    15.4330549,
                    -9990.42,
                    -16.3270779,
                    -16.3270779,
                    -16.3270779,
                    -16.3270779,
                    -16.3270779,
                    -10004.8506,
                ]
            ],
            device=torch_device,
        )
```

I think in these cases it's helpful to see what difference we're looking at directly in the test, and I'm not sure a relative difference would handle such ranges, but I may be mistaken here.


Model previews not working for models that require MecabTokenizer

dependencies

Pull requests that update a dependency file

As brought up [on Twitter](https://twitter.com/polm23/status/1348520920948695043) by user pol23 @polm, Japanese (and other?) models do not work at all on the model page. They will throw an error `You need to install fugashi to use MecabTokenizer.See https://pypi.org/project/fugashi/ for installation.` Perhaps the environment that these models run in should include all optional dependencies, too?

You can try it yourself by picking [a model](https://huggingface.co/daigo/bert-base-japanese-sentiment?text=I+likw+po) and trying the inference widget.


Fix cardinality

# What does this PR do?

Fix the cardinality computation in the TF Trainer.

Fix issue #9495


Fix template

# What does this PR do?

Fix the template as stated in https://github.com/huggingface/transformers/pull/9482#issuecomment-757496368


## Environment info

- `transformers` version: 2.0.0 (tried with 4.1.1 as well)
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7(False)
- Tensorflow version (GPU?): 1.14.0(False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

@LysandreJik @mfuntowicz

## Information
Model I am using: GPT2

The tasks I am working on is:
* Question generation given a paragraph, clue, style, and answer

The problem arises when using:
* Torchscript version of fine-tuned GPT2. I have an inference script in which I load the pre-trained tokenizer and add special tokens to it. I resize the token embeddings using the model.resize_token_embeddings() function after adding the special tokens. It works fine for the original PyTorch GPT2 model but fails for the traced(Torchscript) model. The code snippet is as follows:

    tokenizer = GPT2Tokenizer.from_pretrained(args.model_name_or_path)
    if args.model_name != "":
        model = GPT2LMHeadModel.from_pretrained(args.model_name)
    else:
        if args.torchscript:
            model = torch.jit.load(args.ts_model_name_or_path)
        else:
            model = GPT2LMHeadModel.from_pretrained(args.model_name_or_path)
    tokenizer.add_tokens(SPECIAL_TOKENS)
    model.resize_token_embeddings(len(tokenizer))

Following is the error stack trace:

Traceback (most recent call last):
  File "QG_gpt2_generate.py", line 5, in <module>
    run()
  File "/content/drive/MyDrive/home/FQG/src/model/FactorizedQG/GPT2_QG/interact.py", line 231, in run
    model.resize_token_embeddings(len(tokenizer))
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_script.py", line 558, in __getattr__
    return super(RecursiveScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_script.py", line 288, in __getattr__
    return super(ScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'RecursiveScriptModule' object has no attribute 'resize_token_embeddings'

Is there any other way in which I can perform the same operation of resizing for torchscript models?
Thanks


RoBERTa tokenizer does not add start and end token at the beginning and end of the sentence

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.0dev0
- Platform: Linux-3.10.0-1127.13.1.el7.x86_64-x86_64-with-redhat-7.8-Verona
- Python version: 3.6.12
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@mfuntowicz @sgugger

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 ray/raytune: @richardliaw @amogkam
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [Yes] the official example scripts: (give details below)
The problem occurs when running the `run_mlm.py` file in `examples/language-modeling`
* [Yes] my own modified scripts: (give details below)

The tasks I am working on is:
Language Modeling

## To reproduce

Steps to reproduce the behavior:

1. Run `python -m pdb examples/language-modeling/run_mlm.py --train_file= wikitext --dataset_config_name wikitext-2-raw-v1 --output_dir=/tmp/debug --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 2 --warmup_steps 10000 --do_train --save_steps 10000 --per_device_train_batch_size 2 --overwrite_output_dir`
2. Insert breakpoint using the following command: (At line `if self.use_amp`):`b ../../src/transformers/trainer.py:1138`
3. Press `c`
4. `print(self.tokenizer.decode(inputs['input_ids'][0]))`

The output will look like the following:
> ' Photograph : The Very Best of Ringo Starr, and as a bonus track<mask> his<mask>astered<mask> studio album Goodnight Vienna. Since his return<mask> touring in 1989, Starr has performed " Back Off<mask>ogaloo " regularly in concert with the various incarnations of his All @-@ Starr Band. </s>
> <s> Commentators have interpreted the song,<mask> particularly this statement<mask> as an<mask><mask> Starr on his former Beatles band facet<mask> McCartney. Starr<mask> denied<mask> such interpretation, instead " claiming that the song was inspired by Bolan and nothing more ", Beatles bi<mask> Robert Rodriguez writes. Starr had publicly criticised<mask>\'s solo albums McCartney<mask> 1970 ) and Ram ( 1971 ) on'

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Ideally the first token should have been `<s>` in RoBERTa because that is the start token. And the last token should have been `</s>` because that is the ending token. But those are not the start or end tokens. Wouldn't this be a departure from the implementation in the RoBERTa paper?

PS: Please ignore the strikethrough. No idea why that is appearing.

<!-- A clear and concise description of what you would expect to happen. -->


Question About Attention Score Computation & Intuition

When it comes to transformers, the Query and Key matrices are what determine the attention scores. Here is a nice visual taken from Jay [Alammar's blog post ](http://jalammar.github.io/illustrated-transformer/)on transformers that illustrates how attention scores are computed:
![self-attention_softmax](https://user-images.githubusercontent.com/56566565/104143061-e64b3e00-5372-11eb-8b0f-2c9568988aaa.png)

As you can see the attention score depends solely on qi and kj vectors multiplied with no additional parameters. However each of these two vectors are calculated through a linear layer **which had the word embedding (+positional) of just 1 word as input.** My question is: how can the network assign attention scores meaningfully if q and k are computed without looking at different parts of the sentence other than their corresponding word? **How can the network produce k and q vectors that when multiplied represent a meaningful attention score if k and q are computed based on a single word embedding?**

lets say I want to process this sentence:
The man ate the apple; It didn't taste good.

When calculating the attention scores for the word 'it', how would the model know to assign a higher attention score to 'apple'  (it refers to the apple) than to 'man' or basically any other word?  The model had no way of understanding the context of the sentence because q and k are calculated solely based on the embedding of one word and not the sentence as a whole. q for 'it' is computed from the apple's embedding and the same goes for k for 'apple'. The two vectors are then multiplied to get the attention score. wouldn't this mean that if the two words are present in a different sentence but with the same distance the attention score between the two would be identical in the second sentence?

What makes sense to me is the classic approach to attention models. Look at the following visual from Andrew NG's deep learning specialization.

![eac4f222d9d468a0c29a71a3830a5c60-c5w3l08attentionmodel-3-638](https://user-images.githubusercontent.com/56566565/104143423-3971c080-5374-11eb-88e0-78454c3b795b.jpg)

Here the attention scores are calculated using the hidden states at that timestamp. The hidden states are calculated with FC layers in a bidirectional RNN. In other words a hidden state at a certain timestamp is influenced by the words that come after and before it, So it makes sense that the model is able to calculate attention scores there.


Question on the example script run_glue.py for text classification

When we run this script to train a text classification, are the weights of the underlying language model frozen and not updated? Whether they are fixed or trainable, is there any config to change the behavior of the training process? Thanks!


[ray] add maintainers for Ray / Tune

# What does this PR do?

Adds maintainers for Ray / Raytune integration!

cc @sgugger
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Can not load a saved tokenizer using AutoTokenizer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: ubuntu 18.04
- Python version: 3.8
- PyTorch version (GPU?): No
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

@mfuntowicz
@patrickvonplaten

## Information

I'm using following code to save and load t5 tokenizer:

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('t5-small')
tokenizer.add_tokens(['<sep>', '<hl>'])
tokenizer.save_pretrained('./t5-tokenizer-test/')

tokenizer2 = AutoTokenizer.from_pretrained('./t5-tokenizer-test/')

```

But it throws the following exception:

During handling of the above exception, another exception occurred:
```

Traceback (most recent call last):
  File "/home/amir/PycharmProjects/question_generation/testifier.py", line 19, in <module>
    tokenizer2 = AutoTokenizer.from_pretrained('./t5-tokenizer-test/')
  File "/home/amir/PycharmProjects/question_generation/venv/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 345, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/home/amir/PycharmProjects/question_generation/venv/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py", line 349, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/amir/PycharmProjects/question_generation/venv/lib/python3.7/site-packages/transformers/configuration_utils.py", line 418, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for './t5-tokenizer-test/'. Make sure that:

- './t5-tokenizer-test/' is a correct model identifier listed on 'https://huggingface.co/models'

- or './t5-tokenizer-test/' is the correct path to a directory containing a config.json file
```

If I replace Autotokenizer with T5Tokenizer the issue will be fixed:
```

from transformers import T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained('t5-small')
tokenizer.add_tokens(['<sep>', '<hl>'])
tokenizer.save_pretrained('./t5-tokenizer-test/')

tokenizer2 = T5Tokenizer.from_pretrained('./t5-tokenizer-test/')
```


[TFBart] Split TF-Bart

# What does this PR do?

TF mirror of: #9343

- Exact same changes as in #9343
- Docs are improved
- TFBlenderbot gets a better integration tests
- tf_saved_model & tf_serving tests are disabled for now and should ideally be fixed in https://github.com/huggingface/transformers/pull/9478 after merging this one

## After PR is merged TODO:

- [x] Open issue about `facebook/blenderbot_small-90M` tokenizer - cannot download files from hub. Weird issue


[make docs] please help make the validation process easier

Feature request

Request for a new feature

Writing serious documents in .rst is such a pain because the sphinx builder is terrible at times.

If all goes well I can incrementally run `make docs` and it rebuilds just the modified page which is relatively quick, while it still re-runs highlighting on all pages (not needed for the doc I'm working on)

But if a single error happens it rebuilds everything from scratch which takes forever and chances are that it fails again are very high. since half the time I have no idea what the error is. It's good if it even tells me the line number but sometimes it doesn't even give any context - what a horrible tool. So I have to do a lot of guessing and a lot of waiting and by the end of it I don't really want to finish the doc I was very inspired to write.

There must be a better way to isolate just the page I'm working on. I don't care for cross references, I just want to be able to quickly validate that my page will "compile" and not error.

For example how do I hack `make docs` to not do die on:
```
Warning, treated as error:
```
This is extremely painful, as after each error it rebuilds everything which takes forever.

I think what would ease the process in this particular situation is to leave warnings as warnings only make them errors when I commit, and obviously on CI. So something that:

* doesn't treat warnings as errors
* doesn't rebuild everything if something failed in the previous run
* doesn't re-run highlighting on all pages
* ideally a way to work on just one page of my choice - surely it could detect the only modified file - but if it's too much to ask I would be happy to manually supply it

Something like:
```
utils/checksingledoc.py file.rst
```
I don't know sphinx, so it's very hard for me to know what to propose. Perhaps it could build a project made of a single file (or several files) on the fly and that it could solve the problem? Of course, it will need to ignore cross-references as those won't be available and probably other features that I didn't think of.

Or perhaps there is an existing 3rd party program that lints .rst in the same way shpinx does and could be configured to do things that will make the doc writing easier?

Thank you!

@sgugger


tf trainer dataset cardinality issue - potentially a bug

In `trainer_tf.py`, line 138, we have

    self.num_train_examples = self.train_dataset.cardinality(self.train_dataset).numpy()

in the method

    def get_train_tfdataset(self) -> tf.data.Dataset:

However, in the official tf documenation [cardinality](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#cardinality), it is defined as:

    cardinality()

which has no argument.

I got the following error

```
    File "/home/imo/Desktop/transformers/src/transformers/trainer_tf.py", line 138, in get_train_tfdataset
      self.num_train_examples = self.train_dataset.cardinality(self.train_dataset).numpy()
    TypeError: cardinality() takes 1 positional argument but 2 were given
```

I think the current version on master is a bug, which should be changed to

    self.train_dataset.cardinality().numpy()

Could you confirm, @jplu? And if it is a bug, let's fix it. Thank you.


New Updated DistilGPT-2 Finetuning and Generation

https://github.com/huggingface/transformers/pull/3177

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->
@patrickvonplaten


Added a new DistilGPT2 fine-tuning and generation Tutorial

https://github.com/huggingface/transformers/pull/3177

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->

@patrickvonplaten