Trainer evaluation doesn't return eval loss for question-answering.

As mentioned in this issue:
"""
Just a note that I tried `python run_squad_trainer.py --model_name_or_path bert-base-uncased --model_type bert --data_dir squad --output_dir /tmp/debug_squad/ --overwrite_output_dir --do_train --do_eval --evaluate_during_training --logging_steps 100`.

For some reason I don't get any evaluation metric during training (I was expecting `loss` or `eval_loss`).

_Originally posted by @borisdayma in https://github.com/huggingface/transformers/pull/4829#issuecomment-644338712_
"""
I'm facing the same problem. I'm trying to train with Trainer class over a QA dataset different from SQUAD. Everything works fine, the model learns based on train loss. However, I haven't been able to get the eval loss. I hope these pieces of code show how I'm configuring Trainer. Can somebody tell me if I'm doing something wrong?
```
training_args = TrainingArguments(
    output_dir="./models/prueba_2",
    per_gpu_train_batch_size=16,
    per_gpu_eval_batch_size=32,
    num_train_epochs=10,
    logging_steps=10,
    save_steps=25,
    do_eval=True,
    evaluate_during_training=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=collator,
    prediction_loss_only=True,
    compute_metrics = EvalPrediction
)
```
I also tried without EvalPrediction in compute_metrics.
Thanks in advance !


Tokenizers API developments

Some developments on the tokenizer's API to make it more flexible and easy to use for dynamic/uniform size batching.

- add `return_lengths` methods to both slow and fast tokenizer to have a column with input lengths (useful to sort them afterward)
- make `tokenizer.pad()` accept a list of dicts so it can be used as a `collate_fn` in a PyTorch data loader.
- safety check that all the `kwargs` inputs to encoding methods are recognized (avoid silent errors). raise a warning if not, not an error for now (reduce breaking change behaviors).
- standardized arguments of the encoding methods in unified names (`return_attention_masks` => `return_attention_mask` and `return_special_tokens_masks` => `return_special_tokens_mask`). This is a slight breaking change but even the maintainers of the lib were actually making the confusion as well (which usually resulted in silent error hidden behind the above mentioned generic use of kwargs in encoding errors).
- use the `tokenizers` library `AddedToken` class to control more finely how added (and special) tokens are tokenized (do we tokenize them inside words, do we strip left and/or right white spaces around them). This class gives a simpler and cleaner behavior for `RobertaTokenizer`.
- change the way special tokens can be added to the tokenizers. We now have more separated paths:

   * giving a string at tokenizer initialization or setting an attribute (e.g. `tokenizer.mask_token`) update the relevant attribute in the tokenizer class but doesn't add tokens in the vocabulary. We do that because the initialization of the special tokens is usually done before the backend vocabulary is setup. This automatic and non explicit addition of tokens can also be error-prone and a source of silent-errors.
   * the recommended way to add a token and use it as a special token is to use `add_special_tokens` which will store the attribute AND add the token to the vocabulary if needed and which returns the number of added tokens
   * a new method `tokenizer.sanitize_special_tokens()` can be used to make sure all the special tokens are in the vocabulary and to automatically add them if it's not the case.

This PR should in particular fix cleanly the Roberta specific behavior discussed in:
- https://github.com/huggingface/transformers/pull/2778
- https://github.com/huggingface/transformers/issues/3788

Roberta now behave like GPT2, i.e. without a prefix token. It can still be used in the masked-fill pipeline since with can control the behavior of the mask token with the parameters of `AddedToken` and in particular keep a space after the mask token as required to use the pipeline correctly.


# 🐛 Bug
Loading Fine Tuned BERT Sequence Model after Training
## Information

Hi All,

iam facing following issue while loading pretrained BERT Sequence model with my own data

RuntimeError: Error(s) in loading state_dict for DataParallel:
Missing key(s) in state_dict: "module.out.weight", "module.out.bias".
Unexpected key(s) in state_dict: "bert.embeddings.word_embeddings.weight", "bert.embeddings.position_embeddings.weight", "bert.embeddings.token_type_embeddings.weight", "bert.embeddings.LayerNorm.weight", "bert.embeddings.LayerNorm.bias", "bert.encoder.layer.0.attention.self.query.weight", "bert.encoder.layer.0.attention.self.query.bias", "bert.encoder.layer.0.attention.self.key.weight", "bert.encoder.layer.0.attention.self.key.bias", "bert.encoder.layer.0.attention.self.value.weight", "bert.encoder.layer.0.attention.self.value.bias", "bert.encoder.layer.0.attention.output.dense.weight", "bert.encoder.layer.0.attention.output.dense.bias", "bert.encoder.layer.0.attention.output.LayerNorm.weight", "bert.encoder.layer.0.attention.output.LayerNorm.bias", "bert.encoder.layer.0.intermediate.dense.weight", "bert.encoder.layer.0.intermediate.dense.bias", "bert.encoder.layer.0.output.dense.weight", "bert.encoder.layer.0.output.dense.bias", "bert.encoder.layer.0.output.LayerNorm.weight", "bert.encoder.layer.0.output.LayerNorm.bias", "bert.encoder.layer.1.attention.self.query.weight", "bert.encoder.layer.1.attention.self.query.bias", "bert.encoder.layer.1.attention.self.key.weight", "bert.encoder.layer.1.attention.self.key.bias", "bert.encoder.layer.1.attention.self.value.weight", "bert.encoder.layer.1.attention.self.value.bias", "bert.encoder.layer.1.attention.output.dense.weight", "bert.encoder.layer.1.attention.output.dense.bias", "bert.encoder.layer.1.attention.output.LayerNorm....

any idea about this error

Model I am using (Bert, XLNet ...):

BERT SequenceClassification

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Not able to load the Fined Tuned BERT Model from saved model directory

MODEL = BERTBaseUncased()
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    MODEL = nn.DataParallel(MODEL)
    MODEL.load_state_dict(torch.load(MODEL_PATH, map_location=device))

    MODEL.to(DEVICE)
    MODEL.eval()

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
my own task with my dataset

## To reproduce

Steps to reproduce the behavior:

1. Save the Fined Tuned BERT Sequence  Classification Model
2. Create  MODEL_PATH and config file Path from Saved Model Location
3. Run below code to test the Fined Model using Rest API to test the test data

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
import config
import torch
import flask
import time
from flask import Flask
from flask import request

#import functools
import torch.nn as nn
import joblib
from transformers import BertTokenizer,BertConfig,BertModel

app = Flask(__name__)

MODEL = None
MODEL_PATH = r'/content/model_save/pytorch_model.bin'
config_file = r'/content/model_save/config.json'
DEVICE = "cuda"
PREDICTION_DICT = dict()
memory = joblib.Memory("../input/", verbose=0)

class BERTBaseUncased(nn.Module):
    def __init__(self):
        super(BERTBaseUncased, self).__init__()
        #self.bert = transformers.BertModel.from_pretrained(config.BERT_PATH)
        #config = BertConfig.from_pretrained("bert-base-uncased", output_hidden_states=True)
        config = BertConfig(config_file)
        #self.bert = BertModel(config)
        self.bert = BertTokenizer.from_pretrained("bert-base-uncased", config=config)

        #self.bert  = BertTokenizer.from_pretrained(r"/content/model_save/pytorch_model.bin", do_lower_case=False,encoding="utf-8")
        self.bert_drop = nn.Dropout(0.3)
        self.out = nn.Linear(768, 1)

    def forward(self, ids, mask, token_type_ids):
        _, o2 = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids)
        bo = self.bert_drop(o2)
        output = self.out(bo)
        return output

def predict_from_cache(sentence):
    if sentence in PREDICTION_DICT:
        return PREDICTION_DICT[sentence]
    else:
        result = sentence_prediction(sentence)
        PREDICTION_DICT[sentence] = result
        return result

@memory.cache
def sentence_prediction(sentence):
    tokenizer = config.TOKENIZER
    max_len = config.MAX_LEN
    review = str(sentence)
    review = " ".join(review.split())

    inputs = tokenizer.encode_plus(
        review, None, add_special_tokens=True, max_length=max_len
    )

    ids = inputs["input_ids"]
    mask = inputs["attention_mask"]
    token_type_ids = inputs["token_type_ids"]

    padding_length = max_len - len(ids)
    ids = ids + ([0] * padding_length)
    mask = mask + ([0] * padding_length)
    token_type_ids = token_type_ids + ([0] * padding_length)

    ids = torch.tensor(ids, dtype=torch.long).unsqueeze(0)
    mask = torch.tensor(mask, dtype=torch.long).unsqueeze(0)
    token_type_ids = torch.tensor(token_type_ids, dtype=torch.long).unsqueeze(0)

    ids = ids.to(DEVICE, dtype=torch.long)
    token_type_ids = token_type_ids.to(DEVICE, dtype=torch.long)
    mask = mask.to(DEVICE, dtype=torch.long)

    outputs = MODEL(ids=ids, mask=mask, token_type_ids=token_type_ids)

    outputs = torch.sigmoid(outputs).cpu().detach().numpy()
    return outputs[0][0]

@app.route("/predict")
def predict():
    sentence = request.args.get("sentence")
    start_time = time.time()
    article_prediction = sentence_prediction(sentence)
    response = {}
    response["response"] = {

        "sentence": str(article_prediction),

        "time_taken": str(time.time() - start_time),
    }
    return flask.jsonify(response)

if __name__ == "__main__":
    MODEL = BERTBaseUncased()
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    MODEL = nn.DataParallel(MODEL)
    MODEL.load_state_dict(torch.load(MODEL_PATH, map_location=device))

    MODEL.to(DEVICE)
    MODEL.eval()
    app.run()
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: GPU
- Python version: 3.6
- PyTorch version (GPU?):  1.5.0+cu101
- Tensorflow version (GPU?): 2.x
- Using GPU in script?:Yes
- Using distributed or parallel set-up in script?: Dont Know


OpenAIGPTDoubleHeadsModel Don't have the "labels" attributes as it is described in the documentation

# 🐛 Bug

## Information

Model I am using OpenAIGPTDoubleHeadsModel:

Language I am using the model on English:

The problem arises when using:
* The official example scripts:
I'm using transformer version : '2.11.0'

In the documentation i found that i can pass a "labels" attribute to the OpenAIGPTDoubleHeadsModel Model to be used in the loss:
[Documentation](https://huggingface.co/transformers/model_doc/gpt.html#openaigptdoubleheadsmodel)
But when i pass it he didn't recognize it :

``model(input_ids=input_ids,` token_type_ids=token_type_ids, position_ids=None, head_mask=None, mc_token_ids = mc_token_ids, labels=lm_labels, mc_labels = mc_labels)``

``TypeError                                 Traceback (most recent call last)
<ipython-input-67-cc8f7864d8a7> in <module>()
----> 1 model(input_ids=input_ids, token_type_ids=token_type_ids, position_ids=None, head_mask=None, mc_token_ids = mc_token_ids, labels=lm_labels, mc_labels = mc_labels)
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)
TypeError: forward() got an unexpected keyword argument 'labels' ``

## Question :

Is passing the labels thorough GPT implimentation is depricated ? if yes is there a way to pass it ?
Thank you


Update Conda Release

When trying to get started with transformers using some examples from the model cards, the installation with conda results in an outdated version (2.1.1).

As a result, the example cannot run. However, when using pip to install transformers the newest version (2.11) is correctly used.

I would suggest updating the conda release in order to avoid installing via pip.


Fix TF WarmUp class

This commit fixes the TF WarmUp learning rate scheduler.

The LR shape was wrong due to warmup steps. See linked issue for more details.

Fix #5098


🐛 [TF] `create_optimizer` wrong superposition of learning rate schedules

# 🐛 Bug

When using `create_optimizer`, 2 learning rate schedulers are placed on top of each other (WarmUp and keras Polynomial Decay) :

https://github.com/huggingface/transformers/blob/efeb75b8054cc299698cf8bc09f395ada2660745/src/transformers/optimization_tf.py#L70-L80

But the step in Warmup scheduler is not updated to take into account the warmup steps, which lead to a wrong learning rate :

![image](https://user-images.githubusercontent.com/43774355/84980216-69edd500-b16c-11ea-97f6-44642bdc64ce.png)

---

Expected learning rate shape :

![Dessin sans titre](https://user-images.githubusercontent.com/43774355/84981152-d2d64c80-b16e-11ea-9092-ccc50217fc66.png)


Training the BERTSUM model

Hi, I find that the script can only predict the summaries using the BERTSUM model. Is it possible to train the BERTSUM model using this script?


Can I training a bart model from scratch by transformers?


Addition of VisualBERT

# 🌟 New model addition
## Model description
The VisualBERT model is used for multi-modal processing when the modes of images and text are present. It takes in object detection features from images, and combines them with textual embeddings from the pre-trained BERT models, pre-trained the whole thing on COCO image captioning data, using a similar MLM task as BERT. It has been shown to work well on several multi-modal tasks such as VQA, VCR, NLVR, etc.

<!-- Important information -->

## Open source status

The source code presented along with the paper can be found at
https://github.com/uclanlp/visualbert
* [x] the model implementation is available: (give details)
The model implementation can be found on the GitHub repository, in the models section: https://github.com/uclanlp/visualbert/tree/master/models
This code was provided along with the paper.
Another implementation, which is slightly harder to understand because of complex dependencies, is implemented in the Facebook Research's MMF framework: https://github.com/facebookresearch/mmf/blob/master/mmf/models/visual_bert.py

* [x] the model weights are available: (give details)
The model checkpoints that the authors used are presented as drive links in the given repository, depending on which pre-training we want. There are several links on the README file of the GitHub repository.
* [x] who are the authors: (mention them, if possible by @gh-username)
- Kai-Wei Chang: @KaiWeiChang
- Liunian Harold Li: @liunian-harold-li
- Mark Yatskar
- Da Yin
- Cho-Jui Hsieh
- Kai-Wei Chang

I want to contribute the model myself, please let me know if this is the right avenue for this, and how I can contribute.


Different output from model on CPU and GPU

# 🐛 Bug

I trained models using the `run_glue.py` script and uploaded them to the model hub. I've realized that their output slightly differs between when I do inference on the CPU and the GPU.

Absolute errors are small – on the order of `1e-7` – but that turns out to be too much for my use case.

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: MNLI
* [ ] my own task or dataset: (give details below)

## To reproduce
```python
(torch) qcuda8 04:47 PM > python
Python 3.7.7 (default, Mar 26 2020, 15:48:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> model_cpu = transformers.AutoModelForSequenceClassification.from_pretrained('textattack/bert-base-uncased-MNLI')
>>> model_gpu = transformers.AutoModelForSequenceClassification.from_pretrained('textattack/bert-base-uncased-MNLI').to('cuda')
>>>
>>> premise = "Among these are the red brick Royal Palace, which now houses the Patan Museum (Nepal's finest and most modern museum), and, facing the palace across the narrow brick plaza, eight temples of different styles and sizes."
>>> hypothesis = "The Patan Museum is down the street from the red brick Royal Palace."
>>>
>>> tokenizer = transformers.AutoTokenizer.from_pretrained('textattack/bert-base-uncased-MNLI')
>>> encoded_text = tokenizer.encode_plus((premise, hypothesis), return_tensors='pt')
>>> encoded_text['input_ids']
tensor([[101, 100, 100, 102]])
>>> encoded_text_cuda = {k: v.cuda() for k,v in encoded_text.items()}
>>> model_gpu(**encoded_text_cuda)[0].squeeze().tolist()
[-1.0867613554000854, 0.6688923239707947, 0.30274006724357605]
>>> model_cpu(**encoded_text)[0].squeeze().tolist()
[-1.0867608785629272, 0.6688917279243469, 0.3027404248714447]
```
## Expected behavior

I want the model output between the CPU model and GPU model to be the same.

## Environment info
- `transformers` version: 2.11.0
- Platform: Linux-3.10.0-693.el7.x86_64-x86_64-with-centos-7.4.1708-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


[style] add pandas to setup.cfg

Need to add pandas to setup.cfg. Otherwise, for people that have pandas installed locally isort tries to change `eli5_utils.py` everytime.


[MarianTokenizer] Switch to sacremoses for punc normalization

Attempt at fixing #4491 using @jpcorb20 's solution


encode_plus wrongly tokenizing a symbol

When I used a the bio dataset (linnaeus-IOB) to train bert on, the dataset includes the symbol (>=)  [greater than or equal], the tokenizer is separating them into two tokens

**Expected output is either**:
`>`
`##=`
**or**
`>= `


minor spelling correction in script execution command - movement pruning

actual script name - counts_parameters.py


Is there a helper script to preprocess data for T5 for masked language modeling?

Hi Team

   Thanks for the wonderful HuggingFace library !

I am now working with T5 on my own dataset. I want to know if there is any helper script that can automatically take text and mask a random set of tokens and also generate the expected output sequence for the pretraining unsupervised language modeling task.


Image GPT

# 🌟 New model addition

## Model description

OpenAI just announced Image GPT: https://openai.com/blog/image-gpt/

Although image rendering would be out of scope for Transformers, the RGB generation would still be in scope and it would be best to port the weights to a `GPT2LMModel`.

However, it's not immediately clear here how the tokenization is implemented in the downloaded model. (no separate `vocab.json`)

## Open source status

* [ ] the model implementation is available: https://github.com/openai/image-gpt
* [ ] the model weights are available: see README above
* [ ] who are the authors: @openai


Why does the T5Tokenizer prepend and '_' to every token?

I am using the T5Tokenizer as follows:

```
In [119]: tokenizer = T5Tokenizer.from_pretrained('t5-small')

In [120]: input_ids = tokenizer.encode_plus('I love my dog', return_tensors='pt')

In [121]: [tokenizer.convert_ids_to_tokens([ele]) for ele in input_ids['input_ids'][0]]
Out[121]: [['▁I'], ['▁love'], ['▁my'], ['▁dog']]
```
I even tried another one as follows:

```
In [129]: [tokenizer.convert_ids_to_tokens([ele]) for ele in input_ids['input_ids'][0]]
Out[129]: [['▁I'], ['▁love'], ['▁my'], ['▁school'], ['▁National'], ['x'], ['y'], ['z']]
```
Above, shouldn't 'x', 'y', and 'z' have # prepended to them  as they are part of the same word?

Why do I see an underscore before every token? Is this related to how it is sent inside T5?


SummarizationPipeline: init required task name

Otherwise, can't do:

```python
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
model = AutoModelWithLMHead.from_pretrained("facebook/bart-large-cnn")
p = SummarizationPipeline(model=model, tokenizer=tokenizer)

p("Long boring text to summarize, etc. etc.")
```


Add missing arg in 02-transformers notebook

Add missing `from_tf=True` arg when creating the model `AutoModel.from_pretrained()` to avoid an OSError from loading a PyTorch model from a TF 2.0 checkpoint.

Also fixed two small typos in markdown text.


Update installation page and add contributing to the doc

This PR simplifies the installation page, adds the mention of TF/PT installation with one command only (for CPU) and adds a test that the installation was successful.

The tests mention is moved to CONTRIBUTING. The mention of the tokenization process for OpenAI GPT is moved to that model doc page.

It also adds the contributing guide to the documentation with a simlink (add to fix a few links to make it work).


updated hans eval instructions

I've updated the info regarding how hans evaluation can be carried out.

I've also renamed `run_hans.py` to `test_hans.py` to restore the previous file convention and also indicate that HANS only supports evaluation.


Add header and fix command

This finishes to fix #4742


01_how-to-train.ipynb broken

# 🐛 Bug

## To reproduce

Steps to reproduce the behavior:

1. Go to https://github.com/huggingface/transformers/tree/master/examples
2. Click the colab for `language-modeling`: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
3. Run notebook

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The notebook finishes succesfuly

What I get is:

```
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-5-52625a7c86e5> in <module>()
      1 get_ipython().system('mkdir EsperBERTo')
----> 2 tokenizer.save("EsperBERTo")

/usr/local/lib/python3.6/dist-packages/tokenizers/implementations/base_tokenizer.py in save(self, path, pretty)
    330                 A path to the destination Tokenizer file
    331         """
--> 332         return self._tokenizer.save(path, pretty)
    333
    334     def to_str(self, pretty: bool = False):

Exception: Is a directory (os error 21)
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA


[docs] fix T5 training doc

This PR fixes T5 training doc. In a recent commit `lm_labels` is changed to `labels`. Made the doc changes accordingly. Regarding issue #5079

@patrickvonplaten


How do I pre-train the T5 model in HuggingFace library using my own text corpus?

Hello,

  I understand how the T5 architecture works and I have my own large corpus where I decide to mask a sequence of tokens and replace them with sentinel tokens.

I also understand about the tokenizers in HuggingFace, specially the T5 tokenizer.

Can someone point me to a document or refer me to the class that I need to use to pretrain T5 model on my corpus using the masked language model approach?

Thanks


Add BERT Loses Patience (Patience-based Early Exit) based on the paper https://arxiv.org/abs/2006.04152 and the official implementation https://github.com/JetRunner/PABEE

It's impossible to make PABEE's ALBERT and BERT compatible with the standard API (e.g., `run_glue.py` so I keep the modeling files in a separate directory, under `example/bert_loses_patience`, instead of putting them alongside `modeling_bert.py`)


Several problems with named entites predicted with the ner pipeline

# 🐛 Bug

## Information

Hello,

I am using the `bert-base-cased` model to predict named entities for a bunch of sentences (around 29 900). I am facing 3 main issues :
1. Residual '##' in grouped entities' word field (So they are not well grouped)
2. [UNK] (or [CLS]) tokens inside word fields
3. Missing syllables in the word fields

Model I am using (Bert, XLNet ...): Bert  (`dbmdz/bert-large-cased-finetuned-conll03-english`)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: NER with my own unlabelled dataset

## To reproduce

I didn't find the official example for this so I made my own script with the `TokenClassificationPipeline` :

```Python
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
from transformers import TokenClassificationPipeline

model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

nlp_not_grouped = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=False
)

nlp_grouped = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=True
)

seq1 = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very" \
           "close to the Manhattan Bridge."

seq2 = "In addition , the Blabla Group has completed the acquisition of ISO / TS16949 certification ."

seq3 = "Product sales to the PSA Peugeot CitroÃ«n group totaled â‚¬ 1 , 893 . 6 million in 2012 , down 8 . 1 % "\
"on a reported basis and 10 . 4 % on a like - for - like basis ."

seq4 = "To prepare as best as possible the decisions falling under its responsibilities , Faurecia â€™ s Board of"\
" Directors has set up three committees : c Audit Committee ; c Strategy Committee ; c Appointments and Compensation"\
" Committee ."

sequences = [seq1, seq2, seq3, seq4]

for i, seq in enumerate(sequences):
    ngrouped, grouped = nlp_not_grouped(seq), nlp_grouped(seq)
    print(f"===================== sentence n°{i+1}")
    print("---Sentence---")
    print(seq)
    print("---Not grouped entities---")
    for ngent in ngrouped:
        print(ngent)
    print("---Grouped entities---")
    for gent in grouped:
        print(gent)

```

I have about 29 900 sentences. For each sentence I want to predict all the named entities in it and then locate them in the sentence. Once I have an entity, I use a regex to find it in the original sentence (before the tokenization step) like this :

```Python
start, stop = re.search(re.escape(ent['word']), sent).span()
```

Where `ent['word']` is the text of an entity found in a sentence. For instance, it can be `"London"` for the sentence (sent) `"London is really a great city"`. However I do this later with the grouped entities but since there are errors in it many are discarded because `re.search()` raises an exception (that I catch).

Steps to reproduce the behavior:

You just have to run my script to predict the entities for the four sentences. Here is what I get :

```Python
===================== sentence n°1
---Sentence---
Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore veryclose to the Manhattan Bridge.
---Not grouped entities---
{'word': 'Hu', 'score': 0.9995108246803284, 'entity': 'I-ORG', 'index': 1}
{'word': '##gging', 'score': 0.989597499370575, 'entity': 'I-ORG', 'index': 2}
{'word': 'Face', 'score': 0.9979704022407532, 'entity': 'I-ORG', 'index': 3}
{'word': 'Inc', 'score': 0.9993758797645569, 'entity': 'I-ORG', 'index': 4}
{'word': 'New', 'score': 0.9993405938148499, 'entity': 'I-LOC', 'index': 11}
{'word': 'York', 'score': 0.9991927742958069, 'entity': 'I-LOC', 'index': 12}
{'word': 'City', 'score': 0.9993411302566528, 'entity': 'I-LOC', 'index': 13}
{'word': 'D', 'score': 0.986336350440979, 'entity': 'I-LOC', 'index': 19}
{'word': '##UM', 'score': 0.9396238923072815, 'entity': 'I-LOC', 'index': 20}
{'word': '##BO', 'score': 0.9121386408805847, 'entity': 'I-LOC', 'index': 21}
{'word': 'Manhattan', 'score': 0.9839190244674683, 'entity': 'I-LOC', 'index': 29}
{'word': 'Bridge', 'score': 0.9924242496490479, 'entity': 'I-LOC', 'index': 30}
---Grouped entities---
{'entity_group': 'I-ORG', 'score': 0.9966136515140533, 'word': 'Hugging Face Inc'}
{'entity_group': 'I-LOC', 'score': 0.9992914994557699, 'word': 'New York City'}
{'entity_group': 'I-LOC', 'score': 0.9460329612096151, 'word': 'DUMBO'}
{'entity_group': 'I-LOC', 'score': 0.9881716370582581, 'word': 'Manhattan Bridge'}
===================== sentence n°2
---Sentence---
In addition , the Blabla Group has completed the acquisition of ISO / TS16949 certification .
---Not grouped entities---
{'word': 'B', 'score': 0.9997261762619019, 'entity': 'I-ORG', 'index': 5}
{'word': '##la', 'score': 0.997683048248291, 'entity': 'I-ORG', 'index': 6}
{'word': '##bla', 'score': 0.99888014793396, 'entity': 'I-ORG', 'index': 7}
{'word': 'Group', 'score': 0.9992784261703491, 'entity': 'I-ORG', 'index': 8}
{'word': 'ISO', 'score': 0.9711909890174866, 'entity': 'I-MISC', 'index': 14}
{'word': 'T', 'score': 0.6591967344284058, 'entity': 'I-ORG', 'index': 16}
{'word': '##S', 'score': 0.658642053604126, 'entity': 'I-MISC', 'index': 17}
{'word': '##16', 'score': 0.5059574842453003, 'entity': 'I-MISC', 'index': 18}
{'word': '##9', 'score': 0.5067382454872131, 'entity': 'I-MISC', 'index': 21}
---Grouped entities---
{'entity_group': 'I-ORG', 'score': 0.9988919496536255, 'word': 'Blabla Group'}
{'entity_group': 'I-MISC', 'score': 0.9711909890174866, 'word': 'ISO'}
{'entity_group': 'I-ORG', 'score': 0.6591967344284058, 'word': 'T'}
{'entity_group': 'I-MISC', 'score': 0.5822997689247131, 'word': '##S16'}
===================== sentence n°3
---Sentence---
Product sales to the PSA Peugeot CitroÃ«n group totaled â‚¬ 1 , 893 . 6 million in 2012 , down 8 . 1 % on a reported basis and 10 . 4 % on a like - for - like basis .
---Not grouped entities---
{'word': 'PS', 'score': 0.9970256686210632, 'entity': 'I-ORG', 'index': 5}
{'word': '##A', 'score': 0.9927457571029663, 'entity': 'I-ORG', 'index': 6}
{'word': 'P', 'score': 0.9980151653289795, 'entity': 'I-ORG', 'index': 7}
{'word': '##eu', 'score': 0.9897757768630981, 'entity': 'I-ORG', 'index': 8}
{'word': '##ge', 'score': 0.996147871017456, 'entity': 'I-ORG', 'index': 9}
{'word': '##ot', 'score': 0.9928787350654602, 'entity': 'I-ORG', 'index': 10}
{'word': '[UNK]', 'score': 0.5744695067405701, 'entity': 'I-ORG', 'index': 11}
---Grouped entities---
{'entity_group': 'I-ORG', 'score': 0.934436925819942, 'word': 'PSA Peugeot [UNK]'}
===================== sentence n°4
---Sentence---
To prepare as best as possible the decisions falling under its responsibilities , Faurecia â€™ s Board of Directors has set up three committees : c Audit Committee ; c Strategy Committee ; c Appointments and Compensation Committee .
---Not grouped entities---
{'word': 'F', 'score': 0.9983997941017151, 'entity': 'I-ORG', 'index': 14}
{'word': '##au', 'score': 0.9473735690116882, 'entity': 'I-ORG', 'index': 15}
{'word': '##re', 'score': 0.9604568481445312, 'entity': 'I-ORG', 'index': 16}
{'word': '##cia', 'score': 0.992807149887085, 'entity': 'I-ORG', 'index': 17}
{'word': 'Board', 'score': 0.8452167510986328, 'entity': 'I-ORG', 'index': 20}
{'word': 'of', 'score': 0.5921975374221802, 'entity': 'I-ORG', 'index': 21}
{'word': 'Directors', 'score': 0.6778028607368469, 'entity': 'I-ORG', 'index': 22}
{'word': 'Audi', 'score': 0.9764850735664368, 'entity': 'I-ORG', 'index': 30}
{'word': '##t', 'score': 0.9692177772521973, 'entity': 'I-ORG', 'index': 31}
{'word': 'Committee', 'score': 0.9959701299667358, 'entity': 'I-ORG', 'index': 32}
{'word': 'Strategy', 'score': 0.9705951809883118, 'entity': 'I-ORG', 'index': 35}
{'word': 'Committee', 'score': 0.994032621383667, 'entity': 'I-ORG', 'index': 36}
{'word': 'A', 'score': 0.9764854907989502, 'entity': 'I-ORG', 'index': 39}
{'word': '##oint', 'score': 0.7803319692611694, 'entity': 'I-ORG', 'index': 41}
{'word': '##ments', 'score': 0.7828453779220581, 'entity': 'I-ORG', 'index': 42}
{'word': 'and', 'score': 0.9625542163848877, 'entity': 'I-ORG', 'index': 43}
{'word': 'Co', 'score': 0.9904180765151978, 'entity': 'I-ORG', 'index': 44}
{'word': '##mp', 'score': 0.9140805602073669, 'entity': 'I-ORG', 'index': 45}
{'word': '##ens', 'score': 0.8661588430404663, 'entity': 'I-ORG', 'index': 46}
{'word': '##ation', 'score': 0.9150537252426147, 'entity': 'I-ORG', 'index': 47}
{'word': 'Committee', 'score': 0.9888517260551453, 'entity': 'I-ORG', 'index': 48}
---Grouped entities---
{'entity_group': 'I-ORG', 'score': 0.9747593402862549, 'word': 'Faurecia'}
{'entity_group': 'I-ORG', 'score': 0.7050723830858866, 'word': 'Board of Directors'}
{'entity_group': 'I-ORG', 'score': 0.9805576602617899, 'word': 'Audit Committee'}
{'entity_group': 'I-ORG', 'score': 0.9823139011859894, 'word': 'Strategy Committee'}
{'entity_group': 'I-ORG', 'score': 0.9764854907989502, 'word': 'A'}
{'entity_group': 'I-ORG', 'score': 0.9000368118286133, 'word': '##ointments and Compensation Committee'}
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

For the first sentence (seq1) everything is fine. It's the example of the NER section under Usage section of the documentation : https://huggingface.co/transformers/usage.html#named-entity-recognition

With the other sentences we can see one example of each problem :

### Residual '##' in word pieces

```Python
{'entity_group': 'I-MISC', 'score': 0.9711909890174866, 'word': 'ISO'}
{'entity_group': 'I-ORG', 'score': 0.6591967344284058, 'word': 'T'}
{'entity_group': 'I-MISC', 'score': 0.5822997689247131, 'word': '##S16'}
```

In seq 2, there is `'##S16'` as a word. Obviously, it should have been grouped with the precending entity and form `TS16` even maybe `'ISO / TS16949'` like this :

```Python
{'entity_group': 'I-MISC', 'score': 0.9711909890174866, 'word': 'ISO / TS16949'}
```

### [UNK] tokens in the `word` field

```Python
{'entity_group': 'I-ORG', 'score': 0.934436925819942, 'word': 'PSA Peugeot [UNK]'}
```

Because maybe of the ugly written CitroÃ«n which stands for Citroën. The entity found is `'PSA Peugeot [UNK]'`. In this case it would be better to just put `'PSA Peugeot'` if the last token is identified as [UNK] :

```Python
{'entity_group': 'I-ORG', 'score': 0.934436925819942, 'word': 'PSA Peugeot'}
```

### Syllables lost

For the last sentence we can see that 'Appointments and Compensation Committee' as be splitted into :

```Python
{'entity_group': 'I-ORG', 'score': 0.9764854907989502, 'word': 'A'}
{'entity_group': 'I-ORG', 'score': 0.9000368118286133, 'word': '##ointments and Compensation Committee'}
```

instead of :

```Python
{'entity_group': 'I-ORG', 'score': 0.9000368118286133, 'word': 'Appointments and Compensation Committee'}
```

The entity is not well grouped but more importantly the 'pp' is missing so even if we decided to blend the two groups we wouldn't get the real entity. This problem was first raised here : #4816. I've actually encountered this problem trying to fix the first one : I noticed some entity grouped like this, miss some syllables. The pipeline with `grouped_entity=False` already lost the 'pp' :

```Python
{'word': 'A', 'score': 0.9764854907989502, 'entity': 'I-ORG', 'index': 39}
{'word': '##oint', 'score': 0.7803319692611694, 'entity': 'I-ORG', 'index': 41}
{'word': '##ments', 'score': 0.7828453779220581, 'entity': 'I-ORG', 'index': 42}
```

It seems the way the pipeline blends each tokens is not ok because when I predict the label for each tokens with the code example of the documentation, I get this :

`[('[CLS]', 'O'), ('To', 'O'), ('prepare', 'O'), ('as', 'O'), ('best', 'O'), ('as', 'I-ORG'), ('possible', 'I-ORG'), ('the', 'I-ORG'), ('decisions', 'I-ORG'), ('falling', 'I-ORG'), ('under', 'I-ORG'), ('its', 'I-ORG'), ('responsibilities', 'O'), (',', 'O'), ('F', 'O'), ('##au', 'O'), ('##re', 'O'), ('##cia', 'O'), ('[UNK]', 'O'), ('s', 'O'), ('Board', 'O'), ('of', 'O'), ('Directors', 'O'), ('has', 'O'), ('set', 'O'), ('up', 'O'), ('three', 'O'), ('committees', 'O'), (':', 'O'), ('c', 'O'), ('Audi', 'O'), ('##t', 'O'), ('Committee', 'O'), (';', 'O'), ('c', 'O'), ('Strategy', 'O'), ('Committee', 'O'), (';', 'O'), ('c', 'O'), ('A', 'O'), ('##pp', 'O'), ('##oint', 'O'), ('##ments', 'O'), ('and', 'O'), ('Co', 'O'), ('##mp', 'O'), ('##ens', 'O'), ('##ation', 'O'), ('Committee', 'O'), ('.', 'O'), ('[SEP]', 'O')]`

There are those tokens :

`('A', 'O'), ('##pp', 'O'), ('##oint', 'O'), ('##ments', 'O')` for 'Appointments'

## Environment info

- `transformers` version: 2.11.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0+cpu (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

EDIT : Typos


Colab session crashes on transformers

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
Hi everybody,
Something really strange happens in Colab after the recent updates of the DataCollate class.
I don't know if the two things are correlated, however, after I install the following packages

!git clone https://github.com/huggingface/transformers.git
!pip install ./transformers
!pip install -U nlp

and then I try to load them

import nlp
from transformers import T5Tokenizer

The Colab instance crashes.
Please find below the log.

**WARNING:root:kernel 485c962d-efa0-4103-9c68-ed22abd8839f restarted**


Converting to ONNX doesn't apply to all models

Is it possible to list or highlight which models are ONNX convertible?

On a fresh python environment:

```
python -m pip install -U transformers
python -m  pip install mosestokenizer
```

The transformer version on the machine is `2.11.0`

When trying to convert the https://huggingface.co/transformers/model_doc/marian.html to ONNX, it throws the following error:

```
$ python convert_graph_to_onnx.py --framework pt --model Helsinki-NLP/opus-mt-en-ROMANCE opus-mt-en-romance.onnx

Neither PyTorch nor TensorFlow >= 2.0 have been found. Models won't be available and only tokenizers, configurationand file/data utilities can be used.
ONNX opset version set to: 11

Loading pipeline (model: Helsinki-NLP/opus-mt-en-ROMANCE, tokenizer: Helsinki-NLP/opus-mt-en-ROMANCE)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/tokenization_utils.py:828: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  warnings.warn(
stdbuf was not found; communication with perl may hang due to stdio buffering.
Error while converting the model: 'NoneType' object has no attribute 'from_pretrained'
```

Then after installing pytorch

```
$ python -m pip install -U pytorch

$ python convert_graph_to_onnx.py --framework pt --model Helsinki-NLP/opus-mt-en-ROMANCE opus-mt-en-romance.onnx

ONNX opset version set to: 11
Loading pipeline (model: Helsinki-NLP/opus-mt-en-ROMANCE, tokenizer: Helsinki-NLP/opus-mt-en-ROMANCE)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/tokenization_utils.py:828: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  warnings.warn(
stdbuf was not found; communication with perl may hang due to stdio buffering.

Downloading: 100%|██████████████████████████████| 312M/312M [01:59<00:00, 2.61MB/s]

Error while converting the model: Folder /Users/username/git-stuff/transformers/src/transformers is not empty, aborting conversion
```

Then after creating a new directory:

```

$ python ../convert_graph_to_onnx.py --framework pt --model Helsinki-NLP/opus-mt-en-ROMANCE opus-mt-en-romance.onnx

ONNX opset version set to: 11
Loading pipeline (model: Helsinki-NLP/opus-mt-en-ROMANCE, tokenizer: Helsinki-NLP/opus-mt-en-ROMANCE)

/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/tokenization_utils.py:828: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  warnings.warn(
stdbuf was not found; communication with perl may hang due to stdio buffering.

Using framework PyTorch: 1.5.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
Found output output_1 with shape: {0: 'batch', 1: 'sequence'}
Ensuring inputs are in correct order
decoder_input_ids is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:173: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not padding_mask.any():
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:590: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert embed_dim == self.embed_dim
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:591: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert list(query.size()) == [tgt_len, bsz, embed_dim]
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:633: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_weights.size() == (bsz * self.num_heads, tgt_len, src_len)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:642: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert key_padding_mask is None or key_padding_mask.size()[:2] == (bsz, src_len,)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:654: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_output.size() == (bsz * self.num_heads, tgt_len, self.head_dim)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/torch/onnx/utils.py:736: UserWarning: ONNX export failed on ATen operator triu because torch.onnx.symbolic_opset11.triu does not exist
  warnings.warn("ONNX export failed on ATen operator {} because "
Error while converting the model: Exporting the operator triu to ONNX opset version 11 is not supported. Please open a bug to request ONNX export support for the missing operator.
```

It is understandable that ONNX might not be supporting all models that are available through Huggingface's transformers.

**But is there a way to know which model(s) are ONNX convertible and which aren't?**


how to get complete URLs to weights in 2.11.0

# ❓ Questions & Help

Since it is related to the newest release, I would like to raise the question here. As our servers in the company are not able to access the HF url, we have to download the models locally and upload to the servers. Now seems I couldn't find the links to download `pytorch_model.bin`, `config.json`, `vocab.json`, `merge.txt`.

The only one I can find is https://huggingface.co/facebook/bart-large

But  it only shows:

| File name | Last modified | File size|
|-- | -- | --|
|config.json | Fri, 24 Apr 2020 15:58:48 GMT | 1.2KB|
|pytorch_model.bin | Wed, 12 Feb 2020 19:53:45 GMT | 1.5GB|
|rust_model.ot | Sat, 25 Apr 2020 15:33:01 GMT | 1.9GB|

There is no `vocab.json`, `merge.txt`.  I want to find the complete URLs to these files.


fix typo


🐛 [TFTrainer] Wrong number of optimization steps

# 🐛 Bug

It seems TFTrainer compute the wrong number of optimization steps :

https://github.com/huggingface/transformers/blob/e4aaa4580515446cd5a2972ab42fec0b95819c84/src/transformers/trainer_tf.py#L79

**It does not take into account the gradient accumulation**.

---

I believe this line should be changed to :

```
if self.args.dataloader_drop_last:
    approx = math.floor
else:
    approx = math.ceil
self.train_steps: int = approx(self.num_train_examples / (self.args.train_batch_size * self.args.gradient_accumulation_steps))
```

---

Also, on TPU `drop_remainder` should be called on gradient accumulation steps as well.

https://github.com/huggingface/transformers/blob/e4aaa4580515446cd5a2972ab42fec0b95819c84/src/transformers/trainer_tf.py#L81-L86

Should be changed to :

```
ds = (
    self.train_dataset.cache()
    .shuffle(self.num_train_examples)
    .batch(self.args.train_batch_size * self.args.gradient_accumulation_steps, drop_remainder=self.args.dataloader_drop_last)
    .unbatch()
    .batch(self.args.train_batch_size)
    .prefetch(tf.data.experimental.AUTOTUNE)
)
```

---

@jplu Maybe to add in #5065 ?


glue.py Data Processor Index Error for Large Data

# 🐛 Bug

## Information

I am using `bert-base-uncase` and `roberta-base` for sentence pair classification, following the same exact example concept and data processing as MRPC. I have a large collection of private data that is in the _exact same format of MRPC_, and I am using `run_glue.py`as a base code for running my data.

Everything works great when I have smaller data files (between 10K-200k samples) but the issue happens when I increase the data size (300K and above); I am not sure if the issue is a bug or a memory issue. Here is the issue that happens:

```
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/transformers/data/processors/glue.py", line 200, in _create_examples
text_a = line[3]
IndexError: list index out of range

```

The data, the format and the values are consistent (I am not feeding in an empty file). If I concatenated the data in smaller samples works just fine. Is this an issue with the data processor or some sort of memory issue? it is perhaps important to note that I have not had any issues with using this large dataset for training different models.

## To reproduce

Steps to reproduce the behavior:

1. Have a _title-pair data_ in the exact format of MRPC that is large, say larger than 300K
2. Run the `run_glue.py` with MRPC as the task
and you should get the following error:

## The Error

```
Traceback (most recent call last):
  File "run_glue.py", line 262, in <module>
    main()
  File "run_glue.py", line 140, in main
    train_dataset = GlueDataset(data_args, tokenizer=tokenizer) if training_args.do_train else None
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/transformers/data/datasets/glue.py", line 118, in __init__
    examples = self.processor.get_train_examples(args.data_dir)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/transformers/data/processors/glue.py", line 179, in get_train_examples
    return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/transformers/data/processors/glue.py", line 200, in _create_examples
    text_a = line[3]
IndexError: list index out of range
```

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-4.14.171-105.231.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (True)

Thank you very much for your time and help, in advance.


Errors while running pytest

# ❓ Questions & Help

Hi!

I followed the installation procedures from source for a conda environment.
`make test-examples` returns with the following error:

```
FAILED examples/token-classification/test_ner_examples.py::ExamplesTests::test_run_ner - AssertionError: 2.1329751014709473 not less than 1.5

FAILED examples/test_examples.py::ExamplesTests::test_run_glue - AssertionError: 0.5 not greater than or equal to 0.75
```

I also tried `make test` but get the following errors:

```
FAILED tests/test_modeling_distilbert.py::DistilBertModelTest::test_multigpu_data_parallel_forward - RuntimeError: Caught RuntimeError in replica 0 on device 0.

FAILED tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward - RuntimeError: Gather input tensors must have the same number of dimensions: got 1, ...

FAILED tests/test_modeling_roberta.py::RobertaModelTest::test_multigpu_data_parallel_forward - RuntimeError: Caught RuntimeError in replica 0 on device 0.

FAILED tests/test_modeling_bert.py::BertModelTest::test_multigpu_data_parallel_forward - RuntimeError: Caught RuntimeError in replica 0 on device 0.

FAILED tests/test_modeling_albert.py::AlbertModelTest::test_multigpu_data_parallel_forward - RuntimeError: Caught RuntimeError in replica 0 on device 0.

FAILED tests/test_modeling_xlnet.py::XLNetModelTest::test_multigpu_data_parallel_forward - RuntimeError: Caught RuntimeError in replica 0 on device 0.
```

I followed the setup steps mentioned in the original readme. I am running the tests in a 8 GPU machine.
Please let me know how to fix this.

output of `trasnformers-cli env`

```
- `transformers` version: 2.11.0
- Platform: Linux-5.3.0-1023-aws-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes
```

Thanks!!


Typo


Fix all sphynx warnings

This PR touches a lot of files but nothing that should break anything, it's just there to fix all sphynx warnings. Why? Well some of them are just there to annoy us and don't have any effect, but for roughly half of them, there is something wrong going on in the docs so it's best to fix them all, especially to make it easier to spot new warnings introduced when writing new docs.

The only thing that is a real change is that I removed `members` in the `AdamWeightDecay` because it was documenting its `apply_gradients` method using the docstring from keras which is not sphynx-compatible (it was not rendering properly in our docs). If we really want that method documented (it was the only one), I can rewrite the docstring.


Modify BERT/BERT-descendants to be TorchScript-able (not just traceable)

# 🚀 Feature request

Modify BERT models (src/transformers/modeling_bert.py) to conform to TorchScript requirements, so they can be ``jit.script()``-ed, not just ``jit.trace()``-ed (as is [currently the only supported option](https://huggingface.co/transformers/torchscript.html))

*Note:* I have a working version implementing this, which I would like to contribute.
See below.

## Motivation
A scriptable model would allow for variable-length input, offering big speedup gains and simplification (no need to create different models for different input lengths).

In addition, it would avoid other potential pitfalls with tracing (e.g., code paths that are input dependent and not covered by the tracing example input).

Related issues:
https://github.com/huggingface/transformers/issues/2417
https://github.com/huggingface/transformers/issues/1204
possibly also
https://github.com/huggingface/transformers/issues/1477
https://github.com/huggingface/transformers/issues/902

## Your contribution
I have a working PR that modifies all the models in src/transformers/modeling_bert.py and makes them TorchScript-able. I have not tested it on other models that use BERT components (e.g., albert), but it should be possible to expand the capability to those, as well.

However, it would require some significant work to make it ready for submission: besides formatting, documentation, testing etc., my current version changes the method signatures, and I would need to avoid that to maintain backward-compatibility.

Before putting in that work, I'd like to make sure that such a PR is something you'd be interested in and would be willing to merge in, assuming it meets the requirements.


Tokenization+Transformers works with PyTorch but not TensorFlow on TPU

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): RoBERTa

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. In order to use Huggingface's tokenizer with TensorFlow's data pipeline (`tf.data.Dataset.map()`), the tokenize function must be wrapped in `tf.py_function()` or `tf.numpy_function()` (issue #3851).
2. Neither of these functions are not supported on TPU (tensorflow/tensorflow#30818).
3. This makes it impossible to run Huggingface transformers + tokenizer on a TPU using TensorFlow
4. However, the same tokenization on a TPU works under PyTorch (!)

One workaround is to do tokenization before entering the TF data pipeline, but unfortunately my dataset is too large for that.

Example of code that is necessary, but fails on a TPU:
```
def tokenize_encode_map_fn(text):
    encoded = tf.py_function(tokenize_encode, # tokenize_encode is a wrapper around the Huggingface tokenizer and encoder
                             inp=[text, hypothesis],
                             Tout=[tf.int32, tf.int32])
    return {"input_ids": encoded[0], "attention_mask": encoded[1]}

tf_dataset.map(map_fn)
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Be able to run tokenizer on a TPU with TensorFlow, like you can in PyTorch
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: transformers-2.11.0
- Platform: Google Colab TPU
- Python version: Python 3.6.9
- PyTorch version (GPU?):
- Tensorflow version (GPU?): TensorFlow 2.2.0 / TPU
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


[WIP] TF Trainer with TPUs

This PR is to make the TF trainer fully compliant with the TPUs.

Should fix #5042, #4996 and #4994.


Reorganize documentation

This PR does two things:
- reorganize the doc topics in five sections
- complete the model list in the index

Also I cut all lines at 119 chars (like the code) otherwise it's not readable in visual studio code (and I imagine other viewers), removed the stars from authors since it wasn't pointing to anything (can add them back but we should explain what they mean in that case) and made all authors list comma-separated with one last 'and'.


Non-deterministic training issue on GPU: TF-BERT

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
TF-BERT
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
SST-2
* [ ] my own task or dataset: (give details below)

## To reproduce

In spite of combining learnings from:
* [the "complete recipe" in NVIDIA's slides from gputechconf](https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9911-determinism-in-deep-learning.pdf)
* [a recently suggested workaround](https://github.com/tensorflow/tensorflow/issues/38185#issuecomment-643014439) for non-determinism issues with crossentropy loss

... I am still arriving at the following [short, non-deterministic colab notebook example](https://colab.research.google.com/drive/1VSU8lYFD0E1HKZrIL1MvyIRwAktlSF_t?usp=sharing) to train BERT .

My results for the sum of model weights (as computed with [this suggested function](https://github.com/NVIDIA/tensorflow-determinism/issues/2#issuecomment-548210203)) after training **for only 5 steps** is (differences are **`highlighted`** below):

| | Device | Before training | After training |
| ------------- | ------------- | ------------- | ------------- |
| Run 1  | GPU | -641227.5609667897224  | -641237.442 **`5159916282`** |
| Run 2  | GPU | -641227.5609667897224  | -641237.442 **`3093758523`** |
| | |  | |
| Run 1 | CPU | -641227.5609667301178 | -641238.1506845243275 |
| Run 2 | CPU | -641227.5609667301178 | -641238.1506845243275 |

This variance gets increasingly more pronounced when the model is trained for longer periods of time.
I am expecting a general problem with the computational graph with BERT introducing non-determinism. As a result, this could affect a large part of the huggingface community.

Please keep in mind that determinism is of key importance in certain industries and also a pre-requisite for reproducible research.

Could you please help identify the source of non-determinism and provide guidance on how we can resolve it?

Steps to reproduce the behavior:

1. Execute colab notebook above on the GPU runtime using Tensorflow 2.2.0, observe non-deterministic behavior
2. Execute colab notebook above on the CPU runtime using Tensorflow 2.2.0, observe deterministic behavior

## Expected behavior
Training should be deterministic both on GPU and CPU runtime for TF 2.2.0.

## Environment info
* tensorflow==2.2.0
* nlp==0.2.1

- `transformers` version: 2.11.0
- Platform: Linux, Ubuntu 18.04.3 LTS bionic
- Python version: 3.6.9
- PyTorch version (GPU?): -
- Tensorflow version (GPU?): 2.2.0-gpu
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


What do the following parameters mean during the initialization of T5 model?

Hello,

  I am aware of the general Transformer model and I believe it is the same model used in T5 architecture.

I know we have the input vocab_size which is the total number of vocabulary size. Besides this, the important parameters would be the embedding size (the size of the embedding of each token), the number of layers, the number of heads and others.

Particularly, looking at the T5Config class that is used to initialize a T5 model, what is the d_kv and d_model parameter?

Is d_model the size of the embedding and in which case what is the d_kv? The docstring is not really clear for me.

Thanks for your help.


More flexible wandb support for Trainer

# 🚀 Feature request

**A.** Make it possible to initialize wandb outside Trainer class.
**B.** Add `use_wandb` argument to the Trainer arguments.

## Motivation

**A.** Currently, wandb inside Trainer configuration if very limited. There are only three environment variables `WANDB_WATCH`, `WANDB_PROJECT`, and `WANDB_DISABLED`. (And `WANDB_DISABLED` does not work properly in some cases).

Making it possible to initialize wandb outside Trainer will allow us to:
1. Add custom fields to wandb.config
1. Add tags, notes and just to make configuration as flexible as possible
1. Upload files
1. Use `wandb.log` more safely outside the Transformers code

It will also make the interaction with wandb more clear.

**B.** It is a much more clear interface then an env variable. The question here is which option should have the priority?

## Your contribution

There's a picture in my mind about how to do this and not to destroy backward compatibility. I can make a PR, but maybe need some minor help on writing tests.


Make default_data_collator more flexible and deprecate old behavior

This PR does two things:

- avoid breaking changes when people had a custom `DataCollator` with a `collate_batch` method (there is still the breaking change with `DataCollator` not being a class anymore)
- makes `default_data_collator` more flexible by handling dicts on top of `InputExamples` classes coming from our examples

This fixes #5049


[cleanup] examples test_run_squad uses tiny model

This speeds up the examples tests from 120s to 70s.
Note that this probably understates the difference, since all the relevant models are cached on my local machine.
We can also improve `test_run_glue` in a future PR.
I made an issue discussing improvements this PR does not fix: #5059

### More detail:

Before (master):  119.36s

```bash
============================ slowest test durations ============================
42.69s call     examples/test_examples.py::ExamplesTests::test_run_squad
21.20s call     examples/test_examples.py::ExamplesTests::test_run_glue
19.02s call     examples/token-classification/test_ner_examples.py::ExamplesTests::test_run_ner
13.65s call     examples/test_examples.py::ExamplesTests::test_run_language_modeling
3.80s call     examples/test_examples.py::ExamplesTests::test_generation

```
After: 69.3 Seconds
```bash
============================ slowest test durations ============================
19.12s call     examples/token-classification/test_ner_examples.py::ExamplesTests::test_run_ner
14.60s call     examples/test_examples.py::ExamplesTests::test_run_language_modeling
13.49s call     examples/test_examples.py::ExamplesTests::test_run_glue
3.87s call     examples/summarization/test_summarization_examples.py::TestBartExamples::test_bart_run_sum_cli
3.08s call     examples/translation/t5/test_t5_examples.py::TestT5Examples::test_t5_cli
2.64s call     examples/summarization/test_summarization_examples.py::TestT5Examples::test_t5_cli
2.20s call     examples/summarization/test_summarization_examples.py::TestBartExamples::test_t5_run_sum_cli
1.81s call     examples/test_examples.py::ExamplesTests::test_run_squad
1.67s call     examples/test_examples.py::ExamplesTests::test_generation

```


Error when loading Flaubert model

# 🐛 Bug

## Information
I am trying to run the  example run_ner.pl using the Flaubert model.
But I got this error:

Traceback (most recent call last):
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 191, in _check_seekable
    f.seek(f.tell())
AttributeError: 'NoneType' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/py37/lib/python3.7/site-packages/transformers/modeling_utils.py", line 516, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 549, in _load
    _check_seekable(f)
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 194, in _check_seekable
    raise_err_msg(["seek", "tell"], e)
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 187, in raise_err_msg
    raise type(e)(msg)
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/py37/lib/python3.7/site-packages/transformers/modeling_auto.py", line 1098, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/py37/lib/python3.7/site-packages/transformers/modeling_utils.py", line 519, in from_pretrained
    "Unable to load weights from pytorch checkpoint file. "
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

I got an other error when I added  from_tf=True.

Do you have idea how can I solve this problem.


Examples tests improvements

Examples

Which is related to examples in general

Help wanted

Extra attention is needed, help appreciated

There are a few things about the `examples/` tests that are suboptimal:

1. They never use cuda or fp16, even if they are available.
2. The `@slow` decorator used in the main tests is not importable, so there are no @slow tests.
3. `test_run_glue` uses distilbert-case-cased. It should use a smaller model, one of the `tiny` family [here](https://huggingface.co/models?search=sshleifer/tiny) or a new tiny model.
4. There is no test coverage for TPU.

Any help on any of these fronts would be much appreciated!


Add more tests on tokenizers serialization - fix bugs

Adds more tests on tokenizer serialization (test when adding tokens, special tokens, etc).

Tokenizer's serialization was not thoroughly tested and actually had quite some holes and bugs. Fix related issues.


How can I load the finetuned BART model to memory?

I have finetuned a `facebook/bart-large` model following the example here: https://github.com/huggingface/transformers/blob/master/examples/summarization/finetune.py

As an output I got a `checkpointcheckpoint_ckpt_epoch_0.ckpt` file. How can I create a BartForConditionalGeneration instance with updated weights?


Add pad_to_multiple_of on tokenizers (reimport)

Reimported from #4731.

Introduce `pad_to_multiple_of` on both slow and fast tokenizers. This parameter introduces the "bucketizaton behaviour" also refered as Shape Polymorphism.

This is especially usefull when targetting NN dedicated accelerators such as:
- NVidia Tensor Core (on >= Volta Architecture)
- XLA (PyTorch TPU)
- XLA (Jax / Flax)

Bonus:
- Fix RobertaTokenizer when input is empty `text[0].is_space()` would crash (#3608).

Edit (@thomwolf):
- updated to the new API
- raise a `ValueError` if you want to truncation to a length which is not a multiple of `pad_to_multiple_of`


T5 model for classification doesn't work properly for large number of classes.

T5 model works properly for 50 class classification but when we try for 100 classes it outputs empty string ("") for a large number of test data. Is this model suitable for multi-class classification with a large number of classes? If yes, what do you think might be the problem with what I am doing.


How to cosume movement-pruning .h5 models in QnA pipeline

# ❓ Questions & Help

## Details

Hi, I am working on the movement pruning method for QnA models. I have performed all the [given steps](https://github.com/huggingface/transformers/tree/master/examples/movement-pruning) in order to generate the pruned model and then used [this notebook](https://github.com/huggingface/transformers/blob/master/examples/movement-pruning/Saving_PruneBERT.ipynb) to generate the .h5 model files.

Though I am facing an issue with consuming this model with `QuestionAnsweringPipeline`.

For loading the model; the config file is copied from `BertForQuestionAnswering`, as the pruning repo does not generate any config file.

`BERT_PRUNED_PATH = SERIALIZATION_DIR+'/dbg' +'/BERT_Pruned/'`

`config = BertConfig.from_json_file(BERT_PRUNED_PATH+'config.json')`

*we used squad_sparse.h5 which is renamed to tf_model.h5*

`model_BERT_pruned =
TFBertForQuestionAnswering.from_pretrained(BERT_PRUNED_PATH+'tf_model.h5',config=config)`

**or**

`model_BERT_pruned =
BertForQuestionAnswering.from_pretrained(BERT_PRUNED_PATH+'tf_model.h5',config=config,from_tf=True)
`

config =
{
  "architectures": [
    "BertForQuestionAnswering"
  ],
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

For pipeline,

`
fill_mask_qna_BERT_pruned = pipeline(
  "question-answering",
model=model_BERT_pruned,
tokenizer=tokenizer,
framework="tf"
)
`

Now, when I test the pipeline on questions and context, I get very random answers, probably because the model is not getting loaded in a proper fashion as it should be.

@VictorSanh Can you share the instructions on how to load these .h5 format pruned model using Huggingface modules? Or is there any other way to consume the model.


Fix LR decay in TF Trainer

This PR fixes mainly the issue #5045. It also provides a better alignment with the PT trainer with:

- a `set_seed()` function
- use the `logging_first_step` argument
- better logging message when training and load from checkpoint


# 🐛 Bug
TypeError: function() argument 1 must be code, not str
## Information

@dataclass
class T2TDataCollator(DataCollator):


DataCollator problem

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi everybody
I found an error in the following Colab:
https://colab.research.google.com/drive/1jwXgtOXE8v8_qkiOCbjFQRFC5semK8T7?usp=sharing

Specifically, As far I understand something changed with the implementation of the following snippet:

class T2TDataCollator(DataCollator):
    def collate_batch(self, batch: List) -> Dict[str, torch.Tensor]:
         ..........
<br>

I got the following error: **TypeError: function() argument 1 must be code, not str**

Can you suggest any workarounds?


After I resume learning, loss is greater than prev checkpoint

# ❓ Questions & Help

## Details

I learned albert mlm model.
Often an error occurs (memory exception, ... )
so i resumed learning model from last checkpoint
but, training loss goes back to the beginning.
what do i need to check ?
thank,
![loss ](https://user-images.githubusercontent.com/4244158/84736498-ed7abb00-afe0-11ea-9b30-9e669e327241.png)


How to use 16 token types in pretrained Albert/BERT?

I have a dialogue task and I use token type to distinguish the diffenrent state of the different speeches, but all the pretrained models I can find are of type_vocab_size=2. To accomplish my goal, I have to rewrite many codes in a dirty way. So I want to ask is there an elegant way to restore the pretrained weights and ignore the token type embeddings at the same time? Roughly modifying the type_vocab_size in the given config.json will certainly raise an error


ref #4733

# 🐛 Bug
## Information
Model I am using TFBertEncoder:

Language I am using the model on English:

The problem arises when using:

* [x]  my own modified scripts: (give details below)

The tasks I am working on is:

* [x]  my own task or dataset: (give details below)

## To reproduce
Steps to reproduce the behavior:

1. When I use, TFBertEncoder, I get an error.

Here is my code.

```python
import tensorflow as tf
import numpy as np
from transformers.modeling_tf_bert import BertConfig, TFBertEncoder

print(tf.__name__, tf.__version__)
input_a = tf.keras.layers.Input(shape=(91, 128))

config = BertConfig()
config.hidden_size = 128
config.num_attention_heads = 4

# config.output_attentions = False
# config.output_hidden_states = False

head_mask = [None for _ in range(config.num_hidden_layers)]
encoder_output = TFBertEncoder(config=config)([input_a, None, head_mask])[0]
print(encoder_output.shape)

test_out = tf.keras.layers.Dense(128)(encoder_output)
print(test_out.shape)
```

## Expected behavior
Here is the error:

```
(None, 91, 128)
2020-06-03 11:18:10.160647: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Failed precondition: Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
	 [[{{node output_23/dense/BiasAdd/ReadVariableOp}}]]
Traceback (most recent call last):
  File "D:/python/tx/TEST.py", line 16, in <module>
    a = tf.keras.layers.Dense(128)(encoder_output)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 720, in __call__
    base_layer_utils.create_keras_history(inputs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 187, in create_keras_history
    _, created_layers = _create_keras_history_helper(tensors, set(), [])
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  [Previous line repeated 5 more times]
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 247, in _create_keras_history_helper
    constants[i] = backend.function([], op_input)([])
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3727, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1551, in __call__
    return self._call_impl(args, kwargs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1591, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1692, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 545, in call
    ctx=ctx)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.FailedPreconditionError:  Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
	 [[node output_23/dense/BiasAdd/ReadVariableOp (defined at /python/tx/TEST.py:16) ]] [Op:__inference_keras_scratch_graph_5205]

Function call stack:
keras_scratch_graph
```

## Environment info
* `transformers` version: 2.3.0 (in conda list)
* Platform:
* Python version:3.7
* PyTorch version (GPU?):
* Tensorflow version (GPU?):TF2.1.0(GPU)
* Using GPU in script?:
* Using distributed or parallel set-up in script?:No


TFTrainer does not consider number of epochs when calculating learning rate

# 🐛 Bug
`TFTrainer` does not consider number of epochs when calculating learning rate

## Information

When using `TFTrainer`, learning rate decreases to 0 at the end of the first epoch, even when we want to train on multiple epochs.

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: run_tf_glue with mrpc task
* [ ] my own task or dataset: (give details below)

## To reproduce
`python run_tf_glue.py --model_name_or_path bert-base-cased --task_name MRPC --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir /tmp/test_hf/ --overwrite_output_dir --logging_dir hf --evaluate_during_training --eval_steps 50 --logging_steps 10`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

This is what we get, where learning rate is null after the first epoch:
![image](https://user-images.githubusercontent.com/715491/84722988-39c6ec00-af4a-11ea-9c80-f2213d5a579c.png)

You can refer to [W&B run](https://app.wandb.ai/borisd13/huggingface/runs/2zcsfumy?workspace=user-borisd13) for more details.

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Learning rate should slowly decrease until end of 3rd epoch.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-5.3.0-53-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: Yes (one only)
- Using distributed or parallel set-up in script?: No

@jplu I recorded the issue here so we don't forget to fix it.
Let me know if I can help.


refactor(wandb): consolidate import

This PR consolidates the import logic of wandb as suggested [here](https://github.com/huggingface/transformers/pull/4946#discussion_r440070708)


Fix marian tokenizer save pretrained


❓ [TFTrainer] How to run on 8 TPU cores ?

TensorFlow

Anything TensorFlow

# ❓ Questions & Help

I'm trying to run TFTrainer on 8 TPU cores, but I don't understand how to make it work.

I tried running my script with the flags `--tpu_num_cores 8 --per_device_train_batch_size 8`, expecting each core to handle a batch size of 1.

But when I print the shape of my inputs, I have `[8, x]` instead of `[1, x]`, which leads to memory error.

---

If I start the training with `--tpu_num_cores 8 --per_device_train_batch_size 1`, the shape of inputs is correct (`[1, x]`), but the number of optimization steps computed is not correct (If I have 8k samples, it says I have 8k optimization steps, but I expected 1k steps because I am using 8 TPU cores...).

---

Am I doing something wrong ? **How can I train on 8 TPU cores, with a batch size of 1 for each core ?**


How can I use tokenizer.encode_plus to input and encode 2 sentences - (query,answer) pair for training a BERT binary classifier?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


"AutoTokenizer.from_pretrained" does not work when loading a pretrained MarianTokenizer from a local directory

# 🐛 Bug

## Information

I want to save MarianConfig, MarianTokenizer, and MarianMTModel to a local directory ("my_dir") and then load them:

> import transformers
>
> transformers.AutoConfig.from_pretrained("Helsinki-NLP/opus-mt-en-de").save_pretrained("my_dir")
> transformers.AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de").save_pretrained("my_dir")
> transformers.AutoModelWithLMHead.from_pretrained("Helsinki-NLP/opus-mt-en-de").save_pretrained("my_dir")
>
> config = transformers.AutoConfig.from_pretrained("my_dir")
> tokenizer = transformers.AutoTokenizer.from_pretrained("my_dir")
> model = transformers.AutoModelWithLMHead.from_pretrained("my_dir")

But the above code failed when loading the saved MarianTokenizer from "my_dir":

> Traceback (most recent call last):
>   File "<input>", line 8, in <module>
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 206, in from_pretrained
>     return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 911, in from_pretrained
>     return cls._from_pretrained(*inputs, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 1062, in _from_pretrained
>     tokenizer = cls(*init_inputs, **init_kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_marian.py", line 83, in __init__
>     self.spm_source = load_spm(source_spm)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_marian.py", line 236, in load_spm
>     spm.Load(path)
>   File "/Users/anaconda/lib/python3.6/site-packages/sentencepiece.py", line 367, in Load
>     return self.LoadFromFile(model_file)
>   File "/Users/anaconda/lib/python3.6/site-packages/sentencepiece.py", line 177, in LoadFromFile
>     return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
> TypeError: not a string
>


Ability to pickle/unpickle BatchEncoding pickle (reimport)

Overrides the methods get_state() & set_state() to (respectively) export the content of the underlying data dictionnary and - if defined - the content of encodings.

Unittests added to covert the serialization & deserialization of all the exported properties.

Reimported from #4515


Cannot save and load pretrained MarianTokenizer

# 🐛 Bug

## Information

I want to save a pretrained MarianTokenizer to a local directory ("my_dir") and then load it:
> import transformers
> transformers.AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de").save_pretrained("my_dir")
> tokenizer = transformers.AutoTokenizer.from_pretrained("my_dir")

But the above code failed:

> Traceback (most recent call last):
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 239, in get_config_dict
>     local_files_only=local_files_only,
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/file_utils.py", line 267, in cached_path
>     raise EnvironmentError("file {} not found".format(url_or_filename))
> OSError: file my_dir/config.json not found
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "<input>", line 1, in <module>
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 195, in from_pretrained
>     config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_auto.py", line 196, in from_pretrained
>     config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 252, in get_config_dict
>     raise EnvironmentError(msg)
> OSError: Can't load config for 'my_dir'. Make sure that:
> - 'my_dir' is a correct model identifier listed on 'https://huggingface.co/models'
> - or 'my_dir' is the correct path to a directory containing a config.json file
>


The correct way to save and load pretrained MarianTokenizer?

I want to save a pretrained MarianTokenizer to a local directory ("my_dir") and then load it:
> import transformers
> transformers.AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de").save_pretrained("my_dir")
> tokenizer = transformers.AutoTokenizer.from_pretrained("my_dir")

But the above code failed:

> Traceback (most recent call last):
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 239, in get_config_dict
>     local_files_only=local_files_only,
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/file_utils.py", line 267, in cached_path
>     raise EnvironmentError("file {} not found".format(url_or_filename))
> OSError: file my_dir/config.json not found
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "<input>", line 1, in <module>
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 195, in from_pretrained
>     config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_auto.py", line 196, in from_pretrained
>     config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
>   File "/Users/anaconda/lib/python3.6/site-packages/transformers/configuration_utils.py", line 252, in get_config_dict
>     raise EnvironmentError(msg)
> OSError: Can't load config for 'my_dir'. Make sure that:
> - 'my_dir' is a correct model identifier listed on 'https://huggingface.co/models'
> - or 'my_dir' is the correct path to a directory containing a config.json file
>


Refactor Code samples; Test code samples

Refactoring the code samples in order to prevent copy/pasting the same code samples across classes while updating the model/tokenizer classes and checkpoint names.

- All models now have their docstrings updated.
- Doctest is used for testing
- Fixed a bunch of bugs in all docstrings as well as a few models. All non-cosmetic changes are highlighted below.


update for roberta and xlm

Two changes are done:

Update the inputs with langs during training and evaluation.
Update the token_type_ids for Roberta otherwise it throws an error while creating features.


Training & fine-tuning quickstart

This PR adds a short guide to the docs demonstrating how to train & fine-tune models in native PyTorch, native TF2, and using Trainer.

My aim was not to show how to train on every type of task, but simply to communicate the key points along with a couple of very simple and easy-to-follow examples. More involved examples spanning many tasks are linked at the bottom.


update for roberta and xlm

Two changes are done:
1. Update the inputs with langs during training and evaluation.
2. Update the token_type_ids for Roberta otherwise it throws an error while creating features.


Add DistilBertForMultipleChoice

Another missing model


Some changes to simplify the generation function

This PR proposes to simplify generation in `modeling_utils.py` in the following ways:

1. Removing some redundant code in `_generate_no_beam_search`: finished sequences are padded at generation time, and do not need to be padded again before returning
2. Initializes the cache in its permanent form directly for both `_generate_beam_search` and `_generate_no_beam_search`: this removes the need for a first step test in `modeling_bart.py` and `modeling_t5.py` (and presumably future cached seq2seq architectures)
3. Took all of the logit post-processing out of `_generate_beam_search` and `_generate_no_beam_search` and put it in a single `finalize_generation_logscores` function that can be used in both instead of duplicating the code

The following slow test pass in addition to the basic suite:
```
RUN_SLOW=1 pytest tests/test_modeling_bart.py
RUN_SLOW=1 pytest tests/test_modeling_gpt2.py
RUN_SLOW=1 pytest tests/test_modeling_t5.py
```

#### Small breaking changes
1. The previous versionof `_generate_no_beam_search` seemed to be adding padding twice. We removed is as is seemed redundant, but noting it here just in case.
2. This PR moves the addition of the CTRL length penalty from before to after the `log_softmax` in `_generate_beam_search`. This changes scores a little bit but apparently doesn't drastically alter model behavior. Per @patrickvonplaten :
>  Now, the only thing that will slightly change with this function is that the CTRL enforce penalty for beam search decoding. I tried the new order out on a couple of tensors and the changes are minimal. Also, the repetition penalty is very hacky anyways and we already changed the function from its original formula of the CTRL paper.


Update pipeline examples to doctest syntax

Also fix the values to what's returned. This way we can run
```
python -m doctest README.md
```
to test this codes produces the exact same results. Not sure if we want to include this in our CI in some way.


Add reference to NLP (package) dataset

model card

Related to pretrained model cards


Add reference to NLP dataset

Related to pretrained model cards


Remove old doc page and add note about cache in installation

As discussed offline, removing the old page "Loading Google AI or OpenAI pre-trained weights or PyTorch dump" and move the not about cache in the installation folder (also, making that up to date ^^).