Trainer Evaluates at each step (Not of epoch end) , indentation bug

## Environment info

- `transformers` version: 3.2.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: NA

### Who can help

 Trainer: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
- [ ] the official example scripts:
- [x] my own modified scripts:

The tasks I am working on is:
- [ ] an official GLUE/SQUaD task:
- [x] my own task or dataset:

Basic Single Sentence Classification Dataset loaded via a Dataset class

## To reproduce

Steps to reproduce the behavior:

1. Use the trainer training function with `training_args.evaluation_strategy = EvaluationStrategy.EPOCH`

## Expected behavior

Evaluation should happen after each epoch ends, but instead it happens after each step (batch).

Indentation bug

## Suggested Fix

Move the if condition on line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L829 to before line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L834 and remove 1 indentation block


BufferedWriter takes most of the time

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 Speed and Memory Benchmarks: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
english = pipeline(
    "question-answering",
    model="distilbert-base-uncased-distilled-squad",
    tokenizer="distilbert-base-uncased-distilled-squad"
)

text1 = """It comes as pubs, bars, restaurants and other hospitality venues in England are told they must have a 22:00 closing time from Thursday.

Full details will be set out by the prime minister in Parliament later.

Boris Johnson is meeting the first ministers of Scotland, Wales and Northern Ireland and will address the nation in a live broadcast at 20:00 BST on Tuesday.

As well as the early closing time for hospitality venues, he is expected to announce they will be restricted by law to table service only.
"""

%%prun
english({'question': 'Which country is the news about?', 'context': text1})
```
The profiling result is

```
6256 function calls (6155 primitive calls) in 1.097 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.713    0.713    0.713    0.713 {method 'write' of '_io.BufferedWriter' objects}
   37    0.229    0.006    0.229    0.006 {method 'matmul' of 'torch._C._TensorBase' objects}
   12    0.030    0.002    0.030    0.002 {built-in method matmul}
    5    0.020    0.004    0.020    0.004 {method 'dump' of '_pickle.Pickler' objects}
    6    0.019    0.003    0.019    0.003 {method 'softmax' of 'torch._C._TensorBase' objects}
   33    0.012    0.000    0.012    0.000 {method 'acquire' of '_thread.lock' objects}
    3    0.009    0.003    0.009    0.003 {built-in method posix.waitpid}
    6    0.009    0.002    0.009    0.002 {method 'masked_fill_' of 'torch._C._TensorBase' objects}
    6    0.009    0.001    0.009    0.001 {built-in method torch._C._nn.gelu}
   37    0.006    0.000    0.235    0.006 functional.py:1655(linear)
 94/1    0.005    0.000    0.325    0.325 module.py:710(_call_impl)
...
```

## Expected behavior

Most time is spent on inference such as the method `matmul`.


Trainer.py module 'datasets' has no attribute 'Dataset'

I'm trying to use a Trainer, but I get this error:
```
c:\users\francois\appdata\local\programs\python\python37\lib\site-packages\transformers\trainer.py in __init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, tb_writer, optimizers, **kwargs)
    287
    288         if is_datasets_available():
--> 289             if isinstance(train_dataset, datasets.Dataset):
    290                 self._remove_unused_columns(self.train_dataset, description="training")
    291             if isinstance(eval_dataset, datasets.Dataset):

AttributeError: module 'datasets' has no attribute 'Dataset'
```
My guess is that `datasets.Dataset` should be replaced by `torch.utils.data.Dataset` but I haven't checked the source file. Maybe the person responsible for the `Trainer` development should look into that.

I'm using transformers version 3.2.0 btw.


Error when fine-tune RoBERTa on NSP using Trainer

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-45-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: IDK.
- Using distributed or parallel set-up in script?: IDK.

### Who can help

 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 nlp datasets: [different repo](https://github.com/huggingface/nlp)

## Information

Model I am using RoBERTa trained for Polish lanugage: [polish-roberta](https://github.com/sdadas/polish-roberta), version [robreta_base_transformers](https://github.com/sdadas/polish-roberta/releases/download/models/roberta_base_transformers.zip).

The problem arises when using:
* [ ] my own modified scripts:

```python
from transformers import (BertForNextSentencePrediction,
                          BertTokenizer,
                          RobertaModel, RobertaTokenizer, Trainer,
                          TrainingArguments)
from transformers.data.datasets.language_modeling import TextDatasetForNextSentencePrediction
from transformers.data.data_collator import DataCollatorForNextSentencePrediction
from argparse import ArgumentParser

def parse_args():
    parser = ArgumentParser("Fine-tune RoBERTa in Next Sentence Prediction.")
    parser.add_argument("-m", "--model_path", dest="model_path", required=True, help="Path to RoBERTa model.")
    parser.add_argument("-o", "--output_path", dest="output_path", required=True, help="Path to directory of fine-tuned model.")
    parser.add_argument("-d", "--dataset_path", dest="dataset_path", required=True, help="Path to dataset.")
    args = parser.parse_args()
    return args

if __name__ == "__main__":
    args = parse_args()
    tokenizer = RobertaTokenizer.from_pretrained(args.model_path)
    finetune_model = BertForNextSentencePrediction.from_pretrained(args.model_path)

    training_args = TrainingArguments(
        output_dir=args.output_path,
        num_train_epochs=3,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        warmup_steps=500,
        weight_decay=0.01,
        logging_dir='./logs',
    )

    data_collator = DataCollatorForNextSentencePrediction(
        tokenizer=tokenizer,
        mlm=False,
        block_size=512,
        nsp_probability=0.5,
      )

    train_dataset = TextDatasetForNextSentencePrediction(
        tokenizer=tokenizer,
        file_path=args.dataset_path,
        block_size=512,
    )

    trainer = Trainer(
        model=finetune_model,
        args=training_args,
        train_dataset=train_dataset,
        data_collator=data_collator,
    )

    trainer.train()
    trainer.save_model(args.output_path)

```

The tasks I am working on is:
* [ ] my own task or dataset based on TextDatasetForNextSentencePrediction input format:
```bash
<doc1_turn1>
<doc1_turn2>

<doc2_turn1>
<doc2_turn2>
...
```

## To reproduce

Steps to reproduce the behavior:

1. `python finetune_roberta.py -m <model_dir> -o output/ -d <dataset_path>`

```bash
Special tokens have been added in the vocabulary, make sure the associated word emebedding are fine-tuned or trained.
Some weights of the model checkpoint at roberta_base/ were not used when initializing BertForNextSentencePrediction: ['roberta.embeddings.word_embeddings.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.embeddings.token_type_embeddings.weight', 'roberta.embeddings.LayerNorm.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.0.attention.self.query.weight', 'roberta.encoder.layer.0.attention.self.query.bias', 'roberta.encoder.layer.0.attention.self.key.weight', 'roberta.encoder.layer.0.attention.self.key.bias', 'roberta.encoder.layer.0.attention.self.value.weight', 'roberta.encoder.layer.0.attention.self.value.bias', 'roberta.encoder.layer.0.attention.output.dense.weight', 'roberta.encoder.layer.0.attention.output.dense.bias', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.intermediate.dense.weight', 'roberta.encoder.layer.0.intermediate.dense.bias', 'roberta.encoder.layer.0.output.dense.weight', 'roberta.encoder.layer.0.output.dense.bias', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.self.query.weight', 'roberta.encoder.layer.1.attention.self.query.bias', 'roberta.encoder.layer.1.attention.self.key.weight', 'roberta.encoder.layer.1.attention.self.key.bias', 'roberta.encoder.layer.1.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.value.bias', 'roberta.encoder.layer.1.attention.output.dense.weight', 'roberta.encoder.layer.1.attention.output.dense.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.intermediate.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.bias', 'roberta.encoder.layer.1.output.dense.weight', 'roberta.encoder.layer.1.output.dense.bias', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.2.attention.self.query.weight', 'roberta.encoder.layer.2.attention.self.query.bias', 'roberta.encoder.layer.2.attention.self.key.weight', 'roberta.encoder.layer.2.attention.self.key.bias', 'roberta.encoder.layer.2.attention.self.value.weight', 'roberta.encoder.layer.2.attention.self.value.bias', 'roberta.encoder.layer.2.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.output.dense.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.2.intermediate.dense.weight', 'roberta.encoder.layer.2.intermediate.dense.bias', 'roberta.encoder.layer.2.output.dense.weight', 'roberta.encoder.layer.2.output.dense.bias', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.query.weight', 'roberta.encoder.layer.3.attention.self.query.bias', 'roberta.encoder.layer.3.attention.self.key.weight', 'roberta.encoder.layer.3.attention.self.key.bias', 'roberta.encoder.layer.3.attention.self.value.weight', 'roberta.encoder.layer.3.attention.self.value.bias', 'roberta.encoder.layer.3.attention.output.dense.weight', 'roberta.encoder.layer.3.attention.output.dense.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.intermediate.dense.weight', 'roberta.encoder.layer.3.intermediate.dense.bias', 'roberta.encoder.layer.3.output.dense.weight', 'roberta.encoder.layer.3.output.dense.bias', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.self.query.weight', 'roberta.encoder.layer.4.attention.self.query.bias', 'roberta.encoder.layer.4.attention.self.key.weight', 'roberta.encoder.layer.4.attention.self.key.bias', 'roberta.encoder.layer.4.attention.self.value.weight', 'roberta.encoder.layer.4.attention.self.value.bias', 'roberta.encoder.layer.4.attention.output.dense.weight', 'roberta.encoder.layer.4.attention.output.dense.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.intermediate.dense.weight', 'roberta.encoder.layer.4.intermediate.dense.bias', 'roberta.encoder.layer.4.output.dense.weight', 'roberta.encoder.layer.4.output.dense.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.query.weight', 'roberta.encoder.layer.5.attention.self.query.bias', 'roberta.encoder.layer.5.attention.self.key.weight', 'roberta.encoder.layer.5.attention.self.key.bias', 'roberta.encoder.layer.5.attention.self.value.weight', 'roberta.encoder.layer.5.attention.self.value.bias', 'roberta.encoder.layer.5.attention.output.dense.weight', 'roberta.encoder.layer.5.attention.output.dense.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.intermediate.dense.weight', 'roberta.encoder.layer.5.intermediate.dense.bias', 'roberta.encoder.layer.5.output.dense.weight', 'roberta.encoder.layer.5.output.dense.bias', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.query.bias', 'roberta.encoder.layer.6.attention.self.key.weight', 'roberta.encoder.layer.6.attention.self.key.bias', 'roberta.encoder.layer.6.attention.self.value.weight', 'roberta.encoder.layer.6.attention.self.value.bias', 'roberta.encoder.layer.6.attention.output.dense.weight', 'roberta.encoder.layer.6.attention.output.dense.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.encoder.layer.6.intermediate.dense.weight', 'roberta.encoder.layer.6.intermediate.dense.bias', 'roberta.encoder.layer.6.output.dense.weight', 'roberta.encoder.layer.6.output.dense.bias', 'roberta.encoder.layer.6.output.LayerNorm.weight', 'roberta.encoder.layer.6.output.LayerNorm.bias', 'roberta.encoder.layer.7.attention.self.query.weight', 'roberta.encoder.layer.7.attention.self.query.bias', 'roberta.encoder.layer.7.attention.self.key.weight', 'roberta.encoder.layer.7.attention.self.key.bias', 'roberta.encoder.layer.7.attention.self.value.weight', 'roberta.encoder.layer.7.attention.self.value.bias', 'roberta.encoder.layer.7.attention.output.dense.weight', 'roberta.encoder.layer.7.attention.output.dense.bias', 'roberta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.encoder.layer.7.intermediate.dense.weight', 'roberta.encoder.layer.7.intermediate.dense.bias', 'roberta.encoder.layer.7.output.dense.weight', 'roberta.encoder.layer.7.output.dense.bias', 'roberta.encoder.layer.7.output.LayerNorm.weight', 'roberta.encoder.layer.7.output.LayerNorm.bias', 'roberta.encoder.layer.8.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.query.bias', 'roberta.encoder.layer.8.attention.self.key.weight', 'roberta.encoder.layer.8.attention.self.key.bias', 'roberta.encoder.layer.8.attention.self.value.weight', 'roberta.encoder.layer.8.attention.self.value.bias', 'roberta.encoder.layer.8.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.output.dense.bias', 'roberta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.encoder.layer.8.intermediate.dense.weight', 'roberta.encoder.layer.8.intermediate.dense.bias', 'roberta.encoder.layer.8.output.dense.weight', 'roberta.encoder.layer.8.output.dense.bias', 'roberta.encoder.layer.8.output.LayerNorm.weight', 'roberta.encoder.layer.8.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.query.weight', 'roberta.encoder.layer.9.attention.self.query.bias', 'roberta.encoder.layer.9.attention.self.key.weight', 'roberta.encoder.layer.9.attention.self.key.bias', 'roberta.encoder.layer.9.attention.self.value.weight', 'roberta.encoder.layer.9.attention.self.value.bias', 'roberta.encoder.layer.9.attention.output.dense.weight', 'roberta.encoder.layer.9.attention.output.dense.bias', 'roberta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.intermediate.dense.weight', 'roberta.encoder.layer.9.intermediate.dense.bias', 'roberta.encoder.layer.9.output.dense.weight', 'roberta.encoder.layer.9.output.dense.bias', 'roberta.encoder.layer.9.output.LayerNorm.weight', 'roberta.encoder.layer.9.output.LayerNorm.bias', 'roberta.encoder.layer.10.attention.self.query.weight', 'roberta.encoder.layer.10.attention.self.query.bias', 'roberta.encoder.layer.10.attention.self.key.weight', 'roberta.encoder.layer.10.attention.self.key.bias', 'roberta.encoder.layer.10.attention.self.value.weight', 'roberta.encoder.layer.10.attention.self.value.bias', 'roberta.encoder.layer.10.attention.output.dense.weight', 'roberta.encoder.layer.10.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.encoder.layer.10.intermediate.dense.weight', 'roberta.encoder.layer.10.intermediate.dense.bias', 'roberta.encoder.layer.10.output.dense.weight', 'roberta.encoder.layer.10.output.dense.bias', 'roberta.encoder.layer.10.output.LayerNorm.weight', 'roberta.encoder.layer.10.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.self.query.weight', 'roberta.encoder.layer.11.attention.self.query.bias', 'roberta.encoder.layer.11.attention.self.key.weight', 'roberta.encoder.layer.11.attention.self.key.bias', 'roberta.encoder.layer.11.attention.self.value.weight', 'roberta.encoder.layer.11.attention.self.value.bias', 'roberta.encoder.layer.11.attention.output.dense.weight', 'roberta.encoder.layer.11.attention.output.dense.bias', 'roberta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.encoder.layer.11.intermediate.dense.weight', 'roberta.encoder.layer.11.intermediate.dense.bias', 'roberta.encoder.layer.11.output.dense.weight', 'roberta.encoder.layer.11.output.dense.bias', 'roberta.encoder.layer.11.output.LayerNorm.weight', 'roberta.encoder.layer.11.output.LayerNorm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
- This IS expected if you are initializing BertForNextSentencePrediction from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForNextSentencePrediction from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForNextSentencePrediction were not initialized from the model checkpoint at roberta_base/ and are newly initialized: ['embeddings.word_embeddings.weight', 'embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.LayerNorm.weight', 'embeddings.LayerNorm.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.9.output.dense.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.10.attention.self.value.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.11.output.dense.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.11.output.LayerNorm.bias', 'pooler.dense.weight', 'pooler.dense.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Epoch:   0%|                                                                                | 0/3 [00:00<?, ?it/s/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
                                                                                {'loss': 0.676176025390625, 'learning_rate': 5e-05, 'epoch': 0.3427004797806717, 'step': 500}                                                                        | 499/1459 [04:30<08:09,  1.96it/s]
/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:200: UserWarning: Please also save or load the state of the optimzer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)
                                                                                {'loss': 0.671025390625, 'learning_rate': 4.355171524374517e-05, 'epoch': 0.6854009595613434, 'step': 1000}███████████▎                                              | 999/1459 [08:47<03:53,  1.97it/s]
                                                                                Traceback (most recent call last):███████████████████████████████████████████████████████████████████████████████████████                                           | 1033/1459 [09:06<03:38,  1.95it/s]
  File "finetune_roberta.py", line 75, in <module>
    trainer.train()
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/transformers/trainer.py", line 699, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/transformers/data/data_collator.py", line 358, in __call__
    input_id, segment_id, attention_mask, label = self.create_examples_from_document(doc, i, examples)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/transformers/data/data_collator.py", line 446, in create_examples_from_document
    random_start = random.randint(0, len(random_document) - 1)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/random.py", line 248, in randint
    return self.randrange(a, b+1)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/random.py", line 226, in randrange
    raise ValueError("empty range for randrange() (%d, %d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0, 0, 0)
Epoch:   0%|                                                                                | 0/3 [09:09<?, ?it/s]
Iteration:  71%|████████████████████████████████████████████████████████████████████████████████████████████████████████                                           | 1033/1459 [09:09<03:46,  1.88it/s]
```
## Expected behavior

Model is fine-tuned on NSP taks on given dataset and after that model is saved.


is there a tokenizer only used whitespace for spliting chinese sentence?

i want to use bert masked language model to pre train for chinese sentence,  i have alweady splited chinese sentence into meaninful word, the data file like follows:

我  是  一个  队员
他  不是  一个 合格  的 老师
......

i only want to use whitespace split them, but BertWordPieceTokenizer will split them to character level , the final vocabulary like follows:
{'[SEP]': 3,
 '一': 7,
 '是': 15,
 '员': 12,
 '[CLS]': 2,
 '[UNK]': 1,
 '</S>': 6,
 '[MASK]': 4,
 '<S>': 5,
 '队': 19,
 '的': 17,
 '不': 8,
 '我': 14,
 '他': 10,
 '老': 18,
 '[PAD]': 0,
 '格': 16,
 '个': 9,
 '师': 13,
 '合': 11}
how to correct it ?


[testing] skip decorators: docs, tests, bugs

This PR:

* fixes a bug in `require_torch_and_cuda`
* makes all skip decorators consistent code-wise
* adds a test for testing combinations of skip decorators and other decorators
* clarifies `testing.rst` notes

OK, so other than a small bug in  `require_torch_and_cuda` our skip decorators can be used in any order.

The only problem I found so far is when they are used together with `@parameterized`, which has to come first and skip decorators last. It rewrites test names, to create a unique test name for each parameter group. and then it runs them - it has no idea it may have any skip decorators before it (The decorators get all stacked, and one below has no idea what the one above does).

If you find other unusual decorators, please let me know and I will investigate.

<!-- This line specifies which issue to close after the pull request is merged. -->
Partially fixes #7326

@LysandreJik, @sgugger


Cannot import transformers with TF version 2.1.0

Should Fix

This has been identified as a bug and should be fixed.

TensorFlow

Anything TensorFlow

The installation README says that transformers library requires Tensorflow version >2.0, but I can't seem to import the latest transformers 3.2 release even with Tensorflow v2.1.

```
>>> import transformers
wandb: WARNING W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/amog/dev/ray/lib/python3.7/site-packages/transformers/__init__.py", line 121, in <module>
    from .pipelines import (
  File "/Users/amog/dev/ray/lib/python3.7/site-packages/transformers/pipelines.py", line 47, in <module>
    from .modeling_tf_auto import (
  File "/Users/amog/dev/ray/lib/python3.7/site-packages/transformers/modeling_tf_auto.py", line 45, in <module>
    from .modeling_tf_albert import (
  File "/Users/amog/dev/ray/lib/python3.7/site-packages/transformers/modeling_tf_albert.py", line 24, in <module>
    from .activations_tf import get_tf_activation
  File "/Users/amog/dev/ray/lib/python3.7/site-packages/transformers/activations_tf.py", line 53, in <module>
    "swish": tf.keras.activations.swish,
AttributeError: module 'tensorflow_core.python.keras.api._v2.keras.activations' has no attribute 'swish'
```

Upgrading to TF 2.2 works fine, but I think this should be made more clear in the docs.

cc @jplu @sgugger
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.2.0
- Platform: Mac OS
- Python version: 3.7.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.1.0. On CPU only.
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


data_collator error: AttributeError: 'dict' object has no attribute 'size'

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

I am trying to run a language model that is very similar to the [tutorial ](https://huggingface.co/blog/how-to-train). I have a custom dataset class that returns a dict with fields: dict_keys(['input_ids', 'token_type_ids', 'attention_mask']). When I run the training I get this error message:
```  File "prod2vec/train-from-scratch.py", line 289, in <module>
    sys.exit(main())
  File "prod2vec/train-from-scratch.py", line 265, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/site-packages/transformers/trainer.py", line 456, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.6/site-packages/tqdm/std.py", line 1127, in __iter__
    for obj in iterable:
  File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 35, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.6/site-packages/transformers/data/data_collator.py", line 79, in __call__
    batch = self._tensorize_batch(examples)
  File "/usr/local/lib/python3.6/site-packages/transformers/data/data_collator.py", line 91, in _tensorize_batch
    length_of_first = examples[0].size(0)
AttributeError: 'dict' object has no attribute 'size'
```

The error message is not surprising as examples[0] is a dictionary with previously mentioned three fields. I am curious what and where i am doing a mistake.


[s2s] only save metrics.json from rank zero

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Ensure that integrations are imported before transformers or ml libs

This PR fixes a problem with some 3rd-party integrations that need to be imported before any transformers or other machine learning framework Python modules.

This PR makes the following changes:

1. Moves `import .integrations` in `__init__.py` before any other transformers imports
2. Moves ML imports in .integrations below 3rd-party imports
3. Used math.ceil() rather than numpy.ceil() as that was overkill

Before PR:

* failed with comet_ml

After PR:

* works with comet_ml


Problem loading a dynamic quantized distilbert model.

Hello and thanks for your awesome library,
## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-117-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@VictorSanh
@stefan-it

## Information
I'm trying to optimize a fine-tuned (for token classification, NER) distilBert model through Dynamic Quantization.
I use this line:
```python
quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
```
The model size goes from: 540 MB to 411 MB.

The quantized model works fine when I use it straight away in the script to make predictions, however I'm having trouble saving it and reloading it.
I tried few things, first using save_pretrained:
```python
quantized_model.save_pretrained(quantized_output_dir)
```
And then loading it using:
```python
model = AutoModelForTokenClassification.from_pretrained(quantized_output_dir)
```
When I use it to make predictions, I get the warning:
``
Some weights of the model checkpoint at data/model3/quantized3/ were not used when initializing DistilBertForTokenClassification: ['distilbert.transformer.layer.0.attention.q_lin.scale', 'distilbert.transformer.layer.0.attention.q_lin.zero_point', 'distilbert.transformer.layer.0.attention.q_lin._packed_params.dtype', 'distilbert.transformer.layer.0.attention.q_lin._packed_params._packed_params', 'distilbert.transformer.layer.0.attention.k_lin.scale', 'distilbert.transformer.layer.0.attention.k_lin.zero_point', 'distilbert.transformer.layer.0.attention.k_lin._packed_params.dtype', 'distilbert.transformer.layer.0.attention.k_lin._packed_params._packed_params', 'distilbert.transformer.layer.0.attention.v_lin.scale', 'distilbert.transformer.layer.0.attention.v_lin.zero_point', 'distilbert.transformer.layer.0.attention.v_lin._packed_params.dtype', 'distilbert.transformer.layer.0.attention.v_lin._packed_params._packed_params', 'distilbert.transformer.layer.0.attention.out_lin.scale', 'distilbert.transformer.layer.0.attention.out_lin.zero_point', 'distilbert.transformer.layer.0.attention.out_lin._packed_params.dtype', 'distilbert.transformer.layer.0.attention.out_lin._packed_params._packed_params', 'distilbert.transformer.layer.0.ffn.lin1.scale', 'distilbert.transformer.layer.0.ffn.lin1.zero_point', 'distilbert.transformer.layer.0.ffn.lin1._packed_params.dtype', 'distilbert.transformer.layer.0.ffn.lin1._packed_params._packed_params', 'distilbert.transformer.layer.0.ffn.lin2.scale', 'distilbert.transformer.layer.0.ffn.lin2.zero_point', 'distilbert.transformer.layer.0.ffn.lin2._packed_params.dtype', 'distilbert.transformer.layer.0.ffn.lin2._packed_params._packed_params', 'distilbert.transformer.layer.1.attention.q_lin.scale',
``

For all the layers.
And of course I got wrong predictions because it's as if the model isn't fine-tuned.

I tried saving it using:
```python
torch.save(quantized_model.state_dict(), path)
```
loading it using:
 ```python
config = DistilBertConfig.from_pretrained("distilbert-base-multilingual-cased", num_labels=5)
model = DistilBertForTokenClassification.from_pretrained("distilbert-base-multilingual-cased", config=config)
model.load_state_dict(torch.load(path))
```
and I got this runtime error:
``
RuntimeError: Error(s) in loading state_dict for DistilBertForTokenClassification:
	Missing key(s) in state_dict: "distilbert.transformer.layer.0.attention.q_lin.weight", "distilbert.transformer.layer.0.attention.q_lin.bias", "distilbert.transformer.layer.0.attention.k_lin.weight", "distilbert.transformer.layer.0.attention.k_lin.bias", "distilbert.transformer.layer.0.attention.v_lin.weight", "distilbert.transformer.layer.0.attention.v_lin.bias", "distilbert.transformer.layer.0.attention.out_lin.weight", "distilbert.transformer.layer.0.attention.out_lin.bias", "distilbert.transformer.layer.0.ffn.lin1.weight", "distilbert.transformer.layer.0.ffn.lin1.bias", "distilbert.transformer.layer.0.ffn.lin2.weight", "distilbert.transformer.layer.0.ffn.lin2.bias", "distilbert.transformer.layer.1.attention.q_lin.weight",
	Unexpected key(s) in state_dict: "distilbert.transformer.layer.0.attention.q_lin.scale", "distilbert.transformer.layer.0.attention.q_lin.zero_point", "distilbert.transformer.layer.0.attention.q_lin._packed_params.dtype", "distilbert.transformer.layer.0.attention.q_lin._packed_params._packed_params", "distilbert.transformer.layer.0.attention.k_lin.scale", "distilbert.transformer.layer.0.attention.k_lin.zero_point", "distilbert.transformer.layer.0.attention.k_lin._packed_params.dtype", "distilbert.transformer.layer.0.attention.k_lin._packed_params._packed_params", "distilbert.transformer.layer.0.attention.v_lin.scale", "distilbert.transformer.layer.0.attention.v_lin.zero_point", "distilbert.transformer.layer.0.attention.v_lin._packed_params.dtype", "distilbert.transformer.layer.0.attention.v_lin._packed_params._packed_params", "distilbert.transformer.layer.0.attention.out_lin.scale", "distilbert.transformer.layer.0.attention.out_lin.zero_point", "distilbert.transformer.layer.0.attention.out_lin._packed_params.dtype", "distilbert.transformer.layer.0.attention.out_lin._packed_params._packed_params", "distilbert.transformer.layer.0.ffn.lin1.scale", "distilbert.transformer.layer.0.ffn.lin1.zero_point", "distilbert.transformer.layer.0.ffn.lin1._packed_params.dtype", "distilbert.transformer.layer.0.ffn.lin1._packed_params._packed_params", "distilbert.transformer.layer.0.ffn.lin2.scale", "distilbert.transformer.layer.0.ffn.lin2.zero_point", "distilbert.transformer.layer.0.ffn.lin2._packed_params.dtype", "distilbert.transformer.layer.0.ffn.lin2._packed_params._packed_params", "distilbert.transformer.layer.1.attention.q_lin.scale", "classifier._packed_params.dtype", "classifier._packed_params._packed_params".
``
For all the layers also (didn't put it all to shorten the text).

Here is the text when printing the quantized model:
```JSON
DistilBertForTokenClassification(
  (distilbert): DistilBertModel(
    (embeddings): Embeddings(
      (word_embeddings): Embedding(119547, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (transformer): Transformer(
      (layer): ModuleList(
        (0): TransformerBlock(
          (attention): MultiHeadSelfAttention(
            (dropout): Dropout(p=0.1, inplace=False)
            (q_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (k_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (v_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (out_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (ffn): FFN(
            (dropout): Dropout(p=0.1, inplace=False)
            (lin1): DynamicQuantizedLinear(in_features=768, out_features=3072, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (lin2): DynamicQuantizedLinear(in_features=3072, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        )
        (1): TransformerBlock(
          (attention): MultiHeadSelfAttention(
            (dropout): Dropout(p=0.1, inplace=False)
            (q_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (k_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (v_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (out_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (ffn): FFN(
            (dropout): Dropout(p=0.1, inplace=False)
            (lin1): DynamicQuantizedLinear(in_features=768, out_features=3072, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (lin2): DynamicQuantizedLinear(in_features=3072, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        )
        (2): TransformerBlock(
          (attention): MultiHeadSelfAttention(
            (dropout): Dropout(p=0.1, inplace=False)
            (q_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (k_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (v_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (out_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (ffn): FFN(
            (dropout): Dropout(p=0.1, inplace=False)
            (lin1): DynamicQuantizedLinear(in_features=768, out_features=3072, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (lin2): DynamicQuantizedLinear(in_features=3072, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        )
        (3): TransformerBlock(
          (attention): MultiHeadSelfAttention(
            (dropout): Dropout(p=0.1, inplace=False)
            (q_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (k_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (v_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (out_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (ffn): FFN(
            (dropout): Dropout(p=0.1, inplace=False)
            (lin1): DynamicQuantizedLinear(in_features=768, out_features=3072, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (lin2): DynamicQuantizedLinear(in_features=3072, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        )
        (4): TransformerBlock(
          (attention): MultiHeadSelfAttention(
            (dropout): Dropout(p=0.1, inplace=False)
            (q_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (k_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (v_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (out_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (ffn): FFN(
            (dropout): Dropout(p=0.1, inplace=False)
            (lin1): DynamicQuantizedLinear(in_features=768, out_features=3072, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (lin2): DynamicQuantizedLinear(in_features=3072, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        )
        (5): TransformerBlock(
          (attention): MultiHeadSelfAttention(
            (dropout): Dropout(p=0.1, inplace=False)
            (q_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (k_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (v_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (out_lin): DynamicQuantizedLinear(in_features=768, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (ffn): FFN(
            (dropout): Dropout(p=0.1, inplace=False)
            (lin1): DynamicQuantizedLinear(in_features=768, out_features=3072, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
            (lin2): DynamicQuantizedLinear(in_features=3072, out_features=768, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
          )
          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        )
      )
    )
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): DynamicQuantizedLinear(in_features=768, out_features=5, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
)
```
## Expected behavior
You can successfully load the quantized fine-tuned model to make predictions.
Can be the "DynamicQuantizedLinear" instead of "Linear" be causing this problem ?

Thanks in advance for your help.


Add PRADO model

# 🌟 New model addition

## Model description
PRADO is a model made by google, performing as bert with 100x less parameters

[link to the paper](https://www.aclweb.org/anthology/D19-1506.pdf)
[git to the model code](https://github.com/tensorflow/models/tree/master/research/sequence_projection)

## Open source status

* [X] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [X] who are the authors: (mention them, if possible by @gh-username)
Prabhu Kaliamoorthi / Sujith Ravi / Zornitsa Kozareva


PegasusTokenizer: Newline symbol

Core: Tokenization

Internals of the library; Tokenization.

Ported models generate the `<n>` token at the beginning of sentences, whereas ours do not. The pegasus [original code](https://github.com/google-research/pegasus/blob/master/pegasus/ops/public_parsing_ops.py#L40) replaces `\n` newline symbol with `<n>`. `PegasusTokenizer` should probably do this.

```python
_NEWLINE_SYMBOL = "<n>"
text = tf.strings.regex_replace(text, "\n", _NEWLINE_SYMBOL)
```


Check decorator order

As @stas00 pointed out, the slow decorator is ignore if it's not put last. To make sure we don't make the mistake unintentionally and to fix the places where this si not the case, I wrote a script to check the decorators order and fail on `make quality` if there is a wrong order somewhere.


Mark big downloads slow

This PR adds the slow decorator for models we don't want to download at each CI run.


[s2s] Marian beam search slow for en-de

translation

machine translation utilities and models

Hey @sshleifer (tagging because of translation) - I'm not sure whether I am misunderstanding something or this is an actual issue so apologies, but it seems like validation/eval is significantly slower than training and is a serious bottleneck when fine-tuning translation models.

I am running on Colab with V100, and trying to finetune a MarianMT model on a data set of ~10k sentences of length of up to 300, training on 90% of the data takes about 2 minutes per epoch whereas validation/eval on the remaining 10% of the data takes about 6-7 minutes without any fancy eval flags, no beamsearch etc. This results in fine-tuning being ~5x slower if performing validation every epoch (and still significantly slower even if only performing partial validation or every other epoch etc).

I am using apex and pytorch 1.5.1 as instructed in the readme and in the issues regarding apex fp16 training and bs=16 for both train and validation, different batch sizes did not seem to help. Happy to post more info but the rest is pretty similar to the seq2seq examples.


T5 Cross-attention Decoder - Possible bug with relative_bias

@patrickvonplaten - investigate whether this is a possible bug: https://github.com/google-research/text-to-text-transfer-transformer/issues/415


Add num workers cli arg

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6316


Example Format of Data for token classification

Hi!! I'd like to train the token classification model but I don't know what is the right format of data for token classification training. Thank you.


Test CI with higher timeout

Test.


[Bug Fix] Fix run_squad.py evaluation code doesn't use probabilities

Modification of run_squad.py fine tuning example so it will use the answer correctness probabilities the models produce while evaluating the model and calculating the best thresholds.

Evaluation was done without the evaluated model probabilities but rather with default zero values. It corrupted the evaluation results and the best thresholds (which evaluated always as 0.0).

**Notice: many squad models were evaluated without the probabilities, therefore, the results published in their model cards are possibly wrong.**

Example: [ahotrod/electra_large_discriminator_squad2_512](https://huggingface.co/ahotrod/electra_large_discriminator_squad2_512)

The results with current evaluation script:
```
"exact": 87.09677419354838,
  "f1": 89.98343832723452,
  "total": 11873,
  "HasAns_exact": 84.66599190283401,
  "HasAns_f1": 90.44759839056285,
  "HasAns_total": 5928,
  "NoAns_exact": 89.52060555088309,
  "NoAns_f1": 89.52060555088309,
  "NoAns_total": 5945,
  "best_exact": 87.09677419354838,
  "best_exact_thresh": 0.0,
  "best_f1": 89.98343832723432,
  "best_f1_thresh": 0.0
```

The results after the fix:
```
'exact': 87.00412701086499,
 'f1': 89.77725380276271,
 'total': 11873,
 'HasAns_exact': 83.80566801619433,
 'HasAns_f1': 89.35987422405582,
 'HasAns_total': 5928,
 'NoAns_exact': 90.19343986543313,
 'NoAns_f1': 90.19343986543313,
 'NoAns_total': 5945,
 'best_exact': 87.34102585698643,
'best_exact_thresh': 0.09882385462915344,
 'best_f1': 90.07804792988485,
'best_f1_thresh': 0.09882385462915344
```


Fixes for LayoutLM

Adds the commands from the new script to check for model copies and clean up a bit the docstrings.


Create README.md

model card

Related to pretrained model cards

<!-- add model card to blinoff/roberta-base-russian-v0 -->


Support for Windows in check_copies

This is (hopefully) all what is necessary to make the script `check_copies.py` work on Windows.
@jplu if you can checkout this PR locally and confirm, that would be great!


Memory leak

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.7.0
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed

### Who can help
@LysandreJik, @sgugger, @patrickvonplaten

## Information

Model I am using (Bert, GPT2):

The problem arises when using:
* [ X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X ] my own task or dataset: (give details below)

## To reproduce
When I pretrain or fine tune a model (in my case BERT and GPT2) using torch.distributed.launch, the CPU memory usage will grow up to the memory limit (>500GB) until the first process is killed due to this issue.  If I train bert-base, it takes around 30 epochs until the first process is killed, but when I train gpt-large, it just need 3 epochs until it is killed. Following is the command line I run to train/fine tune the bert-base (similar with gpt2). The script run_language_modeling.py is a copy of transformers/examples/language-modeling/run_language_modeling.py (vers. 3.1.0)

python -m torch.distributed.launch --nproc_per_node=8 \
  ../run_language_modeling.py \
  --output_dir $model_target \
  --model_name_or_path $model_source \
  --config_name $model_source \
  --tokenizer_name $model_source \
  --train_data_file $target_train \
  --eval_data_file $target_test \
  --save_total_limit 5 \
  --block_size 128 \
  --overwrite_output_dir \
  --fp16 \
  --num_train_epochs 50 \
  --do_train --do_eval \
  --per_device_train_batch_size 32 \
  --per_device_eval_batch_size 4 \
  --mlm

## Expected behavior
I would expect that the distributed training run until it is done without any memory issue.
Thanks for checking it.


Text generation with xlnet

how to use xlnet for text generation?


Fixed results of SQuAD-FR evaluation

Related to pretrained model cards

The score for the F1 metric was reported as the Exact Match and vice-versa.

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Adds FSMT to LM head AutoModel

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Create an XLA parameter and fix the mixed precision

This PR adds a new `XLA` parameter to activate/deactivate the XLA compilation and a bug in the mixed precision. These have to be set before the creation of the strategy and float16 is not compliant with TPU, where only bfloat16 is available.


[code quality] new make target that combines style and quality targets

**edit**: this post has been edited to reflect the outcome of the discussion.

Any reason why we don't run `flake8` in `make style`? I find myself needing to run `make style` followed by `make quality` all the time, but I need the latter just for the last 2 checks. Since we have no control over the source code why bother with separating checking and fixing - let's just have one target that fixes and then performs the remaining checks, as we know the first two have been done already.

This PR suggests to create a new target `fixup` that combines the 2 separate fix and check functions into one efficient target,

I will edit the docs if this change resonates with the team.

p.s. if it feels wrong to merge fixing and checking, can we add a 3rd target that is a merged one? `make best`

p.p.s. I know I can make my own alias, I love `make`!


[code quality] fix confused flake8

We run `black  --target-version py35 ...` but flake8 doesn't know that we want this specific `target-version`, so currently with py38 flake8 fails suggesting that black should have reformatted 63 files. Indeed if I run:

```
black --line-length 119 --target-version py38 examples templates tests src utils
```
it indeed reformats 63 files.

The only solution I found is to create a black config file as explained at https://github.com/psf/black#configuration-format, which is what this PR adds.

Now flake8 knows that py35 is the standard and no longer gets confused regardless of the user's python version.

We can now edit out `--line-length 119 --target-version py35` from Makefile and the CI jobs, so that we have one config to rule them all. I pushed that change as well.

@sgugger, @LysandreJik


[s2s] metrics.json is wrong on multigpu

overwritten by last rank to save it.

@nateraw is there a way to check if my module `is_rank_zero` or some such?


Cuda OOM training gpt2-xl with Trainer in multi-GPUs

# ❓ Questions & Help

I am currently trying to finetune the gpt2-xl. I have 2 tesla T4 GPUs. However, I get the CUDA OOM error... when I look at the use of the gpus I see that the first one is full, but the second one still has enough space. Here is my code:

```
from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

from transformers import GPT2LMHeadModel, Trainer, TrainingArguments
model = GPT2LMHeadModel.from_pretrained("gpt2-xl").to(device)

from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer,  TrainingArguments, Trainer
tokenizer = AutoTokenizer.from_pretrained("gpt2-xl")
train_dataset = TextDataset(
          tokenizer=tokenizer,
          file_path='dataset_training.txt',
          block_size=128)

data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=False,
    )

training_args = TrainingArguments(
    output_dir='./results',         # output directory
    num_train_epochs=2,              # total # of training epochs
    per_device_train_batch_size=1,  # batch size per device during training
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',

)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    prediction_loss_only=True,
)

trainer.train()
```
I get "CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 14.73 GiB total capacity; 13.61 GiB already allocated; 31.88 MiB free; 13.98 GiB reserved in total by PyTorch)"

When I run nvidia-smi I see:

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+================
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   75C    P0    34W /  70W |  15047MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   56C    P0    29W /  70W |   9479MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|===============================================
|    0      1756      C   /opt/conda/bin/python                      15037MiB |
|    1      1756      C   /opt/conda/bin/python                       9469MiB |
+-----------------------------------------------------------------------------+
```
**My question is:** Am I making a mistake? or how can a large model be trained with multiple GPUs?


BertModel for 2 category classification - How to evaluate the performance

Hello there, I am building a fine-tuned BERT model for classification (with a linear layer in the end). The prediction should just be 1/0 (Yes, No).

When I am writing the evaluation part, I saw some people online did a F.log_softmax for the logits then use np.argmax to get the predicted label. However, I also saw people directly apply np.argmax on logits without the softmax activation. I am wondering which one should I follow and how to decide that?

Here's my model definition:
```python
class ReviewClassification(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = 2

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

        embedding_size = config.hidden_size

        self.classifier = nn.Linear(embedding_size, len(LABEL_NAME))
        self.init_weights()

    def forward(
            self,
            review_input_ids=None,
            review_attention_mask=None,
            review_token_type_ids=None,
            agent_input_ids=None,
            agent_attention_mask=None,
            agent_token_type_ids=None,
            labels=None,
    ):

        review_outputs = self.bert(
            review_input_ids,
            attention_mask=review_attention_mask,
            token_type_ids=review_token_type_ids,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
        )

        feature = review_outputs[1]  # (batch_size, seq_len) -? Should it be (batch_size, hidden_size)

        # nn.CrossEntropyLoss applies F.log_softmax and nn.NLLLoss internally on your input,
        # so you should pass the raw logits to it.
        logits = self.classifier(feature)

        outputs = (logits,)  # + outputs[2:]  # add hidden states and attention if they are here

        if labels is not None:
            loss_fct = nn.CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            outputs = (loss,) + outputs

        return outputs  # (loss, logits, hidden_states, attentions)
```

Then this is my validation code
```
def model_validate(model, data_loader):
    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    label_prop = data_loader.dataset.dataset.label_prop()

    total_valid_loss = 0

    batch_size = data_loader.batch_size
    num_batch = len(data_loader)

    y_pred, y_true = [], []

    # Evaluate data
    for step, batch in tqdm(enumerate(data_loader), desc="Validation...", total=num_batch):
        b_review_input_ids = batch["review_input_ids"].to(device)
        b_review_attention_mask = batch["review_attention_mask"].to(device)
        b_review_token_type_ids = batch["review_token_type_ids"].to(device)

        b_binarized_label = batch["binarized_label"].to(device)

        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():
            (loss, logits,) = model(review_input_ids=b_review_input_ids,
                                    review_attention_mask=b_review_attention_mask,
                                    review_token_type_ids=b_review_token_type_ids,

                                    labels=b_binarized_label)

        total_valid_loss += loss.item()
        numpy_probas = logits.detach().cpu().numpy()
        y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
        y_true.extend(b_binarized_label.cpu().numpy())
    # End of an epoch of validation

    # put model to train mode again.
    model.train()

    ave_loss = total_valid_loss / (num_batch * batch_size)

    # compute the various f1 score for each label
    report = classification_report(y_true, y_pred, output_dict=True)
    metrics_df = pd.DataFrame(report).transpose()

    metrics_df = metrics_df.sort_index()

    weighted_f1_score = metrics_df.loc['weighted avg', 'f1-score']
    averaged_f1_score = metrics_df.loc['macro avg', 'f1-score']

    return ave_loss, metrics_df, {
        "weighted": weighted_f1_score,
        "averaged": averaged_f1_score,
    }
```

The other version I was trying is:
```
    transfored_logits = F.log_softmax(logits,dim=1)
    numpy_probas = transfored_logits.detach().cpu().numpy()
    y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
    y_true.extend(b_binarized_label.cpu().numpy())

```

The third version I was trying is:
```
   transfored_logits = torch.sigmoid(logits)
    numpy_probas = transfored_logits.detach().cpu().numpy()
    y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
    y_true.extend(b_binarized_label.cpu().numpy())

```

I also don't know how to understand the result. When I see online, people say if I set dim = 1 for log_softmax, the sum of probability for all feature (categories) should = 1. However, giving some examples below:

This is logits output: (for one batch - batch size = 16, num_labels = 2)
tensor([[ 1.1261, -1.8547],
        [ 0.6066, -1.1498],
        [ 1.3667, -2.0078],
        [ 2.0652, -2.6669],
        [ 1.0388, -1.7555],
        [ 0.6801, -1.1652],
        [ 0.8315, -1.3860],
        [ 1.5685, -2.2362],
        [ 0.1150, -0.3344],
        [ 2.0751, -2.6166],
        [ 1.5033, -2.1702],
        [ 0.1115, -0.3096],
        [ 0.8610, -1.4834],
        [ 1.5544, -2.2773],
        [ 2.1014, -2.6533],
        [ 0.7789, -1.3748]], device='cuda:0')

If I apply softmax first, F.log_softmax(logits,dim=1), I get:
tensor([[-0.0495, -3.0302],
        [-0.1593, -1.9157],
        [-0.0337, -3.4082],
        [-0.0088, -4.7409],
        [-0.0594, -2.8537],
        [-0.1467, -1.9920],
        [-0.1033, -2.3209],
        [-0.0220, -3.8267],
        [-0.4935, -0.9429],
        [-0.0091, -4.7008],
        [-0.0251, -3.6985],
        [-0.5046, -0.9257],
        [-0.0916, -2.4360],
        [-0.0214, -3.8531],
        [-0.0086, -4.7632],
        [-0.1098, -2.2635]], device='cuda:0')

The sum per row doesn't sum up to 1 and doesn't look like probability to me.

If I apply sigmoid, torch.sigmoid(logits)
tensor([[0.7551, 0.1353],
        [0.6472, 0.2405],
        [0.7969, 0.1184],
        [0.8875, 0.0650],
        [0.7386, 0.1474],
        [0.6638, 0.2377],
        [0.6967, 0.2000],
        [0.8276, 0.0965],
        [0.5287, 0.4172],
        [0.8885, 0.0681],
        [0.8181, 0.1025],
        [0.5278, 0.4232],
        [0.7029, 0.1849],
        [0.8255, 0.0930],
        [0.8910, 0.0658],
        [0.6854, 0.2018]], device='cuda:0')

It does look like probability more, although it still doesn't sum up to 1.

No matter which version I use, the predicted result is always the same in this case (since my 1 (Yes) Label has a really low incidence rate)
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])


Fix #7304

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7304

Correct order is tensors, name
@LysandreJik you need to teach me how to check the CI on TPU so we catch those rookie mistakes before merging :-)


Wrong arg order for `nested_xla_mesh_reduce` in trainer.py

## Environment info
Python 3.7 on Google Cloud TPUs

### Who can help
@sgugger

## Information
When training examples from [run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py) using Cloud TPUs, we run into this error:
```
TypeError: _xla_rendezvous(): incompatible function arguments. The following argument types are supported:

1. (arg0: int, arg1: str, arg2: str, arg3: List[int]) -> List[bytes]
```

This issue is just due to the wrong arg order in [this line](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L1337) where the [args should be switched](https://github.com/huggingface/transformers/blob/656c27c3a3345d0d2cf31c16f780b573c3dea09a/src/transformers/trainer_utils.py#L162).

This was introduced in:  https://github.com/huggingface/transformers/commit/492bb6aa486856f8243dfeb533ed1b23e996e403

## To reproduce

Steps to reproduce the behavior:

1. Running the provided example on Cloud TPU.

## Expected behavior
Should not fail with `TypeError`.


BART metrics.json and validation checkpoint metrics seem to disagree

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux 4.14
- Python version:3.7.9
- PyTorch version (GPU?):1.6.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: True

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
 examples/seq2seq: @sshleifer
## Information

Model I am using (BART):

The problem arises when using:
* [*] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)
I am using a small subset of https://github.com/pubmedqa/pubmedqa to generate questions from the given passage.

## To reproduce

Steps to reproduce the behavior:

1. Do distributed training
2. Run the finetune script with appropriate arguments.

The saved checkpoint says:
val_avg_rouge2=13.0975-step_count=4.ckpt

However the metrics.json file says that val_avg_rouge2=6.46015 and there are better step_counts in comparison.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The best metric in metrics.json should be chosen for saving checkpoints.


Add possibility to evaluate every epoch

This PR deprecates the argument `evaluate_during_training` and replaces it by `evaluation_strategy` (which supports more than just two values). The goal is to add support to easily evaluate the model every epoch (currently evaluation is done every n steps).

Evaluating every epoch is the most commonly used evaluation strategy, taught in pretty much every course and done in every basic training loop. While it's possible to do that right now, it require a bit of gymnastic (by building the dataloader and grabbing its length or dividing the number of samples in the dataset by the effective batch size). There have been two issues opened about that (but I'm too lazy to track their numbers ^^).

In passing, I made `eval_steps` default to the same as `logging_steps` because it seemed more natural.


[s2s] save hostname with repo info


[s2s] add src_lang kwarg for distributed eval


Create README.md

Related to pretrained model cards


[s2s] s/alpha_loss_encoder/alpha_encoder_loss/

fix to match `distillation.py:        self.alpha_encoder_loss`

@sshleifer


[s2s tests] fix test_run_eval_search

Fix problems with the new test

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7295

@sshleifer


Marian/MBart should not save static position embeddings

Add
```
    keys_to_never_save = [
        "model.encoder.embed_positions.weight",
        "model.decoder.embed_positions.weight",
    ]
```

Probably also applies to XLM, T5 (I think) and a few others.


test_run_eval_search SLOW failure

```bash
RUN_SLOW=1 pytest examples/seq2seq/test_seq2seq_examples.py -k search
```

```
FAILED examples/seq2seq/test_seq2seq_examples.py::test_run_eval_search[patrickvonplaten/t5-tiny-random] - ValueError: could not convert string to float: '--data_dir'
FAILED examples/seq2seq/test_seq2seq_examples.py::test_run_eval_search[sshleifer/tiny-mbart] - ValueError: could not convert string to float: '--data_dir'
```

Investigating


Bert Fine-Tuning on SQuAD with native TF2

I'm trying to run Bert fine tuning on SQuAD with Tensorflow's model.fit function.  I was able to run the tutorial on sequence classification from here https://huggingface.co/transformers/training.html#tensorflow, but a similar set up does not seem to work for question answering.  I am able to get it working with synthetic data, but not real data.  I can never seem to get the data in the right format.  Here is the script I am trying to use

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertForQuestionAnswering.from_pretrained('bert-base-uncased')

data = tfds.load('squad')
examples = SquadV1Processor().get_examples_from_dataset(data)
train_dataset = squad_convert_examples_to_features(examples, tokenizer, max_seq_length=328, doc_stride=128, max_query_length=32, is_training=True, return_dataset='tf')

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

model.compile(optimizer=optimizer, loss=loss)
model.fit(train_dataset, epochs=2, steps_per_epoch=1)

I've tried many other things, but this is the closest I got to something working.  The problem is that the dataset has keys that aren't recognized.  Specifically, I get this error

ValueError: Found unexpected keys that do not correspond to any Model output: dict_keys(['start_positions', 'end_positions', 'cls_index', 'p_mask', 'is_impossible']). Expected: ['output_1', 'output_2']

I might be able to find someway to manually remove these keys, but I feel like there should be some canonical way of getting this to work correctly.  I would appreciate any suggestions.  Thanks in advance.


Support serialized tokenizer in AutoTokenizer

# 🚀 Feature request

Support loading serialized tokenizer (using `Tokenizer.from_file`) in `AutoTokenizer`.

## Motivation

Some standard models use different tokenizers, e.g. `SentencePieceBPETokenizer`. So, there would be no need for implementing new models in `transformers`.


[fsmt] SinusoidalPositionalEmbedding no need to pass device

Just realized there is no need to pass `device` in forward as we already have it in `self.weight`, so simplifying the code a bit.


Fix saving TF custom models

This PR fixes #7277.

It was currently not possible to create a custom model that uses our TF main layers and to save it. A Keras Layer must have a `config` parameter whereas `keras_serializable` was creating a `transformers_config` parameter in the layer.


[s2s] add create student script

This PR adds `create_student.py` script to create the student models for distillbart/pegasus/marian and t5.

```bash
python make_student.py \
  facebook/bart-large-cnn \
  --e 12 \
  --d 6 \
  --save_path student-bart-cnn-12-6 \
```


Fix #7284

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7284

Don't add a keyword argument `"masked_lm_labels"` when`self.mlm` is False, to avoid an input error of the model.


Error importing MBart from transformers

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.29
- Python version: 3.8.2
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 Bart: @sshleifer

## Information

Model I am using: MBart, Bart has the same problem

The problem arises when using:
* [X] the official example scripts: Just copied it from the documentation here: https://huggingface.co/transformers/master/model_doc/mbart.html

The tasks I am working on is:
* [X] my own task or dataset

## To reproduce

Steps to reproduce the behavior:

1. `from transformers import MBartForConditionalGeneration, MBartTokenizer`

## Expected behavior
I would expect MBart to be imported so I could use it, but I get this error instead:

```
ImportError                               Traceback (most recent call last)
<ipython-input-23-20acbef89a41> in <module>
----> 1 from transformers import MBartForConditionalGeneration, MBartTokenizer

ImportError: cannot import name 'MBartForConditionalGeneration' from 'transformers' (/home/memduh/hf/venv/lib/python3.8/site-packages/transformers/__init__.py)
```


"index out of range in self" when calling BertForTokenClassification

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.29
- Python version: 3.8.2
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.  -->

 albert, bert, GPT2, XLM: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): BertForTokenClassification

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Get input_ids, attention_masks, and labels, all of size [32,128] (more generally [batch_size, sequence_length]) from the current batch
2. Send all three inputs to the same device as the model
3. Run the model with the three inputs: ```loss = model(input_ids, attention_masks, labels)```

This is not the code I was running when I encountered the error but running this small chunk reproduces the error for me:

```python

from transformers import BertTokenizer, BertForTokenClassification
import torch as th

batch_size = 32
seq_len = 128

inputs = th.randint(2000, 3000, size=(batch_size, seq_len))
attention_mask = th.randint(0,1,size=(batch_size, seq_len))
labels = th.randint(0,17, size=(batch_size, seq_len))

model = BertForTokenClassification.from_pretrained('bert-base-uncased')
output = model(inputs, attention_mask, labels)
```
and then the error:
```
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    output = model(inputs, attention_mask, labels)
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/transformers/modeling_bert.py", line 1488, in forward
    outputs = self.bert(
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/transformers/modeling_bert.py", line 824, in forward
    embedding_output = self.embeddings(
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/transformers/modeling_bert.py", line 209, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 124, in forward
    return F.embedding(
  File "/home/joakim/.virtualenvs/martino-replication/lib/python3.8/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The model does the forward pass with the current batch and returns the output

<!-- A clear and concise description of what you would expect to happen. -->

## What I've looked at

From previous similar issues like #4153 there was one person who added special tokens to the tokenizer. I don't know what that means but I don't think I've done that. There was also a bug which was patched. In issue #2371 the problem was that a sequence of size greater than 512 was sent into the model. I've printed the size before sending it into the model, and the size is as expected [batch_size, sequence_length], where sequence_length <= 512, so that's not what's happening here either. Finally I tried making sequences of size 512 and sending into the model, in case for some reason the model expected sequence length 512 and nothing else, but as expected that didn't work.


Added RobBERT-v2 model card

Related to pretrained model cards

Hi there!

We [just released](https://twitter.com/pieterdelobelle/status/1308016771744530433) version 2 of our [RobBERT](https://github.com/iPieter/RobBERT) model (= Dutch version of RoBERTa).
In this pull request, we included our model card for this version.

Thanks!


scibert-nli out of dace

## Environment info

- `transformers` version: 3.1.0
- Platform: macOS-10.15.5-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
Owner: @gsarti
Bert owner: @LysandreJik

## Information

By importing the model [here](https://huggingface.co/gsarti/scibert-nli?), two warnings prompt up

## To reproduce

Steps to reproduce the behavior:

1. Just import the model from the pretrained like in the example link

`FutureWarning: The class AutoModelWithLMHead is deprecated and will be removed in a future version. Please use AutoModelForCausalLM for causal language models, AutoModelForMaskedLM for masked language models and AutoModelForSeq2SeqLM for encoder-decoder models.`

`Some weights of BertForMaskedLM were not initialized from the model checkpoint at gsarti/scibert-nli and are newly initialized: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias', 'cls.predictions.transform.dense.weight']`

## Expected behavior

Import without any kind of errors/warnings. I suppose the first warning is due to deprecation and should be solved by importing `AutoModelForMaskedLM` instead (just looking for confirmation and giving the heads-up). The second, it seems that some layers are out of date and not trained, it would be good if the owner could update the model if possible.