Fine tune BERT based models using Trainer fails

## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-45-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes.
- Using distributed or parallel set-up in script?: No.

### Who can help
 Trainer: @sgugger

## Information

I am using pretrained BERT 'bert-base-multilingual-uncased' model and I would like to fine tune it on Next Sentence Prediction task. I followed example given here: [https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py).

## To reproduce

Steps to reproduce the behavior:

1. python3.8 program.py

program.py:
```python
import torch
from transformers import (BertForNextSentencePrediction,
                          BertTokenizer,
                          RobertaModel, RobertaTokenizer, Trainer,
                          TrainingArguments)
from transformers.data.datasets.language_modeling import TextDatasetForNextSentencePrediction
from transformers.data.data_collator import DataCollatorForNextSentencePrediction

if __name__ == "__main__":
    model_dir = "./model/"
    result_model_dir = "./result/"
    logs_directory = './logs'
    dataset_path = 'train.txt' # file in TextDatasetForNextSentencePrediction format
    tokenizer = RobertaTokenizer.from_pretrained('bert-base-multilingual-uncased')
    finetune_model = BertForNextSentencePrediction.from_pretrained('bert-base-multilingual-uncased')

    training_args = TrainingArguments(
        output_dir=result_model_dir,
        num_train_epochs=3,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        warmup_steps=500,
        weight_decay=0.01,
        logging_dir=logs_directory,
    )

    data_collator = DataCollatorForNextSentencePrediction(
        tokenizer=tokenizer,
        mlm=False,
        block_size=512,
        nsp_probability=0.5,
      )

    train_dataset = TextDatasetForNextSentencePrediction(
        tokenizer=tokenizer,
        file_path=dataset_path,
        block_size=512,
    )

    trainer = Trainer(
        model=finetune_model,
        args=training_args,
        train_dataset=train_dataset,
        data_collator=data_collator,
    )

    trainer.train()
    trainer.save_model(result_model_dir)
```
Output in terminal:
```bash
Special tokens have been added in the vocabulary, make sure the associated word emebedding are fine-tuned or trained.
Some weights of the model checkpoint at ./model/ were not used when initializing BertForNextSentencePrediction: ['roberta.embeddings.word_embeddings.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.embeddings.token_type_embeddings.weight', 'roberta.embeddings.LayerNorm.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.0.attention.self.query.weight', 'roberta.encoder.layer.0.attention.self.query.bias', 'roberta.encoder.layer.0.attention.self.key.weight', 'roberta.encoder.layer.0.attention.self.key.bias', 'roberta.encoder.layer.0.attention.self.value.weight', 'roberta.encoder.layer.0.attention.self.value.bias', 'roberta.encoder.layer.0.attention.output.dense.weight', 'roberta.encoder.layer.0.attention.output.dense.bias', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.intermediate.dense.weight', 'roberta.encoder.layer.0.intermediate.dense.bias', 'roberta.encoder.layer.0.output.dense.weight', 'roberta.encoder.layer.0.output.dense.bias', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.self.query.weight', 'roberta.encoder.layer.1.attention.self.query.bias', 'roberta.encoder.layer.1.attention.self.key.weight', 'roberta.encoder.layer.1.attention.self.key.bias', 'roberta.encoder.layer.1.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.value.bias', 'roberta.encoder.layer.1.attention.output.dense.weight', 'roberta.encoder.layer.1.attention.output.dense.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.intermediate.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.bias', 'roberta.encoder.layer.1.output.dense.weight', 'roberta.encoder.layer.1.output.dense.bias', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.2.attention.self.query.weight', 'roberta.encoder.layer.2.attention.self.query.bias', 'roberta.encoder.layer.2.attention.self.key.weight', 'roberta.encoder.layer.2.attention.self.key.bias', 'roberta.encoder.layer.2.attention.self.value.weight', 'roberta.encoder.layer.2.attention.self.value.bias', 'roberta.encoder.layer.2.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.output.dense.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.2.intermediate.dense.weight', 'roberta.encoder.layer.2.intermediate.dense.bias', 'roberta.encoder.layer.2.output.dense.weight', 'roberta.encoder.layer.2.output.dense.bias', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.query.weight', 'roberta.encoder.layer.3.attention.self.query.bias', 'roberta.encoder.layer.3.attention.self.key.weight', 'roberta.encoder.layer.3.attention.self.key.bias', 'roberta.encoder.layer.3.attention.self.value.weight', 'roberta.encoder.layer.3.attention.self.value.bias', 'roberta.encoder.layer.3.attention.output.dense.weight', 'roberta.encoder.layer.3.attention.output.dense.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.intermediate.dense.weight', 'roberta.encoder.layer.3.intermediate.dense.bias', 'roberta.encoder.layer.3.output.dense.weight', 'roberta.encoder.layer.3.output.dense.bias', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.self.query.weight', 'roberta.encoder.layer.4.attention.self.query.bias', 'roberta.encoder.layer.4.attention.self.key.weight', 'roberta.encoder.layer.4.attention.self.key.bias', 'roberta.encoder.layer.4.attention.self.value.weight', 'roberta.encoder.layer.4.attention.self.value.bias', 'roberta.encoder.layer.4.attention.output.dense.weight', 'roberta.encoder.layer.4.attention.output.dense.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.intermediate.dense.weight', 'roberta.encoder.layer.4.intermediate.dense.bias', 'roberta.encoder.layer.4.output.dense.weight', 'roberta.encoder.layer.4.output.dense.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.query.weight', 'roberta.encoder.layer.5.attention.self.query.bias', 'roberta.encoder.layer.5.attention.self.key.weight', 'roberta.encoder.layer.5.attention.self.key.bias', 'roberta.encoder.layer.5.attention.self.value.weight', 'roberta.encoder.layer.5.attention.self.value.bias', 'roberta.encoder.layer.5.attention.output.dense.weight', 'roberta.encoder.layer.5.attention.output.dense.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.intermediate.dense.weight', 'roberta.encoder.layer.5.intermediate.dense.bias', 'roberta.encoder.layer.5.output.dense.weight', 'roberta.encoder.layer.5.output.dense.bias', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.query.bias', 'roberta.encoder.layer.6.attention.self.key.weight', 'roberta.encoder.layer.6.attention.self.key.bias', 'roberta.encoder.layer.6.attention.self.value.weight', 'roberta.encoder.layer.6.attention.self.value.bias', 'roberta.encoder.layer.6.attention.output.dense.weight', 'roberta.encoder.layer.6.attention.output.dense.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.encoder.layer.6.intermediate.dense.weight', 'roberta.encoder.layer.6.intermediate.dense.bias', 'roberta.encoder.layer.6.output.dense.weight', 'roberta.encoder.layer.6.output.dense.bias', 'roberta.encoder.layer.6.output.LayerNorm.weight', 'roberta.encoder.layer.6.output.LayerNorm.bias', 'roberta.encoder.layer.7.attention.self.query.weight', 'roberta.encoder.layer.7.attention.self.query.bias', 'roberta.encoder.layer.7.attention.self.key.weight', 'roberta.encoder.layer.7.attention.self.key.bias', 'roberta.encoder.layer.7.attention.self.value.weight', 'roberta.encoder.layer.7.attention.self.value.bias', 'roberta.encoder.layer.7.attention.output.dense.weight', 'roberta.encoder.layer.7.attention.output.dense.bias', 'roberta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.encoder.layer.7.intermediate.dense.weight', 'roberta.encoder.layer.7.intermediate.dense.bias', 'roberta.encoder.layer.7.output.dense.weight', 'roberta.encoder.layer.7.output.dense.bias', 'roberta.encoder.layer.7.output.LayerNorm.weight', 'roberta.encoder.layer.7.output.LayerNorm.bias', 'roberta.encoder.layer.8.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.query.bias', 'roberta.encoder.layer.8.attention.self.key.weight', 'roberta.encoder.layer.8.attention.self.key.bias', 'roberta.encoder.layer.8.attention.self.value.weight', 'roberta.encoder.layer.8.attention.self.value.bias', 'roberta.encoder.layer.8.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.output.dense.bias', 'roberta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.encoder.layer.8.intermediate.dense.weight', 'roberta.encoder.layer.8.intermediate.dense.bias', 'roberta.encoder.layer.8.output.dense.weight', 'roberta.encoder.layer.8.output.dense.bias', 'roberta.encoder.layer.8.output.LayerNorm.weight', 'roberta.encoder.layer.8.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.query.weight', 'roberta.encoder.layer.9.attention.self.query.bias', 'roberta.encoder.layer.9.attention.self.key.weight', 'roberta.encoder.layer.9.attention.self.key.bias', 'roberta.encoder.layer.9.attention.self.value.weight', 'roberta.encoder.layer.9.attention.self.value.bias', 'roberta.encoder.layer.9.attention.output.dense.weight', 'roberta.encoder.layer.9.attention.output.dense.bias', 'roberta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.intermediate.dense.weight', 'roberta.encoder.layer.9.intermediate.dense.bias', 'roberta.encoder.layer.9.output.dense.weight', 'roberta.encoder.layer.9.output.dense.bias', 'roberta.encoder.layer.9.output.LayerNorm.weight', 'roberta.encoder.layer.9.output.LayerNorm.bias', 'roberta.encoder.layer.10.attention.self.query.weight', 'roberta.encoder.layer.10.attention.self.query.bias', 'roberta.encoder.layer.10.attention.self.key.weight', 'roberta.encoder.layer.10.attention.self.key.bias', 'roberta.encoder.layer.10.attention.self.value.weight', 'roberta.encoder.layer.10.attention.self.value.bias', 'roberta.encoder.layer.10.attention.output.dense.weight', 'roberta.encoder.layer.10.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.encoder.layer.10.intermediate.dense.weight', 'roberta.encoder.layer.10.intermediate.dense.bias', 'roberta.encoder.layer.10.output.dense.weight', 'roberta.encoder.layer.10.output.dense.bias', 'roberta.encoder.layer.10.output.LayerNorm.weight', 'roberta.encoder.layer.10.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.self.query.weight', 'roberta.encoder.layer.11.attention.self.query.bias', 'roberta.encoder.layer.11.attention.self.key.weight', 'roberta.encoder.layer.11.attention.self.key.bias', 'roberta.encoder.layer.11.attention.self.value.weight', 'roberta.encoder.layer.11.attention.self.value.bias', 'roberta.encoder.layer.11.attention.output.dense.weight', 'roberta.encoder.layer.11.attention.output.dense.bias', 'roberta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.encoder.layer.11.intermediate.dense.weight', 'roberta.encoder.layer.11.intermediate.dense.bias', 'roberta.encoder.layer.11.output.dense.weight', 'roberta.encoder.layer.11.output.dense.bias', 'roberta.encoder.layer.11.output.LayerNorm.weight', 'roberta.encoder.layer.11.output.LayerNorm.bias', 'roberta.encoder.layer.12.attention.self.query.weight', 'roberta.encoder.layer.12.attention.self.query.bias', 'roberta.encoder.layer.12.attention.self.key.weight', 'roberta.encoder.layer.12.attention.self.key.bias', 'roberta.encoder.layer.12.attention.self.value.weight', 'roberta.encoder.layer.12.attention.self.value.bias', 'roberta.encoder.layer.12.attention.output.dense.weight', 'roberta.encoder.layer.12.attention.output.dense.bias', 'roberta.encoder.layer.12.attention.output.LayerNorm.weight', 'roberta.encoder.layer.12.attention.output.LayerNorm.bias', 'roberta.encoder.layer.12.intermediate.dense.weight', 'roberta.encoder.layer.12.intermediate.dense.bias', 'roberta.encoder.layer.12.output.dense.weight', 'roberta.encoder.layer.12.output.dense.bias', 'roberta.encoder.layer.12.output.LayerNorm.weight', 'roberta.encoder.layer.12.output.LayerNorm.bias', 'roberta.encoder.layer.13.attention.self.query.weight', 'roberta.encoder.layer.13.attention.self.query.bias', 'roberta.encoder.layer.13.attention.self.key.weight', 'roberta.encoder.layer.13.attention.self.key.bias', 'roberta.encoder.layer.13.attention.self.value.weight', 'roberta.encoder.layer.13.attention.self.value.bias', 'roberta.encoder.layer.13.attention.output.dense.weight', 'roberta.encoder.layer.13.attention.output.dense.bias', 'roberta.encoder.layer.13.attention.output.LayerNorm.weight', 'roberta.encoder.layer.13.attention.output.LayerNorm.bias', 'roberta.encoder.layer.13.intermediate.dense.weight', 'roberta.encoder.layer.13.intermediate.dense.bias', 'roberta.encoder.layer.13.output.dense.weight', 'roberta.encoder.layer.13.output.dense.bias', 'roberta.encoder.layer.13.output.LayerNorm.weight', 'roberta.encoder.layer.13.output.LayerNorm.bias', 'roberta.encoder.layer.14.attention.self.query.weight', 'roberta.encoder.layer.14.attention.self.query.bias', 'roberta.encoder.layer.14.attention.self.key.weight', 'roberta.encoder.layer.14.attention.self.key.bias', 'roberta.encoder.layer.14.attention.self.value.weight', 'roberta.encoder.layer.14.attention.self.value.bias', 'roberta.encoder.layer.14.attention.output.dense.weight', 'roberta.encoder.layer.14.attention.output.dense.bias', 'roberta.encoder.layer.14.attention.output.LayerNorm.weight', 'roberta.encoder.layer.14.attention.output.LayerNorm.bias', 'roberta.encoder.layer.14.intermediate.dense.weight', 'roberta.encoder.layer.14.intermediate.dense.bias', 'roberta.encoder.layer.14.output.dense.weight', 'roberta.encoder.layer.14.output.dense.bias', 'roberta.encoder.layer.14.output.LayerNorm.weight', 'roberta.encoder.layer.14.output.LayerNorm.bias', 'roberta.encoder.layer.15.attention.self.query.weight', 'roberta.encoder.layer.15.attention.self.query.bias', 'roberta.encoder.layer.15.attention.self.key.weight', 'roberta.encoder.layer.15.attention.self.key.bias', 'roberta.encoder.layer.15.attention.self.value.weight', 'roberta.encoder.layer.15.attention.self.value.bias', 'roberta.encoder.layer.15.attention.output.dense.weight', 'roberta.encoder.layer.15.attention.output.dense.bias', 'roberta.encoder.layer.15.attention.output.LayerNorm.weight', 'roberta.encoder.layer.15.attention.output.LayerNorm.bias', 'roberta.encoder.layer.15.intermediate.dense.weight', 'roberta.encoder.layer.15.intermediate.dense.bias', 'roberta.encoder.layer.15.output.dense.weight', 'roberta.encoder.layer.15.output.dense.bias', 'roberta.encoder.layer.15.output.LayerNorm.weight', 'roberta.encoder.layer.15.output.LayerNorm.bias', 'roberta.encoder.layer.16.attention.self.query.weight', 'roberta.encoder.layer.16.attention.self.query.bias', 'roberta.encoder.layer.16.attention.self.key.weight', 'roberta.encoder.layer.16.attention.self.key.bias', 'roberta.encoder.layer.16.attention.self.value.weight', 'roberta.encoder.layer.16.attention.self.value.bias', 'roberta.encoder.layer.16.attention.output.dense.weight', 'roberta.encoder.layer.16.attention.output.dense.bias', 'roberta.encoder.layer.16.attention.output.LayerNorm.weight', 'roberta.encoder.layer.16.attention.output.LayerNorm.bias', 'roberta.encoder.layer.16.intermediate.dense.weight', 'roberta.encoder.layer.16.intermediate.dense.bias', 'roberta.encoder.layer.16.output.dense.weight', 'roberta.encoder.layer.16.output.dense.bias', 'roberta.encoder.layer.16.output.LayerNorm.weight', 'roberta.encoder.layer.16.output.LayerNorm.bias', 'roberta.encoder.layer.17.attention.self.query.weight', 'roberta.encoder.layer.17.attention.self.query.bias', 'roberta.encoder.layer.17.attention.self.key.weight', 'roberta.encoder.layer.17.attention.self.key.bias', 'roberta.encoder.layer.17.attention.self.value.weight', 'roberta.encoder.layer.17.attention.self.value.bias', 'roberta.encoder.layer.17.attention.output.dense.weight', 'roberta.encoder.layer.17.attention.output.dense.bias', 'roberta.encoder.layer.17.attention.output.LayerNorm.weight', 'roberta.encoder.layer.17.attention.output.LayerNorm.bias', 'roberta.encoder.layer.17.intermediate.dense.weight', 'roberta.encoder.layer.17.intermediate.dense.bias', 'roberta.encoder.layer.17.output.dense.weight', 'roberta.encoder.layer.17.output.dense.bias', 'roberta.encoder.layer.17.output.LayerNorm.weight', 'roberta.encoder.layer.17.output.LayerNorm.bias', 'roberta.encoder.layer.18.attention.self.query.weight', 'roberta.encoder.layer.18.attention.self.query.bias', 'roberta.encoder.layer.18.attention.self.key.weight', 'roberta.encoder.layer.18.attention.self.key.bias', 'roberta.encoder.layer.18.attention.self.value.weight', 'roberta.encoder.layer.18.attention.self.value.bias', 'roberta.encoder.layer.18.attention.output.dense.weight', 'roberta.encoder.layer.18.attention.output.dense.bias', 'roberta.encoder.layer.18.attention.output.LayerNorm.weight', 'roberta.encoder.layer.18.attention.output.LayerNorm.bias', 'roberta.encoder.layer.18.intermediate.dense.weight', 'roberta.encoder.layer.18.intermediate.dense.bias', 'roberta.encoder.layer.18.output.dense.weight', 'roberta.encoder.layer.18.output.dense.bias', 'roberta.encoder.layer.18.output.LayerNorm.weight', 'roberta.encoder.layer.18.output.LayerNorm.bias', 'roberta.encoder.layer.19.attention.self.query.weight', 'roberta.encoder.layer.19.attention.self.query.bias', 'roberta.encoder.layer.19.attention.self.key.weight', 'roberta.encoder.layer.19.attention.self.key.bias', 'roberta.encoder.layer.19.attention.self.value.weight', 'roberta.encoder.layer.19.attention.self.value.bias', 'roberta.encoder.layer.19.attention.output.dense.weight', 'roberta.encoder.layer.19.attention.output.dense.bias', 'roberta.encoder.layer.19.attention.output.LayerNorm.weight', 'roberta.encoder.layer.19.attention.output.LayerNorm.bias', 'roberta.encoder.layer.19.intermediate.dense.weight', 'roberta.encoder.layer.19.intermediate.dense.bias', 'roberta.encoder.layer.19.output.dense.weight', 'roberta.encoder.layer.19.output.dense.bias', 'roberta.encoder.layer.19.output.LayerNorm.weight', 'roberta.encoder.layer.19.output.LayerNorm.bias', 'roberta.encoder.layer.20.attention.self.query.weight', 'roberta.encoder.layer.20.attention.self.query.bias', 'roberta.encoder.layer.20.attention.self.key.weight', 'roberta.encoder.layer.20.attention.self.key.bias', 'roberta.encoder.layer.20.attention.self.value.weight', 'roberta.encoder.layer.20.attention.self.value.bias', 'roberta.encoder.layer.20.attention.output.dense.weight', 'roberta.encoder.layer.20.attention.output.dense.bias', 'roberta.encoder.layer.20.attention.output.LayerNorm.weight', 'roberta.encoder.layer.20.attention.output.LayerNorm.bias', 'roberta.encoder.layer.20.intermediate.dense.weight', 'roberta.encoder.layer.20.intermediate.dense.bias', 'roberta.encoder.layer.20.output.dense.weight', 'roberta.encoder.layer.20.output.dense.bias', 'roberta.encoder.layer.20.output.LayerNorm.weight', 'roberta.encoder.layer.20.output.LayerNorm.bias', 'roberta.encoder.layer.21.attention.self.query.weight', 'roberta.encoder.layer.21.attention.self.query.bias', 'roberta.encoder.layer.21.attention.self.key.weight', 'roberta.encoder.layer.21.attention.self.key.bias', 'roberta.encoder.layer.21.attention.self.value.weight', 'roberta.encoder.layer.21.attention.self.value.bias', 'roberta.encoder.layer.21.attention.output.dense.weight', 'roberta.encoder.layer.21.attention.output.dense.bias', 'roberta.encoder.layer.21.attention.output.LayerNorm.weight', 'roberta.encoder.layer.21.attention.output.LayerNorm.bias', 'roberta.encoder.layer.21.intermediate.dense.weight', 'roberta.encoder.layer.21.intermediate.dense.bias', 'roberta.encoder.layer.21.output.dense.weight', 'roberta.encoder.layer.21.output.dense.bias', 'roberta.encoder.layer.21.output.LayerNorm.weight', 'roberta.encoder.layer.21.output.LayerNorm.bias', 'roberta.encoder.layer.22.attention.self.query.weight', 'roberta.encoder.layer.22.attention.self.query.bias', 'roberta.encoder.layer.22.attention.self.key.weight', 'roberta.encoder.layer.22.attention.self.key.bias', 'roberta.encoder.layer.22.attention.self.value.weight', 'roberta.encoder.layer.22.attention.self.value.bias', 'roberta.encoder.layer.22.attention.output.dense.weight', 'roberta.encoder.layer.22.attention.output.dense.bias', 'roberta.encoder.layer.22.attention.output.LayerNorm.weight', 'roberta.encoder.layer.22.attention.output.LayerNorm.bias', 'roberta.encoder.layer.22.intermediate.dense.weight', 'roberta.encoder.layer.22.intermediate.dense.bias', 'roberta.encoder.layer.22.output.dense.weight', 'roberta.encoder.layer.22.output.dense.bias', 'roberta.encoder.layer.22.output.LayerNorm.weight', 'roberta.encoder.layer.22.output.LayerNorm.bias', 'roberta.encoder.layer.23.attention.self.query.weight', 'roberta.encoder.layer.23.attention.self.query.bias', 'roberta.encoder.layer.23.attention.self.key.weight', 'roberta.encoder.layer.23.attention.self.key.bias', 'roberta.encoder.layer.23.attention.self.value.weight', 'roberta.encoder.layer.23.attention.self.value.bias', 'roberta.encoder.layer.23.attention.output.dense.weight', 'roberta.encoder.layer.23.attention.output.dense.bias', 'roberta.encoder.layer.23.attention.output.LayerNorm.weight', 'roberta.encoder.layer.23.attention.output.LayerNorm.bias', 'roberta.encoder.layer.23.intermediate.dense.weight', 'roberta.encoder.layer.23.intermediate.dense.bias', 'roberta.encoder.layer.23.output.dense.weight', 'roberta.encoder.layer.23.output.dense.bias', 'roberta.encoder.layer.23.output.LayerNorm.weight', 'roberta.encoder.layer.23.output.LayerNorm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
- This IS expected if you are initializing BertForNextSentencePrediction from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForNextSentencePrediction from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForNextSentencePrediction were not initialized from the model checkpoint at ./model/ and are newly initialized: ['embeddings.word_embeddings.weight', 'embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.LayerNorm.weight', 'embeddings.LayerNorm.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.9.output.dense.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.10.attention.self.value.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.11.output.dense.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.11.output.LayerNorm.bias', 'encoder.layer.12.attention.self.query.weight', 'encoder.layer.12.attention.self.query.bias', 'encoder.layer.12.attention.self.key.weight', 'encoder.layer.12.attention.self.key.bias', 'encoder.layer.12.attention.self.value.weight', 'encoder.layer.12.attention.self.value.bias', 'encoder.layer.12.attention.output.dense.weight', 'encoder.layer.12.attention.output.dense.bias', 'encoder.layer.12.attention.output.LayerNorm.weight', 'encoder.layer.12.attention.output.LayerNorm.bias', 'encoder.layer.12.intermediate.dense.weight', 'encoder.layer.12.intermediate.dense.bias', 'encoder.layer.12.output.dense.weight', 'encoder.layer.12.output.dense.bias', 'encoder.layer.12.output.LayerNorm.weight', 'encoder.layer.12.output.LayerNorm.bias', 'encoder.layer.13.attention.self.query.weight', 'encoder.layer.13.attention.self.query.bias', 'encoder.layer.13.attention.self.key.weight', 'encoder.layer.13.attention.self.key.bias', 'encoder.layer.13.attention.self.value.weight', 'encoder.layer.13.attention.self.value.bias', 'encoder.layer.13.attention.output.dense.weight', 'encoder.layer.13.attention.output.dense.bias', 'encoder.layer.13.attention.output.LayerNorm.weight', 'encoder.layer.13.attention.output.LayerNorm.bias', 'encoder.layer.13.intermediate.dense.weight', 'encoder.layer.13.intermediate.dense.bias', 'encoder.layer.13.output.dense.weight', 'encoder.layer.13.output.dense.bias', 'encoder.layer.13.output.LayerNorm.weight', 'encoder.layer.13.output.LayerNorm.bias', 'encoder.layer.14.attention.self.query.weight', 'encoder.layer.14.attention.self.query.bias', 'encoder.layer.14.attention.self.key.weight', 'encoder.layer.14.attention.self.key.bias', 'encoder.layer.14.attention.self.value.weight', 'encoder.layer.14.attention.self.value.bias', 'encoder.layer.14.attention.output.dense.weight', 'encoder.layer.14.attention.output.dense.bias', 'encoder.layer.14.attention.output.LayerNorm.weight', 'encoder.layer.14.attention.output.LayerNorm.bias', 'encoder.layer.14.intermediate.dense.weight', 'encoder.layer.14.intermediate.dense.bias', 'encoder.layer.14.output.dense.weight', 'encoder.layer.14.output.dense.bias', 'encoder.layer.14.output.LayerNorm.weight', 'encoder.layer.14.output.LayerNorm.bias', 'encoder.layer.15.attention.self.query.weight', 'encoder.layer.15.attention.self.query.bias', 'encoder.layer.15.attention.self.key.weight', 'encoder.layer.15.attention.self.key.bias', 'encoder.layer.15.attention.self.value.weight', 'encoder.layer.15.attention.self.value.bias', 'encoder.layer.15.attention.output.dense.weight', 'encoder.layer.15.attention.output.dense.bias', 'encoder.layer.15.attention.output.LayerNorm.weight', 'encoder.layer.15.attention.output.LayerNorm.bias', 'encoder.layer.15.intermediate.dense.weight', 'encoder.layer.15.intermediate.dense.bias', 'encoder.layer.15.output.dense.weight', 'encoder.layer.15.output.dense.bias', 'encoder.layer.15.output.LayerNorm.weight', 'encoder.layer.15.output.LayerNorm.bias', 'encoder.layer.16.attention.self.query.weight', 'encoder.layer.16.attention.self.query.bias', 'encoder.layer.16.attention.self.key.weight', 'encoder.layer.16.attention.self.key.bias', 'encoder.layer.16.attention.self.value.weight', 'encoder.layer.16.attention.self.value.bias', 'encoder.layer.16.attention.output.dense.weight', 'encoder.layer.16.attention.output.dense.bias', 'encoder.layer.16.attention.output.LayerNorm.weight', 'encoder.layer.16.attention.output.LayerNorm.bias', 'encoder.layer.16.intermediate.dense.weight', 'encoder.layer.16.intermediate.dense.bias', 'encoder.layer.16.output.dense.weight', 'encoder.layer.16.output.dense.bias', 'encoder.layer.16.output.LayerNorm.weight', 'encoder.layer.16.output.LayerNorm.bias', 'encoder.layer.17.attention.self.query.weight', 'encoder.layer.17.attention.self.query.bias', 'encoder.layer.17.attention.self.key.weight', 'encoder.layer.17.attention.self.key.bias', 'encoder.layer.17.attention.self.value.weight', 'encoder.layer.17.attention.self.value.bias', 'encoder.layer.17.attention.output.dense.weight', 'encoder.layer.17.attention.output.dense.bias', 'encoder.layer.17.attention.output.LayerNorm.weight', 'encoder.layer.17.attention.output.LayerNorm.bias', 'encoder.layer.17.intermediate.dense.weight', 'encoder.layer.17.intermediate.dense.bias', 'encoder.layer.17.output.dense.weight', 'encoder.layer.17.output.dense.bias', 'encoder.layer.17.output.LayerNorm.weight', 'encoder.layer.17.output.LayerNorm.bias', 'encoder.layer.18.attention.self.query.weight', 'encoder.layer.18.attention.self.query.bias', 'encoder.layer.18.attention.self.key.weight', 'encoder.layer.18.attention.self.key.bias', 'encoder.layer.18.attention.self.value.weight', 'encoder.layer.18.attention.self.value.bias', 'encoder.layer.18.attention.output.dense.weight', 'encoder.layer.18.attention.output.dense.bias', 'encoder.layer.18.attention.output.LayerNorm.weight', 'encoder.layer.18.attention.output.LayerNorm.bias', 'encoder.layer.18.intermediate.dense.weight', 'encoder.layer.18.intermediate.dense.bias', 'encoder.layer.18.output.dense.weight', 'encoder.layer.18.output.dense.bias', 'encoder.layer.18.output.LayerNorm.weight', 'encoder.layer.18.output.LayerNorm.bias', 'encoder.layer.19.attention.self.query.weight', 'encoder.layer.19.attention.self.query.bias', 'encoder.layer.19.attention.self.key.weight', 'encoder.layer.19.attention.self.key.bias', 'encoder.layer.19.attention.self.value.weight', 'encoder.layer.19.attention.self.value.bias', 'encoder.layer.19.attention.output.dense.weight', 'encoder.layer.19.attention.output.dense.bias', 'encoder.layer.19.attention.output.LayerNorm.weight', 'encoder.layer.19.attention.output.LayerNorm.bias', 'encoder.layer.19.intermediate.dense.weight', 'encoder.layer.19.intermediate.dense.bias', 'encoder.layer.19.output.dense.weight', 'encoder.layer.19.output.dense.bias', 'encoder.layer.19.output.LayerNorm.weight', 'encoder.layer.19.output.LayerNorm.bias', 'encoder.layer.20.attention.self.query.weight', 'encoder.layer.20.attention.self.query.bias', 'encoder.layer.20.attention.self.key.weight', 'encoder.layer.20.attention.self.key.bias', 'encoder.layer.20.attention.self.value.weight', 'encoder.layer.20.attention.self.value.bias', 'encoder.layer.20.attention.output.dense.weight', 'encoder.layer.20.attention.output.dense.bias', 'encoder.layer.20.attention.output.LayerNorm.weight', 'encoder.layer.20.attention.output.LayerNorm.bias', 'encoder.layer.20.intermediate.dense.weight', 'encoder.layer.20.intermediate.dense.bias', 'encoder.layer.20.output.dense.weight', 'encoder.layer.20.output.dense.bias', 'encoder.layer.20.output.LayerNorm.weight', 'encoder.layer.20.output.LayerNorm.bias', 'encoder.layer.21.attention.self.query.weight', 'encoder.layer.21.attention.self.query.bias', 'encoder.layer.21.attention.self.key.weight', 'encoder.layer.21.attention.self.key.bias', 'encoder.layer.21.attention.self.value.weight', 'encoder.layer.21.attention.self.value.bias', 'encoder.layer.21.attention.output.dense.weight', 'encoder.layer.21.attention.output.dense.bias', 'encoder.layer.21.attention.output.LayerNorm.weight', 'encoder.layer.21.attention.output.LayerNorm.bias', 'encoder.layer.21.intermediate.dense.weight', 'encoder.layer.21.intermediate.dense.bias', 'encoder.layer.21.output.dense.weight', 'encoder.layer.21.output.dense.bias', 'encoder.layer.21.output.LayerNorm.weight', 'encoder.layer.21.output.LayerNorm.bias', 'encoder.layer.22.attention.self.query.weight', 'encoder.layer.22.attention.self.query.bias', 'encoder.layer.22.attention.self.key.weight', 'encoder.layer.22.attention.self.key.bias', 'encoder.layer.22.attention.self.value.weight', 'encoder.layer.22.attention.self.value.bias', 'encoder.layer.22.attention.output.dense.weight', 'encoder.layer.22.attention.output.dense.bias', 'encoder.layer.22.attention.output.LayerNorm.weight', 'encoder.layer.22.attention.output.LayerNorm.bias', 'encoder.layer.22.intermediate.dense.weight', 'encoder.layer.22.intermediate.dense.bias', 'encoder.layer.22.output.dense.weight', 'encoder.layer.22.output.dense.bias', 'encoder.layer.22.output.LayerNorm.weight', 'encoder.layer.22.output.LayerNorm.bias', 'encoder.layer.23.attention.self.query.weight', 'encoder.layer.23.attention.self.query.bias', 'encoder.layer.23.attention.self.key.weight', 'encoder.layer.23.attention.self.key.bias', 'encoder.layer.23.attention.self.value.weight', 'encoder.layer.23.attention.self.value.bias', 'encoder.layer.23.attention.output.dense.weight', 'encoder.layer.23.attention.output.dense.bias', 'encoder.layer.23.attention.output.LayerNorm.weight', 'encoder.layer.23.attention.output.LayerNorm.bias', 'encoder.layer.23.intermediate.dense.weight', 'encoder.layer.23.intermediate.dense.bias', 'encoder.layer.23.output.dense.weight', 'encoder.layer.23.output.dense.bias', 'encoder.layer.23.output.LayerNorm.weight', 'encoder.layer.23.output.LayerNorm.bias', 'pooler.dense.weight', 'pooler.dense.bias', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Epoch:   0%|                                                                                | 0/3 [00:00<?, ?it/sTraceback (most recent call last):                                                                                | 0/1 [00:00<?, ?it/s]
  File "polish_roberta.py", line 77, in <module>
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/transformers/trainer.py", line 707, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/transformers/trainer.py", line 995, in training_step
    outputs = model(**inputs)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/awawrzynski/miniconda3/envs/polish_roberta/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'masked_lm_labels'

Epoch:   0%|                                                                                | 0/3 [00:04<?, ?it/s]
Iteration:   0%|                                                                                | 0/1 [00:04<?, ?it/s]
```

## Expected behavior
Model is trained and after training is finished model is saved into pointed directory.

## Solution
I've followed errors in terminal and find out that excessive parameter is passed into Trainer's training_step function. I've added line that removes this key from dictionary passed into model and it works.

```python
def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) -> torch.Tensor:
        """
        Perform a training step on a batch of inputs.

        Subclass and override to inject custom behavior.

        Args:
            model (:obj:`nn.Module`):
                The model to train.
            inputs (:obj:`Dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.

                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument :obj:`labels`. Check your model's documentation for all accepted arguments.

        Return:
            :obj:`torch.Tensor`: The tensor with training loss on this batch.
        """
        if hasattr(self, "_training_step"):
            warnings.warn(
                "The `_training_step` method is deprecated and won't be called in a future version, define `training_step` in your subclass.",
                FutureWarning,
            )
            return self._training_step(model, inputs, self.optimizer)

        model.train()
        inputs = self._prepare_inputs(inputs)
        inputs.pop("masked_lm_labels") # I added this line and it works.

        if self.args.fp16 and _use_native_amp:
            with autocast():
                outputs = model(**inputs)
                loss = outputs[0]
        else:
            outputs = model(**inputs)
            # We don't use .loss here since the model may return tuples instead of ModelOutput.
            loss = outputs[0]

        if self.args.past_index >= 0:
            self._past = outputs[self.args.past_index]

        if self.args.n_gpu > 1:
            loss = loss.mean()  # mean() to average on multi-gpu parallel training

        if self.args.gradient_accumulation_steps > 1:
            loss = loss / self.args.gradient_accumulation_steps

        if self.args.fp16 and _use_native_amp:
            self.scaler.scale(loss).backward()
        elif self.args.fp16 and _use_apex:
            with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                scaled_loss.backward()
        else:
            loss.backward()

        return loss
```


IXAmBERT model card

model card

Related to pretrained model cards

This PR includes the model card for the IXAmBERT model which has been recently uploaded to the huggingface repository.

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Disable missing weight warning for RobertaForMaskedLM/CamembertForMaskedLM

Fixes #7167, #6193

When loading RobertaForMaskedLM/CamembertForMaskedLM checkpoints, a warning is displayed because "lm_head.decoder.bias" is not loaded. This warning is an artifact of the way the bias are stored in RobertaLMHead.
This PR adds `lm_head.decoder.bias` to the authorized missing keys list for RobertaForMaskedLM.


[seq2seq testing] multigpu test run via subprocess

Tests

Related to tests

This PR is trying to fix the hanging/misbehaving/self-replicating pytest for tests using PL with `gpu>1` (`ddp` backend).

OK, I couldn't figure out how to make `dp` or `ddp_spawn` to work, all kinds of obscure errors inside PL (It doesn't look like these are closely maintained as it's recommended not to use either), so `ddp` it is. I tried to get `dp` to work first, since it doesn't require forking a new process and special handling inside pytest.

Here is a working solution for `ddp`. Bottom line - you have to fork a new process and run the distributed script from it - to get it working with `ddp` - otherwise pytest either hangs or runs itself multiple times, breaking other scripts, a big mess.
I borrow the idea from PL itself https://github.com/PyTorchLightning/pytorch-lightning/blob/master/tests/models/test_gpu.py#L111 - what a better place to find the correct way to test something but from the horse's mouth.

So what I had to do:

* [x] split into `test_seq2seq_examples_multi_gpu.py` as requested
* [x] added `subprocess.Popen` - but then replaced it with the modern `asyncio`- apparently using stdout and stderr pipes with `wait` can still cause a deadlock - but let's see if that works for our needs.
* [x] had to mess with args to correctly convert them into cl args - so many of them! perhaps there is already a helper util that does that - I probably re-invented the wheel
* [x] had to provide a new flag `--overwrite_output_dir` to support multi-gpu processes, as one of the children otherwise creates the output dir and the other fails to do so. instead we create the dir in the parent process.
* [x] there are some issues with accessing module attributes in the distributed env (see details here: https://github.com/huggingface/transformers/issues/5887#issuecomment-695919119) - had to tweak the `lightning_base.py` to not use attributes in 2 accessors. (I didn't check - it's possible I need to adjust other scripts if they use `self.total_steps`) - I'm not 100% sure what is different under ddp - but somehow things behave differently and we have no access to module's attributes unless they are part of the model - see `nn.Module.__getattr__` - might have to do with modules getting pickled. if you have insights I'm all ears.
* [x] the test validation had to be adjust to handle 2 gpus

@sshleifer

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #5887


I want to use the Bert2GPT2 architecture,but my pretrained Bert and GPT2 have different vocabs,so what should I do for the vocabs?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


[wip/dont-merge] pegasus beam search implementation

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


[model card] distlbart-mnli model cards

model card

Related to pretrained model cards

No teacher bart distillation for MNLI

cc @sshleifer


Unable to serialize/save TF2.0 Bert model

Should Fix

This has been identified as a bug and should be fixed.

TensorFlow

Anything TensorFlow

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
https://colab.research.google.com/gist/Douboo/fccd6bcda2e098b10b1c7490f2d8bbf3/untitled3.ipynb#scrollTo=yW4t3uQxjhy1
- `transformers` version: 3.1.0
- Platform: colab
- Python version: 3.7.0
- Tensorflow version (GPU?): 2.3.0
- Using GPU in script?: yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@jplu

## Information

Model I am using (Bert):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
```python
# define model via functional api
def build_model(item_dim, num_layers, num_heads, max_len):
    config = BertConfig(hidden_size=item_dim, num_hidden_layers=num_layers,
                        num_attention_heads=num_heads, intermediate_size=item_dim*4,
                        max_position_embeddings=max_len)
    bert = TFBertMainLayer(config=config)
    inputs_embeds = Input(shape=(max_len, item_dim), dtype=tf.float32, name='inputs')
    inputs = {"inputs_embeds": inputs_embeds}
    # pre-training vectors to bert
    seq_emb = bert(inputs)[0]
    last_token_emb = seq_emb[:, -1, :]
    outputs = Dense(1, activation='sigmoid')(last_token_emb)
    model = Model(inputs=inputs, outputs=outputs)
    return model

model = build_model(item_dim, num_layers, num_heads, max_len)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['acc'])

model.fit(X, y,
          epochs=2, batch_size=128,
          verbose=2
         )

model.save('./my_model')
```
Errors with
`TypeError: ('Not JSON Serializable:', BertConfig {
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 32,
  "initializer_range": 0.02,
  "intermediate_size": 128,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 9,
  "model_type": "bert",
  "num_attention_heads": 1,
  "num_hidden_layers": 1,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 30522
}
)`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
* Successfully saved keras model.
<!-- A clear and concise description of what you would expect to happen. -->


fix unnessasry cpu memory usage when training

Use scalar when recording the aggregated training loss throughout the training procedure in `Trainer`.

The previous version uses `torch.Tensor` to store `tr_loss`. Here, tr_loss is accumulating history across your training loop and takes up lots of CPU memory during training. There is an official pytorch post on **NOT to accumulate history across your training loop**. ([https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory](https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory))

 When I tried to train a sentence classififier with batch size 10 on HotpotQA dataset (for document filtering), the program consumed around 120G CPU memory when I trained for 240,000 steps (roughly 3 epoch over all 900,000 HotpotQA documents), which is unnessasry.

So I think it's better to just use a scalar (as suggested by the official post). Also, the tensor format returned value has never been used in the whole repo. It's a safe fix.


Question about model configuration

# ❓ Questions & Help

Hi,

Now I am trying gpt-2 model to generate sentences. I have a question about the model configuration.

I load model using [this line in the provided code](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py#L221). I found "eos_tokens_ids" and "pad_token_id" both equal to 0. But when I checked the tokenizer, I found the token with id 0 is '!'. So when I use the provided code to generate sentences, sentences always end with '!'. Is there any mistake in the vocabulary or the configuration? Many thanks.


[seq2seq] make it easier to run the scripts

This PR:
* converts scripts into independent executables - easier to run. Docs updates.
* tweaks `./finetune.py` and `./distillation.py` to be self-contained - no need to tweak `PYTHONPATH` via a shell script anymore
* fixes a few more left-overs from `try: import .foo; except: import foo`

Let me know if there any other tweaks to make.

@sshleifer


[Bug/Question] Write With Transformers Implementation vs. Custom Implementation

Hi,

Not sure if this is a bug or perhaps a misimplementation, but i'm comparing the results of using gpt2-large on the 'Write With Transformers' Text Generation example - https://transformer.huggingface.co/doc/gpt2-large, vs. my own implementation of the text generation tool. My use-case is to generate distractor options in an MCQ environment, given a few (15-20) prior examples for the style of distractor questions to generate.

The general format I am implementing is:
`Question: _______ . Answer 1: ___<the correct answer>___ . Answer 2: _____<distractor 1>_____ . Answer 3: ____<distractor 2>_____`

## Write With Transformers
full doc available here - https://transformer.huggingface.co/share/CZqVXdngic

### Model Config
Model size - gpt2/large
Top-p - 0.9
Temperature - 1
Max time - 2.3
### Output
On the 'Write With Transformers' page, I write in the examples:
eg. `Question: How is calorie related to the S.I unit of that quantity?Answer 1: 1 cal = 4.2 J.Answer 2: 1 cal = 3.2 J.Answer 3: 1 cal = 10 J.`

and when I try to generate predictions for the following Question-Answer pair:
`Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W.`

am able to generate a few solid distractors:
`Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2:  1 H. P. = 1. 3 W.  Answer 3 : 1 H . P. =  1. 5 W.`

## Custom Case

### Output
Here's how I create a dataset from the set of questions I've initially written myself.
```
def make_dataset(dataset, epochs):
    total_text = '<|endoftext|>'
    qa = [t for t in dataset]
    for _ in range(epochs):
        random.shuffle(qa)
        total_text += '<|endoftext|>'.join(qa) + '<|endoftext|>'
    return total_text
```

This is the training model params:

```
!python run_language_modeling.py \
    --output_dir=output/$handle \
    --overwrite_output_dir \
    --overwrite_cache \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train --train_data_file=$handle\_train.txt \
    --logging_steps 20 \
    --per_gpu_train_batch_size 1 \
    --num_train_epochs 15
```

num_return_sequences = 5

```
for start in SENTENCES:
    val = !python run_generation.py \
        --model_type gpt2 \
        --model_name_or_path output/$handle \
        --length 40 \
        --num_return_sequences $num_return_sequences \
        --temperature 0.23 \
        --p 0.95 \
        --seed $seed \
        --prompt {'"<|endoftext|>' + start + '"'}
    generated = [val[-1-2*k] for k in range(num_return_sequences)[::-1]]
    print(f'\nStart of sentence: {start}')
    for i, g in enumerate(generated):
        g = g.replace('<|endoftext|>', '')
        print(f'* Generated #{i+1}: {g}')
```

These are my generated mcq pairs:

**Generated #1: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
**Generated #2: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
**Generated #3: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: ______________ = 905 J.Answer 3: 50 J.**
**Generated #4: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
**Generated #5: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**


[Longformer, Bert, Roberta, ...] Fix multi gpu training

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6256.

Issue #6256 shows that distributed training is not possible when the model has layers that are not used at all.
Bert, Roberta and Longformer all have a "pooling_layer" added to their BaseModel (`BertModel`, `RobertaModel`, `LongformerModel`). This pooling layer is however not needed by all the "higher" model extensions. The `LongformerForSequenceClassification`, `...ForTokenClassification` model extensions e.g. simply disregards the pooled output of the BaseModel.
Looking back, I think the pooling layer should not have been part of the BaseModel classes, but is now not really possible to revert the design choice anymore given the number of `BertModel`s already trained with this layer.

Ideally, we should both in TF and PT automatically never instantiate a layer that is never needed.
This is now implemented in PyTorch. In TF loading pretrained weights depends on the `load_weights` function here: https://github.com/huggingface/transformers/blob/38f17037957d325b5540a8031f065e6f23c9e265/src/transformers/modeling_tf_utils.py#L614
The load_weights function **cannot** load a layer which has a different number of weights in the `resolved_archive_file` as `model`.
This means that if we would remove the `pooler` layer for `TFBertForMaskedLM` because it is not needed, the following command:
```
python -c 'from transformers import TFBertForMaskedLM; TFBertForMaskedLM.from_pretrained("bert-base-cased")'
```
would throw the following error:

```
ValueError: Layer #0 (named "bert") expects 197 weight(s), but the saved weights have 199 element(s).
```

 At the moment I don't see how we could keep backwards compatibility in TF without completely replacing the `from_pretrained` functionality and even then I don't think it will be easy to load hdf5 files with more layers than expected in TF.

So I propose the following solution:
Only implement the feature for PT and leave the layer in TF for now because a) It solves the issue b) is 100% backwards compatible.
It would be nice to remove the pooling layers for TF in a future PR, but I think we would have to rewrite `from_pretrained` for TF models in this case (cc @jplu )

@sgugger @LysandreJik - what do you think?


Distilbert classification

Is Maximum Entropy the same as Cross-entropy loss in the context of Distilbert (happy or sad sentiment label) classification?


Multitask pre-training setting

# 🚀 Feature request

Hi everybody!

Is it possible to pre-train T5 model on a mixture of tasks?
Just to be more clear, let's assume I have created a dataset by following the example provided here:
https://huggingface.co/transformers/model_doc/t5.html

Specifically I'm referring to the _**Unsupervised denoising training**_ and _**Supervised training**_.
What I've done is to construct a huge dataset where each tasks is concatenated to the previous one.

Is this enough to experiment with the multitask pertaining setting?


Add model cards for new pre-trained BERTweet-COVID19 models

model card

Related to pretrained model cards

Two new pre-trained models "vinai/bertweet-covid19-base-cased" and "vinai/bertweet-covid19-base-uncased" are resulted by further pre-training the pre-trained model "vinai/bertweet-base" on a  corpus of 23M COVID-19 English Tweets for 40 epochs.


Fix typo in model name


[Bug fix] Fixed target_mapping preparation for XLNet (Pytorch)

The pytorch version of XLNet does not create the `target_mapping` correctly when the batch size > 1.
For generation, the `target_mapping` should be made of zeros, except at the last position (token to be predicted). The current Pytorch implementation is currently only non-zero at the last position of the first batch position:
https://github.com/huggingface/transformers/blob/4f6e52574248636352a746cfe6cc0b13cf3eb7f9/src/transformers/modeling_xlnet.py#L1316

This causes the model to be incorrect when multiple inputs are passed to the model, or when beam search is turned on. This PR fixes the issue by setting the last (sequence) position of `target_mapping` to 1 for all batch positions.

For reference, the Tensorflow implementation seems to be already correct:
https://github.com/huggingface/transformers/blob/4f6e52574248636352a746cfe6cc0b13cf3eb7f9/src/transformers/modeling_tf_xlnet.py#L1158


LXMERT pre-training tasks

# ❓ Questions & Help

Hello, congrats to all contributors for the awesome work with LXMERT! It is exciting to see multimodal transformers coming to hugginface/transformers. Of course, I immediately tried it out and played with the demo.

## LXMERT pre-trained model, trained on what exactly?
__Question:__
Does the line `lxmert_base = LxmertForPreTraining.from_pretrained("unc-nlp/lxmert-base-uncased")` load an already pre-trained LXMERT model on the tasks enumerated in the original paper _“(1) masked crossmodality language modeling, (2) masked object prediction via RoI-feature regression, (3) masked object prediction via detected-label classification, (4) cross-modality matching, and (5) image question answering.”_ [(Tan & Bansal, 2019)](https://arxiv.org/pdf/1908.07490.pdf)? If the pre-training tasks are not all the ones from the paper, would that line load pre-trained weights at all and if yes, on what?

Thanks in advance! 🤗

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**: Here is the link to the [hugginface forum](https://discuss.huggingface.co/t/lxmert-pre-trained-model/1195).


Feature Request: Support Longformer 3D attention mask ?

I try to use 3d attntion mask, but failed.


Changing learning rate for BertModelforTokenClassification

Dear all,

I wanted to set a different learning rate for the linear layer and the Bert model for a BertModelforTokenClassification. How can I do so? This change should be done after loading a locally saved BertModelforTokenClassification model.

Thank You.


[s2s] adjust finetune + test to work with fsmt

integrate FSMT into finetune + add the test

Please note I started tweaking `finetune.py` to make explicit conditionals based on model name/type, rather than try/except. Might be a good idea to make `self.config.model_type` more easily accessible by some quick attribute alias or something...

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7230

Once PR https://github.com/huggingface/transformers/pull/7224 is merged I will have to tweak `s/.weights/weight/` in the test (As we are moving to `nn.Embedding` subclass). So perhaps let's hold off on merging this until after #7224 is in and I pushed the adjustment.

@sshleifer


When I updated my transformers to the latest, the previously trained model loaded with an error

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:'3.1.0'
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


LXMERT visual feature extraction during training/fine-tuning phase

# 🚀 Feature request

Thanks a lot for releasing LXMERT model. In the LXMERT model code samples, the visual feature extraction code (using generalized faster-rcnn: [modeling_frcnn](https://github.com/huggingface/transformers/blob/master/examples/lxmert/modeling_frcnn.py)) only in the inference step is given. However, the visual feature extraction during the training phase is not given. For this reason if we use the same code for fine-tuning, it raises NotImplementedError as the visual feature extraction during training is not implemented. Is it possible to share the visual feature extraction during training?


A confusion about mrc model

# ❓ Questions & Help

- I found all of pretrain models using for mrc use softmax to pred start and end probability.

- If a paragraph has multi-answers, softmax not worked, otherwise, adding some postprocess to combine start and end probability to sort that we can choose n-best answer, but the numbert is  fixed for all paragaph.
- Squad 2.0 has unanswerable questions and i found  data processor define start index == end index == 0 to handle this situation.
- So my confusion is why not use sigmoid instead of softmax in the last layer, if i use sigmoid, i can deal with muti-answers or non-answer easily....


[broken] quantization util for CPU

<!-- This line specifies which issue to close after the pull request is merged. -->

+ completely destroys bart
+ Marian off by one or two words, but not faster.

I wonder why!


[save/load model] authorized keys, no save keys, etc.

We already have:

```
   # modeling_bart:
    authorized_missing_keys = [r"final_logits_bias", r"encoder\.version", r"decoder\.version"]
```
Once https://github.com/huggingface/transformers/pull/7224 is merged we will have:

```
class SomeModel(PretrainedFSMTModel):
    base_model_prefix = "model"
    authorized_missing_keys = [
        "model.encoder.embed_positions.weight",
        "model.decoder.embed_positions.weight",
    ]
    keys_to_never_save= [
        "model.encoder.embed_positions.weight",
        "model.decoder.embed_positions.weight",
    ]
```

I'd like to discuss several things:

1. let's pick consistent intuitive names for the group of these class variables - e.g. both ending with `_keys` or starting with `keys_` and use more descriptive mnemonics?

I suggest:

* `keys_to_ignore_on_load`
* `keys_to_ignore_on_save`

---------

2. why is the current implementation for `authorized_missing_keys` uses a regex search?

```
   # modeling_bart:
    authorized_missing_keys = [r"final_logits_bias", r"encoder\.version", r"decoder\.version"]
    [...]
   # modeling_utils:
            if cls.authorized_missing_keys is not None:
                for pat in cls.authorized_missing_keys:
                    missing_keys = [k for k in missing_keys if re.search(pat, k) is None]
```
when a simple direct comparison `k in list` would work just fine? i.e.: (untested)
```
if cls.authorized_missing_keys
    missing_keys = [k for k in missing_keys if k is not in cls.authorized_missing_keys]
```

it'd make the listing of the keys easier to write/read.
```
    authorized_missing_keys = ["final_logits_bias", "encoder.version", "decoder.version"]
```

-------------------

3. I think we may have an issue with `model.` prefix present in some saved model `state_dict`s and lacking in others - or is it a save/load issue where it has the prefix on the way out, but not back in? Note, that `authorized_missing_keys` doesn't have the `model.` prefix. Perhaps that's why the regex was used - to catch w/ and w/o the prefix?

So perhaps the normalization could happen in the core libs (load?) and the models listing special "needs" keys should all either have the prefix or not.

----

Thank you for your input.


[fsmt] build/test scripts

A few more essential building + testing scripts

@sshleifer


[fsmt] Expanding Positional Embeddings

splitting the comment from @sshleifer in https://github.com/huggingface/transformers/pull/7224#pullrequestreview-492155076 into this separate issue, as the PR hasn't introduced this potential issue. He writes:

> The expansion of embeddings may require a bit more care, but the comment below doesn't prevent merging this PR. You can just delete that logic later if it is bad.

# Expanding Positional Embeddings

```
if max_pos > self.weight.size(0): # recompute/expand embeddings if needed
```

> The reason I haven't autoexpanded the bart positional embeddings so far is that I wanted an error for long sequences that the model would translate poorly, instead of just poor performance. But if you can like concatenate a few en-ru examples and see that performance doesn't plummet it would be good. There is also a theoretical O(seq_len^2) theoretical cost associated with passing longer documents through transformers, so we may not want to encourage longer docs/instead write tooling that, for example, uses moses SentenceSplitter to chunk documents, pass them through the model, and rejoin the results correctly. If it just works with the auto expansion hack then I'm all for it.

@stas00: I shall investigate and will follow up when I got a chance to do so


Add "Fine-tune ALBERT for sentence-pair classification" notebook to the community notebooks

Hello,

I'm adding to the community notebooks a tutorial on fine-tuning ALBERT and other BERT-based models for sentence-pair classification.
The main features of this tutorial are :
[1] End-to-end ML implementation (training, validation, prediction, evaluation)
[2] Easy adaptability to your own datasets
[3] Facilitation of quick experiments with other BERT-based models (BERT, ALBERT, ...)
[4] Quick training with limited computational resources (mixed-precision, gradient accumulation, ...)
[5] Multi-GPU execution
[6] Threshold choice for the classification decision (not necessarily 0.5)
[7] Freeze BERT layers and only update the classification layer weights or update all the weights
[8] Reproducible results with seed settings


[s2s] distributed eval allows num_return_sequences > 1

For more complicated pseudolabeling procedures.


[wip] layernorm eps config

+ Allow config.layernorm_eps to control layernorm epsilon for Bart and all it's children.
+ This is consistent with other models, and makes the code more extensible, at the cost of added complexity.


[s2s] add supported architecures to MD

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


[testing doc] @slow has to be last

Found an issue when `@slow` isn't the last decorator (gets ignored!), so documenting this significance.

@sgugger


[testing] when to @slow and when not to? (huge models download)

Looking at the [CI logs](https://app.circleci.com/pipelines/github/huggingface/transformers/12476/workflows/b4ca9141-e992-4c00-95c6-a6ae7fff63b3/jobs/88844) we do have huge models downloaded (i.e. not @slow):
```
Downloading: 100% 1.16G/1.16G [00:52<00:00, 22.3MB/s]
Downloading: 100% 433M/433M [00:08<00:00, 48.4MB/s]s]
Downloading:  43% 369M/863M [00:08<00:10, 45.4MB/s]
```
so it's very inconsistent. Why not have a whole bunch more of tests not be `@slow` then if we are downloading huge files anyway? A lot of those tests are very fast, other than the download overhead. Or, perhaps, those currently doing huge downloads should be `@slow` in first place?

I'm asking since I was told not to run any fsmt tests with the full model unless it's `@slow` (size ~1.1GB). So it's unclear when it's OK to include huge models in the non-slow test suite and when not to.

Also, here is an alternative approach to think about - why not download large weights while other tests not needing them are running? i.e. fork a process early on on CI after pip installs are done and let it cache the models - then they will be ready to be used by the time the tests that need them get to run. This is an unpolished idea, since one needs to figure out how to re-sort the tests so that these large-model tests aren't run first...


very poor performance of Longformer on SQuAD-like question-answering tasks

## Environment info
any environment can easily re-produce the awful results

- `transformers` version: newest
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.6
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: parallel

### Who can help
Longformer/Reformer: @patrickvonplaten

## Information

Model I am using (Longformer):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run examples/run_squad.py with Longformer-base-4096 Model (with fp16 and gradient_checkpointing enabled)

This results in very awful results than BERT, RoBERTa and any pre-trained models...

E.g., I run it on MARCO data, nothing modify the script. Then the BERT and RoBERTa can easily achieved 50+ F1 score on dev set, while Longformer can only reach **18** points! I believe there are severe bugs for the support of Longformer on SQuAD-like QA tasks. Please confirm that.

## Expected behavior

At least superior than RoBERTa, and marginal superior than BERT models


[example/glue] fix compute_metrics_fn for bart like models

Fixes #7247
@sgugger


[example/glue] run_glue compute metrics fail for bart like models

This PR #7126 introduced multiple predictions for trainer. This breaks the `compute_metrics_fn` of `run_glue.py`  for `bart` like models which return multiple predictions.

For `BartForSequenceClassfication` `p.predictions`  is a `tuple`, so following code fails
https://github.com/huggingface/transformers/blob/1d90d0f386af2af52017d51c421e71a51ec94de0/examples/text-classification/run_glue.py#L154

@sgugger


How to get cross attention weights of decoder when using 'encoderdecodermodel'

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
I'm recently building a encoder-decoder model (Bert2Bert) using `encoderdecodermodel`.
But I found that it is really hard to get cross attention weights of the decoder.
The document of this API said the return of the `forward` function will be the following

- loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) – Languaged modeling loss.

- logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) – Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- past_key_values (List[torch.FloatTensor], optional, returned when use_cache=True is passed or when config.use_cache=True) – List of torch.FloatTensor of length config.n_layers, with each tensor of shape (2, batch_size, num_heads, sequence_length, embed_size_per_head)).

- Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see past_key_values input) to speed up sequential decoding.

- decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) – Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

- Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.

- **decoder_attentions** (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) – Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

- Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads.

- encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) – Sequence of hidden-states at the output of the last layer of the encoder of the model.

- encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) – Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

- Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.

- encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) – Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

- Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads.

The `decoder_attentions` in the above only can return the self-attention weight `(batch_size, num_heads, sequence_length, sequence_length)`. However I want the cross attention weight  `(batch_size, num_heads, input_sequence_length, target_sequence_length)`.  I checked the source code of `encoderdecodermodel` and found the decoder is instantiated from the `AutoModelForCausalLM ` class. I also checked the source code of `AutoModelForCausalLM` and found no way to get cross-attention weight.

Is there any method I can get this cross-attention weight?

@patrickvonplaten Thanks very much! Would be really appreciated if you can help me. It bothers me for a while.


[s2s] distributed_eval edge case

two separate invocations of the script can write to _mp at the same time.
Fix: Can use pid for tmp filename or even just a random number.


fsmt tiny model card + script

Related to pretrained model cards

- a tiny model card (this is used in testing only, so only a very brief doc)
- a script that created that model


Enable pegasus fp16 by clamping large activations

<!-- This line specifies which issue to close after the pull request is merged. -->
+ If encoder layer is about to return inf, clamp to a float **near** the boundary of fp16 range
+ This improves rouge for distil-pegasus-xsum-12-12 in fp16 by 0.1 ROUGE 2
+ I cannot think of how this change could make things worse.
+ Other models' metrics are unaffected!


[s2s] distributed_eval.py saves better speed info


KeyError: 'squeezebert' which from model zoo

This error occurs when using the pretrained model from https://huggingface.co/squeezebert/squeezebert-mnli


Add title to model card

Related to pretrained model cards


Create README.md

Related to pretrained model cards


Update the TF models to remove their interdependencies

This PR tries to remove all the interdependencies across the TF models. All the `.modeling_tf_X import TFXX` where `X` is the name of a model, and `XX` is the name of a layer. All the dependent layers have been rewritten directly in the corresponding file model in order to do not have these import anymore. Only the case where a strict usage without modification is allowed such as `TFCamembert` and `TFXLMRoberta`.


[wip] summarization dataset downloader


is_pretokenized -> is_split_into_words

As discussed offline, replacing `is_pretokenized` in `is_split_into_words` because the users were confused by what the argument represents.

Added deprecation warnings in the functions where the code ends, can add more if you feel I have forgotten some!


[Bug Fix] The actual batch_size is inconsistent with the settings.

In the previous version, the generation of **negative examples** was placed in `Datacollator`, which would cause `batch_size` to be **inconsistent** with the setting during training, resulting in **OOM errors**.
Now I move the negative sample generation process to `TextDataset`, although `TextDataset` will need larger storage space and the reading procedure is more time-consuming, the training will not be interrupted due to OOM errors.

In fact, in my own project, I have used the `Datasets` library you developed, which is very impressive, especially for scenarios with large data scales such as pre-training tasks. I am not sure if it's welcomed to use the `Datasets` library by default in `TextDatasetForNextSentencePrediction`. I can provide a version that depends on the the library, and try to use the library when it is available.


TypeError: 'ByteLevelBPETokenizer' object is not callable

I was trying to train a language model by following this link https://huggingface.co/blog/how-to-train , tokenizer trained successfully but when I'm loading calling this trained tokenizer in LineByLineTextDataset, it showing below error

![image](https://user-images.githubusercontent.com/68846939/93577895-3fedbe00-f9ba-11ea-8de7-70e542744dd6.png)


Fixed typo in README

Found a small typo in README.md and fixed it.


trainer.evaluate() aggregates predictions on GPU and causes CUDA out of memory issues for large datasets

## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-112-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

Trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Use the Trainer for evaluation (`.evaluate()`, `.predict()`) on the GPU with BERT with a large evaluation DataSet where the size of the returned prediction Tensors + Model exceed GPU RAM. (In my case I had an evaluation dataset of 469,530 sentences).
2. Trainer will crash with a CUDA Memory Exception

## Expected behavior

- I would expect the predictions in `predict()` or `evaluate()` from each step would be moved to the CPU device (off the GPU) and then concatenated later. However, the tensors are concatenated while still be on the GPU device, and only converted to CPU numpy arrays after the whole dataset has been predicted/evaluated.

This means that for large evaluation datasets you'll run out of CUDA memory.

It also makes it difficult to pick the batch size to optimize the batch size for the GPU, as you need to allow space for not only the model and inputs, but also all the predictions, which can add up when dealing with large evaluation datasets.

- ie. The problem in the trainer code is that the predictions stay on the GPU [here](https://github.com/huggingface/transformers/blob/67d9fc50d917c63cf67281106214e1d9ae018dff/src/transformers/trainer.py#L1315)
- These tensors get concatenated but stay on the GPU [here](https://github.com/huggingface/transformers/blob/67d9fc50d917c63cf67281106214e1d9ae018dff/src/transformers/trainer_utils.py#L143)
- and then the predictions eventually end up on the CPU [here](https://github.com/huggingface/transformers/blob/67d9fc50d917c63cf67281106214e1d9ae018dff/src/transformers/trainer.py#L1343-L1347)

What this means, is that for larger evaluation datasets, *all* the predictions stay on the GPU, which is a function of how long your evaluation dataset is, and that you often run out of GPU RAM.

A work around is to something like this and run the loop yourself, and predict by batch:

``` py
    preds = []
    for i in tqdm(range(0, len(ds_valid), step)):
        ds_valid = KeywordDataset(tokenizer, df_valid[i:i+step], targets)
        batch_preds = trainer.predict(ds_valid)
        preds.append(batch_preds)
        batch_accuracy = (batch_preds.predictions.argmax(-1) == batch_preds.label_ids).mean()

    np_preds = np.concatenate([pred.predictions for pred in preds], axis=0)
    np_label_ids = np.concatenate([pred.label_ids for pred in preds], axis=0)
    acc = (np_preds.argmax(-1) == np_label_ids).mean()
    print('eval_accuracy = ', acc)
```

It would be nice if it defaulted the cpu (so you don't have to worry about it), and that there is another trainer argument to give the device to aggregate predictions that can be used to override that behaviour if it's important for certain use cases.

Thanks for the amazing work on an amazing library. Working with transformers has never been easier due to the hard work of your team!


[draft][s2s]reload-dl

Placeholder: first test whether you can just send `--reload_dataloaders_every_epoch` from the command line!


test fsmt finetuning

Would be interested to know whether this works with finetune.py.
Just add your tiny guy to
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_seq2seq_examples.py#L378
and edit l382 to set task='translation'.

If it doesn't pass, we can investigate, but I expect it to pass.

Happy for either of us to take this one @stas00

(this is behind tiny fsmt in the queue, but writing it down so I don't forget.)


a possible hack for FSMT's SinusoidalPositionalEmbedding peculiarity

(with normal CIs not running USE_CUDA=1 I completely missed testing this, so found one issue with torchscript tests that I need help with.)

We are talking about FSMT - ported fairseq transformers model.

If I understand correctly their `SinusoidalPositionalEmbedding` was designed so that it won't be part of the model params
https://github.com/pytorch/fairseq/blob/master/fairseq/modules/sinusoidal_positional_embedding.py#L25
most likely so that it won't be part of the `state_dict`, and save space in their already huge 3.3GB model dump (well 13GB actually as they use an ensemble of 4 models). I could be wrong about the reason for this design choice.

I had to copy their implementation, and not use Bart's version, since the pretrained weights rely on it, and the positions it produces are different.

So their `SinusoidalPositionalEmbedding`'s `self.weights` is a normal variable (not a buffer and not a `nn.parameter.Parameter`). They create a dummy buffer `self._float_tensor` to hold the `device`. So when `model.to()` is called, `self._float_tensor` gets the right `device`. During `forward`  `self.weights` gets `to(self._float_tensor)` and all is good. So `self.weights` is kind of a ghost variable. Now you see me and now you don't.

This approach works just fine until we get to `torchscript` - in particular 2 common tests:
```
    def test_torchscript_output_attentions(self):
    def test_torchscript_output_hidden_state(self):
```
which blow up under `USE_CUDA=1`, with:
```
Comparison exception:   Expected all tensors to be on the same device,
but found at least two devices, cuda:0 and cpu!
```
Everything is on `cuda:0` but `SinusoidalPositionalEmbedding`'s `self.weights` are on `cpu` still at this point.

The first time it encounters `self.weights`inside `forward`, before it gets a chance to be moved to the device, torchscript blows up. It wants all variables to be on the same device before `forward`.

## Solution 1

So, I solved this problem with the following hack:
```
class FSMTForConditionalGeneration(PretrainedFSMTModel):
    def to(self, *args, **kwargs):
        super().to(*args, **kwargs)
        self.base_model.to(*args, **kwargs)
        return self

class FSMTModel(PretrainedFSMTModel):
    def to(self, *args, **kwargs):
        super().to(*args, **kwargs)
        self.encoder.embed_positions.to(*args, **kwargs)
        self.decoder.embed_positions.to(*args, **kwargs)
        return self

class SinusoidalPositionalEmbedding(nn.Module):
    def to(self, *args, **kwargs):
        super().to(*args, **kwargs)
        self.weights = self.weights.to(*args, **kwargs)
        return self
```
It's absolutely crazy, but it works.

Basically it forwards `model.to()` call to `SinusoidalPositionalEmbedding`'s `self.weights`, via 3 "bridges".

I thought that each torch module got `to()` called but that doesn't seem to be the case, I think it traverses the model structure instead and doesn't call `to` for each module. Hence the 2 classes are involved to bridge it on.

(and there is also `half()` that needs to be dealt with too, since `model.half()` won't get forwarded to this non-parameter variable either.)

## Solution 2

The second solution is to make `SinusoidalPositionalEmbedding`'s `self.weights` a parameter, but then we have to hack save/load to not save/ignore-on-load `model.encoder.embed_positions.*` and `model.decoder.embed_positions.*` keys.

## Solution 3

The third solution is to save the useless weights (useless as they aren't trained and get calculated deterministically).

Perhaps you can think of other solutions.

Thank you.

@sgugger, @patrickvonplaten, @sshleifer, @LysandreJik


FSMT Training scripts

Hi, thank you very much for your implementation and it helps a lot. I want to train the MT models from scratch. Could you provide some instructions about the training details for FSMT? Thanks a lot. @stas00


Suggestion: Better to change the task name of  "sentiment-analysis" to "text-classification"

There is a task named ``sentiment-analysis`` in``SUPPORTED_TASKS`` used in piepelines.
https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L2509

However, it can be used not only for sentiment analysis but also fo more generic tasks.
Therefore, I think it is better to rename it to ``text-classification`` (or add a task named ``text-classification`` for backward compatibility).


The task name sentiment-analysis


Fix a typo


[fsmt] rewrite SinusoidalPositionalEmbedding + USE_CUDA test fixes + new TranslationPipeline test

These changes are in one PR as they all fix problems for `USE_CUDA=1`

* fix a few `USE_CUDA=1` tests that got skipped previously (was missing `.to(device)`)
* rewrite `SinusoidalPositionalEmbedding` to be a normal `nn.Embedding` subclass with a normal `self.weight` param, but exclude this param from being saved with the `state_dict` since it's not trained, but deterministic
* adjust `PreTrainedModel.save_pretrained` to support models that don't want all of their params saved. (needed for fsmt's `SinusoidalPositionalEmbedding`)
* add a new test using `TranslationPipeline` (well, this one is just a new test)

@sshleifer

This includes fixing: https://github.com/huggingface/transformers/issues/7229


[s2s] remove double assert

Sortish sampler works on multigpu!