How to stop wordpiece-tokenizing in BertTokenizer?

Default BertTokenizer may split one word into two parts or three parts, which is harmful to token labeling task because it will make sentence length lager than input length. It is difficult to fuse these parts back to one representation.
So, how to stop this operation in BertTokenizer? Or there are alternative Tokenizers without spliting words?


F1 socre is zero while loss is about 0.12xx when using run_ner.py to fine tuning bert model

## ❓ Questions & Help

I want to use run_ner.py to do the sequence labeling work, and I make a few class of labels(e.g., B-School_name), then I annotate a few data (about 100 annotations for test) from dataset. Finally, I use run_ner.py to do the fine tuning job, but the f1 score and precision are both zero (but the loss is not). Have I got something wrong?


Unlimited sequence length in Bert for QA


Out of Memory (OOM) when repeatedly running large models

## ❓ Any advice for freeing up GPU memory after training a large model (e.g., roberta-large)?

### System Info

```
Platform Linux-4.4.0-1096-aws-x86_64-with-debian-stretch-sid
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0]
PyTorch 1.3.0
AWS EC2 p3.2xlarge (single GPU)
```

My current objective is to run `roberta-large` multiple training jobs sequentially from the same python script (i.e., for a simple HPO search). Even after deleting all of the objects and clearing the cuda cache after the first training job ends I am still stuck with 41% GPU memory usage (as compared to 15% before starting the training process).

I am attaching a reproducible example to get the error:

### Here is the relevant code

```python
show_gpu('Initial GPU memory usage:')
for i in range(2):
    model, optimizer, scheduler = get_training_obj(params)
    show_gpu(f'{i}: GPU memory usage after loading training objects:')
    for epoch in range(1):
        epoch_start = time.time()
        model.train()
        for batch in dp.train_dataloader:
            xb,mb,_,yb = tuple(t.to(params['device']) for t in batch)
            outputs = model(input_ids = xb, attention_mask = mb, labels = yb)
            loss = outputs[0]
            loss.backward()
            optimizer.step()
            scheduler.step()
            optimizer.zero_grad()
    show_gpu(f'{i}: GPU memory usage after training model:')
    del model, optimizer, scheduler, loss, outputs
    torch.cuda.empty_cache()
    torch.cuda.synchronize()
    show_gpu(f'{i}: GPU memory usage after clearing cache:')
```

### Here is the output and full traceback

```
Initial GPU memory usage: 0.0% (0 out of 16130)

0: GPU memory usage after loading training objects: 14.7% (2377 out of 16130)

0: GPU memory usage after training model: 70.8% (11415 out of 16130)

0: GPU memory usage after clearing cache: 41.8% (6741 out of 16130)

1: GPU memory usage after loading training objects: 50.2% (8093 out of 16130)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-7-20a3bdec1bf4> in <module>()
      8         for batch in dp.train_dataloader:
      9             xb,mb,_,yb = tuple(t.to(params['device']) for t in batch)
---> 10             outputs = model(input_ids = xb, attention_mask = mb, labels = yb)
     11             loss = outputs[0]
     12             loss.backward()

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, labels)
    326                                token_type_ids=token_type_ids,
    327                                position_ids=position_ids,
--> 328                                head_mask=head_mask)
    329         sequence_output = outputs[0]
    330         logits = self.classifier(sequence_output)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask)
    179                                                  token_type_ids=token_type_ids,
    180                                                  position_ids=position_ids,
--> 181                                                  head_mask=head_mask)
    182
    183

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask)
    625         encoder_outputs = self.encoder(embedding_output,
    626                                        extended_attention_mask,
--> 627                                        head_mask=head_mask)
    628         sequence_output = encoder_outputs[0]
    629         pooled_output = self.pooler(sequence_output)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask)
    346                 all_hidden_states = all_hidden_states + (hidden_states,)
    347
--> 348             layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
    349             hidden_states = layer_outputs[0]
    350

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask)
    326         attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
    327         attention_output = attention_outputs[0]
--> 328         intermediate_output = self.intermediate(attention_output)
    329         layer_output = self.output(intermediate_output, attention_output)
    330         outputs = (layer_output,) + attention_outputs[1:]  # add attentions if we output them

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, hidden_states)
    298     def forward(self, hidden_states):
    299         hidden_states = self.dense(hidden_states)
--> 300         hidden_states = self.intermediate_act_fn(hidden_states)
    301         return hidden_states
    302

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in gelu(x)
    126         Also see https://arxiv.org/abs/1606.08415
    127     """
--> 128     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
    129
    130 def gelu_new(x):

RuntimeError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 15.75 GiB total capacity; 14.24 GiB already allocated; 8.88 MiB free; 476.01 MiB cached)
```

### Appendix / complete code to reproduce

```python
import platform; print("Platform", platform.platform())
import sys; print("Python", sys.version)
import torch; print("PyTorch", torch.__version__)

from __future__ import absolute_import, division, print_function
import glob
import logging
import os
import time
import json
import random
import numpy as np
import pandas as pd
from random import sample, seed
import torch
from torch.utils.data import DataLoader, RandomSampler, SequentialSampler,TensorDataset
from torch.utils.data.distributed import DistributedSampler
from transformers import RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer
from transformers import DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer
from transformers import BertConfig, BertForSequenceClassification, BertTokenizer
from transformers import AdamW, WarmupLinearSchedule
from transformers import glue_compute_metrics as compute_metrics
from transformers import glue_output_modes as output_modes
from transformers import glue_processors as processors
from transformers import glue_convert_examples_to_features as convert_examples_to_features
import subprocess

params = {
    'num_epochs': 2,
    'warmup_ratio': 0.06,
    'weight_decay': 0.1,
    'adam_epsilon': 1e-6,
    'model_name': 'roberta-large',
    'max_grad_norm': 1.0,
    'lr': 2e-5,
    'bs': 32,
    'device': 'cuda',
    'task': 'cola',
    'data_dir': '/home/ubuntu/glue_data/CoLA',
    'max_seq_length': 50,
    'metric_name': 'mcc',
    'patience': 3,
    'seed': 935,
    'n': -1,
      }

class DataProcessor():
    '''Preprocess the data, store data loaders and tokenizer'''

    _TOKEN_TYPES = {
        'roberta': RobertaTokenizer,
        'distilbert': DistilBertTokenizer,
        'bert': BertTokenizer,
    }

    def __init__(self, params):
        model_type = params['model_name'].split('-')[0]
        assert model_type in self._TOKEN_TYPES.keys()
        self.tok = self._TOKEN_TYPES[model_type]
        self.params = params
        self.processor = processors[self.params['task']]()
        self.output_mode = output_modes[self.params['task']]
        self.label_list = self.processor.get_labels()

    @staticmethod
    def _convert_to_tensors(features):
        all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
        all_attention_mask = torch.tensor([f.attention_mask for f in features], dtype=torch.long)
        all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
        all_labels = torch.tensor([f.label for f in features], dtype=torch.long)
        return TensorDataset(all_input_ids, all_attention_mask, all_token_type_ids, all_labels)

    def _load_examples(self, tokenizer, evaluate):
        if evaluate:
            examples = self.processor.get_dev_examples(self.params['data_dir'])
        else:
            examples = self.processor.get_train_examples(self.params['data_dir'])
            if self.params['n'] >= 0:
                examples = sample(examples, self.params['n'])
        features = convert_examples_to_features(examples,
                                                tokenizer,
                                                label_list=self.label_list,
                                                max_length=self.params['max_seq_length'],
                                                output_mode=self.output_mode,
                                                pad_on_left=False,
                                                pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
                                                pad_token_segment_id=0)

        return self._convert_to_tensors(features)

    def _define_tokenizer(self):
        return self.tok.from_pretrained(self.params['model_name'], do_lower_case=True)

    def load_data(self):
        tokenizer = self._define_tokenizer()
        self.train_data = self._load_examples(tokenizer, False)
        self.valid_data = self._load_examples(tokenizer, True)
        self.train_n = len(self.train_data)
        self.valid_n = len(self.valid_data)
        self.params['total_steps'] = self.params['num_epochs'] * self.train_n
        return self.params

    def create_loaders(self):
        self.train_dataloader = DataLoader(self.train_data, shuffle=True, batch_size=self.params['bs'])
        self.valid_dataloader = DataLoader(self.valid_data, shuffle=False, batch_size=2*self.params['bs'])

dp = DataProcessor(params)
params = dp.load_data()
dp.create_loaders()

def show_gpu(msg):
    """
    ref: https://discuss.pytorch.org/t/access-gpu-memory-usage-in-pytorch/3192/4
    """
    def query(field):
        return(subprocess.check_output(
            ['nvidia-smi', f'--query-gpu={field}',
                '--format=csv,nounits,noheader'],
            encoding='utf-8'))
    def to_int(result):
        return int(result.strip().split('\n')[0])

    used = to_int(query('memory.used'))
    total = to_int(query('memory.total'))
    pct = used/total
    print('\n' + msg, f'{100*pct:2.1f}% ({used} out of {total})')

# ### Running the Training Loop

def get_training_obj(params):
    config = RobertaConfig.from_pretrained(params['model_name'], num_labels=2)
    model = RobertaForSequenceClassification.from_pretrained(params['model_name'], config=config).to(params['device'])
    no_decay = ['bias', 'LayerNorm.weight']
    gpd_params = [
        {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
         'weight_decay': params['weight_decay']},
        {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
         'weight_decay': 0.0}
        ]
    optimizer = AdamW(gpd_params, lr=params['lr'], eps=params['adam_epsilon'])
    warmup_steps = int(params['warmup_ratio'] * params['total_steps'])
    scheduler = WarmupLinearSchedule(optimizer, warmup_steps=warmup_steps, t_total=params['total_steps'])
    return model, optimizer, scheduler

show_gpu('Initial GPU memory usage:')
for i in range(2):
    model, optimizer, scheduler = get_training_obj(params)
    show_gpu(f'{i}: GPU memory usage after loading training objects:')
    for epoch in range(1):
        epoch_start = time.time()
        model.train()
        for batch in dp.train_dataloader:
            xb,mb,_,yb = tuple(t.to(params['device']) for t in batch)
            outputs = model(input_ids = xb, attention_mask = mb, labels = yb)
            loss = outputs[0]
            loss.backward()
            optimizer.step()
            scheduler.step()
            optimizer.zero_grad()
    show_gpu(f'{i}: GPU memory usage after training model:')
    del model, optimizer, scheduler, loss, outputs
    torch.cuda.empty_cache()
    torch.cuda.synchronize()
    show_gpu(f'{i}: GPU memory usage after clearing cache:')
```


GPT-2 XL


Fix CTRL past

Fixes the issue with the re-usable past in CTRL.


[WIP] Adding Google T5 model

Add Google T5 model:
- paper:  "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" https://arxiv.org/abs/1910.10683
- code: https://github.com/google-research/text-to-text-transfer-transformer

The original model makes heavy use of model and data parallelism to scale the training up to a 11B parameters model. It is based on Mesh TensorFlow (https://github.com/tensorflow/mesh).

We will use a simpler version of model parallelism by simply distributing the layers on several GPUs.

# Workflow for including a model from [README.md](https://github.com/huggingface/transformers/blob/master/templates/adding_a_new_model/README.md)

Here an overview of the general workflow:

- [x] add model/configuration/tokenization classes
- [x] add conversion scripts
- [x] add tests
- [x] finalize

Let's details what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-past or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-past or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [x] copy-past or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this scipt to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] add your models and tokenizer to `pipeline.py`
- [x] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation it-self at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.


BART

# 🌟New model addition

## Model description

method for pre-training seq2seq models by de-noising text. BART outperforms previous work on a bunch of generation tasks (summarization/dialogue/QA), while getting similar performance to RoBERTa on SQuAD/GLUE

## Open Source status

* [ ] the model implementation is available: not yet
* [ ] the model weights are available: not yet
* [ ] who are the authors: @yinhanliu @ernamangoyal


Documentation: Updating docblocks in optimizers.py

## Summary
Updating documentation for the optimizer classes. I explicitly state that the scheduler classes are multiplying the optimizer's learning rate by a changing variable.

## Closes
https://github.com/huggingface/transformers/issues/1712


Fix TFXLNet

PR to fix #1692 (type casting attention mask in TF 2.0 version of XLNet)

cc @mfuntowicz


Do not use GPU when importing transformers

This PR addresses the issue #1507 and the main point is to **not** use GPU already at import time of transformers.

This was caused by the use of class variable `dummy_inputs` that initializes TF by using `tf.constant`. This can be prevented simply by makeing `dummy inputs` a local variable since is not used anywhere else but in `__init__`.

commit messages:
* Make dummy inputs a local variable in TFPreTrainedModel.


add progress bar to convert_examples_to_features

It takes considerate amount of time (~10 min) to parse the examples to features, it is good to have a progress-bar to track this


🌟New model addition: VL-BERT

# 🌟New model addition

## Model description

[VL-BERT: PRE-TRAINING OF GENERIC VISUALLINGUISTIC REPRESENTATIONS](https://arxiv.org/pdf/1908.08530.pdf)

> *We introduce a new pre-trainable generic representation for visual-linguistic tasks,
called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both
visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from
the input image. It is designed to fit for most of the visual-linguistic downstream
tasks. To better exploit the generic representation, we pre-train VL-BERT on the
massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better
align the visual-linguistic clues and benefit the downstream tasks, such as visual
commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single
model on the leaderboard of the VCR benchmark.*

<!-- Important information -->

## Open Source status

* [x] the model implementation is available: https://github.com/jackroos/VL-BERT
* [x] the model weights are available: https://github.com/jackroos/VL-BERT/blob/master/model/pretrained_model/PREPARE_PRETRAINED_MODELS.md

## Additional context

The pre-trainable generic representation for visual-linguistic tasks is becoming more and more important.

<!-- Add any other context about the problem here. -->


TFBertForSequenceClassification.from_pretrained ERROR

### code:
model = TFBertForSequenceClassification.from_pretrained('./bert-base-cased-tf_model.h5')

### Error message:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 212, in from_pretrained
    **kwargs
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/site-packages/transformers/configuration_utils.py", line 154, in from_pretrained
    config = cls.from_json_file(resolved_config_file)
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/site-packages/transformers/configuration_utils.py", line 186, in from_json_file
    text = reader.read()
  File "/DATA/disk1/wutianlong/python/miniconda3/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

### Environment
py:3.6
tf:2.0.0

###
I haved download "bert-base-cased-tf_model.h5" "bert-base-cased-vocab.txt" "bert-base-cased-config.json" in the same dir. when I load model "from_pretrained", I got error above. Please help me what 's wrong and how to fix it. Thanks!


Threads running on evaluation?

Hi Transformers :)

Thanks for the wonderful repo. One quick question. I was running examples/test_examples.py and came to see logs printing multiple times while running in cpu. Is there threads spawned ?
snippet from examples/test_examples.py
**
        testargs = ["run_squad.py",
                    "--train_file=./tests_samples/SQUAD/dev-v2.0-small.json",
                    "--predict_file=./tests_samples/SQUAD/dev-v2.0-small.json",
                    "--model_name=bert-base-uncased",
                    "--output_dir=./tests_samples/temp_dir",
                    "--max_steps=10",
                    "--warmup_steps=2",
                    "--do_train",
                    "--do_eval",
                    "--version_2_with_negative",
                    "--learning_rate=2e-4",
                    "--per_gpu_train_batch_size=2",
                    "--per_gpu_eval_batch_size=1",
                    "--overwrite_output_dir",
                    "--seed=42"]
***

Result where i see multiple logs prints
-----
11/05/2019 07:11:33 - INFO - utils_squad -   *** Example ***
*** Example ***
*** Example ***
11/05/2019 07:11:33 - INFO - utils_squad -   unique_id: 1000000013
unique_id: 1000000013
unique_id: 1000000013
11/05/2019 07:11:33 - INFO - utils_squad -   example_index: 13
example_index: 13
example_index: 13
11/05/2019 07:11:33 - INFO - utils_squad -   doc_span_index: 0
doc_span_index: 0
doc_span_index: 0


resize_token_embeddings doesn't work as expected for BertForMaskedLM

## 🐛 Bug

<!-- Important information -->

Model I am using : **BertForMaskedLM**

Language I am using the model on (English, Chinese....): **multilingual**

The problem arise when using:
* [x] the official example scripts: **BertForMaskedLM**

The tasks I am working on is:
* [x] my own task or dataset: **Finetuning with newly added tokens**.

## To Reproduce

Steps to reproduce the behavior:

```python
# Get model
model = BertForMaskedLM.from_pretrained("bert-base-multilingual-cased")

# define input
input = torch.tensor([[1,2,3,4,5]])

# before experiment
print(f"BERT num embeddings before\t: {model.bert.embeddings.word_embeddings.num_embeddings}")
print(f"LM Decoder num embedding before\t: {model.cls.predictions.decoder.out_features}")
print(f"MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}")

# change embedding size for experiment
model.resize_token_embeddings(119547 + 5)
print(f"\nBERT num embeddings after\t: {model.bert.embeddings.word_embeddings.num_embeddings}")
print(f"LM Decoder num embedding after\t: {model.cls.predictions.decoder.out_features}")
# print failuire
try:
  print(f"MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}")
except RuntimeError:
  print(" ---- Forward Pass failed ---- ")
```

This is the output that I see
```
BERT num embeddings before	: 119547
LM Decoder num embedding before	: 119547
MLM Loss			: 18.269268035888672

BERT num embeddings after	: 119552
LM Decoder num embedding after	: 119547
 ---- Forward Pass failed ----
```

If you see the LM decoder did not change the embedding size, hence won't ever predict the new tokens.

The expectation would be that `resize_token_embeddings` handles the FC inside LM decoder.

I tried also using BertConfig

```python
config = BertConfig.from_pretrained("bert-base-multilingual-cased")
config.vocab_size = 119547 + 5
model = BertForMaskedLM.from_pretrained("bert-base-multilingual-cased", config=config)
```

Which naturally results in error while loading `state_dict`

```
RuntimeError: Error(s) in loading state_dict for BertForMaskedLM:
	size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
	size mismatch for cls.predictions.bias: copying a param with shape torch.Size([119547]) from checkpoint, the shape in current model is torch.Size([119554]).
	size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
```


Roberta embeddings comparison

## ❓ Questions & Help
**SYSTEM**
OS: Linux pop-os 5.0.0
Python version: 3.6.8
Torch version: 1.3.0
Transformers version: 2.1.1
I am running this linux VM with the above software versions on a Windows 10 laptop.

I wanted to compare the cosine similarity between elmo and roberta embeddings for two sequences:
```python
from allennlp.commands.elmo import ElmoEmbedder
from allennlp.data.tokenizers.spacy_tokenizer import SpacyTokenizer
from scipy.spatial.distance import cdist
import numpy as np
import pandas as pd
from transformers import RobertaTokenizer, RobertaModel
import torch

seqA = 'How many cars are in the target parking lot'
seqB = 'How many countries are in the continent of Africa'
comparison_df = pd.DataFrame()
sim_list = []

model_types = ['elmo', 'roberta']
for model_type in model_types:
    sim_list = []
    if model_type == 'elmo':
        tokenizer = SpacyTokenizer()
        model = ElmoEmbedder()
        seqA_tokens = np.array([s.text for s in tokenizer.tokenize(seqA)])
        seqB_tokens = np.array([s.text for s in tokenizer.tokenize(seqB)])

        seqA_embeddings = model.embed_sentence(seqA_tokens)
        seqB_embeddings = model.embed_sentence(seqB_tokens)

        seqA_embeds = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))
        seqB_embeds = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))
        Y = cdist(seqA_embeds, seqB_embeds, 'cosine')[0][0]
        cos_sim = 1.0 - Y
        sim_list.append(cos_sim)

    elif model_type == 'roberta':
        tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
        model = RobertaModel.from_pretrained('roberta-large')
        seqA_tokens = torch.tensor(tokenizer.encode(seqA, add_special_tokens=True)).unsqueeze(0)
        seqB_tokens = torch.tensor(tokenizer.encode(seqB, add_special_tokens=True)).unsqueeze(0)

        with torch.no_grad():
            seqA_embeddings = model(seqA_tokens)[0]
            seqB_embeddings = model(seqB_tokens)[0]

        seqA_embeddings = seqA_embeddings.detach().numpy()
        seqA_embeddings = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))

        seqB_embeddings = seqB_embeddings.detach().numpy()
        seqB_embeddings = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))

        Y = cdist(seqA_embeddings, seqB_embeddings, 'cosine')[0][0]
        cos_sim = 1.0 - Y
        sim_list.append(cos_sim)

    comparison_df[model_type] = sim_list
print(comparison_df)
```

The result of the print statement is:
```
      elmo   roberta
0  0.55092  0.996526
```

I find this strange because, despite having the same number of tokens, I would expect the similarity between the two sequences to not be so close to 1 as roberta is showing. My concern is that I am not accessing the embeddings correctly or not pooling correctly (or both), and was wondering if people here wouldn't mind letting me know what I am doing wrong? Thanks in advance!


glue_convert_examples_to_features not working if no task is provided

## 🐛 Bug

<!-- Important information -->

The problem arise when using:
* [X] my own modified scripts: (give details)

The tasks I am working on is:
* [X] my own task or dataset: (give details)

In these lines:
https://github.com/huggingface/transformers/blob/04c69db399b2ab9e3af872ce46730fbd9f17aec3/transformers/data/processors/glue.py#L66-L69

If no task is provided (even if it's a optional parameter), no processor is selected, label_list is still none.

At least an error should be thrown.


loss is nan, for training on MNLI dataset

## 🐛 Bug

Model I am using (Bert, XLNet....):
** Bert **
bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-cased")

Language I am using the model on (English, Chinese....):
** English **

The problem arise when using:
https://medium.com/tensorflow/using-tensorflow-2-for-state-of-the-art-natural-language-processing-102445cda54a
for MNLI

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: ** MNLI **

## To Reproduce

Steps to reproduce the behavior:

1. create  bert_validation_matched_dataset
example = list(bert_validation_matched_dataset.__iter__())[0]
example
({'attention_mask': <tf.Tensor: id=5022668, shape=(64, 128), dtype=int32, numpy=
  array([[1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         ...,
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>,
  'input_ids': <tf.Tensor: id=5022669, shape=(64, 128), dtype=int32, numpy=
  array([[  101,  8147,  1218, ...,     0,     0,     0],
         [  101, 21637,  1103, ...,     0,     0,     0],
         [  101,  1109,  2570, ...,     0,     0,     0],
         ...,
         [  101,  1109,  7271, ...,     0,     0,     0],
         [  101,  7947,  1685, ...,     0,     0,     0],
         [  101,  1130,  1103, ...,     0,     0,     0]], dtype=int32)>,
  'token_type_ids': <tf.Tensor: id=5022670, shape=(64, 128), dtype=int32, numpy=
  array([[0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         ...,
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>},
 <tf.Tensor: id=5022671, shape=(64,), dtype=int64, numpy=
 array([2, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 2, 0, 2, 0, 1,
        2, 1, 1, 2, 2, 1, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 0, 1, 0, 0, 2, 0,
        2, 2, 1, 2, 2, 2, 2, 2, 2, 0, 2, 0, 0, 2, 1, 2, 2, 1, 2, 0])>)

2. create bert_train_dataset such that
example = list(bert_train_dataset.__iter__())[0]
example
({'attention_mask': <tf.Tensor: id=5023289, shape=(32, 128), dtype=int32, numpy=
  array([[1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         ...,
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>,
  'input_ids': <tf.Tensor: id=5023290, shape=(32, 128), dtype=int32, numpy=
  array([[  101,   144,  2312, ...,     0,     0,     0],
         [  101,   107,  1192, ...,     0,     0,     0],
         [  101,   149, 10844, ...,     0,     0,     0],
         ...,
         [  101,  1109,  2350, ...,     0,     0,     0],
         [  101,  1262,  1173, ...,     0,     0,     0],
         [  101,  1105,  1128, ...,     0,     0,     0]], dtype=int32)>,
  'token_type_ids': <tf.Tensor: id=5023291, shape=(32, 128), dtype=int32, numpy=
  array([[0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         ...,
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>},
 <tf.Tensor: id=5023292, shape=(32,), dtype=int64, numpy=
 array([2, 1, 0, 0, 1, 2, 1, 1, 0, 2, 0, 1, 1, 2, 2, 2, 0, 2, 1, 1, 0, 1,
        1, 2, 0, 1, 1, 0, 2, 2, 2, 1])>)

3. Fit the model
bert_history = bert_model.fit(bert_train_dataset, epochs=1, validation_data=bert_validation_matched_dataset)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
The expected behavior is that loss should not be nan.
Fine-tuning BERT on MNLI
      3/Unknown - 4s 1s/step - loss: nan - accuracy: 0.2031

## Environment
colab


Exceeding max sequence length in Roberta

## ❓ Questions & Help
**SYSTEM**
OS: Linux pop-os 5.0.0
Python version: 3.6.8
Torch version: 1.3.0
Transformers version: 2.1.1
I am running this linux VM with the above software versions on a Windows 10 laptop.

<!-- A clear and concise description of the question. -->
I am interested in comparing the embeddings for two pieces of text from the various models in hf-transformers using some metric (like cosine similarity). I am running into an issue where one piece of text is exceeding the max sequence length for roberta (899 > 512). I was wondering what the best work around for this would be?

```python
from transformers import RobertaTokenizer, RobertaModel
import torch

tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
model = RobertaModel.from_pretrained('roberta-large')
seqA = 'Five college students take time off to spend a peaceful vacation in a remote cabin. A book and audio tape is discovered, and its evil is found to be powerful once the incantations are read out loud. The friends find themselves helpless to stop the evil as it takes them one by one, with only one survivor left with the evil dead and desperately tries to fight to live until morning.'
seqA_tokens = torch.tensor(tokenizer.encode(seqA)).unsqueeze(0)
seqA_embeddings = model(seqA_tokens)[0]

seqB = 'Five Michigan State University students—Ash Williams, his girlfriend, Linda; Ashs sister, Cheryl; their friend Scott; and his girlfriend Shelly—vacation at an isolated cabin in rural Tennessee. Approaching the cabin, the group notices the porch swing move on its own but suddenly stop as Scott grabs the doorknob. While Cheryl draws a picture of a clock, the clock stops, and she hears a faint, demonic voice tell her to "Join us". Her hand becomes possessed, turns pale and draws a picture of a book with a demonic face on its cover. Although shaken, she does not mention the incident. When the cellar trapdoor flies open during dinner, Shelly, Linda, and Cheryl remain upstairs as Ash and Scott investigate. They find the Naturan Demanto, a Sumerian version of the Egyptian Book of the Dead, along with an archaeologists tape recorder. Scott and Ash joke around with the items and take them upstairs. Scott plays a tape of incantations that resurrect a demonic entity. Cheryl yells for Scott to turn off the tape recorder, and a tree branch breaks one of the cabins windows. Later that evening, an agitated Cheryl goes into the woods to investigate strange noises. She gets attacked, stripped, pinned to the ground, and raped by demonically possessed trees. When she manages to escape and returns to the cabin bruised and anguished, Ash agrees to take her back into town, only to discover that the bridge to the cabin has been destroyed. Cheryl panics as she realizes that they are now trapped and the demonic entity will not let them leave. Back at the cabin, Ash listens to more of the tape, learning that the only way to kill the entity is to dismember a possessed host. As Linda and Shelly play Spades, Cheryl correctly calls out the cards, succumbs to the entity, and levitates. In a raspy, demonic voice, she demands to know why they disturbed her sleep and threatens to kill everyone. After she falls to the floor, the group checks on her; Cheryl stabs Linda in the ankle and throws Ash into a shelf. Scott knocks Cheryl into the cellar and locks her inside. Everyone fights about what to do. Shelly becomes paranoid upon seeing Cheryls demonic transformation. She lies down in her room but is drawn to look out of her window, where a demon crashes through and attacks her. Shelly becomes a Deadite and scratches Scotts face. Scott throws her into the fireplace, briefly burning Shellys face. As she attacks him again, Scott bisects part of her wrist with a knife and then she bites off her own mangled hand. Scott stabs her in the back with a Sumerian dagger, apparently killing her. When she reanimates, Scott dismembers her with an axe and buries the remains. Shaken by the experience, he leaves to find a way back to town. He shortly returns mortally wounded; he dies while warning Ash that the trees will not let them escape alive. When Ash checks on Linda, he is horrified to find that she has become possessed. She attacks him, but he stabs her with a Sumerian dagger. Unwilling to dismember her, he buries her instead. She revives and attacks him, forcing him to decapitate her with a shovel. Her headless body bleeds on his face as it tries to rape him, but he pushes it off and retreats to the cabin. Back inside, Ash is attacked by Cheryl, who has escaped the cellar, and the reanimated Scott. He shoots Cheryl several times, gouges Scotts eyes out, and pulls out a branch lodged in Scotts stomach, causing him to bleed out. The Deadites attack, bite, and beat Ash with a fire iron. Ash throws the Naturan Demanto into the fireplace, and the Deadites stop their attack. As the book burns, Scott, Cheryl, and the book gruesomely decompose. Demonic hands protrude from both corpses, and Cheryls decomposed body falls and splatters in front of Ash, leaving him covered in her and Scotts entrails. He hears a voice say "Join Us" but relaxes when it dies away. As day breaks, Ash stumbles outside. Before he can leave, an unseen entity rapidly trails the forest and runs through the cabin, breaking the cabin doors and attacks him from behind.'
seqB_tokens = torch.tensor(tokenizer.encode(seqB)).unsqueeze(0)
seqB_embeddings = model(seqB_tokens)[0]

#code to compare seqA and seqB embeddings using cosine similarity
```

Here is the error:
```python
Token indices sequence length is longer than the specified maximum sequence length for this model (899 > 512). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
  File "/model_embeddings_test.py", line 48, in <module>
     seqB_embeddings = model(seqB_tokens)[0]
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/transformers/transformers/modeling_bert.py", line 692, in forward
    embedding_output = self.embeddings(input_ids, position_ids=position_ids, token_type_ids=token_type_ids)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/transformers/transformers/modeling_roberta.py", line 60, in forward
    position_ids=position_ids)
  File "/transformers/transformers/modeling_bert.py", line 170, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /opt/conda/conda-bld/pytorch_1550796191843/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191
```


GPT2 text generation repeat

## ❓ Questions & Help
**SYSTEM**
OS: Linux pop-os 5.0.0
Python version: 3.6.8
Torch version: 1.3.0
Transformers version: 2.1.1
I am running this linux VM with the above software versions on a Windows 10 laptop.

<!-- A clear and concise description of the question. -->
I am running the following code:
```python
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

sentence = 'Natural language processing tasks are typically approached with'
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
context_tokens = tokenizer.encode(sentence, add_special_tokens=False)
context = torch.tensor(context_tokens, dtype=torch.long)
num_samples = 1
context = context.unsqueeze(0).repeat(num_samples, 1)
generated = context

model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
length = 20
with torch.no_grad():
    for jj in range(5):
        for _ in range(length):
            outputs = model(generated)
            next_token_logits = outputs[0][:, -1, :]
            next_token = torch.argmax(next_token_logits, dim=-1).unsqueeze(-1)
            generated = torch.cat((generated, next_token), dim=1)

out = generated
out = out[:, len(context_tokens):].tolist()
for o in out:
    text = tokenizer.decode(o, clean_up_tokenization_spaces=True)
```
What I was noticing was that GPT2 starts to produce repetitive text (see below) with this approach. I am not sure the best way to prevent this from happening and was wondering if others had any ideas? Thank you in advance!

**OUTPUT**
```
a single task, such as a word search, and the task is then repeated. The task is then repeated for each word in the search.

The task is then repeated for each word in the search. The task is then repeated for each word in the search. The task is then repeated for each word in the search. The task is then repeated for each word in the search. The task is then repeated for each word in the search. The task is then repeated for each word in
```


Fix encode_plus

Add options to control more precisely the output of `encode_plus`.

All the outputs that can't be ingested by a model as deactivated by default.
`token_type_ids` can be ingested by most models and if thus activated by default but can be turned off.

Fix #1532 among others


Fix #1623

Make use of the `--cache_dir` argument in all the examples that include it.

cc @VictorSanh (distillation script) @LysandreJik (all the other scripts)


BUG for XLNet: Low GPU usage and High CPU usage, very low running speed!

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: XLNet used as encoder for seq2seq generation

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Centos 7.6
* Python version: 3.6
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? V100, 16G
* Distributed of parallel setup ? Use multiprocessing for multi GPU training
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Add common getter and setter for input_embeddings & output_embeddings

This PR adds two attributes `input_embeddings` and `output_embeddings` as common properties for all the models.

Simpler to write weights tying.

Also superseed #1598.

cc @rlouf


run_generation.py Runtime error

## ❓ Questions & Help

**Hi everyone,**
I'm  currently trying to execute the run_generation example script.
 However I can not get it to work. Tried to reinstall and played around with the parameters. However the error I get is always the same:

**I'm running the example from the website:**

_python run_generation.py \
    --model_type=gpt2 \
    --model_name_or_path=gpt2_

**It allocates resources etc. However when it is starting to generate I get this error:**

_Namespace(device=device(type='cuda'), length=20, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_samples=1, padding_text='', prompt='', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')
  0%|                                                                                | 0/20 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_generation.py", line 264, in <module>
    main()
  File "run_generation.py", line 249, in main
    device=args.device,
  File "run_generation.py", line 142, in sample_sequence
    outputs = model(**inputs)  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet/CTRL (cached hidden-states)
  File "/home/gstein/myPython/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/gstein/Executables/transformers/transformers/modeling_gpt2.py", line 533, in forward
    head_mask=head_mask)
  File "/home/gstein/myPython/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/gstein/Executables/transformers/transformers/modeling_gpt2.py", line 373, in forward
    input_ids = input_ids.view(-1, input_shape[-1])
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0]_

## Environment

* OS: Ubuntu 16.04.3
* Python version: 3.5.2
* Torch version 1.0
* cloned from Git 4. November 2019:
* Using GPU: Yes/No

##Thoughts?
Does anyone else encounter this behavior?
Or even better: Does somebody has a solution for it?)


Can we fine tune GPT2 using multiple inputs?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hello. I don't think this is possible looking at the code, but to make sure is it possible to use multiple input texts to fine tune GPT-2? For example, I have 5 news articles. Do I submit them all in one document or should I separate it out? I feel like the latter makes more sense especially if the articles are all written differently.

Thanks!


Hello, how to upload a .ckpt file in TFBertForSequenceClassification?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I find the TFBertForSequenceClassification only can upload .h5 file


Retaining unknown token behaver consistency in tokenizer for BERT and XLNET

I found the BERT tokenizer and XLNET tokenizer behave difference.
for example, `"His name is 燚"`
In BERT, it's tokenized as: `['his', 'name', 'is', '[UNK]']`,
but in XLNET, it's tokenized as: `['▁His', '▁name', '▁is', '▁', '燚']`
The difference may adverse to the uniform training frame for BERT and XLNET, and trouble the length-based processing.


add qa and result


I found the BERT tokenizer and XLNET tokenizer behave difference.
for example, `"His name is 燚"`
In BERT, it's tokenized as:` ['his', 'name', 'is', '[UNK]'],`
but in XLNET, it's tokenized as:` ['▁His', '▁name', '▁is', '▁', '燚']`
The difference may adverse to the uniform training frame for BERT and XLNET, and trouble the length-based processing.


How to train from scratch

I would like to train the model from scratch.
How can I drop the trained weight? using the same architecture for Gpt2


How to mask lm_labels and compute loss? --- Finetune gpt2: masking the lm_labels with '-1' and padding increase the perplexity a lot!

Hi,

I wanted to finetune gpt2 in a seq2seq format. For that I followed the same approach used for convAI2 explained [here](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313).
I followed two steps and each step highly increase the ppl.
1) first I masked part of the lm_labels (with '-1') which I didn't wanted to calculate loss for. Doing that I realized the ppl increased but then I also try second step below
2) I padded my input to a maximum length 256 and masked the padded indices as well. Doing that increases the valid ppl (while training) starting from ```perplexity = tensor(6.8035e+10)``` rather than ```perplexity = tensor(17.67)``` when I don't mask lm_labels for padded indices and parts of input sequence that I don't want to compute loss for.
This ppl difference is huge and I am not sure if I have to taken care of sth else beside just masking lm_labels with '-1'?

Here is how I modify ```run_lm_finetuning.py``` script to pad my samples to a same length:
```
def pad(x, padding, padding_length=128):
    return x + [padding] * (padding_length - len(x))

class TextDataset(Dataset):
    def __init__(self, tokenizer, file_path='train', block_size=512):
        assert os.path.isfile(file_path)
        directory, filename = os.path.split(file_path)
        cached_features_file = os.path.join(directory, 'cached_lm_padded' + str(block_size) + '_' + filename)

        if os.path.exists(cached_features_file):
            logger.info("Loading features from cached file %s", cached_features_file)
            with open(cached_features_file, 'rb') as handle:
                self.examples = pickle.load(handle)
        else:
            logger.info("Creating features from dataset file at %s", directory)

            self.examples = []
            with open(file_path, encoding="utf-8") as f:
                text = f.readlines()

            text = [i.strip() for i in text]
            for sent in text:
                tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(sent))

                if len(tokenized_text) > block_size:  # Truncate in block of block_size
                    self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[:block_size]))
                else:
                    self.examples.append(pad(tokenized_text, tokenizer.convert_tokens_to_ids('<pad>'), block_size))
            logger.info("Saving features into cached file %s", cached_features_file)
            with open(cached_features_file, 'wb') as handle:
                pickle.dump(self.examples, handle, protocol=pickle.HIGHEST_PROTOCOL)

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, item):
        return torch.tensor(self.examples[item])
```

And this is how I mask the lm_labels, given labels =input_ids and start and end being my desired indices to mask all indices before start and after end (exclusive):

```
def mask_intervals(labels, start, end):
    x = labels.clone()

    for i in range(x.shape[0]):
        start_index = (labels[i, :] == start).nonzero()
        end_index = (labels[i, :] == end).nonzero()

        x[i, :start_index + 1] = -1
        x[i, end_index:] = -1

    return x
```

I want to know if masking with '-1' suffices or should I change some other part of the code regarding loss computation?


Scheduler documentation blocks subtly wrong

## Summary
Hi, I noticed that in `optimization.py` the way many of the schedulers describe the learning rate is slightly wrong.
For example, `WarmupLinearSchedule` says
```
        Linearly increases learning rate from 0 to 1 over `warmup_steps` training steps.
        Linearly decreases learning rate from 1. to 0. over remaining `t_total - warmup_steps` steps.
```
It actually multiplies the learning rate (set in the optimizer) by this amount. I understand that this is probably implied, but I think the documentation could be a bit clearer on this aspect!

Many thanks,
Dom


transformers module doesn't work with torch compiled on Cuda 10.0?

Hi,

I have been using this repo for long time and it was working okay. Very recently it gave me ```ModuleNotFoundError: No module named 'transformers'``` error and I had to reinstall it from source.
It seems like during installation my torch version (or the Cuda it is compiled on) got updated.
I reinstall my PyTorch with cudatoolkit=10.0 since I have Cuda 10.0 and I cannot update it now.
But I realized doing that, I am not able to import transformers again.

Any idea what should I do?


CTRL does not react to the "seed" argument

Using latest master branch and run_generation.py.
When passing different seeds to GPT-2 it produces different results, as expected,
however CTRL does not.

**GPT-2**
python run_generation.py **--seed=42** --model_type=gpt2 --length=10 --model_name_or_path=gpt2 --prompt="Transformers movie is"
**>>> a great example of bringing early, action-packed**

python run_generation.py **--seed=22** --model_type=gpt2 --length=10 --model_name_or_path=gpt2 --prompt="Transformers movie is"
**>>> a lot of fun. It's a decent one**

**CTRL**
python run_generation.py -**-seed=42** --model_type=ctrl --length=10 --model_name_or_path=ctrl --temperature=0 --repetition_penalty=1.2 --prompt="Links Transformers movie is"
**>>> a big deal for the company, and it has been**

python run_generation.py **--seed=22** --model_type=ctrl --length=10 --model_name_or_path=ctrl --temperature=0 --repetition_penalty=1.2 --prompt="Links Transformers movie is"
**>>> a big deal for the company, and it has been**


Fixing mode in evaluate during training

This fixs the error of not passing mode while using the option of evaluate_during_training option.


The id of the word obtained by tokenizer.encode does not correspond to the id of the word in vocab.txt

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
hello，I encountered a problem，code show as below.
tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')
input_ids=torch.LongTensor(tokenizer.encode("[CLS] the 2018 boss nationals [SEP]"))
print(input_ids)
The result is:tensor([  101,  1996,  2760,  5795, 10342,   102]) , but In the voab.txt, the id of "the" word is 8174, not 1996. Why is this the reason for the berttokenizer token dictionary error?


bugs with run_summarization script

Hi
The evaluation part of this script is missing, I tried to complete it myself but this code does not generation proper string during decoding part for me and does not converege, encoder and decoder weights are not tied as the paper, could you please add the generated text as the output of evaluation code, also please complete the evaluation code, maybe you could use some codes in run_generation.py for this, thanks


Regression Loss

I want `BertForSequenceClassification` to compute regression loss. I load the pre-trained model through
```
config = BertConfig.from_pretrained('bert-base-uncased')
config.num_labels = 1  # Is it proper to set the num_label in this way?
model = BertForSequenceClassification(config)
```

How shall I change the num_labels properly?

Thank you!


unable to import from utils_squad

Hi,
    I am using BertForQuestionAnswering on colab and I have installed
`
!pip install transformers
!pip install pytorch-transformers
`
when I import

` from utils_squad import (read_squad_examples, convert_examples_to_features) `

I get the following error:

ModuleNotFoundError: No module named 'utils_squad'

any solution to this?
Greetings!


## ❓ Questions & Help

Recently I read a tutorial
https://medium.com/tensorflow/using-tensorflow-2-for-state-of-the-art-natural-language-processing-102445cda54a
which you can also see in this notebook
https://colab.research.google.com/drive/16ClJxutkdOqXjBm_PKq6LuuAuLsInGC-

In this tutorial, MRPC dataset is used.

I changed the dataset from MRPC to MNLI and you can check the changes in this notebook, in the corresponding code cells.
https://colab.research.google.com/drive/1mzYkrAW5XUwey4FJIU9SN0Q0RSn0tiDb

with MNLI dataset, I see the following issue (\*) :
 print("Fine-tuning BERT on MRPC")
bert_history = bert_model.fit(bert_train_dataset, epochs=3, validation_data=bert_validation_dataset)
 Fine-tuning BERT on MNLI
Epoch 1/3
    352/Unknown - 511s 1s/step - loss: nan - accuracy: 0.3416
(\*) You can see here that the loss is nan

with MRPC the corresponding output is:
print("Fine-tuning BERT on MRPC")
bert_history = bert_model.fit(bert_train_dataset, epochs=3, validation_data=bert_validation_dataset)
Fine-tuning BERT on MRPC
Epoch 1/3
     15/Unknown - 44s 3s/step - loss: 0.6623 - accuracy: 0.6183 ```

The only differences that I see in those two tutorials is the following: For MNLI there is a match and mismatch validation datasets, and I provide the label_list=['0', '1', '2'] in ``` glue_convert_examples_to_features ```

Could someone help me why this issue occurs?
Thanks


Add `model.train()` line to ReadMe training example

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I added `model.train()` line to ReadMe training example, at this section
https://github.com/huggingface/transformers#optimizers-bertadam--openaiadam-are-now-adamw-schedules-are-standard-pytorch-schedules

```
for batch in train_data:
    model.train()
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()
```

In the API [ https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained ] it says

>The model is set in evaluation mode by default using model.eval() (Dropout modules are deactivated) To train the model, you should first set it back in training mode with model.train()

And I see this is the case for every example in the 'examples' folder, such as this one

https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L227

```
        for step, batch in enumerate(epoch_iterator):
            inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
            inputs = inputs.to(args.device)
            labels = labels.to(args.device)
            model.train()
            outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
            loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
```

There is a `model.train()` in each training loop, before the loss/output line.

Some may develop their training loops based on the readme example, inadvertently deactivating that dropout layers during training.


Interpretation of output from fine-tuned BERT for Masked Language Modeling.

## ❓ Questions & Help
Hi,

I need some help with interpretation of the output from a fine-tuned bert model.
Here is what I have done so far:
I used `run_lm_finetuning.py` to fine-tune 'bert-base-uncased' model on my domain specific text data.

I fine-tuned the model for 3 epochs, and I get the following files after fine-tuning the process:

- config.json
- pytorch_model.bin
- training_args.bin

along with some other files such as :
- special_tokens_map.json
- tokenizer_config.json
- vocab.txt

I am loading this fine-tuned model using the following code:
````
config_class, model_class, tokenizer_class = BertConfig, BertForMaskedLM, BertTokenizer

config = config_class.from_pretrained('path/to/config/dir/config.json')
self.text_feature_extractor = model_class.from_pretrained('path/to/fine-tuned-model/pytorch_model.bin',
                                            from_tf=bool('.ckpt' in 'path/to/fine-tuned-model/pytorch_model.bin'),
                                            config=config)
````

And get output from the model like this:
`raw_out_q = self.text_feature_extractor(q)
`
where q is a padded batch of shape B, SEQ_L tokenized by pre-trained BertTokenizer with the following line of code:
`sentence_indices = self.tokenizer.encode(sentence, add_special_tokens=True)`

The output from the fine-tuned model is a tuple of 2 objects:
First one is of shape `(B, SEQ_L, 30522)`
The second object is however another tuple of 13 tensors: Each tensor is of shape` (B, SEQ_L, 768)
`

Now here are my questions:
How should I interpret this output coming from the model?
For the first tensor, I know vocab_size for bert_base_uncased was 30522, but what does it represent here?
For the second tuple, do these 13 tensors represent output from each bert layer?

For each sentence, I want a sentence-level feature and word-level features. I am aware that sentence-level feature is usually the first position in the sequence (against the "[CLS]" token), but any clarification will be super helpful and much appreciated.

Thanks!
<!-- A clear and concise description of the question. -->


When I uesd gpt2, I got a error

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
`  File "test.py", line 92, in <module>
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 282, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 411, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/site-packages/transformers/tokenization_gpt2.py", line 122, in __init__
    self.encoder = json.load(open(vocab_file, encoding="utf-8"))
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ruiy/anaconda3/envs/pytorch101/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 295508 (char 295507)`


Best practices for Bert passage similarity. Perhaps further processing Bert vectors; Bert model inside another model

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I'm trying develop Bert passage similarity, specifically question/ answer-retrieval. The architecture is pooling bert contextualized embeddings for passages of text, and then doing cos similarity. Basically the architecture is as it's described here

https://github.com/re-search/DocProduct#architecture

Unless I skipped over something, as far as I can tell, there isn't an established way to do this in the Transformer model. It looks like I have to take the Bert output, and do further processing on it. So in this case, Bert would be a component of a larger model.

This is the best I could come up with up some prototype code.

```
!pip install transformers

import torch
from transformers import BertModel, BertTokenizer
from transformers import AdamW
import torch.nn as nn
import torch.nn.functional as F

# Overall model, with Bert model inside it
class OverallModel(nn.Module):
    def __init__(self):
        super(OverallModel, self).__init__()
        self.tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
        self.bertModel = BertModel.from_pretrained('bert-base-uncased')
        self.lossFunction = nn.BCEWithLogitsLoss( reduction = 'none' )

    def forward(self, queries, docs, targets ):
        #Set bert model to train mode
        self.bertModel.train()

        #tokenize queries and pad to maximum length
        queriesT = [self.tokenizer.encode(piece, add_special_tokens=True) for piece in queries]
        widthQ = max(len(d) for d in queriesT)
        queriesP = [d + [0] * (widthQ - len(d)) for d in queriesT]
        queriesTen= torch.tensor(queriesP)

        #tokenize answers and pad to maximum length
        docsT = [ self.tokenizer.encode(piece, add_special_tokens=True) for piece in docs]
        widthD = max(len(d) for d in docsT)
        docsP = [d + [0] * (widthD - len(d)) for d in docsT]
        docsTen= torch.tensor(docsP)

        #Get contextualized Bert embeddings
        bertEmbedsQ = self.bertModel(queriesTen)[0]
        bertEmbedsA = self.bertModel(docsTen)[0]

        #Mean pool embeddings for each text, and normalize to unit vectors
        bertEmbedsQPooled = torch.mean(bertEmbedsQ, 1)
        bertEmbedsAPooled = torch.mean(bertEmbedsA, 1)
        bertEmbedsQPN = F.normalize(bertEmbedsQPooled, p=2, dim=1)
        bertEmbedsAPN = F.normalize(bertEmbedsAPooled, p=2, dim=1)

        #Take dot products of query and answer embeddings, calculate loss with target.
        dotProds = torch.bmm(bertEmbedsQPN.unsqueeze(1), bertEmbedsAPN.unsqueeze(2) ).squeeze()
        indLoss = self.lossFunction( dotProds, targets )
        finalLoss = torch.mean(indLoss)

        return finalLoss

QAmodel = OverallModel()
optimizer = AdamW(QAmodel.parameters())
optimizer.zero_grad()

#Prepare inputs and target
queriesI = ['What do cats eat', 'where do apples come from', 'Who is Mr Rogers', 'What are kites']
docsI = ['they eat catfood', 'from trees' , 'President of Iceland', 'used for diving']
targets = torch.tensor( [1, 1, 0 ,0] , dtype = torch.float32 )

#Caculate loss, gradients, and update weights
loss = QAmodel.forward(queriesI, docsI, targets)
loss.backward()
optimizer.step()
```

This approach seems to be working well. The gradients are flowing all the way through, the weights are updating, etc.

Is this code the best way to train Bert passage similarity.


Why do I get 13 hidden layers?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,

When I try to type
```
model = TFRobertaForSequenceClassification.from_pretrained('roberta-base', output_hidden_states=True)
hidden_layers = model([input])

```

The length of hidden_layers is 13. But base model should have 12 layers, right?
And if I want to get the last hidden layer, should I use "hidden_layers[-1]"?

Thanks in advance!


add_tokens() leading to wrong behavior

Hey folks, I am using GPT2Tokenizer and tried add_tokens() below. But it gives unintended behavior and this is because of #612 in tokenization_utils.py
                sub_text = sub_text.strip()
As you see, dollars is split because of space removal. Could someone look into it please.

```
t1 = GPT2Tokenizer.from_pretrained('gpt2')
t1.tokenize('my value in dignums dollars')
```
Output - ['my', 'Ġvalue', 'Ġin', 'Ġdign', 'ums', '**Ġdollars**']

```
t2 = GPT2Tokenizer.from_pretrained('gpt2')
t2.add_tokens(['dignums'])
t2.tokenize('my value in dignums dollars')
```
Output - ['my', 'Ġvalue', 'Ġin', 'dignums', **'d', 'oll', 'ars'**]


PPLM (squashed)

Update: not the case anymore as #1695 was merged.

~~This also contains https://github.com/huggingface/transformers/pull/1695 to make it easy to test in a stand-alone way, but those changes won't be in the final commit~~


Invalid argument error with TFRoberta on GLUE

## 🐛 Bug

I am unable to get TFRoberta working on the GLUE benchmark.

Model I am using (Bert, XLNet....): TFRoberta

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: *MRPC*
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Copy the following script into a file name `run_glue_task_tf.py`:
```python
import tensorflow as tf
import tensorflow.keras as keras

import tensorflow_datasets.public_api as tfds

import transformers

from absl import app
from absl import flags
from absl import logging

FLAGS = flags.FLAGS
flags.DEFINE_multi_string("tasks", None, "One or more tasks to be used for pretraining")

flags.DEFINE_integer('num_epochs', 3, 'Number of epochs to train')
flags.DEFINE_integer('batch_size', 32, 'Batch size to use for training')
flags.DEFINE_integer('eval_batch_size', 64, 'Batch size to use when evaluating validation/test sets')
flags.DEFINE_boolean('use_xla', False, 'Enable XLA optimization')
flags.DEFINE_boolean('use_amp', False, 'Enable AMP optimization')
flags.DEFINE_integer('max_seq_len', 128, 'Maximum sequence length')
flags.DEFINE_string('model_name', 'bert-base-cased', 'Name of pretrained transformer model to load')

def configure_tf():
    logging.info(('Enabling' if FLAGS.use_xla else 'Disabling') + ' XLA optimization')
    tf.config.optimizer.set_jit(FLAGS.use_xla)
    logging.info(('Enabling' if FLAGS.use_xla else 'Disabling') + ' auto mixed precision (AMP)')
    tf.config.optimizer.set_experimental_options({'auto_mixed_precision': FLAGS.use_amp})

def get_model():
    logging.info('Loading pre-trained TF model from %s', FLAGS.model_name)
    model = transformers.TFAutoModelForSequenceClassification.from_pretrained(FLAGS.model_name)

    opt = keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
    if FLAGS.use_amp:
        logging.debug('Enabling loss scaling')
        opt = keras.mixed_precision.experimental.LossScaleOptimizer(opt, 'dynamic')

    loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True)

    metric = keras.metrics.SparseCategoricalAccuracy('accuracy')

    model.compile(optimizer=opt, loss=loss, metrics=[metric])

    return model

def run_task(task: str, model: keras.Model, tokenizer):
    data, info = tfds.load(task, with_info=True)

    if task.startswith('glue'):
        glue_task = task[len('glue/'):]

        def load_features(split: tfds.core.splits.NamedSplit):
            logging.debug('Converting %s.%s to features', task, split)
            is_xlnet: bool = 'xlnet' in model.name.lower()
            return transformers.glue_convert_examples_to_features(examples=data[split],
                                                                  tokenizer=tokenizer,
                                                                  max_length=FLAGS.max_seq_len,
                                                                  output_mode=transformers.glue_output_modes[glue_task],
                                                                  pad_on_left=is_xlnet,  # Pad on the left for XLNet
                                                                  pad_token=tokenizer.convert_tokens_to_ids(
                                                                      [tokenizer.pad_token])[0],
                                                                  pad_token_segment_id=4 if is_xlnet else 0,
                                                                  task=glue_task)

        train = load_features(tfds.Split.TRAIN)
        valid = load_features(tfds.Split.VALIDATION)
    else:
        raise ValueError('Unsupported task: %s' % task)

    train = train.shuffle(128).batch(FLAGS.batch_size).repeat(FLAGS.num_epochs)
    valid = valid.batch(FLAGS.eval_batch_size).take(FLAGS.eval_batch_size)

    logging.info('Training %s on %s...', model.name, task)
    history = model.fit(x=train, validation_data=valid)

    print(task, 'Performance:')
    print(history)

def main(argv):
    del argv  # Unused.

    logging.debug('Loading tokenizer from %s', FLAGS.model_name)
    tokenizer = transformers.AutoTokenizer.from_pretrained(FLAGS.model_name)
    model = get_model()

    for task in FLAGS.tasks:
        print('-' * 20, task, '-' * 20)
        run_task(task, model, tokenizer)

if __name__ == '__main__':
    app.run(main)
```
2. Run the following command:
```bash
$ python run_glue_task_tf.py --tasks glue/mrpc --model_name roberta-base
```
3. Experience the following error:
```
I1101 14:04:14.134916 46912496418432 run_pretraining.py:76] Training tf_roberta_for_sequence_classification on glue/mrpc...
/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification/roberta/pooler/dense/bias:0'] when minimizing the loss.
W1101 14:04:25.996359 46912496418432 optimizer_v2.py:1029] Gradients do not exist for variables ['tf_roberta_for_sequence_classification/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification/roberta/pooler/dense/bias:0'] when minimizing the loss.
/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:Gradients do not exist for variables ['tf_roberta_for_sequence_classification/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification/roberta/pooler/dense/bias:0'] when minimizing the loss.
W1101 14:04:36.157998 46912496418432 optimizer_v2.py:1029] Gradients do not exist for variables ['tf_roberta_for_sequence_classification/roberta/pooler/dense/kernel:0', 'tf_roberta_for_sequence_classification/roberta/pooler/dense/bias:0'] when minimizing the loss.
2019-11-01 14:04:45.689286: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: indices[28,20] = 1 is not in [0, 1)
         [[{{node tf_roberta_for_sequence_classification/roberta/embeddings/token_type_embeddings/embedding_lookup}}]]
2019-11-01 14:04:45.900194: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: indices[28,20] = 1 is not in [0, 1)
         [[{{node tf_roberta_for_sequence_classification/roberta/embeddings/token_type_embeddings/embedding_lookup}}]]
         [[Shape/_8]]
      1/Unknown - 28s 28s/stepTraceback (most recent call last):
  File "run_pretraining.py", line 96, in <module>
    app.run(main)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "run_pretraining.py", line 92, in main
    run_task(task, model, tokenizer)
  File "run_pretraining.py", line 77, in run_task
    history = model.fit(x=train, validation_data=valid)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 520, in _call
    return self._stateless_fn(*args, **kwds)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1823, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError:  indices[28,20] = 1 is not in [0, 1)
         [[node tf_roberta_for_sequence_classification/roberta/embeddings/token_type_embeddings/embedding_lookup (defined at /data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_distributed_function_58202]

Function call stack:
distributed_function

2019-11-01 14:04:47.519198: W tensorflow/core/kernels/data/generator_dataset_op.cc:102] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
         [[{{node PyFunc}}]]
```

## Expected behavior

The script should train on GLUE when using TFRoberta just as it trains when using TFBert.

## Environment

OS: CentOS-7
Python version: 3.6.9
PyTorch version: 1.2.0
PyTorch Transformers version (or branch): 2.1.1 (master from git)
Using GPU ? Yes
Distributed of parallel setup ? No
Any other relevant information:

## Additional context


model forwards can take an inputs_embeds param


solves several bugs in the summarization codes

Hi,

This pull requests solves several small bugs in this summarization script:

- call the evaluation code
- fix the iterating over batch in eval part
- create the folders before saving encoder/decoder to avoid the crash
- first creates output_dir, then use it during the saving of the model part.
- tokenizer.add_special_tokens_single_sequence does not exist anymore, changed to tokenizer.build_inputs_with_special_tokens
- fix the empty checkpoints in the evaluation
- add evaluate=False in load_and_cache_examples functions, to fix the issue for now, but dataset needs to be split in train/validation
- add missing argument of per_gpu_eval_batch_size

the code needs more features to be added/bugs to be resolved, this just solves the obvious existing bugs, please see my opened bug request #1674 from yesterday on remaining issues.


TFXLNet Incompatible shapes in relative attention

## 🐛 Bug

TFXLNet fails to run due to incompatible shapes when computing relative attention.

Model I am using (Bert, XLNet....):

TFXLNet

Language I am using the model on (English, Chinese....):

English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: *MRPC*
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Fix line 542 in `transformers/modeling_tf_xlnet.py` to address #1692:
```python
            input_mask = 1.0 - tf.cast(attention_mask, dtype=dtype_float)
```
2. Run `run_tf_glue_xlnet.py` script:
```python
import os
import tensorflow as tf
import tensorflow_datasets
from transformers import XLNetForSequenceClassification, TFXLNetForSequenceClassification, glue_convert_examples_to_features, XLNetTokenizer

# script parameters
BATCH_SIZE = 32
EVAL_BATCH_SIZE = BATCH_SIZE * 2
USE_XLA = False
USE_AMP = False

# tf.config.optimizer.set_jit(USE_XLA)
# tf.config.optimizer.set_experimental_options({"auto_mixed_precision": USE_AMP})

# Load tokenizer and model from pretrained model/vocabulary
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = TFXLNetForSequenceClassification.from_pretrained('xlnet-base-cased')

# Load dataset via TensorFlow Datasets
data, info = tensorflow_datasets.load('glue/mrpc', with_info=True)
train_examples = info.splits['train'].num_examples
valid_examples = info.splits['validation'].num_examples

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(
        data['train'],
        tokenizer,
        max_length=512,
        output_mode="classification",
        task='mrpc',
        pad_on_left=True, # pad on the left for xlnet
        pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
        pad_token_segment_id=4
        )

valid_dataset = glue_convert_examples_to_features(
        data['validation'],
        tokenizer,
        max_length=512,
        output_mode="classification",
        task='mrpc',
        pad_on_left=True, # pad on the left for xlnet
        pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
        pad_token_segment_id=4
        )
train_dataset = train_dataset.shuffle(128).batch(BATCH_SIZE).repeat(-1)
valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
if USE_AMP:
    # loss scaling is currently required when using mixed precision
    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, 'dynamic')
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=opt, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
train_steps = train_examples//BATCH_SIZE
valid_steps = valid_examples//EVAL_BATCH_SIZE

history = model.fit(train_dataset, epochs=2, steps_per_epoch=train_steps,
                    validation_data=valid_dataset, validation_steps=valid_steps)

# Save TF2 model
os.makedirs('./save/', exist_ok=True)
model.save_pretrained('./save/')

# Load the TensorFlow model in PyTorch for inspection
pytorch_model = XLNetForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = 'This research was consistent with his findings.'
sentence_1 = 'His findings were compatible with this research.'
sentence_2 = 'His findings were not compatible with this research.'
inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')

pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
print('sentence_1 is', 'a paraphrase' if pred_1 else 'not a paraphrase', 'of sentence_0')
print('sentence_2 is', 'a paraphrase' if pred_2 else 'not a paraphrase', 'of sentence_0')
```

3. See the following error message:
```
2019-11-01 11:57:45.714479: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: Incompatible shapes: [512,512,32,12] vs. [512,1023,32,12]
         [[{{node tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/add_3}}]]
  1/114 [..............................] - ETA: 1:02:58Traceback (most recent call last):
  File "run_tf_glue_xlnet.py", line 63, in <module>
    validation_data=valid_dataset, validation_steps=valid_steps)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 520, in _call
    return self._stateless_fn(*args, **kwds)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1823, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Incompatible shapes: [512,512,32,12] vs. [512,1023,32,12]
         [[node tfxl_net_for_sequence_classification/transformer/layer_._0/rel_attn/add_3 (defined at /data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_distributed_function_67728]
```

## Expected behavior

The script should run just as `run_tf_glue.py`

## Environment

OS: CentOS-7
Python version: 3.6.9
PyTorch version: 1.2.0
PyTorch Transformers version (or branch): 2.1.1 (master from git)
Using GPU ? Yes
Distributed of parallel setup ? No
Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


TFXLNet int32 to float promotion error

## 🐛 Bug

Using TFXLNet on GLUE datasets results in a TypeError when computing the input_mask because the attention_mask is represented as an int32 and is not automatically cast or promoted to a float.

Model I am using (Bert, XLNet....):

TFXLNet

Language I am using the model on (English, Chinese....):

English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Use the attached run_tf_glue_xlnet.py script

```
Traceback (most recent call last):
  File "run_tf_glue_xlnet.py", line 63, in <module>
    validation_data=valid_dataset, validation_steps=valid_steps)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 224, in fit
    distribution_strategy=strategy)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 547, in _process_training_inputs
    use_multiprocessing=use_multiprocessing)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 594, in _process_inputs
    steps=steps)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2419, in _standardize_user_data
    all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2622, in _build_model_with_inputs
    self._set_inputs(cast_inputs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2709, in _set_inputs
    outputs = self(inputs, **kwargs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 842, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in converted code:
    relative to /data/conda/envs/transformers/lib/python3.6/site-packages:

    transformers/modeling_tf_xlnet.py:907 call  *
        transformer_outputs = self.transformer(inputs, **kwargs)
    tensorflow_core/python/keras/engine/base_layer.py:842 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    transformers/modeling_tf_xlnet.py:542 call  *
        input_mask = 1.0 - attention_mask
    tensorflow_core/python/ops/math_ops.py:924 r_binary_op_wrapper
        x = ops.convert_to_tensor(x, dtype=y.dtype.base_dtype, name="x")
    tensorflow_core/python/framework/ops.py:1184 convert_to_tensor
        return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
    tensorflow_core/python/framework/ops.py:1242 convert_to_tensor_v2
        as_ref=False)
    tensorflow_core/python/framework/ops.py:1296 internal_convert_to_tensor
        ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    tensorflow_core/python/framework/tensor_conversion_registry.py:52 _default_conversion_function
        return constant_op.constant(value, dtype, name=name)
    tensorflow_core/python/framework/constant_op.py:227 constant
        allow_broadcast=True)
    tensorflow_core/python/framework/constant_op.py:265 _constant_impl
        allow_broadcast=allow_broadcast))
    tensorflow_core/python/framework/tensor_util.py:449 make_tensor_proto
        _AssertCompatible(values, dtype)
    tensorflow_core/python/framework/tensor_util.py:331 _AssertCompatible
        (dtype.name, repr(mismatch), type(mismatch).__name__))

    TypeError: Expected int32, got 1.0 of type 'float' instead.
```

## Expected behavior

The script should run the same as tf_run_glue.py

## Environment

* OS: CentOS-7
* Python version: 3.6.9
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 2.1.1 (master from git)
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

run_tf_glue_xlnet.py:
```python
import os
import tensorflow as tf
import tensorflow_datasets
from transformers import XLNetForSequenceClassification, TFXLNetForSequenceClassification, glue_convert_examples_to_features, XLNetTokenizer

# script parameters
BATCH_SIZE = 32
EVAL_BATCH_SIZE = BATCH_SIZE * 2
USE_XLA = False
USE_AMP = False

# tf.config.optimizer.set_jit(USE_XLA)
# tf.config.optimizer.set_experimental_options({"auto_mixed_precision": USE_AMP})

# Load tokenizer and model from pretrained model/vocabulary
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = TFXLNetForSequenceClassification.from_pretrained('xlnet-base-cased')

# Load dataset via TensorFlow Datasets
data, info = tensorflow_datasets.load('glue/mrpc', with_info=True)
train_examples = info.splits['train'].num_examples
valid_examples = info.splits['validation'].num_examples

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(
        data['train'],
        tokenizer,
        max_length=512,
        output_mode="classification",
        task='mrpc',
        pad_on_left=True, # pad on the left for xlnet
        pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
        pad_token_segment_id=4
        )

valid_dataset = glue_convert_examples_to_features(
        data['validation'],
        tokenizer,
        max_length=512,
        output_mode="classification",
        task='mrpc',
        pad_on_left=True, # pad on the left for xlnet
        pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
        pad_token_segment_id=4
        )
train_dataset = train_dataset.shuffle(128).batch(BATCH_SIZE).repeat(-1)
valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
if USE_AMP:
    # loss scaling is currently required when using mixed precision
    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, 'dynamic')
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=opt, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
train_steps = train_examples//BATCH_SIZE
valid_steps = valid_examples//EVAL_BATCH_SIZE

history = model.fit(train_dataset, epochs=2, steps_per_epoch=train_steps,
                    validation_data=valid_dataset, validation_steps=valid_steps)

# Save TF2 model
os.makedirs('./save/', exist_ok=True)
model.save_pretrained('./save/')

# Load the TensorFlow model in PyTorch for inspection
pytorch_model = XLNetForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = 'This research was consistent with his findings.'
sentence_1 = 'His findings were compatible with this research.'
sentence_2 = 'His findings were not compatible with this research.'
inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')

pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
print('sentence_1 is', 'a paraphrase' if pred_1 else 'not a paraphrase', 'of sentence_0')
print('sentence_2 is', 'a paraphrase' if pred_2 else 'not a paraphrase', 'of sentence_0')
```

[run_tf_glue_xlnet.zip](https://github.com/huggingface/transformers/files/3798525/run_tf_glue_xlnet.zip)


ALbert Model implementation is finished on squad qa task. but some format is different with huggingface.(specify on albert qa task)

# 🌟New model addition

## Model description

<!-- Important information -->
Hey, seems like your community creates the same one branch for Albert, actually, I also reimplement the code and transfer from tf-hub to PyTorch by referencing your other preview convert_tf_xxx.py code. and also reproduce the author's performance on squad 1.1 for the albert-base model.

The reproduce is show below
Squad 1.1 EM: 79.98, F1 score: 87.98
The paper performance:
Squad 1.1 EM: 82.3,  F1 score: 89.30

## Open Source status

* [x] the model implementation is available: (give details)
* [x] the model weights are available: (give details)

## Additional context

<!-- Add any other context about the problem here. -->
The code is below https://github.com/pohanchi/huggingface_albert.
I just want to ask that I reference your code module from other old model, so what should i do to set my repo public safety.


T5

# 🌟New model addition

[Exploring the Limits of Transfer Learning with a
Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)

## Open Source status

Code and model are open sourced