AlbertTokenizer behavior doesn't match docs

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): AlbertForQuestionAnswering

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Run the code sample from the docs: https://huggingface.co/transformers/v2.2.0/model_doc/albert.html#albertforquestionanswering

```
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForQuestionAnswering.from_pretrained('albert-base-v2')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
input_ids = tokenizer.encode(input_text)
token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# a nice puppet
```

2. Getting the following error:
---------------------------------------------------------------------------
```
ValueError                                Traceback (most recent call last)
<ipython-input-16-2185be87fe39> in <module>
      5 input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
      6 input_ids = tokenizer.encode(input_text)
----> 7 token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
      8 start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
      9 all_tokens = tokenizer.convert_ids_to_tokens(input_ids)

<ipython-input-16-2185be87fe39> in <listcomp>(.0)
      5 input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
      6 input_ids = tokenizer.encode(input_text)
----> 7 token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
      8 start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
      9 all_tokens = tokenizer.convert_ids_to_tokens(input_ids)

ValueError: 102 is not in list
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Code says expected output is "a nice puppet"
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Kaggle kernel, no GPU
* Python version: 3.6.6
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

Some debugging info:
```
print(input_ids)
[2, 2, 72, 23, 2170, 27674, 60, 3, 2170, 27674, 23, 21, 2210, 10956, 3, 3]

tokenizer.decode(input_ids)
'[CLS][CLS] who was jim henson?[SEP] jim henson was a nice puppet[SEP][SEP]'
```
## Additional context

<!-- Add any other context about the problem here. -->


Fixing incorrect link in model docstring

The docstring contains a link to Salesforce/CTRL repo, while the model itself is Facebookresearch/mmbt. It may be the wrong copy\paste.


problem in the doc, in the  "Quick Start" GPT2 example

I am going through the GPT2 example in the doc. Is there a mistake in the "Using the past" code. The main loop to generate text is:

```python
for i in range(100):
    print(i)
    output, past = model(context, past=past)
    token = torch.argmax(output[0, :])
    generated += [token.tolist()]
    context = token.unsqueeze(0)
```

At the first iteration the tensor output as shape [1, 3, 50257], and at all the following iterations it has size [1,50257]. Should the code be:

```python
for i in range(100):
    print(i)
    output, past = model(context, past=past)
    if i==0:
        token = torch.argmax(output[0,-1,:])
    else:
        token = torch.argmax(output[0, :])
    generated += [token.tolist()]
    context = token.unsqueeze(0)
```


run_ner.py load checkpoint issue

Hi,
I just launched the **transformers/examples/run_ner.py** script with my custom model:
`python3 transformers/examples/run_ner.py --data_dir $INPUT_DATA_DIR \
--tokenizer_name $TOKENIZER_FILE_PATH  --output_dir $OUTPUT_DIR --model_type camembert --labels $LABELS_DIR --model_name_or_path $BERT_MODEL  --max_seq_length  $MAX_LENGTH  --num_train_epochs $NUM_EPOCHS --gradient_accumulation_steps $ACCUMULATION_STEPS --per_gpu_train_batch_size $BATCH_SIZE --save_steps $SAVE_STEPS --do_lower_case --do_train --do_eval --do_predict`

Once the data for the train has been loaded, an error appear:
`Traceback (most recent call last):
  File "transformers/examples/run_ner.py", line 567, in <module>
    main()
  File "transformers/examples/run_ner.py", line 496, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer, labels, pad_token_label_id)
  File "transformers/examples/run_ner.py", line 132, in train
    global_step = int(args.model_name_or_path.split('-')[-1].split('/')[0])
ValueError: invalid literal for int() with base 10: 'pytorch_dump_folder'`

Launching the same script a few hours ago the error did not appear, is it something related to the last updates #2134 ?

Thanks in advance.


HuggingFace transformers documentation webpage is blank?

Hello,

Is HuggingFace updating their transformers documentation site (https://huggingface.co/transformers/)?

I looked there to get some information about the HugginFace GPT-2, but for some reason all the contents of the website are gone.

Thank you,


Untrainable dense layer in TFBert. "WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss."

## 🐛 Bug

<!-- Important information -->

I am getting

> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.
> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.
> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.
> WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.

Errors when using the Tensorflow Bert.

For convenience, here is a colab notebook that reproduces the error
https://colab.research.google.com/drive/1fh8l43Mm7g4-yjlun7nTilmuP0SOMOx-

It looks like there's an operation in the TF Bert that does not allow gradients to flow, judging from goolging this issue

https://github.com/tensorflow/probability/issues/467

https://github.com/tensorflow/tensorflow/issues/27949

https://stackoverflow.com/questions/55434653/batch-normalization-doesnt-have-gradient-in-tensorflow-2-0

https://stackoverflow.com/questions/57144586/tensorflow-gradienttape-gradients-does-not-exist-for-variables-intermittently

Model I am using (Bert, XLNet....):

TFBertModel

Language I am using the model on (English, Chinese....):

English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X ] my own modified scripts: (give details)

```
!pip install transformers --quiet
%tensorflow_version 2.x

try:
  %tensorflow_version 2.x
except Exception:
  pass
import tensorflow as tf

from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing import sequence
import os
from tensorflow import keras
from tensorflow.keras.layers import Lambda
from tensorflow.keras import backend as K
from keras.preprocessing.sequence import pad_sequences
from transformers import TFBertModel, BertTokenizer, BertConfig
import numpy as np
from glob import glob
from tqdm import tqdm_notebook

print('TensorFlow:', tf.__version__)

posCites = tf.random.uniform(shape=(3, 4, 768), minval=-1, maxval=1, dtype=tf.dtypes.float32)
negCites = tf.random.uniform(shape=(3, 16, 768), minval=-1, maxval=1, dtype=tf.dtypes.float32)
textInputsIds = tf.random.uniform(shape=(3, 8), minval=0, maxval=200, dtype=tf.dtypes.int32)

dataset = (textInputsIds, posCites, negCites)
batch_size = 3
post_size = 4
neg_size = 16

posLabels = keras.backend.ones(batch_size*post_size)
negLabels = keras.backend.zeros(batch_size*neg_size)

totalLabels = keras.backend.concatenate((posLabels, negLabels), axis=-1)
totalLabels = tf.convert_to_tensor([[totalLabels] * 3])
totalLabels = tf.squeeze(totalLabels)

model = TFBertModel.from_pretrained('bert-base-uncased')
model.compile(
      optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
      loss='binary_crossentropy',
      metrics=['acc'])
labels = tf.constant(np.array([1,0,1]))
model(textInputsIds)[0]

def create_model():
    textInputs = tf.keras.Input(shape=(8,), dtype=tf.int32)

    bert_model = TFBertModel.from_pretrained('bert-base-uncased')

    textOut = bert_model(textInputs)
    textOutMean = tf.reduce_mean(textOut[0], axis=1)

    logits = tf.reduce_sum(textOutMean, axis=-1)
    return tf.keras.Model(inputs=[textInputs], outputs=[logits])

model = create_model()
# model = TFBertModel.from_pretrained('bert-base-uncased')
model.compile(
      optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),
      loss='binary_crossentropy',
      metrics=['acc'])
labels = tf.constant(np.array([1,0,1]))
model.fit(textInputsIds, labels, epochs=100)

```

WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details)

Using Bert as a text encoder, and matching output embeddings with a target.

## To Reproduce

Steps to reproduce the behavior:

Here is a ColabNotebook which contains the code posted above, to recreate the error
https://colab.research.google.com/drive/1fh8l43Mm7g4-yjlun7nTilmuP0SOMOx-

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

Gradients should be calculated for all variables in tfBert.

## Environment

* OS: Linux
* Python version: 3+
* PyTorch version: 1.2+
* PyTorch Transformers version (or branch): same as Pip install
* Using GPU ? colab gpu
* Distributed of parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Possibly related issues

https://github.com/huggingface/transformers/issues/1727


Implement some Python best practices

Improve source code quality with black, isort & flake8.


adding positional embeds masking to TFRoBERTa

Adding positional embeds masking to TFRoBERTa following its addition to the PT model in #1764 to fix PT <=> TF equivalence test


bias weights not used in T5Model

## 🐛 Bug
Running the T5Model on v2.3.0 can show the info message that all bias weights are not used:

> Weights from pretrained model not used in T5Model: ['encoder.block.0.layer.0.layer_norm.bias', 'encoder.block.0.layer.1.layer_norm.bias', 'encoder.block.1.layer.0.layer_norm.bias', 'encoder.block.1.layer.1.layer_norm.bias', 'encoder.block.2.layer.0.layer_norm.bias', 'encoder.block.2.layer.1.layer_norm.bias', 'encoder.block.3.layer.0.layer_norm.bias', 'encoder.block.3.layer.1.layer_norm.bias', 'encoder.block.4.layer.0.layer_norm.bias', 'encoder.block.4.layer.1.layer_norm.bias', 'encoder.block.5.layer.0.layer_norm.bias', 'encoder.block.5.layer.1.layer_norm.bias', 'encoder.final_layer_norm.bias', 'decoder.block.0.layer.0.layer_norm.bias', 'decoder.block.0.layer.1.layer_norm.bias', 'decoder.block.0.layer.2.layer_norm.bias', 'decoder.block.1.layer.0.layer_norm.bias', 'decoder.block.1.layer.1.layer_norm.bias', 'decoder.block.1.layer.2.layer_norm.bias', 'decoder.block.2.layer.0.layer_norm.bias', 'decoder.block.2.layer.1.layer_norm.bias', 'decoder.block.2.layer.2.layer_norm.bias', 'decoder.block.3.layer.0.layer_norm.bias', 'decoder.block.3.layer.1.layer_norm.bias', 'decoder.block.3.layer.2.layer_norm.bias', 'decoder.block.4.layer.0.layer_norm.bias', 'decoder.block.4.layer.1.layer_norm.bias', 'decoder.block.4.layer.2.layer_norm.bias', 'decoder.block.5.layer.0.layer_norm.bias', 'decoder.block.5.layer.1.layer_norm.bias', 'decoder.block.5.layer.2.layer_norm.bias', 'decoder.final_layer_norm.bias']

I think these are to be expected (and that this is actually not a bug), but I'm not sure. https://github.com/huggingface/transformers/issues/180#issuecomment-453937845 mentions that in some cases like this an additional message could be shown indicating whether this is expected behaviour or not but that has not been implemented here.

```python
from transformers import T5Model
import logging

logging.basicConfig(format='%(asctime)s - [%(levelname)s]: %(message)s',
                    datefmt='%d-%b %H:%M:%S',
                    level=logging.INFO)

model = T5Model.from_pretrained('t5-small')

for name, _ in model.named_parameters():
    print(name)
```


Documentation link broken

The README shows strange formatting (where 'Documentation' is put between brackets) for the links to documentation. More importantly, the link to the v2.3.0 documentation is broken (404 not found).


## 🐛 Bug

<!-- Important information -->

hey I just wanted to test BERT for sst2. I have just changed official example script to this:

```
import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/sst2')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='sst-2')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='sst-2')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

# Load the TensorFlow model in PyTorch for inspection
model.save_pretrained('./save/')
pytorch_model = BertForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = "I didn't think this was as absolutely horrible."
inputs_1 = tokenizer.encode_plus(sentence_0, add_special_tokens=True, return_tensors='pt')

pred_1 = pytorch_model(inputs_1['input_ids'], token_type_ids=inputs_1['token_type_ids'])[0].argmax().item()
print("sentence_1 is", pred_1)

```
I'm using  google colab with tensorflow2.0 . the error is :

AttributeError: 'Sst2Processor' object has no attribute 'tfds_map'


Four tests fail when running the full test suite

## 🐛 Bug

```
RUN_SLOW=1 python -m unittest discover -s transformers/tests -p '*_test.py' -t . -v
```

```
======================================================================
ERROR: test_model_from_pretrained (transformers.tests.modeling_tf_albert_test.TFAlbertModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../transformers/transformers/configuration_utils.py", line 160, in from_pretrained
    config = cls.from_json_file(resolved_config_file)
  File ".../transformers/transformers/configuration_utils.py", line 213, in from_json_file
    with open(json_file, "r", encoding='utf-8') as reader:
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/pq/hzv7wgqs5fq0hf1bwzy4mlzr0000gn/T/transformers_test/5b4c66df217ea00b14f607787de616bbff332ae36147a92cd94219160006685a'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../transformers/transformers/tests/modeling_tf_albert_test.py", line 221, in test_model_from_pretrained
    model = TFAlbertModel.from_pretrained(model_name, cache_dir=CACHE_DIR)
  File ".../transformers/transformers/modeling_tf_utils.py", line 249, in from_pretrained
    **kwargs
  File ".../transformers/transformers/configuration_utils.py", line 173, in from_pretrained
    raise EnvironmentError(msg)
OSError: Model name 'albert-base-uncased' was not found in model name list (albert-xxlarge-v2, albert-large-v1, albert-xlarge-v1, albert-base-v2, albert-base-v1, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v1). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/albert-base-uncased/config.json' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

======================================================================
ERROR: test_model_from_pretrained (transformers.tests.modeling_tf_xlm_test.TFXLMModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../transformers/transformers/tests/modeling_tf_xlm_test.py", line 255, in test_model_from_pretrained
    model = XLMModel.from_pretrained(model_name, cache_dir=CACHE_DIR)
NameError: name 'XLMModel' is not defined

======================================================================
FAIL: test_inference_masked_lm (transformers.tests.modeling_roberta_test.RobertaModelIntegrationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../transformers/transformers/tests/modeling_roberta_test.py", line 227, in test_inference_masked_lm
    torch.allclose(output[:, :3, :3], expected_slice, atol=1e-3)
AssertionError: False is not true

======================================================================
FAIL: test_inference_masked_lm (transformers.tests.modeling_tf_roberta_test.TFRobertaModelIntegrationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../transformers/transformers/tests/modeling_tf_roberta_test.py", line 220, in test_inference_masked_lm
    numpy.allclose(output[:, :3, :3].numpy(), expected_slice.numpy(), atol=1e-3)
AssertionError: False is not true

----------------------------------------------------------------------
```


bert（+lstm）+crf

add crf layer for better performance in NER tasks


Extract features aligned to tokens from a BertForQuestionAnswering model

## ❓ Questions & Help
I have a finetuned Bert model on a custom Question Answering task (follow original Tensorflow source), I've successful converted this model and loaded it with `BertForQuestionAnswering`
<!-- A clear and concise description of the question. -->
My question is how to get feature embedding aligned to each token from a pretrained `BertForQuestionAnswering`


NER pipeline missing start/end

Core: Pipeline

Internals of the library; Pipeline.

## 🚀 Feature

2.3 is a great release! Really excited for pipelines.

The feature to add is the start/end positions of the entities.

Additionally, the option to show the recognized entity rather than in subword form would be more user-friendly as an API.

## Motivation

[The release mentions including positions of the entities](https://github.com/huggingface/transformers/releases/tag/v2.3.0). The start/end positions are not in `transformers.py`

## Additional context
This is really exciting and motivated me to use your module. I hope to make a PR in the future.


Recently added pipelines tests should be marked as slow

## 🐛 Bug

Starting today, when running tests, some very large files are downloaded even though I don't enable RUN_SLOW=true.

Some tests in pipelines_test.py should be marked with `@slow` so they don't run unless RUN_SLOW=True.


Training dataset is not available

## ❓ Questions & Help

As stated [here](https://github.com/huggingface/transformers/blob/1ab8dc44b3d84ed1894f5b6a6fab58fb39298fc7/examples/distillation/README.md), the model was trained using Toronto Book Corpus and English Wikipedia. Neither this repository or BERT repository provides links to obtain this data. Upon further investigation the Toronto Book Corpus is no longer public. Please advise on how to get this data.


Fix Camembert and XLM-R `decode` method- Fix NER pipeline alignement

Fix `decode` method for Camembert and XLM-R
Simplify alignement method for NER pipeline


fixing xlm-roberta tokenizer max_length and automodels

Fix missing max token num in XLM-Roberta


BertTokenizer / CamemBERTokenizer `decode` behaviour ?

## 🐛 Bug

Thank you for CamemBERT, it's a great work 😃
Model I am using (Bert, XLNet....): CamemBERT

Language I am using the model on (English, Chinese....): French

The problem arise when using:
* [ ] the official example scripts:
* [x] my own modified scripts: followed by the official documentation at https://huggingface.co/transformers/main_classes/tokenizer.html#

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task:
* [x] my own task or dataset: Show the tokens from the CamemBERT tokenizer.

## To Reproduce

Steps to reproduce the behavior:

1. Tokenize (`encode`) a sentence.
2. Try to `decode` the ids but not working (`TypeError` thrown)

```
tokenizer = CamembertTokenizer.from_pretrained('camembert-base')
ids = tokenizer.encode(sentence)
print(tokenizer.decode(ids))
```
I just followed the documentation for decode which explains :
Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids)).

Traceback :
```
  File "/d/Workspaces/camembert-test/main.py", line 51, in <module>
    tokens_with_transformers_error(sentence)
  File "/d/Workspaces/camembert-test/main.py", line 32, in tokens_with_transformers_error
    print(tokenizer.decode(ids))
  File "/d/Workspaces/camembert-test/.venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1187, in decode
    sub_texts.append(self.convert_tokens_to_string(current_sub_text))
  File "/d/Workspaces/camembert-test/.venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1156, in convert_tokens_to_string
    return ' '.join(self.convert_ids_to_tokens(tokens))
  File "/d/Workspaces/camembert-test/.venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1145, in convert_ids_to_tokens
    tokens.append(self._convert_id_to_token(index))
  File "/d/Workspaces/camembert-test/.venv/lib/python3.7/site-packages/transformers/tokenization_camembert.py", line 146, in _convert_id_to_token
    return self.sp_model.IdToPiece(index - self.fairseq_offset)
TypeError: unsupported operand type(s) for -: 'str' and 'int'
```
Here, convert_tokens_to_string seems to call convert_ids_to_tokens ? Why ?

## Expected behavior

Have the same return statement (or anything else) as this code which seems to works but don't have to...? I am not sure though 😃
```
    tokenizer = CamembertTokenizer.from_pretrained('camembert-base')
    ids = tokenizer.encode(sentence)
    print(tokenizer.convert_tokens_to_string(ids)) # I give list of ids, not list of tokens
```

## Environment

* OS: Linux
* Python version: 3.7
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.2
* Using GPU ? no
* Distributed of parallel setup ? no
* Any other relevant information: /

## Additional context

<!-- Add any other context about the problem here. -->


How to load the finetuned model for retraining from checkpoints in run_squad.py?

Because of bad internet connection and computational issues its hard for us to train a large number of epochs. We're trying to use the run_squad.py script for bangla QA system training. We have trained the model before and have the checkpoints.

```
!python run_squad.py \
  --model_type distilbert \
  --model_name_or_path ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/ \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file ../data/dataset_v_3/train_bangla_samples.json \
  --predict_file ../data/dataset_v_3/valid_bangla_samples.json \
  --version_2_with_negative \
  --per_gpu_train_batch_size 12 \
  --learning_rate 5e-5 \
  --num_train_epochs 1.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --logging_steps 100 \
  --save_steps 100 \
  --fp16 \
  --evaluate_during_training \
  --output_dir ../data/mytrial
```
command produces this error :

```
12/20/2019 14:06:45 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: True
12/20/2019 14:06:47 - INFO - transformers.configuration_utils -   loading configuration file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/config.json
12/20/2019 14:06:47 - INFO - transformers.configuration_utils -   Model config {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "finetuning_task": null,
  "hidden_dim": 3072,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "max_position_embeddings": 512,
  "n_heads": 12,
  "n_layers": 6,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 30522
}

12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Model name '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' not found in model shortcut name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). Assuming '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' is a path or url to a directory containing tokenizer files.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/vocab.txt. We won't load it.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/added_tokens.json. We won't load it.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/special_tokens_map.json. We won't load it.
12/20/2019 14:06:47 - INFO - transformers.tokenization_utils -   Didn't find file ../data/tmp/attempt_with_dataset_v3/checkpoint-5000/tokenizer_config.json. We won't load it.
Traceback (most recent call last):
  File "run_squad.py", line 614, in <module>
    main()
  File "run_squad.py", line 528, in main
    cache_dir=args.cache_dir if args.cache_dir else None)
  File "/content/gdrive/My Drive/huggingfaceattempt/transformers/transformers/tokenization_utils.py", line 302, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/content/gdrive/My Drive/huggingfaceattempt/transformers/transformers/tokenization_utils.py", line 370, in _from_pretrained
    list(cls.vocab_files_names.values())))
OSError: Model name '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). We assumed '../data/tmp/attempt_with_dataset_v3/checkpoint-5000/' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
```
but none of the checkpoint folders have any vocabulary stored. Are we supposed to pass the checkpoint folder path to the model_name_or_path for training again from that checkpoint?


TFDistilBertModelTest.test_pt_tf_model_equivalence thrown while merging after PR

## 🐛 Bug

I've seen that some PRs failed because of the same error, such as #2239 (today) and #2237 (today). If I'm not remember wrong, there is also another PR in the last days involved @rlouf that he had the same error. Even if the changes made not affect `DistilBertModel` such as #2237, this error occurs!
Question: **Is it a bug in Transformers or a bug in our code submitted?**

## To Reproduce

Steps to reproduce the behavior: after submitting a PR to Transformers library, it occurs
```
=================================== FAILURES ===================================
______________ TFDistilBertModelTest.test_pt_tf_model_equivalence ______________

self = <transformers.tests.modeling_tf_distilbert_test.TFDistilBertModelTest testMethod=test_pt_tf_model_equivalence>

    def test_pt_tf_model_equivalence(self):
        if not is_torch_available():
            return

        import torch
        import transformers

        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

        for model_class in self.all_model_classes:
            pt_model_class_name = model_class.__name__[2:]  # Skip the "TF" at the beggining
            pt_model_class = getattr(transformers, pt_model_class_name)

            config.output_hidden_states = True
            tf_model = model_class(config)
            pt_model = pt_model_class(config)

            # Check we can load pt model in tf and vice-versa with model => model functions
            tf_model = transformers.load_pytorch_model_in_tf2_model(tf_model, pt_model, tf_inputs=inputs_dict)
            pt_model = transformers.load_tf2_model_in_pytorch_model(pt_model, tf_model)

            # Check predictions on first output (logits/hidden-states) are close enought given low-level computational differences
            pt_model.eval()
            pt_inputs_dict = dict((name, torch.from_numpy(key.numpy()).to(torch.long))
                                  for name, key in inputs_dict.items())
            with torch.no_grad():
                pto = pt_model(**pt_inputs_dict)
            tfo = tf_model(inputs_dict, training=False)
            tf_hidden_states = tfo[0].numpy()
            pt_hidden_states = pto[0].numpy()
            tf_hidden_states[np.isnan(tf_hidden_states)] = 0
            pt_hidden_states[np.isnan(pt_hidden_states)] = 0
            max_diff = np.amax(np.abs(tf_hidden_states - pt_hidden_states))
>           self.assertLessEqual(max_diff, 2e-2)
E           AssertionError: 3.107201 not less than or equal to 0.02

transformers/tests/modeling_tf_common_test.py:139: AssertionError
```

## Expected behavior

No error was thrown.

## Environment

* OS: **Ubuntu 16.04**
* Python version: **3.6.9**
* PyTorch version: **1.3.1**
* PyTorch Transformers version (or branch): **master**
* Using GPU ? **Indifferent**
* Distributed of parallel setup ? **Indifferent**
* Any other relevant information:


HANS evaluation

Adding the evaluation on the HANS dataset in examples


Readme installation/test order can lead to confusion when running example unit tests

## ❓ Questions & Help

When following the main readme installation/testing instructions in order, it is not mentioned that in order to let the examples tests pass, the separate examples/requirements.txt must be installed.
Thus, `pip install -r ./examples/requirements.txt` should come before `python -m pytest -sv ./transformers/tests/` in the main readme to avoid confusion.

Additionally, the readme line `python -m unittest discover -s examples -p "*test.py" -t examples` cannot find any tests, and produces the following output;

`Ran 0 tests in 0.000s
OK`

I don't think this is the intended behavior - is the line redundant?


Fix out-of-date comments in Transformers examples directory


Removing redundant model weights

## 🐛 This is not a bug, more like implemetation detail

I am using BertForTokenClassification model for my binary token classification problem. If my understanding is right, BertForTokenClassification has one layer on top with num_classes output neurons (one for each class) with softmax activation function + CrossEntropyLoss().

Now, if your problem has >2 classes this is completely fine, but if num_classes=2 you are both modeling P(input = 0) and P(input = 1), and it's easy to see why this is redunant.

num_classes=2 is special case and it should be implemented with only one output neuron with sigmoid activation function + binary cross-entopy.

Please correct me if I am wrong :)


add example for Model2Model in quickstart

As per discussed @LysandreJik


Supoort loading model weights from a single file.

So far, transformers package supports loading a model from a directory, such as
```python
model = BertModel.from_pretrained('./test/saved_model/')  # E.g. model was saved using save_pretrained('./test/saved_model/')
```

Actually, it loads model weight from $directory/pytorch_model.bin.
Is it possible to load model weights from a file?
such as
```python
 tokenizer = BertTokenizer.from_pretrained('./test/saved_model/my_vocab.txt')
```

Because my network is too slow to download models from amazon server.
I have to download the model files manually and put them to transformers.
It is more convenient to load the downloaded model weights from a file than a directory.
```python
model = BertModel.from_pretrained('./test/model_zoo/bert-base-multilingual-cased-pytorch_model.bin')
```
Thank you.


The code used to be clean...

Hi, thanks to all contributors!
From my point of view, recent changes in examples codes, especially run_squad script, are kinda confusing.

May I know what is the reason that we delete old utils_squad.py and store all data processing and evaluating scripts under the transformer/data folder? This makes code look quite messy. Particularly, the example and features classes have been changed a lot.

I was reading the old version scripts for a long long time, and I think it was easy for me to change any components in the whole structure, no matter data preprocessing or introducing new LM.  I can easily change the input format of examples, so I could run codes on other datasets with few lines.

I believe many people have already made task-specific changes on their own codes,  and sometimes some people just want to run codes on new released LMs for testing, so they come here and check. If luckily new LM is available, we would prefer update code by hands instead of the direct downloading whole package again.

But anyway, if changes are compulsory, I am definitely willing to go through it deeply.


Keep even the first of the special tokens intact while lowercasing

This fixes #2220. The order of `all_special_tokens` is random, and the first one of these will get broken by the lowercasing. There are 5 special tokens, so you have a 1 in 5 chance of hitting the problem.


[http] customizable requests user-agent


what is the most efficient way to store all hidden layers' weights?

Hi,

I am following this [post](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) for getting all 12 hidden layers' weights for every token in a sentence.

Consider I have a short text with 2 sentences: `He stole money today. He is fishing on the Mississippi riverbank.`

I want to store 5 + 8 = 13 tokens - all 12 hidden layers weights where each tensor's size=768. So, I will have 13 x 12 = 156 tensors.

I want to save all the weights in a file and I am wondering if I should use `pickle` or `hd5` format (I am working with long text documents.) I am planning to separate two sentences by a blank line, please suggest if any better ways to do it.

Thanks!


Minor/basic text fixes


Trouble loading Albert model

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Albert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details):
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

Using example from docs: https://huggingface.co/transformers/model_doc/albert.html

## To Reproduce
Trying to load the Albert model using the code below:
```
import tensorflow as tf
from transformers import AlbertTokenizer
tokenizer = AlbertTokenizer.from_pretrained('bert-base-uncased')
```
Getting the following error:
```
Traceback (most recent call last):

  File "<ipython-input-4-56254e5f4b51>", line 3, in <module>
    tokenizer = AlbertTokenizer.from_pretrained('bert-base-uncased')

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\tokenization_utils.py", line 302, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\tokenization_utils.py", line 437, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\tokenization_albert.py", line 90, in __init__
    self.sp_model.Load(vocab_file)

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\sentencepiece.py", line 118, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)

RuntimeError: Internal: C:\projects\sentencepiece\src\sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
```

Note, the same code works for bert instead of albert:
> from transformers import *
> tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
>

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Looking for albert model to load without errors
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows 10
* Python version: 3.7
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): pip install transformers (fresh install today 12/19/2019)
* Using GPU ?: Yes
* Distributed of parallel setup ?: N/A
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Add "Train on Valohai" buttons to README

This pull request adds two "Train on Valohai" buttons to README:

![Screenshot from 2019-12-19 16-59-54](https://user-images.githubusercontent.com/2681608/71184210-1284b000-2282-11ea-8bb8-72bf93014672.png)

and...

![Screenshot from 2019-12-19 17-00-07](https://user-images.githubusercontent.com/2681608/71184213-13b5dd00-2282-11ea-84c7-393afbbae7ea.png)

When clicked, it will automatically create a project on Valohai and let you run the project examples without any further setup. Effectively the same as "Deploy to Heroku" button if you are familiar with that.

## The flow looks like this (after login):

![Screenshot from 2019-12-19 17-00-19](https://user-images.githubusercontent.com/2681608/71184512-88891700-2282-11ea-855b-5a187ef2e700.png)
![Screenshot from 2019-12-19 17-00-34](https://user-images.githubusercontent.com/2681608/71184528-8d4dcb00-2282-11ea-8228-6322236360a9.png)
![Screenshot from 2019-12-19 17-05-31](https://user-images.githubusercontent.com/2681608/71184531-8e7ef800-2282-11ea-970a-aefa5d127069.png)
![Screenshot from 2019-12-19 17-06-18](https://user-images.githubusercontent.com/2681608/71184536-9048bb80-2282-11ea-98b0-7fd371a78677.png)


[REVIEW] Updated a out-of-date comment in run_lm_finetuning.py


[REVIEW] Updated comments in run_lm_finetuning.py

I've added DistilBERT and CamemBERT models in the description of the models that can be used for fine-tuning a LM model on a custom dataset.


[REVIEW] Removed duplicate XLMConfig, XLMForQuestionAnswering and XLMTokenizer in run_squad.py

Before this PR, in run_squad.py at lines 58-65 were the following:
```
MODEL_CLASSES = {
    'bert': (BertConfig, BertForQuestionAnswering, BertTokenizer),
    'xlnet': (XLNetConfig, XLNetForQuestionAnswering, XLNetTokenizer),
    'xlm': (XLMConfig, XLMForQuestionAnswering, XLMTokenizer),
    'distilbert': (DistilBertConfig, DistilBertForQuestionAnswering, DistilBertTokenizer),
    'albert': (AlbertConfig, AlbertForQuestionAnswering, AlbertTokenizer),
    'xlm': (XLMConfig, XLMForQuestionAnswering, XLMTokenizer)
}
```

After this PR, I've **removed** the last (key, value) pair in the `MODEL_CLASSES` dictionary because it contains _xlm_ as key **two** times.


Need pretrained XLNET on Squad which can be loaded from_pre_trained

## ❓ Questions & Help
It will be great ,if someone could share their username/model for XLNET pretrained on squad
<!-- A clear and concise description of the question. -->


Can we remove force_download=True from tests?

## 🚀 Feature

I would like to remove `force_download=True` from tests and rely on the cache to keep cached files up to date.

## Motivation

Currently, several tests use `force_download=True` on large model files, e.g.:

https://github.com/huggingface/transformers/blob/9c58b236ef5fbbe5d0cbde4932eb342a73eaa0dc/transformers/tests/modeling_tf_auto_test.py#L49

This prevents caching large models within a test run and across test runs, which is very painful when working on these tests in a local environment. This is the main reason why I'm filing this issue.

In fact, it's so painful that these tests are marked as slow and skipped by default. As a consequence, we're not getting as much value from them as we could. If we downloaded each model only once, perhaps we could run them in CI.

I assume that `force_download=True` was added for robustness, to make sure the cache doesn't contain stale files.

If that's correct, then I believe I can be safely removed, because the current implementation of the cache in `file_utils.py` is sufficiently robust to keep the cache up to date.

The entry point is here:

https://github.com/huggingface/transformers/blob/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b/transformers/configuration_utils.py#L157-L158

which goes here:

https://github.com/huggingface/transformers/blob/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b/transformers/file_utils.py#L192-L196

which always gets an ETag (and, I guess, fails without an Internet connection):

https://github.com/huggingface/transformers/blob/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b/transformers/file_utils.py#L288-L289

As a consequence, you never hit the only situation where the cache may use stale files: if it cannot get an ETag (either because there's no Internet connection, or because the HTTP server doesn't provide an ETag).

(As a side note, file_utils.py is needlessly complex, given that all transformers files are stored on S3.)

Am I missing a reason for `force_download=True`? Did you add it because you encountered another issue?


Updated typo on the link

Updated documentation due to typo


tokenizer of bert-base-uncased gives an incorrect split

I found the following code gives an incorrect split

`tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')`
`text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"`
`tokenized_text = tokenizer.tokenize(text)`
`print(' '.join(tokenized_text))`

> [ cl ##s ] who was jim henson ? [ sep ] jim henson was a puppet ##eer [ sep ]

The correct one, based on the Quick tour example, seems to be

> [CLS] who was jim henson ? [SEP] jim heson was a puppet ##eer [SEP]


When i run the script run_tf_ner.py, i got ValueError: Expected floating point type, got <dtype: 'int32'>.

I have tried tf 2.0.0a0, 2.0.0b0, 2.0.0b1, but the same error was reported.

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Traceback (most recent call last):
  File "run_tf_ner.py", line 615, in <module>
    app.run(main)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "run_tf_ner.py", line 517, in main
    cache_dir=args['cache_dir'] if args['cache_dir'] else None)
  File "/home/zpchen/transformers-master/transformers/modeling_tf_utils.py", line 303, in from_pretrained
    ret = model(model.dummy_inputs, training=False)  # build the network with dummy inputs
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 712, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/zpchen/transformers-master/transformers/modeling_tf_bert.py", line 1011, in call
    outputs = self.bert(inputs, **kwargs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 712, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/zpchen/transformers-master/transformers/modeling_tf_bert.py", line 547, in call
    embedding_output = self.embeddings([input_ids, position_ids, token_type_ids, inputs_embeds], training=training)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 709, in __call__
    self._maybe_build(inputs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1966, in _maybe_build
    self.build(input_shapes)
  File "/home/zpchen/transformers-master/transformers/modeling_tf_bert.py", line 122, in build
    initializer=get_initializer(self.initializer_range))
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 389, in add_weight
    aggregation=aggregation)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 713, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 154, in make_variable
    shape=variable_shape if variable_shape else None)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 260, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 221, in _variable_v1_call
    shape=shape)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 60, in getter
    return captured_getter(captured_previous, **kwargs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py", line 1250, in creator_with_resource_vars
    return self._create_variable(*args, **kwargs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/distribute/one_device_strategy.py", line 76, in _create_variable
    return next_creator(*args, **kwargs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 199, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2502, in default_variable_creator
    shape=shape)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 264, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 464, in __init__
    shape=shape)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 608, in _init_from_args
    initial_value() if init_from_fn else initial_value,
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 134, in <lambda>
    init_val = lambda: initializer(shape, dtype=dtype)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/init_ops_v2.py", line 341, in __call__
    dtype = _assert_float_dtype(dtype)
  File "/home/zpchen/anaconda7/lib/python3.6/site-packages/tensorflow/python/ops/init_ops_v2.py", line 769, in _assert_float_dtype
    raise ValueError("Expected floating point type, got %s." % dtype)
ValueError: Expected floating point type, got <dtype: 'int32'>.


corrected typo in example for t5 model input argument

For the T5Model the argument name of the input has to be specified explicitly since the forward function is defined as
`def forward(self, **kwargs):`
and can therefore only handle keyworded arguments such as `input_ids=inputs_ids`.


Support running tests in parallel

At this point, after `pip install pytest-xdist`, I'm getting a **2.5x speedup** running tests locally on my 2016 MBP (2,9 GHz Quad-Core Intel Core i7):

- `python -m pytest -n auto -s -v ./transformers/tests/` runs in slightly less than 2 minutes
- `python -m pytest -s -v ./transformers/tests/` takes slightly more than 5 minutes

Furthermore, Circle CI gets a **2,15x speedup**, going from [7:30 minutes](https://circleci.com/workflow-run/4ca98875-2d5c-438b-951f-4939d2f3cfc9) to [3:30 minutes](https://circleci.com/workflow-run/ad7094bb-0a0b-404d-ba3f-bd18b37f98bd).

The bottleneck is now the examples, which take a bit less than 3:30 to run, even with parallelization.

This PR adds a new dependency: filelock. You'll need to `pip install -e .` for local development again after it's merged.

This is now ready for review.

----

EDIT - test run time jumped up after I rebased on top of master mostly because of #2246.


Error while loading Pretrained Enocder and Decoder transformers

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Define a model and save it.
when you try to save the model initially you'll get the following error #2196 (create a folder manually encoder and decoder)
2. And try to load the model.
you'll get following error

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guest_1/anaconda3/envs/my_env/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 146, in from_pretrained
    encoder_pretrained_model_name_or_path, *model_args, **kwargs_encoder
  File "/home/guest_1/anaconda3/envs/my_env/lib/python3.6/site-packages/transformers/modeling_auto.py", line 159, in from_pretrained
    "'xlm', 'roberta, 'ctrl'".format(pretrained_model_name_or_path))
ValueError: Unrecognized model identifier in /models/m2m/. Should contains one of 'bert', 'openai-gpt', 'gpt2', 'transfo-xl', 'xlnet', 'xlm', 'roberta, 'ctrl'

```
This is because [here](https://github.com/huggingface/transformers/blob/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b/transformers/modeling_auto.py#L170) the model prefix is checked to load the pre-trained model. So rather than just saving the directory as encoder and decoder save it with model prefix.

You could do something like this
```
        encoder_dir = self.encoder.base_model_prefix + "encoder"
        decoder_dir = self.encoder.base_model_prefix + "decoder"
        self.encoder.save_pretrained(os.path.join(save_directory, encoder_dir))
        self.decoder.save_pretrained(os.path.join(save_directory, decoder_dir))
```
 But another potential bug the above code will lead to is that almost all the transformers has the same prefix. You need to rename all the transformers with their respective transformers prefix.
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: ubuntu
* Python version: 3.6
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? doesn't matter
* Distributed of parallel setup ? doesn't matter
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Return overflowing tokens if max_length is not given

## 🚀 Feature

The tokenizer can return overflowing tokens in the `encode_plus` method if 1. `return_overflowing_tokens=True`; and 2. if `max_length` is given.  I imagine that it could also be useful to allow to return the overflowing tokens when a max_length is not given but when the input is longer than the model's max_seq_len. As an example: if the input is 600 tokens long, and the model supports up to 512, then the tokenizer will cut down the input to 512 anyway, so the superfluous 88 tokens can then be returned.

Example showing that the overflowing tokens are not returned, even though the input is trimmed. Would expect the trimmed tokens to be returned in the `overflowing_tokens` field.

```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = 'I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas. I like bananas.'

encoded_inputs = tokenizer.encode_plus(text,
                                       return_tensors='pt',
                                       pad_to_max_length=True,
                                       return_overflowing_tokens=True,
                                       return_special_tokens_mask=True)
print(encoded_inputs.keys())
print(encoded_inputs['input_ids'].size())
# dict_keys(['special_tokens_mask', 'input_ids', 'token_type_ids', 'attention_mask'])
# torch.Size([1, 512])
```


XLM run_squad errors with size mismatch

## 🐛 Bug

<!-- Important information -->

Model: XLM_mlm_ende_1024

Language: English

Using: the official example scripts: ``run_squad.py``

## To Reproduce

Steps to reproduce the behavior:

1. install dependecies and download squad v1.1 data; pull, install transformers from github master.
2. run ``run_squad.py`` with the following args

```
--model_type xlm --model_name_or_path xlm-mlm-ende-1024 --do_train --do_eval --train_file ./squad_data/train-v1.1.json --predict_file ./squad_data/dev-v1.1.json --per_gpu_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./debug_xlm
```

Ultimate error:
```
size mismatch for transformer.embeddings.weight: copying a param with shape torch.Size([64699, 1024]) from checkpoint, the shape in current model is torch.Size([30145, 1024]).
```
Full inout ouput below.

Expected behavior: finetune xlm for squad.

## Environment

* OS: OpenSuse 15.0
* Python version: 3.6
* PyTorch version: torch.__version__ = '1.3.1+cpu'
* PyTorch Transformers version (or branch): (just transformers now?) 2.2.2
* Using GPU ? nope
* Distributed of parallel setup ? Ummm n/a

Relates to previous issues (possibly):

* [I've had with XLM](https://github.com/huggingface/transformers/issues/2038)
* Similar looking [error](https://github.com/huggingface/transformers/issues/594)

## Additional context

```python ./transformers/examples/run_squad.py --model_type xlm --model_name_or_path xlm-mlm-ende-1024 --do_train --do_eval --train_file ./squad_data/train-v1.1.json --predict_file ./squad_data/dev-v1.1.json --per_gpu_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./debug_xlm
12/18/2019 15:47:23 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, 16-bits training: False
12/18/2019 15:47:23 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-ende-1024-config.json from cache at /HOME/.cache/torch/transformers/8f689e7cdf34bbebea67ad44ad6a142c9c5144e5c19d989839139e0d47d1ed74.0038e5c2b48fc777632fc95c3d3422203693750b1d0845a511b3bb84ad6d8c29
12/18/2019 15:47:23 - INFO - transformers.configuration_utils -   Model config {
  "asm": false,
  "attention_dropout": 0.1,
  "bos_index": 0,
  "causal": false,
  "dropout": 0.1,
  "emb_dim": 1024,
  "embed_init_std": 0.02209708691207961,
  "end_n_top": 5,
  "eos_index": 1,
  "finetuning_task": null,
  "gelu_activation": true,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "id2lang": {
    "0": "de",
    "1": "en"
  },
  "init_std": 0.02,
  "is_decoder": false,
  "is_encoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "lang2id": {
    "de": 0,
    "en": 1
  },
  "layer_norm_eps": 1e-12,
  "mask_index": 5,
  "max_position_embeddings": 512,
  "max_vocab": -1,
  "min_count": 0,
  "n_heads": 8,
  "n_langs": 2,
  "n_layers": 6,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_index": 2,
  "pruned_heads": {},
  "same_enc_dec": true,
  "share_inout_emb": true,
  "sinusoidal_embeddings": false,
  "start_n_top": 5,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "first",
  "summary_use_proj": true,
  "torchscript": false,
  "unk_index": 3,
  "use_bfloat16": false,
  "use_lang_emb": true,
  "vocab_size": 30145
}

12/18/2019 15:47:24 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-ende-1024-vocab.json from cache at /HOME/.cache/torch/transformers/6771b710c1daf9d51643260fdf576f6353369c3563bf0fb12176c692778dca3f.2c29a4b393decdd458e6a9744fa1d6b533212e4003a4012731d2bc2261dc35f3
12/18/2019 15:47:24 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-ende-1024-merges.txt from cache at /HOME/.cache/torch/transformers/85d878ffb1bc2c3395b785d10ce7fc91452780316140d7a26201d7a912483e44.42fa32826c068642fdcf24adbf3ef8158b3b81e210a3d03f3102cf5a899f92a0
12/18/2019 15:47:25 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-ende-1024-pytorch_model.bin from cache at /HOME/.cache/torch/transformers/ea4c0bbee310b490decb2b608a4dbc8ed9f2e4a103dd729ce183770b0fef698b.119d74257b953e5d50d73555a430ced11b1c149a7c17583219935ec1bd37d948
12/18/2019 15:47:28 - INFO - transformers.modeling_utils -   Weights of XLMForQuestionAnswering not initialized from pretrained model: ['qa_outputs.start_logits.dense.weight', 'qa_outputs.start_logits.dense.bias', 'qa_outputs.end_logits.dense_0.weight', 'qa_outputs.end_logits.dense_0.bias', 'qa_outputs.end_logits.LayerNorm.weight', 'qa_outputs.end_logits.LayerNorm.bias', 'qa_outputs.end_logits.dense_1.weight', 'qa_outputs.end_logits.dense_1.bias', 'qa_outputs.answer_class.dense_0.weight', 'qa_outputs.answer_class.dense_0.bias', 'qa_outputs.answer_class.dense_1.weight']
12/18/2019 15:47:28 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in XLMForQuestionAnswering: ['pred_layer.proj.weight', 'pred_layer.proj.bias']
Traceback (most recent call last):
  File "./transformers/examples/run_squad.py", line 614, in <module>
    main()
  File "./transformers/examples/run_squad.py", line 532, in main
    cache_dir=args.cache_dir if args.cache_dir else None)
  File "/HOME/sandpit/transformers/transformers/modeling_utils.py", line 486, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for XLMForQuestionAnswering:
        size mismatch for transformer.embeddings.weight: copying a param with shape torch.Size([64699, 1024]) from checkpoint, the shape in current model is torch.Size([30145, 1024]).
```


T5 - Finetuning of an EncoderDecoder Model

Hello,

I know that the T5 implementation is quite new, but is there already code to finetune and lateron decode from the T5 model?

As I understand most of your models are no EncoderDecoder models, so I guess that the default pipeline / code is not working for T5, is that right?

Could you point me to a script / command / piece of code for finetuning T5?


Fine-tuning TF models on Colab TPU

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,

I am trying to fine-tune TF BERT on Imdb dataset on Colab TPU. Here is the full notebook:

https://colab.research.google.com/drive/16ZaJaXXd2R1gRHrmdWDkFh6U_EB0ln0z

Can anyone help me what I am doing wrong?
Thanks


Fast tokenizers

I am opening this PR to track the integration of `tokenizers`.

At the moment, we created two new classes to represent the fast version of both GPT2 and Bert tokenizers. There are a few breaking changes compared to the current `GPT2Tokenizer` and `BertTokenizer`:
  - `add_special_token` is now specified during initialization
  - truncation and padding options are also setup during initialization

By default, `encode_batch` pads everything using the longest sequence, and `encode` does not pad at all. If `pad_to_max_length=True`, then we pad everything using this length.
If a `max_length` is specified, then everything is truncated according to the provided options. This should work exactly like before.

In order to try these, you must `pip install tokenizers` in your virtual env.


training a new BERT tokenizer model

## ❓ Questions & Help

I would like to train a new BERT model.
There are some way to train BERT tokenizer (a.k.a. wordpiece tokenizer) ?


```glue_convert_examples_to_features``` for sequence labeling tasks

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
I would like a function like ```glue_convert_examples_to_features``` for sequence labelling tasks.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
The motivation is that I need much better flexibility for  sequence labelling tasks. Its not enough to have a final, and decided for me model, for this task. I want just the features (a sequence of features/embeddings I guess).

## Additional context
This can be generalized to any dataset with a specific format.

<!-- Add any other context or screenshots about the feature request here. -->


Fix segmentation fault

Fix segmentation fault that started happening yesterday night.
Following the fix from #2205 that could be reproduced using circle ci ssh access.

~Currently fixing the unforeseen event with Python 2.~ The error with Python 2 was due to Regex releasing a new version (2019.12.17) that couldn't be built on Python 2.7.


Transformers Encoder and Decoder Inference

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Error while doing inference.
```
from transformers import PreTrainedEncoderDecoder, BertTokenizer
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased',''bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
encoder_input_ids=tokenizer.encode("Hi How are you")
import torch
ouput = model(torch.tensor( encoder_input_ids).unsqueeze(0))
```
and the error is
```
TypeError: forward() missing 1 required positional argument: 'decoder_input_ids'
```
During inference why is decoder input is expected ?

Let me know if I'm missing anything?
## Environment

OS: ubuntu
Python version: 3.6
PyTorch version:1.3.0
PyTorch Transformers version (or branch):2.2.0
Using GPU ? Yes
Distributed of parallel setup ? No
Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Segmentation fault when GPT2-chinese import transformers

## ❓ Questions & Help
CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
GPU: Tesla P40
OS Platform: Ubuntu 16.04.3 LTS
transformers version: 2.2.2
TensorFlow version: 2.0.0
PyTorch version: 1.3.1
Python version: 3.6.2

Hi
When I trained the model in [GPT2-chinese](https://github.com/Morizeyao/GPT2-Chinese), it imported transformers and got an error - Segmentation fault(core dumped)
```
(python36) user@ubuntu:~/projects/GPT2-Chinese$ python -Xfaulthandler train.py
I1218 00:39:09.690756 139724474922816 file_utils.py:33] TensorFlow version 2.0.0 available.
Fatal Python error: Segmentation fault

Current thread 0x00007f1423b1b740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 922 in create_module
  File "<frozen importlib._bootstrap>", line 560 in module_from_spec
  File "<frozen importlib._bootstrap>", line 648 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "/home/wangjun/.conda/envs/python36/lib/python3.6/site-packages/torch/__init__.py", line 81 in <module>
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 655 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "/home/wangjun/.conda/envs/python36/lib/python3.6/site-packages/transformers/file_utils.py", line 38 in <module>
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 655 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "/home/wangjun/.conda/envs/python36/lib/python3.6/site-packages/transformers/__init__.py", line 20 in <module>
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 655 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "train.py", line 1 in <module>
Segmentation fault (core dumped)
```
So I found the error came from line 33 'import torch' in 'transformers/transformers/file_utils.py'.
Then I swapped the position where tensorflow and torch are imported and the error disappeared.
I wonder whether it is a bug of transformer or torch and what caused it.


PRs error which occurs many times in the last days

## 🐛 Bug

I've seen that many recent PRs (e.g. [2201](https://github.com/huggingface/transformers/pull/2201), [2203](https://github.com/huggingface/transformers/pull/2203), [2190](https://github.com/huggingface/transformers/pull/2190), [2189](https://github.com/huggingface/transformers/pull/2189), ...) have encountered the same error reported below. All the PRs I've mentioned above passed 7 different checks, but not the **`ci/circleci: build_py3_torch_and_tf`**.
```
python -m pytest -sv ./transformers/tests/ --cov
============================= test session starts ==============================
platform linux -- Python 3.5.9, pytest-5.3.2, py-1.8.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /home/circleci/transformers
plugins: cov-2.8.1
collecting ... Fatal Python error: Segmentation fault

Current thread 0x00007f0afaba9740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 938 in create_module
  File "<frozen importlib._bootstrap>", line 577 in module_from_spec
  File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/usr/local/lib/python3.5/site-packages/torch/__init__.py", line 81 in <module>
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 697 in exec_module
  File "<frozen importlib._bootstrap>", line 673 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/home/circleci/transformers/transformers/file_utils.py", line 38 in <module>
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 697 in exec_module
  File "<frozen importlib._bootstrap>", line 673 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/home/circleci/transformers/transformers/__init__.py", line 20 in <module>
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 697 in exec_module
  File "<frozen importlib._bootstrap>", line 673 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 957 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 943 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "<frozen importlib._bootstrap>", line 222 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 943 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 968 in _find_and_load
  File "/usr/local/lib/python3.5/site-packages/py/_path/local.py", line 701 in pyimport
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 492 in _importtestmodule
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 424 in _getobj
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 248 in obj
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 440 in _inject_setup_module_fixture
  File "/usr/local/lib/python3.5/site-packages/_pytest/python.py", line 427 in collect
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 254 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 234 in from_call
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 254 in pytest_make_collect_report
  File "/usr/local/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/site-packages/_pytest/runner.py", line 373 in collect_one_node
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 717 in genitems
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 720 in genitems
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 492 in _perform_collect
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 454 in perform_collect
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 255 in pytest_collection
  File "/usr/local/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 245 in _main
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 196 in wrap_session
  File "/usr/local/lib/python3.5/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
  File "/usr/local/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/usr/local/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/lib/python3.5/site-packages/_pytest/config/__init__.py", line 92 in main
  File "/usr/local/lib/python3.5/site-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/local/lib/python3.5/runpy.py", line 85 in _run_code
  File "/usr/local/lib/python3.5/runpy.py", line 193 in _run_module_as_main

Received "segmentation fault" signal
```
## Expected behavior

When a new PR comes into Transformers, it does not generate this bug.

## Environment

* OS: **Ubuntu 16.04**
* Python version: **3.6.9**
* PyTorch version: **1.3.1**
* PyTorch Transformers version (or branch): **master** (installed with `pip install transformers`)
* Using GPU **Indifferent**
* Distributed of parallel setup **Indifferent**
* Any other relevant information:


fix: wrong architecture count in README

Just say “the following” so that this intro doesn't so easily fall out of date :)


weights not initialised in pre-trained Roberta

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Robert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: see below

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: I am importing pre-trained models

## To Reproduce

When running the following code:

```
import logging
logging.basicConfig(level=logging.INFO)
from transformers import RobertaForMaskedLM
pt_m = RobertaForMaskedLM.from_pretrained('roberta-base')
```

I am getting the following messages in the log:

```
INFO:transformers.modeling_utils:loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin from cache at /home/bartosz/.cache/torch/transformers/228756ed15b6d200d7cb45aaef08c087e2706f54cb912863d2efe07c89584eb7.49b88ba7ec2c26a7558dda98ca3884c3b80fa31cf43a1b1f23aef3ff81ba344e
INFO:transformers.modeling_utils:Weights of RobertaForMaskedLM not initialized from pretrained model: ['lm_head.decoder.weight']
INFO:transformers.modeling_utils:Weights from pretrained model not used in RobertaForMaskedLM: ['lm_head.weight']
```

In particular, I am concerned that the weight of the LM head were not initialised from the stored values.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

The model should load and all weights should get initialized from the pre-trained model.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Linux
* Python version: 3.7
* PyTorch version: 1.2.1
* PyTorch Transformers version (or branch): 2.2.2
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

If I comment out these two lines the messages disappear:

https://github.com/huggingface/transformers/blob/f061606277322a013ec2d96509d3077e865ae875/transformers/modeling_utils.py#L445-L446

<!-- Add any other context about the problem here. -->


[WAITING YOUR REVIEW] Issue #2196: now it's possible to save PreTrainedEncoderDecoder objects correctly to file system

**Details implemented in this PR:**

- [X] Create the output directory (whose name is passed by the user in the "save_directory" parameter) where it will be saved encoder and decoder, if not exists.
- [X] Empty the output directory, if it contains any files or subdirectories.
- [X] Create the "encoder" directory inside "save_directory", if not exists.
- [X] Create the "decoder" directory inside "save_directory", if not exists.
- [X] Save the encoder and the decoder in the previous two directories, respectively.


run_ner.py example fails

I am trying to run the run_ner.py example described here: (https://huggingface.co/transformers/examples.html#named-entity-recognition)

When running the example I get the following exception:
```
I1217 09:36:10.744300 14416 file_utils.py:40] PyTorch version 1.3.1 available.
W1217 09:36:11.329299 14416 run_ner.py:422] Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
I1217 09:36:11.863193 14416 configuration_utils.py:160] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert_base_uncased/config.json from cache at s3_cache\67674340071d93960fbc3eb74cf1d0b51de232689bfc75d63e3f4ab1c9a052f9
Traceback (most recent call last):
  File ".\run_ner.py", line 531, in <module>
    main()
  File ".\run_ner.py", line 441, in main
    cache_dir=args.cache_dir if args.cache_dir else None)
  File "C:\Program Files\Python\Python37\lib\site-packages\transformers\configuration_utils.py", line 163, in from_pretrained
    config = cls.from_json_file(resolved_config_file)
  File "C:\Program Files\Python\Python37\lib\site-packages\transformers\configuration_utils.py", line 196, in from_json_file
    return cls.from_dict(json.loads(text))
  File "C:\Program Files\Python\Python37\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\Python\Python37\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
A JSON files does not seemed to be found. Can someone tell me which file this is?


How to add traditional features for transformers?

## ❓ Questions & Help

I don't know whether models like Bert can capture some traditional features or not, such as the number of nouns in a sentence, the number of words in a sentence, etc. But I want to add these features to Bert or xlnet for training. Can anyone help me?


How to output labels for GLUE test set

## ❓ Questions & Help

Thanks for your great work.

I have done some modification based on your library. I'd like to test it on GLUE test set (not dev set). Is there any way I can do it? Do you have a script for output GLUE submission files?

Thanks.


XLNet fine-tuning speed (Multi-label classification)

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi!

I'm wondering whether my speed of fine-tuning is normal.

Training is taking **2~3hours per epoch**.

I am using [fast-bert](https://github.com/kaushaltrivedi/fast-bert) to train multi-label classifier, with
- model = xlnet-base-cased
- max sequence length = 512 tokens
- mem states using = no
- gpu = V100 * 4
- training data amount = 110000 ~ 120000
- validation data amount = 30000 ~ 40000
- evaluation = per each epoch, per every 100 steps
- apex, f16 used

Thank you so much!


Error while saving Pretrained model for Encoder and decoder

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Define a model and try to save the model then the error occurs. This is because the encoder and decoder model is save in different directories and the directories are not created.
In order to handle this, you need to check if the given path exists then the encoder folder exists if not create a folder above this [line](https://github.com/huggingface/transformers/blob/3f5ccb183e3cfa755dea2dd2afd9abbf1a0f93b8/transformers/modeling_encoder_decoder.py#L169)

Code to reproduce:
```
 from transformers import PreTrainedEncoderDecoder
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased', 'bert-base-uncased', cache_dir='../transformers/cache')
model.save_pretrained(final_model_output)
```
Error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guest_1/anaconda3/envs/my_env/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 167, in save_pretrained
    self.encoder.save_pretrained(os.path.join(save_directory, "encoder"))
  File "/home/guest_1/anaconda3/envs/my_env/lib/python3.6/site-packages/transformers/modeling_utils.py", line 239, in save_pretrained
    assert os.path.isdir(save_directory), "Saving path should be a directory where the model and configuration can be saved"
AssertionError: Saving path should be a directory where the model and configuration can be saved
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

    OS: ubuntu
    Python version: 3.6
    PyTorch version:1.3.0
    PyTorch Transformers version (or branch):2.2.0
    Using GPU ? Yes
    Distributed of parallel setup ? No
    Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Fixing checks test pr, will be closed


Improve TextDataset building/tokenization (6x faster; Enable large dataset file usage)

## This PR:
- Chunks the reading of the dataset file used to create a `TextDataset` for training, if you used a file of any larger size (for my case I had 3.5GB txt file I was able to get finish in ~45 min) the program would just hang at `f.read()` 😢
- Speeds up `lowercase_text` in the `BasicTokenizer` with a simpler regex scheme
- Add `@functools.lru_cache()` to several functions responsible for acting on individual chars
- Use multiprocessing to drastically speed up tokenization inside `TextDataset` constructor

## Checkouts & Performance Profiling
Benchmark script I used to clock speeds 👉 [gist](https://gist.github.com/mttcnnff/f192d8933f2a8d2b58e14b53841c4080)

Comparison script I used to compare results 👉 [gist](https://gist.github.com/mttcnnff/503f9facd3c317e7efc61a43295a335f)

### Performance on master with no changes on a ~16.5MB txt file (~1 min):

### Performance after all changes applied on same ~16.5MB txt file (~10 seconds):


Fine-tuning GPT2 or BERT and adding new vocabulary?

## ❓ Questions & Help

I am fine-tuning the GPT2 on a domain-specific corpus and I was wondering if there is a way to add new vocabulary to the model. I am checking the ```vocab.json``` file after saving the model into ```output``` directory and don't find the any new vocabulary in there.


Bug fix: PyTorch loading from TF and vice-versa

Fix loading a PyTorch model from TF and vice-versa when model architectures are not the same

Fix #2109


Numpy compatibility for sentence piece


Adding Finnish BERT.

We have trained BERT-base on Finnish text and wish to have it included in the library. Both cased and uncased models are available. You can see the paper [here](https://arxiv.org/abs/1912.07076) and a website for the model can be found [here](http://turkunlp.org/FinBERT/).

These changes passed all the relevant tests (\*auto\*, \*common\*, \*_bert_test\*) including `test_model_from_pretrained` with the Finnish models.


Add support for XLM-RoBERTa

Hi,

this model adds support for the recently released XLM-RoBERTa model from the Facebook AI team.

XLM-RoBERTa is described in the ["Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) paper from Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.

The model itself is integrated into `fairseq` library and weights for the base and large XLM-R are available, see example [here]().

## Results

### NER

This PR also extends the `run_ner` script to support XLM-R. Results for NER (CoNLL datasets):

#### Base model

| Model      | English             | Dutch               | Spanish             | German              | Avg.
| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | -------------------
| Paper      |   -   (dev) / 91.95 |   -   (dev) / 91.21 |   -   (dev) / 88.46 |   -   (dev) / 83.65 |   -   (dev) / 88.82
| Reproduced | 95.31 (dev) / 91.20 | 91.66 (dev) / 91.37 | 85.23 (dev) / 88.15 | 87.11 (dev) / 84.02 | 89.83 (dev) / 88.69

#### Large model

| Model      | English             | Dutch               | Spanish             | German              | Avg.
| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | -------------------
| Paper      |   -   (dev) / 92.74 |   -   (dev) / 93.25 |   -   (dev) / 89.04 |   -   (dev) / 85.53 |   -   (dev) / 90.14
| Reproduced | 96.84 (dev) / 92.80 | 94.02 (dev) / 94.41 | 88.94 (dev) / 89.30 | 88.60 (dev) / 86.04 | 92.10 (dev) / 90.64

Parameters used for reproducing the paper results: 20 training epochs with a learning rate of `5.0e-6` and a batch size of 16. Only one run is reported here.

## Tasks

* [x] Upload model to 🤗/ Transformers S3
* [x] Add support for base model (convert script needs to be adjusted)
* [x] Report results for NER (CoNLL datasets)
* [x] Add XLM-R to `Auto*` interfaces
* [x] Check tokenization methods


About QuestionAnswering on SQuAD2.0 Dataset

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Do anyone understand the paper " QuestionAnswering on SQuAD2.0 Dataset" in section 5.1. It says
 "As we increase the number of epochs in training, the performance of the answerable questions is improved while the performance for the non-answerable questions drop hugely"
<img width="393" alt="擷取" src="https://user-images.githubusercontent.com/32416416/70907720-88033d00-2044-11ea-9ab9-1f2437aefd11.PNG">
I want to know why cause this situation?
And it also says  One possible solution is that, the no-answer indicator is only from the ﬁrst [CLS] token, the value of attentions to the [CLS] token may be much weaker than the word-word attention. Hence the Transformer may focus less on the attention associated with the [CLS] token.
what does this mean about " the Transformer may focus less on the attention associated with the [CLS] token."
Can anyone help? Thans a lot


Output diverging on different GPUs using same prompt?

## 🐛 Bug

Wondering if anyone else is noticing this, or I'm missing something.

GPT-2 transformer run_generation.py

Running on both AWS p2 and p3, which have different GPUs.

Same text seed, same numerical seed (default:  42)

The output is identical for a long segment, and then suddenly diverges, picking a different word on the P3, and then carrying on in a different direction after that.

I'll do some more investigation and try to post examples.  But first:  Has anyone else noticed anything like this?  Is this normal behavior?


summarization code is incomplete

Hi
in the summarization code you have removed all the training part, why is that?
Solely evaluating an existing model does not really have any point.
While I really find this repo great, incomplete work like this summarization folder, defenitely degrade from the dignity of this repo. I greatly appreciate either remove this summarization folder, or properly implementing it.


RuntimeError: CUDA error: device-side assert triggered when using Roberta

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
Roberta
Language I am using the model on (English, Chinese....):
English
The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

```
class QuestModel(nn.Module):
    def __init__(self, n_classes=30):
        super(QuestModel, self).__init__()
        self.model_name = 'QuestModel'
        self.bert_model = models[MODEL_NAME].from_pretrained(MODEL_NAME)
        self.fc = nn.Linear(LINEAR_LAYER[MODEL_NAME], n_classes)
        self.dropout = nn.Dropout(p=0.2)

    def forward(self, ids, seg_ids):
        attention_mask = (ids > 0).float()
        layers, pool_out = self.bert_model(input_ids=ids, token_type_ids=seg_ids, attention_mask=attention_mask)

        out = self.dropout(pool_out)
        logit = self.fc(out)
        return logit

......
for i, (ids, seg_ids, labels) in enumerate(data_loader):
            ids, seg_ids, labels = ids.cuda(), seg_ids.cuda(), labels.cuda()
            outputs = data_parallel(model, (ids, seg_ids))
            scores = torch.sigmoid(outputs)
            loss = loss_fn(outputs, labels)
            # loss = custom_loss(pred, y_batch.to(device))
            preds.append(outputs.cpu().numpy())
            original.append(labels.cpu().numpy())

            avg_loss += loss.item() / len(data_loader)
......
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)
Google quest QA label

## To Reproduce

Steps to reproduce the behavior:

1. I have a script that worked fine when I use bert from transformers package
2. Then I change tokenizer and model to roberta
3. Always got the following errors

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

```
/opt/conda/conda-bld/pytorch_1565272279342/work/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [130,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "/mnt/home/dunan/Learn/Kaggle/google_quest/kaggle_google_quest/train.py", line 267, in <module>
    train(args)
  File "/mnt/home/dunan/Learn/Kaggle/google_quest/kaggle_google_quest/train.py", line 138, in train
    score, val_loss = predict(model, val_loader)
  File "/mnt/home/dunan/Learn/Kaggle/google_quest/kaggle_google_quest/train.py", line 252, in predict
    preds.append(outputs.cpu().numpy())
RuntimeError: CUDA error: device-side assert triggered
```

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: CentOS
* Python version: 3.6
* PyTorch version: 1.2
* PyTorch Transformers version (or branch): 2.2.2
* Using GPU ? Yes
* Distributed of parallel setup ? All happened when I using 1, 2, 4 Gpu
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


T5Tokenizer: Using cls_token, but it is not set yet.

## 🐛 Bug

Model I am using (Bert, XLNet....): T5

## To Reproduce

Steps to reproduce the behavior:

1. Load T5Tokenizer
2. Try getting the CLS or SEP token: `tokenizer.sep_token` or `tokenizer.cls_token`
3. An error will be raised "Using cls_token, but it is not set yet."

Running the latest commit on the master branch.

I imagine that the T5 implementation is not complete yet, but I thought I'd point it out anyway. I figured that T5 expects the same input as BERT, as stated in the source:

https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/modeling_t5.py#L651-L658

As an aside, I saw this incorrect mention of XLNetTokenizer in the T5Tokenizer. Probably overlooked while working on adding _yet another model_ to transformers. You guys are crazy!

https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/tokenization_t5.py#L112-L120


Unit of the prediction scores of a language model

I have used the base transformer models for downstream tasks for a while now but I haven't had the time to dig into how the models were actually trained. When looking at the *ForMaskedLM models, I can see the return tuple contains `prediction_scores` for each token.

> prediction_scores: torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)
> Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

One could get the probabilities of vocabulary items by running a SoftMax over these prediction_scores, but my question is: what are these outputs themselves, what is their unit? In other words: during training, how were these outputs used? Since they are the primary output in the tuple, I suppose they were used in the loss function. At first I expected these to be **perplexity** but since they are returned before any softmax (and perplexity is 2^entropy), I don't see how that can be true. Still, these scores seem to be used to to get the most likely masked token in the [quickstart](https://huggingface.co/transformers/quickstart.html#bert-example). So if it's not probability and not perplexity, then what is its unit?


sts-b task score is far worse than other GLUE tasks

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello,
I'm evaluating my GPT style model pretrained on TEXT8 dataset with GLUE.
Below is the evaluation result.

```
CoLA | SST-2 |     MRPC    |    QQP    |    STS-B    | MNLI | QNLI | RTE  | WNLI
19.1 |   85  | 82.5 / 71.6 | 78.4 / 82 | 41.6 / 39.4 | 62.8 | 74.6 | 57.4 | 56.3
```

Compare to other GLUE tasks, STS-B scores seem far worse than baseline in leader board.
Does anyone know why model works bad on STS? Thanks a lot.


Conda version is not the latest

## 🚀 Feature

The conda package in conda forge channel (v2.1.1) is not the latest released version (v2.2.2) so the ALBERT model is missing from the package.

## Motivation

In conda environment we need the latest packages containing the ALBERT model.

## Additional context

Also, for the ALBERT model in its code it has printing comments for each 12 layers. It shouldn't be there I guess in a production ready version.


Pretty sure patch in Pull Request #1313 is incorrect

## 🐛 Bug

The bug was introduced in pull request #1313

If a stop_token is specified, but does not exist in **text**, then the last character of the text string is trimmed off.  text.find will return -1 in that case, which seems to remove the last character from the string.

Example:
--stop_token="wake"

`text = tokenizer.decode(o, clean_up_tokenization_spaces=True)`

text = ' fishing with two ladies and the same two ladies had been fishing with him and all three ladies had run'

` text = text[: text.find(args.stop_token) if args.stop_token else None]`

text = ' fishing with two ladies and the same two ladies had been fishing with him and all three ladies had ru'

See how that final 'n' is trimmed off, even though "wake" does not occur in this string?

The fix should be to first find out if stop_token occurs in the text before doing the trimming, maybe:

```
if args.stop_token :
    loc = text.find(args.stop_token)
    if loc != -1 :+1:
        text = text[:loc]
```


Should I always use bert as a teacher to distillation distilbert as a student?

## ❓ Questions & Help
Should I always use bert as a teacher to distill distilbert as a student?
Is it fine RoBERTa model as a teacher to distill [distilbert](https://github.com/huggingface/transformers/blob/master/transformers/modeling_distilbert.py)?
I assume roberta and distilbert use the same tokenizer and dataloader method.


Tokenize with offsets

Similar purpose to https://github.com/huggingface/transformers/pull/1274 (which I also used for most of the testing) but different approach.
It keeps track of token offsets by trying to progressively tokenize the text character by character, and consume matching tokens along the way.
It returns just the start of a span. Tests were added for ALBERT, CTRL and T5.
I think this implementation is more generic and simpler to understand, and the results are very good.