a problem occur when I train Chinese distilgpt2 model

### When I was training a new model from zero to one, the following questions appeared, please help me answer them, thank you very much!

![image](https://user-images.githubusercontent.com/36957508/73609835-53573f00-460c-11ea-87dc-dc68e4829212.png)
![image](https://user-images.githubusercontent.com/36957508/73609850-7124a400-460c-11ea-8abc-bff4117484a1.png)
C:\Users\gaochangkuan\Desktop\transformers-master\examples\distillation>python train.py --student_type gpt2 --student_config training_configs/distilgpt2.json --teacher_type gpt2 --teacher_name distilgpt2 --alpha_ce 5.0 --alpha_mlm 2.0 --alpha_cos 1.0 --alpha_clm 0.0 --mlm --freeze_pos_embs  --data_file data/binarized_text.bert-base-chinese.pickle --token_counts data/token_counts.bert-base-chinese.pickle  --dump_path model   --force
02/02/2020 22:26:54 - INFO - transformers.file_utils - PID: 27864 -  PyTorch version 1.4.0+cpu available.
02/02/2020 22:27:02 - INFO - utils - PID: 27864 -  Experiment will be dumped and logged in model
02/02/2020 22:27:02 - INFO - utils - PID: 27864 -  Param: Namespace(adam_epsilon=1e-06, alpha_ce=5.0, alpha_clm=0.0, alpha_cos=1.0, alpha_mlm=2.0, alpha_mse=0.0, batch_size=5, checkpoint_interval=4000, data_file='data/binarized_text.bert-base-chinese.pickle', dump_path='model', force=True, fp16=False, fp16_opt_level='O1', freeze_pos_embs=True, freeze_token_type_embds=False, gradient_accumulation_steps=50, group_by_size=True, initializer_range=0.02, is_master=True, learning_rate=0.0005, local_rank=0, log_interval=500, master_port=-1, max_grad_norm=5.0, mlm=True, mlm_mask_prop=0.15, mlm_smoothing=0.7, multi_gpu=False, n_epoch=3, n_gpu=0, restrict_ce_to_mask=False, seed=56, student_config='training_configs/distilgpt2.json', student_pretrained_weights=None, student_type='gpt2', teacher_name='distilgpt2', teacher_type='gpt2', temperature=2.0, token_counts='data/token_counts.bert-base-chinese.pickle', warmup_prop=0.05, weight_decay=0.0, word_keep=0.1, word_mask=0.8, word_rand=0.1)
Using cache found in C:\Users\gaochangkuan/.cache\torch\hub\huggingface_pytorch-pretrained-BERT_master
02/02/2020 22:27:12 - INFO - transformers.configuration_utils - PID: 27864 -  loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-config.json from cache at C:\Users\gaochangkuan\.cache\torch\transformers\8a3b1cfe5da58286e12a0f5d7d182b8d6eca88c08e26c332ee3817548cf7e60a.3767c74c8ed285531d04153fe84a0791672aff52f7249b27df341dbce09b8305
02/02/2020 22:27:12 - INFO - transformers.configuration_utils - PID: 27864 -  Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "directionality": "bidi",
  "do_sample": false,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 21128
}

02/02/2020 22:27:22 - INFO - transformers.tokenization_utils - PID: 27864 -  loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-vocab.txt from cache at C:\Users\gaochangkuan\.cache\torch\transformers\8a0c070123c1f794c42a29c6904beb7c1b8715741e235bee04aca2c7636fc83f.9b42061518a39ca00b8b52059fd2bede8daa613f8a8671500e518a8c29de8c00
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Special tokens {'unk_token': 100, 'sep_token': 102, 'pad_token': 0, 'cls_token': 101, 'mask_token': 103}
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Loading data from data/binarized_text.bert-base-chinese.pickle
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Loading token counts from data/token_counts.bert-base-chinese.pickle (already pre-computed)
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Splitting 124 too long sequences.
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Remove 2840 too short (<=11 tokens) sequences.
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Remove 0 sequences with a high level of unknown tokens (50%).
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  30807 sequences
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Data loader created.
02/02/2020 22:27:22 - INFO - utils - PID: 27864 -  Loading student config from training_configs/distilgpt2.json
02/02/2020 22:27:22 - INFO - transformers.configuration_utils - PID: 27864 -  loading configuration file training_configs/distilgpt2.json
02/02/2020 22:27:22 - INFO - transformers.configuration_utils - PID: 27864 -  Model config GPT2Config {
  "architectures": null,
  "attn_pdrop": 0.1,
  "bos_token_id": 0,
  "do_sample": false,
  "embd_pdrop": 0.1,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_epsilon": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 6,
  "n_positions": 1024,
  "num_beams": 1,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 21128
}

02/02/2020 22:27:24 - INFO - utils - PID: 27864 -  Student loaded.
02/02/2020 22:27:24 - INFO - transformers.configuration_utils - PID: 27864 -  loading configuration file E:\GPT2_Text_generation\GPT2-Chinese-master\GPT2Model\config.json
02/02/2020 22:27:24 - INFO - transformers.configuration_utils - PID: 27864 -  Model config GPT2Config {
  "architectures": null,
  "attn_pdrop": 0.1,
  "bos_token_id": 0,
  "do_sample": false,
  "embd_pdrop": 0.1,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_epsilon": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 10,
  "n_positions": 1024,
  "num_beams": 1,
  "num_labels": 1,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": true,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 21128
}

02/02/2020 22:27:24 - INFO - transformers.modeling_utils - PID: 27864 -  loading weights file E:\GPT2_Text_generation\GPT2-Chinese-master\GPT2Model\pytorch_model.bin
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  Teacher loaded from distilgpt2.
21128 21128
768 768
1024 1024
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  Initializing Distiller
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  Using [0, 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63, 67, 71, 75, 79, 83, 87, 91, 95, 99, 103, 107, 111, 115, 119, 123, 127, 131, 135, 139, 143, 147, 151, 155, 159, 163, 167, 171, 175, 179, 183, 187, 191, 195, 199, 203, 207, 211, 215, 219, 223, 227, 231, 235, 239, 243, 247, 251, 255, 259, 263, 267, 271, 275, 279, 283, 287, 291, 295, 299, 303, 307, 311, 315, 319, 323, 327, 331, 335, 339, 343, 347, 351, 355, 359, 363, 367, 371, 375, 379, 383, 387, 391, 395, 399, 403, 407, 411, 415, 419, 423, 427, 431, 435, 439, 443, 447, 451, 455, 459, 463, 467, 471, 475, 479, 483, 487, 491, 495, 499, 503, 507, 511, inf] as bins for aspect lengths quantization
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  Count of instances per bin: [1267 1907 1866 1702 1584 1483 1380 1237 1205 1101 1047  974  882  854
  758  672  598  583  593  519  492  453  444  414  371  338  352  305
  290  298  260  250  260  214  210  200  189  190  149  153  121  125
  124  106  116  105   87  100   78  103   73   70   74   78   65   70
   52   43   46   51   48   38   49   28   32   41   34   27   29   31
   28   39   28   23   25   26   17   25   23   12   20   17   17   20
    8   12   15   16    8   11   11   10   13   11    3    9    8    5
    9    5    6    6    6    4   10    4    6    3    3    3    2    4
    3    6    4    3    7    2    6    9    1    2    6    2    3  134]
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  Using MLM loss for LM step.
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  --- Initializing model optimizer
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  ------ Number of trainable parameters (student): 58755072
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  ------ Number of parameters (student): 59541504
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  --- Initializing Tensorboard
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  Starting training
02/02/2020 22:27:27 - INFO - utils - PID: 27864 -  --- Starting epoch 0/2
-Iter:   0%|                                                                                | 0/6162 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train.py", line 329, in <module>
    main()
  File "train.py", line 324, in main
    distiller.train()
  File "C:\Users\gaochangkuan\Desktop\transformers-master\examples\distillation\distiller.py", line 355, in train
    self.step(input_ids=token_ids, attention_mask=attn_mask, lm_labels=lm_labels)
  File "C:\Users\gaochangkuan\Desktop\transformers-master\examples\distillation\distiller.py", line 385, in step
    input_ids=input_ids, attention_mask=attention_mask
**ValueError: too many values to unpack (expected 2)**


# 🐛 Bug

## Information

Model I am using (TFBertModel):

Language I am using the model on (English):

Also I'm using `tensorflow==2.1.0` and `transformers==2.3.0`

The problem arises when using:
* [x] the official example scripts: (give details below)
I'm trying to use the `optimization_tf.create_optimizer` from the source code.

The tasks I am working on is:
* [x] my own task or dataset: (give details below)
Just a text classification task.

## To reproduce

Steps to reproduce the behavior:

1. Try to use the model the regular way
2. When running the model with "optimization_tf.create_optimizer"

## Environment info

```
/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py in _process_single_batch(model, inputs, targets, output_loss_metrics, sample_weights, training)
    271                         loss_scale_optimizer.LossScaleOptimizer):
    272             grads = model.optimizer.get_unscaled_gradients(grads)
--> 273           model.optimizer.apply_gradients(zip(grads, trainable_weights))
    274       else:
    275         logging.warning('The list of trainable weights is empty. Make sure that'

TypeError: apply_gradients() missing 1 required positional argument: 'clip_norm'
```

## How I am able to run

On the class `AdamWeightDecay` and the method `apply_gradients` I just call the  supper function like this:

```
def apply_gradients(self, grads_and_vars, name=None):
    return super().apply_gradients(grads_and_vars)
```

but as you can see I'm not using the `clip_norm` as the source example uses.

Is there a  way to use the original source function as described in the source code?


Removed unused fields in DistilBert TransformerBlock

A few fields in the TransformerBlock are unused - this small PR cleans it up.


DistributedDataParallel for multi-gpu single-node runs in run_lm_finetuning.py

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
Modify `run_lm_finetuning.py` with DDP for multi-gpu single-node jobs.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->
In it's current state, `run_lm_finetuning.py` does not run in DDP for multi-gpu single-node training jobs. This results in all but the first GPU having very low utilization (as low as 50%, when the first one is in the high 80%) due to the way simple DP works. Once implemented, the load would be more evenly balanced across all the GPUs.
## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
I can help implementing this feature, but would need guidance on what should/shouldn't be modified to get this working properly.


Can't pickle local object using the finetuning example.

I was testing out the finetuning example from the repo:

`python run_lm_finetuning.py --train_data_file="finetune-output/KantText.txt" --output_dir="finetune-output/hugkant" --model_type=gpt2 --model_name_or_path=gpt2 --do_train --block_size=128`

While saving the checkpoint, it gives the following error:

```

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 790, in <module>
    main()
  File "run_lm_finetuning.py", line 740, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 398, in train
    torch.save(scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 209, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 134, in _with_file_like
    return body(f)
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 209, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 282, in _save
    pickler.dump(obj)
AttributeError: Can't pickle local object 'get_linear_schedule_with_warmup.<locals>.lr_lambda'
```


Fix typo in examples/utils_ner.py

`"%s-%d".format()` -> `"{}-{}".format()`


Load from tf2.0 checkpoint fail

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Download tf2.0 checkpoint from https://storage.googleapis.com/cloud-tpu-checkpoints/bert/keras_bert/uncased_L-12_H-768_A-12.tar.gz
2. unpack the model tar.gz to `bert_models` folder
3. start an iPython console and type following codes:
```python
import tensorflow as tf
from transformers import TFBertModel, BertConfig
config = BertConfig.from_json_file("./bert_models/uncased_L-12_H-768_A-12/bert_config.json")
model = TFBertModel.from_pretrained("./bert_models/uncased_L-12_H-768_A-12/bert_model.ckpt.index", config=config)
```
4. I check the original code from tf2.0 and found they didn't implement model.load_weights when by_name is True. Error is following:
NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment

* OS: CentOS Linux release 7.4.1708 (Core)
* Python version: 3.7.6
* PyTorch version: 1.3.1
* `transformers` version (or branch):
* Using GPU ? Yes
* Distributed or parallel setup ? No
* Any other relevant information:


# ❓ Questions & Help
What is the input for TFBertForSequenceClassification?
## Details
I have a simple multiclass text data on which I want to train the BERT model.

From docs I have found the input format of data:
```a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids])```

In my understanding:
`input_ids`- tokenized sentences, generated from BERT tokenizer.
`attention_mask`- As name suggests it is attention mask. I should use it to mask out padding tokens. Please correct me if I am wrong.
Now what is `token_type_ids'? is it necessary?

When I tried to print output_shape of the model? I got:
`AttributeError: The layer has never been called and thus has no defined output shape.`
So, let's say my dataset has 5 classes. Does this model expect one-hot encoded vector of shape [BATCH_SIZE, CLASSES] for .fit() method?

Also if I don't use .from_pretrained() method, will it load an untrained model?


How to make transformers examples use GPU?

# ❓ Questions & Help

I'm training the run_lm_finetuning.py with wiki-raw dataset. The training seems to work fine, but it is not using my GPU. Is there any flag which I should set to enable GPU usage?

## Details
I'm training the run_lm_finetuning.py with wiki-raw dataset. The training seems to work fine, but it is not using my GPU. Is there any flag which I should set to enable GPU usage?


# 🐛 Bug

## Running run_lm_finetuning.py on bert-base-uncased with wikitext-2-raw does not work.

Model I am using (Bert, XLNet ...): Bert - bert-base-uncased

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [*] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [*] an official GLUE/SQUaD task: train language model on wikitext
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Installed Transformers from the source (git pull and then pip install). Downloaded Wikitext-2 raw dataset.
2. Ran this command ""python run_lm_finetuning.py --output_dir=output --model_type=bert --model_name_or_path=bert-base-uncased --do_train  --train_data_file=E:\\Code\\data\\wikitext-2-raw\\wiki.train.raw --do_eval --eval_data_file=E:\\Code\\data\\wikitext-2-raw\\wiki.test.raw --mlm""
3. This fails in train() method. I haven't touched the code. Stacktraces below:

python run_lm_finetuning.py --output_dir=output --model_type=bert --model_name_or_path=bert-base-uncased --do_train  --train_data_file=E:\\Code\\data\\wikitext-2-raw\\wiki.train.raw --do_eval --eval_data_file=E:\\Code\\data\\wikitext-2-raw\\wiki.test.raw --mlm
2020-01-31 21:51:38.831236: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
01/31/2020 21:51:40 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
01/31/2020 21:51:40 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json not found in cache or force_download set to True, downloading to C:\Users\athre\AppData\Local\Temp\tmp91_kkef0
01/31/2020 21:51:40 - INFO - transformers.file_utils -   copying C:\Users\athre\AppData\Local\Temp\tmp91_kkef0 to cache at C:\Users\athre\.cache\torch\transformers\4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.8f56353af4a709bf5ff0fbc915d8f5b42bfff892cbb6ac98c3c45f481a03c685
01/31/2020 21:51:40 - INFO - transformers.file_utils -   creating metadata file for C:\Users\athre\.cache\torch\transformers\4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.8f56353af4a709bf5ff0fbc915d8f5b42bfff892cbb6ac98c3c45f481a03c685
01/31/2020 21:51:40 - INFO - transformers.file_utils -   removing temp file C:\Users\athre\AppData\Local\Temp\tmp91_kkef0
01/31/2020 21:51:40 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at C:\Users\athre\.cache\torch\transformers\4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.8f56353af4a709bf5ff0fbc915d8f5b42bfff892cbb6ac98c3c45f481a03c685
01/31/2020 21:51:40 - INFO - transformers.configuration_utils -   Model config {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 30522
}

01/31/2020 21:51:40 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache or force_download set to True, downloading to C:\Users\athre\AppData\Local\Temp\tmpx8hth4qr
01/31/2020 21:51:41 - INFO - transformers.file_utils -   copying C:\Users\athre\AppData\Local\Temp\tmpx8hth4qr to cache at C:\Users\athre\.cache\torch\transformers\26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
01/31/2020 21:51:41 - INFO - transformers.file_utils -   creating metadata file for C:\Users\athre\.cache\torch\transformers\26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
01/31/2020 21:51:41 - INFO - transformers.file_utils -   removing temp file C:\Users\athre\AppData\Local\Temp\tmpx8hth4qr
01/31/2020 21:51:41 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at C:\Users\athre\.cache\torch\transformers\26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
01/31/2020 21:51:41 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin not found in cache or force_download set to True, downloading to C:\Users\athre\AppData\Local\Temp\tmpy8kf8hkd
01/31/2020 21:54:14 - INFO - transformers.file_utils -   copying C:\Users\athre\AppData\Local\Temp\tmpy8kf8hkd to cache at C:\Users\athre\.cache\torch\transformers\aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
01/31/2020 21:54:14 - INFO - transformers.file_utils -   creating metadata file for C:\Users\athre\.cache\torch\transformers\aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
01/31/2020 21:54:14 - INFO - transformers.file_utils -   removing temp file C:\Users\athre\AppData\Local\Temp\tmpy8kf8hkd
01/31/2020 21:54:14 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at C:\Users\athre\.cache\torch\transformers\aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
01/31/2020 21:54:16 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
01/31/2020 21:54:18 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=510, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=True, do_train=True, eval_all_checkpoints=False, eval_data_file='E:\\\\Code\\\\data\\\\wikitext-2-raw\\\\wiki.test.raw', evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_steps=-1, mlm=True, mlm_probability=0.15, model_name_or_path='bert-base-uncased', model_type='bert', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='output', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=500, save_total_limit=None, seed=42, server_ip='', server_port='', should_continue=False, tokenizer_name=None, train_data_file='E:\\\\Code\\\\data\\\\wikitext-2-raw\\\\wiki.train.raw', warmup_steps=0, weight_decay=0.0)
01/31/2020 21:54:18 - INFO - __main__ -   Loading features from cached file E:\\Code\\data\\wikitext-2-raw\bert_cached_lm_510_wiki.train.raw
01/31/2020 21:54:18 - INFO - __main__ -   ***** Running training *****
01/31/2020 21:54:18 - INFO - __main__ -     Num examples = 4664
01/31/2020 21:54:18 - INFO - __main__ -     Num Epochs = 1
01/31/2020 21:54:18 - INFO - __main__ -     Instantaneous batch size per GPU = 4
01/31/2020 21:54:18 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
01/31/2020 21:54:18 - INFO - __main__ -     Gradient Accumulation steps = 1
01/31/2020 21:54:18 - INFO - __main__ -     Total optimization steps = 1166
Epoch:   0%|                                                                                | 0/1 [00:00<?, ?it/s]C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.                                                                                | 0/1166 [00:00<?, ?it/s]
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [9,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [10,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [12,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [15,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [18,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [21,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [22,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [24,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [25,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [26,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [28,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [29,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [30,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 790, in <module>
    main()
  File "run_lm_finetuning.py", line 740, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 356, in train
    loss.backward()
  File "E:\Code\torch_env\lib\site-packages\torch\tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "E:\Code\torch_env\lib\site-packages\torch\autograd\__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered
Epoch:   0%|                                                                                | 0/1 [00:00<?, ?it/s]
Iteration:   0%|

## Expected behavior

Training should start.

## Environment

* OS: Windows 10
* Python version:  3.7
* PyTorch version: 1.4 stable
* `transformers` version (or branch): Latest (Jan-31-2020)
* Using GPU ? Yes
* Distributed or parallel setup ? Only 1 GPU
* Any other relevant information:


DistilBERT does not support token type ids, but the tokenizers produce them

```Python
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("distilbert-base-uncased-distilled-squad")
>>> tokenized = tokenizer.encode_plus("I ate a clock yesterday.", "It was very time consuming.")
>>> tokenized
{'input_ids': [101, 1045, 8823, 1037, 5119, 7483, 1012, 102, 2009, 2001, 2200, 2051, 15077, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

>>> model = transformers.AutoModel.from_pretrained("distilbert-base-uncased-distilled-squad")
>>> model(**tokenized)
TypeError: forward() got an unexpected keyword argument 'token_type_ids'
```

In contrast, RoBERTa also does not support token type ids, but its forward method still takes the parameter, and its tokenizer produces type ids that are all zero.


Store Model cards in the repo


Add TF2 version of FlauBERT

Hello,

I worked today to add the new FlauBERT model in TF2 version. Translated models are available in:
```
jplu/tf-flaubert-base-cased
jplu/tf-flaubert-large-cased
jplu/tf-flaubert-small-cased
jplu/tf-flaubert-base-uncased
```


CLI script to gather environment info

I noticed that all too often people leave the "Environment" section in their issue empty. However, things such as the version number of PT/TF and `transformers` itself are very useful to know when trying to debug things.

This PR adds a small script to the existing CLI workflow. Running `python transformers-cli info` will output something like this:

```
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 2.4.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

Note that GPUs being available through the DL framework (`GPU?`) are retrieved automatically, but that users still have to specify whether or not they are actually using the GPU.

In addition, the relevant issue templates have been updated to direct users to the script.


Typo on markdown link in README.md


Albert language model fine tuning not running run_lm_finetuning.py

# 🐛 Bug

## Information

Model I am using (Albert(all types)):

Language I am using the model on (English):

The problem arises when using:
* [ ] the official example scripts: (give details below)
the code returns memory allocation problems when run with any version from albert. i tried to reduce the sequence length and batch size to a minum setting, but the issue still arises. my setting and the minimized setting both run normally with bert or roberta, the issue arises only when i change the model to Albert.
an example:
`tcmalloc: large alloc 1951195136 bytes == 0x7f750f664000 @  0x7f76efbf8887 0x7f764c2a1b79 0x7f764c29fb0f 0x7f764c29fc33 0x7f764c26a155 0x7f764c26837e 0x7f764c26bbb1 0x7f764c2606df 0x50a8af 0x50c5b9 0x509d48 0x50aa7d 0x50c5b9 0x508245 0x509642 0x595311 0x5a067e 0x50d966 0x58efc9 0x4c9546 0x5886f4 0x58892e 0x551b81 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x50a080 0x50aa7d 0x50c5b9 0x508245`

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)
language model finetuning for albert
## To reproduce

Steps to reproduce the behavior:

1. in run_lm_finetuning add:
` from transformers import (AlbertConfig,
    AlbertForMaskedLM,
    AlbertTokenizer,
)`
2.add to MODEL_CLASSES dictionary:
`    "albert": (AlbertConfig, AlbertForMaskedLM, AlbertTokenizer),`
3. add file text.txt, a similar txt file to the wiki dataset that's mentioned in the docs.
4.run the finetuning script:
`python transformers/examples/run_lm_finetuning.py \
    --output_dir=output \
    --model_type=albert \
    --model_name_or_path=albert-base-v1 \
    --do_train \
    --train_data_file test.txt \
    --block_size 50 \
    --per_gpu_train_batch_size 2 \
    --max_steps 520000 \
    --weight_decay 0.01 \
    --logging_steps 5000 \
    --mlm`

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment

* OS: Google colab
* Python version: 3.7
* PyTorch version: 1.3.1
* `transformers` version (or branch): latest
* Using GPU ? yes
* Distributed or parallel setup ? no
* Any other relevant information:


Missing `do_sample` argument for run_generation example

# ❓ Questions & Help

It seems the arguments `k`, `p`, `temperature` are disabled because `do_sample` is set to False by default. Thus, [run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py)  will always use greedy decoding no matter how the `k`, `p`, `temperature` are set, which is kind of misleading.

I think `do_sample` argument should be included in the code.


get_linear_schedule_with_warmup method can't be found in optimization.py

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

My transformer was downloaded from anaconda cloud by using command "conda install -c conda-forge transformers", when I tried to use AdamW as what you show in the example, I found there is no **get_linear_schedule_with_warmup** in the **transformers.optimization.py** file for me to create a scheduler.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* `transformers` version (or branch):
* Using GPU ?
* Distributed or parallel setup ?
* Any other relevant information:


AutoModel fails to load FlauBERT with `output_hidden_states`

MWE:

```python
import transformers

model = transformers.AutoModel.from_pretrained("flaubert-base-cased", output_hidden_states=True)
```

Tested on rev 5a6b138 fails with

```console
Traceback (most recent call last):
  File "mwe.py", line 3, in <module>
    model = transformers.AutoModel.from_pretrained("flaubert-base-cased", output_hidden_states=True)
  File "<redacted>/transformers/modeling_auto.py", line 377, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "<redacted>/transformers/modeling_utils.py", line 463, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'output_hidden_states'
```

This works when loading directly from `transformers.FlaubertModel`.


Input file format for examples/run_lm_finetuning.py

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I wanted to employ the `examples/run_lm_finetuning.py` from the transformers repository on a pretrained Bert model. However, from following the documentation it is not evident how a corpus file should be structured (apart from referencing the Wiki-2 dataset). I've tried

- One document per line (multiple sentences)
- One sentence per line. Documents are separated by a blank line (this I found in some older pytorch-transformers documentation)

By looking at the code of `examples/run_lm_finetuning.py` it is not directly evident how sequence pairs for the Next Sentence Prediction objective are formed. Would the --line-by-line option help here? I'd be grateful, if someone could give me some hints how a text corpus file should look like.

Many thanks and cheers,

nminds

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
[SO link](https://stackoverflow.com/questions/60001698/how-exactly-should-the-input-file-be-formatted-for-the-language-model-finetuning)


Regarding distlbert uncased model's size

I finetuned the distlbert uncased model, and I had a thought that since it is a lower layer model, it should have less weight. But, to my surprise, I find that the model generated after finetuning, I saved it as :
      tf.saved_model.save(model, "./tempdir/distilbert/2/")
and the tf model got saved.
this model has a very high size(of 810 Mb), although it should be less .And the original bert mdoel which is a large model has a size of 410 Mb.
Please look into the matter


how can i finetune BertTokenizer?

Is it possible to fine tune BertTokenizer so that the new vocab.txt file which it uses gets updated on my custom dataset? or do i need to retrain the bert model from scratch for the same?


Hardware requirements for BERT QA inference

Hi,

I am using **bert-large-uncased-whole-word-masking-finetuned-squad** model for QA inference.

I used my laptop's CPU to build the pipeline and try it out.

Now I want to deploy it, and so I would like to know what is the minimum hardware requirement?(If I use the same settings as in your usage example script)?

I am more interested in the **minimum size of the GPU**.
Of course I don't want it to be too slow. Is there any studies on this matter, or measurement?

Thanks!


Correct PyTorch distributed training command in examples/README.md

Running the command currently detailed in the documentation yields
`58.4/70.3 EM/F1` with Ubuntu 18.04 and torch 1.4.0, not `86.9/93.1` as promised. It also looks wrong because we're using a cased model with `--do_lower_case`.

Switching it to match the PyTorch distributed training example given in the main README gives me the approximately-correct `86.68/93.03` results.


Config: reference array of architectures


Issue about pipeline of sentiment-analysis

Hi,
I tried the `pipeline` code on the README:
```
from transformers import pipeline

nlp = pipeline('sentiment-analysis')
print(nlp('We are very happy to include pipeline into the transformers repository.'))
```

However, it shows the following error:
```
I0131 01:02:23.627610 4420611520 file_utils.py:35] PyTorch version 1.1.0.post2 available.
I0131 01:02:28.316742 4420611520 file_utils.py:362] https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache or force_download set to True, downloading to /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmp_lgvnb3l
I0131 01:02:30.348834 4420611520 file_utils.py:377] copying /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmp_lgvnb3l to cache at /Users/yantong/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
I0131 01:02:30.349704 4420611520 file_utils.py:381] creating metadata file for /Users/yantong/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
I0131 01:02:30.350285 4420611520 file_utils.py:390] removing temp file /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmp_lgvnb3l
I0131 01:02:30.350615 4420611520 tokenization_utils.py:398] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /Users/yantong/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
I0131 01:02:31.742653 4420611520 file_utils.py:362] https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-config.json not found in cache or force_download set to True, downloading to /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmp7wk7u8_r
I0131 01:02:32.793608 4420611520 file_utils.py:377] copying /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmp7wk7u8_r to cache at /Users/yantong/.cache/torch/transformers/437d6b3001e14ea1853bcee09a1b2557f230862c5a03d3ebd78a4cdb94a79020.7a412cd94061214ced4285ea8f65100868e4c9757c85781d11a83acd01fa14a4
I0131 01:02:32.794027 4420611520 file_utils.py:381] creating metadata file for /Users/yantong/.cache/torch/transformers/437d6b3001e14ea1853bcee09a1b2557f230862c5a03d3ebd78a4cdb94a79020.7a412cd94061214ced4285ea8f65100868e4c9757c85781d11a83acd01fa14a4
I0131 01:02:32.794360 4420611520 file_utils.py:390] removing temp file /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmp7wk7u8_r
I0131 01:02:32.794815 4420611520 configuration_utils.py:185] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-config.json from cache at /Users/yantong/.cache/torch/transformers/437d6b3001e14ea1853bcee09a1b2557f230862c5a03d3ebd78a4cdb94a79020.7a412cd94061214ced4285ea8f65100868e4c9757c85781d11a83acd01fa14a4
I0131 01:02:32.795076 4420611520 configuration_utils.py:199] Model config {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "finetuning_task": "sst-2",
  "hidden_dim": 3072,
  "id2label": {
    "0": "NEGATIVE",
    "1": "POSITIVE"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "NEGATIVE": 0,
    "POSITIVE": 1
  },
  "max_position_embeddings": 512,
  "n_heads": 12,
  "n_layers": 6,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 30522
}

I0131 01:02:33.892992 4420611520 file_utils.py:362] https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-modelcard.json not found in cache or force_download set to True, downloading to /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmpgd57b64v
I0131 01:02:35.001976 4420611520 file_utils.py:377] copying /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmpgd57b64v to cache at /Users/yantong/.cache/torch/transformers/57ded08a298ef01c397973781194aa0abf6176e6f720f660a2b93e8199dc0bc7.455d944f3d1572ab55ed579849f751cf37f303e3388980a42d94f7cd57a4e331
I0131 01:02:35.002447 4420611520 file_utils.py:381] creating metadata file for /Users/yantong/.cache/torch/transformers/57ded08a298ef01c397973781194aa0abf6176e6f720f660a2b93e8199dc0bc7.455d944f3d1572ab55ed579849f751cf37f303e3388980a42d94f7cd57a4e331
I0131 01:02:35.002778 4420611520 file_utils.py:390] removing temp file /var/folders/bg/126twh7d29bdtfqxjf_s2kr40000gn/T/tmpgd57b64v
I0131 01:02:35.003092 4420611520 modelcard.py:154] loading model card file https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-modelcard.json from cache at /Users/yantong/.cache/torch/transformers/57ded08a298ef01c397973781194aa0abf6176e6f720f660a2b93e8199dc0bc7.455d944f3d1572ab55ed579849f751cf37f303e3388980a42d94f7cd57a4e331
I0131 01:02:35.003378 4420611520 modelcard.py:192] Model card: {
  "caveats_and_recommendations": {},
  "ethical_considerations": {},
  "evaluation_data": {},
  "factors": {},
  "intended_use": {},
  "metrics": {},
  "model_details": {},
  "quantitative_analyses": {},
  "training_data": {}
}

I0131 01:02:36.353755 4420611520 modeling_utils.py:406] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-pytorch_model.bin from cache at /Users/yantong/.cache/torch/transformers/f62a0baccbff4fbb83b3b6c63168af997d5aea02fc1a8ea2ab0a26dd79ac6517.461f3160566473d3587f9f4776a5131b1ed527b0d5fccb4b5f06003f457154bc
Traceback (most recent call last):
  File "/Users/yantong/Library/Python/3.7/lib/python/site-packages/transformers/modeling_utils.py", line 415, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location='cpu')
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/serialization.py", line 581, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: unexpected EOF, expected 75086145 more bytes. The file might be corrupted.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/yantong/PycharmProjects/Attribute_Inference_Attack_Reviews/transformer_pipelines.py", line 5, in <module>
    nlp = pipeline('sentiment-analysis')
  File "/Users/yantong/Library/Python/3.7/lib/python/site-packages/transformers/pipelines.py", line 905, in pipeline
    model = model_class.from_pretrained(model, config=config, **model_kwargs)
  File "/Users/yantong/Library/Python/3.7/lib/python/site-packages/transformers/modeling_auto.py", line 601, in from_pretrained
    return DistilBertForSequenceClassification.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
  File "/Users/yantong/Library/Python/3.7/lib/python/site-packages/transformers/modeling_utils.py", line 417, in from_pretrained
    raise OSError("Unable to load weights from pytorch checkpoint file. "
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
```

I tried downloading the `distilbert-base-uncased-finetuned-sst-2-english-pytorch_model.bin` and copied to the `.cache/torch/transformers` directory, but it still doesn't work.

OS: MacOS
transformers version: 2.3.0

Could somebody help me fix this issue? Thanks in advance!


Add layerdrop to Flaubert

This PR adds `layerdrop` to Flaubert.


German Bert tokenizer does not recognize (some) special characters (!,?,...)

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

**Bert**

Language I am using the model on (English, Chinese ...):

**German**

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

`tokenizer = BertTokenizer.from_pretrained('bert-base-german-cased')`
`tokenizer.encode("Hallo!", add_special_tokens=False)`

returns:

`[5850, 26910, 2]`

where `2` is the index of '[UNK]' (unknown) token. Same would happen with 'Hallo?'. In the vocab.txt we have tokens '##!' and '##?' but not '?' or '!'. On the other hand some special characters are recognized, like ':' or ';'.

## Expected behavior

At least such common characters as '!' and '?' should be recognized by tokenizer

## Environment

* OS: Ubuntu
* Python version: 3.7
* `transformers` version (or branch): 2.3


distilbert_multilingual_cased model for multiple language

By when distilbert_multilingual_cased model is going to be released.because I tried to finetune the
distilbert_multilingual_cased model , but it said "OSError: file distilbert-base-multilingual-cased not found", which means the above-mentioned model is not included in the list. Or let me know if I am doing any wrong assumption?


TFCamembertModel

Hi,

I wanted to use the tensorflow version of Camembert : TFCamembertModel, but the implememtation is not available with the v2.3.0 version : https://huggingface.co/transformers/v2.3.0/model_doc/camembert.html.

But TFCamembertModel  seems to be available with another version of transformers : https://huggingface.co/transformers/model_doc/camembert.html.

Is this a new or old version of the library ?

Any way, have you already succeeded importing TFCamembertModel ?

Thanks a lot !


Issue with my profile on the upload/share models webpage

Hello,

The URL of my [profile](https://huggingface.co/jplu) on the upload/share models webpage looks like if there was a model called `jplu`. Any idea why?

Thanks.


How to add a fc classification head to BertForQA to make a MTL-BertForQA model?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Does loss  function in the run_tf_ner.py takes logits or probabilities?

The model used for ner model is TFBertForTokenClassification in case of Tensorflow. According to documentation this model produces logits for every token. But the loss function that is used in `run_tf_ner.py` is `tf.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE)` which has default argument `from_logits=False`. In this case the loss function expects probabilities but not the logits. So i would think that correct loss function should have additional argument `from_logits=True`. I'm building my own code using the `run_ner_tf.py` as an example, that is why i want to understand it. Am i missing something here? Because it looks like people can train the model and get some reasonable results...


Add classifier dropout in ALBERT

As mentioned in the original [paper](https://arxiv.org/pdf/1909.11942.pdf), they separated the dropout rates of the transformer cells and the classifier, moreover, in V2 the dropouts are 0 (expect for the classifier, again).

Current implementation does not supports this and models are not training well (can't reproduce results of GLUE benchmark using V2 models). I manually updated these values and got V2 models converging.

This issue was raised in #2337 and also mentioned in https://github.com/google-research/ALBERT/issues/23

I added a separate parameter in the config file and update the sequence classification head.

Please also update the configuration of ALBERT V2 models ([base](https://huggingface.co/albert-base-v2), [large](https://huggingface.co/albert-large-v2), [xlarge](https://huggingface.co/albert-xlarge-v2)) in your repository.
More specifically, the configuration of the **attention and hidden dropout rates** of ALBERT V2 models in your repository as well (see as in https://tfhub.dev/google/albert_base/3, https://tfhub.dev/google/albert_large/3, https://tfhub.dev/google/albert_xlarge/3 and https://tfhub.dev/google/albert_xxlarge/3)


Bug in consecutive creation of tokenizers with different parameters

# 🐛 Bug
Creating a tokenizer with do_lower_case set to True overwrites it for the consecutive creation.
## Information

Model I am using: bert-base-cased

Language I am using the model on (English, Chinese ...):
English

## To reproduce
```python
from transformers import AutoTokenizer

text = "Hello there!"

tokenizer_first = AutoTokenizer.from_pretrained("bert-base-cased")
print(tokenizer_first.tokenize(text))

tokenizer_forced_lowercase = AutoTokenizer.from_pretrained("bert-base-cased", do_lower_case=True)

print(tokenizer_forced_lowercase.tokenize(text))

tokenizer_second = AutoTokenizer.from_pretrained("bert-base-cased")
print(tokenizer_second.tokenize(text))
```
The output on my machine ubuntu 18.04, transformers 2.3.0 installed just now from the repo:
```
['Hello', 'there', '!']
['hello', 'there', '!']
['hello', 'there', '!']
```

Steps to reproduce the behavior:
Execute code snippet from above.

## Expected behavior
Expected output:
```
['Hello', 'there', '!']
['hello', 'there', '!']
['Hello', 'there', '!']
```
## Environment

* OS: ubuntu 18.04
* Python version: Python 3.7.3
* PyTorch version:  1.2.0
* `transformers` version (or branch): 2.3.0 just installed from master


Flaubert

From PR #2632 by [formiel](https://github.com/formiel).

This PR adds [FlauBERT](https://github.com/getalp/Flaubert). Most of the code is derived from XLM (there are some new features in FlauBERT such as pre_norm and layerdrop).

The failing tests were fixed.


Trouble fine tuning Huggingface GPT-2 on Colab — Assertion error

[Cross posted from SO]

I wish to fine tune Huggingface's GPT-2 transformer model on my own text data. I want to do this on a Google Colab notebook. However, it doesn't seem to work.

I install the various bits and pieces via the Colab:

```
!git clone https://github.com/huggingface/transformers
%cd transformers
!pip install .
!pip install -r ./examples/requirements.txt
```

Following the example, I upload the suggested WikiText sample data to the for training and run the suggested CLI commands in the notebook.

```
!export TRAIN_FILE=wiki.train.raw
!export TEST_FILE=wiki.test.raw

!python run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE
```

This chugs along for a bit, but then I get an assertion error:

```
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 790, in <module>
    main()
  File "run_lm_finetuning.py", line 735, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 149, in load_and_cache_examples
    return TextDataset(tokenizer, args, file_path=file_path, block_size=args.block_size)
  File "run_lm_finetuning.py", line 88, in __init__
    assert os.path.isfile(file_path)
AssertionError
```
When I run this script via CLI on my own machine it works fine, with the problem that it takes forever to do anything. Why does Colab present this specific problem? Thanks!


Best weights/models after fine-tuning gpt2

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
I am fine-tuning gpt2 on a new dataset and it checkpoints after every 50 iterations and saves the model.  It stretches my local storage to extreme and would want to delete redundant models. So my queries are following:

1) Is it possible to only save the best weights (which gave the lowest perplexity/loss on evaluation data)?
2) When we run_generation.py and passed the directory of our fine-tuned model, which model weights are actually used for generation? ( as there are so many checkpoint folders with model weights).
3) And hence, related to above 2 questions how does the model consider "the best" model/weights from the fine-tuned model directory we pass as arguments? and can we just mention the chekcpoint folder as well in run_generation.py?

Thanks


Integrate fast tokenizers library inside transformers

Integrate the BPE-based tokenizers inside transformers.

- [x] Bert (100% match)
- [x] DistilBert (100% match)
- [x] OpenAI GPT (100% match)
- [x] GPT2 (100% match if no trailing \n)
- [x] Roberta (100% match if no trailing \n)
- [x] TransformerXL
- [x] CTRL (No binding will be provided).

Added priority for Tokenizer with fast implementation in `AutoTokenizer` this is done through a new mapping (name: class) -> (name: Tuple[class, class]) which represents both the Python and Rust implementation classes. if no Rust implementation is available, it is simply set to None. AutoTokenizer will try to pick the Rust class if not None, otherwise it defaults to the Python one.

Added some matching tests which basically checks that there is a huge % of element wise matching tokens. This is set arbitrary to 0.05 (5%) _[i.e. at max, 5% of differences between Python and Rust]_.

Added parameter `return_offsets_mapping=False` over encoding methods which will return the offset mapping if using a Rust tokenizer. If using a Python tokenizer, a warning message is displayed through the module logger and the argument is discarded.


Fine tuning XLMRoberta for Question Answering

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

I'm trying to fine tune XLM Roberta for Question Answering in tensorflow version, and the question is do i need to convert the pytorch pretrained model to tensorflow because 'xlm-roberta-case' = "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-base-pytorch_model.bin", if so how can i do it? i tried to use load_pytorch_model_in_tf2_model() but i had errors!!
 thank you for help in advance.


bert-base-uncased have weird result on Squad 2.0

I followed the example to fine-tuning BERT on SQuAD2.0:
https://huggingface.co/transformers/examples.html#fine-tuning-bert-on-squad1-0

I run the code as follow:
```
python /content/drive/My\ Drive/squad2/run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file /content/drive/My\ Drive/squad2/train-v2.0.json \
  --predict_file /content/drive/My\ Drive/squad2/dev-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/drive/My\ Drive/squad2_model/
```
However, I got weird results as follow:

```
Results: {'exact': 40.722648024930514, 'f1': 44.3783712849203, 'total': 11873, 'HasAns_exact': 81.5620782726046, 'HasAns_f1': 88.88400847939587, 'HasAns_total': 5928, 'NoAns_exact': 0.0, 'NoAns_f1': 0.0, 'NoAns_total': 5945, 'best_exact': 50.11370336056599, 'best_exact_thresh': 0.0, 'best_f1': 50.11370336056599, 'best_f1_thresh': 0.0}
```
'NoAns_exact' and 'NoAns_f1' are zero.

Do I miss anything when running the example code?


is SOP(sentence order prediction) implemented?

# ❓ Questions & Help
I am reviewing huggingface's version of Albert.

However, I cannot find any code or comment about SOP.

I can find NSP(Next Sentence Prediction) implementation from modeling_from src/transformers/modeling_bert.py.

Is SOP inherited from here with SOP-style labeling? or Is there anything I am missing?

## Details
https://stackoverflow.com/questions/59961023/is-sopsentence-order-prediction-implemented


Remove unnecessary `del` in run_tf_glue.py example

Platform: Ubuntu 18.04 (Linux-4.15.0-1054-aws-x86_64-with-Ubuntu-18.04-bionic)
Python: 3.6.9
PyTorch: 1.4.0
TensorFlow: 2.0.0

Running `./examples/run_tf_glue.py` gives `KeyError: 'special_tokens_mask'`.

Diving into the code, it looks like there's an optional keyword argument in [`encode_plus()`](https://github.com/huggingface/transformers/blob/9d87eafd118739a4c121d69d7cff425264f01e1c/src/transformers/tokenization_utils.py#L834) named `return_special_tokens_mask` that defaults to False. I'm guessing that this argument was added recently and the example just needs to be updated?


models and tokenizers trained with pytorch_pretrained_bert are not compatible with transformers

# 📚 Migration

## Information

The models and tokenizers in transformers 2.3.0 are backward incompatible with pytorch_pretrained_bert 0.6.2.

## Details

```
>>> import transformers
>>> transformers.__version__
'2.3.0'
>>> import pytorch_pretrained_bert
>>> pytorch_pretrained_bert.__version__
'0.6.2'
>>> from pytorch_pretrained_bert import OpenAIGPTTokenizer
>>> tokenizer = OpenAIGPTTokenizer.from_pretrained("runs/mymodel")
ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.
>>> len(tokenizer)
40483
>>> from transformers import OpenAIGPTTokenizer
>>> tokenizer = OpenAIGPTTokenizer.from_pretrained("runs/mymodel")
ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.
>>> len(tokenizer)
40478
>>>
```

`runs/mymodel` contains a model that was trained using pytorch_pretrained_bert 0.6.2, specifically with the `transfer-learning-conv-ai` repo.

Expected behavior from transformers 2.3.0: `len(tokenizer)` must be 40483, like with pytorch_pretrained_bert 0.6.2.

## Environment

* OS: Amazon Linux
* Python version: 3.6
* PyTorch version:
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch): 0.6.2
* `transformers` version (or branch): 2.3.0
* Using GPU? Yes
* Distributed or parallel setup? N/A

## Checklist

- [Y] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [Y] I checked if a related official extension example runs on my machine.

-------

UPDATE: this issue is also the case for models, see below:

```
>>> from pytorch_pretrained_bert import OpenAIGPTLMHeadModel
>>> model = OpenAIGPTLMHeadModel.from_pretrained("runs/mymodel")
>>> from transformers import OpenAIGPTLMHeadModel
>>> model = OpenAIGPTLMHeadModel.from_pretrained("runs/mymodel")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 486, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for OpenAIGPTLMHeadModel:
	size mismatch for transformer.tokens_embed.weight: copying a param with shape torch.Size([40483, 768]) from checkpoint, the shape in current model is torch.Size([40478, 768]).
```


How to get .ckpt files for tensorflow  DistilBERT model

`model.save_pretrained('dir')` tf_model.h5 how to get .ckpt files for it


<!-- A clear and concise description of the question. -->

I recently downloaded the [ALBERT_base_v2](https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz) TF pretrained model and converted it to a pytorch with the following code:
`(base) enoch@enoch-pc:~/dl_repos/transformers/src/transformers$ python convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path '/home/enoch/Documents/experiment/ALBERT_pretrained_models/albert_base_v2/albert_base/model.ckpt-best.index' --albert_config_file '/home/enoch/Documents/experiment/ALBERT_pretrained_models/albert_base_v2/albert_base/albert_config.json' --pytorch_dump_path '/home/enoch/Documents/experiment/albert_base_v2.ckpt'`

However, when I run

`tokenizer = AlbertTokenizer.from_pretrained('albert_base_v2.ckpt')`

I get
```
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<timed exec> in <module>

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    307
    308         """
--> 309         return cls._from_pretrained(*inputs, **kwargs)
    310
    311     @classmethod

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
    456         # Instantiate tokenizer.
    457         try:
--> 458             tokenizer = cls(*init_inputs, **init_kwargs)
    459         except OSError:
    460             raise OSError(

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_albert.py in __init__(self, vocab_file, do_lower_case, remove_space, keep_accents, bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, mask_token, **kwargs)
    109
    110         self.sp_model = spm.SentencePieceProcessor()
--> 111         self.sp_model.Load(vocab_file)
    112
    113     @property

~/anaconda3/lib/python3.7/site-packages/sentencepiece.py in Load(self, filename)
    116
    117     def Load(self, filename):
--> 118         return _sentencepiece.SentencePieceProcessor_Load(self, filename)
    119
    120     def LoadOrDie(self, filename):

RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
```

Please, how do I use or load the downloaded pretrained models? Thank you.


Multiple token IDs for same token

## ❓ Questions & Help

I am using GPT2Tokenizer. I observed that some tokens are duplicated in the vocabulary with a space appended to them in the beginning. For example, there exist two separate tokens - `can` and `<space>can`. Both of them are mapped to different token IDs - `5171` and `460` respectively.

Although, both -
```
can_token_id = tokenizer.encode(' can', add_special_tokens=False)
```
and
```
can_token_id = tokenizer.encode('can', add_special_tokens=False)
```
return can_token_id as `[5171]`

Is this an intended property of the tokenizer? Is there a mapping available between what token IDs are just a space appended version of other tokenIDs?


standardize CTRL BPE files - upload models to S3

This PR:
- update CTRL BPE files (`vocab.json` and `merges.txt`) to use a single format for sub-word splitting (selected to use `</w>` at the end of words)
- upload CTRL updated vocabulary files and pytorch model (not updated) to AWS.

cc @mfuntowicz


Updates to the templates

This PR updates the existing GitHub templates. Main changes are:

- motivate users to post general question on Stack Overflow, tagged [huggingface-transformers](https://stackoverflow.com/questions/tagged/huggingface-transformers)
- removing the 'additional context' section as it might not add much and just bloats the template
- changed references to pytorch-transformers to transformers

closes https://github.com/huggingface/transformers/issues/2529


Add check to verify existence of pad_token_id

In batch_encode_plus we have to ensure that the tokenizer has a pad_token_id so that, when padding, no None values are added as padding. That would happen with gpt2, openai, transfoxl.

closes https://github.com/huggingface/transformers/issues/2640


'Embedding' object has no attribute 'shape'

## ❓ Questions & Help

**version**
tensorflow : 2.0.0
tensorflow-gpu : 2.0.0
torch : 1.3.1
transformers : 2.3.0

Also I'm using **google Colab**

I want to convert tf pretrained-model(for Korean) to pytorch model.

I just tried below code
**config = BertConfig.from_json_file(BERT_PATH+'/config.json')
model = BertForPreTraining.from_pretrained(BERT_PATH, from_tf=True, config=config)**

However,
![image](https://user-images.githubusercontent.com/38174055/73248452-58745280-41f6-11ea-9335-e35b46df801a.png)
this error comes out.. ;(

in the BERT_PATH,
-config.json
-model.ckpt.data-00000-of-00001
-model.ckpt.meta
-model.ckpt.index
-vocab.txt

Can anyone help me?
I found some similar issues, but it didn't help...


[Umberto] model shortcuts

cc @loretoparisi @simonefrancia

see #2485


PPLM with Tensorflow

## ❓ Questions & Help

Hello,

I am still quite new to the library, so I do apologize if the answer is straightforward.

The latest release (https://github.com/huggingface/transformers/releases/tag/v2.3.0) mentions the inclusion of PPLM as a new architecture, both as Pytorch and TF.

I can't however seem to figure out how to import **PPLM** as a **Tensorflow** model. Any help would be greatly appreciated.

Thank you


[FIX] #2658 Inconsistent values returned by batch_encode_plus and enc…

As ticket describe, when using batch_encode_plus, instead of encode_plus, tokens type and mask are different. They should be the same using batch processing or not. Proposed fix here solve the issue


Inconsistent values returned by batch_encode_plus and encode_plus

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): bert-base-uncased

The problem arise when using:
* batch_encode_plus and encode_plus with pad_to_max_length & max_length

## To Reproduce

Minimal exemple :

I compare tokens created via batch or not, and they are different for **masks** and **types**
```
pretrained = 'bert-base-uncased'
tokenizer = FixedAutoTokenizer.from_pretrained(pretrained)
model = AutoModel.from_pretrained(pretrained)
text = "My"
text1 = "features are ok"

mylist = list()
mylist.append(text)
mylist.append(text1)

#####################################################################################################
batch_encoding = tokenizer.batch_encode_plus(mylist,
                                             return_tensors='pt',
                                             add_special_tokens=False)

######################################################################################################
text_encoding = tokenizer.encode_plus(text,
                                      return_tensors='pt',
                                      add_special_tokens=False,
                                      max_length=3,
                                      pad_to_max_length=True)

print("\n--Batch Encoding \n")
print(batch_encoding['input_ids'])
print(batch_encoding['token_type_ids'])
print(batch_encoding['attention_mask'])
print("\n--One-by-one encoding\n")
print(text_encoding['input_ids'])
print(text_encoding['token_type_ids'])
print(text_encoding['attention_mask'])
```

It gives

```
--Batch Encoding

tensor([[2026,    0,    0],
        [2838, 2024, 7929]])
tensor([[0, 1, 1],
        [0, 0, 0]])
tensor([[1, 1, 1],
        [1, 1, 1]])

--One-by-one encoding

tensor([[2026,    0,    0]])
tensor([[0, 0, 0]])
tensor([[1, 0, 0]])
```

## Expected behavior

It should return the same value, like following

```
--Batch Encoding

tensor([[2026,    0,    0],
        [2838, 2024, 7929]])
tensor([[0, 0, 0],
        [0, 0, 0]])
tensor([[1, 0, 0],
        [1, 1, 1]])

--One-by-one encoding

tensor([[2026,    0,    0]])
tensor([[0, 0, 0]])
tensor([[1, 0, 0]])
```

## Environment

* OS: MacOs Mojave (reproduced also on Ubuntu)
* Python version: 3.7
* PyTorch version: torch==1.3.1
* PyTorch Transformers version (or branch): transformers==2.3.0
* Using GPU ? no


Add `return_special_tokens_mask` to `batch_encode_plus()`

Proposal to add the keyword argument `return_special_tokens_mask` to the method `batch_encode_plus()` to match the functionality of `encode_plus()`. The implementation simply adds the argument in the `encode_plus()` call, so it inherits its implementation and should be compatible with other changes to the `batch_encode_plus()` arguments.


Using Transformers for a Sequence with Multiple Variables at Each Step

## ❓ Questions & Help

I have sequence data which I want to classify and predict future sequence. However, I   know that there are a few additional features which aso affect the subsequent values, each to a different extent. So I have multiple features for each step of input sequence. However, the output sequence can be just one feature for each step.

Input - [(x1, y1, z1), (x2,y2,z2), (x3,y3,z3)]
Output - [x4, x5, x6] or classification (a,b or c)

I do not want to concatenate x,y, and z as they have varying effects on the label and I think concatenating would give an erroneous result.

Need advise on whether Transformers, and specifically Huggingface Transformers can be used for this use case?

Thanks.
<!-- -->


Fix AutoModelForQuestionAnswering for Roberta

When using `AutoModelForQuestionAnswering()` to load a Roberta model, we are currently instantiating a `BertForQuestionAnswering` class. This is happening because `RobertaConfig` is an instance of `BertConfig` (due to inheritance) and there's no other mapping for Roberta in here:

https://github.com/huggingface/transformers/blob/bac51fba3a6b96f02f482e9a352601242b200e47/src/transformers/modeling_auto.py#L176-L184


Add keyword arguments to batch_encode_plus() to match encode_plus()

## 🚀Consistent Keyword arguments for batch_encode_plus() to match encode_plus()
 Currently, features such as `return_special_tokens_mask` that are available for the `encode_plus()` method are not available for `batch_encode_plus()`. It would be nice if all keyword arguments worked in a similar fashion.

## Motivation
`batch_encode_plus()` is extremely useful to seamlessly tokenize batches, however, the lack of features forces to fall back to encode_plus() and an _uglier_ implementation.


Fix token_type_ids for XLM-R


Fix importing unofficial TF models with extra optimizer weights

Hi:)

I was trying to convert the BERT `tf model` to `torch model`, and tf model has *extra optimizer weights* ([This file](https://drive.google.com/file/d/1mNDA-SNCsnu60wzKVe_Y3k-dq3LoDHB2/view) is the one I've tried to convert).

But it encounters the error, and I've printed the parameters' name in tf model.

<img width="686" alt="Screen Shot 2020-01-27 at 10 21 50 PM" src="https://user-images.githubusercontent.com/28896432/73183937-f87e9d00-415e-11ea-9317-2611aecdefe7.png">

I've found out the instead of "adam_v" for "adam_m, names are saved as "AdamWeightDecayOptimizer" or "AdamWeightDecayOptimizer_1".

There was a similar issue that also encountered the issue that I had. ([Issue Link from DeepPavlov repo](https://github.com/deepmipt/DeepPavlov/issues/863))

I can't find out the exact reason why the parameters are named as "AdamWeightDecayOptimizer" or "AdamWeightDecayOptimizer_1" instead of "adam_v" or "adam_m", but it might be safe to cover all the exceptional cases:)


XLNET SQuAD2.0 Fine-Tuning - What May Have Changed?

## ❓ Questions & Help

I fine-tuned XLNet_large_cased on SQuAD 2.0 last November 2019 with Transformers V2.1.1 yielding satisfactory results:
```
xlnet_large_squad2_512_bs48
{
  "exact": 82.07698138633876,
  "f1": 85.898874470488,
  "total": 11873,
  "HasAns_exact": 79.60526315789474,
  "HasAns_f1": 87.26000954590184,
  "HasAns_total": 5928,
  "NoAns_exact": 84.54163162321278,
  "NoAns_f1": 84.54163162321278,
  "NoAns_total": 5945,
  "best_exact": 83.22243746315169,
  "best_exact_thresh": -11.112004280090332,
  "best_f1": 86.88541353813282,
  "best_f1_thresh": -11.112004280090332
}
```
![loss_graph](https://user-images.githubusercontent.com/44321615/73140980-b73d9e80-4033-11ea-9320-d4fa633c9fd7.jpg)

with script:
```
#!/bin/bash

export OMP_NUM_THREADS=6
RUN_SQUAD_DIR=/media/dn/dssd/nlp/transformers/examples
SQUAD_DIR=${RUN_SQUAD_DIR}/scripts/squad2.0
MODEL_PATH=${RUN_SQUAD_DIR}/runs/xlnet_large_squad2_512_bs48

python -m torch.distributed.launch --nproc_per_node=2 ${RUN_SQUAD_DIR}/run_squad.py \
  --model_type xlnet \
  --model_name_or_path xlnet-large-cased \
  --do_train \
  --train_file ${SQUAD_DIR}/train-v2.0.json \
  --predict_file ${SQUAD_DIR}/dev-v2.0.json \
  --version_2_with_negative \
  --num_train_epochs 3 \
  --learning_rate 3e-5 \
  --adam_epsilon 1e-6 \
  --max_seq_length 512 \
  --doc_stride 128 \
  --save_steps 2000 \
  --per_gpu_train_batch_size 1 \
  --gradient_accumulation_steps 24 \
  --output_dir ${MODEL_PATH}

CUDA_VISIBLE_DEVICES=0 python ${RUN_SQUAD_DIR}/run_squad.py \
  --model_type xlnet \
  --model_name_or_path ${MODEL_PATH} \
  --do_eval \
  --train_file ${SQUAD_DIR}/train-v2.0.json \
  --predict_file ${SQUAD_DIR}/dev-v2.0.json \
  --version_2_with_negative \
  --max_seq_length 512 \
  --per_gpu_eval_batch_size 48 \
  --output_dir ${MODEL_PATH}
$@
```
After upgrading Transformers to Version 2.3.0 I decided to see if there would be any improvements in the fine-tuning results using the same script above.  I got the following results:
```
xlnet_large_squad2_512_bs48
Results: {
'exact': 45.32131727448834,
'f1': 45.52929325627209,
'total': 11873,
'HasAns_exact': 0.0,
'HasAns_f1': 0.4165483859174251,
'HasAns_total': 5928,
'NoAns_exact': 90.51303616484441,
'NoAns_f1': 90.51303616484441,
'NoAns_total': 5945,
'best_exact': 50.07159100480081,
'best_exact_thresh': 0.0,
'best_f1': 50.07229287739689,
'best_f1_thresh': 0.0}
```
No learning takes place:
![loss](https://user-images.githubusercontent.com/44321615/73141342-de966a80-4037-11ea-92c7-bdbf2035b51b.jpg)

Looking for potential explanation(s)/source(s) for the loss of performance.  I have searched Transformer releases and issues for anything pertaining to XLNet with no clues.  Are there new fine-tuning hyperparameters I've missed that now need to be assigned, or maybe didn't exist in earlier Transformer versions?  Any PyTorch/Tensorflow later version issues?  I may have to recreate the Nov 2019 environment for a re-run to verify the earlier results, and then incrementally update Transformers, PyTorch, Tensorflow, etc.?

Current system configuration:
OS: Linux Mint 19.3 based on Ubuntu 18.04. 3 LTS and Linux Kernel 5.0
GPU/CPU: 2 x NVIDIA 1080Ti / Intel i7-8700
Seasonic 1300W Prime Gold Power Supply
CyberPower 1500VA/1000W battery backup

Transformers: 2.3.0
PyTorch: 1.3.0
TensorFlow: 2.0.0
Python: 3.7.5