loss function error when running run_lm_finetuning.py file

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): Multilingual model (trying to finetune with Bengali)

The problem arise when using:
* [run_lm_finetuning.py] the official example scripts: I wanted to fine tune the multilingual bert model on Bengali text.

## To Reproduce

Steps to reproduce the behavior:

1. Running the script mentioned using the command I used below and Bengali txt. For now I simply put Bengali wikidump in a text file.

student_1@gpuserver:~/aysha_anis_thesis/thesis$ python3 run_lm_finetuning.py --output_dir=lm_out --model_type=bert --model_name_or_path=bert-base-multilingual-cased --do_train --train_data_file=wikiText.txt --mlm --save_total_limit=2
2020-01-26 22:28:55.372480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-01-26 22:28:55.377886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
/usr/lib/python3/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
01/26/2020 22:28:56 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: False
01/26/2020 22:28:57 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json from cache at /home/student_1/.cache/torch/transformers/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.83b0fa3d7f1ac0e113ad300189a938c6f14d0588a4200f30eef109d0a047c484
01/26/2020 22:28:57 - INFO - transformers.configuration_utils -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 119547
}

01/26/2020 22:28:59 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt from cache at /home/student_1/.cache/torch/transformers/96435fa287fbf7e469185f1062386e05a075cadbf6838b74da22bf64b080bc32.99bcd55fc66f4f3360bc49ba472b940b8dcf223ea6a345deb969d607ca900729
01/26/2020 22:29:00 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-pytorch_model.bin from cache at /home/student_1/.cache/torch/transformers/5b5b80054cd2c95a946a8e0ce0b93f56326dff9fbda6a6c3e02de3c91c918342.7131dcb754361639a7d5526985f880879c9bfd144b65a0bf50590bddb7de9059
01/26/2020 22:29:04 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
01/26/2020 22:29:06 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=510, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=False, do_train=True, eval_all_checkpoints=False, eval_data_file=None, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=50, max_grad_norm=1.0, max_steps=-1, mlm=True, mlm_probability=0.15, model_name_or_path='bert-base-multilingual-cased', model_type='bert', n_gpu=2, no_cuda=False, num_train_epochs=1.0, output_dir='lm_out', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=50, save_total_limit=2, seed=42, server_ip='', server_port='', should_continue=False, tokenizer_name=None, train_data_file='wikiText.txt', warmup_steps=0, weight_decay=0.0)
01/26/2020 22:29:06 - INFO - __main__ -   Loading features from cached file bert_cached_lm_510_wikiText.txt
01/26/2020 22:29:08 - INFO - __main__ -   ***** Running training *****
01/26/2020 22:29:08 - INFO - __main__ -     Num examples = 103717
01/26/2020 22:29:08 - INFO - __main__ -     Num Epochs = 1
01/26/2020 22:29:08 - INFO - __main__ -     Instantaneous batch size per GPU = 4
01/26/2020 22:29:08 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 8
01/26/2020 22:29:08 - INFO - __main__ -     Gradient Accumulation steps = 1
01/26/2020 22:29:08 - INFO - __main__ -     Total optimization steps = 12965
Epoch:   0%|                                                                                | 0/1 [00:00<?, ?it/s/home/student_1/.local/lib/python3.6/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [9,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [10,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [12,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [15,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [21,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [22,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [23,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [25,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [28,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [29,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 785, in <module>
    main()
  File "run_lm_finetuning.py", line 735, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 353, in train
    loss.backward()
  File "/home/student_1/.local/lib/python3.6/site-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/student_1/.local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [10,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [12,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [18,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [21,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [22,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [23,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [24,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [25,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [26,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [28,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [29,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
Epoch:   0%|                                                                                | 0/1 [00:02<?, ?it/s]
Iteration:   0%|                                                                                | 0/12965 [00:02<?, ?it/s]


Using a Model without any pretrained data

## ❓ Questions & Help

<!-- Sorry for a very basic question. Can I use your library without any pertained data? For example, I want to use a BERT transformer model, but using only my corpus of data. In the docs, I only see examples using pretrained models. Thanks. -->


run_lm_finetuning.py for GPT2 throw error "Using pad_token, but it is not set yet."

I used the official setting.
```bash
python transformers/examples/run_lm_finetuning.py \
    --output_dir=gpt2_q_model \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=txt/{q_files[0]} \
```

But it says the padding id was not set.

```python
ERROR - transformers.tokenization_utils -   Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "transformers/examples/run_lm_finetuning.py", line 785, in <module>
    main()
  File "transformers/examples/run_lm_finetuning.py", line 735, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "transformers/examples/run_lm_finetuning.py", line 330, in train
    for step, batch in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.6/dist-packages/tqdm/_tqdm.py", line 979, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "transformers/examples/run_lm_finetuning.py", line 231, in collate
    return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/utils/rnn.py", line 384, in pad_sequence
    out_tensor = sequences[0].data.new(*out_dims).fill_(padding_value)
TypeError: fill_() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor value)
      didn't match because some of the arguments have invalid types: (NoneType)
 * (Number value)
      didn't match because some of the arguments have invalid types: (NoneType)
```


Question Answering with Japanese

## ❓ Questions & Help
Hi @singletongue,

I am trying to use Question-Answering for Japanese, however I could not find any model trained for that.
I tried with the available models but the results were way off (as expected...).
Any suggestions on available models, or other library that already handle QnA with Japanese?

If it is supposed to work as-is, could you share a simple example?

Thank you in advance!


glue.py: AttributeError: 'numpy.str_' object has no attribute 'text_a'

when I am executing the glue data conversion i.e.

`sequences = glue_convert_examples_to_features(X_train, tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
`
I'm getting this error:

> I0126 11:57:07.862119 16252 glue.py:70] Using label list ['0', '1'] for task mrpc
> I0126 11:57:07.863118 16252 glue.py:73] Using output mode classification for task mrpc
> I0126 11:57:07.864120 16252 glue.py:80] Writing example 0
> ---------------------------------------------------------------------------
>
> AttributeError                            Traceback (most recent call last)
>
> <ipython-input-11-621d8071aa9a> in <module>
>       2 #test_sequences = [tokenizer.encode(xxt,add_special_tokens=True) for xxt in X_test]
>       3 #val_sequences = [tokenizer.encode(xxt,add_special_tokens=True) for xxt in X_val]
> ----> 4 sequences = glue_convert_examples_to_features(df['cleantext'], tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
>       5 val_sequences = glue_convert_examples_to_features(X_val, tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
>       6 test_sequences = glue_convert_examples_to_features(X_val, tokenizer, max_length=MAX_SEQUENCE_LENGTH, task='mrpc')
>
> d:\anaconda3\envs\t2\lib\site-packages\transformers\data\processors\glue.py in glue_convert_examples_to_features(examples, tokenizer, max_length, task, label_list, output_mode, pad_on_left, pad_token, pad_token_segment_id, mask_padding_with_zero)
>      84
>      85         inputs = tokenizer.encode_plus(
> ---> 86             example.text_a,
>      87             example.text_b,
>      88             add_special_tokens=True,
>
> AttributeError: 'str' object has no attribute 'text_a'

As tokenizer I'm using
`tokenizer = BertTokenizer.from_pretrained('bert-base-cased')`

and my numpy version is 1.18.1, tensorflow version 2.1.0 (base 2.1.0), transformers version 2.3.0


How to load locally saved tensorflow DistillBERT model

I have got tf model  for DistillBERT by the following python line

> `import tensorflow as tf
from transformers import DistilBertTokenizer, TFDistilBertModel
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = TFDistilBertModel.from_pretrained('distilbert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"), dtype="int32")[None, :]  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]`

> These lines have been executed successfully. But I am facing error with model.save()

> `model.save("DSB/DistilBERT.h5")`
> `model.save("DSB")`
> `model.save("DSB/")`
> all the above 3 line gives errors
![Screenshot (288)](https://user-images.githubusercontent.com/48640299/73131673-852d3c00-4035-11ea-88fc-2641887bb951.png)
![Screenshot (289)](https://user-images.githubusercontent.com/48640299/73131716-46e44c80-4036-11ea-8a12-3ec00b309341.png)

> but downlines works
>`model.save_pretrained("DSB")`
this saves 2 file tf_model.h5 and config.json
>`model.save_weights("DSB/DistDistilBERT_weights.h5")`
this  also have saved the file
![Screenshot (290)](https://user-images.githubusercontent.com/48640299/73131807-b3138000-4037-11ea-91ac-d2f4106e24d4.png)
> but I am not able to re-load this locally saved model any how, I have tried with all down-lines it gives error

> `from tensorflow.keras.models import load_model
from transformers import DistilBertConfig, PretrainedConfig
from transformers import TFPreTrainedModel
config = DistilBertConfig.from_json_file('DSB/config.json') conf2=PretrainedConfig.from_pretrained("DSB")
config=TFPreTrainedModel.from_config("DSB/config.json")`
> all these load configuration , but I am unable to load model , tried with all down-line
> `model=TFPreTrainedModel.from_pretrained("DSB")`

> `model=PreTrainedModel.from_pretrained("DSB/tf_model.h5", from_tf=True, config=config)`

> `model=TFPreTrainedModel.from_pretrained("DSB/")`

> ` model=TFPreTrainedModel.from_pretrained("DSB/tf_model.h5", config=config)`

![Screenshot (291)](https://user-images.githubusercontent.com/48640299/73131818-d3433f00-4037-11ea-910f-9265008b3d77.png)

> NotImplementedError                       Traceback (most recent call last)
<ipython-input-28-7f562f1af321> in <module>()
      1 from transformers import TFPreTrainedModel
----> 2 model=TFPreTrainedModel.from_pretrained("DSB/tf_model.h5", config=config)
      3 #config=TFPreTrainedModel.from_config("DSB/config.json")
      4 #model=TFPreTrainedModel.from_pretrained("DSB/")

2 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    309             return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file, allow_missing_keys=True)
    310
--> 311         ret = model(model.dummy_inputs, training=False)  # build the network with dummy inputs
    312
    313         assert os.path.isfile(resolved_archive_file), "Error retrieving file {}".format(resolved_archive_file)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    820           with base_layer_utils.autocast_context_manager(
    821               self._compute_dtype):
--> 822             outputs = self.call(cast_inputs, *args, **kwargs)
    823           self._handle_activity_regularization(inputs, outputs)
    824           self._set_mask_metadata(inputs, outputs, input_masks)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py in call(self, inputs, training, mask)
    710     """
    711     if not self._is_graph_network:
--> 712       raise NotImplementedError('When subclassing the `Model` class, you should'
    713                                 ' implement a `call` method.')
    714

NotImplementedError: When subclassing the `Model` class, you should implement a `call` method.


XLNet run_squad.py IndexError: tuple index out of range

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English (xlnet-base-cased)

The problem arise when using:
* [x] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: run_squad.py
* [ ] my own task or dataset: (give details)

## To Reproduce

CUDA_VISIBLE_DEVICES=0,1,2,3 python run_squad.py \
  --model_type xlnet \
  --model_name_or_path xlnet-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --version_2_with_negative \
  --train_file /data/medg/misc/phuongpm/squadv2/train-v2.0.json \
  --predict_file /data/medg/misc/phuongpm/squadv2/dev-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 10000 \
  --output_dir /scratch/phuongpm/tuned/squad_xlnet/

## Expected behavior

Epoch:   0%|                                                                        | 0/2 [00:00<?, ?it/sTraceback (most recent call last):                                                | 0/2791 [00:00<?, ?it/s]
  File "run_squad.py", line 837, in <module>
    main()
  File "run_squad.py", line 776, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 221, in train
    inputs.update({"is_impossible": batch[7]})
IndexError: tuple index out of range
Epoch:   0%|                                                                        | 0/2 [00:00<?, ?it/s]
Iteration:   0%|                                                                 | 0/2791 [00:00<?, ?it/s]

## Environment

* OS: Linux
* Python version: 3.6
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch):
commit babd41e7fa07bdd764f8fe91c33469046ab7dbd1
Author: Lysandre <lysandre.debut@reseau.eseo.fr>
Date:   Fri Jan 24 17:06:55 2020 -0500
* Using GPU ? Yes
* Distributed or parallel setup ?
* Any other relevant information:


BERT LOSS FUNCTION

My question is that can I use KLDivLoss instead of CrossEntropyLoss when I fine-tune BERT for classification? the reason for that is that I want to pass the weight of each class(e.g for binary classification, instead of 1 or 0 I will pass the probability distribution )

Thank you in advance


Scrambled dimensions on output of forward pass

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: see attached minimum working example.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: see attached minimum working example.

## To Reproduce

Steps to reproduce the behavior:

1. Run the minimal working example (see below) with the command: `python xl_mwe.py`
2. Observe the following output:
```
Embedded batch: torch.Size([3, 13, 300])
XLNet output  : torch.Size([13, 3, 300])
```
3. Per the documentation, the correct dimensions for the output should have been [3, 13, 300]. From the documentation of `last_hidden_state` in `XLNetModel.forward`: last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)):

4. While constructing the minimal working example, I also observed another bug. If I change d_model to 25 (d_model = 300 in the code below) and n_heads to 5 (default is 10 in the code below), I get an error from einsum:

```
Traceback (most recent call last):
  File "xl_mwe.py", line 43, in <module>
    main()
  File "xl_mwe.py", line 37, in main
    xlnet_output = xlnet(inputs_embeds=embedded_batch)[0]
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\transformers\modeling_xlnet.py", line 858, in forward
    head_mask=head_mask[i])
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\transformers\modeling_xlnet.py", line 436, in forward
    head_mask=head_mask)
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\transformers\modeling_xlnet.py", line 383, in forward
    k_head_r = torch.einsum('ibh,hnd->ibnd', r, self.r)
  File "C:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torch\functional.py", line 202, in einsum
    return torch._C._VariableFunctions.einsum(equation, operands)
RuntimeError: size of dimension does not match previous size, operand 1, dim 0
```

## Minimal working example

```
import numpy as np
import torch
from transformers import XLNetConfig, XLNetModel, XLNetTokenizer

def embed(input_str, dims=25, fix_len=-1):
  result = []
  for word in input_str.split():
    result.append(np.random.rand(dims))
  if fix_len > -1:
    result = result[0: fix_len]
    if len(result) < fix_len:
      result = result + [np.zeros(dims)] * (fix_len - len(result))
  return result

def embed_batch(batch, dims=25, fix_len=-1):
  return np.stack([embed(x, dims, fix_len) for x in batch], axis=0)

def main():
  batch = [
      "Hello, how are you doing?",
      "Please go to the store and buy some bread.",
      "Trump was not exonerated by the Mueller report."
  ]

  d_model = 300
  config = XLNetConfig(d_model=d_model, n_head=10)
  xlnet = XLNetModel(config)

  embedded_batch = embed_batch(batch, dims=d_model, fix_len=13)
  embedded_batch = torch.from_numpy(embedded_batch).float()
  print(f"Embedded batch: {embedded_batch.shape}")

  xlnet_output = xlnet(inputs_embeds=embedded_batch)[0]
  print(f"XLNet output  : {xlnet_output.shape}")

if __name__ == "__main__":
  main()
```

## Environment

* OS: Windows 10
* Python version: 3.7.4
* PyTorch version:  1.2.0
* PyTorch Transformers version (or branch):  2.3.0
* Using GPU ? Yes
* Distributed or parallel setup ? No

## Additional context

<!-- Add any other context about the problem here. -->


```
import tensorflow as tf
from transformers import DistilBertTokenizer, TFDistilBertModel

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = TFDistilBertModel.from_pretrained('distilbert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```

###################
these line of codes gives the error

ImportError: cannot import name 'TFDistilBertModel'

############

![Screenshot (274)](https://user-images.githubusercontent.com/48640299/73121265-6e87d600-3f9e-11ea-8f44-ec8619b42835.png)
![Screenshot (275)](https://user-images.githubusercontent.com/48640299/73121268-78113e00-3f9e-11ea-9ef2-e77a26170178.png)


batch_encode_plus not working for GPT2, OpenAI, TransfoXL when returning PyTorch tensors

## 🐛 Bug

`batch_encode_plus` does not work on GPT2, OpenAI, and TransfoXL when returning PyTorch tensors. Note that the code does work when leaving out the `return_tensors` argument. In that case, the output of `encoded` looks normal.

## To Reproduce
```python
from transformers import *

TOKENIZERS = {
    'albert': (AlbertTokenizer, 'albert-base-v1'),
    'bert': (BertTokenizer, 'bert-base-uncased'),
    'distilbert': (DistilBertTokenizer, 'distilbert-base-uncased'),
    'gpt2': (GPT2Tokenizer, 'gpt2'),
    'openai': (OpenAIGPTTokenizer, 'openai-gpt'),
    'roberta': (RobertaTokenizer, 'roberta-base'),
    'transfoxl': (TransfoXLTokenizer, 'transfo-xl-wt103'),
    'xlm': (XLMTokenizer, 'xlm-mlm-enfr-1024'),
    'xlnet': (XLNetTokenizer, 'xlnet-base-cased')
}

text = ['I like bananas and cookies .',
        'You are not what I thought you were , though .',
        'Cookies are awesome .']

for tok_name, (tok_cls, tok_default) in TOKENIZERS.items():
    tokenizer = tok_cls.from_pretrained(tok_default)
    try:
        encoded = tokenizer.batch_encode_plus(text, return_tensors='pt')
    except Exception as e:
        print(f"{tok_name} failed: {e}")
```

Output on latest master:

```
gpt2 failed: Could not infer dtype of NoneType
openai failed: Could not infer dtype of NoneType
transfoxl failed: Could not infer dtype of NoneType
```

## Environment

* OS: Windows 10
* Python version: 3.7.3
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): latest master
* Using GPU ? yes
* Distributed or parallel setup ? no


## ❓ Questions & Help

<!-- error comes from modeling_xlnet.py file -->

i get this error :
---------------------------------------------------------------------------
```
AttributeError                            Traceback (most recent call last)

<ipython-input-80-01c16e13fe9a> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', "gkf = GroupKFold(n_splits=5).split(X=df_train.question_body, groups=df_train.question_body)\n\nvalid_preds = []\ntest_preds = []\nfor fold, (train_idx, valid_idx) in enumerate(gkf):\n    \n    # will actually only do 2 folds (out of 5) to manage < 2h\n    if fold in [0, 2]:\n\n        train_inputs = [inputs[i][train_idx] for i in range(len(inputs))]\n        train_outputs = outputs[train_idx]\n\n        valid_inputs = [inputs[i][valid_idx] for i in range(len(inputs))]\n        valid_outputs = outputs[valid_idx]\n        \n        K.clear_session()\n        model = create_model()\n        optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)\n        #optimizer = AdamW(lr=1e-4)\n        model.compile(loss=bce_dice_loss, optimizer=optimizer)\n        model.fit(train_inputs, train_outputs, epochs=6, batch_size=6)\n        # model.save_weights(f'bert-{fold}.h5')\n        valid_preds.append(model.predict(valid_inputs))\n        test_preds.append(model.predict(test_inputs))\n        \n        rho_val = compute_spearmanr_ignore_nan(valid_outputs, valid_preds[-1])\n        print('validation score = ', rho_val)\n        model.save_weights(f'/content/drive/My Drive/quest/validation-{rho_val}-fold-{fold}.hdf5')")

5 frames

</usr/local/lib/python3.6/dist-packages/decorator.py:decorator-gen-60> in time(self, line, cell, local_ns)

<timed exec> in <module>()

/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py in forward(self, input_ids, attention_mask, mems, perm_mask, target_mapping, token_type_ids, input_mask, head_mask, inputs_embeds)
    726             raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    727         elif input_ids is not None:
--> 728             input_ids = input_ids.transpose(0, 1).contiguous()
    729             qlen, bsz = input_ids.shape[0], input_ids.shape[1]
    730         elif inputs_embeds is not None:

AttributeError: 'Tensor' object has no attribute 'transpose'
```
when i try xlnet but i don't get error when i try bert

code i am using :

```py
from transformers import XLNetConfig, XLNetModel,XLNetTokenizer
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

def compute_spearmanr_ignore_nan(trues, preds):
    rhos = []
    for tcol, pcol in zip(np.transpose(trues), np.transpose(preds)):
        rhos.append(spearmanr(tcol, pcol).correlation)
    return np.nanmean(rhos)

def create_model():
    q_id = tf.keras.layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
    a_id = tf.keras.layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)

    q_mask = tf.keras.layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
    a_mask = tf.keras.layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)

    q_atn = tf.keras.layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
    a_atn = tf.keras.layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)

    #config = BertConfig() # print(config) to see settings
    config = XLNetConfig()
    config.output_hidden_states = False # Set to True to obtain hidden states
    # caution: when using e.g. XLNet, XLNetConfig() will automatically use xlnet-large config

    # normally ".from_pretrained('bert-base-uncased')", but because of no internet, the
    # pretrained model has been downloaded manually and uploaded to kaggle.
    #bert_model = TFBertModel.from_pretrained(BERT_PATH+'bert-base-uncased-tf_model.h5', config=config)
    #bert_model = TFBertModel.from_pretrained('xlnet-base-cased')
    #bert_model = XLNetModel(config)
    bert_model = XLNetModel.from_pretrained('xlnet-large-cased')

    # if config.output_hidden_states = True, obtain hidden states via bert_model(...)[-1]
    q_embedding = bert_model(q_id, attention_mask=q_mask, token_type_ids=q_atn)[0]
    a_embedding = bert_model(a_id, attention_mask=a_mask, token_type_ids=a_atn)[0]

    q = tf.keras.layers.GlobalAveragePooling1D()(q_embedding)
    a = tf.keras.layers.GlobalAveragePooling1D()(a_embedding)

    x = tf.keras.layers.Concatenate()([q, a])

    x = tf.keras.layers.Dropout(0.2)(x)

    x = tf.keras.layers.Dense(30, activation='sigmoid')(x)

    model = tf.keras.models.Model(inputs=[q_id, q_mask, q_atn, a_id, a_mask, a_atn,], outputs=x)

    return model
```


Get Warning Message: Unable to convert output to tensors format pt

I am running the following code:

```
from transformers.modeling_tf_bert import TFBertForSequenceClassification
pytorch_model = TFBertForSequenceClassification.from_pretrained('./save/')

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = "This research was consistent with his findings."
sentence_1 = "His findings were compatible with this research."
sentence_2 = "His findings were not compatible with this research."

inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')
```
Get Warning Message:

```
WARNING:transformers.tokenization_utils:Unable to convert output to tensors format pt, PyTorch or TensorFlow is not available.
WARNING:transformers.tokenization_utils:Unable to convert output to tensors format pt, PyTorch or TensorFlow is not available.
```

Then, when I run:

```
pred_1 = pytorch_model(inputs_1['input_ids'], token_type_ids=inputs_1['token_type_ids'])[0].argmax().item()
pred_2 = pytorch_model(inputs_2['input_ids'], token_type_ids=inputs_2['token_type_ids'])[0].argmax().item()

print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")
```

I get the error:  `AssertionError: Too many inputs.`


Add AutoModelForPreTraining

Add `AutoModelForPretraining` and `TFAutoModelForPretraining` classes which will load the full model used for pretraining (guarantee we should have all the pre-trained weights).

This class can be used for instance to convert between an original PyTorch and a TF2.0 models while being sure that all the pretrained weights are converted:

```python
# PyTorch => TF 2.0 (save TF 2.0 weights from PT weights)
tf_model = TFAutoModelForPretraining.from_pretrained('my-model', from_pt=True)
tf_model.save_pretrained()

# TF 2.0 => PyTorch (save PT weights from TF 2.0 weights)
pt_model = AutoModelForPretraining.from_pretrained('my-model', from_tf=True)
pt_model.save_pretrained()
```


Gradient checkpointing with GPT2DoubleHeadsModel

## ❓ Questions & Help

I've been trying to fine-tune `GPT2DoubleHeadsModel` using `gpt2-large` and `gpt2-xl` on the [Topical-Chat](https://github.com/alexa/alexa-prize-topical-chat-dataset) dataset.

I'm finding that loading even a single example into memory is difficult with the larger versions of GPT-2. I found [this](https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255) Medium post by @thomwolf which suggests that gradient checkpointing would be effective at handling this situation.

Is there a gradient-checkpointed version of the code in `GPT2DoubleHeadsModel` or the underlying `GPT2Model` that could be used as-is? I'm trying to do this myself by editing `modeling_gpt2.py`, but I'm facing issues.

https://github.com/huggingface/transformers/blob/babd41e7fa07bdd764f8fe91c33469046ab7dbd1/src/transformers/modeling_gpt2.py#L478-L480

Specifically, I added a checkpoint in the above line like this:

`outputs = checkpoint(block, hidden_states, layer_past, attention_mask, head_mask[i])`

NOTE: I had to remove the key names since it looks like checkpoint does not support key-value arguments, only positional. This might lead to compatibility issues, I'd love to know thoughts on this as well.

This is using the official PyTorch [checkpoint](https://pytorch.org/docs/stable/checkpoint.html). I'm also considering trying [this](https://github.com/csrhddlam/pytorch-checkpoint/blob/master/checkpoint.py) other implementation for checkpoint since I read somewhere that it is supposed to be faster than the official implementation.

With the official PyTorch implementation, I'm getting the following error:
`CheckpointFunctionBackward.forward: expected Variable (got list) for return value 0.`

[This](https://discuss.pytorch.org/t/checkpoint-didnt-support-list-output/16957/3) thread on the PyTorch forums seems to suggest that this error arises when attempting to use `torch.utils.checkpoint` with modules that return a variable number of tensors, which is the case with `Block` within `GPT2Model`.

Could @thomwolf, @LysandreJik or anyone else in the Hugging Face team please help with this? Thanks!


Improving generation

Fix #2554

TODO:
- add tests on generation

TODO potential:
- this PR could be used to fix #2415 and fix #2482 as well
- add TF 2.0 support for generation


AutoModels Documentation


Details on T5's current integration status

Hi all,

Regarding Google's T5 model, here is a quick summary of the status:
* the core model is in the library and some people have started to use it, but:
  - while the operations are identical or very similar (einsum vs. matmul), there is quite a significantly higher relative error between this model's PT hidden-state and the mesh-tensorflow hidden-state (in particular compared to our previous TF => PT model conversions).
  - our guess is that this comes for a combination of bfloat16 vs. fp32, einsum+model parallelism vs. matmul, plus the fact that we are not masking the hidden-states at each layer as the original implementation do (this should not matter much though).
  - as a consequence, we are waiting to be able to confirm it's performances on a GLUE fine-tuning before having a wider communication on its addition to the library.
* the full integration with GLUE tests requires a few features that we still need to add:
  - a decoding mechanism,
  - a pre/post-processing for GLUE to use it in text-to-text setting, and
  - a model parallelism feature
^^ we plan to work on these in February (from the more general view of having better encoder-decoder support in the library).

cc @julien-c @LysandreJik @sshleifer @patrickvonplaten


Add FlauBERT: Unsupervised Language Model Pre-training for French

This PR adds [FlauBERT](https://github.com/getalp/Flaubert). Most of the code is derived from XLM (there are some new features in FlauBERT such as `pre_norm` and `layerdrop`).

`make test` had 1 failure related to BERT and not to FlauBERT:

> [gw0] FAILED tests/test_configuration_auto.py::AutoConfigTest::test_pattern_matching_fallback

`make style` passed.

`make quality` passed.


CamembertTokenizer cannot be pickled

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):  Camembert

Language I am using the model on (English, Chinese....): French

The problem arise when using my own modified scripts:
  I have a nn.Module and, within this module, I store the tokenizers
  I can normally save these tokenizers easily, but CamemberTokenizer
  gives me a  **TypeError: can't pickle SwigPyObject objects**

The tasks I am working on consists in creating a model and saving it using torch.save()

## To Reproduce
```
import torch
from transformers import  CamembertTokenizer, BertTokenizer

class MyModelCamembert(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.cheese =  CamembertTokenizer.from_pretrained('camembert-base')

    def forward(self, x):
        return 1

class MyModelBert(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.cheese = BertTokenizer.from_pretrained('bert-base')

    def forward(self, x):
        return 1

# with bert it works
no_cheese = MyModelBert()
torch.save(no_cheese, "~/bert.pkl")

# with camembert it doesn't
cheese = MyModelCamembert()
torch.save(cheese, "~/camembert.pkl")
```

Steps to reproduce the behavior:

1. Try to save a module containing a tokenizer using the torch.save()
2. Find out it works for Bert but not for Camembert

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
Stack Trace when saving CamembertTokenizer

Traceback (most recent call last):
  File "/home/swqh0332/Desktop/blablapy.py", line 28, in <module>
    torch.save(cheese, "/home/swqh0332/camembert.pkl")
  File "/home/swqh0332/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 260, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/home/swqh0332/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 185, in _with_file_like
    return body(f)
  File "/home/swqh0332/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 260, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/home/swqh0332/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 332, in _save
    pickler.dump(obj)
TypeError: can't pickle SwigPyObject objects

## Expected behavior

I would like to be able to pickle the CamembertTokenizer as I it is possible with the other models

## Environment

* OS: Ubuntu 18
* Python version: 3.6.9
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.2
* Using GPU ? No
* Distributed or parallel setup ? No


Pad token for GPT2 and OpenAIGPT models

## ❓ Questions & Help

I noticed that out of all the models `pad_token` is not set for only `OpenAIGPTModel` and `GPT-2Model`.
I get a warning: `Using pad_token, but it is not set yet.` and `pad_token_id` is `None`

Is there any specific reason why is that so?

If not, what is the appropriate padding token to be used for these models?

Thanks


Question about Architecture of BERT for QA

## ❓ Questions & Help

I have a question about the architecture of Bert for QA.

In Bert forward function

``` python
class BertForQuestionAnswering(BertPreTrainedModel):
    def __init__(self, config):
        super(BertForQuestionAnswering, self).__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

        self.init_weights()

    @add_start_docstrings_to_callable(BERT_INPUTS_DOCSTRING)
    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        start_positions=None,
        end_positions=None,
    ):

        outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
        )

        sequence_output = outputs[0]

        logits = self.qa_outputs(sequence_output) # The line I don't understand
        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        outputs = (start_logits, end_logits,) + outputs[2:]
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(start_positions.size()) > 1:
                start_positions = start_positions.squeeze(-1)
            if len(end_positions.size()) > 1:
                end_positions = end_positions.squeeze(-1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            ignored_index = start_logits.size(1)
            start_positions.clamp_(0, ignored_index)
            end_positions.clamp_(0, ignored_index)

            loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
            start_loss = loss_fct(start_logits, start_positions)
            end_loss = loss_fct(end_logits, end_positions)
            total_loss = (start_loss + end_loss) / 2
            outputs = (total_loss,) + outputs

        return outputs  # (loss), start_logits, end_logits, (hidden_states), (attentions)

```

I think logits are from linear layer (this is from bert output)

And start_loss and end_loss is calculated by the logits ( just splited by 2)

But, I read BERT article, But It describes

![image](https://user-images.githubusercontent.com/18682286/73066556-abde5c00-3ee9-11ea-90d2-0ff6ed976048.png)

It looks like the model have to use only spans of the paragraph in last layer.

But, I can't get it how the model can know where's start/end span is?

So can you explain it?

It will be really helpful to me if you answer it.


Albert on QQP inference

While using Albert model trained on QQP data, i am using following code for inference.
How to manage two sentences and two labels (0,1) like QQP?

ffrom transformers import AlbertTokenizer, AlbertForSequenceClassification
import torch

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForSequenceClassification.from_pretrained('albert-base-v2')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, logits = outputs[:2]


Why does the hidden state of the same input token change every time I call the same GPT2 model?

Hello,

Say I fixed my input to the GPT2 model:

```python
input_ids = test_i[:,0]
input_ids = torch.tensor(input_ids.tolist()).unsqueeze(0)
```

Then I try to retrieve the hidden state vector of the last token:
```python
tst_hidden_states = best_model(input_ids)[3][1][0, (test_i.size()[0] - 1), :].detach()
tst_hidden_states[0:5]
>>>tensor([-0.0146,  0.0718, -0.0297, -0.0000, -0.0315])
```

but when I repeat the above process with the exactly same input, the hidden state of the last token keeps changing:

```python
tst_hidden_states = best_model(input_ids)[3][1][0, (test_i.size()[0] - 1), :].detach()
tst_hidden_states[0:5]
>>> tensor([-0.0146,  0.0000, -0.0297, -0.0212, -0.0315])
```
Given that I didn't change the model, I don't understand why the hidden state of the same input and the same token keeps changing at each turn. How can I prevent the hidden state from changing?

Thank you,


BertModel output the same embedding during Evaluation

## ❓ Questions & Help

During evaluation, my text model outputs the same embedding regardless of the token id. The following is my model.

```
class BertTextEncoderFactory(nn.Module):
    def __init__(self, embedding_dim = 256, model_name_or_path = None, backbone ='bert'):
        super(BertTextEncoderFactory, self).__init__()

        if (backbone == 'bert'):
            self.encoder = BertForRetrival(embedding_dim, 'bert-base-uncased')
```
```
class BertForRetrival(nn.Module):
    def __init__(self, single_embedding_dim = 256, model_name_or_path = 'bert-base-uncased'):
        super(BertForRetrival, self).__init__()

        self.config = BertConfig.from_pretrained(model_name_or_path)
        self.bert = BertModel.from_pretrained(model_name_or_path, config=self.config)

        self.single_embedding_dim = single_embedding_dim
        self.dropout = nn.Dropout(self.config.hidden_dropout_prob)

        self.embedding_layer = nn.Sequential(nn.Linear(self.config.hidden_size, self.config.hidden_size),
                                            nn.LeakyReLU(),
                                            self.dropout,
                                            nn.Linear(self.config.hidden_size, int(self.config.hidden_size / 2)),
                                            nn.LeakyReLU(),
                                            self.dropout,
                                            nn.Linear(int(self.config.hidden_size / 2), self.single_embedding_dim),
                                            nn.ReLU())

        self.init_weights(self.embedding_layer)

    def init_weights(self, module):
    	for m in module.modules():
            if type(m) == nn.Linear:
        	    torch.nn.init.xavier_uniform_(m.weight)
        	    m.bias.data.fill_(0.001)

    def forward(self, input_ids, attention_mask=None, token_type_ids=None):
        outputs = self.bert(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
        print(len(outputs))
        print(outputs[0].size())
        print(outputs)

        first_token_tensor = outputs[0][:,0]
        pooled_output = outputs[1]
        pooled_output = self.dropout(pooled_output)
        single_embedding = self.embedding_layer(pooled_output)

        return single_embedding
```

Model initialization
`text_model = BertTextEncoderFactory(embedding_dim = 256, model_name_or_path = None, backbone = 'bert') `

After I finetuned the model, I save it to a checkpoint.
`torch.save(text_model, './pretrainedcheckpoint/checkpoint.pth.tar')`

During evaluation, I  first initialize the model in the following way
`text_model = BertTextEncoderFactory(embedding_dim = 256, model_name_or_path = None, backbone = 'bert') `

The output of sample ids is the following. So far so good.
```
text_model.eval()
ids = torch.tensor([[101, 14378, 102]], dtype=torch.long)
text_model(ids, None, None)
```

Ouput is the following
```
2
torch.Size([1, 3, 768])
tensor([[[-0.6077,  0.1454, -0.1540,  ...,  0.0763,  0.5157,  0.4968],
         [ 0.7466, -0.3633, -0.0637,  ...,  0.0403,  0.5987,  0.2889],
         [ 0.9683,  0.0883, -0.3452,  ...,  0.2865, -0.6153, -0.1851]]],
       grad_fn=<NativeLayerNormBackward>)
```

Then, I load the model with trained weights.
```
import os
resume = './pretrainedcheckpoint/checkpoint.pth.tar'
if os.path.isfile(resume):
    checkpoint = torch.load(resume)
text_model.load_state_dict(checkpoint)
```
`<All keys matched successfully>`

Now, given the same token id sequences, the model output three exactly same embedding.
```
text_model.eval()
ids = torch.tensor([[101, 14378, 102]], dtype=torch.long)
text_model(ids, None, None)
```
```
2
torch.Size([1, 3, 768])
tensor([[[-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618]]],
       grad_fn=<NativeLayerNormBackward>)
```

As you can see, the embedding for the three tokens are all the same !!!!
```
tensor([[[-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618],
         [-0.3972,  0.2239, -0.3335,  ..., -0.4338,  0.4992, -0.0618]]],
```


Pipeline error when creating a model without a model card json file (on Windows)

Core: Pipeline

Internals of the library; Pipeline.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
Bert
Language I am using the model on (English, Chinese....):
English
The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)
```python
import sys
from transformers import pipeline

if __name__ == '__main__':
    print("start")
    nlp_ft = pipeline('question-answering', model=r'C:\Users\a652726\PycharmProjects\src\data\raw\qa\wwm-bert-uncased-finetuned-squad',
                      tokenizer='bert-large-uncased')
```

Results in the ValueError: no modelcard.json file (which i do not have)
My  fix (hack):
In modelcard.py replace (line 164):

```python
        except EnvironmentError:
            if pretrained_model_name_or_path in ALL_PRETRAINED_CONFIG_ARCHIVE_MAP:
```
with
```python
        except (EnvironmentError, ValueError):
            if pretrained_model_name_or_path in ALL_PRETRAINED_CONFIG_ARCHIVE_MAP:
```
This results in:

```python
            logger.warning("Creating an empty model card.")
```
And everything works fine after this.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details)
 Changing QA pipeline to work on a fixed set of spans (like multiple choice QA task, or classification)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:  Windows
* Python version: 3.7
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): master (post 2.3)
* Using GPU ? no
* Distributed or parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


How to merge TFDistilBertForSequenceClassification with another tf.Keras model

## ❓ Questions & Help

In TensorFlow 2, what is the recommended way to merge `TFDistilBertForSequenceClassification` (or any other Transformer model) with another `tf.keras` model?

In other words, I'd like to do something like this:

```
merged_out = keras.layers.concatenate([other_model.output, distilbert_model.output])
merged_out = layers.Dense(1)(merged_out)
combined_model = keras.Model([other_model.input] + distilbert_model.input, merged_out)
```
The above produces an error because `distilbert_model.output` is not accessible in the same way as vanilla tf.Keras models:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-48-b15ccf9c0221> in <module>()
----> 1 merged_out = keras.layers.concatenate([other_model.output, distilbert_model.output])

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in output(self)
   1574     """
   1575     if not self._inbound_nodes:
-> 1576       raise AttributeError('Layer ' + self.name + ' has no inbound nodes.')
   1577     return self._get_node_attribute_at_index(0, 'output_tensors', 'output')
   1578

AttributeError: Layer tf_distil_bert_for_sequence_classification has no inbound nodes.
```


QA pipeline run-time error when there is no answer

Internals of the library; Pipeline.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
Bert large uncased SQUAD /finetuned on SQUAD2.0 and my dataset
Language I am using the model on (English, Chinese....):
English
The problem arise when using:
* [x ] the official example scripts: (give details)
```python
from transformers import pipeline
nlp_ft = pipeline('question-answering',  model='/data/bert/divorce_qa/wwm-bert-uncased-finetuned-squad', tokenizer='bert-large-uncased')
            nlp_ft({
    'question': "is it raining?",
    'context': ''
})
```
Converting examples to features: 100%|██████████| 1/1 [00:00<00:00, 2486.25it/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-c7eb59b211f0> in <module>
      1 nlp_ft({
      2     'question': "is it raining?",
----> 3     'context': ''
      4 })

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    657                     # Retrieve the score for the context tokens only (removing question tokens)
    658                     fw_args = {k: torch.tensor(v) for (k, v) in fw_args.items()}
--> 659                     start, end = self.model(**fw_args)
    660                     start, end = start.cpu().numpy(), end.cpu().numpy()
    661

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, start_positions, end_positions)
   1265                             position_ids=position_ids,
   1266                             head_mask=head_mask,
-> 1267                             inputs_embeds=inputs_embeds)
   1268
   1269         sequence_output = outputs[0]

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    687                 extended_attention_mask = attention_mask[:, None, None, :]
    688         else:
--> 689             raise ValueError("Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format(input_shape, attention_mask.shape))
    690
    691         # Since attention_mask is 1.0 for positions we want to attend and 0.0 for

ValueError: Wrong shape for input_ids (shape torch.Size([0])) or attention_mask (shape torch.Size([0]))

Another case (with topk):
```python
            nlp_ft({
    'question': "met with client  to discuss her house.",
    'context': 'snow'
}, topk=5)
```
Converting examples to features: 100%|██████████| 1/1 [00:00<00:00, 998.17it/s]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-21-bdcc590f57c7> in <module>
      2     'question': "met with client  to discuss her house.",
      3     'context': 'snow'
----> 4 }, topk=5)

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    684                     'answer': ' '.join(example.doc_tokens[feature.token_to_orig_map[s]:feature.token_to_orig_map[e] + 1])
    685                 }
--> 686                 for s, e, score in zip(starts, ends, scores)
    687             ]
    688         if len(answers) == 1:

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/pipelines.py in <listcomp>(.0)
    684                     'answer': ' '.join(example.doc_tokens[feature.token_to_orig_map[s]:feature.token_to_orig_map[e] + 1])
    685                 }
--> 686                 for s, e, score in zip(starts, ends, scores)
    687             ]
    688         if len(answers) == 1:

KeyError: 255

Is SQUAD2.0 no answer (the span is the CLS token) implemented?

A more difficult problem:
The same span 0:42 appears in the answers twice  with different probabilities:
```python
            nlp_ft({
    'question': 'not divorcing?',
    'context': 'My son loves ice cream. I hired a lawyer.  The weather is beautiful'
}, topk=20)
```
Converting examples to features: 100%|██████████| 1/1 [00:00<00:00, 608.13it/s]
[{'score': 6.8867944232487335e-15,
  'start': 24,
  'end': 42,
  'answer': 'I hired a lawyer.'},
 {'score': 5.837555144012473e-15,
  'start': 0,
  'end': 42,
  'answer': 'My son loves ice cream. I hired a lawyer.'},
 {'score': 5.104124911876544e-15,
  'start': 26,
  'end': 42,
  'answer': 'hired a lawyer.'},
 {'score': 4.480651773242456e-15,
  'start': 0,
  'end': 23,
  'answer': 'My son loves ice cream.'},
 {'score': 3.9003458779199145e-15,
  'start': 34,
  'end': 42,
  'answer': 'lawyer.'},
 {'score': 3.683094227838328e-15,
  'start': 24,
  'end': 66,
  'answer': 'I hired a lawyer. The weather is beautiful'},
 {'score': 3.637585671341453e-15, 'start': 0, 'end': 6, 'answer': 'My son'},
 {'score': 3.4290555803043746e-15,
  'start': 3,
  'end': 42,
  'answer': 'son loves ice cream. I hired a lawyer.'},
 {'score': 3.1219554896279132e-15,
  'start': 0,
  'end': 66,
  'answer': 'My son loves ice cream. I hired a lawyer. The weather is beautiful'},
 {'score': 2.963547353528702e-15,
  'start': 32,
  'end': 42,
  'answer': 'a lawyer.'},
 {'score': 2.8398875189274633e-15,
  'start': 43,
  'end': 66,
  'answer': 'The weather is beautiful'},
 {'score': 2.7297131068172943e-15,
  'start': 26,
  'end': 66,
  'answer': 'hired a lawyer. The weather is beautiful'},
 {'score': 2.631992946944041e-15,
  'start': 3,
  'end': 23,
  'answer': 'son loves ice cream.'},
 {'score': 2.4540475046027306e-15,
  'start': 17,
  'end': 42,
  'answer': 'cream. I hired a lawyer.'},
 {'score': 2.3835331682965698e-15,
  'start': 24,
  'end': 42,
  'answer': 'I hired a lawyer.'},
 {'score': 2.351548698805639e-15,
  'start': 7,
  'end': 42,
  'answer': 'loves ice cream. I hired a lawyer.'},
 {'score': 2.136764987640859e-15, 'start': 3, 'end': 6, 'answer': 'son'},
 {'score': 2.0859256871447656e-15,
  'start': 34,
  'end': 66,
  'answer': 'lawyer. The weather is beautiful'},
 {'score': 2.0203893789166256e-15,
  'start': 0,
  'end': 42,
  'answer': 'My son loves ice cream. I hired a lawyer.'},
 {'score': 1.9864633437892544e-15, 'start': 24, 'end': 25, 'answer': 'I'}]

* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:  Ubuntu 18.04.3 LTS
* Python version: 3.7.4
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): master 2.3 (or later)
* Using GPU ? no
* Distributed or parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


tokenizer.add_tokens not working

## 🐛 Bug

<!-- Important information -->
I tried to add new tokens in vocabulary using tokenizer.add_tokens() and then called model() according to the code given in `BertForMaskedLM` class definition. The code is given below:
```

from transformers import BertForMaskedLM, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')

tokenizer.add_tokens(['[SPECIAL_TOKEN_1]', '[SPECIAL_TOKEN_2]'])
model.resize_token_embeddings(len(tokenizer))

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
```

I get the following error:
`RuntimeError: The size of tensor a (30524) must match the size of tensor b (30522) at non-singleton dimension 2`


Documentation markup for model descriptions

## 🐛 Bug

Looking at [the documentation](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification), it seems something went wrong in markup land. In some models (but not all, e.g. BertModel, BertForMaskedLM, BertForNextSentencePrediction), the model description (i.e. the first paragraph) is split up in one highlighted line (grey background) and the rest of the text is regular. Seems like a formatting issue in the source code.


Document which heads are pretrained and which aren't

## 🚀 Feature

I was going through the documentation and I realised I never thought about the different heads in much detail (I always start from the base model and built on top of that). Now that I did, I wonder whether users (mistakenly?) assume that models such as `BertForQuestionAnswering` have a pretrained head. I am assuming that these heads are _not_ pretrained but that it is a convenience to have an architecture that can be finetuned on downstream tasks. If I am correct, it might be useful to highlight for these kind of models that the heads are not pretrained.

That being said, when I run

```python
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
model.eval()
torch.set_grad_enabled(False)

for name, parameters in model.named_parameters():
    if 'classifier' in name:
        print(name)
        print(parameters)
```

I get the notice that "Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']", but still:

```
classifier.weight
Parameter containing:
tensor([[-0.0098,  0.0137,  0.0275,  ..., -0.0221, -0.0190,  0.0156],
        [ 0.0144,  0.0016,  0.0084,  ...,  0.0055,  0.0221, -0.0145]],
       requires_grad=True)
````

I've been peaking into the rabbit hole, but I can't seem to find where this random initialisation occurs.

If you agree, I can put some time in putting that in the documentation but I might need help or at least a review. Perhaps this can even be automated? That would be awesome.


Adding scibert in the list of pre-trained models?

# 🌟New model addition

## Model description

Would it be possible/is it in the pipeline to add SCIBERT as one of the pre-trained models for Bert? Could be as simple as adding it to the `BERT_PRETRAINED_MODEL_ARCHIVE_MAP`.

## Open Source status

Scibert is available on its own repository (https://github.com/allenai/scibert). The advantage of adding it to the map is that we don't need to download it ad-hoc every time we want to use it, it would be cached in the same repository, etc.

* [x] the model implementation is available:
* [x] the model weights are available:
* [x] who are the authors: Iz Beltagy and Kyle Lo and Arman Cohan authored the ALlenAI

## Additional context

<!-- Add any other context about the problem here. -->


summarization codes

Hi
I greatly appreciate to add also possibilities to train the summarization codes from scratch. I see only evaluation part in the codes. Does this also work for training?
thanks a lot for your response.
Kind regards
Rabeeh


TF Models have no attribute .train() or .eval()

## 🐛 Bug

Using any of the TF models I am unable to set the **.eval()** or **.train()** properties. In addition, when loading from a pre-trained path (which the documentation seems to imply would mean that the models will be set to eval mode) I see non deterministic outputs given the same input indicating the models do not have dropout turned off.

Basic example:

```
model = TFBert.from_pretrained('path_to_model_directory', from_pt=True)
model.eval() #### This errors with "TFBertModel object has no attribute 'eval'"
tokenizer = BertTokenizer.from_pretrained('path_to_model_directory')
inputs = tokenizer.encode_plus('Dummy text here.', return_tensors='tf')['input_ids']

print(model(inputs)) ## these outputs
print(model(inputs)) ## **will not** be the same

print(model(inputs, training=False)) ## these outputs
print(model(inputs, training=False)) ## **will** be the same
```

Any help would be greatly appreciated!

## Environment

* OS: Windows
* Python version: 3.6
* PyTorch version: 1.4
* PyTorch Transformers version (or branch): 2.3
* Using GPU ? Yes


Adaptive Attention Span for Transformers

# 🌟New model addition

## Model description

<!-- Important information -->

## Open Source status

* [x] the model implementation is available: https://github.com/facebookresearch/adaptive-span
* [x] the model weights are available: get_pretrained.sh
* [x] who are the authors: Facebook Research

## Additional context

No additional dependencies required.


Question answering pipeline fails with long context

Internals of the library; Pipeline.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Question Answering Pipeline / Distilbert

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [x] the official example scripts: (give details): Based on the sample pipeline code form here: https://github.com/huggingface/transformers#quick-tour-of-pipelines
* [] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce
I think I found a bug in the pipeline code. It fails when there's a long context in a list. See below:

```
from transformers import pipeline
nlp = pipeline('question-answering')

long_str = 'These are some irrelevant words. ' * 100
long_str = 'Pipeline have been included in the huggingface/transformers repository. ' + long_str

#Works
nlp(
    {
    'question': 'What is the name of the repository ?',
    'context': 'Pipeline have been included in the huggingface/transformers repository. '
    },
    {
    'question': 'What is the name of the repository ?',
    'context': 'Pipeline have been included in the huggingface/transformers repository. '
    }
)

#Long context by itself - works
nlp(
    {
    'question': 'What is the name of the repository ?',
    'context': long_str
    })

#Long context in a list - fails
nlp(
    {
    'question': 'What is the name of the repository ?',
    'context': long_str
    },
    {
    'question': 'What is the name of the repository ?',
    'context': 'Pipeline have been included in the huggingface/transformers repository. '
    }
)
```

Here's the error message:
```
Converting examples to features: 100%|██████████| 2/2 [00:00<00:00, 87.19it/s]
Traceback (most recent call last):

  File "<ipython-input-3-e795fc7f26bf>", line 8, in <module>
    'context': 'Pipeline have been included in the huggingface/transformers repository. '

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\pipelines.py", line 686, in __call__
    for s, e, score in zip(starts, ends, scores)

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\pipelines.py", line 686, in <listcomp>
    for s, e, score in zip(starts, ends, scores)

IndexError: index 0 is out of bounds for axis 0 with size 0
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

Would like to get the answer for the second example.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows
* Python version:  Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
* PyTorch version: N/A
* PyTorch Transformers version (or branch): master
* Using GPU ?
* Distributed or parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Missing module "startlette" when calling transformers-cli

Internals of the library; Pipeline.

Calling `transformers-cli` in terminal returns error ``ModuleNotFoundError: No module named 'starlette'``.

I assume starlette should be added into dependencies in setup.py and it will be a quick fix.


XLnet memory usage for long sequences

## ❓ Questions & Help

Hello,
I have some questions regarding how the XLnet memory and output work in this implementation.

1. As it's been mentioned before, by default, XLnet doesn't use memory. So, how is this possible that it accepts long sequences as input (in other words, why there isn't any limit on the number of input tokens), unlike BERT, for example, that will only accept 512 tokens.

2. If I set the memory length to 512, and feed the XLnet with 512 tokens at a time (for a 1024 sequence length), also pass the memory in each step. (like the example code below) Will the final output of the network, includes all the information from the whole sequence?

```
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
model = XLNetModel.from_pretrained('xlnet-large-cased')
mems = None
for i in range(2):
    input_ids = torch.tensor(tokenizer.encode(text[i])).unsqueeze(0)
    outputs = model(input_ids, mems=mems)
    mems = outputs[1]
```

To be clear, I want to use XLnet for long text summarization. So I need to feed the XLnet output to a decoder part and need a fixed-length representation.


Error in fine tuning Roberta for QA

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,
I tried to fine tune RobertaModel for question answering task, i implemented TFRobertaForQuestionAnswering but when i run the training script i got this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,11] = 1 is not in [0, 1) [Op:ResourceGather] name: tf_roberta_for_question_answering/tf_roberta_model/roberta/embeddings/token_type_embeddings/embedding_lookup/

Here is my class TFRobertaForQuestionAnswering
```python
from transformers import TFRobertaPreTrainedModel, RobertaConfig, TFRobertaModel
import tensorflow as tf

class TFRobertaForQuestionAnswering(TFRobertaPreTrainedModel):

    config_class = RobertaConfig
    #pretrained_model_archive_map = ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
    base_model_prefix = "roberta"

    def __init__(self, config, *inputs, **kwargs):
        config.vocab_size = config.vocab_size + 2
        super().__init__(config, *inputs, **kwargs)

        def get_initializer(initializer_range=0.02):
            """Creates a `tf.initializers.truncated_normal` with the given range.
            Args:
                initializer_range: float, initializer range for stddev.
            Returns:
                TruncatedNormal initializer with stddev = `initializer_range`.
            """
            return tf.keras.initializers.TruncatedNormal(stddev=initializer_range)

        self.num_labels = config.num_labels
        self.roberta = TFRobertaModel(config)
        self.qa_outputs = tf.keras.layers.Dense(
            config.num_labels, kernel_initializer=get_initializer(config.initializer_range), name="qa_outputs")

    def call(
        self,
        input_ids,
        start_positions=None,
        end_positions=None,
        ** kwargs
    ):

        outputs = self.roberta(
            input_ids,
            ** kwargs
        )

        sequence_output = outputs[0]

        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = tf.split(logits, 2, axis=-1)
        start_logits = tf.squeeze(start_logits, -1)
        end_logits = tf.squeeze(end_logits, -1)

        outputs = (start_logits, end_logits,) + outputs[2:]
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(tf.size(start_positions)) > 1:
                start_positions = tf.squeeze(start_positions, -1)
            if len(tf.size(end_positions)) > 1:
                end_positions = tf.squeeze(end_positions, -1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            #ignored_index = tf.size(start_logits, 1)
            #with tf.Session() as sess:
            #    scalar = ignored_index.eval()
            #tf.clip_by_value(start_positions, 0, ignored_index)
            #tf.clip_by_value(end_positions, 0, ignored_index)

            loss_fct = tf.keras.losses.SparseCategoricalCrossentropy(from_logits = True)
            start_loss = loss_fct(start_logits, start_positions)
            end_loss = loss_fct(end_logits, end_positions)
            total_loss = (start_loss + end_loss) / 2
            outputs = (total_loss,) + outputs

        return outputs  # (loss), start_logits, end_logits, (hidden_states), (attentions)
```

and here is my training script:

```python
import tensorflow as tf
from transformers import squad_convert_examples_to_features, SquadV2Processor, RobertaTokenizer
from modelRoberta import TFRobertaForQuestionAnswering
from pathlib import Path
import six
import numpy as np

MAX_QUERY_LENGTH = 64
MAX_SEQ_LENGTH = 384
MAX_DOC_STRIDE = 128
MAX_ANSWER_LENGTH = 64
N_TOK_FOR_CONTEXT = 20

def get_shape_list(tensor):
  shape = tensor.shape.as_list()

  non_static_indexes = []
  for (index, dim) in enumerate(shape):
    if dim is None:
      non_static_indexes.append(index)

  if not non_static_indexes:
    return shape

  dyn_shape = tf.shape(tensor)
  for index in non_static_indexes:
    shape[index] = dyn_shape[index]
  return shape

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
model = TFRobertaForQuestionAnswering.from_pretrained("roberta-base")

squad = SquadV2Processor()

train_examples = squad.get_train_examples(data_dir=Path(__file__).parent, filename="train.json")
test_examples = squad.get_dev_examples(data_dir=Path(__file__).parent, filename="test.json")

train_dataset = squad_convert_examples_to_features(train_examples[:1], tokenizer=tokenizer, max_seq_length=MAX_SEQ_LENGTH,
                                                   doc_stride=MAX_DOC_STRIDE, max_query_length=MAX_QUERY_LENGTH,
                                                   is_training=True, return_dataset='tf')
test_dataset = squad_convert_examples_to_features(test_examples[:1], tokenizer=tokenizer, max_seq_length=MAX_SEQ_LENGTH,
                                                   doc_stride=MAX_DOC_STRIDE, max_query_length=MAX_QUERY_LENGTH,
                                                   is_training=False, return_dataset='tf')

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')

batch_size = 1
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)
epochs = 3
for epoch in range(epochs):
  print('Start of epoch %d' % (epoch,))

  for step, batch in enumerate(train_dataset):

    with tf.GradientTape() as tape:

          inputs_batch = {
              "inputs_ids": batch[0]['input_ids'],
              "token_type_ids": batch[0]['token_type_ids'],
          }

          start_position = batch[1]['start_position']
          end_position = batch[1]['end_position']
          seq_length = get_shape_list(inputs_batch['inputs_ids'])[1]
          start_positions = tf.one_hot(start_position, on_value=1.0, off_value=0.0, depth=seq_length, dtype=tf.float32)
          end_positions = tf.one_hot(end_position,on_value=1.0, off_value=0.0, depth=seq_length, dtype=tf.float32)

          outputs = model(inputs_batch['inputs_ids'], token_type_ids= inputs_batch['token_type_ids'], training=True)  # Logits for this minibatch

          start_logits, end_logits = outputs[:2]
          start_logits = tf.nn.log_softmax(start_logits, axis=-1)
          end_logits = tf.nn.log_softmax(end_logits, axis=-1)

          seq_height = get_shape_list(inputs_batch['inputs_ids'])[0]
          start_logits = tf.keras.backend.reshape(start_logits, shape=(seq_height*seq_length, 1))
          end_logits = tf.keras.backend.reshape(end_logits, shape=(seq_height * seq_length, 1))
          start_positions = tf.keras.backend.reshape(start_positions, shape=(seq_height * seq_length, 1))
          end_positions = tf.keras.backend.reshape(end_positions, shape=(seq_height * seq_length, 1))
          start_loss = -tf.reduce_mean(tf.reduce_sum(start_positions * start_logits, axis=-1))
          end_loss = -tf.reduce_mean(tf.reduce_sum(end_positions * end_logits, axis=-1))
          loss_value = (start_loss + end_loss) / 2

         grads = tape.gradient(loss_value, model.trainable_weights)

        optimizer.apply_gradients(zip(grads, model.trainable_weights))

    if step % 200 == 0:
        print('Training loss (for one batch) at step %s: %s' % (step, float(loss_value)))
        print('Seen so far: %s samples' % ((step + 1) * batch_size))

context = "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse ('Norman' comes from 'Norseman') raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries."
question = "In what country is Normandy located?"

en_plus = tokenizer.encode_plus(context, question, add_special_tokens=True)
en = en_plus['input_ids']
token_type_ids = en_plus['token_type_ids']
input_ids = tf.constant([en])

segments_tensors = tf.constant([token_type_ids])

outputs = model(input_ids)

start_scores, end_scores = outputs[:2]

ss = tf.argmax(start_scores.numpy()[0]).numpy()
es = tf.argmax(end_scores.numpy()[0]).numpy()

answer = tokenizer.decode(en[ss: es+1], clean_up_tokenization_spaces=True)
print(ss)
print(es)
print(answer)

model.save_pretrained('./save/')
```

Thanks in advance for helping me.


Finetuning my language model

## ❓ Questions & Help

I have a problem about a finetuning of my own language model ([model](https://mxmdownloads.s3.amazonaws.com/umberto/umberto-commoncrawl-cased-v1.tar.gz) and [sentencepiece](https://mxmdownloads.s3.amazonaws.com/umberto/umberto-commoncrawl-cased-v1-sentencepiece.bpe.model)). I'm trying to use [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py).

When it tokenizes the text of the training set, the program enters a loop without giving feedback.
It blocks at [Line 105](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L105) in the _tokenize_ function.

```python
            ...
            logger.info("Creating features from dataset file at %s", directory)

            self.examples = []
            with open(file_path, encoding="utf-8") as f:
                text = f.read()

            tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))

            for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
                self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size]))
            # Note that we are loosing the last truncated example here for the sake of simplicity (no padding)
            # If your dataset is small, first you should loook for a bigger one :-) and second you
            # can change this behavior by adding (model specific) padding.
            ...
```
I have tried to wait for more than 1 hour, but nothing. Using BERT instead of my model, the finetuning starts (even if after several minutes with no log meanwhile).

I launched this:
```bash
python3 run_lm_finetuning.py \
--train_data_file /path/to/train.txt \
--eval_data_file /path/to/eval.txt \
--output_dir /path/to/output \
--mlm \
--do_train \
--do_eval \
--model_type roberta \
--model_name_or_path /path/to/my/model \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 8 \
--overwrite_output_dir \
--overwrite_cache \
--max_steps 500000 \
--block_size 128\
--save_steps 50000 \
--eval_all_checkpoints
```
My impression is that something is wrong with the tokenization on the model, any ideas?


run_ner.py huge discrepancy between eval and predict (or "dev" and "test" evaluation modes)

## ❓ Questions & Help

I'm comparing two different pretrained Bert models on the NER task. One is a bert-base-multilingual-cased model, which works fine, consistently, as one would expect. The other is our own pretrained multilingual Bert, which is trained on fewer languages and has so far shown better results on those few languages. However, when running run_ner.py from here, it doesn't evaluate consistently. The `dev.txt` dataset is completely identical to the `test.txt`dataset, so I'd expect identical results, but I get
```
--do_eval
f1 = 0.7603833865814698
loss = 0.06039658671007991
precision = 0.7531645569620253
recall = 0.7677419354838709
```
and
```
--do_predict
f1 = 0.025925925925925925
loss = 0.41404916612165316
precision = 0.030434782608695653
recall = 0.02258064516129032
```
I also tried `--evaluate_during_training` flag and I get "solid" results already from very few steps and identical to `--do_eval` at the end of training, but `--do_predict` are always much worse. Surprisingly, though, this doesn't occur with bert-base-multilingual-cased model even if I save it to disk and point to that folder. Additionally if under the clause if args.predict I change the mode from "test" to "dev" in function evaluate() I get good results. I repeat, dev and test are identical files that differ only in name.

The problem occurs on any max sequence length, after deleting cache, etc. No errors are displayed during training or evaluation/prediction.
Please help if you have any ideas what might be going wrong.


Bad Results with Albert

## ❓ Questions & Help
Trying to understand why is the cosine similarity between tokens with Albert way bad in comparison to DistilBert.
Any inferences on the same would be helpful.
Thanks in advance.

Embeddings constructed for a token by summing the last 4 encoded layers.
Distance metric: cosine

Results with DistilBert

Results with Albert


Bug in the command line tool: os.DirEntry not supported in Python 3.5

Hi,

[This line](https://github.com/huggingface/transformers/blob/1a8e87be4e2a1b551175bd6f0f749f3d2289010f/src/transformers/commands/user.py#L162) will cause a syntax error in Python 3.5 as `os.DirEntry` does not exist.

You should either update the code for backward compatibility or update the README replacing 3.5+ by 3.6 (I believe the former would be preferred).

Best regards.


Fix inconsistency between T5WithLMHeadModel's doc and it's behavior

The doc string for `T5WithLMHeadModel` is currently inconsistent with it's behavior.

The doc string says that the forward method ignores indices of -1 when computing the loss; however, the method instead ignores indices of -100. This pull request changes the method to ignore indices of -1, making the two consistent.

It's worth noting, there is a [commit](https://github.com/huggingface/transformers/commit/1b59b57b57010e6119282f3dbf37f8c7c6d6313e#diff-7370db3a19209bf984cc40925aaf2b71) by @thomwolf that changed this value from -1 to -100, though I couldn't find why the change was made. Thomas, perhaps you remember if the change is still important? If it is, I can instead update this PR to change the doc string.


Upload CLI: on Windows, uniformize paths/urls separators