examples/lm_finetuning/simple_lm_finetuning.py crashes with cublas runtime error

## Possible bug: simple_lm_finetuning.py crashes with cublas runtime error🐛 Bug

<!-- Important information -->
### TL;DR
I'm trying to finetune the existing English **bert-base-uncased** model according to the examples in `examples/lm_finetuning/README.md` on IMDB data, but fail. The `simple_lm_finetuning.py` script crashes with a cublas runtime error.

### VERBOSE

I have tried the following on a local machine, as well as on GCP, with two different datasets.

I have formatted the input data according to the specification in `examples/lm_finetuning/README.md` and stored it in a file called `imdb_corpus_1.txt` (essentially using the information kindly provided in https://medium.com/dsnet/running-pytorch-transformers-on-custom-datasets-717fd9e10fe2 wrt to the data and preprocessing)

To reproduce the issue, run the following on a suitable dataset.

The command:
```
~/pytorch-transformers/examples/lm_finetuning$ python3 simple_lm_finetuning.py --train_corpus ~/imdb_corpus_1_small.txt --bert_model bert-base-uncased --do_lo
wer_case --output_dir finetuned_lm --do_train
```

results in output ending with the following stacktrace:
```
raceback (most recent call last):
  File "simple_lm_finetuning.py", line 641, in <module>
    main()
  File "simple_lm_finetuning.py", line 591, in main
    outputs = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 694, in forward
    head_mask=head_mask)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 623, in forward
    head_mask=head_mask)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 344, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 322, in forward
    attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 279, in forward
    self_outputs = self.self(input_tensor, attention_mask, head_mask)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 199, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/fredrik_olsson/venv/pytorch-transformers/lib/python3.7/site-packages/torch/nn/functional.py", line 1371, in linear
    output = input.matmul(weight.t())
RuntimeError: cublas runtime error : resource allocation failed at /pytorch/aten/src/THC/THCGeneral.cpp:216
```

*However*, the following command, also pertaining to finetuning, successfully executes:

```
~/pytorch-transformers/examples$ python3 run_lm_finetuning.py --train_data_file ~/imdb_corpus.txt --output_dir fredriko --model_name_or_path bert-base
-uncased --mlm --do_train --do_lower_case --evaluate_during_training --overwrite_output_dir
```

## Environment

* OS: ubuntu
* Python version: python 3.7.3
* PyTorch version: torch 1.2.0
* PyTorch Transformers version (or branch): pytorch-transformers 1.2.0
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:


Write With Transformer adding spaces?

This issue didn't happen before, but now whenever you use the autocomplete it always adds a space to the beginning, even when a space is not needed. I.E: when adding a comma / period to the end of a sentence, when starting a new line, or (most egregiously) when finishing a word that was only partially written.

As an aside, I wouldn't mind waiting a few extra seconds to get auto-fill suggestions that are longer than two words.


Running XLNet on Squad

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

This is the padding problem. In GLUE codes in the examples, the padding for XLNet is on the left of the input, but in Squad codes, the padding is on right. I was wondering which one is correct.

Also, the inputs of `convert_examples_to_features` are different in Glue and Squad, where Squad uses most of default values like `pad_token, sep_token, pad_token_segment_id and cis_token_segment_id`, but Glue use the value of `tokenizer`. which one is correct?

Or the example codes are out-of-date? Thanks


Max encoding length + corresponding tests

The encoding function eases the encoding of sequences across tokenizers. The addition of the `head_mask` return further removes the pressure on the user to manually check the added special tokens.

There is currently no easy method to truncate the encoded  sequences while keeping the special tokens intact. This PR aims to change this by providing a `max_length` flag to be passed to the encoding function. This flag works even when no special tokens are involved (e.g. for GPT-2).

The second sequence is truncated while the first stays intact. If the first sequence is longer than the specified maximum length, a warning is sent and no sequences are truncated.


Why you need DistilBertModel class?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

You have `DistilBertModel`, `DistilBertForSequenceClassification`, etc. in `modeling_distilbert.py`. Why you need these classes? How about using `BertModel`, `BertForSequenceClassification`, etc.?

I found the weight names are different (e.g., `transformer.layer.0.attention.q_lin.weight`and `bert.encoder.layer.0.attention.self.query.weight`), but I think it's better to use the same weight names.

(A conversion script from distilbert weights to normal bert weights is useful to use `BertModel`.)

Thanks.


R-BERT implementation

## 🚀 Feature

An implementation of the R-BERT architecture for relationship classification

## Motivation

Hi @Huggingface. A recent paper describes an architecture for relationship classification called [R-BERT](https://arxiv.org/pdf/1905.08284.pdf), which claims SOTA performance on the Semeval 2010 Task 8 challenge. However, no code was provided with the paper. [I’ve written an implementation of this](https://github.com/azdatascience/pytorch-transformers/blob/rbert/examples/run_semeval.py) and can confirm it produces very good results (F1=89.07 using the official Semeval scoring script). Is this suitable for merging into the pytorch-transformers repo, or should it exist as a separate package?


fixed: hard coding for max and min number will out of range in fp16, which will cause nan.


model_type for gpt

## ❓ Questions & Help

To run openai-gpt model in run_glue.py what should we provide as model_type? The model classes contains
MODEL_CLASSES = {
    'bert'
    'xlnet'
    'xlm'
    'roberta'
}


KnowBert

As has amazingly and remarkably become a standard response to a new model announcement, will this new transformer model be implemented:
https://arxiv.org/pdf/1909.04164.pdf - KnowBert at EMNLP19.


breaking change

Great job, just in case it went unnoticed:
from revision 995e38b7af1aa325b994246e1bfcc7bf7c9b6b4f
to revision 2c177a87eb5faab8a0abee907ff75898b4886689

examples are broken due to changed orders of parameters in
pytorch_transformers/modeling_bert.py

```
<     def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None,
<                 position_ids=None, head_mask=None):
<         outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,
<                             attention_mask=attention_mask, head_mask=head_mask)
---
>     def forward(self, input_ids, attention_mask=None, token_type_ids=None,
>                 position_ids=None, head_mask=None, labels=None):
>
```


Different performance between pip install vs. download zip code

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi guys,
I meet a weird  problem. Basically I am using BERT for sentence pair classification, such as MNLI, RTE tasks.

I installed the pytorch_transformers by "pip install pytorch-transformers" and

> from pytorch_transformers.modeling_bert import BertForSequenceClassification

everything works fine, the performance on RTE can reach 70%.

Instead, I downloaded the zip code to local since I want to do some modification about the BertForSequenceClassification in my future projects. (let's say I renamed the unzipped folder into "my_pytorch_transformers"), and now import as:

> from my_pytorch_transformers.modeling_bert import BertForSequenceClassification

Now, everything was not the same; the performance was just 53%, and it showed difference from the beginning of iterations.

So, what's wrong here?  I thought maybe some initializations are different between the default installation and my local code?  But I always used the pretrained BERT "bert-large-uncased", ....I can not figure out where the difference comes from.

Thanks for any hints


unconditional generation with run_generation.py

## ❓ Questions & Help

Is it possible to have unconditional generation with ``` run_generation.py``` ? I realized the previous script ```run_gpt2.py``` had this option.
Can we use the same ```start_token``` trick?


Can pytorch-transformers be used to get XLM sentence embeddings for multiple languages?

## ❓ Questions & Help

I tried to create a class to get the XLM sentence embeddings in multiple languages.
```
class XLMSentenceEmbeddings(pt.XLMPreTrainedModel):
    def __init__(self, config):
        super(XLMSentenceEmbeddings, self).__init__(config)
        self.transformer = pt.XLMModel(config)
    def forward(self, input_ids, lengths=None, position_ids=None, langs=None, token_type_ids=None,
                attention_mask=None, cache=None, labels=None, head_mask=None):
        transformer_outputs = self.transformer(input_ids, lengths=lengths, position_ids=position_ids, token_type_ids=token_type_ids, langs=langs, attention_mask=attention_mask, cache=cache, head_mask=head_mask)
    return transformer_outputs[0][:, 0, :]
```

But if I try this code I get the same result if I set the language ids to Dutch or if I don't.
```
tokenizer = pt.XLMTokenizer.from_pretrained('xlm-mlm-100-1280')
xlm_model = XLMSentenceEmbeddings.from_pretrained('xlm-mlm-100-1280')

sentence = 'een nederlandstalige zin'
lang = 'nl'
input_ids = torch.tensor(tokenizer.encode(sentence, lang=lang)).unsqueeze(0)
with torch.no_grad():
    xlm_model.eval()
    output_without_lang = xlm_model(input_ids).numpy()
    output_with_lang = xlm_model(input_ids, langs=xlm_model.transformer.config.lang2id[lang]*torch.ones(input_ids.size(), dtype=int)).numpy()

np.sum(output_without_lang - output_with_lang)
```
The output of this sum at the end always returns zero.

If I change the config to use the language embeddings like so: `config.use_lang_emb = True` my results are random each time which seems to suggest the embeddings are not included in the model. Should I change a different config? Or is the only way to get the sentence embeddings to train the model again?


Special tokens / XLNet

it is necessary to add [CLS], [SEP] tokens in case of XLNet transformers?

Thanks!

*I used only the tokenizer.encode() function even if the sample had several sentences and I didn't set any special tokens. I think it was not the right way, isn't? It was done for a classification task.


Fixing typo in gpt2 for doc site's class link


ModuleNotFoundError in distillation/scripts/binarized_data.py

Hello,

importing logger (still? I've seen previous issues, but this is the first time I'm running the code myself) throws the ModuleNotFoundError.

```
Traceback (most recent call last):
  File "examples/distillation/scripts/binarized_data.py", line 25, in <module>
    from examples.distillation.utils import logger
ModuleNotFoundError: No module named 'examples.distillation'
```


how to finetuning with roberta-large

## 🐛 Bug

<!-- Important information -->

Model I am using (Roberta-large):

Language I am using the model on (English, ):

The problem arise when using:
* [ ] my own modified scripts: (give details)
export CUDA_VISIBLE_DEVICES=2
export GLUE_DIR=/home/zhaoguangxiang/bert/glue_data
DATA=MNLI
NUM_CLASSES=3
LR=1e-5
MAX_SENTENCES=32
TOTAL_NUM_UPDATES=123873
WARMUP_UPDATES=7432
for seed in 42
do
    python3 examples/run_glue.py \
    --model_type bert \
    --model_name_or_path https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-pytorch_model.bin \
    --task_name ${DATA} \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/${DATA} \
    --save_steps 10000 \
    --logging_steps 1000 \
    --max_seq_length 512 \
    --max_steps ${TOTAL_NUM_UPDATES}  --warmup_steps ${WARMUP_UPDATES} --learning_rate ${LR} \
    --per_gpu_eval_batch_size 32   \
    --per_gpu_train_batch_size 32   \
    --seed ${seed} \
    --output_dir checkpoint/roberta_${DATA}_output_seed${seed}/
done
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)

## To Reproduce

Steps to reproduce the behavior:

1. run scripts
2. i will see
![image](https://user-images.githubusercontent.com/17742385/64612608-cca88800-d406-11e9-82d8-3e45e02f8577.png)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

load or download roeberta and s tart  training

## Environment

* PyTorch Transformers version (or branch): latest
* Using GPU ? yes
* Distributed of parallel setup ? no


BLUE

In this PR:
- I add BertForMultiLabelClassification, RobertaForTokenClassification, RobertaForMultiLabelClassification.
- I add examples for Finetuning the BERT, RoBERTa models for tasks on BLUE (https://github.com/ncbi-nlp/BLUE_Benchmark). BLUE (Biomedical Language Understanding Evaluation) is similar to GLUE, but for Biomedical data. The "run_blue", "utils_blue" are customized from "run_glue", "utils_glue", but more sufficient, because it contains not only sequence classification, but also token classification, multi-label classification. People may also have more options for examples of fine-tuning BERT/RoBERTa.
- I also add test function to test_examples as well as test data (HOC)


Issue in fine-tuning distilbert on Squad 1.0

When I tried to finetune the distillbert (using run_squad.py in examples folder), the model reaches the F1 score of 17.43 on dev set but you have mentioned that the F1 score is 86.2. Can you help me with what I am doing wrong at the time of fine-tuning?

Below is the command that I am using
python ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path /root/distilbert_training \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file train-v1.1.json \
    --predict_file dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir /root/output/ \
    --per_gpu_eval_batch_size=3 \
    --per_gpu_train_batch_size=3


Roberta for squad

Hi,

Please add Roberta for squad.

Thanks
Mahesh


Quick questions about details

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Can someone explain the difference between `mem_len`,  `mlen`, and `ext_len` in `TransfoXLModel`? While the documentation has stated below

![Screen Shot 2019-09-10 at 11 18 40](https://user-images.githubusercontent.com/23093968/64583914-110f3600-d3bd-11e9-9571-5e9c607a1744.png)

Unfortunately, I'm still confused especially `mem_len` and `mlen`. Thank you so much.


❓ How to finetune `token_type_ids` of RoBERTa ?

## ❓ Questions & Help

RoBERTa model does not use `token_type_ids`.

However it is mentioned in the documentation :

> you will have to train it during finetuning

Indeed, I would like to train it during finetuning. I tried to load the model with :

`model = RobertaModel.from_pretrained('roberta-base', type_vocab_size=2)`

But I received the error :

> RuntimeError: Error(s) in loading state_dict for RobertaModel:
	size mismatch for roberta.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([1, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).

---

So **how can I create my RoBERTa model from the pretrained checkpoint, in order to finetune the use of `token ids` ?**


Fix to prevent crashing on assert len(tokens_b)>=1

Thank you for the awesome library! One little issue that I have sometimes with somewhat "noisy" text is that Bert tokenizer fails to process some weird stuff. In such a case, one unexpectedly gets no tokens and the converter fails on assert with a message like this one:
```
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]                      Traceback (most recent call last):
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 354, in <module>
    main()
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 350, in main
    create_training_file(docs, vocab_list, args, epoch)
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 276, in create_training_file
    whole_word_mask=args.do_whole_word_mask, vocab_list=vocab_list)
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 244, in create_instances_from_document
    assert len(tokens_b) >= 1
AssertionError

```


Can't reproduce XNLI zero-shot results from MBERT in Chinese

## ❓ Questions & Help

Hi guys,
I am trying to reproduce the XNLI zero-shot transfer results from MBERT.
With the same code and same checkpoint but different language for the test set, I am not able to reproduce the results for Chinese, Arabic, and Urdu. Does anyone encounter the same problem? Thanks!

Model |English | Chinese | Arabic | German | Spanish | Urdu
-- |-- | -- | -- | -- | -- | --
From MBERT github page | 81.4 | 63.8 | 62.1 | 70.5 | 74.3 | 58.3
My results | 82.076 | 36.088 | 35.327 | 68.782 | 70.419 | 35.170


Unable to load DistilBertModel after training

## ❓ Questions & Help

I'm following the example to train a DistilBert model from scratch from:  examples/distillation/README.md

I perform the training step:

```
python examples/distillation/train.py --dump_path ser_dir/sm_training_1 --data_file data/sm_bin_text.bert-base-uncased.pickle --token_counts data/sm_token_counts.bert-base-uncased.pickle --force --fp16
```

Which completes successfully, however the `dump_path` doesn't contain a `pytorch_model.bin` file so I cannot load in the model:

```py
from pytorch_transformers import DistilBertModel

DistilBertModel.from_pretrained("ser_dir/sm_training_1")
Model name 'ser_dir/sm_training_1' was not found in model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad). We assumed 'ser_dir/sm_training_1/pytorch_model.bin' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
...
OSError: file ser_dir/sm_training_1/pytorch_model.bin not found
```

Content of serialization directory:

```sh
ls ser_dir/sm_training_1/
checkpoint.pth  config.json  git_log.json  log  model_epoch_0.pth  model_epoch_1.pth  model_epoch_2.pth  parameters.json
```

I also tried to load it as a regular `BertModel`. Is there something else I need to do to load in a basic .bin file then apply the checkpoint weights on top? I cannot find any tutorials or examples specifying the next steps on this process.

Thanks!


How to deal with oov tokens with pretrained models

## ❓ Questions & Help

Can you, please, give an advice on how to handle out of vocabulary word? Just to use `[UNK]` token or there is a way to add this token to vocabulary and thus to train embedding for it?
Also, I noticed that oov words by default return multiple tokens. In my task (sequence tagging) I would like to save token-label correspondence.


changes in evaluate function in run_lm_finetuning.py

changed the return value of `evaluate` function from `results` to `result` and also removed unused empty dict `results`


Trying to fix the head masking test

Reviving this PR from @LysandreJik which tried to fix the head masking failing test by making random seed accessible anywhere within the common tests.


class DistilBertForMultiLabelSequenceClassification()

## 🚀 Feature

Distil BERT For Multi-Label Sequence Classification

## Motivation

To do multi-label text classification using DistilBERT

## Additional context

None


Question on the position embedding of DistilBERT

## ❓ Questions & Help

To the best of my knowledge, sinusoidal position embedding is used in the training procedure of DistilBERT, which is computed by [create_sinusoidal_embeddings](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_distilbert.py#L52).

When I compute the position embedding by using `create_sinusoidal_embeddings` with `n_pos=512` and `dim=768`, I got the following position embedding tensor:

```python
Parameter containing:
tensor([[ 0.0000e+00,  1.0000e+00,  0.0000e+00,  ...,  1.0000e+00,
          0.0000e+00,  1.0000e+00],
        [ 8.4147e-01,  5.4030e-01,  8.2843e-01,  ...,  1.0000e+00,
          1.0243e-04,  1.0000e+00],
        [ 9.0930e-01, -4.1615e-01,  9.2799e-01,  ...,  1.0000e+00,
          2.0486e-04,  1.0000e+00],
        ...,
        [ 6.1950e-02,  9.9808e-01,  5.3551e-01,  ...,  9.9857e-01,
          5.2112e-02,  9.9864e-01],
        [ 8.7333e-01,  4.8714e-01,  9.9957e-01,  ...,  9.9857e-01,
          5.2214e-02,  9.9864e-01],
        [ 8.8177e-01, -4.7168e-01,  5.8419e-01,  ...,  9.9856e-01,
          5.2317e-02,  9.9863e-01]])
```

However, when I looked into the position embedding from the pre-trained DistilBERT checkpoint files (`distilbert-base-uncased` and `distilbert-base-uncased-distilled-squad`,), I got the position embedding tensor as follows:

```python
tensor([[ 1.7505e-02, -2.5631e-02, -3.6642e-02,  ...,  3.3437e-05,
          6.8312e-04,  1.5441e-02],
        [ 7.7580e-03,  2.2613e-03, -1.9444e-02,  ...,  2.8910e-02,
          2.9753e-02, -5.3247e-03],
        [-1.1287e-02, -1.9644e-03, -1.1573e-02,  ...,  1.4908e-02,
          1.8741e-02, -7.3140e-03],
        ...,
        [ 1.7418e-02,  3.4903e-03, -9.5621e-03,  ...,  2.9599e-03,
          4.3435e-04, -2.6949e-02],
        [ 2.1687e-02, -6.0216e-03,  1.4736e-02,  ..., -5.6118e-03,
         -1.2590e-02, -2.8085e-02],
        [ 2.6413e-03, -2.3298e-02,  5.4922e-03,  ...,  1.7537e-02,
          2.7550e-02, -7.7656e-02]])
```

I am wondering if I missed or misunderstood something in details. Why is there a difference between these two position embedding tensors? I think the sinusoidal position embedding should be unchanged during training.

Thanks a lot!


Bert output last hidden state

## ❓ Questions & Help

Hi,

Suppose we have an utterance of length 24 (considering special tokens) and we right-pad it with 0 to max length of 64.
If we use Bert pertained model to get the last hidden states, the output would be of size [1, 64, 768].
Can we use just the first 24 as the hidden states of the utterance? I mean is it right to say that the output[0, :24, :] has all the required information?
I realized that from index 24:64, the outputs has float values as well.


Remove duplicate hidden_states of the last layer in BertEncoder in modeling_bert.py

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
For  "class BertEncoder" in modeling_bert.py, remove duplicate hidden_states of the last layer

## Motivation

![image](https://user-images.githubusercontent.com/2401439/64491964-fb0c5300-d2a0-11e9-9873-d86ee9041db3.png)

Bert-Base models have 12 layers instead of 13 layers. But when config.output_hidden_states is true, "len(all_hidden_states)" is printed 13 instead of 12. It seems that the two lines under "# Add last layer" is improper, since the last layer's hidden_states are already added.


[RuntimeError: sizes must be non-negative]  : XLnet, Large and Base

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu
* Python version: 3.5.2
* PyTorch version: 1.20, torch 0.4.1
* PyTorch Transformers version (or branch): latest
* Using GPU ? YES
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context
The issue reads almost exactly like this - https://github.com/huggingface/pytorch-transformers/issues/924
Except - the problem still appears to persist as I have pytorch 1.2 installed./
To be sure I have to use CODA 9.0 - not sure if that is causing the issue. Note - I can run - XLM, Bert and Roberta in the exact same data and code -jut swapping out the modelname .
<!-- Add any other context about the problem here. -->


Citing DistilBERT

Currently, my understanding is citing the repo/codebase should be done by via a link (i.e. in the paper as a footnote) as there is no citation (i.e. in BibTeX style) yet.
For citing DistilBERT (the released model and distillation approach), how should this be done?


Hi there, is bert-large-uncased-whole-word-masking-finetuned-squad trained for Squad 1.0 or 2.0?

I mean, whether the training data contains examples with no answer?


## ❓ Questions & Help

Hi,
I am running a modified version of ```run_lm_finetuning.py```, it was working fine and model checkpoints have been saved, until the last step of the first epoch (9677/9678), where I got this error:

```
Traceback (most recent call last):████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 9677/9678 [2:01:24<00:00,  1.36it/s]
  File "my_run_lm_finetuning.py", line 588, in <module>
    main()
  File "my_run_lm_finetuning.py", line 542, in main
    global_step, tr_loss = train(args, train_dataset, model, bert_model_fintuned, tokenizer, bert_tokenizer)
  File "my_run_lm_finetuning.py", line 260, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, enc_output, labels=labels)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 144, in forward
    return self.gather(outputs, self.output_device)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 156, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 67, in gather
    return gather_map(outputs)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 54, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 68, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/cuda/comm.py", line 166, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: Gather got an input of invalid size: got [2, 3, 12, 256, 64], but expected [2, 4, 12, 256, 64] (gather at /opt/conda/conda-bld/pytorch_1544199946412/work/torch/csrc/cuda/comm.cpp:227)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7f3c52b7fcc5 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: torch::cuda::gather(c10::ArrayRef<at::Tensor>, long, c10::optional<int>) + 0x4d8 (0x7f3c936eaba8 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #2: <unknown function> + 0x4f99de (0x7f3c936ed9de in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #3: <unknown function> + 0x111e36 (0x7f3c93305e36 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #14: THPFunction_apply(_object*, _object*) + 0x5dd (0x7f3c9350140d in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
```
Note that in this experiment I used a fine-tuned version of Bert (I fine-tuned it using your previous script in lm_finetune folder) and there I have the ```max_seq_length =256```, however when running this (```run_lm_finetuning.py```) , I have ```block_size=128```.

Any idea of what is the error for?


fix tokenize(): potential bug of splitting pretrained tokens with newly added tokens

In the tokenizer base class, `split_on_token()` attempts to split input text by each of the added tokens. Because it uses `text.split(tok)`, it may accidentally split a token in the pretrained vocabulary at the middle.

For example a new token "ht" is added to the vocabulary. Then "light" will be split into `["lig", ""]`. But as "light" is a token in the pretrained vocabulary, it probably should be left intact to be processed by `self._tokenize()`.

Hence in this pull request, `text.split()` is replaced with `re.split()`, which will split only at word boundaries (`[^A-Za-z0-9_]` in regular expression). This behavior can be enabled by specifying a new `tokenize()` argument: `additional_tokens_as_full_words_only=True` (default: False). If it's specified in `tokenizer.encode(text, ...)`, it will still take effect, as this argument will be passed down to `tokenize()`.

On languages that have no or different word boundaries as above (such as Chinese or Japanese), this behavior may produce undesirable results, and the user can revert to the old `text.split()` by not specifying `additional_tokens_as_full_words_only` (it will take the default value `False`).

An explanation of the argument `additional_tokens_as_full_words_only` has been added to the docstring of `tokenize()`. A test function `test_add_partial_tokens_tokenizer()` has been added to `tokenization_bert_test.py`.


How to set the weight decay in other layers after BERT output?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I notice that we should set weight decay of bias and LayerNorm.weight to zero and set weight decay of other parameter in BERT to 0.01. But how to set the weight decay of other layer such as the classifier after BERT? Thanks


Fixing head masking test

Try to fix the Nan in head masking tests by removing them.


Is there any sample code for fine-tuning BERT on sequence labeling tasks, e.g., NER on CoNLL-2003?

## ❓ Questions & Help
Is there any sample code for fine-tuning BERT on sequence labeling tasks, e.g., NER on CoNLL-2003, using BertForTokenClassification?
<!-- A clear and concise description of the question. -->


Cut off sequences of length greater than max_length= 512 for roberta

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
Roberta uses max_length of 512 but text to tokenize is variable length. Is there an option to cut off source text to maximum length during tokenization process?
## Motivation
Most text are not fixed sizes but are still required for end goal of using roberta on new datasets. The roberta model will throw an error when it encounters a text size > 512. Any help is appreciated to allow tokenization and limit the size to maximum length for pretrained encoders. Alternative is to let the pretrained encoder adapt by increasing positional embedding size so an error is not thrown or use average pooling. I m not sure how this would be implemented and allow fine tuning.
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
I can't seem to figure out how to circumvent the text length problem as new data text length can go as large as 2500 or more but roberta is only 512 max.
## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


Better examples

Refactored the examples section: removed old and not up-to-date examples and added new examples for fine-tuning and generation.

The `examples` file is not in the `/doc/source` folder anymore but in the `/examples` folder. It is therefore visible when users open the folder on GitHub.

Note: In order to generate the current documentation, a symlink has to be done between the `examples/README.md` file to a `docs/source/examples.md`. The corresponding documentation has been added to the documentation README, alongside the command necessary to create the symlink.

**The new examples are visible on: http://lysand.re/examples.html**


Fine-tuned RoBERTa models on CPU

## ❓ Questions & Help

Because of using FusedLayerNorm as BertLayerNorm from apex library after finetuning a saved model isn't possible to use on CPU with apex and Cuda installed. What will be the easiest way to run finetuned models on a server without a GPU? I see that the easiest way to just use the python version of BertLayerNorm during training but maybe there is another way.


LSTM returns nan after using the pretrained BERT embedding as input

Hello ,
 i'm using the pretrained Bert model (from pytorch-transformers) to get the contextual embedding of a written text ,  i summed the last 4 hidden layers outputs (i red that the concatenation of the last four layers usually produce the best results )
than i use a LSTM layer with attention to get the paragraph level embedding from the word embedding produced by the Bert model
the output should be a score range [-1,1]
I tried the rmsprop , adam ... optimizers with MSEloss and always after just  few batches iterations the lstm layer produces nan values .
 any suggestions will be greatly appreciated


How to fine tune small dataset?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Most people test bert on large dataset, but when it comes to small dataset, I assume the fine tune process and batch size maybe different. Besides, the dataset domain is twitter domain, which is kind of different from BERT pretrained corpus.
Could anyone gives some suggestions on finetuning BERT on small twitter dataset? Thanks in advance for any help.


Finetuning distilbert-base-uncased

## ❓ Questions & Help
When trying to finetune distilbert-base-uncased on my own dataset
I receive the following error message:
ERROR - pytorch_transformers.tokenization_utils -   Model name 'distilbert-base-uncased' was not found in model name list (bert-base-uncased,bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc).

How do I add distilbert-base-uncased to the model name list (I assume I will somehow have to modify PYTORCH-TRANSFORMERS_CACHE)

code used:
python simple_lm_finetuning.py \
--train_corpus dataset.txt \
--bert_model distilbert-base-uncased \
--do_lower_case \
--output_dir finetuned_lm/ \
--do_train


run_squad.py predictions

## ❓ Can anyone explain how start_logit and end_logit value is been determined ? And how it can used to measure the reliability of the answer span
In some issues I saw that if start_logit and end_logit = -1 then it is concluded to be unanswerable question.

There are some cases where I get the

> "1": [
{
"text": "hover your mouse pointer",
"probability": 1.0,
"start_logit": -12.987695693969727,
"end_logit": -12.40383529663086
}
],

In this place, what these values of start_logit and end_logit actually mean ? Since logit is natural log of odds, what is considered here as odds ?

In some cases, if the number of nbest predictions is 1, then [here in utils_squad.py](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_squad.py#L613-L614) the answer span is made `empty`, does it means that it is the right answer ?

            if len(nbest)==1:
                nbest.insert(0,
                    _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))

But in the [same file](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_squad.py#L618-L620), when there is no nbest predictions, again the answer span is made `empty`

        if not nbest:
            nbest.append(
                _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))

For two different situations why the same answer is been set ?


How to set the token_type_ids in XLNet correctly?

Hi, I am fine-tuning an XLNet, and I want to use type embedding to indicate different parts of a sequence. I am facing a difficulty that `“indices are selected in the vocabulary (unlike BERT which has a specific vocabulary for segment indices)”` which is the description of `token_type_ids` in the official document. Does that mean the type embedding and token embedding share the same vocabulary? In that case, how can I select the right indices for the types?  If I use `0` and `1` for types, is there a collision between types and the special tokens (like `UNK`)?

Thanks in advance!


convert_roberta_checkpoint_to_pytorch.py 514 max position?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I m not sure if this is intentional but why is max position 514. I m assuming the origingal roberta model is 512 like bert or is this incorrect? This is the only place i find this reference as well.


the best way to cut the upper layers

Salut,

what would be the best way to cut the upper layers of transformers (from 12 cutting 5 upper layer results 7 layers model for the use)?

Best regards


Fix typo


Can't trace any model with pytorch-transformers 1.2

## 🐛 Bug

I get an error whenever I try to trace any model from pytorch-transformers 1.2.0.
When I roll back to 1.1 everything is fine.

## To Reproduce

```python
from pytorch_transformers import BertModel
import torch

model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
model.to('cuda')
model.eval()
ids = torch.LongTensor([[1, 2, 3]]).cuda()
tok = torch.zeros_like(ids)
att = torch.ones_like(ids)
torch.jit.trace(model, (ids, tok, att, ids))
```

This script produces the following error:

```
Traceback (most recent call last):
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 545, in run_mod_and_filter_tensor_outputs
    outs = wrap_retval(mod(*_clone_inputs(inputs)))
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
RuntimeError: r ASSERT FAILED at /pytorch/aten/src/ATen/core/jit_type.h:142, please report a bug to PyTorch. (expect at /pytorch/aten/src/ATen/core/jit_type.h:142)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f273aa91441 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f273aa90d7a in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
frame #2: std::shared_ptr<c10::DimensionedTensorType const> c10::Type::expect<c10::DimensionedTensorType const>() + 0x140 (0x7f27397ff810 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #3: torch::jit::fuser::compileKernel(torch::jit::fuser::KernelSpec const&, torch::jit::fuser::ArgSpec const&, std::vector<long, std::allocator<long> > const&, c10::Device) + 0xa5a (0x7f27397fbdca in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #4: torch::jit::fuser::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&, std::string*) + 0x5b0 (0x7f2739803c20 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #5: torch::jit::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x13 (0x7f2739733bc3 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #6: <unknown function> + 0xb2b066 (0x7f273973d066 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #7: <unknown function> + 0xa8ebe6 (0x7f27396a0be6 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #8: torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x22 (0x7f273969c202 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #9: <unknown function> + 0xa7685d (0x7f273968885d in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #10: <unknown function> + 0x457617 (0x7f277a2cd617 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x130d0c (0x7f2779fa6d0c in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #14: python3() [0x4fbfce]
frame #16: python3() [0x574db6]
frame #20: python3() [0x4ec2e3]
frame #22: python3() [0x4fbfce]
frame #24: python3() [0x574db6]
frame #27: python3() [0x5401ef]
frame #30: python3() [0x4ec3f7]
frame #33: python3() [0x5401ef]
frame #35: python3() [0x53fc97]
frame #37: python3() [0x53fc97]
frame #39: python3() [0x60cb42]
frame #44: __libc_start_main + 0xf0 (0x7f279df7d830 in /lib/x86_64-linux-gnu/libc.so.6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "session.py", line 10, in <module>
    torch.jit.trace(model, (ids, tok, att, ids))
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 702, in trace
    _check_trace([example_inputs], func, executor_options, traced, check_tolerance, _force_outplace)
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/autograd/grad_mode.py", line 43, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 583, in _check_trace
    traced_outs = run_mod_and_filter_tensor_outputs(module, inputs, 'trace')
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 551, in run_mod_and_filter_tensor_outputs
    ' with test inputs.\nException:\n' + indent(str(e)))
torch.jit.TracingCheckError: Tracing failed sanity checks!
Encountered an exception while running the trace with test inputs.
Exception:
	r ASSERT FAILED at /pytorch/aten/src/ATen/core/jit_type.h:142, please report a bug to PyTorch. (expect at /pytorch/aten/src/ATen/core/jit_type.h:142)
	frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f273aa91441 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
	frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f273aa90d7a in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
	frame #2: std::shared_ptr<c10::DimensionedTensorType const> c10::Type::expect<c10::DimensionedTensorType const>() + 0x140 (0x7f27397ff810 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #3: torch::jit::fuser::compileKernel(torch::jit::fuser::KernelSpec const&, torch::jit::fuser::ArgSpec const&, std::vector<long, std::allocator<long> > const&, c10::Device) + 0xa5a (0x7f27397fbdca in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #4: torch::jit::fuser::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&, std::string*) + 0x5b0 (0x7f2739803c20 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #5: torch::jit::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x13 (0x7f2739733bc3 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #6: <unknown function> + 0xb2b066 (0x7f273973d066 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #7: <unknown function> + 0xa8ebe6 (0x7f27396a0be6 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #8: torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x22 (0x7f273969c202 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #9: <unknown function> + 0xa7685d (0x7f273968885d in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #10: <unknown function> + 0x457617 (0x7f277a2cd617 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
	frame #11: <unknown function> + 0x130d0c (0x7f2779fa6d0c in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
	<omitting python frames>
	frame #14: python3() [0x4fbfce]
	frame #16: python3() [0x574db6]
	frame #20: python3() [0x4ec2e3]
	frame #22: python3() [0x4fbfce]
	frame #24: python3() [0x574db6]
	frame #27: python3() [0x5401ef]
	frame #30: python3() [0x4ec3f7]
	frame #33: python3() [0x5401ef]
	frame #35: python3() [0x53fc97]
	frame #37: python3() [0x53fc97]
	frame #39: python3() [0x60cb42]
	frame #44: __libc_start_main + 0xf0 (0x7f279df7d830 in /lib/x86_64-linux-gnu/libc.so.6)
```

## Environment

* OS: Ubuntu 16.04.6 LTS
* Python version: Python 3.5.2
* PyTorch version:  '1.1.0'
* PyTorch Transformers version (or branch): '1.2.0'
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:
I installed pytorch-transformers with pip:
```
pip3 install --user pytorch-transformers==1.2
```


[2.0] TF 2.0 support

Currently converted models:
- [x] BERT
- [x] GPT-2
- [x] XLNet
- [x] XLM
- [x] Transformer-XL
- [x] GPT
- [x] RoBERTa
- [x] DistilBert

With TF 2.0 Keras imperative interface and Eager, the workflow and models are suprisingly similar:
```python
import numpy
import torch
import tensorflow as tf
from pytorch_transformers import BertModel, TFBertModel, BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
pytorch_model = BertModel.from_pretrained('bert-base-uncased')
tf_model = TFBertModel.from_pretrained('bert-base-uncased')

text = "[CLS] Who was Jim Henson ? Jim [MASK] was a puppeteer [SEP]"
tokens = tokenizer.encode(text)

pytorch_inputs = torch.tensor([tokens])
tf_inputs = tf.constant([tokens])

with torch.no_grad():
    pytorch_outputs = pytorch_model(pytorch_inputs)

tf_output = tf_model(tf_inputs, training=False)

numpy.amax(numpy.abs(pytorch_outputs[0].numpy() - tf_output[0].numpy()))
# >>> 2.861023e-06 => we are good,  a few 1e-6 is the expected difference
# between TF and PT arising from internals computation ops
```

The convention is to use the same name for classes as the original PyTorch classes but prefixed with `TF`.

If you want to install and use this development branch, you should install from the `tf2` branch like this:
- install TF 2.0: `pip install tensorflow==2.0.0-rc0`
- install pytorch-transformers from the `tf2` branch: `pip install https://github.com/huggingface/pytorch-transformers/archive/tf2.zip`

TO-DO / not forget:
- [ ] check weights initialization
- [x] add weights tying
- [x] add example with losses using `model.compile` /`model.fit`
- [ ] take care of having the two possible gelu implementations for Bert
- [ ] untangle Transfo-XL tokenizer from `torch.load` and `torch.save`
- [x] test that all dropout modules are desactivated when training=False (check determinism)
- [ ] clean up our FP16 support (for PyTorch as well) with (i) an adjustment of masking values and (ii) an adjustment of LayerNorm epsilon (add an attribute in configuration files).


Learning word-pieces garble the predictions

## 🐛 Bug

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The tasks I am working on is:
* [ x] my own task or dataset: Sentiment Analysis on chatbot conversations

Word-pieces change the prediction completely, I fail to understand why.
If i do a token to token similarity also, by considering the token vector to be the average of wordpieces or the first token vector to see if they carry the same context, they don't.
If I had lemmatized the training set, these pieces would have not learnt anything really.
But that is not intuitive.

[CLS] token vector is fed to the classifier. Changes need to be done internally.
How can I handle such cases?
Any leads would be helpful.
Thanks in advance.


[2.0] - Split configuration and modeling files

Refactor to split configuration and modeling files so we can share configuration easily between various frameworks.

This PR is quite annoying to rebase so we should probably merge it pretty soon.


Distributed device ordinal question

## ❓ Questions & Help

In the following line, we set the device ordinal to be local rank. However, suppose we have four independent nodes each with only one GPU. Then the 4th node (rank 3) will execute this line with the device ordinal 3, but it really only has 1 GPU, so it'll be invalid to ask for the GPU 3. So won't that break things? Shouldn't it be `torch.device("cuda", 0)`?

https://github.com/huggingface/pytorch-transformers/blob/0287d264e913e10018a95a2723115dc9121e5fc6/examples/run_glue.py#L403


Fixing TransformerXL bool issue #1169

Fixing #1169 regarding using uint or bool masks in Transformer-XL and PyTorch 1.1.0 and 1.2.0.

Hopefully, this solution will be compatible upward with the future PyTorch releases.


How to fine-tune xlnet on SQuAD with the parameter setting provided in the paper?

From [here](https://arxiv.org/pdf/1906.08237.pdf) on page 16, it seems we should set Layer-wise lr decay to 0.75. However, I didn't find a way to do so in `run_squad.py`. Could someone provide a sample command line that could run this fine-tune task with the given parameters?

Thanks!


Fix loading of question answering bert from tf weights.

I've got an attribute error when loading pretrained tf weights for question answering (bert) in `load_tf_weights_in_bert` at:
```
elif l[0] == 'squad':
     pointer = getattr(pointer, 'classifier')
```
since `BertForQuestionAnswering` does not have a 'classifier' attribute but `qa_outputs`. I've added a try except, which resolves the error.


RoBERTa/GPT2 tokenization

Hi,

I've one question regarding to the tokenization logic.

I'm using the RoBERTa tokenizer from `fairseq`:

```python
In [15]: tokens = roberta.encode("Berlin and Munich have a lot of puppeteer to see .")

In [16]: tokens
Out[16]:
tensor([    0, 26795,  2614,     8, 10489,    33,    10,   319,     9, 32986,
         9306,   254,     7,   192,   479,     2])
```

Interestingly, Berlin will be splitted into two subwords (with ids 26795 and 2614).

When I use the `pytorch-transformer` implementation:

```
In [21]: tokens = tokenizer.tokenize("<s>Berlin and Munich have a lot of puppeteer to see .</s>")

In [22]: indexed_tokens = tokenizer.convert_tokens_to_ids(tokens)

In [23]: indexed_tokens
Out[23]: [0, 5459, 8, 10489, 33, 10, 319, 9, 32986, 9306, 254, 7, 192, 479, 2]
```

Berlin is not splitted 😅

The `roberta.encode` method will return one subword for Berlin, when I start the sentence with a space - which tokenizer is correct here 🤔


[2.0] Reodering arguments for torch jit #1010 and future TF2.0 compatibility

Torch jit (cf #1010) and TF 2.0 (cf #1104) are more strict than PyTorch on having a specific order of arguments for easy use.

This PR refactor the order of the keyword arguments to make them as natural as possible.

This will be a breaking change for people using positional order to input keyword arguments in the forward pass of the models, hence is delayed to the 2.0 release.


How to finetune DistilBERT on custom data?

## ❓ Questions & Help

I want to build classifier by using DistillBERT. I would like to know how to finetune it on the custom dataset and build classifier on it.
Thansk!


how to get distilbert-base-uncased-distilled-squad?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I think it is initialize a six layer bert and distill with a 12 layer bert, then save the checkpoint file, is it right?


Finetuning BertModel to extract textual features for VQA shows bad results

## ❓ Questions & Help
I am trying to use Bert as a textual feature extractor for VQA.

This is the code for tokenizing question text in VQA.
```
    self.tokenizer = pytorch_transformers.BertTokenizer.from_pretrained(
            'bert-base-uncased')
    q = self.tokenizer.encode(q_text, add_special_tokens=True)
```

This code is for extracting features from Bert.
```

  self.bert = pytorch_transformers.BertModel.from_pretrained('bert-base-uncased',
                                                                   output_attentions=True)
    question_mask_cls = torch.arange(lengths[0]).to(self.device)
    lengths_cls = lengths
    q_mask_cls = question_mask_cls[None, :] < lengths_cls[:, None]
    q_mask_cls = q_mask_cls.to(torch.float32)
    question_embed_t, _, src_attn_list = self.bert(question_padded,
                                                 attention_mask=q_mask_cls)

    output, tgt_attn_list, tgt_src_attn_list = self.q_decoder(
            tgt=obj_feature_bbox_cls.permute(1, 0, 2),
                       memory=question_embed_t.permute(1, 0, 2),
                       memory_key_padding_mask=memory_key_padding_mask,
                       tgt_key_padding_mask=tgt_key_padding_mask)
```

If I free the parameters of Bert, it gives better results. But when I fine tune the whole model, the model does not seem to learn. I've tried the Adam optimizer of pytorch and AdamW provided by this repository. Both of them does not work.

```
optimizer = pytorch_transformers.optimization.AdamW(model.parameters(), lr=3e-5)
```

![image](https://user-images.githubusercontent.com/8081512/64243428-4ab9ea00-cf42-11e9-8e36-70121e0453ae.png)

The orange curve shows the model with Bert parameters freezed, while the pink and skyblue curve shows the model that trains Bert parameters.

Are there any potential issues I am missing?
<!-- A clear and concise description of the question. -->


how to use 'spiece.model' to create the xlnet_tokenizer

## ❓ Questions & Help
For some reason, my computer can not connect to Internet, which means I can not use the code "tokenizer = tokenizer_class.from_pretrained('xlnet-base-cased', do_lower_case = True)" to create the tokenizer.
The Piece model (spiece.model)  is used for (de)tokenization, how can I use it to create the tokenizer?
I have tried :
`
from pytorch_transformers import XLNetTokenizer
config = {
'vocab_path' :  path.sep.join([BASIC_DIR,'pretrained/pytorch_xlnet_pretrained/spiece.model'])
        }
tokenizer = XLNetTokenizer.from_pretrained(config['vocab_path'], do_lower_case = True)
`
it doesn't work.


Fix reference of import in XLM tokenization


Roberta tokenizer fails on certain unicode characters

## 🐛 Bug

<!-- Important information -->

Model I am using: `Roberta`:

Language I am using the model on (English, Chinese....): English

The problem arise when using:
- The `roberta-base` tokenizer and tokenizing unicode accents
```
from pytorch_transformers import *

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
phrase = "I visited the Côte d'Azur"
for word in phrase.split():
    print(tokenizer.tokenize(word))

```
this outputs:

```
['I']
['vis', 'ited']
['the']
['C', 'Ã´', 'te']
['d', "'", 'Az', 'ur']
```

## Expected behavior
```
['I']
['vis', 'ited']
['the']
['C', 'ô', 'te']
['d', "'", 'Az', 'ur']
```
## Environment

* OS: MacOS 10.14.6
* Python version: 3.6
* PyTorch version: 1.1.0.post2
* PyTorch Transformers version (or branch): 1.1.0
* Using GPU ? no
* Distributed of parallel setup ? no
* Any other relevant information: xlnet and bert do not face this same tokenization issue. It appears that the issue comes from the gpt2-tokenizer

## Additional context

<!-- Add any other context about the problem here. -->


BertEncoder head_mask not subscript-able error when not passed

## 🐛 Bug

BertEncoder takes in head_mask parameters with default value of None but on https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L431, ith index is checked without checking if head_mask is None. If by default nothing is passed this results in error.

## Fix
Check head_mask is not None in https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L431


## ❓ Questions & Help

Using do_eval from run_glue.py uses the cached result. I want to evaluate my fine-tuned models and I can't find any guide on how to do so. Anybody that can point me in the right direction?


[README] link to Write With Transformer

Tomorrow we'll release a new version of Write With Transformer that's gonna let you:
- experiment with different models (gpt2, xlnet) and/or model checkpoints (example for gpt2: small, large, arxiv)
- Share links to your documents.

With those two changes transfomer.huggingface.co is graduating to being an official demo for `pytorch-transformers`'s text generation capabilities.


XLnet output attentions doesn't work

## 🐛 Bug

Model I am using - XLNet.

Language I am using the model on English.

The problem arise when using the flag `output_attentions=True`

## To Reproduce

## Expected behavior
When executing
`        outputs= self.model(input_ids)
`
I would expect to have a tuple with outputs and attentions but it fails.
The problem probably roots from the lines:
`            if self.output_attentions:
                attentions.append(outputs[2])
`
I receive Nones instead of the attentions, or sometimes the error:
`IndexError: tuple index out of range`
For the following line in the forward function:
`attentions.append(outputs[2])`
## Environment

* OS: Windows
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): Master branch version from 31.08.19
* Using GPU: Yes


Convert RoBERTa to TF checkpoint

Can we use the "convert_pytorch_checkpoint_to_tf" script to convert the RoBERTa checkpoint to the Tensorflow ckpt? Thanks.


## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): DistilBertModel

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: I am trying to run the sample in the given examples within a notebook. Specifically the example in [DistilBERT's Example README](https://github.com/huggingface/pytorch-transformers/tree/master/examples/distillation)

## To Reproduce

Steps to reproduce the behavior:

1. Create a new notebook with a Python 3.7 interpretter
2. Type in the following code:
```
!pip install pytorch-transformers
import torch
from pytorch_transformers.tokenization_distilbert import DistilBertTokenizer
from pytorch_transformers.modeling_distilbert import DistilBertModel
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```
3. Run & receive the error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-48-3aa5cae06e9c> in <module>
      1 tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
----> 2 model = DistilBertModel.from_pretrained('distilbert-base-uncased')
      3
      4 input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)
      5 outputs = model(input_ids)

/mnt/c/Users/.../venv/lib/python3.7/site-packages/pytorch_transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    472             assert model.config.output_attention == True
    473             # Loading from a TF checkpoint file instead of a PyTorch model (slower)
--> 474             config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json')
    475             model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)
    476

/mnt/c/Users/.../venv/lib/python3.7/site-packages/pytorch_transformers/modeling_distilbert.py in __init__(self, config)
    486         self.transformer = Transformer(config) # Encoder
    487
--> 488         self.init_weights()
    489
    490     def _resize_token_embeddings(self, new_num_tokens):

/mnt/c/Users/.../venv/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    589                 return modules[name]
    590         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 591             type(self).__name__, name))
    592
    593     def __setattr__(self, name, value):

AttributeError: 'DistilBertModel' object has no attribute 'init_weights'
```

## Expected behavior
Code would execute. Not sure if init_weights should be inherited or if it's a type (there is [_init_weights ](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_distilbert.py#L403)in the DistilBertTokenizer class)

## Environment

* OS: Win10 (with WSL)
* Python version: 3.7.4
* PyTorch version: 1.2
* PyTorch Transformers version (or branch): mainline
* Using GPU ? no
* Distributed of parallel setup ? parallel


Updated GLUE script. New feature: Binary mask creation from the tokenizer's encoding.

The new `tokenizer.encode(seq_0, seq_1, add_special_tokens=True)` method makes life easier when building sequences. However, it makes it harder to create binary masks as the different sequence lengths are unknown. As a feature, I have therefore added a flag to the encode function so that it can output binary masks.

Example:
```py
from pytorch_transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
seq_0 = "This is the one"
seq_1 = "This is the last"

input_ids, mask = tokenizer.encode(seq_0, seq_1, add_special_tokens=True, output_mask=True)
# input_ids: [ 101, 1188, 1110, 1103, 1141,  102, 1188, 1110, 1103, 1314, 102]
# mask:      [   0,    0,    0,    0,    0,    0,    1,    1,    1,    1,   1]
```

It works for BERT, RoBERTa, XLM, and XLNet. I have refactored the GLUE example with this method. It greatly simplifies input creation.

I have added an additional unit test to the `commontests` suite. Furthermore, in order to make sure the tokenization was correct I compared against the original input creation of the GLUE script to make sure every encoded sequence remained the same.


DistilBERT Loss Function Choice and further query on extending to GPT2.

## ❓ Questions & Help

Can you describe the motiavtion behind scaling the KLDivLoss by squared temperature ?

https://github.com/huggingface/pytorch-transformers/blob/50792dbdcccd64f61483ec535ff23ee2e4f9e18d/examples/distillation/distiller.py#L331

When applying the same logic for GPT-2 distillation, I did the following

``` python
    def training_step(self, data_batch, batch_i):
        """
        Lightning calls this inside the training loop
        :param data_batch:
        :return:
        """
        # forward pass

        token_ids, lengths = data_batch
        orig_loss_ce, s_logits = self.student(input_ids=token_ids, labels=token_ids)[:2]     # (bs, seq_length, voc_size)

        self.teacher.eval() # Required to do this every time.
        with torch.no_grad():
            t_logits = self.teacher(input_ids=token_ids)[0] # (bs, seq_length, voc_size)

        loss_kl = self.kld_loss_fct(F.log_softmax(s_logits/self.temperature, dim=-1),
                                    F.softmax(t_logits/self.temperature, dim=-1)) * (self.temperature)**2
        loss_kl /= s_logits.shape[1]
        loss = self.alpha_kl * loss_kl
        if self.alpha_orig_ce > 0.:
            loss += self.alpha_orig_ce * orig_loss_ce
        if self.alpha_mse > 0.:
            loss_mse = self.mse_loss_fct(s_logits, t_logits)/s_logits.size(0) # Reproducing batchmean reduction
            loss += self.alpha_mse * loss_mse

        # in DP mode (default) make sure if result is scalar, there's another dim in the beginning
        if self.trainer.use_dp:
            loss = loss.unsqueeze(0)

        output = OrderedDict({
            'loss': loss
        })

        # can also return just a scalar instead of a dict (return loss_val)
        return output

```
I found that the distillBert implementation lead to high initial loss range for kl (130-180) depending on the average sequence length per batch while cross entropy was in range of 4-5.
So I scaled the loss_kl by the total timesteps in the batch. (My batches don't have any masked tokens). Training did converge to similar perplexity as teacher on the held out set of toronto books.
Is my method motivated, or am I applying the KL wrongly in the GPT2 scenario necessiating the scaling ?


DistilBERT baseline

Nice work on DistillBert! I was wondering if you had baseline numbers for the student model trained without the distillation objective? Seems like a very important baseline to understand if distillation was useful.


DistilBERT training is killed because OOM

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am trying DistilBERT training. The training script (train.py) had gradually consumed CPU memory, and the training was killed because OOM in about one day (the available CPU memory is 96GB).
I used one GPU for the training.

Do you have any idea? Thanks in advance.


added tokens may split a normal token into halves

In the tokenizer base class, split_on_token() attempts to split input text by each of the added tokens. Because it uses text.split(tok), it may accidentally split a token at the middle.

For example a new token "ht" is added to the vocabulary. Then "light" will be split into "lig" and "". But as "light" is in the original vocabulary, it should be left intact to be processed by self._tokenize().

Hence I'd suggest to replace it with re.split, which will split only at word boundaries ([0-9a-zA-Z_]).

But in languages whose word boundaries are different from English, this behavior may be undesirable and the user can revert to the old text.split(). It is controlled by a newly added flag split_added_on_word_boundary.


How to install previous versions of pytorch-transformers

Hi
I am using some codes requiring old versions of your work, do you mind telling me
how to install old version of this repository?
thanks
Best
Julia


merge


May I get the details of Bert pre-train procedure?

## ❓ Questions & Help
I want to measure the time consume in BERT pre-train procedure. May I ask some questions here:
1. What is the pre-train data? Is that same in paper : BooksCorpus & English Wiki.
2. Is there some code for pre-train can I utilize?
3. May I know how much epochs should I train when pre-train 'base' version model or 'large' version model?


Fix byte-level BPE decoding error when using added tokens

This PR fixes a mismatch between regular unicode added tokens and byte-level BPE tokens when doing decoding.

Wrong behavior reported in #1133.

Also adds regression tests.


Write with Transformer doesn't show 774M model?

According to [this twitter post](https://twitter.com/huggingface/status/1166368535221870592), Write With Transformer has been updated to include the Large model.

However, the Model & Decoder Settings only shows Small and Medium options for model size.


apex fp16 FusedLayerNorm type issues

#564  🐛 Bug

I seem to be getting the following error each time I try to train with APEX/fp16 with BERT finetuning. It happened with my own scripts and I also see this with repository's standard `finetune_on_pregenerated.py` which was recently updated. The error diagnostics seem to indicate an issue with the `FusedLayerNorm`. To further confirm: doing a local mod where I replaced the definition of BertLayerNorm with

```BertLayerNorm = torch.nn.LayerNorm```

The change resolves this issue (while, in my case, not noticeably changing the performance).. Apex docs are a bit raw but the most recent set does not suggest to manually manipulate optimizers or layer definitions, perhaps we should just stick to the BertLayerNorm definition as described above?

```
Traceback (most recent call last):
  File "ash3/tune_bert.py", line 101, in <module>
    main(sys.argv[1:])
  File "ash3/tune_bert.py", line 47, in main
    pregenerate(init)
  File "ash3/tune_bert.py", line 85, in pregenerate
    finetune_on_pregenerated(tune_args)
  File "/home/madvillain/gitlab/ai/ash3/ash3/finetuning/finetune_on_pregenerated.py", line 292, in main
    outputs = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 785, in forward
    prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 533, in forward
    prediction_scores = self.predictions(sequence_output)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 501, in forward
    hidden_states = self.transform(hidden_states)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 483, in forward
    hidden_states = self.LayerNorm(hidden_states)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/apex/normalization/fused_layer_norm.py", line 159, in forward
    input, self.weight, self.bias, self.normalized_shape,self.eps)
  File "/home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/apex/normalization/fused_layer_norm.py", line 25, in forward
    input_, ctx.normalized_shape, weight_, bias_, ctx.eps)
RuntimeError: expected scalar type Half but found Float (data<c10::Half> at /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/include/ATen/core/TensorMethods.h:1386)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7f6af587edc5 in /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Half* at::Tensor::data<c10::Half>() const + 0x2c6 (0x7f6abeb8aa36 in /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #2: cuda_layer_norm(at::Tensor*, at::Tensor*, at::Tensor*, at::Tensor*, int, int, c10::ArrayRef<long>, at::Tensor*, at::Tensor*, double) + 0x3ed (0x7f6abeb87dcd in /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #3: layer_norm_affine(at::Tensor, c10::ArrayRef<long>, at::Tensor, at::Tensor, double) + 0x27a (0x7f6abeb7985a in /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #4: <unknown function> + 0x196c4 (0x7f6abeb866c4 in /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #5: <unknown function> + 0x16e0a (0x7f6abeb83e0a in /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
<omitting python frames>
frame #12: THPFunction_apply(_object*, _object*) + 0x691 (0x7f6b24b0a081 in /home/madvillain/miniconda3/envs/ash3/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
```

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [* ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [* ] an official GLUE/SQUaD task: (give the name) finetune_on_pregenerated.py
* [ ] my own task or dataset: (give details)

## Expected behavior

no failures

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6
* PyTorch version: 1.1.0, 1.2.0
* PyTorch Transformers version (or branch): 1.1.0
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information: cudatoolkit 10.0, APEX git hash code: 53eae1986320d016ee7b347d78839dd5e96e7e93


Can't get GPT2tokenizer to load correctly

## ❓ Questions & Help

Hi, I'm a fairly new coder and I'm hitting a roadblock that I just don't understand – maybe someone here can help me, but I figure it's worth asking the community.

I'm trying to run run_lm_finetuning.py, and the tokenizer doesn't seem to be loading correctly. I'm getting this error: `AttributeError: 'GPT2Tokenizer' object has no attribute 'max_len_single_sentence'`

I've looked at the code, and there clearly is a `max_len_single_sentence` attribute in the init, but I can't get to it. I've even tried simply loading a GPT2-tokenizer into a jupyter notebook and trying to get the value, and it has the same error.

I assume I've done something wrong, I just can't figure out what. In case it helps, I've put my entire traceback below.

Any ideas? Thanks!

```
python examples/run_lm_finetuning.py --train_data_file='HFlongs1000.txt' --output_dir='pytorch-transformers/HFOutput' --model_type='gpt2' --tokenizer_name='gpt2' --model_name_or_path='gpt2'
09/01/2019 06:44:36 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
09/01/2019 06:44:36 - INFO - pytorch_transformers.modeling_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /home/jupyter/.cache/torch/pytorch_transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
09/01/2019 06:44:36 - INFO - pytorch_transformers.modeling_utils -   Model config {
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "vocab_size": 50257
}

09/01/2019 06:44:36 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /home/jupyter/.cache/torch/pytorch_transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
09/01/2019 06:44:36 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /home/jupyter/.cache/torch/pytorch_transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
Traceback (most recent call last):
  File "examples/run_lm_finetuning.py", line 497, in <module>
    main()
  File "examples/run_lm_finetuning.py", line 431, in main
    args.block_size = tokenizer.max_len_single_sentence  # Our input block size will be the max possible for the model
AttributeError: 'GPT2Tokenizer' object has no attribute 'max_len_single_sentence'```


How to use BERT or word embedding for e-commerce product classification.

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to classify products on an e-commerce site. For example, if we take Amazon for example if a product name is iPhone XS then it should be categorized to Electronics -> mobile, very straight forward, however, the problem comes when we train the model on clothes and many other sports items.
For example: "George - George Men's Cargo Short" found on walmart is being classified as SPORTS & OUTDOOR, FASHION. However, it should be classified to Clothes.

Currently, we have tried text CNN but I'm very positive that BERT or other word embeddings can enhance the performance.

Base Code:
https://github.com/brightmart/text_classification
However, it appears that TextCNN is better than BERT as per the author of this repository. Does anyone know what's the ideal way to approach this problem?
![image](https://user-images.githubusercontent.com/7957331/64069995-c22b0900-cc24-11e9-8077-21313e38c970.png)


Attribute errors with pytorch_transformers tests

## 🐛 Bug

<!-- Important information -->

Model I am using (from the official repo):

Language I am using the model on (English, Yoruba,Igbo, Hausa etc):

The problem arise when using:
* [ ] the official example scripts: (give details): the test run script

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)=>yes

## To Reproduce

Steps to reproduce the behavior:

1.python -m pytest -sv ./pytorch_transformers/tests/

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

`pytorch_transformers/tests/modeling_transfo_xl_test.py::TransfoXLModelTest::test_transfo_xl_lm_head FAILED
pytorch_transformers/tests/modeling_transfo_xl_test.py::TransfoXLModelTest::test_transfo_xl_model FAILED
pytorch_transformers/tests/tokenization_xlnet_test.py::XLNetTokenizationTest::test_tokenizer_no_lower PASSED

=================================== FAILURES ===================================
__________________ TransfoXLModelTest.test_attention_outputs ___________________

self = <pytorch_transformers.tests.modeling_transfo_xl_test.TransfoXLModelTest testMethod=test_attention_outputs>

    def test_attention_outputs(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

        for model_class in self.all_model_classes:
            config.output_attentions = True
            config.output_hidden_states = False
            model = model_class(config)
            model.eval()
>           outputs = model(**inputs_dict)

pytorch_transformers/tests/modeling_common_test.py:73:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
    outputs = self._forward(input_ids, mems=mems, head_mask=head_mask)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = TransfoXLModel(
  (word_emb): AdaptiveEmbedding(
    (emb_layers): ModuleList(
      (0): Embedding(10, 32)
      (1):... LayerNorm(torch.Size([32]), eps=1e-05, elementwise_affine=True)
      )
    )
  )
  (pos_emb): PositionalEmbedding()
)
dec_inp = tensor([[19, 69, 72, 42, 32, 34, 52, 38, 81, 71, 81, 47, 44],
        [22, 12,  3, 26, 63, 25, 64, 52, 79, 71, 17, 16,...   [82, 26, 62, 95, 55, 79,  8, 90, 33, 83, 64, 53, 68],
        [ 7, 57, 63, 40, 74, 77, 50, 77, 19,  7, 53, 38, 19]])
mems = [tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0.,...   [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]]])]
head_mask = [None, None, None, None, None]

    def _forward(self, dec_inp, mems=None, head_mask=None):
        qlen, bsz = dec_inp.size()

        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
        # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads] (a head_mask for each layer)
        # and head_mask is converted to shape [num_hidden_layers x qlen x klen x bsz x n_head]
        if head_mask is not None:
            if head_mask.dim() == 1:
                head_mask = head_mask.unsqueeze(0).unsqueeze(0).unsqueeze(0).unsqueeze(0)
                head_mask = head_mask.expand(self.n_layer, -1, -1, -1, -1)
            elif head_mask.dim() == 2:
                head_mask = head_mask.unsqueeze(1).unsqueeze(1).unsqueeze(1)
            head_mask = head_mask.to(dtype=next(self.parameters()).dtype) # switch to fload if need + fp16 compatibility
        else:
            head_mask = [None] * self.n_layer

        word_emb = self.word_emb(dec_inp)

        mlen = mems[0].size(0) if mems is not None else 0
        klen = mlen + qlen
        if self.same_length:
            all_ones = word_emb.new_ones(qlen, klen)
            mask_len = klen - self.mem_len
            if mask_len > 0:
                mask_shift_len = qlen - mask_len
            else:
                mask_shift_len = qlen
            dec_attn_mask = (torch.triu(all_ones, 1+mlen)
>                   + torch.tril(all_ones, -mask_shift_len)).bool()[:, :, None] # -1
E           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError

>           outputs = model(**inputs)

pytorch_transformers/tests/modeling_common_test.py:185:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
    outputs = self._forward(input_ids, mems=mems, head_mask=head_mask)
E           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError
_________________ TransfoXLModelTest.test_hidden_states_output _________________

self = <pytorch_transformers.tests.modeling_transfo_xl_test.TransfoXLModelTest testMethod=test_hidden_states_output>

    def test_hidden_states_output(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

        for model_class in self.all_model_classes:
            config.output_hidden_states = True
            config.output_attentions = False
            model = model_class(config)
            model.eval()
>           outputs = model(**inputs_dict)

pytorch_transformers/tests/modeling_common_test.py:249:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
    outputs = self._forward(input_ids, mems=mems, head_mask=head_mask)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError
__________________ TransfoXLModelTest.test_transfo_xl_lm_head __________________

self = <pytorch_transformers.tests.modeling_transfo_xl_test.TransfoXLModelTest testMethod=test_transfo_xl_lm_head>

    def test_transfo_xl_lm_head(self):
        self.model_tester.set_seed()
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
>       output_result = self.model_tester.create_transfo_xl_lm_head(*config_and_inputs)

pytorch_transformers/tests/modeling_transfo_xl_test.py:201:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pytorch_transformers/tests/modeling_transfo_xl_test.py:142: in create_transfo_xl_lm_head
    lm_logits_1, mems_1 = model(input_ids_1)
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1349: in forward
    transformer_outputs = self.transformer(input_ids, mems=mems, head_mask=head_mask)
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py:493: in __call__
    result = self.forward(*input, **kwargs)
pytorch_transformers/modeling_transfo_xl.py:1253: in forward
E           AttributeError: 'Tensor' object has no attribute 'bool'

pytorch_transformers/modeling_transfo_xl.py:1145: AttributeError
=============================== warnings summary ===============================
-- Docs: http://doc.pytest.org/en/latest/warnings.html
======= 5 failed, 206 passed, 10 skipped, 36 warnings in 171.71 seconds ========
`

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
Seamless execution!!!
## Environment
==============NVSMI LOG==============

Timestamp                           : Sat Aug 31 11:09:33 2019
Driver Version                      : 418.67
CUDA Version                        : 10.1

Attached GPUs                       : 1
GPU 00000000:00:04.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla

* OS:Ubuntu 18.04
* Python version:3.6
* PyTorch version:1.1.0
* PyTorch Transformers version (or branch):https://github.com/huggingface/pytorch-transformers
* Using GPU ?yes
* Distributed of parallel setup ?no
* Any other relevant information:

Why do these attribute errors occur?


How to add new pre-trained model pytorch-transformers

## ❓ Questions & Help

Pytorch-transformers is a great library. I like that it does one thing, give access to the pre-trained SOTA models for NLP.
I and my team we want to help and start contributing:
- at first, want to add Polish BERT model like https://github.com/huggingface/pytorch-transformers/pull/688

But we do not know how to do this :(
Is there any guide or procedure that shows what should be changed in order to add a new model?
We would be grateful if someone guides us.


ImportError: cannot import name 'DistilBertModel'

## 🐛 Bug

Can you update the pypi package? I cannot import DistilBERT on pytorch_transformers==1.1.0


Roberta for NER task

## ❓ Questions & Help

Hello, is there a way to use Roberta model for NER task? Is there a script somewhere?

Thank you.


Dependency errors when trying to use gpt2 using pytorch hub.

It started today, yesterday it was working fine. When I try to download `gpt2` model from pytorch hub repository, as follows:

```python
torch.hub.load('huggingface/pytorch-pretrained-BERT', 'gpt2Tokenizer', 'gpt2')
```

I get the following error: `ModuleNotFoundError: No module named 'sacremoses'`. If I add that dependency manually then I get another error: `ModuleNotFoundError: No module named 'sentencepiece'`. Then I add `sentencepiece` dependency manually just to get another error: `RuntimeError: Cannot find callable gpt2Tokenizer in hubconf`. And this error seems to be related to API changes.

I'm using a Google Colab GPU instance.

If this is not the right place to post this issue, please re-direct me to the proper source to post the issue. Thanks.


distillation: fix ModuleNotFoundError error in token counts script

Hi,

I'm currently trying out "distillation" 😅

This PR a `ModuleNotFoundError` message in the `token_counts.py` script (same error was recently fixed in 803c1cc4eacd38f1b854578d7d717b5e4a1ada47 🤗


[Help] how to make a constrained text generation

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
What I need is to make a constrained text generation via XLNet or GPT-2:

Input:  No one has the intention of building a wall.
Constraint: the output should include two pre-defined key words: 'No one' and 'construct'.
Expected output(e.g.): No one has the intention, a wall to construct.
(with a text length being predefined).

I found some reference like followings,
https://awslabs.github.io/sockeye/inference.html#lexical-constraints

but it is too complecate to transfer. Could u give me some advice?

thx a log!


XLNet bias fix on resize embeddings (cf #1124)

Fixed an issue where the linear layer bias wouldn't be resized along the weight resize when there was an embedding matrix resize with XLNet (cf #1124).

This fix works for any model that needs to tie its weights between an embedding layer & a linear layer if . that linear layer has a bias.


Large Memory Layers

## 🚀 Feature

Implement models with Large Memory Layers from this paper: https://arxiv.org/pdf/1907.05242.pdf

## Motivation

These models seem very promising.


--seed does not change the fintuning results of the xlnet model

## 🐛 Bug

<!-- Important information -->

Model I am using  (XLNet):

Language I am using the model on (English):

The problem arise when using:
* [ ] the official example scripts
gpu=3
seed=1
task=MRPC
bsz=32
learning_rate=5e-5
max_steps=800
warmup_steps=200
save_steps=400
export CUDA_VISIBLE_DEVICES=${gpu}
export GLUE_DIR=/home/zhaoguangxiang/bert/glue_data
python3 ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
    --do_eval   \
    --task_name=${task}     \
    --data_dir=${GLUE_DIR}/${task}  \
    --output_dir=checkpoint/xl_${task}_seed${seed}/   \
    --max_seq_length=128   \
    --per_gpu_eval_batch_size=${bsz}   \
    --per_gpu_train_batch_size=${bsz}   \
    --gradient_accumulation_steps=1 \
    --max_steps=${max_steps}  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --save_steps ${save_steps} \
    --learning_rate ${learning_rate} \
    --warmup_steps=${warmup_steps}
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)

## To Reproduce

Steps to reproduce the behavior:

1. change the seed
2. results does not change, and keep acc=0.8774509803921569 for every seed

## Environment

* OS:linux
* Python version:3.6
* PyTorch version: 1.2
* PyTorch Transformers version (or branch): latest
* Using GPU ?  1 * TITAN RTX


Problem with optimizers after migration

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): Russian

The problem arise when using:
The optimizers. I tried the default parameters from the example with max_grad_norm = 1.0 and lr = 2e-5.

```
warmup_proportion = float(num_warmup_steps) / float(num_total_steps)  # 0.1

### Previously BertAdam optimizer was instantiated like this:
optimizer = BertAdam(model.parameters(), lr=lr, schedule='warmup_linear', warmup=warmup_proportion, t_total=num_total_steps)
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    optimizer.step()

### In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
    scheduler.step()
    optimizer.step()
```

The tasks I am working on is:
* [ ] my own task or dataset: private dataset on comments from client support

Details of the issue:

After migration I see that the model converges more slowly and failed to obtain such an accuracy as it was before. In my multi-class classification dataset I get about 0.59 accuracy while the previous version resulted in about 0.63 accuracy after convergence. Do the optimizers in both versions are equivalent? If not, how can I make them absolutely the same?

## Environment

* OS: Ubuntu
* Python version: 3.6
* PyTorch version: 1.0
* PyTorch Transformers version (or branch): 1.0
* Using GPU ? Yes
* Distributed of parallel setup ? No


regarding #1026 pull request

Dear Thomas,
This is regarding my #1026 pull request, here is my understanding of  the reproducibility issue I was getting:

- on line 451,  in the codes tokenizer is reloaded without setting do_lower_case, then if you use both do_train+do_eval, you will get different results than if you do do_eval only on the same directory since if you use only do_eval only, tokenizer is read from line 408 where do_lower_case is considered.

- the second issue I see is that if you do both do_train and do_eval you read tokenizer from the output_dir, but if you do only do_eval you read tokenizer from args.model_name_or_path which can be different and could results in different results, so this is better to reload the tokenizer once from output_dir during the evaluation and remove it from training part.

thanks.
Best regards,
Rabeeh


How to load pretraind XLM model

## ❓ Questions & Help

Facebook recently released a new pre-trained language model (17 and 100) (https://github.com/facebookresearch/XLM#pretrained-cross-lingual-language-models)

I want to load 17 language model.

Could someone guide me on how to achieve this? The straightforward way didn't work for me.

I have downloaded the 3 files  listed on facebook GitHub:
- model - https://dl.fbaipublicfiles.com/XLM/mlm_17_1280.pth
- bpe codes - https://dl.fbaipublicfiles.com/XLM/codes_xnli_17
- vocabulary - https://dl.fbaipublicfiles.com/XLM/vocab_xnli_17

and saved them in folder '/home/ksopyla/xlm/mlm17l'. Then I have tried to load the model with XLMModel.form_pretrained function

`
model = XLMTokenizer.from_pretrained('/home/ksopyla/xlm/mlm17/')
`
got
`
Model name '/home/ksopyla/xlm/mlm17' was not found in model name list (xlm-mlm-en-2048, xlm-mlm-ende-1024, xlm-mlm-enfr-1024, xlm-mlm-enro-1024, xlm-mlm-tlm-xnli15-1024, xlm-mlm-xnli15-1024, xlm-clm-enfr-1024, xlm-clm-ende-1024). We assumed '/home/ksopyla/xlm/mlm17/config.json' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
  File "/home/ksopyla/.vscode/extensions/ms-python.python-2019.8.30787/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/home/ksopyla/.vscode/extensions/ms-python.python-2019.8.30787/pythonFiles/lib/python/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/ksopyla/.vscode/extensions/ms-python.python-2019.8.30787/pythonFiles/lib/python/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/ksopyla/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/ksopyla/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/ksopyla/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ksopyla/dev/document_embeddings/xlm_from_pretraind.py", line 34, in <module>
    output_hidden_states=True,
  File "/home/ksopyla/.local/share/virtualenvs/szrek-data-PaoX74GN/lib/python3.7/site-packages/pytorch_transformers/modeling_utils.py", line 430, in from_pretrained
    **kwargs
TypeError: cannot unpack non-iterable NoneType object
`

I suspect that I should change the file names and adjust the vocab file format. But I can't find it in the documentation.


About distilled the SQuAD?

## ❓ Questions & Help

Thank you for your excellent work. I want to know that have you released the distilled model code on SQuAD dataset? And how to set the max sequence length on teacher model and student model? Are they the same length?


Update apex fp16 implementation

As the issue I raised here: https://github.com/huggingface/pytorch-transformers/issues/1143
I updated the implementation of apex fp16, following latest Apex version's document.


fix: hard coding for max number

fp16 max number is 65504, the original 1e30 will cause Nan in fp16