Generated text makes no sense. Trying to auto-generate sentences like https://transformer.huggingface.co/doc/distil-gpt2

I would like to replicate the text-generation behavior of https://transformer.huggingface.co/doc/distil-gpt2 on my MacOS. I went to that URL and typed "A doula is a person" and kept tabbing for auto-text generation, and got this:
```
A doula is a person who cares for a child or infant. A doula is not a midwife. But the two are not entirely independent. Why did I chose a doula and not myself? There are two good reasons: First, since my daughter was still growing , I had the luxury of time to learn as much knowledge about breastfeeding as I could. Second, I have the opportunity
```

Quite impressive. But now, I want to replicate this in Python on my system so I can auto-generate text that makes sense. I did the following:
1. Installed tensor flow
2. installed pytorch
3. installed huggingface :D
4.  Based on the recommendation in the [Examples section here](https://huggingface.co/transformers/examples.html#language-generation), I then ran:
```
python run_generation.py --model_type=gpt2 --model_name_or_path=gpt2
```

and gave it the same sample input:
```
Model prompt >>> A doula
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.24it/s]
 hatter with a broad laugh and a guffaw over her stomach are eminently adorable. In
Model prompt >>> A doula is a person
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  8.94it/s]
 who wishes to be free and independent while having her resources donated to society.

Gender Non-

```
As you can see, I get gibberish back, the sentences and context makes no sense. What am I missing?


Is there a way of finetuning DistilGPT2?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello! I am trying to finetune GPT2 for a project of mine but my aim is to deploy the model on a server and hence, I would like the final model file to be small. I was thinking of using the Distil* models especially the DistilGPT2 but couldnt find it in the run_lm_finetuning.py script.

Currently, is finetuning distilGPT2 supported? If yes, then how should I do it?


CUDA out of memory on loss.backward when fine-tuning GPT2 (117M)

## ❓ Questions & Help

File "run_lm_finetuning.py", line 551, in <module>
    main()
  File "run_lm_finetuning.py", line 503, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 240, in train
    loss.backward()
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 150, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 198.00 MiB (GPU 0; 5.93 GiB total capacity; 4.64 GiB already allocated; 54.94 MiB free; 233.05 MiB cached)

I encounter the above error with my **1060 GTX 6GB Nvidia, on the GPT-2 small model. The training configs are:
batch size = 1
gradient accumulation steps = 1024** (I've started without gradient accumulation, then tried accumulation based on an old issue from this repo, then from small values I went up to this value, but the error always occurs).

**If i run with no gradient accumulation, I get this instead:**
 File "run_lm_finetuning.py", line 228, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 549, in forward
    inputs_embeds=inputs_embeds)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 460, in forward
    head_mask=head_mask[i])
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 232, in forward
    head_mask=head_mask)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 182, in forward
    x = self.c_attn(x)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/antonio/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 488, in forward
    x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.93 GiB total capacity; 4.77 GiB already allocated; 12.81 MiB free; 154.93 MiB cached)

Can you please give me a little hint on how to overcome this error, or a little hope to run gpt-2 small on 6 GB of GPU.

Thanks


Trouble running fine tuned language model script

## 🐛 Bug

I'm having trouble running the finetuning script. When I run

```

TRAIN_FILE='/home/khev/research/transformers/data/wikitext-2-raw/wiki.train.raw'
TEST_FILE='/home/khev/research/transformers/data/wikitext-2-raw/wiki.test.raw'

%run run_lm_finetuning.py \
    --output_dir=output \
    --model_type=bert \
    --model_name_or_path=bert-base-uncased\
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm

```

I get the following error:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/research/transformers/examples/run_lm_finetuning.py in <module>
    545
    546 if __name__ == "__main__":
--> 547     main()

~/research/transformers/examples/run_lm_finetuning.py in main()
    497             torch.distributed.barrier()
    498
--> 499         global_step, tr_loss = train(args, train_dataset, model, tokenizer)
    500         logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
    501

~/research/transformers/examples/run_lm_finetuning.py in train(args, train_dataset, model, tokenizer)
    222         epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
    223         for step, batch in enumerate(epoch_iterator):
--> 224             inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
    225             inputs = inputs.to(args.device)
    226             labels = labels.to(args.device)

~/research/transformers/examples/run_lm_finetuning.py in mask_tokens(inputs, tokenizer, args)
    147     probability_matrix = torch.full(labels.shape, args.mlm_probability)
    148     special_tokens_mask = [tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()]
--> 149     probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
    150     masked_indices = torch.bernoulli(probability_matrix).bool()
    151     labels[~masked_indices] = -1  # We only compute loss on masked tokens

RuntimeError: Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'
```

Info:

Python 3.7.3
Torch 1.1.0
Ubuntu 18.04.3 LTS


Using multiple inputs for GPT-2

## ❓ Questions & Help
A current project I am working on, to utilize a tuned approach in text generation, is creating a cover letter based off of a job description and some personal qualifications. From researching, I found the convAI blog that you used and thought to apply the concept with OpenAIGPTDoubleHeadsModel. I added special tokens to the vocabulary for delimiters and segment indicators to include word, position and segment embeddings. I have the persona ('some personal qualifications'), turned history into job description and the reply was "I am qualified for this position because...". From my understanding, the attention will help relate the job description and persona to the text generation.  Now I am stuck when it comes to fine tuning the model. I have a bunch of cover letters to use scraped from the web. I have fine tuned a base model and it outputs some coherent cover letters but I want to incorporate the skills needed in the job description for a more accurate an specific cover letter. Am I on the right track and if so how do I fine tune the model on the cover letters? Any suggestions or ideas are welcomed.
<!-- A clear and concise description of the question. -->


Best way to fine tune GPT-2 in order to create a custom text generator?

Hello to everyone, and thanks for this wonderful work.

I am new to this library and I would appreciate an help for a task that i want to accomplish, just to know if I am acting right, to create a custom english text generator, such that giving it an input (title/sentence) it would generate 200-300 words based on that input.

My questions are:
1) I have prepared my dataset (each input is composed basically by the title and the corpus), what file should I look for to fine-tune GPT-2: run_lm_finetuning.py ? How many epochs/iterations do you suggest to run to fine tune? How large the dataset should be?

2) Once I have fine-tuned GPT-2, how to generate my custom text giving as input a title/sentence and using the fine-tuned model?

Thanks a lot


computing self-attention for tokens in a sentence

## ❓ Questions & Help

Hi,

Please refer to this [issue](https://github.com/google-research/bert/issues/914).

Thanks!


Sample a constant number of tokens for masking in LM finetuning

@thomwolf @LysandreJik @julien-c Re-creating this PR as the original PR https://github.com/huggingface/transformers/pull/1555 that I created about a month ago has become quite stale. Please review.

For Masked LM fine-tuning, I think both the original BERT and RoBERTa implementations uniformly sample x number of tokens in *each* sequence for masking (where x = mlm_probability * 100 * sequence_length)
However, The current logic in run_lm_finetuning.py does an indepdendent sampling (from bernoulli distribution) for each token in the sequence. This leads to variance in the number of masked tokens (with the average number still close to x%).

The below example illustrates an extreme case, of the current logic, where no token in the input sequence is masked.

```
In [1]: import numpy as np
   ...: import torch
   ...: from transformers import BertTokenizer
   ...:
   ...: mlm_probability = 0.15
   ...: tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
   ...:
   ...: tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode('please mask me, o lord!', add_special_tokens=True))
   ...:
   ...: input_ids = tokenizer.convert_tokens_to_ids(tokens)
   ...:
   ...: inputs = torch.Tensor([input_ids])
   ...:
   ...: labels = inputs.clone()
   ...:
   ...: probability_matrix = torch.full(labels.shape, mlm_probability)
   ...:
   ...: special_tokens_mask = [tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()]
   ...: probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
   ...: masked_indices = torch.bernoulli(probability_matrix).bool()
   ...:
   ...:

In [2]: masked_indices
Out[2]: tensor([[False, False, False, False, False, False, False, False, False]])

```

This PR modifies the logic so the percentage of masked tokens is constant (at x).

Separately, the existing and the new masking logic both rely on boolean tensors of pytorch.
So, this also updates README to include the minimum pytorch version needed. (1.2.0)


LM Fine-tuning for XLNET?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi there, is there any plan to add support for fine-tuning XLNET? This one currently isn't available in run_lm_finetuning.py.

Thanks!


Update conversion script to convert XLM-R

It requires a few changes to the existing RoBERTa conversion script because `fairseq` has updated the RoBERTa model definition.

Specifically, the definition of multiheaded attention is changed at https://github.com/pytorch/fairseq/blob/2a9b4ec2374574cd0315f95e126788e4fe795f0d/fairseq/modules/multihead_attention.py#L42

With this new script, you should be able to convert the XLM-R model released at https://github.com/pytorch/fairseq/tree/master/examples/xlmr and requested by #1769.

I have successfully converted the two released model weights.


Fix special tokens addition in decoder #1807

Fixes the issue detailed in #1807

Added special tokens should not be present when decoding with the `skip_special_tokens` flag set to `True`.
Using the `convert_tokens_to_ids` checks whether those added tokens are present and removes them if necessary.


NameError: name 'DUMMY_INPUTS' is not defined - From TF to PyTorch

## 🐛 Bug

I'm using TFBertForSequenceClassification, Tensorflow 2.0.0b0, PyTorch is up-to-date and the code from Hugging Face README.md:

```
import tensorflow as tf
import tensorflow_datasets
from transformers import *

tf.compat.v1.enable_eager_execution()

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

model.save_pretrained('/home/rubens_gmail_com/tf')
```
The model trains properly, but after weights are saved as `.h5` and I try to load to `BertForSequenceClassification.from_pretrained`, the following error shows up:

```
pytorch_model = BertForSequenceClassification.from_pretrained('/home/rubens_gmail_com/tf',from_tf=True)

NameError                                 Traceback (most recent call last)
<ipython-input-25-7a878059a298> in <module>
      1 pytorch_model = BertForSequenceClassification.from_pretrained('/home/rubens_gmail_com/tf',
----> 2                                                               from_tf=True)

~/.local/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    357                 try:
    358                     from transformers import load_tf2_checkpoint_in_pytorch_model
--> 359                     model = load_tf2_checkpoint_in_pytorch_model(model, resolved_archive_file, allow_missing_keys=True)
    360                 except ImportError as e:
    361                     logger.error("Loading a TensorFlow model in PyTorch, requires both PyTorch and TensorFlow to be installed. Please see "

~/.local/lib/python3.7/site-packages/transformers/modeling_tf_pytorch_utils.py in load_tf2_checkpoint_in_pytorch_model(pt_model, tf_checkpoint_path, tf_inputs, allow_missing_keys)
    199
    200     if tf_inputs is None:
--> 201         tf_inputs = tf.constant(DUMMY_INPUTS)
    202
    203     if tf_inputs is not None:

NameError: name 'DUMMY_INPUTS' is not defined
```

If I run:
```
pytorch_model = BertForSequenceClassification.from_pretrained('/home/rubens_gmail_com/tf/tf_model.h5',from_tf=True)
```
I get:

```
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-35-4146e5f25140> in <module>
----> 1 pytorch_model = BertForSequenceClassification.from_pretrained('/home/rubens_gmail_com/tf/tf_model.h5',from_tf=True)

~/.local/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    285                 cache_dir=cache_dir, return_unused_kwargs=True,
    286                 force_download=force_download,
--> 287                 **kwargs
    288             )
    289         else:

~/.local/lib/python3.7/site-packages/transformers/configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    152
    153         # Load config
--> 154         config = cls.from_json_file(resolved_config_file)
    155
    156         if hasattr(config, 'pruned_heads'):

~/.local/lib/python3.7/site-packages/transformers/configuration_utils.py in from_json_file(cls, json_file)
    184         """Constructs a `BertConfig` from a json file of parameters."""
    185         with open(json_file, "r", encoding='utf-8') as reader:
--> 186             text = reader.read()
    187         return cls.from_dict(json.loads(text))
    188

/opt/anaconda3/lib/python3.7/codecs.py in decode(self, input, final)
    320         # decode input (taking the buffer into account)
    321         data = self.buffer + input
--> 322         (result, consumed) = self._buffer_decode(data, self.errors, final)
    323         # keep undecoded input until the next call
    324         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

```

I'm using Python 3.7.4, 8 x V100 GPU, Anaconda environment on a Debian on GCP. Any ideas on how to overcome this issue ?


Why do language modeling heads not have activation functions?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

This is more of a question about the transformer architecture in general than anything else. I noticed that, in `modeling_openai.py`, for example, the `self.lm_head()` module is just a linear layer. Why is it sufficient to use a linear transformation here for the language modeling task? Would there by any advantages/disadvantages to throwing a `gelu` or something on the end and then calling the output of that the logits?

```python
class OpenAIGPTLMHeadModel(OpenAIGPTPreTrainedModel):
    r"""
        **labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
            Labels for language modeling.
            Note that the labels **are shifted** inside the model, i.e. you can set ``labels = input_ids``
            Indices are selected in ``[-1, 0, ..., config.vocab_size]``
            All labels set to ``-1`` are ignored (masked), the loss is only
            computed for labels in ``[0, ..., config.vocab_size]``
    Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs:
        **loss**: (`optional`, returned when ``labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``:
            Language modeling loss.
        **prediction_scores**: ``torch.FloatTensor`` of shape ``(batch_size, sequence_length, config.vocab_size)``
            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
        **hidden_states**: (`optional`, returned when ``config.output_hidden_states=True``)
            list of ``torch.FloatTensor`` (one for the output of each layer + the output of the embeddings)
            of shape ``(batch_size, sequence_length, hidden_size)``:
            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
        **attentions**: (`optional`, returned when ``config.output_attentions=True``)
            list of ``torch.FloatTensor`` (one for each layer) of shape ``(batch_size, num_heads, sequence_length, sequence_length)``:
            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
    Examples::
        tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
        model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids, labels=input_ids)
        loss, logits = outputs[:2]
    """
    def __init__(self, config):
        super(OpenAIGPTLMHeadModel, self).__init__(config)
        self.transformer = OpenAIGPTModel(config)
        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)

        self.init_weights()

    def get_output_embeddings(self):
        return self.lm_head

    def forward(self, input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None,
                labels=None):
        transformer_outputs = self.transformer(input_ids,
                                               attention_mask=attention_mask,
                                               token_type_ids=token_type_ids,
                                               position_ids=position_ids,
                                               head_mask=head_mask,
                                               inputs_embeds=inputs_embeds)
        hidden_states = transformer_outputs[0]
        lm_logits = self.lm_head(hidden_states)

        outputs = (lm_logits,) + transformer_outputs[1:]
        if labels is not None:
            # Shift so that tokens < n predict n
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = labels[..., 1:].contiguous()
            # Flatten the tokens
            loss_fct = CrossEntropyLoss(ignore_index=-1)
            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
                            shift_labels.view(-1))
            outputs = (loss,) + outputs

        return outputs  # (loss), lm_logits, (all hidden states), (all attentions)
```


XLMForSequenceClassification - help with zero-shot cross-lingual classification

## ❓ Questions & Help
Hi guys
According to XLM description (https://github.com/facebookresearch/XLM?fbclid=IwAR0-ZJpmWmIVfR20fA2KCHgrUU3k0cMUyx2n_V9-9C8g857-nhavrfBnVSI#pretrained-cross-lingual-language-models), we could potentially do XLNI by training in `en` dataset and do inference in other language:
```
XLMs can be used to build cross-lingual classifiers. After fine-tuning an XLM model
on an English training corpus for instance (e.g. of sentiment analysis, natural language
inference),  the model is still able to make accurate predictions at test time in other
languages, for which there is very little or no training data.  This approach is usually
referred to as "zero-shot cross-lingual classification".
```

I'm doing some tests w/ a dataset here at work, and I'm able to train the model with *XLMForSequenceClassification* and when I test it in `en`, performance looks great!
However, when I try to pass another language and do some inference, performance on other language (the so-called zero shot XLNI) does not look fine.

Here are the configurations I'm using:
```python
config = XLMConfig.from_pretrained('xlm-mlm-tlm-xnli15-1024')
config.num_labels = len(list(label_to_ix.values()))
config.n_layers = 6
tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-tlm-xnli15-1024')
model = XLMForSequenceClassification(config)
```
I trained the forward pass passing the `lang` vector with `en` language:
```python
langs = torch.full(ids.shape, fill_value = language_id, dtype = torch.long).cuda()
output = model.forward(ids.cuda(), token_type_ids=tokens.cuda(), langs=langs, head_mask=None)[0]
```
For inference, I'm passing now other languages, by using following function:
```python
def get_reply(msg, language = 'en'):
    features = prepare_features(msg, zero_pad = False)
    language_id = tokenizer.lang2id[language]
    ids = torch.tensor(features['input_ids']).unsqueeze(0).cuda()
    langs = torch.full(ids.shape, fill_value = language_id, dtype = torch.long).cuda()
    tokens = torch.tensor(features['token_type_ids']).unsqueeze(0).cuda()
    output = model.forward(ids,token_type_ids=tokens, langs=langs)[0]
    _, predicted = torch.max(output.data, 1)
    return list(label_to_ix.keys())[predicted]
```
BTW, my `prepare_features` function is quite simple, just use the `encode_plus` method and then zero pad for a given sentence size:
```python
def prepare_features(seq_1, zero_pad = False, max_seq_length = 120):
    enc_text = tokenizer.encode_plus(seq_1, add_special_tokens=True, max_length=300)
    if zero_pad:
        while len(enc_text['input_ids']) < max_seq_length:
            enc_text['input_ids'].append(0)
            enc_text['token_type_ids'].append(0)
    return enc_text
```
I've noticed in file (https://github.com/huggingface/transformers/blob/master/transformers/modeling_xlm.py) that the forward loop for `XLMModel` does add `lang` vector as sort of an offset for the embeddings:
```python
tensor = tensor + self.lang_embeddings(langs)
```
I'm wondering if I'm passing the `lang` vector in right format. I choose the model with *tlm* language model on purpose because I was guessing that it benefits of the translation language model of the XLM. Have you guys experiment already with zero-shot text classification? Any clue my inference in other languages is not working?
Any insight will be greatly appreciated!
Cheers,
Roberto


Whether it belongs to the bug of class trainedtokenizer decode?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

![1](https://user-images.githubusercontent.com/16183570/68684557-a4bed780-05a3-11ea-8d06-1c5638be6ac6.png)
![2](https://user-images.githubusercontent.com/16183570/68684572-abe5e580-05a3-11ea-9620-847264507fae.png)


Extracting First Hidden States

## ❓ Questions & Help

For example, right now in order to extract the first hidden states of DistilBert model,

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased', **output_hidden_states=True**)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
**first_hidden_states = outputs[1][0]  # The first hidden-state**

However, I can only extract the first layer hidden states after running through the entire model; all 7 layers (which is unnecessary if I only want the first).

Is there a way I could use the model to only extract the hidden states of the first layer and stop there? I am looking to further optimize the inference time of DistilBert.

Thank you :)


## ❓ Questions & Help
![image](https://user-images.githubusercontent.com/33107884/68678361-c5356480-0598-11ea-8e2b-c3b419c6af52.png). "RuntimeError: CUDA error: device-side assert triggered" occurs. My model is as follows:
```
class TextClassify(nn.Module):
    def __init__(self, bert, kernel_size, word_dim, out_dim, num_class, dropout=0.5):
        super(TextClassify, self).__init__()
        self.bert = bert
        self.cnn = nn.Sequential(nn.Conv1d(word_dim, out_dim, kernel_size=kernel_size, padding=1), nn.ReLU(inplace=True))
        self.drop = nn.Dropout(dropout, inplace=True)
        self.fc = nn.Linear(out_dim, num_class)

    def forward(self, word_input):
        batch_size = word_input.size(0)
        represent = self.xlnet(word_input)[0]
        represent = self.drop(represent)
        represent = represent.transpose(1, 2)
        contexts = self.cnn(represent)
        feature = F.max_pool1d(contexts, contexts.size(2)).contiguous().view(batch_size, -1)
        feature = self.drop(feature)
        feature = self.fc(feature)
        return feature
```
How can I solve it?


fix multi-gpu eval in torch examples

Although batch_size for eval is updated to include multiple GPUs, DataParallel is missing from the model and hence doesn't use multi-GPUs. This PR allows DataParallel (multi-GPU) model in eval.


fix run_squad.py during fine-tuning xlnet on squad2.0

The following is a piece of code in forward function of xlnet model, which obviously is the key point of training the model on unanswerable questions using cls token representations.  But the default value of tensor `is_impossible`(using to indicate whether this example is answerable) is none, and we also hadn't passed this tensor into forward function. That's the problem.

```
 if cls_index is not None and is_impossible is not None:
                # Predict answerability from the representation of CLS and START
                cls_logits = self.answer_class(hidden_states, start_positions=start_positions, cls_index=cls_index)
                loss_fct_cls = nn.BCEWithLogitsLoss()
                cls_loss = loss_fct_cls(cls_logits, is_impossible)
                total_loss += cls_loss * 0.5
```

I added the `is_impossible` tensor to TensorDataset and model inputs, and got a reasonable result,

{
"exact": 80.4177545691906,
"f1": 84.07154997729623,
"total": 11873,
"HasAns_exact": 77.59784075573549,
"HasAns_f1": 84.83993323200234,
"HasAns_total": 5928,
"NoAns_exact": 84.0874684608915,
"NoAns_f1": 84.0874684608915,
"NoAns_total": 5945
}

My running command:
```
python run_squad.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train \
    --do_eval \
    --version_2_with_negative \
    --train_file data/train-v2.0.json \
    --predict_file data/dev-v2.0.json \
    --learning_rate 3e-5 \
    --num_train_epochs 4 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ./wwm_cased_finetuned_squad/ \
    --per_gpu_eval_batch_size=2  \
    --per_gpu_train_batch_size=2   \
    --save_steps 5000
```
I run my code with 4 nvidia GTX 1080ti gpus and pytorch==1.2.0.


pip cannot install transformers with python version 3.8.0

## ❓ Questions & Help

    The error message looks like this,

`    ERROR: Command errored out with exit status 1:
     command: 'c:\users\enderaoe\appdata\local\programs\python\python38-32\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Enderaoe\\AppData\\Local\\Temp\\pip-install-g7jzfokt\\sentencepiece\\setup.py'"'"'; __file__='"'"'C:\\Users\\Enderaoe\\AppData\\Local\\Temp\\pip-install-g7jzfokt\\sentencepiece\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\Enderaoe\AppData\Local\Temp\pip-install-g7jzfokt\sentencepiece\pip-egg-info'
         cwd: C:\Users\Enderaoe\AppData\Local\Temp\pip-install-g7jzfokt\sentencepiece\
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Enderaoe\AppData\Local\Temp\pip-install-g7jzfokt\sentencepiece\setup.py", line 29, in <module>
        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
      File "c:\users\enderaoe\appdata\local\programs\python\python38-32\lib\codecs.py", line 905, in open
        file = builtins.open(filename, mode, buffering)
    FileNotFoundError: [Errno 2] No such file or directory: '..\\VERSION'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.`

    Should I lower my python version, or there is any other solution?


run_glue.py RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:3

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)  : transformers/examples/run_glue.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) :  MRPC
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
I've tested using
python -m pytest -sv ./transformers/tests/
python -m pytest -sv ./examples/
and it works fine without couple of tesks.

2.
after test, i downloaded glue datafile via
https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e
and tried run_glue.py

pip install -r ./examples/requirements.txt
export GLUE_DIR=/path/to/glue
export TASK_NAME=MRPC

3.
python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/$TASK_NAME/

and i got this error.

`11/11/2019 21:10:50 - INFO - __main__ -     Total optimization steps = 345
Epoch:   0%|                                                                                | 0/3 [00:00<?, ?it/sTraceback (most recent call last):                                                             | 0/115 [00:00<?, ?it/s]
  File "./examples/run_glue.py", line 552, in <module>
    main()
  File "./examples/run_glue.py", line 503, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "./examples/run_glue.py", line 146, in train
    outputs = model(**inputs)
  File "/home/insublee/anaconda3/envs/py_torch4/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/insublee/anaconda3/envs/py_torch4/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 146, in forward
    "them on device: {}".format(self.src_device_obj, t.device))
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:3`
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: ubuntu16.04LTS
* Python version:                                                      3.7.5
* PyTorch version:                                                     1.2.0
* PyTorch Transformers version (or branch):             2.1.1
* Using GPU ?                                                           4-way 2080ti
* Distributed of parallel setup ?                                cuda10.0 cudnn 7.6.4
* Any other relevant information:

## Additional context
thank you.


Exact and F1 score do not increase when fine-tunes XLM on the SQuAD dataset

## ❓ Questions & Help

I am trying to fine-tune XLM on the SQuAD dataset.
The command is as following:
[CUDA_VISIBLE_DEVICES=0 python run_squad.py --model_type xlm --model_name_or_path xlm-mlm-tlm-xnli15-1024 --do_train --do_eval --train_file $SQUAD_DIR/train-v1.1.json --predict_file $SQUAD_DIR/dev-v1.1.json --per_gpu_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 100 --max_seq_length 384 --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training --output_dir /home/weihua/Sqad/transformers/xlm_out/ ]
![WeChat Image_20191112113346](https://user-images.githubusercontent.com/43492059/68640277-2da52700-0542-11ea-981b-c748e918e1d8.png)

All the parameters I set are in accordance with the example in the pytorch-transformers document. But Exact and F1 scores have barely increased. And the loss of each epoch also drops very slowly. Each epoch drops by about 0.01.
![68477603-d6b10080-0268-11ea-947f-995b157da8d3](https://user-images.githubusercontent.com/43492059/68640296-3f86ca00-0542-11ea-93b3-db1bd64539b2.png)

Is there a problem with my parameter settings? Or do I need to adjust some parts of the model when I use XLM?


## 🐛 Bug

<!-- Important information -->

Model I am using XLM:

Language I am using the model on English:

The problem arise when using:
 [CUDA_VISIBLE_DEVICES=0 python run_squad.py   --model_type xlm   --model_name_or_path xlm-mlm-tlm-xnli15-1024   --do_train   --do_eval   --train_file $SQUAD_DIR/train-v1.1.json   --predict_file $SQUAD_DIR/dev-v1.1.json   --per_gpu_train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 100   --max_seq_length 384   --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training  --output_dir /home/weihua/Sqad/transformers/xlm_out/ ]
![WeChat Image_20191112113346](https://user-images.githubusercontent.com/43492059/68639732-752ab380-0540-11ea-8543-7279a40d553a.png)

The tasks I am working on is an official SQUaD task.

All the parameters I set are in accordance with the example in the pytorch-transformers document. But Exact and F1 scores have barely increased. And the loss of each epoch also drops very slowly. Each epoch drops by about 0.01.
![68477603-d6b10080-0268-11ea-947f-995b157da8d3](https://user-images.githubusercontent.com/43492059/68640118-b96a8380-0541-11ea-8ac1-8e0e72cdc001.png)

Is there a problem with my parameter settings? Or do I need to adjust some parts of the model when I use XLM?


Add an LSTM and CNN layer on top of BERT embeddings for sentiment analysis task

I am trying to add an LSTM and a convolutional layer on top of my BERT embeddings using the Transformers package in Tensorflow for a sentiment analysis task. Does someone know how I can go about that?


TF: model forwards can take an inputs_embeds param

see https://github.com/huggingface/transformers/pull/1695 (non-TF)


Fix GPT2LMHeadModel.from_pretrained(from_tf=True)

GPT2LMHeadModel.from_pretrained(from_tf=True) doesn't work because pointer points to the GPT2LMHeadModel instance, not the GPT2Model instance.

This bug causes errors like:

'GPT2LMHeadModel' object has no attribute 'h'


RuntimeError: Connection timed out in Single node Multi GPU training

I am trying to pre-train DistilBERT with single node multigpu as given here https://github.com/huggingface/transformers/tree/master/examples/distillation.

I have set the IP address of my gcp instance and port number. I am getting this error. Any solutions?

> File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 400, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: Connection timed out


Confused by GPT2DoubleHeadsModel example

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Your library is really helpful. And I have 2 questions about the example of GPT2DoubleHeadsModel.

1.
In source code of model,

> choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]

Here the comment said that "number of choices: 2". So is it a single sample or two samples? What are the two label classes?

2.
What task is the GPT2DOUBLEHeadsModel trained on? I don't find the information from the document, issues or the original paper.

Please help me. Thank you in advance.


MNLI: BERT No Training Progress

## 🐛 Bug

I am using BERT and have successfully set up pipelines for 7/8 GLUE tasks, and I find comparably good accuracy on all of them. However, for MNLI task, training loss does not converge at all. I am correctly using 3 classes with `num_classes`. In fact, I have even tried reduced the scope of MNLI to a 2-class problem (entailment vs neutral/contradiction) for testing purposes to see if this is the issue, and the model fails to converge on this training task as well. However, the exact same code (with only the dataset switched) works for MRPC, for example.

Model I am using (Bert, XLNet....): **BERT**

Language I am using the model on (English, Chinese....): **English**

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: **see co-lab**

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: **MNLI**
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

See linked co-lab for minimum viable example
https://colab.research.google.com/drive/1Thdrpvp0uX2TCaCpqsLoUVn7DuRLlILQ

## Expected behavior

I expect to see good training progress; instead, training loss does not converge.

## Environment

* OS: **Linux**
* Python version: **3.6**
* PyTorch version: **1.3.0**
* PyTorch Transformers version (or branch): **2.1.1**
* Using GPU ? **Yes**
* Distributed of parallel setup ? **No, single GPU**
* Any other relevant information: **None**


DistilBERT for token classification

Hi,

this PR adds a `DistilBertForTokenClassification` implementation (mainly inspired by the BERT implementation) that allows to perform sequence labeling tasks like NER or PoS tagging.

Additionally, the `run_ner.py` example script was modified to fully support DistilBERT for NER tasks.

I did a small comparison between BERT (large, cased), RoBERTa (large, cased) and DistilBERT (base, uncased) with the same hyperparameters as specified in the [example documentation](https://huggingface.co/transformers/examples.html#named-entity-recognition) (one run):

| Model | F-Score Dev | F-Score Test
| --------------------------------- | ------- | --------
| `bert-large-cased`            | 95.59 | 91.70
| `roberta-large`                  | 95.96 | 91.87
| `distilbert-base-uncased` | 94.34 | 90.32

Unit test for the `DistilBertForTokenClassification` implementation is also added.


## ❓ Questions & Help

When I use Bert, the "token indices sequence length is longer than the specified maximum sequence length for this model (1017 > 512)" occurs. How can I solve this error?


transformers vs pytorch_pretrained_bert giving different scores for BertForNextSentencePrediction

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM,BertForNextSentencePrediction
tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')
BertNSP=BertForNextSentencePrediction.from_pretrained('bert-base-uncased')

text1 = "How old are you?"
text2 = "The Eiffel Tower is in Paris"

text1_toks = ["[CLS]"] + tokenizer.tokenize(text1) + ["[SEP]"]
text2_toks = tokenizer.tokenize(text2) + ["[SEP]"]
text=text1_toks+text2_toks
indexed_tokens = tokenizer.convert_tokens_to_ids(text)
segments_ids = [0]*len(text1_toks) + [1]*len(text2_toks)

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
BertNSP.eval()
prediction = BertNSP(tokens_tensor, segments_tensors)
prediction=prediction[0] # tuple to tensor
print(predictions)
softmax = torch.nn.Softmax(dim=1)
prediction_sm = softmax(prediction)
print (prediction_sm)

output:
tensor([[ 2.1772, -0.8097]], grad_fn=)
tensor([[0.9923, 0.0077]], grad_fn=)

Now when trying with pytorch_pretrained_bert :

!pip install pytorch_pretrained_bert
import torch
import pytorch_pretrained_bert
from pytorch_pretrained_bert import BertTokenizer, BertAdam, BertForNextSentencePrediction

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')

text1 = "How old are you?"
text2 = "The Eiffel Tower is in Paris"

text1_toks = ["[CLS]"] + tokenizer.tokenize(text1) + ["[SEP]"]
text2_toks = tokenizer.tokenize(text2) + ["[SEP]"]
text=text1_toks+text2_toks
print(text)
indexed_tokens = tokenizer.convert_tokens_to_ids(text)
segments_ids = [0]*len(text1_toks) + [1]*len(text2_toks)

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
print(indexed_tokens)
print(segments_ids)

model.eval()
prediction = model(tokens_tensor, segments_tensors)

print(prediction)
softmax = torch.nn.Softmax(dim=1)
prediction_sm = softmax(prediction)
print (prediction_sm)

output:
tensor([[-2.3808, 5.4018]], grad_fn=)
tensor([[4.1673e-04, 9.9958e-01]], grad_fn=)

1. which is the correct score here ? is the softmax score from transformers  model correct or pytorch_pretrained_bert  model correct ?

2.Also the output of model in pytorch_pretrained_bert is tensor but output of model in transformers is tuple .Why again is it like this ?


BertForMultipleChoice QuickTour issue with weights?

## 🐛 Bug

<!-- Important information -->

Model I am using (BertForMultipleChoice):

Language I am using the model on (English.):

The problem arise when using:
* [x] the official example scripts:
Arises when running through the last piece of example code found here:
https://github.com/huggingface/transformers#quick-tour
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Run the sample code from the last loop of the Quick Tour
2. Need to supply directory names, see my code (below) for how I did that...
3. When BertForMultipleChoice runs the line `reshaped_logits = logits.view(-1, num_choices)` in modeling_bert.py we get a runtime error `RuntimeError: shape '[-1, 16]' is invalid for input of size 1`

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
# Each architecture is provided with several class for fine-tuning on down-stream tasks, e.g.
BERT_MODEL_CLASSES = [BertModel, BertForPreTraining, BertForMaskedLM, BertForNextSentencePrediction,
                      BertForSequenceClassification, BertForMultipleChoice, BertForTokenClassification,
                      BertForQuestionAnswering]
```
```
# All the classes for an architecture can be initiated from pretrained weights for this architecture
# Note that additional weights added for fine-tuning are only initialized
# and need to be trained on the down-stream task
pretrained_weights = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(pretrained_weights)
for model_class in BERT_MODEL_CLASSES:

    print("Processing", model_class.__name__, "...")

    # Store class name as target directory
    model_dir_name = model_class.__name__+"/"

    # Load pretrained model/tokenizer
    model = model_class.from_pretrained(pretrained_weights)

    # Models can return full list of hidden-states & attentions weights at each layer
    model = model_class.from_pretrained(pretrained_weights,
                                        output_hidden_states=True,
                                        output_attentions=True)
    input_ids = torch.tensor([tokenizer.encode("Let's see all hidden-states and attentions on this text")])
    all_hidden_states, all_attentions = model(input_ids)[-2:]

    # Models are compatible with Torchscript
    model = model_class.from_pretrained(pretrained_weights, torchscript=True)
    traced_model = torch.jit.trace(model, (input_ids,))

    save_directory = 'BERT_test/'+ model_dir_name
    if not os.path.isdir(save_directory):
        os.mkdir(save_directory)

    # Simple serialization for models and tokenizers
    model.save_pretrained(save_directory)  # save
    model = model_class.from_pretrained(save_directory)  # re-load
    tokenizer.save_pretrained(save_directory)  # save
    tokenizer = BertTokenizer.from_pretrained(save_directory)  # re-load

    # SOTA examples for GLUE, SQUAD, text generation...
```
The error:
```
I1111 20:47:30.383128 21676 modeling_utils.py:383] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at C:\Users\User\.cache\torch\transformers\aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
I1111 20:47:33.363116 21676 modeling_utils.py:453] Weights of BertForMultipleChoice not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
I1111 20:47:33.365122 21676 modeling_utils.py:456] Weights from pretrained model not used in BertForMultipleChoice: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-da6597bd484d> in <module>()
     19                                         output_attentions=True)
     20     input_ids = torch.tensor([tokenizer.encode("Let's see all hidden-states and attentions on this text")])
---> 21     all_hidden_states, all_attentions = model(input_ids)[-2:]
     22
     23     # Models are compatible with Torchscript

G:\Anaconda3\envs\pytorch1\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

g:\deeplearning\huggingface\transformers\transformers\modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels)
   1096         pooled_output = self.dropout(pooled_output)
   1097         logits = self.classifier(pooled_output)
-> 1098         reshaped_logits = logits.view(-1, num_choices)
   1099
   1100         outputs = (reshaped_logits,) + outputs[2:]  # add hidden states and attention if they are here

RuntimeError: shape '[-1, 16]' is invalid for input of size 1

```
## Expected behavior
Should run without error!

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows 10
* Python version: 3.6.6
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


BertForNextSentencePrediction is giving high score for non similar sentences .

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM,BertForNextSentencePrediction
tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')
BertNSP=BertForNextSentencePrediction.from_pretrained('bert-base-uncased')

text1 = "How old are you?"
text2 = "The Eiffel Tower is in Paris"

text1_toks = ["[CLS]"] + tokenizer.tokenize(text1) + ["[SEP]"]
text2_toks = tokenizer.tokenize(text2) + ["[SEP]"]
text=text1_toks+text2_toks
print(text)
indexed_tokens = tokenizer.convert_tokens_to_ids(text1_toks + text2_toks)
segments_ids = [0]*len(text1_toks) + [1]*len(text2_toks)

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
print(indexed_tokens)
print(segments_ids)
BertNSP.eval()
prediction = BertNSP(tokens_tensor, segments_tensors)
prediction=prediction[0]  # tuple to tensor
print(predictions)
softmax = torch.nn.Softmax(dim=1)
prediction_sm = softmax(prediction)
print (prediction_sm)

o/p of predictions
tensor([[ 2.1772, -0.8097]], grad_fn=<AddmmBackward>)

o/p of prediction_sm
tensor([[0.9923, 0.0077]], grad_fn=<SoftmaxBackward>)

why is the score still high 0.9923 even after apply softmax ?


Invalid argument with CTRLModel

## 🐛 Bug

<!-- Important information -->

Model I am using (CTRLModel):

Language I am using the model on (English):

The problem arise when using:
* [X ] the official example scripts: (give details)

The tasks I am working on is:
* [X ] the Quick Tour for transformers

## To Reproduce

Steps to reproduce the behavior:

1. run the quick tour as found here https://github.com/huggingface/transformers#quick-tour
2. in the loop `for model_class, tokenizer_class, pretrained_weights in MODELS` it will crash out when processing the CTRLModel
```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-5-25349833af91> in <module>
      5
      6     tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
----> 7     model = model_class.from_pretrained(pretrained_weights)
      8
      9     # Encode text

g:\deeplearning\huggingface\transformers\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    389
    390         if state_dict is None and not from_tf:
--> 391             state_dict = torch.load(resolved_archive_file, map_location='cpu')
    392
    393         missing_keys = []

G:\Anaconda3\envs\fastai\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    384         f = f.open('rb')
    385     try:
--> 386         return _load(f, map_location, pickle_module, **pickle_load_args)
    387     finally:
    388         if new_fd:

G:\Anaconda3\envs\fastai\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    578     for key in deserialized_storage_keys:
    579         assert key in deserialized_objects
--> 580         deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
    581         if offset is not None:
    582             offset = f.tell()

OSError: [Errno 22] Invalid argument

```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## What should have happened
Should have run to completion...

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version: 3.3.9
[EDIT] - type on the Python, its 3.6.9.
* PyTorch version: 1.2.0+cu92
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? Yes
* Distributed of parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


a BertForMaskedLM.from_pretrained error

## 🐛 Bug

<!-- Important information -->

Model I am using (BertForMaskedLM):

Language I am using the model on (Chinese):

When i want to do a limit vocabulary fine-tune in BerForMaskedLM, the new config/ vocab and model.bin in word_embedding are all been change.

for example:

`config = BertConfig.from_pretrained('bert-base-chinese')`
`config.vocab_size = 2`
`model_dict = torch.load(old_model_path)`
`model_dict['bert.embeddings.word_embeddings.weight'] = model_dict['bert.embeddings.word_embeddings.weight'][:2]`
`torch.save(model_dict, new_model_path)`

`model = BertForMaskedLM.from_pretrained(new_model_path, config=config)`

But when i use the from_pretrained code for the model, an errors has appeared:

` Error(s) in loading state_dict for BertForMaskedLM:
	size mismatch for cls.predictions.bias: copying a param with shape torch.Size([21128]) from checkpoint, the shape in current model is torch.Size([2]).
	size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([21128, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).`

And i have try to use the BertModel.from_pretrained, the error does not appear, and use the code to check the data shape.

`torch.load(new_model_path)['bert.embeddings.word_embeddings.weight'].shape `
it appear   torch.Size([2, 768])


"Write with Transformer" source code?

## ❓ Questions & Help
Hello, can't find source code of it. May you help, please?
<!-- A clear and concise description of the question. -->


Unclear documentation for special_tokens_mask

According to [the docs](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.get_special_tokens_mask), the `special_tokens_mask` returned by e.g. `encode_plus` should have

> 0 for a special token, 1 for a sequence token

Yet when I try

```python
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer.encode_plus("This is a test sentence", add_special_tokens=True)
```

I get

```python
{'special_tokens_mask': [1, 0, 0, 0, 0, 0, 1],
 'input_ids': [101, 2023, 2003, 1037, 3231, 6251, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0]}
```

Which — to me — would suggest that it is in fact 1 for special tokens and 0 for sequence tokens.
The implementations for [BERT](https://github.com/huggingface/transformers/blob/700331b5ece63381ad1b775fc8661cf3ae4493fd/transformers/tokenization_bert.py#L210) and [RoBERTa](https://github.com/huggingface/transformers/blob/1c542df7e554a2014051dd09becf60f157fed524/transformers/tokenization_roberta.py#L110) support this, so should the documentation be made clearer or is it just me who understood it backward?


How to measure similarity of words?

I want to know it BERT based contextual embedding can be used to measure similarity of identical words in different contexts.
And, can I estimate a threshold for that.


model = GPT2LMHeadModel.from_pretrained(args.model_path) try loads in json format

## 🐛 Bug

Initial a model in this code:

```
logging.info('loading model from: {}'.format(args.model_path))
    model = GPT2LMHeadModel.from_pretrained(args.model_path)
```

It seems inside transforms reading our model in json format which cause error:

```
INFO 11-10 16:18:49 generate.py:167 - loading model from: ./weights/ProseInChinese/pytorch_model.bin
I1110 16:18:49.226168 139982371047232 configuration_utils.py:148] loading configuration file ./weights/ProseInChinese/pytorch_model.bin
Traceback (most recent call last):
  File "generate.py", line 225, in <module>
    main()
  File "generate.py", line 168, in main
    model = GPT2LMHeadModel.from_pretrained(args.model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 287, in from_pretrained
    **kwargs
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 154, in from_pretrained
    config = cls.from_json_file(resolved_config_file)
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 186, in from_json_file
    text = reader.read()
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
```


Dose the file /examples/run_lm_finetuning.py provide a demo to pre-train a BERT

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I'm curious that if the file run_lm_finetuning.py can pre-train a BERT. But if so, why the file named finetuning instead of pre-training? Thanks!


Problems when restoring the pretrain weights for TFbert

## 🐛 Bug

<!-- Important information -->

Model I am using Bert:

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.when I try to reload the weights for TFbert by using:
    model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased')
2. I got the error
    ValueError: Expected floating point type, got <dtype: 'int32'>.
3. Does anybody know how to fix this?

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Multi GPU dataparallel crash

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: run_lm_finetuning
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: I am finetuning the GPT2 model with a dataset that I have used in the past to fine-tune the  BERT model among others

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
raceback (most recent call last):████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 7587/7588 [1:42:01<00:00,  1.26it/s]
  File "run_lm_finetuning.py", line 551, in <module>
    main()
  File "run_lm_finetuning.py", line 503, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 228, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 153, in forward
    return self.gather(outputs, self.output_device)
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather
    res = gather_map(outputs)
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 55, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 68, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/cuda/comm.py", line 165, in gather
    return torch._C._gather(tensors, dim, destination)
**RuntimeError: Gather got an input of invalid size: got [2, 2, 12, 1024, 64], but expected [2, 3, 12, 1024, 64] (gather at /opt/conda/conda-**bld/pytorch_1565272279342/work/torch/csrc/cuda/comm.cpp:226)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7f47a13d5e37 in /home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: torch::cuda::gather(c10::ArrayRef<at::Tensor>, long, c10::optional<int>) + 0x3c7 (0x7f4720c61327 in /home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #2: <unknown function> + 0x5fa742 (0x7f47a420f742 in /home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #3: <unknown function> + 0x1c8316 (0x7f47a3ddd316 in /home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #14: THPFunction_apply(_object*, _object*) + 0x98f (0x7f47a40024bf in /home/dev/deeplearning/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

## Expected behavior

it crashes at the last batch all the time. I expect it to move to the next epoch

## Environment

* OS:
* Python version: 3.6
* PyTorch version: 1.2
* PyTorch Transformers version (or branch):
* Using GPU ? 4 Quadro 8000
* Distributed of parallel setup ? parallel
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


from_pretrained: convert DialoGPT format

DialoGPT checkpoints have "lm_head.decoder.weight" instead of "lm_head.weight".

(see: https://www.reddit.com/r/MachineLearning/comments/dt5woy/p_dialogpt_state_of_the_art_conversational_model/f6vmwuy?utm_source=share&utm_medium=web2x)


Could you support albert?

## ❓ Questions & Help

For many students,who haven't large GPU,so they will use small model(eg albert),hence I hope you will support loading albert, Tkanks so much!


Extracting the output layer of HuggingFace GPT2DoubleHeadsModel

Hello,

Suppose that I have two GPT2DoubleHeadsModel (let’s call it model A and B).
Is there any way that I can:

1. take the hidden state of a given input at the n-th layer of the model A and feed it directly into the output layer of model B to compute output

AND

2. Take the output obtained from 1. and calculate the loss based on an appropriate label

Some coding example would be a great help.

Thank you,


pip install transformers not downloading gpt2-xl

Following the release of 1.5 B parameter model, I attempted to upgrade my version using the following command:

pip install transformers --upgrade

The installed library does not have gpt2-xl, and throws this error when I try calling it:

OSError: Model name 'gpt2-xl' was not found in model name list (gpt2, gpt2-medium, gpt2-large, distilgpt2). We assumed 'gpt2-xl' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

Please update the pip installer to allow the easy installation of the latest model


For HuggingFace GPT2DoubleHeadsModel, is there a way to directly provide a hidden state for an input?

Hello,
Say I have two custom trained HuggingFace GPT2DoubleHeadsModels (Model 1 and 2).
 I want to take an hidden state of m-th layer of model 1, and use that hidden state as my input for model 2.
Is this possible with HuggingFace GPT2DoubleHeadsModels?

Some coding example would be a great help!

Thank you,


[WIP] BertAbs summarization

This PR builds on the encoder-decoder mechanism to do abstractive summarizaton. Contributions:

- A BeamSearch class that takes any `PreTranedEncoderDecoder` as an input;
- A script `run_summarization.py` that allows to pre-train the model and generate summaries.

Note that to save the checkpoints I had to add a parameter in the `save_pretrained_model` method, but I am not sure this is the best solution.


Fix run_squad.py

The `run_squad.py` was looking in the output directory for `config.json` when really it was one level lower in the checkpoint directories.


Error when Fine-tuning XLM on SQuA

## 🐛 Bug

<!-- Important information -->

Model I am using (XLM):

Language I am using the model on (English and Chinese):

The problem arise when using:

[CUDA_VISIBLE_DEVICES=2 python run_squad.py   --model_type xlm  --model_name_or_path xlm-mlm-tlm-xnli15-1024   --do_train   --do_eval     --train_file $SQUAD_DIR/Crosslingual_QZH_train.json   --predict_file $SQUAD_DIR/Crosslingual_QZH_valid.json   --per_gpu_train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 100   --max_seq_length 384   --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training  --output_dir /home/weihua/Sqad/transformers/xlm_out/ ]

error:
Traceback (most recent call last):
  File "run_squad.py", line 569, in <module>
    main()
  File "run_squad.py", line 515, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 179, in train
    results = evaluate(args, model, tokenizer)
  File "run_squad.py", line 275, in evaluate
    args.version_2_with_negative, tokenizer, args.verbose_logging)
  File "/home/weihua/Sqad/transformers/examples/utils_squad.py", line 814, in write_predictions_extended
    final_text = get_final_text(tok_text, orig_text, tokenizer.do_lower_case,
AttributeError: 'XLMTokenizer' object has no attribute 'do_lower_case'

![WeChat Image_20191108182243](https://user-images.githubusercontent.com/43492059/68469281-d0188e00-0254-11ea-928a-8ab4a7140838.png)
![WeChat Image_20191108182248](https://user-images.githubusercontent.com/43492059/68469288-d3ac1500-0254-11ea-9116-9f0dd15b65cd.png)

How can I deal with this problem?


Only init encoder_attention_mask if stack is decoder

We currently initialize `encoder_attention_mask` when it is `None`,
whether the stack is that of an encoder or a decoder. Since this
may lead to bugs that are difficult to tracks down later, I added a condition
that assesses whether the current stack is a decoder.


[XLM-R] by Facebook AI Research

# 🌟New model addition

## Model description

Yesterday, Facebook has released _open source_ its new NLG model called **XLM-R** (XLM-RoBERTa) on [arXiv](https://arxiv.org/abs/1911.02116). This model uses self-supervised training techniques to achieve state-of-the-art performance in **cross-lingual understanding**, a task in which a model is trained in one language and then used with other languages without additional training data. Our model improves upon previous multilingual approaches by incorporating **more training data and languages** — including so-called low-resource languages, which lack extensive labeled and unlabeled data sets.

## Open Source status

* [ ] the model implementation is available: [here](https://github.com/pytorch/fairseq/blob/master/fairseq/models/roberta/model.py) under the **XLMRModel** Python class (row 198)
* [ ] the model weights are available: **Yes**, [here](https://github.com/pytorch/fairseq/blob/master/examples/xlmr/README.md) more details
* [ ] who are the authors: **FacebookAI Research** (Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov)

## Additional context

Facebook says these two sentences about this new model in their [blog](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/?__xts__[0]=68.ARAEwOOYcfFnZxSeyb88a9qwrCTeTF8aO869NMP3lKL7Xp6aYnOGmS1j9jO6wYt89qDPAiHkpVboO3nuo_GhLR2GM8uVTx2q8m3HFehnIkQjn96QtyD33FxpHipTbT4hC1bohM6Y8tNQirCLKZ28VTRiHBbEpgEP1cr9NuwRUkyVfS0KcQpXv0GFLRFQt4MJvU-nnmrMbBN56MXQ7pi3IGHLVmanXhJbgQFJu7Utq80xa8EGVR3b5SqAZgN6yL7jVwKkcYpnHVdgdJJ6dLdXRm46xK03XbxxrL8ghktmSxXyzhykfPEo_akj0u06syf3HYBsZReDdF178xiEZIgn_2VXIA&__tn__=-UK-R):

> XLM-R represents an important step toward our vision of providing the best possible experience on our platforms for everyone, regardless of what language they speak

> We hope to improve the performance of multilingual models created by the research community, particularly systems that use self-supervised training methods to better understand low-resource languages.

> XLM-R has been trained on **2.5T of data** across **100 languages** data filtered from Common Crawl


BERT: Uncased vocabulary has 30,552 tokens whereas cased has 28,996 tokens. Why this difference?

## ❓ Questions & Help

I was expecting cased vocabulary to be larger than uncased as it needs to include both tokens with capitalizations and without. But, to the contrary, I found the cased vocabulary to be smaller. May I know reason for this?


GPT2 - XL

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. In the documentation, it's given how to load GPT2-XL. But I'm getting an error when I try to load is GPT2 XL available as of now or it would take some time?

Error:
```
>>> model = GPT2DoubleHeadsModel.from_pretrained('gpt2-xl')
Model name 'gpt2-xl' was not found in model name list (gpt2-large, gpt2, gpt2-medium). We assumed 'gpt2-xl' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/cshome/anandhpe/.local/lib/python3.5/site-packages/pytorch_transformers/modeling_utils.py", line 492, in from_pretrained
    **kwargs
  File "/cshome/anandhpe/.local/lib/python3.5/site-packages/pytorch_transformers/modeling_utils.py", line 194, in from_pretrained
    raise e
  File "/cshome/anandhpe/.local/lib/python3.5/site-packages/pytorch_transformers/modeling_utils.py", line 180, in from_pretrained
    resolved_config_file = cached_path(config_file, cache_dir=cache_dir, force_download=force_download, proxies=proxies)  File "/cshome/anandhpe/.local/lib/python3.5/site-packages/pytorch_transformers/file_utils.py", line 124, in cached_path
    raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file gpt2-xl not found

```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:Ubuntu
* Python version:3.7
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch):
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Added additional training utils for run_squad.py

Added different LR schedules and hyperparameters for beta1 and beta2 for AdamW


Fix run_bertology.py

Make imports and `args.overwrite_cache` match `run_glue.py`.


Bug-fix: Roberta Embeddings Not Masked

## Summary
I replace the code that makes the position ids with logic closer to the original fairseq `make_positions` function. It wasn't clear to me what to do in the event that the embeddings are passed in directly through `inputs_embeds` so I resorted to the old methodology and just generating a positional id for all inputs.

## Closes
#1761


Fixed training for TF XLNet

This PR makes the following changes and fully fixes `model.fit()` training for XLNet. It now works, and was tested with a slightly modified `run_tf_glue.py`, and also tested with XLA, AMP and tf.distribute

- Fix dtype error with `input_mask` and `attention_mask`
- `rel_attn_core()` does not work properly in non-eager mode, resulting in shape errors. `tf.shape` is now used to obtain the shape of `ac` instead of calling `ac.shape`, and training now works in non-eager mode.

@thomwolf


Perplexity for (not-stateful) Transformer - Why is it still fair to compare to RNN?

Greetings,

It is clear to me how to compute perplexity for RNNs as RNNs are a stateful model.

In general, given a very long document, I believe we need to: 1. chunk the document into a sequence of chunks, and we compute the cross entropy loss for each of the chunks, before taking the average for all chunks and take exponential.

Because RNNs are a stateful model, we can use what we have from the last RNN hidden state regarding to previous chunk to initialize hidden state for RNN that handles the next chunk. Because of this, I think RNN computes the "right" perplexity for a long document.

Meanwhile Transformer are not a stateful model. For this reason there is no such things as "transferring" hidden state from previous chunk to next chunk I guess. For this reason I think Transformers are not really computing the perplexity for a long document. It compute something different.

So why is it still fair to compare two models regarding to perplexity for long documents?
Plus, if I create a non-stateful RNN (e.g. as in Keras I can set stateful=False), will the perplexity the non-stateful model computes without "transferring" hidden state be the right perplexity we need? Is it comparable to the stateful RNN?

Thank you!


Roberta Positional Embeddings Not Masked

## Bug

Hi, I think for the RoBERTa model, the positional embeddings are created slightly wrong. In both this library and in `fairseq` the positional embeddings are sequential integers starting from `padding_idx + 1`. However, in `fairseq.utils.make_positions` all indicies of the input that correspond to the padding index are masked as they go into the positional embedding layer.

## Example

```python
import torch
padding_idx = 0

tensor = torch.as_tensor([
    [1, 13, 54, 0, 0],
    [1, 55, 12, 24, 0]
])
```

### In Fairseq

```
from fairseq import utils
utils.make_positions(tensor, padding_idx = padding_idx)
>>> tensor([
        [1, 2, 3, 0, 0],
        [1, 2, 3, 4, 0]
        ])
```

The positional sequence begins at padding_idx + 1 but all occurrences of the padding index in the input tensor are unmanipulated. This means the position embeddings are only adjusted for tokens which are not padding.

### In Transformers

This code comes from [modeling_roberta.py#L65](https://github.com/huggingface/transformers/blob/master/transformers/modeling_roberta.py#L65)
```python
# Position numbers begin at padding_idx+1. Padding symbols are ignored.
# cf. fairseq's `utils.make_positions`
position_ids = torch.arange(self.padding_idx+1, seq_length+self.padding_idx+1, dtype=torch.long, device=input_ids.device)
position_ids = position_ids.unsqueeze(0).expand_as(input_shape)
```

The output tensor of these function calls do not retain the masked tokens:
```
>>> tensor([
        [1, 2, 3, 4, 5],
        [1, 2, 3, 4, 5]
        ])
```
## Complete Example

```python

from fairseq import utils

tensor = torch.as_tensor([
    [1, 13, 54, 0, 0],
    [1, 55, 12, 24, 0]
])

def fairseq_make_positions(tensor, padding_idx):
    mask = tensor.ne(padding_idx).int()
    return (
        torch.cumsum(mask, dim=1).type_as(mask) * mask
    ).long() + padding_idx

def transformers_make_positions(input_ids, padding_idx):
    input_shape = input_ids.size()
    seq_length = input_shape[1]
    position_ids = torch.arange(padding_idx+1, seq_length+padding_idx+1, dtype=torch.long, device=input_ids.device)
    position_ids = position_ids.unsqueeze(0).expand(input_shape)
    return position_ids

print(fairseq_make_positions(tensor, padding_idx = 0))
print(transformers_make_positions(tensor, padding_idx = 0))
```
Output:
```
# Fairseq:
tensor([[1, 2, 3, 0, 0],
        [1, 2, 3, 4, 0]])
# Transformers:
tensor([[1, 2, 3, 4, 5],
        [1, 2, 3, 4, 5]])
```

## Expected behavior
I expect all padded tokens in the input tensor to be padded as they go into the positional embeddings layer as well as the token embeddings layer.

## Additional context

Happy to make a PR if this is agreed to be a bug, but this will mean the positional embeddings train slightly differently to the existing `RobertaEmbeddings` module.


'RuntimeError: CUDA error' is occured when encoding text with pre-trained model on cuda

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): I am trying to use Bert encoder

Language I am using the model on (English, Chinese....): English

There are two problems when I use BERT for encoding text to vectors
Here is Error log

First,
```
File "/home/jcw/VAIRLtext/Text2Vec.py", line 192, in encode
    embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 627, in forward
    head_mask=head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 348, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 326, in forward
    attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 283, in forward
    self_outputs = self.self(input_tensor, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 202, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/functional.py", line 1374, in linear
    output += bias
RuntimeError: CUDA error: device-side assert triggered
```

Second,
```
File "/home/jcw/VAIRLtext/Text2Vec.py", line 192, in encode
    embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 627, in forward
    head_mask=head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 348, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 326, in forward
    attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 283, in forward
    self_outputs = self.self(input_tensor, attention_mask, head_mask)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/transformers/modeling_bert.py", line 202, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/jcw/tmp/anaconda3/envs/VAIRLtext/lib/python3.7/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
```

These two errors are coming from BERT encode :(
here is my code
```
class Text2Vec(object):
     def __init__(self, args):
          self.max_lens = args.max_lens
          self.tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')
          self.encoder = self.get_encoder('bert-base-multilingual-cased', args.num_layers)
          self.encoder.to('cuda')
          self.pad_token = self.tokenizer.convert_tokens_to_ids(self.tokenizer.pad_token)

     def get_encoder(self, encoder_type, num_layers):
          model = AutoModel.from_pretrained(encoder_type)
          model.eval()
          model.encoder.layer = torch.nn.ModuleList([layer for layer in model.encoder.layer[:num_layers]])
          return model

     def padding(self, arr, pad_token):
          lens = torch.LongTensor([len(a) for a in arr])
          padded = torch.ones(len(arr), self.max_lens) * pad_token
          mask = torch.zeros(len(arr), self.max_lens)
          for i, a in enumerate(arr):
               padded[i, :lens[i]] = torch.tensor(a)
               mask[i, :lens[i]] = 1
          return padded, mask

     def encode(self, text):
          text = sent_tokenize(text)
          tokens = [self.tokenizer.encode(a, add_special_tokens=True) for a in text]
          padded_tokens, masks = self.padding(tokens, self.pad_token)

          padded_tokens = padded_tokens.to('cuda')
          masks = masks.to('cuda')

          self.encoder.eval()
          with torch.no_grad():
               embeddings = self.encoder(padded_tokens, attention_mask=masks)[0]

          embed = []
          masks = list(torch.split(masks, 1, dim=0))
          embeddings = list(torch.split(embeddings, 1, dim=0))
          for embedding, mask in zip(embeddings, masks):
               masked_embed = embedding * mask.squeeze()[:, None]
               embed.append(masked_embed)

          return torch.cat(embed, dim=0)
```

What am I missing? please advise to me!!

## Environment
* OS: ubuntu 16.04
* Python version: Python 3.7.4
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? titan-xp
* Distributed of parallel setup ? not yet
* Any other relevant information:


NameError: name 'DUMMY_INPUTS' is not defined

## ❓ Can someone help me with resolving this error?
<!-- A clear and concise description of the question. -->
![image](https://user-images.githubusercontent.com/10863620/68379756-7771c400-0174-11ea-8d8d-7b72159c555f.png)


the BertModel have the class "BertForTokenClassification", why XLNetModel don't have the class

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


Is it possible fine tune XLNet?

## ❓ Questions & Help

Hello, is there currently a script or recommended process to fine tune XLNet? Currently I am writing my own and using https://mccormickml.com/2019/09/19/XLNet-fine-tuning/ as an inspiration but overall I am still pretty new at this. I managed to get it to run but it seems to be taking forever so just thought I'd check.

Thanks! Any help would be great.

<!-- A clear and concise description of the question. -->


Is it okay to define and use new model by using only a part of full GPT model blocks? Or anyone tried to do so?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,

As far as I know, the smallest GPT2 model that comes with pretrained weight has 768 hidden size with 12 transformer blocks.

However, I'm considering to define 'smaller' GPT model with only 1 or 2 transformer blocks and load those weight from corresponding blocks in the pretrained GPT weight.

This is basically because the original model size is too big to fit in my gpu environment.

so I'm wondering if it will work or not. I think it would be at least better than start training from randomly initialized weight, but not sure about it.

So I'd like to ask how you think.
Thanks!


How to add weighted CrossEntropy loss in sequence classification task?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am doing a sequence classification task base on run_ner.py. My dataset is an imbalenced dataset and apperantly 'O' label appears far more than other labels. I'm wondering if it is helpful to use weighted CrossEntropy loss.
If the answer is 'yes', how to add weight to the loss function? Currently I hard code the weight in BertForSequenceClassification class (in modeling_bert.py). I feel it is not a smart way to do it. Do you have some advice on how to add the weight infomation?
Many thanks.


Out of Memory Error (OOM) only during evaluation phase of run_lm_finetuning.py and run_glue.py

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): roberta-large

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* the official example scripts: run_lm_finetuning.py and run_glue.py

The tasks I am working on is:
* my own task or dataset: my own dataset which I have put into the correct format for `run_lm_finetuning` and the `SST-2` task for `run_glue.py`

## To Reproduce

When I use the official `run_lm_finetuning.py` and `run_glue.py` scripts with `roberta-large` on my own data, I run into out of memory (`OOM`) issues during `evaluation`. `Training completes successfully` with a `batch_size` of `1` and a `max_senquence_length` of `512`. However, when `evaluating` the 4th or 5th checkpoint, I get an `OOM` error. I tested using `distributed` training with `FP16` on `2 Tesla P100-SXM2 16 GB` and `2 Tesla P100-PCIE 12 GB` with the same results. I'm currently testing on `2 Tesla M40 24GB`; however, I don't always have access to these GPUs; so,  this is untenable long term.

## Expected behavior

Since it finishes training and evaluation of the first few checkpoints with the aforementioned setting, I would expect it to be able to evaluate every checkpoint without error. However, it seems that there is a bug in the evaluation code. I assume it is not properly clearing the memory after testing each checkpoint.

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6.8
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU Yes,  Tesla M40 24GB, Tesla P100-SXM2 16 GB, Tesla P100-PCIE 12 GB
* Distributed of parallel setup 2x of each separate GPU


Added Mish Activation Function

Mish is a new activation function proposed here - https://arxiv.org/abs/1908.08681
It has seen some recent success and has been adopted in SpaCy, Thic, TensorFlow Addons and FastAI-dev.
All benchmarks recorded till now (including against ReLU, Swish and GELU) is present in the repository - https://github.com/digantamisra98/Mish
Might be a good addition to experiment with especially in the Bert Model.


Subtokens in BPE in GPT2

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi, I have a question.

For a word represented by more than 1 subtoken in BPE, I wonder what's the principle to divide it into subtokens?

I use GPT2 small version for example. "bookshelf" is divided into "books","he" and "if".(index for encoder is [3835, 258, 1652]).  But actually "book" and "shelf" are represented by single subtoken.(book is 1492, shelf is 18316). Why is "bookshelf" divided into 3 parts instead of "book" and "shelf"?

Thank you!


input token embedding issues

1. During fine-tuning an XLNet, does the word embeddings kept constant (learned from the pre-trained model) or are they initialized randomly but learned contextually during fine-tuning itself?

2. And how would I set my own word embeddings during fine-tuning (XLNetForSequenceClassification)? I have my own token embeddings that I want to feed for my XLNetForSequenceClassification and is it possible to keep this embeddings constant during fine-tuning?
get_input_embeddings and set_input_embeddings functions does not seem to work with XLNetForSequenceClassification Class.

Thanks!


How to calculate memory requirements of different GPT models?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

What's the best way to calculate how much GPU ram is needed to finetune a given GPT size? The GPT-xl model is 6.5GB worth of parameters but clearly this isn't the full story when it comes to fine-tuning. I've tried running a batch size of 1 on a 11GB 1080ti but that ran out of memory. My next step is to test it on a 2080ti with FP16 training enabled to further reduce memory requirements. It would be great if there was some back of the envelope calculation to see in advance if this would work or if instead I should go straight to 16GB Titan V. Or maybe even that's too small!


gpt2 generation crashes when using `past` for some output lengths

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): gpt2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

I am using `run_generation.py` to generate text from gpt2, but with the code slightly changed to make use of `past` to cache hidden states.  It crashes in some cases.

## To Reproduce

Steps to reproduce the behavior:

Apply the three line change to the `run_generation.py` script:

```
$ git checkout f88c104d8 .   # current head of master
$ git diff
diff --git a/examples/run_generation.py b/examples/run_generation.py
index 2d91766..bfbf68a 100644
--- a/examples/run_generation.py
+++ b/examples/run_generation.py
@@ -113,9 +113,10 @@ def sample_sequence(model, length, context, num_samples=1, temperature=1, top_k=
     context = context.unsqueeze(0).repeat(num_samples, 1)
     generated = context
     with torch.no_grad():
+        past = None
         for _ in trange(length):

-            inputs = {'input_ids': generated}
+            inputs = {'input_ids': generated, 'past': past}
             if is_xlnet:
                 # XLNet is a direct (predict same token, not next token) and bi-directional model by default
                 # => need one additional dummy token in the input (will be masked), attention mask and target mapping (see model docstring)
@@ -136,6 +137,7 @@ def sample_sequence(model, length, context, num_samples=1, temperature=1, top_k=
                 inputs["langs"] = torch.tensor([xlm_lang] * inputs["input_ids"].shape[1], device=device).view(1, -1)

             outputs = model(**inputs)  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet/CTRL (cached hidden-states)
+            past = outputs[1]
             next_token_logits = outputs[0][:, -1, :] / (temperature if temperature > 0 else 1.)

             # repetition penalty from CTRL (https://arxiv.org/abs/1909.05858)
```

```
$ python examples/run_generation.py --prompt "Who was Jim Henson ? Jim Henson was a" --model_type gpt2 --model_name_or_path gpt2 --length 40
...
11/06/2019 11:38:36 - INFO - __main__ -   Namespace(device=device(type='cpu'), length=40, model_name_or_path='gpt2', model_type='gpt2', n_gpu=0, no_cuda=False, num_samples=1, padding_text='', prompt='Who was Jim Henson ? Jim Henson was a', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')
 88%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                | 35/40 [00:04<00:00,  8.68it/s]
Traceback (most recent call last):
  File "examples/run_generation.py", line 262, in <module>
    main()
  File "examples/run_generation.py", line 247, in main
    device=args.device,
  File "examples/run_generation.py", line 139, in sample_sequence
    outputs = model(**inputs)  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet/CTRL (cached hidden-states)
  File "/Users/joelb/anaconda3/envs/huggingface/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/joelb/views/transformers/transformers/modeling_gpt2.py", line 546, in forward
    inputs_embeds=inputs_embeds)
  File "/Users/joelb/anaconda3/envs/huggingface/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/joelb/views/transformers/transformers/modeling_gpt2.py", line 438, in forward
    position_embeds = self.wpe(position_ids)
  File "/Users/joelb/anaconda3/envs/huggingface/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/joelb/anaconda3/envs/huggingface/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/Users/joelb/anaconda3/envs/huggingface/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 1024 out of table with 1023 rows. at /Users/distiller/project/conda/conda-bld/pytorch_1570710797334/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

The crash does not occur with the default `--length 20`.

I'm trying to make this work for faster generation speed.  So I tested the change with a smaller length that does not crash, but I observe the speed with caching is slightly slower than without.

## Environment
```
Platform Darwin-18.7.0-x86_64-i386-64bit
Python 3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)]
PyTorch 1.3.0
Tensorflow 2.0.0
```

## Additional context

<!-- Add any other context about the problem here. -->


Released OpenAI GPT-2 1.5B model

## 🚀 Feature

- [ ] Blog: [https://openai.com/blog/gpt-2-1-5b-release/](url)
- [ ] Code: [https://github.com/openai/gpt-2](url)
- [ ] Dataset: [https://github.com/openai/gpt-2-output-dataset](url)

## Motivation

A bigger model for text generation which is more human than the OpenAI GPT-2 774M parameters

## Additional context

Maybe this is not the most requested feature by AI researchers. Maybe, it's better to implement and standardize **new** NLG models!


How to use gpt-2-xl with run_generation.py

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am currently running the run_generation.py and putting the model as gpt-2, however I am not satisfied with the results frm the small model.

How can I insert the big model instead?


Fixing models inputs_embeds