Ram utilisation of DistilBERT

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I was checking the memory consumption of RoBERTa and DistilBERT. I found there is no significant change in memory usage. Although Inference time is around 1sec for DistilBERT and for RoBERTa is 2sec.
Memory usage on CPU:
Port 9000: DistilBERT
Port 9002: RoBERTa
![compute](https://user-images.githubusercontent.com/18630864/65751708-e8ab6980-e128-11e9-93a7-937cc0211009.png)

Have you guys seen any significant change in memory usage or am I missing something here?


Errors when using fp16 with traced models

## 🐛 Bug

When I run

```
roberta_model = RobertaForMaskedLM.from_pretrained("roberta-base", torchscript=True)
roberta_model.cuda()
roberta_model.half()
traced_model = torch.jit.trace(roberta_model, (r_input_ids))
```

I get the following error

`
Expected object of scalar type Float but got scalar type Half for argument #2 'mat2'
`

When I attempt to load a normally traced model

```
loaded_model = torch.jit.load("traced_roberta_cuda.pt")
loaded_model.cuda()
loaded_model.half()
loaded_model(r_input_ids )
```

I get

`RuntimeError: expected device cuda:0 and dtype Float but got device cuda:0 and dtype Half
`

Is there a way to use fp16 with traced models? It happened with BertForSequenceClassification, RobertaForSequenceClassification and RobertaForMaskedLM.

## Environment
* Models tested on: Bert and Roberta:
* Language: English
* OS: Ubuntu 18.04
* Python version: 3.6.9
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 2.0.0


RobertaTokenizer documentation is off with the new transformers library

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Roberta

Language I am using the model on (English, Chinese....): NA

The problem arise when using:
* [ ] the official example scripts:

The tasks I am working on is:
NA

## To Reproduce

Steps to reproduce the behavior:

In the documentation for the tokenization_roberta.py, it says in the RobertaTokenizer class

```
    RoBERTa BPE tokenizer, derived from the GPT-2 tokenizer. Peculiarities:
        - Byte-level Byte-Pair-Encoding
        - Requires a space to start the input string => will add a space is there isn't.
          As a consequence, this tokenizer `encode` and `decode` method will not conserve
          the absence of a space at the beginning of a string: `tokenizer.decode(tokenizer.encode("Hello")) = " Hello"
```
However, with using the new transformers library, when I run this example I get
```
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

tokenizer.decode(tokenizer.encode("Hello"))
"Hello"
```
The leading space seems to no longer present as it was in pytorch_transformers, however if (per the source code) if I add the arg add_prefix_space = True, then it outputs with the leading space. Just a tiny fix to hopefully help out anyone else who gets confused by it.

Thanks and love the new updates to the library!


With the latest update to `Transformers`, has the function been removed? I still see it in the code, but I run into the error:

`AttributeError: 'RobertaTokenizer' object has no attribute 'add_special_tokens_sentences_pair'`


Examples in Colab

Hi all , does anyone have a Colab sample to share ?


Size mismatch when loading pretrained model

I'm seeing this:

```
In [1]: import pytorch_transformers
In [2]: m=pytorch_transformers.AutoModel.from_pretrained('roberta-base')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-7a33f5ecb345> in <module>
----> 1 m=pytorch_transformers.AutoModel.from_pretrained('roberta-base')
/opt/anaconda3/lib/python3.7/site-packages/pytorch_transformers/modeling_auto.py in from_pretrained(cls, pretrained
_model_name_or_path, *model_args, **kwargs)
    240             return DistilBertModel.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
    241         elif 'roberta' in pretrained_model_name_or_path:
--> 242             return RobertaModel.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
    243         elif 'bert' in pretrained_model_name_or_path:
    244             return BertModel.from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
/opt/anaconda3/lib/python3.7/site-packages/pytorch_transformers/modeling_utils.py in from_pretrained(cls, pretraine
d_model_name_or_path, *model_args, **kwargs)
    592         if len(error_msgs) > 0:
    593             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 594                                model.__class__.__name__, "\n\t".join(error_msgs)))
    595
    596         if hasattr(model, 'tie_weights'):
RuntimeError: Error(s) in loading state_dict for RobertaModel:
        size mismatch for roberta.embeddings.position_embeddings.weight: copying a param with shape torch.Size([514
, 768]) from checkpoint, the shape in current model is torch.Size([512, 768]).
```


Why is the vocabulary of token_type_ids and input_ids shared?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

https://github.com/huggingface/transformers/blob/17ea43cf985829634bd86b36b44e5410c6f83e36/transformers/modeling_gpt2.py#L421

In GPT2Model, forward method, it seems the vocabulary of token_type_ids and input_ids is shared. I checked the vocabulary table, 0 and 1 corresponds to the exclamation sign and the quote sign. What is the reason of sharing the vocabulary? Is it on purpose?


Extending `examples/` to TensorFlow

## 🚀 Feature

Hi, thanks for putting in the tremendous effort for TensorFlow-PyTorch interoperability! Would those scripts in the `examples/` be soon extended to Tensorflow as well?

## Motivation

I (and presumably many others) rely on the examples to quickly experiment with models and ideas. Extending the examples to Tensorflow would be hugely helpful, and should help the codebase reach a broader set of audiences.

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


faster dataset building

Now it takes around 1 minute to process 20mb and it takes forever for 200mb dataset (it's non-linear). This is a fix to make it linear.


Completed the documentation with TF2


Optimize XLNet model to generate embedding of long documents

We experiment generating embeddings with TransformerXL and XLnet. Our documents have 5000 to 80000 characters each.

We got an average of 0.8 second per document with TransformerXL and 1.3 second per document with XLNet.

To optimize XLNet we found that using only 200 tokens per call is optimal.
A ratio around 350 tokens/second with xlnet-base-cased and 100 tokens/second with large.

Any idea if XLNet can be optimize?

    tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
    model = XLNetModel.from_pretrained("xlnet-base-cased")

    outputs = model(text_tokens, mems=mems)


Typo in modeling_bert file

I was looking at the code of BertModel adapted for different tasks here
https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py

I noticed a small typo in line 882
`self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)`
I think it should be either `self.num_labels` or `config.num_labels` in the second argument

The complete function
```
def __init__(self, config):
        super(BertForSequenceClassification, self).__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)

        self.init_weights()
```


[FIX] fix run_generation.py to work with batch_size > 1

I expended the `top_k_top_p_filtering` function, and by that the`run_generation.py` script to work with num_samples > 1.

This can be expended by scattering the sorted tensors.
First pull request in this repository, so let me know if I need to do anything else :)

Cheers, Matan.


pytorch-transformers returns output of 13 layers?

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....): BertModel

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [x] my own task or dataset: (give details)

Details of the issue:

I am using pytorch-transformers for the rather unconventional task of regression (one output). In my research I use BERT and I'm planning to try out the other transformers as well. When I started, I got good results with `pytorch-pretrained-bert`. However, running the same code with `pytorch-transformers` gives me results that are a lot worse.

In the original code, I use the output of the model, and concatenate the last four layers - as was proposed in the BERT paper. The architecture that I used looks like this:

```python
from pytorch_pretrained_bert.modeling import BertModel
import torch
from torch import nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.bert_model = BertModel.from_pretrained('bert-base-uncased')
        self.pre_classifier = nn.Linear(3072, 512)
        self.dropout = nn.Dropout(0.2)
        self.classifier = nn.Linear(512, 1)

    def forward(self, bert_ids, bert_mask):
        all_bert_layers, _ = self.bert_model(bert_ids, attention_mask=bert_mask)
        print('hidden_states', len(all_bert_layers))
        # concat last four layers
        out = torch.cat(tuple([all_bert_layers[i] for i in [-1, -2, -3, -4]]), dim=-1)
        print('output', out.size())

        # Pooling by also setting masked items to zero
        bert_mask = bert_mask.unsqueeze(2)
        # Multiply output with mask to only retain non-paddding tokens
        out = torch.mul(out, bert_mask)
        print('output', out.size())

        # First item ['CLS'] is sentence representation
        out = out[:, 0, :]
        print('pooled_output', out.size())

        out = self.pre_classifier(out)
        print('pre_classifier', out.size())

        out = self.dropout(out)
        print('dropout', out.size())

        out = self.classifier(out)
        print('classifier', out.size())

        return out
```

When porting this to `pytorch-transformers`, the main thing was that now we get a tuple back from the model *and* we have to explicitly ask to get all hidden states back. As such, the converted code looks like this:

```python
from pytorch_transformers import BertModel
import torch
from torch import nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.bert_model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
        self.pre_classifier = nn.Linear(3072, 512)
        self.dropout = nn.Dropout(0.2)
        self.classifier = nn.Linear(512, 1)

    def forward(self, bert_ids, bert_mask):
        out, _ = self.bert_model(input_ids=bert_ids, attention_mask=bert_mask)
        hidden_states = out[2]
        print('hidden_states', len(hidden_states))

        out = torch.cat(tuple([hidden_states[i] for i in [-1, -2, -3, -4]]), dim=-1)
        print('output', out.size())

        # Pooling by also setting masked items to zero
        bert_mask = bert_mask.unsqueeze(2)
        # Multiply output with mask to only retain non-paddding tokens
        out = torch.mul(out, bert_mask)
        print('output', out.size())

        # First item ['CLS'] is sentence representation
        out = out[:, 0, :]
        print('pooled_output', out.size())

        out = self.pre_classifier(out)
        print('pre_classifier', out.size())

        out = self.dropout(out)
        print('dropout', out.size())

        out = self.classifier(out)
        print('classifier', out.size())

        return out
```

As I said before, this leads to *very* different results. Seeding cannot be the issue, since I set all seeds manually in both cases, like this:

```python
def set_seed():
    torch.manual_seed(3)
    torch.cuda.manual_seed_all(3)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(3)
    random.seed(3)
    os.environ['PYTHONHASHSEED'] = str(3)
```

I have added the print statements as a sort of debugging and I quickly found that there is a fundamental difference between the two architectures. The *hidden_states* print statement will yield `12` for pytorch-pretrained-bert and `13` for `pytorch-transformers`! I am not sure how that relates, but I would assume that this could be the starting point to start looking.

I have tried comparing the created models, but in both cases the encoder consists of 12 layers, so I am not sure why `pytorch-transformers` returns 13? What's the extra one?

Going through the source code, it seems that the first hidden_state (= last hidden_state from the embeddings) is included. Is that true?

https://github.com/huggingface/pytorch-transformers/blob/7c0f2d0a6a8937063bb310fceb56ac57ce53811b/pytorch_transformers/modeling_bert.py#L340-L352

Even so, since the embeddings would be the first item in all_hidden_states, the last four layers should be the same still. Therefore, I am not sure why there is such a big difference in the results of the above two. If you spot any faults, please advise.

## Environment

* OS: Win 10
* Python version: 3.7
* PyTorch version: 1.2
* PyTorch Transformers version (or branch):
* Using GPU ? Yes, CUDA 10
* Distributed of parallel setup ? No

## Checklist

- [x] I have read the migration guide in the readme.


## ❓ Questions & Help

Is the UI code for https://transformer.huggingface.co open source?


Loading errors for BERT base on GPU with PyTorch 0.4.1


GLUE Script for Tensorflow


Sequence Classification pooled output vs last hidden state

## ❓ Questions & Help

Why in BertForSequenceClassification do we pass the pooled output to the classifier as below from the source code
```python
outputs = self.bert(input_ids,
                    attention_mask=attention_mask,
                    token_type_ids=token_type_ids,
                    position_ids=position_ids,
                    head_mask=head_mask)

pooled_output = outputs[1]

pooled_output = self.dropout(pooled_output)
logits = self.classifier(pooled_output)
```
but in RobertaForSequenceClassification we do not seem to pass the pooler output?
```python
outputs = self.roberta(input_ids,
                       attention_mask=attention_mask,
                       token_type_ids=token_type_ids,
                       position_ids=position_ids,
                       head_mask=head_mask)
sequence_output = outputs[0]
logits = self.classifier(sequence_output)
```

I thought we would pass the pooled_output in both cases to the classifier?


Pytorch/TF2 determinism

Check to see if the models have the same results when in eval mode (pt) or when training=False (tf)


## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
bert-large-uncased

Language I am using the model on (English, Chinese....):
English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

I am using a modified version of the run_lm_finetuning.py and amp at optimization level o1.  Level O2 runs without issue.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

I am finetuning on a dataset of sentences, with a min length of 10 tokens, using padding and the mask_tokens function given by the repo. Input and labels are padded per specs and of type=LongTensor (torch.Size([4, 200]) torch.Size([4, 200])) with batch size of 4.

## To Reproduce

Steps to reproduce the behavior:

1. When I run without amp, training works as intended.  If I train at amp level O2, training runs as intended.
2. Running with amp level O1 leads to an error: RuntimeError: expected scalar type Half but found Float
3. I have verified that there is no model.eval() and that scaled_loss and clip_grad_norm_ calls are the same as prescribed in the example.
3. Examples of model initialization, train loop, and error are below.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Training to complete without error.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:  Google Colab
* Python version:  3.6
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? Y
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

**Model/Optimizer code:**
```
    model = model_class.from_pretrained(args.model_name_or_path, from_tf=bool('.ckpt' in args.model_name_or_path), config=config)
    model.to(args.device)

    # Prepare optimizer and schedule (linear warmup and decay)
    #no_decay = ['bias', 'LayerNorm.weight']
    no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': args.weight_decay},
        {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
        ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, weight_decay=args.weight_decay)

    scheduler = WarmupLinearSchedule(optimizer, warmup_steps=args.warmup_steps, t_total=t_total)

    if args.fp16:
        model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)
```

**Import Part of Train Loop:**
```
        for step, batch in enumerate(epoch_iterator):
            inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else mask_labels(batch, tokenizer, args)

            inputs = inputs.to(args.device)
            labels = labels.to(args.device)

            model.train()

            loss, _ = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)

            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel training
            if args.gradient_accumulation_steps > 1:
                loss = loss / args.gradient_accumulation_steps

            if args.fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()

            tr_loss += loss.item()

            if (step + 1) % args.gradient_accumulation_steps == 0:
                print('Clipping Grad Norm. fp16:', args.fp16)
                if args.fp16:
                  torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
                else:
                  torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)
                scheduler.step()  # Update learning rate schedule
                optimizer.step()
                optimizer.zero_grad()
                global_step += 1
```

**Error:**
```
RuntimeError                              Traceback (most recent call last)

<ipython-input-7-627529edcd83> in <module>()
     10 ft_data.fp16_opt_level = 'O1'
     11
---> 12 run_finetune(ft_data)
     13
     14 print('Finetuned BERT model loaded.')

12 frames

<ipython-input-6-39df7e7a4180> in run_finetune(args)
    337         torch.distributed.barrier()
    338
--> 339     global_step, tr_loss = train(args, train_dataset, model, tokenizer)
    340     logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
    341

<ipython-input-6-39df7e7a4180> in train(args, train_dataset, model, tokenizer)
    193             model.train()
    194
--> 195             loss, _ = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
    196
    197             if args.n_gpu > 1:

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, masked_lm_labels)
    767
    768         sequence_output = outputs[0]
--> 769         prediction_scores = self.cls(sequence_output)
    770
    771         outputs = (prediction_scores,) + outputs[2:]  # Add hidden states and attention if they are here

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, sequence_output)
    417
    418     def forward(self, sequence_output):
--> 419         prediction_scores = self.predictions(sequence_output)
    420         return prediction_scores
    421

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, hidden_states)
    406
    407     def forward(self, hidden_states):
--> 408         hidden_states = self.transform(hidden_states)
    409         hidden_states = self.decoder(hidden_states) + self.bias
    410         return hidden_states

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_bert.py in forward(self, hidden_states)
    388         hidden_states = self.dense(hidden_states)
    389         hidden_states = self.transform_act_fn(hidden_states)
--> 390         hidden_states = self.LayerNorm(hidden_states)
    391         return hidden_states
    392

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py in forward(self, input)
    157         if self.elementwise_affine:
    158           return FusedLayerNormAffineFunction.apply(
--> 159               input, self.weight, self.bias, self.normalized_shape,self.eps)
    160         else:
    161           return FusedLayerNormFunction.apply(input, self.normalized_shape, self.eps)

/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py in forward(ctx, input, weight, bias, normalized_shape, eps)
     23     bias_ = bias.contiguous()
     24     output, mean, invvar = fused_layer_norm_cuda.forward_affine(
---> 25         input_, ctx.normalized_shape, weight_, bias_, ctx.eps)
     26     ctx.save_for_backward(input_, weight_, bias_, mean, invvar)
     27     return output

RuntimeError: expected scalar type Half but found Float (data<c10::Half> at /usr/local/lib/python3.6/dist-packages/torch/include/ATen/core/TensorMethods.h:1821)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f0840be9273 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: c10::Half* at::Tensor::data<c10::Half>() const + 0x3ee (0x7f08298ccf8e in /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #2: cuda_layer_norm(at::Tensor*, at::Tensor*, at::Tensor*, at::Tensor*, int, int, c10::ArrayRef<long>, at::Tensor*, at::Tensor*, double) + 0x4c5 (0x7f08298ca745 in /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
```
There are a total of 63 frames which output in the error that have been truncated here.


[Proposal] GLUE processors included in library


A Micro BERT

## ❓ Questions & Help

Hello,
Has anyone solved a problem like this, or knows of a solution:
I want to pre-train BERT on a custom dataset, but this data is much smaller than the one used by Google.
So is it possible to train it on a "micro" bert with much lesser layers, etc.
Thanks in advance


How to build a Text-to-Feature Extractor based on Fine-Tuned BERT Model

I have now tried for several days to solve an issue I have...

I need to make a feature extractor for a project I am doing, so I am able to translate a given sentence e.g. "My hat is blue" into a vector of a given length e.g. 768. That vector will then later on be combined with several other values for the final prediction in e.g. a random forest algorithm.

My dataset contains a text column + a label column (with 0 and 1 values) + several other columns that are not of interest for this problem.

I know how to do make that feature extractor using word2vec, Glove, FastText and pre-trained BERT/Elmo Models. That works okay.

Now I want to improve the text-to-feature extractor by using a FINE-TUNED BERT model, instead of a PRE-TRAINED BERT MODEL. I want to fine-tune the BERT model on my dataset and then use that new BERT model to do the feature extraction. I am NOT INTERESTED in using the bert model for the predictions themselves! Only for the feature extraction. How can i do that? I think i need the run_lm_finetuning.py somehow, but simply cant figure out how to do it.

I could really need some help...

P.S. I have already created a binary classifier using the text information to predict the label (0/1), by adding an additional layer. Could I in principle use the output of the previous layers, in evaluation mode, as word embeddings? If I can, then I am not sure how to get the output of those in evaluation mode.


parameter never_split not added in BasicTokenizer's tokenize

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
Bert

Language I am using the model on (English, Chinese....):
English

The problem arise when using:
* [ ] my own modified scripts:  I need to add some special tokens that will not been split during tokenizing. And my special tokens contain punc, like [E1]. This will be split in _run_split_on_punc() if omit parameter never_split in this function.

## To Reproduce

Steps to reproduce the behavior:

1.
 omit parameter never_split when invoke self._run_split_on_punc(token)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

```python
    # tokenization_bert.py
    def tokenize(self, text, never_split=None):
       ....
        orig_tokens = whitespace_tokenize(text)
        split_tokens = []
        for token in orig_tokens:
            if self.do_lower_case and token not in never_split:
                token = token.lower()
                token = self._run_strip_accents(token)

            split_tokens.extend(self._run_split_on_punc(token))
        output_tokens = whitespace_tokenize(" ".join(split_tokens))

        return output_tokens
```
Modify
```python
            split_tokens.extend(self._run_split_on_punc(token))
```
to
```python
            split_tokens.extend(self._run_split_on_punc(token, never_split))
```
wolud solve this problem.


Using pytorch-transformer to reimplement the "Attention is all you need" paper

## ❓ Questions & Help
I use this repo for a long time but I realized even though the name is PyTorch transformers I can't find an easy way to re-implement the original paper of "Attention is all you need" with pretrained model. Can someone help me?


Why does padding affect the embedding results for XLNet? Pre-padding returns different embeddings than post-padding. Which one should be used?

## ❓ Questions & Help

Hello, I am confused with different results of XLNet depending on padding. For Bert, padding doesn't affect the outputs, but for XLNet with **pre** padding (which I saw in https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L281), returns very different results for the same sentence, with and without padding. The difference between no padding and post padding for XLNet returns similar results. Why did "pre" padding is used for run_glue.py? Does XLNet expect post padding or pre padding? Is there any document I am missing to clarify those important distinctions?

Here is a demonstration of the differences for pre, post padding for BERT and XLNet: https://colab.research.google.com/drive/1PCiU3icdfUB-nrLbrKAhgpePoIcFRlqX

Thanks,
Osman


BertForQuestionAnswering output to predict text

<!-- A clear and concise description of the question. -->

In predict mode, BertForQuestionAnswering model output a tuple like below, how to get the text answer interactively
```
tensor([[ 0.4691,  0.3912, -0.3447,  0.9756,  0.7171,  0.3746,  0.5273,  0.3756,
          0.2083,  0.4130,  0.2145,  0.1327,  0.7265,  0.4678,  0.6294,  0.3284]])
```


This is my code for Roberta:

```

# coding: utf-8

# In[4]:

import pandas as pd
import numpy as np
import json, re
from tqdm import tqdm_notebook
from uuid import uuid4

## Torch Modules
import torch
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader

## PyTorch Transformer
from pytorch_transformers import RobertaModel, RobertaTokenizer
from pytorch_transformers import RobertaForSequenceClassification, RobertaConfig

# In[21]:

import pandas as pd
train_data = pd.read_csv("walmart_input/test/train.csv")
test_data  = pd.read_csv("walmart_input/test/test.csv")
dataset    = pd.concat([train_data, test_data])
test_data.head()

# In[46]:

total_length = len(dataset)

# In[47]:

label_to_ix = {}
for label in dataset.PT:
    if label not in label_to_ix:
        label_to_ix[label]=len(label_to_ix)

total_pt_count = len(list(set(list(train_data["PT"]))))

# In[48]:

config            = RobertaConfig.from_pretrained('roberta-base')
config.num_labels = len(list(set(list(train_data["PT"]))))

print(f"Total length of dataset: {total_length} \n Total PT count: {total_pt_count} \n {config}")
# In[36]:

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model     = RobertaForSequenceClassification(config)

def prepare_features(seq_1, max_seq_length = 300,
             zero_pad = False, include_CLS_token = True, include_SEP_token = True):
    ## Tokenzine Input
    tokens_a = tokenizer.tokenize(seq_1)

    ## Truncate
    if len(tokens_a) > max_seq_length - 2:
        tokens_a = tokens_a[0:(max_seq_length - 2)]
    ## Initialize Tokens
    tokens = []
    if include_CLS_token:
        tokens.append(tokenizer.cls_token)
    ## Add Tokens and separators
    for token in tokens_a:
        tokens.append(token)

    if include_SEP_token:
        tokens.append(tokenizer.sep_token)

    input_ids = tokenizer.convert_tokens_to_ids(tokens)
    ## Input Mask
    input_mask = [1] * len(input_ids)
    ## Zero-pad sequence lenght
    if zero_pad:
        while len(input_ids) < max_seq_length:
            input_ids.append(0)
            input_mask.append(0)
    return torch.tensor(input_ids).unsqueeze(0), input_mask

# In[38]:

class Intents(Dataset):
    def __init__(self, dataframe):
        self.len = len(dataframe)
        self.data = dataframe

    def __getitem__(self, index):
        title = self.data.title[index]
        label = self.data.PT[index]
        X, _  = prepare_features(title)
        y = label_to_ix[self.data.PT[index]]
        return X, y

    def __len__(self):
        return self.len

print("FULL Dataset: {}".format(dataset.shape))
print("TRAIN Dataset: {}".format(train_data.shape))
print("TEST Dataset: {}".format(test_data.shape))
training_set = Intents(train_data)
testing_set = Intents(test_data)

training_set.__getitem__(0)[0].shape
model(training_set.__getitem__(0)[0])

# In[65]:

## Training Params
if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
    #model = nn.DataParallel(model)

## Training Params
device = torch.device("cuda:0,1" if torch.cuda.is_available() else "cpu")
model  = model.cuda()
model = nn.DataParallel(model,device_ids=[0,1],dim=1)
model.to(device)
# Parameters
params = {'batch_size': 1,
          'shuffle': True,
          'num_workers': 2}

training_loader = DataLoader(training_set, **params)
testing_loader  = DataLoader(testing_set, **params)

# In[66]:

loss_function = nn.CrossEntropyLoss()
learning_rate = 1e-02
optimizer = optim.Adam(params =  model.parameters(), lr=learning_rate)

## Test Forward Pass
inp = training_set.__getitem__(0)[0].cuda()

#print(inp)
output = model(inp)[0]
torch.max(output.data, 1)

# In[ ]:

import time
start_time    = time.time()
max_epochs    = 2
model         = model.train()
for epoch in tqdm_notebook(range(max_epochs)):
    print("EPOCH -- {}".format(epoch))
    for i, (sent, label) in enumerate(training_loader):
        optimizer.zero_grad()
        sent = sent.squeeze(0)
        if torch.cuda.is_available():
          sent = sent.cuda()
          label = label.cuda()
        print("CUDA detail:")
        print(sent)
        print(label)
        output = model.forward(sent)[0]
        _, predicted = torch.max(output, 1)
        print(f" - {i}.) {predicted}")
        loss = loss_function(output, label)
        loss.backward()
        optimizer.step()

        if i%100 == 0:
            correct = 0
            total = 0
            for sent, label in testing_loader:
                sent = sent.squeeze(0)
                if torch.cuda.is_available():
                  sent = sent.cuda()
                  label = label.cuda()
                output = model.forward(sent)[0]
                _, predicted = torch.max(output.data, 1)
                total += label.size(0)
                correct += (predicted.cpu() == label.cpu()).sum()
            accuracy = 100.00 * correct.numpy() / total
            print('Iteration: {}. Loss: {}. Accuracy: {}%'.format(i, loss.item(), accuracy))

timetaken    = format(float((time.time() - start_time)),'.3f')
print(timetaken)

torch.save(model.state_dict(), 'roberta_state_dict_on_new_data_MAY2019.pth')

```
Here I am trying to run the code on Multiple GPUs(2-P100). But I keep getting this warning:
```

A sequence with no special tokens has been passed to the RoBERTa model. This model
requires special tokens in order to work. Please specify add_special_tokens=True in
 your encoding.

```

I am not sure, what causing this issue. But when I run it using single (i.e: after removing DataParallel wrapper), it doesn't give this warning.

Any help would be appreciated.

-thanks.


BertTokenizer provides wrong encode function for Japanese BERT

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BertTokenizer

Language I am using the model on (English, Chinese....): Japanese

I tried to load the tokenizer for Bert from pretrained [Bert for Japanese](http://nlp.ist.i.kyoto-u.ac.jp/index.php?BERT%E6%97%A5%E6%9C%AC%E8%AA%9EPretrained%E3%83%A2%E3%83%87%E3%83%AB). The tokenizer recognize "が" the same as "か"　although both appears in vocab file. Here is example code:

```
from pytorch_transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('vocab.txt')
token_ids = tokenizer.encode('が')
print('token_ids: ', token_ids)
print(tokenizer.decode(token_ids))

token_ids = tokenizer.encode('か')
print('token_ids: ', token_ids)
print(tokenizer.decode(token_ids))
```
the result:
```
token_ids:  [90]
か
token_ids: [90]
か
```
The vocab.txt file can be downloaded from [here](https://drive.google.com/open?id=1f3k9GcyqEIjjFo8EgqqaOQmiSXxT1hqF):
I also found that BertTokenizer also mis-recognized: 'て' and 'で', 'ば' and 'は'


How to predict missing word [MASK] using Robert

I am reading the docs and I still cannot figure out how to I predict missing word in a sentence using Robert. With bert this is described at https://huggingface.co/pytorch-transformers/quickstart.html

    # Mask a token that we will try to predict back with `BertForMaskedLM`
    masked_index = 8
    tokenized_text[masked_index] = '[MASK]'
    assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a',     'puppet', '##eer', '[SEP]']

Robert example is something I do not understand. What is the output of that?

    import torch
    from pytorch_transformers import RobertaTokenizer, RobertaForMaskedLM

    tokenizer = RobertaTokenizer.from_pretrained('roberta-
    base',cache_dir="/var/software/Models/robert/")
    model = RobertaForMaskedLM.from_pretrained('roberta-
    base',cache_dir="/var/software/Models/robert/")
    input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
    outputs = model(input_ids, masked_lm_labels=input_ids)
    loss, prediction_scores = outputs[:2]
    print("prediction_scores",prediction_scores,len(prediction_scores))

     output:
    A sequence with no special tokens has been passed to the RoBERTa model. This model requires
    special tokens in order to work. Please specify add_special_tokens=True in your encoding.
    prediction_scores tensor([[[33.6519, -3.9080, 24.2591,  ...,  2.8165,  4.9966, 12.8938],
         [ 5.8086, -4.2237, 16.1383,  ..., -1.0431, -0.8348,  3.5343],
         [ 0.3336, -4.1881, 10.7825,  ...,  0.7295,  0.9056,  3.7928],
         [ 0.2897, -4.4614,  8.1219,  ..., -3.9978,  0.1261, -1.4313],
         [ 3.3684, -4.0727, 10.7862,  ...,  1.7704, -2.2975,  3.9174],
         [ 2.0526, -4.9519, 18.1501,  ..., -4.2190, -5.0759,  1.4990]]],
       grad_fn=<AddBackward0>) 1


Remove unnecessary use of FusedLayerNorm

Fix #1172


How to preprocess my own data to use RoBERTa of Multiple GPUs

Hey,

I am bit naive using deep learning of text-classification, my data **(.csv)** consist of basically two columns:

- Text
- Labels

As per basic objective, model should take unseen text and predict label _(variable y)_ from the trained labels.

**I followed this tutorial to train RoBERTa algorithm:**
- [https://colab.research.google.com/drive/1xg4UMQmXjDik3v9w-dAsk4kq7dXX_0Fm](https://colab.research.google.com/drive/1xg4UMQmXjDik3v9w-dAsk4kq7dXX_0Fm)

Here the input format is universal (Train.tsv and Test.tsv) with 2 columns (which I mentioned above). The only problem is that this code doesn't utilize the multiple GPUs **(I even tried the DataParallel wrapper)**.
Somehow, I found the repository of the pytorch-transformer where they have given an example of how to utilize multiple GPUs and train the RoBERTa model i.e:

- [https://github.com/huggingface/pytorch-transformers/tree/master/examples](https://github.com/huggingface/pytorch-transformers/tree/master/examples)
This repository example takes the input-data in the wiki.text data format and they have provided the link [ Pretraining RoBERTa using your own data ](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md#pretraining-roberta-using-your-own-data) and **yet I don't find it of any use as it just talks about the format what they think is right not about the transforming of the standard format (i.e: .CSV)**

Here is the sample:
```
 = Valkyria Chronicles III =

 Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to as Valkyria Chronicles III outside Japan , is a tactical role @-@ playing video game developed by Sega and Media.Vision for the PlayStation Portable . Released in January 2011 in Japan , it is the third game in the Valkyria series . <unk> the same fusion of tactical and real @-@ time gameplay as its predecessors , the story runs parallel to the first game and follows the " Nameless " , a penal military unit serving the nation of Gallia during the Second Europan War who perform secret black operations and are pitted against the Imperial unit " <unk> Raven " .

```

I mean, why? do they have to use this format, why can't they go with standard format of classification algorithm? and if they want to then why they don't provide the detail regarding the data and adjusting the personal data corresponding to the algorithm intake?

Just wanted to know, how to preprocess the data according to multiple GPUs RoBERTa example, any help would be appreciated.

thanks.


Add option to use a 'stop token'

This will be used to truncate the output text to everything till right before the 'stop token'. If the 'stop token' is not found, then the whole text will be returned based on the specified 'length'.


In BertForSequenceClassification, why is loss initialised in every forward?

Looking at [the source](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L902-L910) I can see that the correct loss function is initialized in each call to forward.

https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L902-L910

Can you explain why? Why isn't the loss function set up as part of `init()`? Is there any advantage of always re-initialising it on each forward?

Edit: I see that you do this in other parts as well, e.g. the ReLU layer in distilbert: https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_distilbert.py#L598


RoBERTa : add_special_tokens=True

I set add_special_tokens=True
but I still get:
A sequence with no special tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.


Redundant sep_token_extra option for RoBERTa Fine-tuning

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): RoBERTa

Language I am using the model on (English, Chinese....): English

## Context

I was reading the code on RoBERTa fine-tuning and noticed the [`sep_token_extra` option](https://github.com/huggingface/pytorch-transformers/search?q=sep_token_extra&unscoped_q=sep_token_extra), which looks like a misinterpretation of a sentence from the original paper.

The current implementation [added an extra `[SEP]` to the RoBERTa input compared to BERT](https://github.com/huggingface/pytorch-transformers/blob/d8923270e6c497862f990a3c72e40cc1ddd01d4e/examples/utils_glue.py#L453), which seems wrong. Check out:

1. The [Facebook language model format](https://github.com/pytorch/fairseq/blob/e75cff5f2c1d62f12dc911e0bf420025eb1a4e33/fairseq/data/legacy/masked_lm_dataset.py#L193)
2. A related [Twitter discussion](https://twitter.com/VictoriaLinML/status/1175596109009321986)

The tasks I am working on is:
Fine-tuning RoBERTa on downstream tasks

## Code Sample
https://github.com/huggingface/pytorch-transformers/search?q=sep_token_extra&unscoped_q=sep_token_extra


Best loss

I am building a classifier by adopting codes from `run_glue.py`. There is plenty of optimization in training and tuning hyperparameters. Could anyone explain the difference between loss, tr_loss and logging_loss in these parts?
https://github.com/huggingface/pytorch-transformers/blob/a2d4950f5c909f7bb4ea7c06afa6cdecde7e8750/examples/run_glue.py#L120
https://github.com/huggingface/pytorch-transformers/blob/a2d4950f5c909f7bb4ea7c06afa6cdecde7e8750/examples/run_glue.py#L134
Thank you.


Planned support for new Grover 1.5B models?

Thanks for the great repo.

Just wondering if there's any planned support for the new Grover 1.5B models?

https://github.com/rowanz/grover (original 1.5B now available via download_model.py)

https://github.com/vanyacohen/opengpt2-1.5B-gpu-inference (slightly different variation)

Cheers


mask_tokens sometimes masks special tokens

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): RoBERTa

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details) run_lm_finetuning
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

the mask_token function in run_lm_finetuning script sometimes masks the special tokens.  This leads roberta model to throw a warning message:
A sequence with no special tokens has been passed to the RoBERTa model. "
                           "This model requires special tokens in order to work. "
                           "Please specify add_special_tokens=True in your encoding

I would prevent the first and last tokens being masked by adding the below lines immediately after masked_indices are calculated.

masked_indices[:, 0] = False
masked_indices[:, -1] = False

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Which model is best to used for language model rescoring for ASR

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello all, I am wanting to use this library to rescore an output from an automatic speech recognition model. Still learning a lot about language model so out of curiosity for anyone who's tried, which model has given you the best performance? Looking to a language model that can predict score the probability of a sentence most effectively.


Dataset format and Best Practices For Language Model Fine-tuning

## ❓ Questions & Help

Hi, thanks for making this code base available!

I have two questions, one on the input format of for fine-tuning the language model on custom dataset, and one on (unreasonably-)long data preprocessing time. Thanks in advance for any help!

- I'm trying to fine-tune the BERT Model on an extra dataset, and am using the `run_lm_finetuning.py` script in the `examples/` directory. However, I'm having trouble locating instructions on the proper format of the input data. There used to be some instructions in the `examples/lm_finetuning/` directory, but they seem deprecated now.

- As a start, I followed the `run_lm_finetuning.py` example and changed nothing but `--train_data_file` argument with a bigger text file (arbitrary format). The training, however, hangs on the data preprocessing part for about 10 hours, and the last standard output is shown below.

```
pytorch_transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (164229992 > 512). Running this sequence through the model will result in indexing errors
```


max_len_single_sentence should be max_len - 2 for RoBERTa

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): RoBERTa

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details) run_lm_finetuning
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details) . Language model finetuning

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
While using language model finetuning for roberta-base, I got an error cublas runtime error : resource allocation failed at /pytorch/aten/src/THC/THCGeneral.cpp:216.

When I checked the tokenized dataset, I observed that it had 514 tokens, i.e. 512 coming from max_len_single_sentence plus 2 special tokens.  RoBERTa tokenizer should have max_len_single_sentence set to 510 just like the one in BERT.

max_len_single_sentence = max_len - 2

## Environment

* OS: Ubuntu
* Python version: 3.7
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): Master
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->
R


Getting an unexpected EOF when trying to download 'bert-large-uncased-whole-word-masking-finetuned-squad' model.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details): BertForQuestionAnswering
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name): SQuaD
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. from pytorch_transformers import BertForQuestionAnswering
2. model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Rectified Adam + LARS

## 🚀 Feature
There has been a lot of buzz around the new Radam and Ralamb (Radam + LARS) optimizers, and I was wondering if it could also be implemented in pytorch-transformers.

## Motivation

It seems to have consistent performance improvements. It also seems to handle different learning rates a lot better. And with LARS it also allows for really large batch sizes without regressing.

## Additional context

https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20
https://github.com/mgrankin/over9000
https://twitter.com/jeremyphoward/status/1162118545095852032
https://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b


RBERT implementation

As per #1250, this PR describes an additional classification head for BERT for relationship classification tasks. This work is originally documented in [this paper](https://arxiv.org/pdf/1905.08284.pdf). In addition, the new head can be used with RoBERTa, producing a new SOTA as far as I know....

I have included a new example script and associated utils file that demonstrate how it can be used:
```run_semeval.py```, and updated the README.md in ```examples``` accordingly.

Note, contrary to what I said in the original issue, there is no need for new tokenisation classes - rather, strings simply need to be preprocessed with entity delimiting characters prior to tokenisation, and the input ID's of these characters passed to the classification head (see included example for details)


❓ Why the criterion of XLNet LMHeadModel use ignore_index = -1 ?

In the XLNetLMHeadModel, the criterion used to compute the loss uses `ignore_index=-1` :

https://github.com/huggingface/pytorch-transformers/blob/9f995b99d4c4067662c3bd4f1274315c0839deeb/pytorch_transformers/modeling_xlnet.py#L927-L931

**Why ?**
Isn't it supposed to ignore the padding index ID, i.e. 5 ?


What is the best CPU inference acceleration solution for BERT now?

Thank you very much.
Thank you very much.
Thank you very much.


fix annotation


add support for file I/O

sometimes we need to process multiple prompts from a file and generate multiple sequences. also, writing results to a file would be less verbose and fast.


Added ValueError for duplicates in list of added tokens

Very small addition to raise an error if the list of tokens passed to `add_tokens` contains duplicates. This otherwise raises cryptic errors down the line. Happy to update it to `Warning` if someone believes there's any reason for duplicates to be allowed here.


Where are BERT's pretrained Embeddings loaded?

I am trying to better understand the difference between the different types of embeddings that BERT uses (from the BERT paper: token, segment, position). For this purpose, I was hoping to put some print statement in the `pytorch_transformers` source code to see how the IDs change into vector representations for each type of embedding.

First of all I am confused about the embeddings that `pytorch_transformers` uses. Going through the source code for [`BertEmbeddings`](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L142-L171) I can see

- word embeddings
- position embeddings
- token type embeddings

What are these _token type_ embeddings? Are they the same as segment embeddings?

Secondly, during my quest for better understanding what's going on, I couldn't figure out where the pretrained embedding models are loaded, or even where they are downloaded. I am curious to see the vocab list of all types of embeddings, but I couldn't find them anywhere.

Any pointers?


Delete n_special reference in docstring

I don't think the `n_special` param is used, even in `**kwargs`.


cannot import name 'XLNetForMultipleChoice' but python can import

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert):

Language I am using the model on (English):

when I use the following command to run run_multiple_choice.py, like:
'''
python examples/run_multiple_choice.py --model_type bert --task_name race --model_name_or_path bert_large --do_train --do_eval --do_lower_case --data_dir $RACE_DIR --learning_rate 5e-5 --num_train_epochs 3 --max_seq_length 80 --output_dir models_bert/race_base --per_gpu_eval_batch_size=16 --per_gpu_train_batch_size=16 --gradient_accumulation_steps 2 --overwrite_output
'''
it gives the error information:
![image](https://user-images.githubusercontent.com/18585628/65219953-479b2e00-daec-11e9-9ecf-fd85bc8e5e34.png)
but when I in my python environment to import the package , it has no problem!
![image](https://user-images.githubusercontent.com/18585628/65220000-626da280-daec-11e9-935b-4cd951474b39.png)
what's wrong with the run_multiple_choice.py?
## Environment

* OS:
* Python version: 3.6.2
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? yes
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Fine Tuning GPT2 on wikitext-103-raw

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Running pytorch-transformers\examples\run_lm_finetuning.py. This is stuck at  load_and_cache_examples step. I just see message like
  WARNING - pytorch_transformers.tokenization_utils -   This tokenizer does not make use of special tokens. The sequence has been returned with no modification.
The train file has 1.8M rows with this rate it would take few days to just tokenize and cache training data.
Is this expected?  Did anyone face this before?

Thanks in advance for your help.


traced_model

when I ran: traced_model = torch.jit.trace(model, (input_ids,))
I got:
/home/jhy/py3.6/lib/python3.6/site-packages/torch/tensor.py:389: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  'incorrect results).', category=RuntimeWarning)
Traceback (most recent call last):
  File "/home/jhy/project/xlnet/src/xlnet_test.py", line 13, in <module>
    traced_model = torch.jit.trace(model, (input_ids,))
  File "/home/jhy/py3.6/lib/python3.6/site-packages/torch/jit/__init__.py", line 772, in trace
    check_tolerance, _force_outplace, _module_class)
  File "/home/jhy/py3.6/lib/python3.6/site-packages/torch/jit/__init__.py", line 904, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)
RuntimeError: Tracer cannot infer type of (tensor([[[-0.9993,  0.2632, -0.6305,  ..., -0.3520, -1.2041, -1.5944],
         [ 4.5358,  2.6032, -1.4790,  ...,  2.1211,  1.6621, -0.9913],
         [ 2.0586,  2.1398,  0.6811,  ...,  1.9191,  0.0836, -1.2848],
         ...,
         [-1.4818,  0.5329,  0.5212,  ...,  0.6176,  1.7843, -1.8773],
         [-2.8784,  1.9871,  0.5379,  ...,  1.3778,  1.0554, -1.3039],
         [-4.1723,  1.3071,  0.6565,  ...,  1.2515,  1.6618, -0.8640]]],
       grad_fn=<PermuteBackward>), (None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None))
:Cannot infer type of a None value (toTraceableIValue at /pytorch/torch/csrc/jit/pybind_utils.h:268)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f8ea599c273 in /home/jhy/py3.6/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x44e288 (0x7f8ea69db288 in /home/jhy/py3.6/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #2: <unknown function> + 0x4bdda2 (0x7f8ea6a4ada2 in /home/jhy/py3.6/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #3: <unknown function> + 0x4d1d81 (0x7f8ea6a5ed81 in /home/jhy/py3.6/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x1d3ef4 (0x7f8ea6760ef4 in /home/jhy/py3.6/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: _PyCFunction_FastCallDict + 0x288 (0x566ad8 in /home/jhy/py3.6/bin/python)
frame #6: /home/jhy/py3.6/bin/python() [0x5067b0]
frame #7: _PyEval_EvalFrameDefault + 0x4de (0x50729e in /home/jhy/py3.6/bin/python)
frame #8: /home/jhy/py3.6/bin/python() [0x504232]
frame #9: /home/jhy/py3.6/bin/python() [0x505e83]
frame #10: /home/jhy/py3.6/bin/python() [0x5066f0]
frame #11: _PyEval_EvalFrameDefault + 0x4de (0x50729e in /home/jhy/py3.6/bin/python)
frame #12: /home/jhy/py3.6/bin/python() [0x504232]
frame #13: /home/jhy/py3.6/bin/python() [0x505e83]
frame #14: /home/jhy/py3.6/bin/python() [0x5066f0]
frame #15: _PyEval_EvalFrameDefault + 0x4de (0x50729e in /home/jhy/py3.6/bin/python)
frame #16: /home/jhy/py3.6/bin/python() [0x504232]
frame #17: PyEval_EvalCode + 0x23 (0x6022e3 in /home/jhy/py3.6/bin/python)
frame #18: /home/jhy/py3.6/bin/python() [0x647fa2]
frame #19: PyRun_FileExFlags + 0x9a (0x64806a in /home/jhy/py3.6/bin/python)
frame #20: PyRun_SimpleFileExFlags + 0x197 (0x649d97 in /home/jhy/py3.6/bin/python)
frame #21: Py_Main + 0x5c2 (0x63c352 in /home/jhy/py3.6/bin/python)
frame #22: main + 0xe9 (0x4dbcb9 in /home/jhy/py3.6/bin/python)
frame #23: __libc_start_main + 0xf0 (0x7f8eabcff830 in /lib/x86_64-linux-gnu/libc.so.6)
frame #24: _start + 0x29 (0x5cb639 in /home/jhy/py3.6/bin/python)


MemoryError on run_lm_finetuning.py

Previous versions of finetune_on_pregenerated.py had a `--reduce_memory` parameter to keep memory requirements from going overboard, it seems like it is no longer available in the new run_lm_finetuning.py file?


Adding Adapters

From: https://arxiv.org/pdf/1902.00751.pdf

Open to feedback!

* Implementing adapters requires a couple more hyperparameters that need to go into the BertConfig. Do let me know if there is an alternative to modifying the core Config object (maybe a subclass would work better?)
* If `use_adapter` is False, the adapter modules are not created, so there should be no issue with changes in `state_dicts`/weights if they're not enabled.
* Added a utility function for extracting the adapter parameters from the model, to facilitate tuning only the adapter layers. In practice, a user should tune the adapter layers (+layer norm) and the final classifier layers, the latter of which varies depending on the model.
* I believe this should work seamlessly with RoBERTa.


Typo with LM Fine tuning script

Typo with LM Fine tuning script


TransfoXLLMHeadModel compatibility with pytorch 1.1.0

TransfoXLLMHeadModel._forward uses torch.Tensor.bool, which is not present in pytorch 1.1.0


Evaluation result.txt path suggestion

## 🚀 Feature

At pytorch-transformers/examples/**run_lm_finetuning**.py and **run_glue**.py
There is a line ```output_eval_file = os.path.join(eval_output_dir, "eval_results.txt")```

When setting evaluate_during_training **True**, `output_eval_file` will keep being overwritten.

I think `output_eval_file` can be assigned like

 `output_eval_file = os.path.join(eval_output_dir, prefix, "eval_results.txt")`

Meanwhile, in the main() function

```result = evaluate(args, model, tokenizer, prefix=global_step)```

change into
```
result = evaluate(args, model, tokenizer, prefix=checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else "")
```
Just a little suggestion


GPT2 Tokenizer Decoding Adding Space

## 🐛 Bug

The GPT-2 tokenizer's decoder now adds a space at the beginning of the string upon decoding.

(Potentially causing #1254)

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Run the following code:

```bash
from pytorch_transformers.tokenization_gpt2 import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.decode(tokenizer.encode("test phrase"))
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

The expected decoded string is "test phrase". However, currently it produces " test phrase".

## Environment

* OS: OSX
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): master (#e768f2322abd2a2f60a3a6d64a6a94c2d957fe89)

* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:


Fix fp16 masking in PoolerEndLogits

Necessary to run xlnet squad fine-tuning with `--fp16 --fp16_opt_level="O2"`, otherwise loss is immediately `NaN` and fine-tuning cannot proceed.

Similar to #1249


Is training from scratch possible now?

Do the models support training from scratch, together with original (paper) parameters?


start_position=0 in utils_squad.py when span is impossible

Hi,

https://github.com/huggingface/pytorch-transformers/blob/e768f2322abd2a2f60a3a6d64a6a94c2d957fe89/examples/utils_squad.py#L340-L351

when answer is out of span, start_position should be cls_index rather than 0 as L350

And in

https://github.com/huggingface/pytorch-transformers/blob/e768f2322abd2a2f60a3a6d64a6a94c2d957fe89/examples/run_squad.py#L253-L259

when using multi gpu and set evaluate_during_training=True, we may add
`model_tmp = model.module if hasattr(model, 'module') else model` in order to get `model_tmp.config. start_n_top` rather than `model.config.start_n_top`


FineTuning using single sentence document

Hello

roBERTa is not using the next sentence prediction objective. I want to fine-tune the pre-trained model on an unlabelled corpus of domain-specific text (ULMFIT style intermediate pretraining).
The bottleneck is my examples are single short sentences, instead of a document with multiple sentences

The INPUT format here says it requires a file with one sentence per line and one blank line between documents. For me, each document has a single sentence
https://github.com/huggingface/pytorch-transformers/tree/b62abe87c94f8df4d5fdc2e9202da651be9c331d/examples/lm_finetuning

The last time I raised an issue this was an expected behavior I know
Re : https://github.com/huggingface/pytorch-transformers/issues/272

How to do it now :) ? Any help is appreciated .


connection limit of pregenerate_training_data.py

## ❓ Questions & Help

Inside pregenerate_training_data.py, one can use multiprocessing to process each epoch in parallel.

However, since the communication between process is limited by pickle limit size. We can only transfer arguments less than 1GB and argument: docs are very likely to exceed this limit.

I save the docs as pickle file and ask every process to read on their own.


'Default process group is not initialized' Error

I get this error when finetuning bert using code in `lm_finetuning/` folder, when I try to run it several GPUs.
```
Traceback (most recent call last):
  File "finetune_on_pregenerated.py", line 330, in <module>
    main()
  File "finetune_on_pregenerated.py", line 323, in main
    if  n_gpu > 1 and torch.distributed.get_rank() == 0  or n_gpu <=1 :
  File "/opt/anaconda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/opt/anaconda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized
```

sample training data(*epoch_0.json*):
```
{"tokens": ["[CLS]", "i", "'", "ve", "got", "the", "[MASK]", "scenario", "in", "mind", "do", "you", "[MASK]", "[MASK]", "##k", "[SEP]", "i", "prefer", "that", "over", "red", "##dit", "[SEP]"], "segment_ids": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1], "is_random_next": false, "masked_lm_positions": [6, 12, 13], "masked_lm_labels": ["perfect", "have", "ki"]}
{"tokens": ["[CLS]", "she", "message", "##d", "me", "suggesting", "i", "was", "ignorant", "because", "i", "[MASK]", "##t", "know", "the", "feeling", "and", "restriction", "that", "panties", "[MASK]", "on", "women", "[MASK]", ".", ".", ".", "seriously", ".", ".", "[MASK]", "panties", "she", "[MASK]", "[MASK]", "men", "don", "##t", "know", "how", "bad", "it", "is", "to", "wear", "panties", "because", "society", "[MASK]", "##t", "let", "women", "speak", "up", "about", "it", "[SEP]", "[MASK]", "yourself", "lucky", ".", ".", "[MASK]", "bullet", "dodged", "[SEP]"], "segment_ids": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1], "is_random_next": false, "masked_lm_positions": [11, 20, 22, 23, 30, 33, 34, 48, 57, 62], "masked_lm_labels": ["didn", "put", "women", ".", ".", "said", "that", "won", "consider", "."]}
{"tokens": ["[CLS]", "[MASK]", "enough", "my", "first", "name", "[MASK]", "actually", "lisa", "i", "[MASK]", "##t", "ha", "[MASK]", "minded", "[MASK]", "b", "##ds", "##m", "vi", "##ds", "at", "12", "13", "not", "made", "to", "do", "actual", "sex", "like", "u", "said", "but", "the", "[MASK]", "displayed", "play", "whipped", "on", "[MASK]", "vi", "##ds", "i", "think", "the", "pe", "##dos", "[MASK]", "'", "ve", "enjoyed", "watching", "[MASK]", "[SEP]", "this", "probably", "[MASK]", "[MASK]", "'", "t", "what", "[MASK]", "had", "in", "mind", "though", "sorry", "but", "i", "thought", "it", "funny", "when", "the", "first", "word", "was", "lisa", "an", "that", "'", "s", "my", "emil", "[SEP]"], "segment_ids": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], "is_random_next": false, "masked_lm_positions": [1, 6, 10, 13, 15, 35, 40, 48, 53, 57, 58, 62, 84], "masked_lm_labels": ["funny", "is", "would", "##v", "doing", "being", "the", "would", "them", "is", "n", "u", "name"]}
```

*epoch_0_metrics.json*
```
{"num_training_examples": 3, "max_seq_len": 256}
```

Reproducing:
```
export CUDA_VISIBLE_DEVICES=6,7

python3 finetune_on_pregenerated.py --pregenerated_data training/ --bert_model bert-base-uncased --do_lower_case --output_dir finetuned_lm/ --epochs 1 --train_batch_size 16
```
The code works fine when training on one single GPU.

Thanks.


No language embedding weights in pre-trained xlm models.

I'm trying to train a one-shot classification model using the given XLM pre-trained weights.
However, I noticed that for both `xlm-mlm-17-1280` and `xlm-mlm-100-1280`, That I kept receiving the warning `weights of XLMForSequenceClassification not initialized from pre-trained model: ['lang_embeddings.weight']`. I then looked into the state_dict of those two checkpoints and saw that indeed there were no weights that matched that key. Should that weight exist in the state_dict somewhere?


Write with Transformer: Please, add an autosave to browser cache!

## 🚀 Feature

In Write with Transformer, the writing should be periodically saved to the browser cache, so that if the user accidentally refreshes the page, their work that they may have spent hours on won't be lost.

## Motivation

I just lost several hours worth of writing because I accidentally refreshed the page.

## Additional context

:(


Implement fine-tuning BERT on CoNLL-2003 named entity recognition task

I added a script for fine-tuning BERT on the CoNLL-2003 named entity recognition task, as an example for token classification. This was requested in #1216.
I followed the structure of the run_glue example, and implemented the data processing in a way suitable for all transformer models (although currently token classification is only implemented for BERT).
The training procedure is as described in the original BERT paper: Only the first sub-token of each CoNLL-tokenized word is classified and contributes to the loss, the remaining sub-tokens are ignored.


Fixes #1263, add tokenization_with_offsets, gets tokens with offsets in the original text

This is similar to the utils_squad approach to getting offsets for the tokens but can also be used in other places where the tokens should have a correspondence to the original text to fix #1263.


ModuleNotFoundError: No module named 'pytorch_transformers.modeling' using convert_pytorch_checkpoint_to_tf.py

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
BERT
Language I am using the model on (English, Chinese....):
Chinese
The problem arise when using:
* [✔ ] the official example scripts: (give details)
run convert_pytorch_checkpoint_to_tf.py to generate the tf check point

## To Reproduce

Steps to reproduce the behavior:

```
python3  /Users/xxx/py3ml/lib/python3.6/site-packages/pytorch_transformers/convert_pytorch_checkpoint_to_tf.py
    Traceback (most recent call last):
  File "/Users/xxx/py3ml/lib/python3.6/site-packages/pytorch_transformers/convert_pytorch_checkpoint_to_tf.py", line 23, in <module>
    from pytorch_transformers.modeling import BertModel
ModuleNotFoundError: No module named 'pytorch_transformers.modeling'
```

## Expected behavior
convert it
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: macOS 10.14
* Python version: 3.6
* PyTorch version: 1.2
* PyTorch Transformers version (or branch):1.2.0
* Using GPU ? no
* Distributed of parallel setup ? no
* Any other relevant information:
I change the code to
`from pytorch_transformer import BertModel`
It works fine

## Additional context

<!-- Add any other context about the problem here. -->


How long does it take? (BERT Model Finetuning using Masked ML objective)

I am about to finetune a multilingual BERT model using English and Chinese text from the legal domain.

My corpus is around 27GB, how long should I expect to train 3 epochs (default parameters) using a Google TPU?


get NaN loss when I run the example code run_squad.py

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I use the example run_squad.py code, and use the Readme's hyper-parameters, but I got nan loss when I trained a few batches. And I use the `autograd.detect_anomaly()` want to catch that.

The more information is below:
> File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
self._target(*self._args, **self._kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 1211, in forward
    attention_mask=attention_mask, head_mask=head_mask)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 713, in forward
    head_mask=head_mask)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 434, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 414, in forward
    intermediate_output = self.intermediate(attention_output)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 386, in forward
    hidden_states = self.intermediate_act_fn(hidden_states)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 145, in gelu
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
Traceback (most recent call last):
  File "run_squad.py", line 544, in <module>
    main()
  File "run_squad.py", line 490, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 165, in train
    loss.backward()
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.

My environment PyTorch version: torch 1.1.0, pytorch-transformers 1.2.0, and use 4 Titan X gpus for train. I don't know why the official code cause this result, could someone help me about that...


BERT returns different embedding for same sentence

I am using pre-trained BERT for creating features, for same sentence it produces different result in two different runs. Do we have to set some random state to produce consistent result? I am using pytorch-transformers for reading pre-trained model.


could you add an option to transfer variables from float32 to float16 in GPT2 model to reduce model size and accelerate the inference speed

## 🚀 Feature
could you add an option to transfer variables from float32 to float16 in GPT2 model
## Motivation
reduce model size and accelerate the inference speed
## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


How to use pytorch-transformers for transfer learning?

## ❓ Questions & Help

```python
pretrained_weights='bert-base-uncased'
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)

features = []
input_ids = torch.tensor([self.tokenizer.encode(phrase, add_special_tokens=True)])
with torch.no_grad():
    # Models outputs are tuples
    outputs = self.model(input_ids)
    last_hidden_states = outputs[0]
    avg_pool_hidden_states = np.average(last_hidden_states[0], axis=0))
    return avg_pool_hidden_states
```

I am working on Sentence Similarity problem, given sentence S, find similar sentences W. So I want to encode sentence S and find the closest top-K sentences from W.

I have read the documentations, but I want to confirm few things how do I do. the following:
- How do I get the second to last layer?
- Am I average pooling the last hidden states correctly?


Accuracy not increasing with BERT Large model

I experimented with  `BERT_base_cased` and `BERT_large_cased` model for multi class text classification. With BERT_base_cased, I got satisfactory results. When I tried with BERT_large_cased model, the accuracy is same for all the epochs

```
Epoch: 01 | Epoch Time: 0m 57s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.200893470219204	2.790178544819355	4.977107011354887	3.6057692021131516
Epoch: 02 | Epoch Time: 0m 57s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.085730476038797	2.287946455180645	4.954357807452862	3.6057692021131516
Epoch: 03 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.019492668764932	2.901785634458065	4.961122549497164	3.6057692021131516
Epoch: 04 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.0052995937211175	3.57142873108387	4.9535566843473	3.6057692021131516
Epoch: 05 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.003523528575897	3.23660708963871	4.9652618261484	3.6057692021131516
Epoch: 06 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.010107040405273	3.29241082072258	4.96296108686007	3.6057692021131516
Epoch: 07 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.028377030576978	2.678571455180645	4.94510478239793	3.6057692021131516
Epoch: 08 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.04387321642467	2.901785634458065	4.9411917466383715	3.6057692021131516
Epoch: 09 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.027528064591544	3.18080373108387	4.940045246711144	3.6057692021131516
Epoch: 10 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.023407867976597	3.29241082072258	4.940378886002761	3.6057692021131516
Epoch: 11 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.015415557793209	3.125	4.939220135028545	3.6057692021131516
Epoch: 12 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.018008896282741	3.29241082072258	4.9386150653545675	3.6057692021131516
Epoch: 13 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.003824523517063	2.957589365541935	4.938107490539551	3.6057692021131516
Epoch: 14 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.003440124647958	3.069196455180645	4.93824944129357	3.6057692021131516
Epoch: 15 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.012082431997571	3.069196455180645	4.9383643590486965	3.6057692021131516
Epoch: 16 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.009286454745701	3.01339291036129	4.93832148038424	3.6057692021131516
Epoch: 17 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.006769972188132	2.901785634458065	4.937925778902494	3.6057692021131516
Epoch: 18 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.006464583533151	3.125	4.937762847313514	3.6057692021131516
Epoch: 19 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.004164610590253	2.957589365541935	4.937491783728967	3.6057692021131516
Epoch: 20 | Epoch Time: 0m 57s
*******train_loss,train_acc,valid_loss,valid_ac**********
5.013612789767129	2.957589365541935	4.937890896430383	3.6057692021131516
Epoch: 21 | Epoch Time: 0m 58s
*******train_loss,train_acc,valid_loss,valid_ac**********
4.997398240225656	2.511160634458065	4.937900726611797	3.6057692021131516
```
With `BERT_base_cased`, there is no such problem. But with `BERT_large_cased`, why accuracy is same in all the epochs? Any help is really appreciated............. @thomwolf  @nreimers


Fine-tune distilbert-base-uncased under run_glue

## ❓ Questions & Help

I am a bit confused on how to fine-tune distilbert. I see three options

1. Fine-tune bert for the task and then use distillation.distiller
2. Fine-tune bert for the task and then use distillation.train
3. Fine-tune distilbert-base-uncased directly for the task using run_glue.py

I tried the 3rd option under run_glue.py as follows

add distilbert to MODEL_CLASSES
```MODEL_CLASSES = {
    'bert': (BertConfig, BertForSequenceClassification, BertTokenizer),
    'distilbert': (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
    'xlnet': (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer),
    'xlm': (XLMConfig, XLMForSequenceClassification, XLMTokenizer),
    'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer),
}
```
and add a flag `--model_type=distilbert`

Which of the three methods above should be used?


different results shown each time when I run the example code for BertForMultipleChoice

When I ran the following example provided for BertForMultipleChoice in the documentation, I've got different results each time when I run it. Does it mean that BertForMultipleChoice is only provided to fine-tune the BERT model with RocStories/SWAG like datasets, and no pretrained models (after the fine-tuning) are provided?

________________________
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertConfig
from pytorch_transformers import BertForNextSentencePrediction, BertForMultipleChoice
import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMultipleChoice.from_pretrained('bert-base-uncased')
model.eval()
choices = ["Hello, my dog is cute", "Hello, my cat is amazing"]
input_ids = torch.tensor([tokenizer.encode(s) for s in choices]).unsqueeze(0)  # Batch size 1, 2 choices
labels = torch.tensor(1).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, classification_scores = outputs[:2]
print(classification_scores)


Error running openai-gpt on ROCstories

## 🐛 Bug

<!-- Important information -->

The model I am using: OpenAIGPT

The language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: When I try to run examples/single_model_script/run_openai_gpt.py I get this error:

```
Traceback (most recent call last):
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 288, in <module>
    main()
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 158, in main
    model = OpenAIGPTDoubleHeadsModel.from_pretrained(args.model_name, num_special_tokens=len(special_tokens))
  File "/home/rohola/Codes/Python/pytorch-transformers/pytorch_transformers/modeling_utils.py", line 330, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'num_special_tokens'
```

The tasks I am working on is:
*  ROCstories

## To Reproduce

Steps to reproduce the behavior:

1. Just run the "run_openai_gpt.py "

## Environment

* OS: Ubuntu 16.04
* Python version: 3.6
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): The last commit
* Using GPU: True
* Distributed of parallel setup: No
* Any other relevant information:

## Additional context

Even when I remove that argument I get another error:
```
Traceback (most recent call last):
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 288, in <module>
    main()
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 224, in main
    losses = model(input_ids, mc_token_ids, lm_labels, mc_labels)
  File "/home/rohola/Codes/Python/pytorch-transformers/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rohola/Codes/Python/pytorch-transformers/pytorch_transformers/modeling_openai.py", line 601, in forward
    head_mask=head_mask)
  File "/home/rohola/Codes/Python/pytorch-transformers/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rohola/Codes/Python/pytorch-transformers/pytorch_transformers/modeling_openai.py", line 425, in forward
    hidden_states = inputs_embeds + position_embeds + token_type_embeds
RuntimeError: The size of tensor a (78) must match the size of tensor b (16) at non-singleton dimension 1
```


Offsets in original text from tokenizers

## 🚀 Feature

A new method for tokenizers: tokenize_with_offsets. In addition to returning the tokens, it returns the spans in the original text that the tokens correspond to.
After
tokens, offsets = tokenizer.tokenize_with_offsets(text)
then
tokens[i] maps to text[offsets[i, 0]:offsets[i, 1]]

## Motivation

I find it useful to be able to get the spans in the original text where the tokens come from. This is useful for example in extractive question answering, where the model predicts a sequence of tokens, but the user would like to see a highlighted passage.

## Additional context

I have a version of this in a fork: https://github.com/michaelrglass/pytorch-transformers
There is a test (regression_test_tokenization_with_offsets) that verifies the tokenization with offsets gives the same tokenization for many models - still working on XLM.
The test data I used is available from https://ibm.box.com/s/228183fe95ptn8eb9n0zq4i2y7picq4r

Since this touches several files, and works for the majority (but not all) of the tokenizers, I thought creating an issue for discussion would be better than an immediate pull request.


run_generation.py  'encode' error for gpt2 and xlnet

Hello:

Been using nshepperd's tf repo and various excellent forks for fine-tuning and inference without issue.

Wanted to check out py-torch transformers and compare. First test is simple conditional sampling from the pytorch models:

python3 run_generation.py --model_type=xlnet --length=20 --model_name_or_path=models/xlnet-large-cased

Loads config.json and model weights then Model prompt >>>

Enter anything: Hello there huggingface. What's up?

then ...

Traceback (most recent call last):
  File "run_generation.py", line 195, in <module>
    main()
  File "run_generation.py", line 175, in main
    context_tokens = tokenizer.encode(raw_text)
AttributeError: 'NoneType' object has no attribute 'encode'

Same error when switching to gpt2, in this case  gpt-large.

Is there a modification to the script required?  Is the terminal syntax incorrect?

Can't move on to fine-tuning until basic inference sorted out.

Tested both pytorch 1.2.0 and 1.1.0.  Same error.

    OS: ubuntu 18.04
    Python version: python 3.6.8
    PyTorch version: torch 1.2.0 tested and changed to 1.1.0 to match transformer
    PyTorch Transformers version (or branch): pytorch-transformers 1.2.0 tested and changed to 1.1.0 to match above
    Using GPU ? yes
    Distributed of parallel setup ? no

All help appreciated. Would like to test distributed fine-tuning but need the basic repo working first on local machine.

Cheers


SequenceSummary / quenstion regarding summary types

in the class SequenceSummary(nn.Module) which is a part of {BERT, XLNet}ForSequenceClassification:

https://github.com/huggingface/pytorch-transformers/blob/32e1332acf6fd1ad372b81c296d43be441d3b0b1/pytorch_transformers/modeling_utils.py#L643-L644

in case of XLNet we can see that the last token will be taken, for BERT it is otherwise the first one.
https://github.com/huggingface/pytorch-transformers/blob/32e1332acf6fd1ad372b81c296d43be441d3b0b1/pytorch_transformers/modeling_utils.py#L692

Why only the first one or the last one? Why not to apply the max or average pooling over all token from the output?

Thanks a lot!


XLNet tokenizer returns empty list instead of string for some indexes

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

XLNet

Language I am using the model on (English, Chinese....):

English

The problem arise when using:
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Create tokenizer: tokenizer = XLNetTokenizer.from_pretrained(path)
2. Try to decode index 4: tokenizer.decode(4)
3. You get empty list [] although expected is a string

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

Decode method of the tokenizer should return a string and not a list.

## Environment

* OS:
* Python version: 3.7
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.2.0


Cannot install the library

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 659, in unpack_http_url
    hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 882, in _download_http_url
    _download_url(resp, link, content_file, hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 603, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "/usr/lib/python2.7/dist-packages/pip/utils/hashes.py", line 46, in check_against_chunks
    for chunk in chunks:
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 571, in written_chunks
    for chunk in chunks:
  File "/usr/lib/python2.7/dist-packages/pip/utils/ui.py", line 139, in iter
    for x in it:
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 560, in resp_read
    decode_content=False):
  File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py", line 436, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py", line 384, in read
    data = self._fp.read(amt)
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/filewrapper.py", line 63, in read
    self._close()
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/filewrapper.py", line 50, in _close
    self.__callback(self.__buf.getvalue())
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/controller.py", line 275, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/serialize.py", line 87, in dumps
    ).encode("utf8"),
MemoryError


fix padding_idx of RoBERTa model

The padding index of the pretrained RoBERTa model is 1, and 0 is assigned to `<s>` token. The padding index of the current RoBERTa model is set to be 0, therefore `<s>` is treated as padding.
This PR aims to fix this problem.


Training time increased from 45 min per epoch to 6 hours per epoch in colab

## 📚 Migration

<!-- Important information -->

Model I am using (Bert): PyTorch-pretrained-Bert and BERT in pytorch-transformers

Language I am using the model on : English

The problem arise when using:
*  the official example scripts: run_squad.py in pytorch-transformers

The tasks I am working on is:
* an official GLUE/SQUaD task: Fine Tuning SQuAD

Details of the issue:

Hi finetuned model over SQuAD with the following code in colab:
%run run_squad.py --bert_model bert-base-uncased \
                  --do_train \
                  --do_predict \
                  --do_lower_case \
                  --fp16 \
                  --train_file SQUAD_DIR/train-v1.1.json  \
                  --predict_file SQUAD_DIR/dev-v1.1.json \
                  --train_batch_size 12 \
                  --learning_rate 3e-5 \
                  --num_train_epochs 2.0 \
                  --max_seq_length 384 \
                  --doc_stride 128 \
                  --output_dir debug_squad9

The above code took 45 min for one epoch with  "exact_match": 81.94891201513718, "f1": 89.02481046726041. I ran this code(PyTorch-pretrained-Bert) somewhere between July 20th - 27th 2019.

And now the same file with the above code(PyTorch-pretrained-Bert) is taking the 6 hours for one epoch. Why is that?

I have tested with pytorch-transformers as well, It is also taking 6 hours for one epoch.

I am unable to understand. Is there any change with apex or implementation ??

## Environment

* OS: google colab
* Python version: Python 3.6.8
* PyTorch version: 1.2
* PyTorch Transformers version (or branch): the existing version pytorch-transformers
* Using GPU ? - Yes I guess K80 single GPU
* Distributed of parallel setup ? No
* Any other relevant information:
- The above environment details on based on the exiting run which took 6 hours for one epoch.
- I haven't collected the environment information when I tested it for the first time over PyTorch-pretrained-Bert.

## Checklist

- [yes ] I have read the migration guide in the readme.
- [yes ] I checked if a related official extension example runs on my machine.

## Additional context

Is there anything new that I am missing from pytorch-Transformers or implementation change in PyTorch-pretrained-Bert ??


## 🚀 Feature

Could you please implement a Adafactor optimizer? :)

( https://arxiv.org/abs/1804.04235 )

## Motivation

In contrast to Adam it requires much less GPU memory.
I tried to use the FairSeq implementation for the pytorch-transformers, but I'm no expert and I couldn't get it done.

Could you please do that? :)

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->