[WIP] Refactor Tokenizers creation to support in-memory initialization

As pointed out in #916 currently tokenizers ask for the path from where they'll load the required vocabulary files.

This PR allows tokenizers to take their vocab from data living in-memory and cold-storage.

Implementations details:

- All tokenizer now have a specific ${TokenizerName}Vocab dataclass holding all the required information to run the model.
- All ${TokenizerName}Vocab dataclass provide a from_pretrained method in charge of reading necessary files
- All tokenizer now take as first argument vocabs which has to ${TokenizerName}Vocab instance.
- All model now have a static member vocab_class which points to the desired ${TokenizerName}Vocab data class

- Some ${TokenizerName}Vocab.from_pretrained share loading routines and thus code is currently duplicated across all of them. It might be possible to refactor to use a generic method that handles such loading.

- [x] Bert
- [x] Transformer XL
- [x] GPT
- [x] GPT-2
- [x] XLNet
- [x] XLM
- [x] RoBERTa
- [x] DistilBERT


fix adding special tokens

Currently there is a bug when adding `additional_special_tokens` in the form of a tuple, instead of a list. To reproduce:

```python
from pytorch_transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
tokenizer.add_special_tokens({"additional_special_tokens": ("@a@", "@b@")})
tokenizer.all_special_tokens
```

Results in:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-81549ce398a5> in <module>()
----> 1 tokenizer.all_special_tokens

~/GitHub/pytorch-transformers/pytorch_transformers/tokenization_utils.py in
all_special_tokens(self)
    677         set_attr = self.special_tokens_map
    678         for attr_value in set_attr.values():
--> 679             all_toks = all_toks + (attr_value if isinstance(attr_val
ue, (list, tuple)) else [attr_value])
    680         all_toks = list(set(all_toks))
    681         return all_toks

TypeError: can only concatenate list (not "tuple") to list
```


Idea to improve DistilBERT

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

Based on https://medium.com/huggingface/distilbert-8cf3380435b5, you are using KL_loss to train the student from the teacher.

You can do something a little bit different there.

Loss = KL_loss(teacher, student when teacher is right) and CE(when the teacher is wrong).
Therefore, the student should converge to be better than the teacher by no learning from its mistake.

Or even better, you could correct the teacher by artificially telling him the GT;
By replacing in the teacher prediction the GT class probability value by 1 and then renormalizing
by the sum of probs (here: some of sort of artificial smooth).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

Improve DistillBert performance.

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


What is the relationship between `run_lm_finetuning.py` and the scripts in `lm_finetuning`?

## ❓ Questions & Help

It looks like there are now two scripts for running LM fine-tuning. While `run_lm_finetuning` seems to be newer, the documentation in `lm_finetuning` seems to indicate that there is more subtlety to generating the right data for performing LM fine-tuning in the BERT format. Does the new script take this into account?

Sorry if I'm missing something obvious!


Closing bracket is missing in token_counts.py for DistilBERT

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): DistilBERT

The problem arise when using:
* [x] the official example scripts: (give details)

I'm trying DistilBERT, and get an "invalid syntax error" when I run `examples/distillation/scripts/token_counts.py`.

A closing bracket seems missing at line 27.

```
    parser.add_argument("--data_file", type=str, default="data/dump.bert-base-uncased.pickle",
                        help="The binarized dataset."
```

## Environment

* OS: Linux
* Python version: 3.6.9
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 1.1.0
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:


Documentation auto-deploy

Documentation is now deployed automatically. @thomwolf @julien-c


GPT2-large fails to load the tokenizer

## 🐛 Bug

Using: GPT2-Large

## To Reproduce
When I load the gpt2-large model, in the same way as gpt2 and gpt2-medium I get an NoneType when loading the tokenizer.

```
self.tokenizer = GPT2Tokenizer.from_pretrained("gpt2-large", bos_token="_start_", unk_token='_unk_', eos_token="_eos_", sep_token="_delimiter_", cls_token="_classify_", pad_token='_pad_' )
self.model = GPT2LMHeadModel.from_pretrained("gpt2-large")
```
At this point in the debugger self.tokenizer == None is True
The issue is clear when I try to use the tokenizer.
```
File "gpt2_train.py", line 70, in load
    num_added_toks = self.tokenizer.add_special_tokens(special_tokens_dict)
AttributeError: 'NoneType' object has no attribute 'add_special_tokens'
```

## Expected behavior

Get a tokenizer object

## Environment

* OS: Linux
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0 (last github pull)
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information:


Attention values occasionally exceed 1 in BertModel

Core: Modeling

Internals of the library; Models.

```Python
outputs = self.model(x, attention_mask = x_mask)  # Models outputs are now tuples
print(outputs[2][-1].max())
print((outputs[2][-1]>1).sum().item()) # Number of attention values > 1
print((outputs[2][-1]>-1).sum().item()) # Total number of attention values
```
```
tensor(1.0750, device='cuda:7', grad_fn=<MaxBackward1>)
1545
480000
```


How to finetune GPT2

## ❓ Questions & Help

Hi all,

I would like to finetune the pretrained gpt2 model with a newspapers dataset. Do you know how would that be possible? I haven't found any train scipt for gpt2.

Thanks a lot.


where can i assign step in function lr_lambda of Class WramupLinearSchedule?

As i know LambdaLR's get_lr function recieve last_epoch as parameter, so where does it get the step of current training?

class WarmupLinearSchedule(LambdaLR):
    """ Linear warmup and then linear decay.
        Linearly increases learning rate from 0 to 1 over `warmup_steps` training steps.
        Linearly decreases learning rate from 1. to 0. over remaining `t_total - warmup_steps` steps.
    """
    def __init__(self, optimizer, warmup_steps, t_total, last_epoch=-1):
        self.warmup_steps = warmup_steps
        self.t_total = t_total
        super(WarmupLinearSchedule, self).__init__(optimizer, self.lr_lambda, last_epoch=last_epoch)

    def lr_lambda(self, step):
        if step < self.warmup_steps:
            return float(step) / float(max(1, self.warmup_steps))
        return max(0.0, float(self.t_total - step) / float(max(1.0, self.t_total - self.warmup_steps)))


Why still using old implementation of apex fp16

## 🚀 Feature

According to Nvidia apex fp16 documentary: https://nvidia.github.io/apex/amp.html

The implementation of new apex version doesn't require using FP16_Optimizer wrapping over FusedAdam. So I wonder why the team still kept the old implementation. Is there any special reason ? If not, I will make a pull request of implementation due to new apex version


FP16_Optimizer is not an Optimizer when fp_16

## 🐛 Bug

When using fp_16, the scheduler should be:

scheduler = WarmupLinearSchedule(optimizer.optimizer, warmup_steps=args.warmup_steps,
                                         t_total=num_train_optimization_steps)

if not, that will be "TypeError: FP16_Optimizer is not an Optimizer"


Small modification of comment in the run_glue.py example

Add RoBERTa to the comment as it was not explicit that RoBERTa don't use token_type_ids.


Can't Using Binarization Script for DistilBERT

## 🐛 Bug

<!-- Important information -->

I'm currently using DistilBERT and running into issues when I run scripts/binarized_data.py. I get the following error:

Traceback (most recent call last):
  File "scripts/binarized_data.py", line 25, in <module>
    from ..utils import logger
ValueError: attempted relative import beyond top-level package

I haven't modified anything within the package.


Need multiple capabilities

Need both Generative Finetuning and Distilling Capabilities


loss explosion

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am using Bert-CRF model to do Named Entity Recognition task. I am using the average of the last four layers as the input of CRF model.  But the loss will increase and become Nan in a few batches. Anyone meet that problem before? Any suggestions will be appreciated!


Cannot import DistilBert classes

Tried installing from the master, and couldn't do it.
![image](https://user-images.githubusercontent.com/347398/63883240-91c73e80-c988-11e9-9a22-241ac48479c3.png)


swap order of optimizer.step() and scheduler.step()

The current code results in the following warning:

```
UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
```


distilbert: fix number of hidden_size

Hi,

this PR corrects the return value of the `hidden_size` function (which should be the dimension size, as it is used in all other models) :)


Schedulers cause memory accumulation across folds in cross-validation?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am facing a strange issue when using the schedulers available in this library within a cross-validation loop. Basically, in each fold, I initialize a new model, optimizer, and scheduler. GPU memory accumulates until I eventually get a CUDA out of memory issue.

The simplest example I could come up with to reproduce the error is:

```python
import torch
from pytorch_transformers import WarmupConstantSchedule, WarmupCosineSchedule, WarmupLinearSchedule, WarmupCosineWithHardRestartsSchedule

# In my actual project, this is a for loop over the k-folds of k-fold cross-validation.
# In this example I use a while just to demonstrate the OOM error.
while True:
    net = torch.nn.Linear(10000, 10000)
    net = net.cuda()

    optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
    scheduler = WarmupCosineWithHardRestartsSchedule(optimizer, 1, 1000)

    # I also tried all the other schedulers. Same issue.
    # scheduler = WarmupConstantSchedule(optimizer, 1)
    # scheduler = WarmupCosineSchedule(optimizer, 1, 1000)
    # scheduler = WarmupLinearSchedule(optimizer, 1, 1000)

    del net, optimizer, scheduler
```

This will run until it (very quickly) uses up all 12GB on my Titan XP GPU. To make sure it was truly the initialization of the scheduler, I also tested

```python
import torch
from pytorch_transformers import WarmupCosineWithHardRestartsSchedule

while True:
    net = torch.nn.Linear(10000, 10000)
    net = net.cuda()

    optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)

    del net, optimizer
```

And did not see the memory accumulation or OOM error.

My question(s) is/are:

- Is this a known problem?
- Am I doing something dumb?
- How might I use a new scheduler for each fold of k-fold cross-validation in a way that doesn't lead to this issue?

Thanks a lot.


GPT2 Tokenizer decoding fails when the added tokens include a space

## 🐛 Bug

After adding a new token that contains a space to the GPT2 tokenizer, the tokenizer produces an error at decoding time (see example code below). My current workaround is to preprocess that token to remove spaces before adding it and to postprocess the token after decoding. But I thought I'd share this in case this is something that the library can warn against (e.g. added tokens should not include spaces) or even support.

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Run the following code:

```python
from pytorch_transformers.tokenization_gpt2 import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.add_tokens(["special token"])
encoded = tokenizer.encode("special token")
tokenizer.decode(encoded)
```

2. Currently, I get the error:
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-5-f47101f92e14> in <module>
----> 1 tokenizer.decode(encoded)

~/miniconda3/lib/python3.7/site-packages/pytorch_transformers/tokenization_utils.py in decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces)
    665             token_ids, skip_special_tokens=skip_special_tokens
    666         )
--> 667         text = self.convert_tokens_to_string(filtered_tokens)
    668         if clean_up_tokenization_spaces:
    669             text = self.clean_up_tokenization(text)

~/miniconda3/lib/python3.7/site-packages/pytorch_transformers/tokenization_gpt2.py in convert_tokens_to_string(self, tokens)
    187         """ Converts a sequence of tokens (string) in a single string. """
    188         text = ''.join(tokens)
--> 189         text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors=self.errors)
    190         return text
    191

~/miniconda3/lib/python3.7/site-packages/pytorch_transformers/tokenization_gpt2.py in <listcomp>(.0)
    187         """ Converts a sequence of tokens (string) in a single string. """
    188         text = ''.join(tokens)
--> 189         text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors=self.errors)
    190         return text
    191

KeyError: ' '
```
## Expected behavior

I expect the decoder to return the string `"special token"`

## Environment

* OS: OSX
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): master (d06c5a2a0acd8525d969a8f8f5b968ec0ec110b4)
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:


How to split consecutive numbers?

## ❓ Questions & Help

   In some NER datasets of BIO(ES) format, each number of a consecutive number string labeled with a corresponding tag, e.g., "All Jhon need is only 10 yuan" will be labeled as "O, PER, O, O, O, O, O, O". In this case, "10" is labeled as "O, O". But in **BertTokenizer** and **PreTrainedTokenizer**, I don't find(or know?)  effective params to deal with this situation.
<!-- A clear and concise description of the question. -->


mems output in XLNet

## 🐛 Bug

Hi,
I am trying to use memory of the last forward pass (mems arg) with XLNet. I am getting a tuple of None as mems output instead of a tuple of tensor. The same code with TransformerXL is running fine. Am I doing anything wrong or is this a bug ? Below is a short code snippet to reproduce the error.
Many thanks,
A

```python
import torch
from pytorch_transformers import TransfoXLTokenizer, TransfoXLModel, XLNetTokenizer, XLNetModel

text = ['This is the first sentence. ', 'And this is another one']

# transformer-XL
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLModel.from_pretrained('transfo-xl-wt103')
mems = None
for i in range(2):
    input_ids = torch.tensor(tokenizer.encode(text[i])).unsqueeze(0)
    outputs = model(input_ids, mems=mems)
    mems = outputs[1]
# RUNS OK

# XLNet
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
model = XLNetModel.from_pretrained('xlnet-large-cased')
mems = None
for i in range(2):
    input_ids = torch.tensor(tokenizer.encode(text[i])).unsqueeze(0)
    outputs = model(input_ids, mems=mems)
    mems = outputs[1]
# We get tuple of None in first model output, second forward crashes.
# File "/home/asors/anaconda3/envs/psco/lib/python3.7/site-packages/pytorch_transformers/modeling_xlnet.py", line 858, in forward
#     mlen = mems[0].shape[0] if mems is not None else 0
```

<!-- Important information -->

Model I am using: XLNet
Language I am using the model on (English, Chinese....):

* OS: Ubuntu
* Python version: 3.7
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0


Output of BertModel does not match fixed feature vectors extracted from the last hidden layer

I want to finetune bert in my task. However, the bert ouput does not match the fixed feature vectors.

I extracted the fixed feature vectors like this:
I use `extract_features.py` to  extract the fixed feature vectors of last hidden layer (layer -1). The command line is below:
`python extract_features.py --input_file=input.txt --output_file=output.json --vocab_file=model_path/vocab.txt --bert_config_file=model_path/bert_config.json --init_checkpoint=model_path/bert_model.ckpt --layers=-1 --max_seq_length=128 --batch_size=8`

The bert model output extracted like below:
The load model code is like below:
 `model_dict = model.state_dict()`
`#load pretrained released bert model`
  `pretrained_dict = torch.load( 'pytorch_model.bin')`
  `pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}`
`#update params`
  `model_dict.update(pretrained_dict)`
  `model.load_state_dict(model_dict)`

The init model code is like below:
`class A(nn.Module):`
` def __init__(self):`
` self.config = BertConfig.from_json_file('config.json')`
` self.bert = BertModel(self.config)`
`def inference(self, input_ids):`
` all_encoder_layers, _ = self.bert(input_ids, token_type_ids=None, attention_mask=input_mask)`
`return all_encoder_layers[-1]`
(I've omitted some irrelevant code.)

Then I ouput tensor all_encoder_layers[-1].
all_encoder_layers[-1] doesn't match the feature vectors extracted by `extract_features.py` .

I checked the params in my model. Bert params have been loaded and it is consistent with pretrained params. Also. the input sequence is consistent.
 Can anybody help me? Is there any settings I forget?


Fine-tuning (BERT & RoBERTa) base outperforms large

## ❓ Questions & Help

On all datasets I have used so far, base always outperforms large after the fine-tune. This is true for both BERT and RoBERTa.

Why is that? Am I doing something wrong? Does large require far more epochs to train or a different learning rate?


cannot import name 'RobertaConfig

## 🐛 Bug

When I run run_glue.py with the roberta model I get an ImportError: cannot import name 'RobertaConfig'

I can't run the run_glue.py with any model since it cannot import the RobertaConfig on line 34.

Any ideas why?


DistilBERT

Preparing the release for DistilBERT (smaller, faster, lighter, cheaper version of BERT)


Bert initialization

When I train bert model from scratch, it can not convergence and the loss does not decrease. Still don't work even try different learning rate many times.

But It works when I tried the tf version. I checked code and there have no much difference except the initialization.

So does anybody have some ideas about this?


UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1176: character maps to <undefined>

## 🐛 Bug

<!-- Important information -->
```UnicodeDecodeError``` when using vocab file generated by ```GPT2Tokenizer```.  Specifically, I created an instance of the ```GPT2Tokenizer``` by calling ```from_pretrained('gpt2')``` then saved the vocab and merges file for that instance to a local directory.  When creating a new ```GPT2Tokenizer``` from the saved files I encounter a ```UnicodeDecodeError``` when reading from the vocab file.

Model I am using (Bert, XLNet....): GPT2 Tokenizer

Language I am using the model on (English, Chinese....): N/A

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

```python
pretrained_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
vocab_file, merges_file = pretrained_tokenizer.save_vocabulary('.')
new_tokenizer = GPT2Tokenizer(vocab_file, merges_file) # <- UnicodeDecodeError occurs here
```

## Expected behavior

I expect ```new_tokenizer``` to initialize a tokenizer with the same behavior as ```pretrained_tokenizer```.

## Environment

* OS: Windows 10
* Python version: 3.6.8
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0

## Additional context

seems likely to be a bug during encoding in ```save_vocabulary```


XLNet resize embedding size ERROR

## ❓ Questions & Help

I add new tokens to XLNetLMHeadModel and use resize function
```
tokenizer.add_tokens(["<token1>", "<token2>"])
model.resize_token_embeddings(len(tokenizer))
```
But when running, the following error occurs:
```
Traceback (most recent call last):
 ...
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "../pytorch_transformers/modeling_xlnet.py", line 1059, in forward
    logits = self.lm_loss(transformer_outputs[0])
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/nas/home/jsun/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1410, in linear
    output += bias
RuntimeError: The size of tensor a (32003) must match the size of tensor b (32000) at non-singleton dimension 2
```

It is because in `resize_token_embeddings`, it changes the embedding size and calls `tie_weight` function to resize the LM head weight. But it forgot to change the size of `bias`, since XLNet has
```
self.lm_loss = nn.Linear(config.d_model, config.n_token, bias=True)
```
while other models have bias=False


Extracting Features Example

## ❓ Questions & Help

Hello.

Sorry, if my question is out of date or i just didn't find it, but i'm looking for the example/extract_features.py
that was supposed to be in in this repo (as mentioned in this stackoverflow  [post](https://stackoverflow.com/questions/55369821/how-to-train-a-neural-network-model-with-bert-embeddings-instead-of-static-embed)) and can't find it anymore. Was it just in an earlier release and got scrapped?

Thank you in advance for any help