Distilling GPT2 with gives OOM

## ❓ Questions & Help

Distilling GPT2 gives OOM what is the best way to fit both teacher student in single GPU and train?
Tried reducing batch size but that itself results into an error.

  File "train.py", line 285, in main
    distiller.train()
  File "trabsformersexamples\distillation\distiller.py", line 340, in train
    self.step(input_ids=token_ids, attention_mask=attn_mask, lm_labels=lm_labels)
  File "trabsformersexamples\distillation\distiller.py", line 378, in step
    s_logits, _, s_hidden_states = self.student(input_ids=input_ids, attention_mask=None)            # (bs, seq_length, voc_size)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "conda\conda\envs\pytorch\lib\site-packages\transformers\modeling_gpt2.py", line 549, in forward
    inputs_embeds=inputs_embeds)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "conda\conda\envs\pytorch\lib\site-packages\transformers\modeling_gpt2.py", line 439, in forward
    inputs_embeds = self.wte(input_ids)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "conda\conda\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.IntTensor instead (while checking arguments for embedding)


Tokenizing/Loading Data for GPT-2 (1 example per line)

I want to finetune GPT-2 on a dataset that has one example per line, and the examples all have different length. What is the best way to alter the TextDataset class to allow for this? I understand GPT-2 requires fixed length inputs but I'm not sure how to apply the attention mask to achieve this? Also do I need to add a bos/eos token to each line?

Appreciate the help


Update Squ

Update https://github.com/huggingface/transformers/blob/master/examples/run_squad.py

Implemented solution from

https://github.com/huggingface/transformers/issues/1837#issuecomment-554206349


`overwrite_cache` argument in `run_lm_finetuning.py` not used at all

There is an unused argument in `run_lm_finetuning.py`

https://github.com/huggingface/transformers/blob/e70cdf083ddb8bfe298d43e6d70d698a3a2f56d3/examples/run_lm_finetuning.py#L418

Is it forgotten?


Cleanup TPU bits from `run_glue.py`

TPU runner is currently implemented in:
https://github.com/pytorch-tpu/transformers/blob/tpu/examples/run_glue_tpu.py.

We plan to upstream this directly into `huggingface/transformers`
(either `master` or `tpu`) branch once it's been more thoroughly tested.


error on bert.fit for Squad dataset

Need more information

Further information is requested

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
The size of tensor a (384) must match the size of tensor b (12) at non-singleton dimension 1.
On passing values to fit function in modelling/bert.py facing above error.


fixes issue with unrecognized arguments for AdamW

as suggested in https://github.com/huggingface/transformers/issues/830


Correction for the tuple problem

This fixes the problem described and corrected in  https://github.com/huggingface/transformers/issues/831


explain how to successfully run examples in readme and doc


Is there a straightforward way to classify documents at the sentence level, while using surrounding sentences for context?

## ❓ Questions & Help

I'm curious how one would go about classifying documents at the sentence level using the Sequence Classification classes where the classification of a document's sentence would use other sentences in the document for context.

For example, if a document says, "I truly hate this product. Thanks [company name]!" The two sentences should be classified as negative. However, if you classify each sentence separately, "I truly hate this product" would be classified as negative, while "Thanks [company name]!" would likely be classified as positive.

I understand I could classify the document as a whole, but I'm looking for a more granular level of text classification.

Any guidance on making adjustments for this need would be greatly appreciated.

Thanks!


Using GPU for gpt2-xl

I want to use gpt2-xl using my PC's GPU (NVIDIA GeForce GTX 1070). The usual way to do this is via conda, but it appears that the latest version of gpt-2 is not available for conda. For example, the following installation commands don't work:
```
conda install transformers
conda install git+https://github.com/huggingface/transformers
```
The latest version of transformers for conda is a[ month old,](https://anaconda.org/conda-forge/transformers) and therefore doesn't include gpt2-xl. How can I use my machine's GPU to run gpt2-xl without using conda? Will there be conda support soon?


save_pretrained on CamembertTokenizer

## 🐛 Bug

This is probably something you already know, but `save_pretrained` for `CamembertTokenizer` seems not working at the moment.

Here is the error message I get:

```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-38-52393f32f8a0> in <module>
      1 results = {}
      2 if args.do_eval and args.local_rank in [-1, 0]:
----> 3     tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
      4     checkpoints = [args.output_dir]
      5     if args.eval_all_checkpoints:

/mnt/azmnt/code/Users/adm/transformers/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    281
    282         """
--> 283         return cls._from_pretrained(*inputs, **kwargs)
    284
    285

/mnt/azmnt/code/Users/adm/transformers/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
    410
    411         # Instantiate tokenizer.
--> 412         tokenizer = cls(*init_inputs, **init_kwargs)
    413
    414         # Save inputs and kwargs for saving and re-loading with ``save_pretrained``

/mnt/azmnt/code/Users/adm/transformers/transformers/tokenization_camembert.py in __init__(self, vocab_file, bos_token, eos_token, sep_token, cls_token, unk_token, pad_token, mask_token, additional_special_tokens, **kwargs)
     55         self.max_len_sentences_pair = self.max_len - 4  # take into account special tokens
     56         self.sp_model = spm.SentencePieceProcessor()
---> 57         self.sp_model.Load(str(vocab_file))
     58         # HACK: These tokens were added by fairseq but don't seem to be actually used when duplicated in the actual
     59         # sentencepiece vocabulary (this is the case for <s> and </s>

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/sentencepiece.py in Load(self, filename)
    116
    117     def Load(self, filename):
--> 118         return _sentencepiece.SentencePieceProcessor_Load(self, filename)
    119
    120     def LoadOrDie(self, filename):

OSError: Not found: "None": No such file or directory Error #2

```

Best


GPT2 Tokenizer Special Token ID Bug

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2 Tokenizer

The problem arise when using:
my own modified script: The problem arises when I try to add special tokens to the GPT2 tokenizer, specifically a pad token and a sep token.

The tasks I am working on is:
* [ ] my own task or dataset: summarization on the xsum dataset, however, the current bug does not actually affect the model, but the pre-processing.

## To Reproduce
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

from transformers import AutoTokenizer
encoder = AutoTokenizer.from_pretrained('gpt2')
encoder.add_special_tokens({'pad_token': '[PAD]', 'sep_token': '[SEP]'})
print(encoder.sep_token, encoder.sep_token_id)
print(encoder.pad_token, encoder.pad_token_id)
print('All special tokens:', encoder.all_special_tokens)
print('All special ids:', encoder.all_special_ids)

## Expected behavior

The expected output of this script is below:
[SEP] 50258
[PAD] 50257
All special tokens: ['<|endoftext|>', '[PAD]', '[SEP]']
All special ids: [50256, 50257, 50258]

The actual output is:
[SEP] 50258
[PAD] 50257
All special tokens: ['<|endoftext|>', '[PAD]', '[SEP]']
All special ids: [50256, 50256, 50256]

As you can see, All special IDS do not match the actual ids of the [SEP] and [PAD] token. Not sure why this is the case. Am I misunderstanding something about how add_special_tokens works?

## Environment

* OS: Ubuntu 18.04
* Python version: 3.7
* PyTorch version: 1.0.0
* PyTorch Transformers version (or branch): 2.1.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information: None

## Additional context

<!-- Add any other context about the problem here. -->


Wrong definition of the `logging_steps` parameter at the `run_lm_finetuning.py`

Hi,
`logging_steps` parameter defined as `"Log every X updates steps."` but it's been affected by `gradient_accumulation_steps` as follows:

https://github.com/huggingface/transformers/blob/f3386d938348628c91457fc7d8650c223317a053/examples/run_lm_finetuning.py#L243-L254

Therefore, if you, e.g., set logging_steps=1000 and  gradient_accumulation_steps=5, it'll log in every 5000 steps. That affects `evaluate_during_training` in a not intended way.


F score 0 in combining RoBERTa and BiLSTM

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to stack an LSTM on top of RoBERTa model for binary classification problem
I tried two moods , freezing BERT embedding and Fine-tuning
In case of freezing the embedding I get around 57% F-score compared to regular fine tune BERT which got me 81%
When I tried to unfreeze the embedding the f score is always 0 ,
Most probably I am doing something wrong, but I can't spot it I would appreciate some help

Model part

```
class RoBERTaLSTMClassifier(nn.Module):
    def __init__(self, bert_config, num_classes, hidden_size=None, dropout=0.5):
        """
        bert: pretrained bert model
        num_classes: the number of num_classes
        hidden_size: the number of hiddens which will be used by LSTM layer
        dropout: dropout rate
        """
        super(RoBERTaLSTMClassifier, self).__init__()
        self.num_classes = num_classes
        self.model = RobertaModel(bert_config)
        if hidden_size is None: self.hidden_size = bert_config.hidden_size
        else: self.hidden_size = hidden_size
        self.lstm = nn.LSTM(bert_config.hidden_size, self.hidden_size, bidirectional=True,batch_first=True)
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Linear(self.hidden_size * 2, 1)
        self.softmax = nn.Softmax()
        ## add sigmoid non linearity for binary classification
        self.sig           = nn.Sigmoid()

    def forward(self, input_ids, attention_mask, current_batch_size, hidden):
        """
        all_layers: whether or not to return all encoded_layers
        return: logits in the following format (batch_size, num_classes)
        """
        #with torch.no_grad():
             ## freeze embedding from BERT
        outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)
       # last hidden state is input to the LSTM
        output, (hidden_h, hidden_c) = self.lstm(outputs[0], hidden)

        output_hidden = torch.cat((hidden_h[0], hidden_h[1]), dim=1) #[B, H*2]

        logits = self.classifier(self.dropout(output_hidden)) #[B, C]
        sig_out = self.sig(logits).view(current_batch_size, -1)

        ## get the last batch output
        sig_out = sig_out[:, -1] # get last batch of labels
        hidden = (hidden_h, hidden_c)

        return sig_out, hidden

    def init_bilstm_hidden(self, batch_size):
        h0 = torch.zeros(2, batch_size, self.hidden_size).to(device) # 2 for bidirection
        c0 = torch.zeros(2, batch_size, self.hidden_size).to(device)

        return (h0, c0)
```

The training loop part
```
from sklearn.metrics import f1_score
from tqdm import tqdm, trange
import numpy as np
lr=0.001
roberta_conf = RobertaConfig.from_pretrained('roberta-base')

num_classes = 2
hidden_size = 256

LSTMRoBERTaModel =  RoBERTaLSTMClassifier(roberta_conf, num_classes=num_classes,hidden_size= hidden_size,dropout=0.5)
criterion = nn.BCELoss() ## binary cross entropy
optimizer = torch.optim.Adam(LSTMRoBERTaModel.parameters(), lr=lr)

epochs = 5
counter = 0
max_grad_norm = 1.0

nb_tr_examples, nb_tr_steps = 0, 0
for _ in trange(epochs, desc="Epoch"):
    LSTMRoBERTaModel.cuda()
    LSTMRoBERTaModel.train()
    tr_loss = 0
    y_preds = []
    y_true = []
    hidden_init = LSTMRoBERTaModel.init_bilstm_hidden(batch_size=bs)
    h = hidden_init
    for step, batch in enumerate(train_dataloader):
        batch = tuple(t.to(device) for t in batch)
        b_input_ids, b_input_mask, b_labels = batch
        current_batch_size = b_input_ids.size()[0]
        ##
        ## need to ask why converting to tuple
        h = tuple([each.data for each in h])

        ## forward pass
        preds, h = LSTMRoBERTaModel.forward(b_input_ids, b_input_mask, current_batch_size,h)

        loss = criterion(preds.squeeze(),b_labels.float())

        # track train loss
        tr_loss += loss.item()
        nb_tr_examples += b_input_ids.size(0)
        nb_tr_steps += 1

        # gradient clipping
        torch.nn.utils.clip_grad_norm_(parameters=LSTMRoBERTaModel.parameters(), max_norm=max_grad_norm)

        loss.backward()
        optimizer.step()
        LSTMRoBERTaModel.zero_grad()
    # print train loss per epoch
    print("\nTrain loss: {}".format(tr_loss/nb_tr_steps))

    LSTMRoBERTaModel.eval()
    eval_loss, eval_accuracy = 0, 0
    nb_eval_steps, nb_eval_examples = 0, 0

    val_h = LSTMRoBERTaModel.init_bilstm_hidden(bs)

    for batch in dev_dataloader:
        batch = tuple(t.to(device) for t in batch)
        b_input_ids, b_input_mask, b_labels = batch
        current_batch_size = b_input_ids.size()[0]
        with torch.no_grad():

             preds, val_h = LSTMRoBERTaModel.forward(b_input_ids, b_input_mask, current_batch_size, val_h)
             loss = criterion(preds.squeeze(),b_labels.float())

        eval_loss += loss
        y_preds.extend(np.round(preds.data.cpu()))
        y_true.extend(b_labels.data.cpu())
        #print(preds[2], b_labels[2] )
        #eval_accuracy += f1_score(torch.tensor.numpy(b_labels.float), toch.tensor.numpy(preds))
        nb_eval_examples += b_input_ids.size(0)
        nb_eval_steps += 1
    eval_loss = eval_loss/nb_eval_steps
    print("Validation loss: {}".format(eval_loss))
    print("F1 - Score: {}".format(f1_score(y_true,y_preds)))
    #print("F1- Score: {}".format(eval_accuracy/nb_eval_steps))
```


XLNetForSequenceClassification and CLS token

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,
As we can see in the official XLNet code ([https://github.com/zihangdai/xlnet]()), the [SEP] and [CLS] tokens are added before the zero-padding. (in convert_single_sequence() in classifier_utils.py)
(example : "This is my sequence ! [SEP][CLS]")
What is strange is that in the XLNetModelForSequenceClassification, we use a SequenceSummary which will keep the last value from the hidden state, i.e. the last token from padding (which doesn't correspond to the real [CLS] token)? (As I can see, it's working like that in the official code too).

Maybe someone here can explain me if this is an error, or if I forgot something ?


convert list to set in tokenize().split_on_tokens()

As [issue 1830](https://github.com/huggingface/transformers/issues/1830), I meet the same question
when i add some special_tokens in Tokenizer.
But I think it is property **self.all_special_tokens** that slow the code.
property self.all_special_tokens will **be called so many time** when we added some special token.

An easy way to solve this problem is to create a temporary Set.

 In my implementation, it faster about 10 times when 207 special tokens are added, I do not get a precise number because of multiprocessing : )


question about 'add_prefix_space' of encode method

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi, I have a question about the 'add_prefix_space' parameter of encode method. I'll use small gpt2 for example.

> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
> print(tokenizer.encode("my",add_prefix_space=True)) #616
> print(tokenizer.encode("my",add_prefix_space=False)) #1820

I think 616 is the index for the word 'my'(' my') since there's a space before letter 'm'.
And 1820 is the index for '-me' which is part of a word. Am I right?

Thank you!


run_squad hangs for small max_seq_length

## 🐛 Bug

I am experimenting with minimal examples to understand how sequences are generated, setting `max_seq_length` very low to trigger document chunking, and I hit this bug:

When `max_seq_length` is smaller than the query, `max_tokens_for_doc` becomes negative causing an infinite loop.

https://github.com/huggingface/transformers/blob/f3386d938348628c91457fc7d8650c223317a053/examples/utils_squad.py#L241-L258

Obviously having `max_seq_length` smaller than the query isn't very useful, and I should be setting `max_query_length`, But it would still be nice to have an assertion catch this.

### Steps to reproduce
1. `rm $SQUAD_DIR/cached*`
2. train with `--max_seq_length 16`
```
python examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --do_lower_case \
    --do_train \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --per_gpu_train_batch_size 12 \
    --learning_rate 3e-5 \
    --num_train_epochs 2.0 \
    --max_seq_length 16 \
    --doc_stride 8 \
    --output_dir /tmp/squad_seq_len_bug
```


Is this a bug?

Hello,
I tried to import this:

`from transformers import AdamW, get_linear_schedule_with_warmup`
but got error : model not found
but when i did this, it worked:
```
from transformers import AdamW
from transformers import WarmupLinearSchedule as get_linear_schedule_with_warmup
```
however when I set the scheduler like this :
```
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total)
```
I got this error :
```
__init__() got an unexpected keyword argument 'num_warmup_steps'
```


Can the HuggingFace GPT2DoubleHeadsModel either for regular language modelling or solving multiple-choice questions? or is it only for solving multiple-choice questions?

Hello,

According to the HuggingFace Transformer's website (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel), GPT2DoubleHeadsModel is the GPT2 Model transformer with a language modelling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks.

Does this mean that the GPT2DoubleHeadsModel can be used for both the regular language modelling tasks (predicting the next token) and also for solving multiple-choice questions?

Or does this mean that GPT2DoubleHeadsModel can be used to test machines on the multiple-choice type questions only?

Thank you,


Mean does not exist in TF2


Clarifications about the Quick-tour of the fine-tuning scripts?

## ❓ Questions & Help

The [quick tour](https://github.com/huggingface/transformers#quick-tour-of-the-fine-tuningusage-scripts) mentions an example of fine-tuning on different GLUE tasks.

My task/trained model has nothing to do with GLUE tasks, i.e.: I already have a pre-trained  pytorch model file and the formatted corpus files (train.tsv, dev.tsv and test.tsv).

Is the run_glue.py script meant to be edited by me for doing a new *NON GLUE* fine-tuning?

Or it is a matter of adjusting parameters on the CLI?

If none of them applies, where I can find an example of fine-tuning with support for "customized" task?


Disparitry with Fairseq Roberta implementation for predicting the mask token

Core: Modeling

Internals of the library; Models.

## ❓ Questions & Help
When experimenting with the fill mask functionality on the fairseq repo I realized there is a disparity with the results I get from huggingface implementation. Wondering if there is a mismatch between the model released here and their latest release.

Thanks


German DistilBERT

Hi,

this PR adds the German DistilBERT to the library 🤗

Thanks to the Hugging Face team (incl. hardware support) the German DistilBERT was trained on 1/2 of the data that was used for training the [German DBMDZ BERT](https://github.com/dbmdz/german-bert) model for ~4 days.

Evaluation on NER tasks (German CoNLL and GermEval) shows a performance difference of 1.3% on average compared to the German BERT model.

---

Remaining tasks:

* [x] Model, configuration and vocab is already uploaded to S3, only file permissions need to be adjusted


License for the pretrained DistilBert Models

Hi Team,
It would be a great help if the license for the pre-trained models is mentioned in the repository.
and particularly for Distil Models.


Understanding feature creation

## ❓ Questions & Help

Hi,

I am trying to understand feature creation when using run_lm_finetuning.py. If I let the script download the tokenizer files, everything works fine. If I manually assign a folder for the tokenizer with vocab.json and merges.txt downloaded from Hugging Face,  I get an empty dataframe during feature creation.

`11/19/2019 16:36:20 - INFO - __main__ -   Creating features from dataset file at /data/test
11/19/2019 16:41:53 - INFO - __main__ -   Saving features into cached file /data/test/roberta-base_cached_lm_999999999998_moleculenet_roberta_train.csv
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 558, in <module>
    main()
  File "run_lm_finetuning.py", line 510, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 175, in train
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
  File "/usr/local/conda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 94, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0`


XLNet for Token classification

Hi,

this PR adds a XLNet based token classifier for PyTorch `XLNetForTokenClassification` and TensorFlow `TFXLNetForTokenClassification` with unit test that allows performing sequence labeling tasks like NER or PoS tagging.


Pre-training a smaller version of BERT on own data

## ❓ Questions & Help
Hello,
<!-- A clear and concise description of the question. -->
I want to load the first 3-4 layers of BERT, pre-train those on my own data and then fine-tune the model on the target task. Is this possible?

I took a look at the [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) and changed the lines 478-479:

`    config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
                                          cache_dir=args.cache_dir if args.cache_dir else None)`
to
`    config = BertConfig(num_hidden_layers=3)
`
I'm not sure, if this change is sufficient and if the first or last three layers will be taken from the pre-trained BERT?

Is the _next sentence prediction_ pre-training task implemented somewhere ?

Thanks in Advance


XLNet for Token classification

Hi,

this PR adds a `TFXLNetForTokenClassification` implementation and unit test that allows to perform sequence labeling tasks like NER or PoS tagging.


How to fine-tune BERT on a large training dataset?

## ❓ Questions & Help

Hi,

I want to fine-tune BERT on a large training dataset. With around 1.5million training examples, this is currently consuming around 60GB of RAM. Is there any way to reduce the RAM usage or load the training examples in parts? Thanks!.


BertForTokenClassification for NER . what is the conclusion of  this output ?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi ,
Im trying to perform NER using BertForTokenClassification  .I saw this sample code in transformers GIT page.

from transformers import BertForTokenClassification
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForTokenClassification.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
print(labels)
outputs = model(input_ids, labels=labels)
loss, scores = outputs[:2]

output loss:
tensor(0.5975, grad_fn=<NllLossBackward>)

output scores:

tensor([[[-0.1622,  0.1824],
         [-0.1552, -0.0534],
         [-0.3032, -0.1166],
         [-0.2453, -0.1182],
         [-0.4388, -0.1898],
         [-0.3159, -0.1067]]], grad_fn=<AddBackward0>)

1.When i printed the loss and score i got below values .Now how should i infer this output ? what dose these value represent for  performing NER ? what should i do to get the NER tags for the sentence "Hello, my dog is cute" .

2.i referred few NER codes in GIT using BERT  and they have  humongous line of code written for performing the NER . Is there any simple way to perform NER using bert ?  like how Flair library has very simple method for performing the NER task ?


Run bert for multi-classification but loss never decrease

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I copy the code in example/run_glue.py for text classification task with bert model. All i change in this code is changed to a multi-classification task. I didn't change the main stream of this code. however, after 80 epoches running, i just found the eval result didn't change from the begging to the end. Next time i print the loss and i found the loss nearly unchanged from the begining to the end. i change loss from 1e-5 to 1e-3 and it is not the case, the loss is still nearly unchanged. i just sum the loss together in an single epoch, and after several epochs it only changed from 4703 to 4700 . So that's my problem and any ideas will be appreciated .

`

import torch
from transformers import BertForSequenceClassification, BertTokenizer, InputExample, AdamW, WarmupLinearSchedule
from torch.utils.data import DataLoader, Dataset, SequentialSampler, RandomSampler, TensorDataset
from torch.utils.data.distributed import DistributedSampler
import random
import numpy as np
import os, pickle
import argparse
from tqdm import tqdm, trange
import copy, json
import glob
from apex import amp
from transformers import glue_convert_examples_to_features as convert_examples_to_features
from Bert_eval import evaluate

def train(args, model, tokenizer):
    with open("../../data/train_texts", "r") as fr:
        texts = fr.readlines()
    with open("../../data/train_labels", "r") as fr:
        labels = fr.readlines()
    examples = load_dataset(texts, labels)
    label_list = ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]
    features = convert_examples_to_features(examples, tokenizer,
                                 label_list=label_list,
                                 output_mode="classification")
    cached_path = "cached_file"
    torch.save(features, cached_path)
    all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
    all_attention_mask = torch.tensor([f.attention_mask for f in features], dtype=torch.long)
    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
    all_labels = torch.tensor([f.label for f in features], dtype=torch.long)
    train_dataset = TensorDataset(all_input_ids, all_attention_mask, all_token_type_ids, all_labels)
    train_dataloader = DataLoader(train_dataset, batch_size=args.train_batch_size)
    no_decay = ['bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': args.weight_decay},
        {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
        ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    scheduler = WarmupLinearSchedule(optimizer, warmup_steps=args.warmup_steps, t_total=args.t_total)

    model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)

    # multi-gpu training (should be after apex fp16 initialization)
    if args.n_gpu > 1:
        model = torch.nn.DataParallel(model)
    # Distributed training (should be after apex fp16 initialization)
    if args.local_rank != -1:
        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank],
                                                          output_device=args.local_rank,
                                                          find_unused_parameters=True)
    # Train!
    print("***** Running training *****")
    print("  Num examples = %d", len(train_dataset))
    print("  Num Epochs = %d", args.num_train_epochs)
    print("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
    print("  Total optimization steps = %d", args.t_total)
    global_step = 0
    train_iterator = trange(int(args.num_train_epochs), desc="Epoch")#, disable=args.local_rank not in [-1, 0])
    set_seed(args)
    for _ in train_iterator:
        tr_loss = 0.0
        print("global_step: ", global_step)
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        print("optimizer: ", optimizer)
        for step, batch in enumerate(epoch_iterator):
            # print("batch: ", len(batch))
            model.train()
            batch = tuple(t.to(args.device) for t in batch)
            inputs = {'input_ids':      batch[0],
                    'attention_mask': batch[1],
                    'token_type_ids': batch[2],
                    'labels':         batch[3]}

            outputs = model(**inputs)
            loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
            print("loss:", loss)
            loss.backward()
            tr_loss += loss.item()
            if (step + 1) % args.gradient_accumulation_steps == 0:
                if args.fp16:
                    torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
                else:
                    torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)

                optimizer.step()
                scheduler.step()  # Update learning rate schedule
                model.zero_grad()
                global_step += 1
        output_dir = os.path.join(args.output_dir, 'checkpoint-{}'.format(global_step))
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)
        result = evaluate(args, model, tokenizer)
        print("result: ", result)
        model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training
        model_to_save.save_pretrained(output_dir)
        torch.save(args, os.path.join(output_dir, 'training_args.bin'))

def load_dataset(lines, labels):
    """
        convert examples for the training sets for document classification
    """
    examples = []
    for (i, (line, label)) in enumerate(zip(lines, labels)):
        line = line.strip()
        label = label.strip()
        # label = str(i % 2)
        guid = i
        examples.append(
            InputExample(guid=guid, text_a=line, label=label)
        )
    return examples

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--data_dir", type=str, default="./data/",
                        help="data path for train or test")
    parser.add_argument("--train_batch_size", type=int, default=32, help="training batch size")
    parser.add_argument("--t_total", type=int, default=100, help="training epoch")
    parser.add_argument("--weight_decay", default=0.0, type=float,
                        help="Weight decay if we apply some.")
    parser.add_argument("--output_mode", default="classification", type=str,
                        help="task name.")
    parser.add_argument("--learning_rate", default=1e-3, type=float,
                        help="The initial learning rate for Adam.")
    parser.add_argument("--max_grad_norm", default=1.0, type=float,
                        help="Max gradient norm.")
    parser.add_argument("--num_train_epochs", default=100, type=float,
                        help="Total number of training epochs to perform.")
    parser.add_argument("--max_steps", default=-1, type=int,
                        help="If > 0: set total number of training steps to perform. Override num_train_epochs.")
    parser.add_argument("--warmup_steps", default=0, type=int,
                        help="Linear warmup over warmup_steps.")
    parser.add_argument("--n_gpu", type=int, default=1, help="number of gpus to run")
    parser.add_argument('--logging_steps', type=int, default=50,
                        help="Log every X updates steps.")
    parser.add_argument('--save_steps', type=int, default=5000,
                        help="Save checkpoint every X updates steps.")
    parser.add_argument("--per_gpu_eval_batch_size", default=256, type=int,
                        help="task name.")
    parser.add_argument("--do_train", action="store_true", help="train model flag")
    parser.add_argument("--do_eval", action="store_true", help="eval model flag")
    parser.add_argument("--eval_all_checkpoints", action='store_true',
                        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number")
    parser.add_argument("--no_cuda", action='store_true',
                        help="Avoid using CUDA when available")
    parser.add_argument('--overwrite_output_dir', action='store_true',
                        help="Overwrite the content of the output directory")
    parser.add_argument('--overwrite_cache', action='store_true',
                        help="Overwrite the cached training and evaluation sets")
    parser.add_argument('--seed', type=int, default=42,
                        help="random seed for initialization")
    parser.add_argument("--adam_epsilon", default=1e-8, type=float,
                        help="Epsilon for Adam optimizer.")
    parser.add_argument('--fp16', action='store_true',
                        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit")
    parser.add_argument('--fp16_opt_level', type=str, default='O1',
                        help="For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']."
                             "See details at https://nvidia.github.io/apex/amp.html")
    parser.add_argument("--local_rank", type=int, default=-1,
                        help="For distributed training: local_rank")
    parser.add_argument('--gradient_accumulation_steps', type=int, default=1,
                        help="Number of updates steps to accumulate before performing a backward/update pass.")
    parser.add_argument("--output_dir", default="./checkpoint", type=str,
                        help="The output directory where the model predictions and checkpoints will be written.")
    parser.add_argument("--eval_checkpoint", type=str, default="1730", help="the checkpoint to reload")

    # parser.add_argument("--gpu", type=int, default=0, help="choose gpu device")
    args = parser.parse_args()
    if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        args.n_gpu = torch.cuda.device_count()
    else:  # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        torch.distributed.init_process_group(backend='nccl')
        args.n_gpu = 1
    args.device = device
    model_class = BertForSequenceClassification
    tokenizer_class = BertTokenizer
    pretrained_weights = "bert-base-chinese"
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights, num_labels=10)#, output_hidden_states=True, output_attentions=True)
    if torch.cuda.is_available():
        model.to(args.device)
    train(args, model, tokenizer)
    if True:
        # Create output directory if needed
        if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]:
            os.makedirs(args.output_dir)
        print("model saved")
        model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training
        model_to_save.save_pretrained(args.output_dir)
        tokenizer.save_pretrained(args.output_dir)
        print("model saved into ", args.output_dir)
        # Good practice: save your training arguments together with the trained model
        torch.save(args, os.path.join(args.output_dir, 'training_args.bin'))

        # Load a trained model and vocabulary that you have fine-tuned
        model = model_class.from_pretrained(args.output_dir)
        tokenizer = tokenizer_class.from_pretrained(args.output_dir)
        model.to(args.device)
    results = {}
    if True:
        tokenizer = tokenizer_class.from_pretrained(args.output_dir)#, do_lower_case=args.do_lower_case)
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = list(os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + '/**/' + WEIGHTS_NAME, recursive=True)))
        for checkpoint in checkpoints:
            global_step = checkpoint.split('-')[-1] if len(checkpoints) > 1 else ""
            prefix = checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else ""

            model = model_class.from_pretrained(checkpoint)
            model.to(args.device)
            result = evaluate(args, model, tokenizer, prefix=prefix)
            result = dict((k + '_{}'.format(global_step), v) for k, v in result.items())
            results.update(result)
    print("result: ", results)

`


tensorflow2.0 does not has mean, but reduce mean

## 🐛 Bug
tensorflow2.0 does not has a mean function but only a reduce_mean().
```
output = tf.mean(hidden_states, axis=1)
```
should be:
```
output = tf.reduce_mean(hidden_states, axis=1)
```
<!-- Important information -->

Model I am using: TFXLMForSequenceClassification with summary_type == 'mean'

Language I am using the model on (English, Chinese....): any

The problem arise when using:
when loading the model, due to the lazy initialisation of tf2.0 you have to run the model with dummy input.
```
TFXLMForSequenceClassification.from_pretrained(os.path.join(args.model_path, "xlm-mlm-17-1280-tf_model.h5"), config=config)
```

The tasks I am working on is:
* [ x] my own task or dataset: I want to finetune the model on a custom english dataset and transfer it to multiple language

## To Reproduce

Steps to reproduce the behavior:
1. load config file and add summary info
```
config = XLMConfig.from_pretrained(os.path.join(args.data_path, "xlm-mlm-17-1280-config.json"))
config.summary_use_proj = True
config.summary_type = 'mean'
config.summary_proj_to_labels = True
config.num_labels = len(LABELS)
config.summary_activation = "tanh"
```
2. load the model
```
model = TFXLMForSequenceClassification.from_pretrained(os.path.join(args.model_path, "xlm-mlm-17-1280-tf_model.h5"), config=config)
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

You should see an error stating that tf2.0 does not has a mean function.
File "transformers/modeling_tf_utils.py", line 442:
    ``output = tf.mean(hidden_states, axis=1)`` should be ``output = tf.reduce_mean(hidden_states, axis=1)``

```AttributeError: module 'tensorflow' has no attribute 'mean'```

## Environment

* OS: Ubuntu16.04
* Python version: 3.7.5
* Tensorflow version: 2.0
* Transformers version (or branch): 2.1.1
* Using GPU: yes
* Distributed of parallel setup : not distributed, only single GPU

## Additional context

<!-- Add any other context about the problem here. -->


How do I train OpenAIGPTDoubleHeadsModel from scratch?

## ❓ Questions & Help

It looks like I cannot train `OpenAIGPTDoubleHeadsModel` from scratch because it necessarily needs to be initialized using the `from_pretrained()` method. How should I initialize this model to be able to train it from scratch?

As a hack, perhaps I could initialize it using `from_pretrained()` and then reset the pre-trained initialization so any fine-tuning would essentially be the same as pre-training?


save all 12 layers outputs for each token

Hi,

I am following [this](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) post to create sentence vectors from the last 4 layers or sum all 12 layers and so on.

Considering a given document has 10,000 tokens and I am planning to use  "bert_base_uncased", I was wondering how to save all 12 layers for each token efficiently. Also, further, how to retrieve these layers in order to build sentence vectors.

Thanks!


Better TensorFlow 2 examples

This PR aims to improve the TensorFlow examples of the library. It aims to use many tf-specific paradigms, including but not limited to Keras API, AMP, XLA, `tensorflow_datasets`, `tf.train.Features` and `tf.train.Examples`, model saving using Keras callbacks.

Two examples are specifically targeted:

- [x]  `run_tf_glue.py`
- [ ]  `run_tf_squad.py`

Aims to have a similar architecture to the PyTorch examples: using a parser with similar arguments so as to be used from a terminal.

Feedback is most welcome.


[WIP] Add support for CamembertForTokenClassification

Hi,

this PR adds a `CamembertForTokenClassification` implementation, so that fine-tuning of NER models is possible.

Tasks:

* [x] Implement `CamembertForTokenClassification`
* [x] Add `CamembertForTokenClassification` to module import list
* [x] Add support for CamemBERT model in `run_ner.py` example


Adds CamemBERT to Model architectures list


Conversion of [Model]ForSequenceClassification logits to probabilities

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I noticed that whenever I would convert logits coming from the model to probabilities using the following equation:

probability = e^logit/(1 + e^logit)

The probabilities for my two classes do not add up to 1. Why is this?


XLM Masked Word Prediction

## ❓ Questions & Help

Does anyone knows how to mask a word in a sentence and then get predictions with probabilities from XLM models


multitask learning

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I would like to apply multitask learning, using for example two tasks, and the one (and maybe both) of the two tasks is sequential labeling like NER. To my understanding, in order to apply the library on such a task, the way to go is with BertForTokenClassification. Is that correct? But also what I think is that I do not have enough flexibility to use it/ adapt it to create a multitask model.

Could you share your thoughts on that? Any help would be much appreciated.


Some questions about the abstractive summarization code.

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, thanks for sharing the code,  i was confused about some code and implements when i reading it.
a) In run_summarization_finetuning.py,
![image](https://user-images.githubusercontent.com/9161371/69008385-bf6ac500-0984-11ea-8f02-72806bb68675.png)
should cls_token_id be replaced by sep_token_id?
b) In utils_summarization.py
![image](https://user-images.githubusercontent.com/9161371/69008407-17a1c700-0985-11ea-8ff4-a16cee796213.png)
This will make sequence [cls] a b [sep] c d [sep] encoded as 0 0 0 1 1 1 0, should the order be changed?
c) In the decoder, the deocder_mask only mask the pad token, i think look_ahead_mask is more accurate for we can't see the future words in advance.

Looking forward to reply. Thanks!


run glue problem

## ❓ Questions & Help

How to fix this problem
![image](https://user-images.githubusercontent.com/5669444/69006304-2f1e8700-0968-11ea-9ebf-0fe4c4c19f83.png)


typo "deay" -> "decay"

a typo has been fixed.


XLNet model params for Question answering

Model: XLNet for QA
model(parameters ...)
RuntimeError: CUDA error: device-side assert triggered


The acc of RACE is always low by roberta model

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
My scripts like follows:
python ./examples/run_multiple_choice.py \
        --model_type roberta \
        --task_name race \
        --model_name_or_path roberta-base \
        --do_train \
        --do_eval \
        --do_lower_case \
        --do_test \
        --data_dir $RACE_DIR \
        --learning_rate 5e-5 \
        --num_train_epochs 5 \
        --max_seq_length 512 \
        --output_dir ./race_base_1115 \
        --per_gpu_eval_batch_size=2 \
        --per_gpu_train_batch_size=2 \
        --gradient_accumulation_steps 1 \
        --overwrite_output_dir \
        --logging_steps 1000\
        --save_steps 1000\
        --evaluate_during_training \

but my acc result is always under 30%, both test and evaluate :
eval_acc = 0.2833400891771382
eval_loss = 1.386294308898901

Maybe it be that the token length is too long?  More than 3W input sentences' length is longer than 512.  It would truncated context.
But this problem didn't happen in fairseq(original Roberta code). i wanna know why.
Ask for help. Thanks a lot.


Can't run convert_roberta_original_pytorch_checkpoint_to_pytorch.py

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
when i want to convert my model which prertrained in fairseq, the error like follows:

2019-11-16 23:53:48.119139: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-16 23:53:48.148610: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2097740000 Hz
2019-11-16 23:53:48.151302: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557dc08b8900 executing computations on platform Host. Devices:
2019-11-16 23:53:48.151342: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
loading archive file ./checkpoint_best.pt
extracting archive file ./checkpoint_best.pt to temp dir /tmp/tmpz46rlxeu
Traceback (most recent call last):
  File "./transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py", line 178, in <module>
    args.classification_head
  File "./transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py", line 46, in convert_roberta_checkpoint_to_pytorch
    roberta = FairseqRobertaModel.from_pretrained(roberta_checkpoint_path)
  File "/home/liangchen/.conda/envs/py36lc/lib/python3.6/site-packages/fairseq/models/roberta/model.py", line 139, in from_pretrained
    **kwargs,
  File "/home/liangchen/.conda/envs/py36lc/lib/python3.6/site-packages/fairseq/hub_utils.py", line 33, in from_pretrained
    model_path = file_utils.load_archive_file(model_name_or_path)
  File "/home/liangchen/.conda/envs/py36lc/lib/python3.6/site-packages/fairseq/file_utils.py", line 80, in load_archive_file
    with tarfile.open(resolved_archive_file, 'r:' + ext) as archive:
  File "/home/liangchen/.conda/envs/py36lc/lib/python3.6/tarfile.py", line 1588, in open
    raise CompressionError("unknown compression type %r" % comptype)
tarfile.CompressionError: unknown compression type 'pt'

Ask for help. Thanks a lot!


run_finetuning resize token embeddings

Please see #1848


Hi,
When you run `run_lm_finetuning.py` with a custom tokenizer via `--tokenizer_name tokenizer` parameter, you need to resize model embedding according to the new tokenizer. Otherwise you are going to get `CUDA runtime error (59) : device-side assert triggered` or `Assertion 'srcIndex < srcSelectDimSize' failed`.

I fixed this issue and going to create a PR if it's okay?
Thanks!


Update modeling_utils.py by adding "DUMMY_INPUTS" after "logger" variable.

This Pull Request fixes a probable bug found using Transformers in Anaconda, using TFBertForSequenceClassification, Tensorflow 2.0.0b0, according to:

https://github.com/huggingface/transformers/issues/1810#issuecomment-554572471


fix summary_type value of SequenceSummary

from #1845


summary_type value  of SequenceSummary is incorrect

https://github.com/huggingface/transformers/blob/0477b307c7501ea76e01b03cb387a2312db752b3/transformers/modeling_utils.py#L731

I think `if hasattr(config, 'summary_use_proj')` is incorrect, `if hasattr(config, 'summary_type')` is correct.


Rebase and merge Louismartin/camembert


"This tokenizer does not make use of special tokens." warning

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I just updated to the latest version transformers. Now when I use tokenizer to encode word, it always show the warning "This tokenizer does not make use of special tokens."

Is there any way to hide that warning? Thank you.


xlm-mlm-17-1280 model masked word prediction

Hi
I would like some help with how to use pretrained xlm-mlm-17-1280 model to get predictions for masked word prediction. I have followed http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/ for BERT mask prediction and it is working. Could you help me with how to use xlm-mlm-17-1280 model for word prediction. I need to get prediction for Turkish Language which is one of the languages in 17 languages


BPE error when fine-tuning a CTRL model

Hi,

cc @keskarnitish

I'm using a slightly modified version of the exapmles/run_lm_finetuning.py code to fine-tune a CTRL model and getting this BPE warning:

```
WARNING - transformers.tokenization_ctrl -   Saving vocabulary to /home/ubuntu/data/ctrl/pytrained4/merges.txt: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!
```

Is this something I should be concerned about?


Sampling sequence generator for transformers

[@thomwolf version]: In this PR I introduce a flexible API to generate sequences for the models of the library provided with LM heads.
The API is designed to have a very flexible and quite exhaustive API:
- with/without a prompt
- with/without beam search
- with/without greedy decoding/sampling
- with any (and combination) of top-k/top-p/penalized repetitions

Only single-stack architectures are currently supported.

Here is an example generating 4 continuations of a prompt with using a beam search of 3 for each continuation and token sampling at each step with top-p filtering and repetition penalization until we reach a <unk> token or a max length of 50.

```python
import torch
from transformers import OpenAIGPTLMHeadModel, OpenAIGPTTokenizer

tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')

input_ids = tokenizer.encode("This is the", return_tensors='pt')

output = model.generate(input_ids,
                            max_length=50,
                            do_sample=True,
                            num_beams=3,
                            temperature=0.7,
                            top_k=0,
                            top_p=0.8,
                            repetition_penalty=1.4,
                            bos_token_id=None,
                            pad_token_id=None,
                            eos_token_ids=tok.unk_token_id,
                            batch_size=None,
                            length_penalty=None,
                            num_return_sequences=4)[0]

for j in range(len(output)):
    print(tok.decode(output[j].tolist()))
```

To-add features:
- Add support for encoder-decoder architectures
- Add support for past and cached states


Add support for Japanese BERT models by cl-tohoku

This PR adds new BERT models for Japanese text.
Details of the models can be found in [this repository](https://github.com/cl-tohoku/bert-japanese).

Since the way of tokenization is Japanese-specific, a new file is added for the tokenizers.
And, the locations of the files (models, configs, etc.) should be updated when the files are uploaded to S3. (How could I do this?)


resize_token_embeddings not implemented for TFBert*

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hey,

I noticed for the Tensorflow 2 implementation of BERT, the `resize_token_embeddings` function is not implemented, while the Pytorch BERT class does.

### Questions:
* Is the implementation of `resize_token_embeddings` for TF models expected?
  * I'd be happy to help with this
* Any other solutions to resizing token embedding?
  * Possibly manually updating the config, vocab file, and weights to support the added tokens

Example:

```
special_tokens = {
    'additional_special_tokens': ['[SPC]']
}
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
tokens_added = tokenizer.add_special_tokens(special_tokens)

model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
model.resize_token_embeddings(len(tokenizer))
```
Output: `  File ".../transformers/modeling_tf_utils.py", line 115, in resize_token_embeddings
    raise NotImplementedError`

### Source:

`TFPreTrainedModel`
https://github.com/huggingface/transformers/blob/74ce8de7d8e0375a9123f9542f3483f46cc8df9b/transformers/modeling_tf_utils.py#L117

Here I think the implementation would look similar to the PyTorch model. Updating the embeddings specifically for BERT would then occur in TFBertMainLayer.resize_token_embeddings.

```
def resize_token_embeddings(self, new_num_tokens=None):
        """ Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size.
        Take care of tying weights embeddings afterwards if the model class has a `tie_weights()` method.

        Arguments:

            new_num_tokens: (`optional`) int:
                New number of tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end.
                If not provided or None: does nothing and just returns a pointer to the input tokens ``tf.Variable`` Module of the model.

        Return: ``tf.Variable``
            Pointer to the input tokens Embeddings Module of the model
        """
        base_model = getattr(self, self.base_model_prefix, self)  # get the base model if needed
        model_embeds = base_model._resize_token_embeddings(new_num_tokens)
        if new_num_tokens is None:
            return model_embeds

        # Update base model and current model config
        self.config.vocab_size = new_num_tokens
        base_model.vocab_size = new_num_tokens

        # Tie weights again if needed
        if hasattr(self, 'tie_weights'):
            self.tie_weights()

        return model_embeds
```

`TFBertMainLayer`
https://github.com/huggingface/transformers/blob/74ce8de7d8e0375a9123f9542f3483f46cc8df9b/transformers/modeling_tf_bert.py#L472

Thanks,

Ali Momin


Error started happening today: ImportError: cannot import name 'get_linear_schedule_with_warmup'

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ x] the official example scripts: (give details)

The squad script:
https://github.com/huggingface/transformers/blob/master/examples/run_squad.py

The tasks I am working on is:
* [X ] an official GLUE/SQUaD task: (give the name)

Squad 2.0 from https://rajpurkar.github.io/SQuAD-explorer/

* [ ] my own task or dataset: (give details)

## To Reproduce

```

!pip install transformers
import urllib.request

url = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json'
urllib.request.urlretrieve(url, 'train-v2.0.json')

url = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json'
urllib.request.urlretrieve(url, 'dev-v2.0.json')
!wget 'https://raw.githubusercontent.com/huggingface/transformers/master/examples/run_squad.py'
!wget 'https://raw.githubusercontent.com/huggingface/pytorch-transformers/master/examples/utils_squad.py'
!wget 'https://raw.githubusercontent.com/huggingface/pytorch-transformers/master/examples/utils_squad_evaluate.py'

# $SQUAD_DIR/train-v1.1.json

SQUAD_Train = '/content/train-v2.0.json'
SQUAD_Dev = '/content/dev-v2.0.json'

!python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file '$SQUAD_Train' \
  --predict_file '$SQUAD_Dev' \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --version_2_with_negative \
  --output_dir /tmp/debug_squad/

```

Results in

>ImportError: cannot import name 'get_linear_schedule_with_warmup'

I ran it yesterday and it worked fine, but today it's not working.

For convenience, here's a colab notebook with the code

https://colab.research.google.com/drive/1tNisXX5siuNnkuEQ-X_XdEdDTtdJ0qeL


Model parallelism support?

Hi. I'm running into memory issues when trying to fine tune the CTRL language model on a 16 GB GPU. Is there any built-in support for splitting models across more than one GPU? I suppose that mapping the input embedding layer to a different GPU or even the CPU would do the trick.


Parallell compute failing for finetuning GPT2 using GPU

## ? Questions & Help
**SYSTEM**
OS: Ubuntu 18.04
Python version: 3.6.8
Torch version: 1.0.0
Transformers version: 2.1.1

I am trying to finetune GPT2 using a GPU and am getting an error. I run the `run_lm_finetuning.py` example script with the following parameters:

```python
python run_lm_finetuning.py \
--output_dir=/home/user/gpt2_finetuned_isr/model \
--model_type=gpt2 \
--model_name_or_path=gpt2 \
--do_train \
--train_data_file=/path/to/train/data/train_data.txt \
--do_eval \
--eval_data_file=/path/to/test/data/test_data.txt
```

Here is the error:
```python
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/user/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html
11/14/2019 11:01:48 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
11/14/2019 11:01:49 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /home/user/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
11/14/2019 11:01:49 - INFO - transformers.configuration_utils -   Model config {
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

11/14/2019 11:01:50 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /home/user/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
11/14/2019 11:01:50 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /home/user/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
11/14/2019 11:01:50 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at /home/user/.cache/torch/transformers/4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
/home/user/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:117: UserWarning:
    Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
11/14/2019 11:02:10 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=1024, cache_dir='', config_name='', device=device(type='cuda'), do_eval=True, do_lower_case=False, do_train=True, eval_all_checkpoints=False, eval_data_file='/home/user/gpt2_finetuned_isr/finetune_gpt2_test.txt', evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, local_rank=-1, logging_steps=50, max_grad_norm=1.0, max_steps=-1, mlm=False, mlm_probability=0.15, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='/home/user/gpt2_finetuned_isr/model', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=50, save_total_limit=None, seed=42, server_ip='', server_port='', tokenizer_name='', train_data_file='/home/user/gpt2_finetuned_isr/finetune_gpt2_train.txt', warmup_steps=0, weight_decay=0.0)
11/14/2019 11:02:10 - INFO - __main__ -   Creating features from dataset file at /home/user/gpt2_finetuned_isr
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (1003332 > 1024). Running this sequence through the model will result in indexing errors
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.
11/14/2019 11:02:16 - WARNING - transformers.tokenization_utils -   This tokenizer does not make use of special tokens. Input is returned with no modification.

BTW, I get TONS of these tokenizer warnings... I won't paste them here as they say the same thing...

11/14/2019 11:02:16 - INFO - __main__ -   Saving features into cached file /home/user/gpt2_finetuned_isr/cached_lm_1024_finetune_gpt2_train.txt
11/14/2019 11:02:16 - INFO - __main__ -   ***** Running training *****
11/14/2019 11:02:16 - INFO - __main__ -     Num examples = 979
11/14/2019 11:02:16 - INFO - __main__ -     Num Epochs = 1
11/14/2019 11:02:16 - INFO - __main__ -     Instantaneous batch size per GPU = 4
11/14/2019 11:02:16 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
11/14/2019 11:02:16 - INFO - __main__ -     Gradient Accumulation steps = 1
11/14/2019 11:02:16 - INFO - __main__ -     Total optimization steps = 245
Epoch:   0%|                                                                                | 0/1 [00:00<?, ?it/sTraceback (most recent call last):                                                                                | 0/245 [00:00<?, ?it/s]
  File "run_lm_finetuning.py", line 545, in <module>
    main()
  File "run_lm_finetuning.py", line 497, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 228, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 533, in forward
    head_mask=head_mask)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 385, in forward
    position_ids = torch.arange(past_length, input_ids.size(-1) + past_length, dtype=torch.long, device=input_ids.device)
RuntimeError: parallel_for failed: no kernel image is available for execution on the device
Epoch:   0%|                                                                                | 0/1 [00:00<?, ?it/s]
Iteration:   0%|                                                                                | 0/245 [00:00<?, ?it/s]
```

I see the pytorch warning about the GPU, but it still seems available to use:
```python
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.current_device()
/home/user/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:117: UserWarning:
    Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
0
>>> torch.cuda.device(0)
<torch.cuda.device object at 0x7fb833751d30>
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'Quadro 4000'
>>> torch.cuda.is_available()
True
```

Any guidance is much appreciated!


Where is Model2Model PreTrainedEncoderDecoder in run_summerization_finetune

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


As of now the tokenizers output a specific warning when an encoded sequence is longer than the maximum specified sequence length, which is model-specific:

```
Token indices sequence length is longer than the specified maximum sequence length for this model (X > 1024). Running this sequence through the model will result in indexing errors
```

It is currently in the `convert_tokens_to_ids` and this leads to two issues:

- using `encode` or `encode_plus` methods with a `max_length` specified will still output that warning as the `convert_tokens_to_ids` method is used before the truncation is done. (cf #1791)
- since `prepare_for_model` was introduced, I personally feel that all modifications related to the model should happen in that method and not in `tokenize` or `convert_tokens_to_ids`.

This PR aims to slightly change the behavior so that both aforementioned issues may be solved by putting the warning in the `prepare_for_model` method if no `max_length` is specified.


replace LambdaLR scheduler wrappers by function

Custom schedulers are currently initiated by wrapping Pytorch's LambdaLR
class and passing a method of the wrapping class to the __init__
function of LambdaLR. This approach is not appropriate for several
reasons:

1. one does not need to define a class when it only defines a
__init__() method;
2. instantiating the parent class by passing a method of the child class
creates a cyclical reference which leads to memory leaks. See issues #1742 and #1134.

In this commit we replace the wrapper classes with functions that
instantiate `LambdaLR` with a custom learning rate function. We use a
closure to specify the parameter of the latter. We also do a bit of
renaming within the function to explicit the behaviour and removed
docstrings that were subsequently not necessary.


sum() is replaced by itertools.chain.from_iterable()

sum() is the leanest method to flatten a string list, so it's been replaced by itertools.chain.from_iterable() . Please check #1830


GPT2 tokenizer is so slow because of sum()

## 🐛 Bug
Hi,
As the discussion started in that  #1621 issue, GPT2 tokenization is so slow even with 50MB of the dataset.

I'm using `run_lm_finetuning.py` and here are the steps to reproduce the problem:

- Have a dataset not an even bigger one. 20MB of a dataset is enough.
- Call `run_lm_finetuning.py` to train (finetune) the dataset. Here are my parameters:
```
--train_data_file "/train/datafile" \
--eval_data_file "/eval/datafile" \
--output_dir "/train/model" \
--model_type gpt2 \
--model_name_or_path distilgpt2 \
--cache_dir "/train/cache" \
--do_train \
--evaluate_during_training \
--per_gpu_train_batch_size 1 \
--per_gpu_eval_batch_size 1 \
--gradient_accumulation_steps 5 \
--overwrite_output_dir \
--seed 99
```
- You'll see it'll spend 20+ mins (depends on your cpu) to tokenize just 50MB of a text file.

I dug into `huggingface/transformers` 's codebase and profiled the tokenization process. And it is obvious that this summation drains the time:
https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L644

I run profiler and here is the result:

```
73791524 function calls in 1566.379 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    27157    0.083    0.000    0.109    0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)
    27157    0.065    0.000    0.128    0.000 _bootlocale.py:33(getpreferredencoding)
    81471    0.070    0.000    0.327    0.000 locale.py:589(setlocale)
    27157    0.422    0.000    0.876    0.000 locale.py:647(getpreferredencoding)
    27157    0.363    0.000    5.997    0.000 regex.py:328(findall)
    27157    0.662    0.000    1.682    0.000 regex.py:434(_compile)
  4815114    8.744    0.000   16.641    0.000 tokenization_gpt2.py:139(bpe)
  2030532    1.116    0.000    1.887    0.000 tokenization_gpt2.py:149(<lambda>)
    27157   22.702    0.001  110.038    0.004 tokenization_gpt2.py:180(_tokenize)
 25459527    5.702    0.000    5.702    0.000 tokenization_gpt2.py:194(<genexpr>)
 10242602    1.764    0.000    1.764    0.000 tokenization_gpt2.py:195(<genexpr>)
  1377876    1.678    0.000    1.975    0.000 tokenization_gpt2.py:91(get_pairs)
    95205    0.526    0.000    0.910    0.000 tokenization_utils.py:1043(special_tokens_map)
    95205    0.932    0.000    1.987    0.000 tokenization_utils.py:1055(all_special_tokens)
        1    0.119    0.119 1566.379 1566.379 tokenization_utils.py:615(tokenize)
    40789    0.099    0.000    0.169    0.000 tokenization_utils.py:623(split_on_token)
        1    0.287    0.287 1566.260 1566.260 tokenization_utils.py:641(split_on_tokens)
    54417    0.698    0.000  112.123    0.002 tokenization_utils.py:659(<genexpr>)
    27157    0.063    0.000    0.063    0.000 {built-in method _locale.nl_langinfo}
    81471    0.252    0.000    0.252    0.000 {built-in method _locale.setlocale}
   761640    0.384    0.000    0.384    0.000 {built-in method builtins.getattr}
    54314    0.022    0.000    0.022    0.000 {built-in method builtins.hasattr}
   516605    0.150    0.000    0.150    0.000 {built-in method builtins.isinstance}
  1821447    0.159    0.000    0.159    0.000 {built-in method builtins.len}
   472563    3.469    0.000    5.355    0.000 {built-in method builtins.min}
        1 1453.081 1453.081 1565.204 1565.204 {built-in method builtins.sum}
  2043214    0.297    0.000    0.297    0.000 {method 'add' of 'set' objects}
   456488    0.055    0.000    0.055    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
  4815114    1.169    0.000    1.169    0.000 {method 'encode' of 'str' objects}
  5550977   16.572    0.000   18.336    0.000 {method 'extend' of 'list' objects}
    27157    3.952    0.000    3.952    0.000 {method 'findall' of '_regex.Pattern' objects}
  2057689    0.784    0.000    0.784    0.000 {method 'get' of 'dict' objects}
   735863    0.233    0.000    0.233    0.000 {method 'index' of 'tuple' objects}
  4894984   38.307    0.000   44.010    0.000 {method 'join' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
  4855903    1.365    0.000    1.365    0.000 {method 'split' of 'str' objects}
    68048    0.009    0.000    0.009    0.000 {method 'strip' of 'str' objects}
    95205    0.024    0.000    0.024    0.000 {method 'values' of 'dict' objects}

```
I turned it into this by removing `sum()`
```
(self._tokenize(token, **kwargs) if token not \
                    in self.added_tokens_encoder and token not in self.all_special_tokens \
                    else [token] for token in tokenized_text)
```
and here is the profiler result:

```
   73275678 function calls in 121.030 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    27157    0.058    0.000    0.076    0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)
    27157    0.041    0.000    0.084    0.000 _bootlocale.py:33(getpreferredencoding)
    81471    0.058    0.000    0.211    0.000 locale.py:589(setlocale)
    27157    0.330    0.000    0.625    0.000 locale.py:647(getpreferredencoding)
    27157    0.267    0.000    4.996    0.000 regex.py:328(findall)
    27157    0.434    0.000    1.160    0.000 regex.py:434(_compile)
  4815114    9.797    0.000   18.875    0.000 tokenization_gpt2.py:139(bpe)
  2030532    1.270    0.000    2.100    0.000 tokenization_gpt2.py:149(<lambda>)
    27157   24.693    0.001  119.272    0.004 tokenization_gpt2.py:180(_tokenize)
 25459527    6.204    0.000    6.204    0.000 tokenization_gpt2.py:194(<genexpr>)
 10242602    1.975    0.000    1.975    0.000 tokenization_gpt2.py:195(<genexpr>)
  1377876    2.002    0.000    2.328    0.000 tokenization_gpt2.py:91(get_pairs)
    68050    0.287    0.000    0.475    0.000 tokenization_utils.py:1043(special_tokens_map)
    68050    0.507    0.000    1.061    0.000 tokenization_utils.py:1055(all_special_tokens)
        1    0.031    0.031  121.030  121.030 tokenization_utils.py:615(tokenize)
    27263    0.077    0.000    0.158    0.000 tokenization_utils.py:623(split_on_token)
        1    0.178    0.178  120.999  120.999 tokenization_utils.py:641(split_on_tokens)
        1    0.330    0.330  120.350  120.350 tokenization_utils.py:659(<listcomp>)
    27157    0.043    0.000    0.043    0.000 {built-in method _locale.nl_langinfo}
    81471    0.148    0.000    0.148    0.000 {built-in method _locale.setlocale}
   544400    0.188    0.000    0.188    0.000 {built-in method builtins.getattr}
    54314    0.014    0.000    0.014    0.000 {built-in method builtins.hasattr}
   407985    0.092    0.000    0.092    0.000 {built-in method builtins.isinstance}
  1807921    0.181    0.000    0.181    0.000 {built-in method builtins.len}
   472563    3.992    0.000    6.092    0.000 {built-in method builtins.min}
  2043214    0.326    0.000    0.326    0.000 {method 'add' of 'set' objects}
   456488    0.064    0.000    0.064    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
  4815114    1.259    0.000    1.259    0.000 {method 'encode' of 'str' objects}
  5550977   18.064    0.000   20.040    0.000 {method 'extend' of 'list' objects}
    27157    3.569    0.000    3.569    0.000 {method 'findall' of '_regex.Pattern' objects}
  2057689    0.839    0.000    0.839    0.000 {method 'get' of 'dict' objects}
   735863    0.273    0.000    0.273    0.000 {method 'index' of 'tuple' objects}
  4894984   41.821    0.000   48.026    0.000 {method 'join' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
  4842377    1.597    0.000    1.597    0.000 {method 'split' of 'str' objects}
    54522    0.007    0.000    0.007    0.000 {method 'strip' of 'str' objects}
    68050    0.012    0.000    0.012    0.000 {method 'values' of 'dict' objects}
```

You can see 121 seconds vs 1566 seconds.  It is 12x times faster without `sum()`. Okay lets discuss do we need `sum()`? Actually, not. Because the `sum()` just flattens the array with the leanest way and there are far more efficient ways. See that [answer](https://stackoverflow.com/a/953097) on StackOverflow.
Also as written in official python [doc](https://docs.python.org/3/library/functions.html#sum) , `sum()` is developed for numbers rather than strings.

So I replaced `sum()` with `list(itertools.chain.from_iterable(text))` as follows and run profiler.

```
return list(itertools.chain.from_iterable((self._tokenize(token, **kwargs) if token not \
                    in self.added_tokens_encoder and token not in self.all_special_tokens \
                    else [token] for token in tokenized_text)))
```
Here is the result:

```
         73791524 function calls in 114.720 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    27157    0.045    0.000    0.060    0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)
    27157    0.035    0.000    0.067    0.000 _bootlocale.py:33(getpreferredencoding)
    81471    0.045    0.000    0.159    0.000 locale.py:589(setlocale)
    27157    0.277    0.000    0.502    0.000 locale.py:647(getpreferredencoding)
    27157    0.237    0.000    4.258    0.000 regex.py:328(findall)
    27157    0.346    0.000    0.929    0.000 regex.py:434(_compile)
  4815114    8.703    0.000   16.973    0.000 tokenization_gpt2.py:139(bpe)
  2030532    1.171    0.000    1.923    0.000 tokenization_gpt2.py:149(<lambda>)
    27157   22.988    0.001  112.449    0.004 tokenization_gpt2.py:180(_tokenize)
 25459527    5.708    0.000    5.708    0.000 tokenization_gpt2.py:194(<genexpr>)
 10242602    1.755    0.000    1.755    0.000 tokenization_gpt2.py:195(<genexpr>)
  1377876    1.595    0.000    1.900    0.000 tokenization_gpt2.py:91(get_pairs)
    95205    0.345    0.000    0.565    0.000 tokenization_utils.py:1043(special_tokens_map)
    95205    0.581    0.000    1.236    0.000 tokenization_utils.py:1055(all_special_tokens)
        1    0.022    0.022  114.720  114.720 tokenization_utils.py:615(tokenize)
    40789    0.103    0.000    0.182    0.000 tokenization_utils.py:623(split_on_token)
        1    0.583    0.583  114.698  114.698 tokenization_utils.py:641(split_on_tokens)
    54417    0.248    0.000  113.314    0.002 tokenization_utils.py:659(<genexpr>)
    27157    0.032    0.000    0.032    0.000 {built-in method _locale.nl_langinfo}
    81471    0.111    0.000    0.111    0.000 {built-in method _locale.setlocale}
   761640    0.219    0.000    0.219    0.000 {built-in method builtins.getattr}
    54314    0.012    0.000    0.012    0.000 {built-in method builtins.hasattr}
   516605    0.097    0.000    0.097    0.000 {built-in method builtins.isinstance}
  1821447    0.166    0.000    0.166    0.000 {built-in method builtins.len}
   472563    3.855    0.000    5.777    0.000 {built-in method builtins.min}
        1    0.000    0.000    0.000    0.000 {built-in method from_iterable}
  2043214    0.305    0.000    0.305    0.000 {method 'add' of 'set' objects}
   456488    0.058    0.000    0.058    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
  4815114    1.104    0.000    1.104    0.000 {method 'encode' of 'str' objects}
  5550977   17.434    0.000   19.189    0.000 {method 'extend' of 'list' objects}
    27157    3.092    0.000    3.092    0.000 {method 'findall' of '_regex.Pattern' objects}
  2057689    0.759    0.000    0.759    0.000 {method 'get' of 'dict' objects}
   735863    0.243    0.000    0.243    0.000 {method 'index' of 'tuple' objects}
  4894984   41.030    0.000   46.738    0.000 {method 'join' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
  4855903    1.396    0.000    1.396    0.000 {method 'split' of 'str' objects}
    68048    0.009    0.000    0.009    0.000 {method 'strip' of 'str' objects}
    95205    0.013    0.000    0.013    0.000 {method 'values' of 'dict' objects}
```

It significantly improves the speed as seen in the difference between 114 seconds and 1566 seconds.

I'm going to create a pull request if everything is clear?
Thank you for your effort.


NotImplementedError when using TFDistilBertModel

## 🐛 Bug

<!-- Important information -->

I have this error when I train a model with TFDistilBertModel. The same code works if I change to BERT for example. The transformers is loaded with `TFDistilBertModel.from_pretrained("distilbert-base-uncased")`

```
  File "/home/<user>/miniconda3/envs/nlp/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in converted code:
    relative to /home/<user>:

    ../models.py:68 call  *
        bert_hidden_states = self.model(
    miniconda3/envs/nlp/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py:842 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    miniconda3/envs/nlp/lib/python3.7/site-packages/transformers/modeling_tf_distilbert.py:545 call  *
        outputs = self.distilbert(inputs, **kwargs)
    miniconda3/envs/nlp/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py:842 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    miniconda3/envs/nlp/lib/python3.7/site-packages/transformers/modeling_tf_distilbert.py:431 call  *
        raise NotImplementedError

    NotImplementedError:

```
## Environment

* OS: Manjaro
* Python version: 3.7.5
* TF: 2.0
* Using GPU ? Yes

<!-- Add any other context about the problem here. -->


GPT2 tokenizer is so slow because of regex.findall

## 🐛 Bug
Hi,
As the discussion started in that  #1621 issue, GPT2 tokenization is so slow even with 50MB of the dataset.

I'm using `run_lm_finetuning.py` and here are the steps to reproduce the problem:

- Have a dataset not an even bigger one. 50MB of a dataset is enough.
- Call `run_lm_finetuning.py` to train (finetune) the dataset. Here are my parameters:
```
--train_data_file "/train/datafile" \
--eval_data_file "/eval/datafile" \
--output_dir "/train/model" \
--model_type gpt2 \
--model_name_or_path distilgpt2 \
--cache_dir "/train/cache" \
--do_train \
--evaluate_during_training \
--per_gpu_train_batch_size 1 \
--per_gpu_eval_batch_size 1 \
--gradient_accumulation_steps 5 \
--overwrite_output_dir \
--seed 99
```
- You'll see it'll spend 20+ mins (depends on your cpu) to tokenize just 50MB of a text file.

I dug into `huggingface/transformers` 's codebase and profiled the tokenization process. And it seems like this regex drains the time:
https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_gpt2.py#L193

I run cProfiler and here is the result:

```
19537 function calls in 0.077 seconds
   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)
        1    0.000    0.000    0.077    0.077 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 _bootlocale.py:33(getpreferredencoding)
        3    0.000    0.000    0.000    0.000 locale.py:589(setlocale)
        1    0.000    0.000    0.000    0.000 locale.py:647(getpreferredencoding)
        1    0.000    0.000    0.035    0.035 regex.py:328(findall)
        1    0.000    0.000    0.000    0.000 regex.py:434(_compile)
     1457    0.004    0.000    0.006    0.000 tokenization_gpt2.py:139(bpe)
      350    0.000    0.000    0.000    0.000 tokenization_gpt2.py:149(<lambda>)
        1    0.009    0.009    0.077    0.077 tokenization_gpt2.py:191(plot)
     6536    0.002    0.000    0.002    0.000 tokenization_gpt2.py:196(<genexpr>)
     3240    0.003    0.000    0.003    0.000 tokenization_gpt2.py:197(<genexpr>)
      607    0.001    0.000    0.001    0.000 tokenization_gpt2.py:91(get_pairs)
        1    0.000    0.000    0.000    0.000 {built-in method _locale.nl_langinfo}
        3    0.000    0.000    0.000    0.000 {built-in method _locale.setlocale}
        1    0.000    0.000    0.077    0.077 {built-in method builtins.exec}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.hasattr}
        5    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
      332    0.000    0.000    0.000    0.000 {built-in method builtins.len}
       86    0.001    0.000    0.001    0.000 {built-in method builtins.min}
      350    0.000    0.000    0.000    0.000 {method 'add' of 'set' objects}
       85    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
     1457    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
     1594    0.007    0.000    0.010    0.000 {method 'extend' of 'list' objects}
        1    0.035    0.035    0.035    0.035 {method 'findall' of '_regex.Pattern' objects}
      351    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
      137    0.000    0.000    0.000    0.000 {method 'index' of 'tuple' objects}
     1474    0.014    0.000    0.016    0.000 {method 'join' of 'str' objects}
     1457    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}
```

P.s. Here `plot()` is the intermadiate function I named in order to profile and it covers that section:
```
            for token in re.findall(self.pat, text):
                if sys.version_info[0] == 2:
                    token = ''.join(self.byte_encoder[ord(b)] for b in token) # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)
                else:
                    token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8')) # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)
                bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(' '))
```

Is there anyone having that issue? Haven't anybody tried to finetune GPT2 with a dataset larger than 300MB?
It's been 39 hours and my tokenization still in progress. Is it normal or could we optimize it?
Thanks!


How to get all layers(12) hidden states of BERT?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I tried to set the output_hidden_states=True, but only got 3 layers of the hidden states of model outputs for BERT, but theoricaly it should be 12, how can I get that?


Regarding Fine-Tuning for Abstractive Summarization

Hi,

What result did you obtain with your Abstractive Summarization code for CNN-DM? Also, is it implemented based on this paper: https://arxiv.org/abs/1908.08345 ?

It will be great if you provide some details regarding it.


合并


Issue testing run_squad.py example

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am trying to run the squad example here

https://github.com/huggingface/transformers/blob/master/examples/run_squad.py

Using the Squad data from here

https://rajpurkar.github.io/SQuAD-explorer/

But I get this error

>ValueError: For training, each question should have exactly 1 answer.

Here is my code

```
!pip install transformers
import urllib.request

url = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json'
urllib.request.urlretrieve(url, 'train-v2.0.json')

url = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json'
urllib.request.urlretrieve(url, 'dev-v2.0.json')

!wget 'https://raw.githubusercontent.com/huggingface/transformers/master/examples/run_squad.py'
!wget 'https://raw.githubusercontent.com/huggingface/pytorch-transformers/master/examples/utils_squad.py'
!wget 'https://raw.githubusercontent.com/huggingface/pytorch-transformers/master/examples/utils_squad_evaluate.py'

# $SQUAD_DIR/train-v1.1.json

SQUAD_Train = '/content/train-v2.0.json'
SQUAD_Dev = '/content/dev-v2.0.json'

!python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file '$SQUAD_Train' \
  --predict_file '$SQUAD_Dev' \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

```

Here is the error message

```
11/14/2019 05:01:20 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
11/14/2019 05:01:20 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json from cache at /root/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.d7a3af18ce3a2ab7c0f48f04dc8daff45ed9a3ed333b9e9a79d012a0dedf87a6
11/14/2019 05:01:20 - INFO - transformers.configuration_utils -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}

11/14/2019 05:01:21 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt from cache at /root/.cache/torch/transformers/5e8a2b4893d13790ed4150ca1906be5f7a03d6c4ddf62296c383f6db42814db2.e13dbb970cb325137104fb2e5f36fe865f27746c6b526f6352861b1980eb80b1
11/14/2019 05:01:21 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin from cache at /root/.cache/torch/transformers/35d8b9d36faaf46728a0192d82bf7d00137490cd6074e8500778afed552a67e5.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2
11/14/2019 05:01:24 - INFO - transformers.modeling_utils -   Weights of BertForQuestionAnswering not initialized from pretrained model: ['qa_outputs.weight', 'qa_outputs.bias']
11/14/2019 05:01:24 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
11/14/2019 05:01:27 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', device=device(type='cuda'), do_eval=True, do_lower_case=True, do_train=True, doc_stride=128, eval_all_checkpoints=False, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=3e-05, local_rank=-1, logging_steps=50, max_answer_length=30, max_grad_norm=1.0, max_query_length=64, max_seq_length=384, max_steps=-1, model_name_or_path='bert-base-cased', model_type='bert', n_best_size=20, n_gpu=1, no_cuda=False, null_score_diff_threshold=0.0, num_train_epochs=2.0, output_dir='/tmp/debug_squad/', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=8, per_gpu_train_batch_size=12, predict_file='/content/dev-v2.0.json', save_steps=50, seed=42, server_ip='', server_port='', tokenizer_name='', train_file='/content/train-v2.0.json', verbose_logging=False, version_2_with_negative=False, warmup_steps=0, weight_decay=0.0)
11/14/2019 05:01:27 - INFO - __main__ -   Creating features from dataset file at /content/train-v2.0.json
Traceback (most recent call last):
  File "run_squad.py", line 569, in <module>
    main()
  File "run_squad.py", line 514, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
  File "run_squad.py", line 307, in load_and_cache_examples
    version_2_with_negative=args.version_2_with_negative)
  File "/content/utils_squad.py", line 152, in read_squad_examples
    "For training, each question should have exactly 1 answer.")
ValueError: For training, each question should have exactly 1 answer.
```

For convenience, here's a colab notebook with the code that you can run

https://colab.research.google.com/drive/1tNisXX5siuNnkuEQ-X_XdEdDTtdJ0qeL


## ❓ Questions & Help
I am wondering how to use convert_pytorch_checkpoint_to_tf2.py to convert pytorch checkpoint to tensorflow checkpoint.  I found Comparing-PT-and-TF-models.ipynb have an example to use pytorch_pretrained_bert. But It doesn't work for convert_pytorch_checkpoint_to_tf2. Could you please give me any help?
<!-- A clear and concise description of the question. -->


CamemBERT

Code for the CamemBERT model, adapted from https://github.com/pytorch/fairseq/tree/master/examples/camembert