XLnet output_attentions=True raises an exception

## 🐛 Bug

I am working on conditional sentences probabilities based on [this code](https://github.com/huggingface/transformers/issues/917#issuecomment-525297746) and whenever `output_attentions=True` and `target_mapping` is provided, there is an exception thrown.

Model I am using (Bert, XLNet....):  XLNet ('xlnet-base-cased')

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] my own modified scripts: Setting `output_attentions=True`, throws an exception: `AttributeError: 'tuple' object has no attribute 'permute'`.

The tasks I am working on is:
* [x] my own task or dataset: Using just some sample text

## To Reproduce

Here is a [Google Colab notebook](https://colab.research.google.com/drive/1fkNB0Aqlhtvo3CcHWQ6IqxmCm2Qt9Etn) where the issue can be reproduced as well. Just run all cells.

**Code:**
```python
# https://github.com/huggingface/transformers/issues/917#issuecomment-525297746

import torch
from transformers import XLNetTokenizer, XLNetLMHeadModel
import numpy as np
from scipy.special import softmax

PADDING_TEXT = """In 1991, the remains of Russian Tsar Nicholas II and his family
(except for Alexei and Maria) are discovered.
The voice of Nicholas's young son, Tsarevich Alexei Nikolaevich, narrates the
remainder of the story. 1883 Western Siberia,
a young Grigori Rasputin is asked by his father and a group of men to perform magic.
Rasputin has a vision and denounces one of the men as a horse thief. Although his
father initially slaps him for making such an accusation, Rasputin watches as the
man is chased outside and beaten. Twenty years later, Rasputin sees a vision of
the Virgin Mary, prompting him to become a priest. Rasputin quickly becomes famous,
with people, even a bishop, begging for his blessing. <eod> """

text = "The dog is very cute."

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = XLNetLMHeadModel.from_pretrained('xlnet-base-cased', output_attentions=True)

tokenize_input = tokenizer.tokenize(PADDING_TEXT + text)
tokenize_text = tokenizer.tokenize(text)

sum_lp = 0.0
for max_word_id in range((len(tokenize_input)-len(tokenize_text)), (len(tokenize_input))):

    sent = tokenize_input[:]

    input_ids = torch.tensor([tokenizer.convert_tokens_to_ids(sent)])

    perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
    perm_mask[:, :, max_word_id:] = 1.0

    target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)
    target_mapping[0, 0, max_word_id] = 1.0

    with torch.no_grad():
        next_token_logits, attentions = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)

    word_id = tokenizer.convert_tokens_to_ids([tokenize_input[max_word_id]])[0]
    predicted_prob = softmax(np.array(next_token_logits[0][-1]))
    lp = np.log(predicted_prob[word_id])

    sum_lp += lp

print("sentence logprob =", sum_lp)
```

**Stacktrace:**
```shell
AttributeError                            Traceback (most recent call last)
<ipython-input-5-6490f5f4333c> in <module>()
     38
     39     with torch.no_grad():
---> 40         next_token_logits, attentions = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)
     41
     42     word_id = tokenizer.convert_tokens_to_ids([tokenize_input[max_word_id]])[0]

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py in forward(self, input_ids, attention_mask, mems, perm_mask, target_mapping, token_type_ids, input_mask, head_mask, inputs_embeds, labels)
    952                                                input_mask=input_mask,
    953                                                head_mask=head_mask,
--> 954                                                inputs_embeds=inputs_embeds)
    955
    956         logits = self.lm_loss(transformer_outputs[0])

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py in forward(self, input_ids, attention_mask, mems, perm_mask, target_mapping, token_type_ids, input_mask, head_mask, inputs_embeds)
    879             outputs = outputs + (hidden_states,)
    880         if self.output_attentions:
--> 881             attentions = tuple(t.permute(2, 3, 0, 1).contiguous() for t in attentions)
    882             outputs = outputs + (attentions,)
    883

/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlnet.py in <genexpr>(.0)
    879             outputs = outputs + (hidden_states,)
    880         if self.output_attentions:
--> 881             attentions = tuple(t.permute(2, 3, 0, 1).contiguous() for t in attentions)
    882             outputs = outputs + (attentions,)
    883

AttributeError: 'tuple' object has no attribute 'permute'
```

## Expected behavior

The model should output the logits for each token and the attention values across layers, heads, and tokens.

## Environment

* OS: 18.04.3 LTS (Bionic Beaver)
* Python version: 3.6.8
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? No
* Distributed of parallel setup ? N/A
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Why is the weight of linear layer tied to the input embeddings in OpenAIGPTLMHeadModel?

## ❓ Questions & Help
Yes the original GPT paper also uses same `W_e` as both token embedding matrix and linear weight, and seems that many succeeding models like GPT-2, XLNet also use the same matrix. In my perspective, the token embedding matrix and the weight in linear layer have nothing related (though they have both the same shape). Could you please explain a bit of that?


Worse F1 on squad2 with finetune+distil distilroberta-base than just finetune

Hi there,
I am trying to finetune distilroberta on squad2. First, I simply used the _distilroberta-base_ model and finetuned it on the squad2 dataset using `run_squad.py`, which gave me **74/71 F1/EM**. It's a lot worse than the roberta-base accuracy.

Currently, I am trying to finetune+distil (from roberta-base squad2 finetuned model) using `run_squad_w_distillation.py`. My roberta-base squad2 finetuned model has around 83/80 F1/EM. However, when I try to finetune+distil _distilroberta-base_ with the finetuned roberta-base as teacher, I only get around **63/60 F1/EM**. Maybe my hyperparams are way off or I need to train longer? Here's my current config:
- learning_rate=3e-5
- total_batch_size=16
- num_train_epochs=2
- max_seq_length=384

I left all other hyperparams as default.

I also checked out some predictions and it seems the model most of the time predicts _no answer_ as the best answer. In cases where it actually predicts an answer, the accuracy is not that bad.

Would be awesome to get some feedback on this, as I am trying to do inference on CPU and a distilled model would greatly benefit me in this case.

Cheers


Facing AttributeError: 'DataParallel' object has no attribute 'resize_token_embeddings'

## 🐛 AttributeError: 'DataParallel' object has no attribute 'resize_token_embeddings'

<!-- Important information -->

I'm facing AttributeError: 'DataParallel' object has no attribute 'resize_token_embeddings' while performing fine-tuning by using run_lm_finetuning.py.

Following are the arguments:

 python run_lm_finetuning.py --train_data_file=sample_text.txt --model_type=gpt2 --model_name_or_path=gpt2 --output_dir=op --mlm --do_train --overwrite_output_dir --do_lower_case --save_steps=50

I tried to change the model but faced same error :

## Detailed error message

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 556, in <module>
    main()
  File "run_lm_finetuning.py", line 508, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 218, in train
    model.resize_token_embeddings(len(tokenizer))
  File "/nipa/anaconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'DataParallel' object has no attribute 'resize_token_embeddings'


When training QA models, albert-xxlarge-v2 uses much more GPU mem than Bert-large

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,
         When I used run_square.py to train QA model, I found that albert-xxlarge-v2 uses much more GPU mem than Bert-large. Specifically, when using Bert large, I can set `Max_sequence_ength = 512`, `bash_size = 12`. But when I use albert-xlarge-v2, I can only set `Max_sequence_length` = 512, `bash_size = 6`.
In fact, the number of parameters of Albert XXLarge is much less than that of Bert large, and the size of model file is the same. Why does Albert-xxlarge occupy more GPU memory when training QA model? Is it caused by more head parameters?


Will you add XLNet text-generation feature ?

## ❓ Questions & Help

There is `run_generation.py` in example now. Do you have a plan to add feature of  complete lm_finetune and inference? Just like GPT-2.

Thanks


Possible error in the HuggingFace Transformers documentation?

Hello,

According to HuggingFace Transformers documentation website (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel), under the GPT2DoubleHeadsModel, it defines the output lm_prediction_scores as the following:

`lm_prediction_scores: torch.FloatTensor of shape (batch_size, num_choices, sequence_length, config.vocab_size)`

To me this doesn't make sense. Shouldn't the dimension of `lm_prediction_scores` for `GPT2DoubleHeadsModel` be just `(batch_size, sequence_length, config.vocab_size)`? [no `num_choices` in the middle]

Thank you,


Saving and resuming

Here's my basic implementation of the saving and resuming improvements discussed in #1960. So far, I've only modified the `run_lm_finetuning` example, but if my changes are approved I can update the rest of the examples as well.

There are three main changes:

1. The example now saves the optimizer, scheduler, and tokenizer every `save_steps` iterations.
2. The example now checks whether training is being continued from a checkpoint, and if so, looks for a saved optimizer and scheduler and loads them in.
3. The example checks whether training is being continued from a checkpoint, and if so, gets the global step of the checkpoint and continues training from the last saved global step.


Fine Tuning Bert for Q&A

This is more of a theoretical question.  I would like to use a Bert Model trained on SQUAD 2.0 then train it further on my domain's Q&A dataset.

How would I do that.  I've read through the code.  As I understand it, I see that the BertforQuestionAnswering would be what I would need to use loaded with a model that is fine tuned on Squad so the weights match the architecture.

But now I want to further fine tune this model to include Q&A training data from my target domain.  How would I do that?


run_squad.py for tf

Is there any version of the script for fine tunning on Squad using tensorflow?


[WIP] Squad refactor

This PR aims to refactor SQuAD to make it usable with all models with question answering heads, and without having to build the entire tokenization pipeline as it is currently done.

- It is based on processors that manage data, similarly to the GLUE processors. The two new processors are `SquadV1Processor` and `SquadV2Processor`. They'll probably be merged into a single `SquadProcessor` as the difference between the two versions is minimal.
- It leverages powerful abstractions made for the `run_glue` refactor a few months ago that greatly simplified the tokenization pipeline
- It can be interfaced with the package `tensorflow_datasets`.
- It better respects the library-wide naming, with `attention_mask` instead of `input_mask` and `token_type_ids` instead of `segment_ids`, among others.
- Introduces padding to `encode` and `encode_plus`, alongside tests.

It is still a work on progress but some aspects of it are working.

### Left to do

- [x] Add the processors to `__init__.py`
- [x] Patch the evaluation so that it leverages the current interface
- [x] Patch the evaluation so that it may work with tfds
- [x] Modify the run arguments to reflect the changes
- [x] Remove the `only_first` argument which would only be used for testing
- [x] Update tests running the `run_squad.py` script
- [x] Include the padding location in the tokenizers and reflect the changes in the feature converter
- [x] Test that all current models can train and evaluate (BERT, RoBERTa, XLNet, XLM)
- [x] Add the last models (DistilBERT, ALBERT, ...)
- [x] Return datasets (maybe only pytorch TensorDataset for now)
- [x] Documentation
- [x] Short examples showcasing the simply usage in the processors section.
- [x] Patch the evaluation for impossible questions

### Running sample

Here's the major difference from the user's perspective. Initially, to obtain the examples which were then converted to features, the user had to do as follows (taken from the current `run_squad.py`), which only works for BERT/XLNet/DistilBERT/ALBERT:

```py

examples = read_squad_examples(
    input_file=input_file,
    is_training=not evaluate,
    version_2_with_negative=args.version_2_with_negative
)

features = convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=args.max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=args.max_query_length,
    is_training=not evaluate,
    cls_token_segment_id=2 if args.model_type in ['xlnet'] else 0,
    pad_token_segment_id=3 if args.model_type in ['xlnet'] else 0,
    cls_token_at_end=True if args.model_type in ['xlnet'] else False,
    sequence_a_is_doc=True if args.model_type in ['xlnet'] else False
)
```
In order to obtain the exact same results, the user now has to do as follows, which will be completely model independant once the `sequence_a_is_doc` is integrated in our sequence pair tokenization methods:

```py
processor = SquadV1Processor()
examples = processor.get_dev_examples("examples/squad") if evaluate else processor.get_train_examples("examples/squad")
features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=args.max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=args.max_query_length,
    is_training=not evaluate,
    sequence_a_is_doc=True if args.model_type in ['xlnet'] else False
)
```

The same can be done by using TFDS instead, removing the need to specify a file. The two initial lines now become:

```py
tfds_examples = tensorflow_datasets.load("squad")["validation"] if evaluate else tensorflow_datasets.load("squad")["train"]
examples = SquadV1Processor().get_examples_from_dataset(tfds_examples)
```


add special tokens

Hello
I tried to add special tokens to bert tokenizer via add_special_tokens:
```
tokenizer.add_special_tokens({'additional_special_tokens':['SS']})
```
But I got CUDA error
```
CUDA error: device-side assert triggered
```
The code runs without adding additional_special_tokens!
Any idea?


Error when running the script 'run_summarization_finetuning.py'

## 🐛 Bug

<!-- Important information -->

Model I am using Bert:

Language I am using the model on English:

The problem arise when using:
* [x] the official example scripts: run_summarization_finetuning.py

The tasks I am working on is:
* [x] other task or dataset: summarization, cnn-dailymail dataset(stories)

## To Reproduce

Steps to reproduce the behavior:

1.Download cnn-dailymail dataset from [official website](https://cs.nyu.edu/~kcho/DMQA/)(only 'stories' for 'cnn' & 'dailymail').
2.Use the path which refers to the location of dataset to specify parameter '--data_dir' and specify '--output_dir' with my own dir.
3.Run the script 'run_summarization_finetuning.py' with command 'CUDA_VISIBLE_DEVICES=0 python run_summarization_finetuning.py --data_dir xxx --output_dir xxxx' via pycharm. (anonymous path 'xxx' & 'xxxx')

Traceback (most recent call last):
  File "/root/.pycharm_helpers/pydev/pydevd.py", line 1741, in <module>
    main()
  File "/root/.pycharm_helpers/pydev/pydevd.py", line 1735, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/root/.pycharm_helpers/pydev/pydevd.py", line 1135, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/root/transformers/examples/run_summarization_finetuning.py", line 487, in <module>
    main()
  File "/root/transformers/examples/run_summarization_finetuning.py", line 452, in main
    global_step, tr_loss = train(args, model, tokenizer)
  File "/root/transformers/examples/run_summarization_finetuning.py", line 219, in train
    for step, batch in enumerate(epoch_iterator):
  File "/root/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py", line 1002, in __iter__
    for obj in iterable:
  File "/root/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 560, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/root/transformers/examples/run_summarization_finetuning.py", line 74, in collate
    encode_for_summarization(story, summary, tokenizer) for story, summary in data
  File "/root/transformers/examples/run_summarization_finetuning.py", line 74, in <listcomp>
    encode_for_summarization(story, summary, tokenizer) for story, summary in data
  File "/root/transformers/examples/utils_summarization.py", line 143, in encode_for_summarization
    for line in story_lines
  File "/root/transformers/examples/utils_summarization.py", line 143, in <listcomp>
    for line in story_lines
AttributeError: 'BertTokenizer' object has no attribute 'add_special_tokens_single_sequence'

## Expected behavior

Complete the missing function 'add_special_tokens_single_sequence' in BertTokenizer and other related 'Tokenizer', such as xlnet, roberta...

## Environment

* OS: Ubuntu 5.4.0-6ubuntu1~16.04.9
* Python version: python3.7(anaconda)
* PyTorch version:1.1.0
* PyTorch Transformers version (or branch):transformers-2.2.0
* Using GPU ?Yes
* Distributed of parallel setup ?No


Transformers for WebNLG tasks

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Can we leverage GPT-2 pre-trained model for WebNLG tasks ?http://webnlg.loria.fr/pages/challenge.html

The WebNLG challenge consists in mapping data to text
similar to what is being done in https://github.com/tyliupku/wiki2bio.


update all tf.shape and tensor.shape to shape_list

We need to use the special method `shape_list` from `modeling_tf_utils` to be sure we can get TF 2.0 tensor shapes both in eager and non-eager mode.

This PR fixes this for all TF 2.0 models and templates.


AlbertForQuestionAnswering

Hello! Thanks for adding Albert so quickly! I have a problem with Albert answering a simple question from the Huggingface default example:
```
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForQuestionAnswering.from_pretrained('albert-base-v2')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
input_ids = tokenizer.encode(input_text)
token_type_ids = [0 if i <= input_ids.index(3) else 1 for i in range(len(input_ids))] # for albert [SEP] token has id 3
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
```
It actually shows empty output because
```
torch.argmax(start_scores), torch.argmax(end_scores)+1
## (tensor(7), tensor(6))
```
For other versions of Albert I also get some nonsense results :(
Thanks in advance


Modify position_embeddings from pre_trained model

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

When I load a model like below:
`model1 = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
```
BertForSequenceClassification(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(30522, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): BertEncoder(
      (layer): ModuleList(...
```

I want to change Embedding size from 512 to 1024, but when I try to add like this and get an error:
`model = BertForSequenceClassification.from_pretrained('bert-base-uncased', max_position_embeddings=1024)`
> RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
	size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([1024, 768]).

May I know how to change configs of pre-trained model layers?


'convert_tf_checkpoint_to_pytorch.py' file is missing

Hi all,
I pre-trained BERT base model on my domain-specific corpus using ```https://github.com/google-research/bert```  ```create_pretraining_data.py``` and ```run_pretraining.py```.
Now, I want to use it with this ```pytorch-transformers```. I saw from this page https://devhub.io/repos/huggingface-pytorch-pretrained-BERT that there is conversion script from tf checkpoints to ```pytorch_model.bin``` called ```convert_tf_checkpoint_to_pytorch.py``` but the file no longer exists.
Does anyone have solution? Thanks!


Merge pull request #1 from huggingface/master

merge


How can we view different versions of documentation?

## ❓ Questions & Help

How can I select a specific version of transformers in the documentation located here: https://huggingface.co/transformers/index.html?


Albert Hyperparameters for Fine-tuning SQuAD 2.0

## ❓ Questions & Help

I want to fine-tune `albert-xxlarge-v1` on SQuAD 2.0 and am in need of optimal hyperparameters.  I did not find any discussion in the Albert original paper regarding suggested fine-tuning hyperparameters, as is provided in the XLNet original paper.  I did find the following hard-coded parameters in the Google-research Albert `run_squad_sp.py` code:
```
'do_lower_case' = True
'train_batch_size' = 32
'predict_batch_size' = 8
'learning_rate' = 5e-5
'num_train_epochs' = 3.0
'warmup_proportion' = 0.1
```
With fine-tuning on my 2x GPUs taking ~69 hours, I'd like to shrink the number of fine-tuning iterations necessary to attain optimal model performance.  Anyone have a bead on the optimal hyperparameters?

Also, Google-research comments in `run_squad_sp.py` state that `warmup_proportion` is "Proportion of training to perform linear learning rate warmup for."  "E.g.,  0.1 = 10% of training".  Since 3 epochs, batch size = 32 while fine-tuning SQuAD 2.0 results in approximately 12.5K total optimization steps, would I set `--warmup_steps = 1250` when calling Transformers' run_squad.py?

Thanks in advance for any input.


Changes to S3 Roberta / RobertaForSequenceClassification

Hello,
I'm wondering if there have been any changes in either the pretrained Roberta model or the configuration of RobertaForSequenceClassification within the past month or so. I am initializing it as `RobertaForSequenceClassification.from_pretrained(...)` and running it as demonstrated in `run_glue.py`.

For a custom dataset, I am noticing that the fine-tune accuracy has decreased by several percentage points on newly trained models as opposed to ~ 1 month ago. This happens consistently (i.e I've tried retraining multiple times) and using the same hyperparams.

In addition, I've noticed that the new training times are cut by ~1/2, so the model seems to train faster, but is less performant.


How to persist cloud-based transformers

## ❓ Questions & Help

I'm using this repo in the cloud and attempting to persist the model fails as it seems HDF5 and thus h5py doesn't support that paradigm per https://github.com/h5py/h5py/issues/925

What is the recommended method of saving the model in this scenario? Thanks!


add add_special_tokens=True for input examples

According to #1957 , in some versions of transformers, add_special_tokens is not set to True by default. In that case, the example code is wrong as input_ids will missing [CLS] and [SEP] tokens. It's better to pass add_special_tokens=True in the example explicitly.


Bert Tensor Dimensions

## 🐛 Bug

My code was working on 2.1 it throws an error in 2.2:

    transformers\modeling_tf_bert.py:777 call
        outputs = self.bert(inputs, **kwargs)
    transformers\modeling_tf_bert.py:512 call
        attention_mask = tf.fill(input_shape, 1)
    tensorflow_core\python\ops\array_ops.py:171 fill
        result = gen_array_ops.fill(dims, value, name=name)
    tensorflow_core\python\ops\gen_array_ops.py:3602 fill
        "Fill", dims=dims, value=value, name=name)
    tensorflow_core\python\framework\op_def_library.py:545 _apply_op_helper
        (input_name, err))

    ValueError: Tried to convert 'dims' to a tensor and failed. Error: Cannot convert a partially known TensorShape to a Tensor: (None, 72)

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: predict function

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

## Environment

* OS:
* Python version: 3.7.4
* PyTorch version: -
* Tensorflow :  2.0.0
* Using GPU : Yes
* Distributed of parallel setup : No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Implemented concurrent encoding and converting of sequences for data binarization


AlbertPreTrainedModel class is not available in release v2.2.0

## ❓ Questions & Help

In the release v2.2.0, the AlbertForSequenceClassification class inherits the AlbertPreTrainedModel class as "class AlbertForSequenceClassification(AlbertPreTrainedModel)".

However, in the doc, this pre-trained model is not documented. And in the released v2.2.0 transformer, the AlbertPreTrainedModel class is not available to be imported.

This is not a big issue since we can use BertPreTrainedModel class instead (like RoBERTa). But it should be consistent. Especially in the doc under Albert, there is a class called TFBertForPretraining which makes confusion for users.


Trouble running 'bert-base-multilingual-cased'

## ❓ Questions & Help

Hi,
I got trouble when I apply Quickstart BERT example to korean news text.

but when I run this code using model 'bert-base-multilingual-cased',

```{.python}
# Predict hidden states features for each layer
with torch.no_grad():
    # See the models docstrings for the detail of the inputs
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
    # Transformers models always output tuples.
    # See the models docstrings for the detail of all the outputs
    # In our case, the first element is the hidden state of the last layer of the Bert model
    encoded_layers = outputs[0]
# We have encoded our input sequence in a FloatTensor of shape (batch size, sequence length, model hidden dimension)
assert tuple(encoded_layers.shape) == (1, len(indexed_tokens), model.config.hidden_size)
```

 I sometimes got error like below,

```{.python}
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-39-b8e77f8ffc14> in <module>
      2 with torch.no_grad():
      3     # See the models docstrings for the detail of the inputs
----> 4     outputs = model(tokens_tensor, token_type_ids=segments_tensors)
      5     # Transformers models always output tuples.
      6     # See the models docstrings for the detail of all the outputs

//anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

//anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    712             head_mask = [None] * self.config.num_hidden_layers
    713
--> 714         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    715         encoder_outputs = self.encoder(embedding_output,
    716                                        attention_mask=extended_attention_mask,

//anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

//anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    176             inputs_embeds = self.word_embeddings(input_ids)
    177         position_embeddings = self.position_embeddings(position_ids)
--> 178         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    179
    180         embeddings = inputs_embeds + position_embeddings + token_type_embeddings

//anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

//anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

//anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: index out of range: Tried to access index 2 out of table with 1 rows. at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

Any help will be greatly appreciated.
Thanks!

Info:
OS: MacOsX 10.14.6 (Mojave)
python : 3.7
PyTorch : 1.3.1


Fix issue: #1962, input shape problem

Hi,

To Fix #1962
The input's shape seem to cause error in 2.2.0 version tf_albert_model

Hope that it can help.


XLMForTokenClassification

# 🌟New model addition

## Model description
As the XLM version for token classification (NER tasks) was not implemented, I followed a similar path as for BertForTokenClassification, and it seems to work. The reason for this is that BERT multilingual works horribly for WikiNER in spanish, achieving only 55% F1 with much fine-tuning, against the 87% of Spacy. Therefore, I'm trying to improve this metric and for that purpose I decided to use XLM, which is trained only on 15 different languages, not on more than 100. There's still one think that my model implementation lacks, and that is the fact that model dimension has to be set manually. I've been trying to add d_model to XLMConfig and then pass this config to my class, but it says XLMModel has no attribute d_model. If anyone can help me out with that I'd appreciate that.
<!-- Important information -->

The code:

class XLMForTokenClassification(XLMModel):

    def __init__(self, config, d_model=1024):
        super(XLMForTokenClassification, self).__init__(config)
        self.num_labels = config.num_labels
        self.xlm = XLMModel(config)
        self.dropout = nn.Dropout(config.dropout)
        self.classifier = nn.Linear(d_model, config.num_labels)
        self.init_weights()

    def forward(self, input_ids=None, attention_mask=None, langs=None, token_type_ids=None,
               position_ids=None, head_mask=None, labels=None):

        outputs = self.xlm(input_ids,
                          attention_mask=attention_mask,
                          langs=langs,
                          token_type_ids=position_ids,
                          head_mask=head_mask)

        sequence_output = self.dropout(outputs[0])
        logits = self.classifier(sequence_output)

        outputs = (logits, ) + outputs[2:] #add hidden states and attention if they are here

        if labels is not None:

            loss_fct = nn.CrossEntropyLoss()
            #only keep active parts of the loss
            if attention_mask is not None:

                active_loss = attention_mask.view(-1) == 1
                active_logits = logits.view(-1, self.num_labels)[active_loss]
                active_labels = labels.view(-1)[active_loss]
                loss = loss_fct(active_logits, active_labels)

            else:
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

            outputs = (loss, ) + outputs

        return outputs

## Open Source status

* [x] the model implementation is available: (give details)
It is uploaded above.
* [ ] the model weights are available: (give details)
* [x] who are the authors: (mention them)
Alejandro Vaca Serrano

## Additional context

<!-- Add any other context about the problem here. -->


How to increase model saving checkpoint from 50 to 1000?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When I run the script `run_squad.py` , it creates model check points every 50th iteration. How to increase model saving checkpoint from 50 to 1000. Where should I edit the code ?


Did the underlying pre-trained models change somehow?

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: I'm just loading the pre-trained bert-base-uncased model

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: document classification

## To Reproduce

Steps to reproduce the behavior:

1. `PRETRAINED_WEIGHTS = "bert-base-uncased"`
2. `model = TFBertForSequenceClassification.from_pretrained(PRETRAINED_WEIGHTS)`
3. see just below
```
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
sparse_categorical_accuracy = tf.keras.metrics.SparseCategoricalAccuracy("train_accuracy")
model.compile(optimizer=optimizer,
              loss=loss,
              metrics=[sparse_categorical_accuracy])
```
4. see just below
```
history = model.fit([train_input_ids, train_input_masks, train_segment_ids],
                    train_labels,
                    epochs=EPOCHS,
                    batch_size=BATCH_SIZE,
                    validation_data=([val_input_ids, val_input_masks, val_segment_ids],
                                     val_labels),
                    use_multiprocessing=True,
                    verbose=1)
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

That the model begins training.

## Environment

* OS: Databricks (linux?)
* Python version: 3.7
* PyTorch version: I'm using the TF-flavor models
* PyTorch Transformers version (or branch): v2.1.1 & v2.2.0 (see additional context)
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context
In databricks I was not pinning the transformers version so it was installing the latest. I have always pinned the `mlflow` version to 1.4.0 though. I tried with what the latest prior release of `transformers` was (2.1.1) and still got the same error where this worked flawlessly before. The error is below and it specifies it is an `mlflow` issue, though in reality I think it may have something to do with the pretrained model that is loaded when we specify `bert-base-uncased`. It seems this underlying model changed independently of the latest release of `transformers`? Are the pre-trained models from some public Google repository or are they Huggingface-specific?

Thanks again for supporting TF 2, this repo has been a blessing!

Traceback:

```
UserWarning: Logging to MLflow failed: Saving the model to HDF5 format requires the model to be a Functional model or a Sequential model. It does not work for subclassed models, because such models are defined via the body of a Python method, which isn't safely serializable. Consider saving to the Tensorflow SavedModel format (by setting save_format="tf") or using `save_weights`.
  try_mlflow_log(mlflow.keras.log_model, self.model, artifact_path='model')

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Incompatible shapes: [4,12,512,512] vs. [4,1,1,0]
	 [[node tf_bert_for_sequence_classification/bert/encoder/layer_._0/attention/self/add (defined at /databricks/python/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
	 [[cluster_1_1/xla_compile]]
	 [[cluster_0_1/merge_oidx_1/_22]]
  (1) Invalid argument:  Incompatible shapes: [4,12,512,512] vs. [4,1,1,0]
	 [[node tf_bert_for_sequence_classification/bert/encoder/layer_._0/attention/self/add (defined at /databricks/python/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
	 [[cluster_1_1/xla_compile]]
0 successful operations.
0 derived errors ignored. [Op:__inference_distributed_function_37652]

Function call stack:
distributed_function -> distributed_function
```
<!-- Add any other context about the problem here. -->


TFBertModel ValueError: Tried to convert 'dims' to a tensor and failed. Error: Cannot convert a partially known TensorShape to a Tensor

The code below was fine when transformers 2.1.1

but after I update to transformers 2.2.0
```
model = TFBertForSequenceClassification.from_pretrained('bert-base-chinese', num_labels=5)
model.summary()

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

model_fit = model.fit(train_input_ids, train_label,
                      batch_size=8, epochs=1,
                    validation_data=(validation_input_ids, validation_label)
                   )
```
```
ValueError: Tried to convert 'dims' to a tensor and failed.
Error: Cannot convert a partially known TensorShape to a Tensor: (None, 512)
```


Problems with running 'run_lm_finetuning.py' with bert

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
My rough modification configuration (local)
    root_path = "F://IdeaProjects/transformers"
    bert_path = "F://BERT/chinese_L-12_H-768_A-12" （Downloaded from the 'https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip'）
    ## Required parameters
    parser.add_argument("--train_data_file", default=os.path.join(root_path, "data/train.txt"),...)
    parser.add_argument("--output_dir", default=os.path.join(root_path, "output")...)
    ## Other parameters
    parser.add_argument("--eval_data_file", default=os.path.join(root_path, "data/dev.txt"),...)
    parser.add_argument("--model_type", default="bert"...)
    parser.add_argument("--model_name_or_path", default=bert_path, ...)
    parser.add_argument("--mlm", action='store_true', default=True,...)
    parser.add_argument("--mlm_probability", type=float, default=0.15,...)
    parser.add_argument("--config_name", default=os.path.join(bert_path, "bert_config.json"),..)
    parser.add_argument("--tokenizer_name", default=bert_path, ...)

- [ Is my configuration correct?]
 I changed this line, 'model = model_class.from_pretrained(args.model_name_or_path,
from_tf=True,#bool('.ckpt' in args.model_name_or_path),' Because my filename does not have this character（'.ckpt'）.

Then running 'run_lm_finetuning.py' causes the problem in the 'modeling_bert.py' file of the 'load_tf_weights_in_bert' method：
     'BertOnlyMLMHead' object has no attribute 'bias'


Improving model saving and resuming

## 🚀 Feature

The official transformer examples should make it easier to continue training a model that suddenly stopped (e.g. vm gets preempted in the middle of a training run).

To do this, the examples should be updated to save the `optimizer` and `scheduler` states to the `output_dir` as well as the current epoch to disk at the end of each epoch in a `training_state.pt` file. This way, the user could choose to continue training from a previous model checkpoint, but would continue training from the saved epoch and would use the saved tokenizer, optimizer, and scheduler.

## Motivation

If your VM gets preempted in the middle of a training run, you won't be able to properly continue training the model since the scheduler will be reset and the current learning rate will be lost.

## Additional context

If anyone is interested in this, I can implement the feature and start a pull request.

Related issues:

- https://github.com/huggingface/transformers/issues/1925
- https://github.com/huggingface/transformers/issues/839


update Roberta checkpoint conversion

- update to fix fairseq Roberta checkpoint conversion

Fairseq had removed `in_proj_weight` and `in_proj_bias` from the self attention module:
https://github.com/pytorch/fairseq/commit/4c6b689eebe66a53717dacf28cba7a11b6ffa64f

- create save directory if not exist


run_ner.py --do_predict inference mode errors. Right data format?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello again,

I'm here to bother you one more time.

I fine-tuned preloaded BioBERT weights on a custom dataset to run biomedical NER.

Now I want to use the model for inference mode on a 'raw' set of documents. I renamed this set 'test.txt' and formatted it the following way (documents are separated by '-DOCSTART- (num_doc)' lines):

```
to O
be O
referred O
to O
the O
location O
of O
the O
disease O
in O
the O
skeletal O
structures O
examined O
; O

unchanged O
the O
areas O
of O
bone O
rarefaction O
reported O
to O
the O
sternum O
as O
a O
result O
of O
median O
sternotomy O
. O
```

I had to add the 'fake' labels on the right and place a space " " between col1 and col2.

The error I now get is:
```
Traceback (most recent call last):
  File "run_ner.py", line 531, in <module>
    main()
  File "run_ner.py", line 522, in main
    output_line = line.split()[0] + " " + predictions[example_id].pop(0) + "\n"
IndexError: list index out of range
```
Many thanks again.


Do we need to add [CLS] and [SEP] for BertForMaskedLM ?

## ❓ Questions & Help

https://github.com/huggingface/transformers/blob/cc7968227e08858df4a5c618c739e1a3ca050196/transformers/modeling_bert.py#L837-L841

Seems like the example is wrong?


get_linear_schedule_with_warmup Scheduler

Hello,

When I try to execute the line of code below, Python gives me an import error:

```js
from pytorch_transformers import (GPT2Config, GPT2LMHeadModel, GPT2DoubleHeadsModel,
                                  AdamW, get_linear_schedule_with_warmup)

ImportError: cannot import name 'get_linear_schedule_with_warmup' from 'pytorch_transformers' (/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/pytorch_transformers/__init__.py)
```

What should I then import to use the linear scheduler with warm up?

Thank you,


run_squad.py crashes during do_eval

## 🐛 Bug

When running run_squad.py as provided in the README once training is complete the prediction/evaluation component of the script crashes.

No prediction files are written are written.

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details) example squad fine tuning

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) official squad dev set 1.1
* [ ] my own task or dataset: (give details) the checkpoint was made using a custom training dataset in squad format, but it appears to be an eval bug

## To Reproduce

Steps to reproduce the behavior:

1. finish training as specified in README
2. I ran with this command
CUDA_VISIBLE_DEVICES=10,11,12,13,14,15 python -m torch.distributed.launch --nproc_per_node=6 ./examples/run_squad.py     --model_type bert     --model_name_or_path bert-large-uncased-whole-word-masking     --do_train     --do_eval     --do_lower_case     --train_file /data/data/SQUAD/train-v1.1json     --predict_file /data/data/SQUAD/dev-v1.1.json     --learning_rate 3e-5     --num_train_epochs 2     --max_seq_length 384     --doc_stride 128     --output_dir models/wwm_uncased_finetuned_squad_supp/     --per_gpu_eval_batch_size=6       --per_gpu_train_batch_size=6 --save_steps 500

I also ran with just do_eval using the same model and it produced the same error

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

11/26/2019 18:29:01 - INFO - __main__ -   Saving features into cached file /data/data/SQUAD/cached_dev_bert-large-uncased-whole-word-masking_384
11/26/2019 18:29:20 - INFO - __main__ -   ***** Running evaluation  *****
11/26/2019 18:29:20 - INFO - __main__ -     Num examples = 10833
11/26/2019 18:29:20 - INFO - __main__ -     Batch size = 6
Evaluating: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 301/301 [00:48<00:00,  6.20it/s]
11/26/2019 18:30:09 - INFO - __main__ -     Evaluation done in total 48.868770 secs (0.004511 sec per example)
11/26/2019 18:30:09 - INFO - utils_squad -   Writing predictions to: models/wwm_uncased_finetuned_squad_supp/predictions_.json
11/26/2019 18:30:09 - INFO - utils_squad -   Writing nbest to: models/wwm_uncased_finetuned_squad_supp/nbest_predictions_.json
Traceback (most recent call last):
  File "./examples/run_squad.py", line 573, in <module>
    main()
  File "./examples/run_squad.py", line 562, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "./examples/run_squad.py", line 284, in evaluate
    args.version_2_with_negative, args.null_score_diff_threshold)
  File "/home/clong/git/transformers/examples/utils_squad.py", line 532, in write_predictions
    result = unique_id_to_result[feature.unique_id]
KeyError: 1000000000
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 253, in <module>
    main()
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 249, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', './examples/run_squad.py', '--local_rank=5', '--model_type', 'bert', '--model_name_or_path', 'bert-large-uncased-whole-word-masking', '--do_eval', '--do_lower_case', '--train_file', '/data/data/SQUAD/train-v1.1.json', '--predict_file', '/data/data/SQUAD/dev-v1.1.json', '--learning_rate', '3e-5', '--num_train_epochs', '2', '--max_seq_length', '384', '--doc_stride', '128', '--output_dir', 'models/wwm_uncased_finetuned_squad_supp/', '--per_gpu_eval_batch_size=6', '--per_gpu_train_batch_size=6', '--save_steps', '500']' returned non-zero exit status 1.

## Expected behavior

no crashing and predictions written

## Environment

* OS: Ubuntu 18.04 in NVIDIA pytorch container
* Python version: 3.6.9 anaconda
* PyTorch version: 1.3.0 custom nvidia version
* PyTorch Transformers version (or branch): pip install
* Using GPU ? yes
* Distributed of parallel setup ? using 6 of 16 gpu's on system
* Any other relevant information:


BertForMultipleChoice

## ❓ Questions & Help
Shape Error when using BrtForMultipleChoice

![image](https://user-images.githubusercontent.com/26097066/69660490-0ff9b680-1046-11ea-89a3-fa60b769be97.png)

Below is the model i used:

![image](https://user-images.githubusercontent.com/26097066/69660530-26a00d80-1046-11ea-9cbd-b71d56a8bd22.png)


the output type of TFBertModel is weird

```
model = TFBertModel.from_pretrained('bert-base-chinese')
model.summary()

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

predictions = model.predict(validation_input_ids)
print(type(predictions))
print(predictions.shape)
```
```
<class 'list'>
AttributeError: 'list' object has no attribute 'shape'
```
The type is weird.
It is a (N, 512, 768) shape numpy array inside a List.

I had to take it out from the List
```
predictions = predictions[0]
print(predictions.shape)
```
```
(8359, 512, 768)
```


suggest to track repo w/ https rather than ssh

cf #1943 we tell users to track the repository via https rather than ssh (as it requires us to enable ssh).


Benchmark not replicable

## ❓ Questions & Help
Hello. I wanted to test if everything is allright with my downloads and so I ran the code snippet you provided in the section **Fine-tuning Bert model on the MRPC classification task** in the main README file (the only difference being the number of gpus-I use 4). However, my evaluation results are well below the ones you mention. I get
```
acc = 0.8725490196078431
acc_and_f1 = 0.888829254329469
f1 = 0.9051094890510949
```
in my terminal.

There is also no output file in the specified folder.

Do you know what could cause this?

Thanks for answer
MS


word or sentence embedding from BERT model

How can I extract embeddings for a sentence or a set of words directly from pre-trained models (Standard BERT)? For example, I am using Spacy for this purpose at the moment where I can do it as follows:

sentence vector:
`sentence_vector = bert_model("This is an apple").vector`

word_vectors:
```
words = bert_model("This is an apple")
word_vectors = [w.vector for w in words]
```

I am wondering if this is possible directly with huggingface pre-trained models (especially BERT).


Can i train my own text corpus

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I get the idea of using `from_pretrained` but Can i train my own text corpus and then get the weights? If So how?


Should I use `attention_mask`?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
#### My Target
1. Transfer sentences to `ids`
2. Padding `ids` when
3. Encode `ids` to vector(`last_hidden_states`)
4. Put this vector to my own downstream model.

#### My code
```py
original_text = "Hello world!"
ids = model.encode(original_text)
padding_mask = [1] * len(ids)  # Padding mask
while ids < max_length:
    ids.append(0)  # Padding
    padding_mask.append(0)  # Mask using 0

# === Use `attention_mask` ===
outputs = model(ids, attention_mask=padding_mask)
last_hidden_states = outputs[0]  # Get vector such as '<CLS>'
```

#### But...
In official demo code, **it does not use padding mask**:
```py
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
# === Not use `attention_mask` ===
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```
So, why? Should I use `attention_mask`?
Thanks!


Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
fine tuning bert  use wikitext2


Fixed typo

Changed `emove` to `remove`


When using the Bert model

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
pregenerate_training_data.py and  finetune_on_pregenerated.py Is it in the project.If so, where?


tokenization progress made more sensible via tqdm

Because tokenization takes a relatively long time, having progress being visualized via tqdm would be nice.


git@github.com: Permission denied (publickey) when fetching

When you wanted to sync forked repository with base, as described [here](https://github.com/huggingface/transformers/blame/aa92a184d2b92faadec975139ad55e2ae749362c/CONTRIBUTING.md#L140)

You get:
```
➜  transformers (master) ✗ git fetch upstream
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```
[This](https://stackoverflow.com/a/34081606) solution introduced on stackoverflow fixes the problem. [This](https://stackoverflow.com/questions/13509293/git-fatal-could-not-read-from-remote-repository#comment85002398_34081606) one also says:

> If the repo owner has not set up ssh keys then you will likely have this issue. The fix as indicated is to use https instead, or have the repo owner set up ssh

Could you please fix this (by setting up ssh?), in order to make contributing easy?
Thanks!


Wrong paraphrase in the TF2/PyTorch README example.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):  TFBertForSequenceClassification

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: https://github.com/huggingface/transformers#quick-tour-tf-20-training-and-pytorch-interoperability
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Sequence Classification
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Run the attached script.
2. Observe
```
$ /Users/igor/projects/ml-venv/bin/python /Users/igor/projects/transformers-experiments/paraphrasing_issue.py
2019-11-25 08:58:53.985213: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fed57a2be00 executing computations on platform Host. Devices:
2019-11-25 08:58:53.985243: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
INFO:absl:Overwrite dataset info from restored data version.
INFO:absl:Reusing dataset glue (/Users/igor/tensorflow_datasets/glue/mrpc/0.0.2)
INFO:absl:Constructing tf.data.Dataset for split None, from /Users/igor/tensorflow_datasets/glue/mrpc/0.0.2
Train for 115 steps, validate for 7 steps
Epoch 1/2
  4/115 [>.............................] - ETA: 1:22:04 - loss: 0.6936  5/115 [>.............................] - ETA: 1:18:44 - loss: 0.6876  6/115 [>.............................] - ETA: 1:16:01 - loss: 0.6760115/115 [==============================] - 4587s 40s/step - loss: 0.5850 - accuracy: 0.7045 - val_loss: 0.4695 - val_accuracy: 0.8137
Epoch 2/2
115/115 [==============================] - 4927s 43s/step - loss: 0.3713 - accuracy: 0.8435 - val_loss: 0.3825 - val_accuracy: 0.8358
**sentence_1 is a paraphrase of sentence_0
sentence_2 is a paraphrase of sentence_0**
```
3. Wonder why.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

# Load the TensorFlow model in PyTorch for inspection
model.save_pretrained('./save/')
pytorch_model = BertForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = "This research was consistent with his findings."
sentence_1 = "His findings were compatible with this research."
sentence_2 = "His findings were not compatible with this research."
inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')

pred_1 = pytorch_model(inputs_1['input_ids'], token_type_ids=inputs_1['token_type_ids'])[0].argmax().item()
pred_2 = pytorch_model(inputs_2['input_ids'], token_type_ids=inputs_2['token_type_ids'])[0].argmax().item()

print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")
```

## Expected behavior

```
sentence_1 is a paraphrase of sentence_0
sentence_2 is not a paraphrase of sentence_0
```

## Environment

* OS: MacOS
* Python version: 3.7.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch):   last commit afaa33585109550f9ecaaee4e47f187aaaefedd0 as of Sat Nov 23 11:34:45 2019 -0500.
* Using GPU ? nope
* Distributed of parallel setup ? single machine
* Any other relevant information: TF version is 2.0.0


NER - sciBERT weights not initialized.

## ❓ Questions & Help

<!-- Custom weights not initialized when training NER on dataset. -->

Hi all,

first of all thanks for this awesome interface.

Coming to the issue:
I am trying out NER on the Anatem dataset, using Google Colab's GPU.
I imported SciBERT (and BioBERT) models with the solutions provided in issue [457](https://github.com/huggingface/transformers/issues/457).

For clarity, batch_size is 8 because when set to 16 the GPU goes into seg fault.

the scritp is the following. I am into the `transformers/examples` folder
```
!python3 run_ner.py --data_dir ../datasets/ \
--model_type bert \
--labels ../datasets/labels.txt \
--model_name_or_path ../scibert_model \
--output_dir ../results_scibert \
--max_seq_length  512 \
--num_train_epochs 3 \
--per_gpu_train_batch_size 8 \
--save_steps 750 \
--seed 1 \
--do_train \
--do_eval \
--do_predict
```
And the warning is:
```
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   Model name '../scibert_model' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). Assuming '../scibert_model' is a path or url to a directory containing tokenizer files.
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   Didn't find file ../scibert_model/added_tokens.json. We won't load it.
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   Didn't find file ../scibert_model/special_tokens_map.json. We won't load it.
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   Didn't find file ../scibert_model/tokenizer_config.json. We won't load it.
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   loading file ../scibert_model/vocab.txt
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   loading file None
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   loading file None
11/25/2019 16:47:14 - INFO - transformers.tokenization_utils -   loading file None
11/25/2019 16:47:14 - INFO - transformers.modeling_utils -   loading weights file ../scibert_model/pytorch_model.bin
11/25/2019 16:47:20 - INFO - transformers.modeling_utils -   Weights of BertForTokenClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
11/25/2019 16:47:20 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in BertForTokenClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']
```

Could you please explain the meaning of this? I have read other issues about it, but I didn't really grasp the meaning and the solution.

Thank you very much!


Add TF2 NER example

Hi,

Here my small contribution. I have implemented the TF2 version of the NER example already existing in the repo. I tried to have an implementation as close as possible of the Pytorch version.

Let me know for any needed changes :)


Abruptly model training was stopped

I'm fine tuning NER custom dataset using BERT cased model. I'm using script of examples/run_ner.py to fine tune.  **In 2nd epoch training was stopped abruptly without displaying any error.**

Epoch:  67%|██████▋   | 2/3 [2:48:41<1:24:21, 5061.32s/it]
Iteration:  98%|█████████▊| 3734/3816 [1:22:30<01:48,  1.32s/it]
Iteration:  98%|█████████▊| 3735/3816 [1:22:31<01:47,  1.33s/it]

**The training was stopped at 98%.**

Training details are given here:
no. of training sentences: 122095, batch size=32, num_epochs=3, save_steps=750, GPU server: Tesla K40m

Could anybody help me out how to solve this issue and please let me know if you need any further information.

Thanks in advance


Load output file from fine-tuned bert language model

Hi,
I have fine-tuned bert cased language model using **run_lm_finetuning.py**. 'output' is the output file directory and '**bert-base-cased.txt**' is another file created by the model.

1. Does the .txt file mentioned has the output of fine-tuned model??
2. If so, how should I open the file?? I am getting 'UTF-8' encoding issue with the file.

Thank you.


access to the vocabulary

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Is there any way we can get access to the vocabulary in GPT2? Like a list: [subtoken1, subtoken2, ...subtoken 10000...]

Thank you in advance!


how to output specific layer of TFBertForSequenceClassification, or add layer?

how to output the last layer of TFBertForSequenceClassification?

I want to output the layer before classifier (Dense)

```
Model: "tf_bert_for_sequence_classification"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
bert (TFBertMainLayer)       multiple                  102267648
_________________________________________________________________
dropout_37 (Dropout)         multiple                  0
_________________________________________________________________
classifier (Dense)           multiple                  3845
=================================================================
Total params: 102,271,493
Trainable params: 102,271,493
Non-trainable params: 0
```

I tried tf.keras function
```
dense1_layer_model = Model(inputs=model.input, outputs=model.get_layer('bert').output)
```

It didnt worked.


attention_mask added, not multiplied ... is this correct?

https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L233

This line adds (+ operator) the attention mask. I wonder whether this is correct, as I would have very much expected the mask to be multiplied.


Download model too slow, is there any way

in run_lm_finetuning.py
transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin not found in cache or force_download set to True, downloading to ....


Can I use HF XLNet to make a Model that Predicts Backwards?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I would like to create a model that lets me generate multi sentence text sequences but backwards, given the tail end of some input text.

Could I do that using the HF XLNet framework?

I am new to this stuff so if this is possible would you be able to please give me some general pointers on how to go go about doing this?

Grateful for any advice!

Cheers
Fred


Using GPT-2 XL

Hi, Im trying to use the pretrained gpt-xl
but I get the following error:
OSError: Model name 'gpt2-xl' was not found in tokenizers model name list (gpt2, gpt2-medium, gpt2-large, distilgpt2). We assumed 'gpt2-xl' was a path or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.

Im following an example from the documentation.

      7 # Load pre-trained model tokenizer (vocabulary)
---> 8 tokenizer = GPT2Tokenizer.from_pretrained('gpt2-xl')
      9 # Load pre-trained model (weights)
     10 model = GPT2LMHeadModel.from_pretrained('gpt2-xl')
Any Idea why?

Thank you. :)


Using model output by transformers (v2.0) in older versions (0.4.0 or 1.0.0)

## ❓ Questions & Help

I have a bert model finetuned on in-domain data on the latest version of the package (`2.0`). I would now like to use this in some code that is written with the older version of the package (say `0.4.0` or `1.0.0`). Would this be possible?

I tried pointing the code which imports version `0.4.0` to the model output and it gave an error saying `bert_config.json` not found, but there was a `config.json` in the model folder. I renamed `config.json` file and made the code run again and it seems to run.

Am I on the right track? Is this all I have to do to get it run?


BERT bertviz


configuration of the optimizer

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....):
Bert

Language I am using the model on (English, Chinese....):
English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)
Sequence labeling task

Details of the issue:
So, what is the issue:
I have a code that works with ```pytorch-pretrained-bert==0.4.0```
with the following setup for the optimizer:
```
FULL_FINETUNING = True
if FULL_FINETUNING:
    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'gamma', 'beta']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate': 0.01},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate': 0.0}
    ]
else:
    param_optimizer = list(model.classifier.named_parameters())
    optimizer_grouped_parameters = [{"params": [p for n, p in param_optimizer]}]
optimizer = Adam(optimizer_grouped_parameters, lr=3e-5)
```
With this configuration I have an f1-score near to  68% from the beginning.

But with transformers, and migrating to something like (taken from documentation):
```
um_training_steps = 1000
num_warmup_steps = 100
warmup_proportion = float(num_warmup_steps) / float(num_training_steps)  # 0.1
#optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps)
```
I stay in an  f1-score near to 14%.
How can I simulate the former functionality to get back to to the better f1-score?

The changes between the two versions are
```
> num_training_steps = 1000
> num_warmup_steps = 100
> warmup_proportion = float(num_warmup_steps) / float(num_training_steps)  # 0.1
> #optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
> scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps)
>
181c199
< epochs = 50
---
> epochs = 10
195c213
<                      attention_mask=b_input_mask, labels=b_labels)
---
>                      attention_mask=b_input_mask, labels=b_labels)[0]
204a223,224
>         #optimizer.step()
>         torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
205a226,227
>         scheduler.step()
>
220c242
<                                   attention_mask=b_input_mask, labels=b_labels)
---
>                                   attention_mask=b_input_mask, labels=b_labels)[0]
222c244
<                            attention_mask=b_input_mask)
---
>                            attention_mask=b_input_mask)[0]
276d297
```
Also in the documentation https://huggingface.co/transformers/migration.html
suggests the following order
```
    scheduler.step()
    optimizer.step()
```
but that raises a warning from latest version of pytorch which wants the opposite order.

<!-- A clear and concise description of the migration issue. If you have code snippets, please provide it here as well. -->

## Environment

* OS:
* Python version: 3.6
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): latest
* Using GPU ? yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Checklist

- [x] I have read the migration guide in the readme.
- [ ] I checked if a related official extension example runs on my machine.

<!-- Add any other context about the problem here. -->


Split on punc should receive never_split list

When tokenizing when a never_split token that contains any punctuation, such as [ or ] they currently get split when they shouldn't be.


Mask probability in run_lm_finetuning.py

   Hi:

 I don't understand why 0.5 is used when replacing masked input tokens with a random word. I think the probability should be 0.1? Or, the positions replaced with [MASK] shall already be stripped out before using 0.5. I think it is a small bug here? Thanks!

     # 80% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK])
    indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
    inputs[indices_replaced] = tokenizer.convert_tokens_to_ids(tokenizer.mask_token)

    # 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random]

    # The rest of the time (10% of the time) we keep the masked input tokens unchanged
    return inputs, labels

Best Regards,
Qi


How to process ARC dataset with HuggingFace GPT2

Hello,

I am interested in processing ARC dataset with HuggingFace GPT2.
The ARC dataset (http://nlpprogress.com/english/question_answering.html) is a question answering, which contains 7,787 genuine grade-school level, multiple-choice science questions. The dataset also comes with the full corpus of texts extracted from various articles that explains various scientific concepts that can be used to solve these 7,787 multiple choice questions (i.e. this full-corpus is not in the multiple choice format; it's just a series of excerpts from various articles)

I am assuming I'd have to use the GPT2DoubleHeadsModel to process this ARC dataset, since it is a set of multiple-choice questions. However, I also need to somehow train my GPT2DoubleHeadsModel based on the contents of the full corpus of texts that contains excerpts from various scientific articles, since GPT2DoubleHeadsModel wouldn't have acquired any scientific knowledge prior to processing this dataset.

But the thing is, the corpus of series of scientific articles on which I am interested in training my GPT2DoubleHeadsModel with is not written in a multiple-choice format -- is it possible to train only those parts of the GPT2DoubleHeadsModel that are responsible for language modelling with the series of scientific articles, and then fine-tune the entire component of the GPT2DoubleHeadsModel with the training data from the multiple choice questions?

If it is possible, how can I do it?

Thank you,


Need a Restore training mechenisim in run_lm_finetuning.py

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

## Motivation
When training run_lm_finetuning.py for a long time, a restore training feature should be added.

Otherwise, states of sheduler and optimizer are changed when restart.

For example, when it breaks at step checkpoint-30000, it will restart at step 0 with initial learning rate and other configs. This is really troublesome.

Thanks, please.

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


TypeError: convert_examples_to_features() got an unexpected keyword argument 'sequence_a_is_doc'

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert):

Language I am using the model on (English):

The problem arise when using:
```
!python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-large-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file train-v1.1.json \
  --predict_file dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Fine-tuning on SQuAD

## Environment

* OS:
* Python version: 3.6
* PyTorch version: 1.3.1'
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ? Yes


Step restarts from step 0 when reload from an existing checkpoint?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, everyone

I am totally new to Transformers, it's really a good solution. :-)

Question:

When I reload my program from a break point (existing checkpoint, say checkpoint-30000),
what I expect is that, from Tensorboard, I can see the program restarting from 30001.

However, it restarts from step 0, although parameters are up to date.

Does any config point to this problem?

Or any easy solutions?

Thanks.


update


FileNotFoundError when running run_squad.py

## ❓ Questions & Help

I tried fine-tuning BERT on squad on my local computer. The script I ran was

```
python3 ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \
```
But I get an error with regards to the `train-v1.1.json` not being found. The full output is

```
I1122 20:03:40.218862 4637015488 tokenization_utils.py:375] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-vocab.txt from cache at /Users/maxtian/.cache/torch/transformers/b3a6b2c6d7ea2ffa06d0e7577c1e88b94fad470ae0f060a4ffef3fe0bdf86730.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
I1122 20:03:40.596048 4637015488 modeling_utils.py:383] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-pytorch_model.bin from cache at /Users/maxtian/.cache/torch/transformers/66cc7a7501e3499efedc37e47b3a613e0d3d8d0a51c66224c69f0c669b52dcfb.ae11cc7f2a26b857b76b404a908c7abad793f88bf8ad95caecff154da87994b1
I1122 20:03:54.460903 4637015488 modeling_utils.py:453] Weights of BertForQuestionAnswering not initialized from pretrained model: ['qa_outputs.weight', 'qa_outputs.bias']
I1122 20:03:54.461247 4637015488 modeling_utils.py:456] Weights from pretrained model not used in BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']
I1122 20:03:54.473404 4637015488 run_squad.py:504] Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', device=device(type='cpu'), do_eval=True, do_lower_case=True, do_train=True, doc_stride=128, eval_all_checkpoints=False, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=3e-05, local_rank=-1, logging_steps=50, max_answer_length=30, max_grad_norm=1.0, max_query_length=64, max_seq_length=384, max_steps=-1, model_name_or_path='bert-large-uncased-whole-word-masking', model_type='bert', n_best_size=20, n_gpu=0, no_cuda=False, null_score_diff_threshold=0.0, num_train_epochs=2.0, output_dir='../models/wwm_uncased_finetuned_squad/', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=3, per_gpu_train_batch_size=3, predict_file='/dev-v1.1.json', save_steps=50, seed=42, server_ip='', server_port='', tokenizer_name='', train_file='/train-v1.1.json', verbose_logging=False, version_2_with_negative=False, warmup_steps=0, weight_decay=0.0)
I1122 20:03:54.474577 4637015488 run_squad.py:308] Creating features from dataset file at /train-v1.1.json

```

And I get the following error

```
Traceback (most recent call last):
  File "./examples/run_squad.py", line 573, in <module>
    main()
  File "./examples/run_squad.py", line 518, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
  File "./examples/run_squad.py", line 311, in load_and_cache_examples
    version_2_with_negative=args.version_2_with_negative)
  File "/Users/maxtian/Desktop/Python_Projects/transformers/examples/utils_squad.py", line 114, in read_squad_examples
    with open(input_file, "r", encoding='utf-8') as reader:
FileNotFoundError: [Errno 2] No such file or directory: '/train-v1.1.json'
```


CTRLTokenizer not consistent with the fastBPE tokenizer used in Salesforce/CTRL

## 🐛 Bug

cc @keskarnitish

<!-- Important information -->

I am using the transformers CTRL re-implementation to fine-tune the original pre-trained model released by Salesforce https://github.com/salesforce/ctrl.

When the input text consists of newline characters, the tokenization of the Transformers tokenizer differs from the one used by Salesforce CTRL.

## To Reproduce
Salesforce tokenization:
```
import fastBPE
import re
bpe = fastBPE.fastBPE('codes', 'vocab')
line = 'This is one sentence.\nAnd this is another sentence!\n'
tokenized_line = bpe.apply([line])[0]
tokenized_line = re.findall(r'\S+|\n', tokenized_line)
toks = list(filter(lambda x: x != u'@@', tokenized_line))
print(toks)

['This', 'is', 'one', 'sentenc@@', 'e.@@', '\n', 'And', 'this', 'is', 'another', 'sent@@', 'ence@@', '!@@', '\n']
```

Transformers tokenization:
```
from transformers import CTRLTokenizer
tokenizer = CTRLTokenizer.from_pretrained('ctrl', do_lower_case=False)
toks = tokenizer.tokenize(line)
print(toks)

['This', 'is', 'one', 'sentenc@@', 'e.@@', '\n@@', 'And', 'this', 'is', 'another', 'sent@@', 'ence!']
```

Also, I get this issue with double space in the input text:
```
line = 'And also a problem with more than one consecutive  space'
tokenized_line = tokenizer.tokenize(line)

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-37-747954838180> in <module>
      1 line = 'And also a problem with more than one consecutive  space'
----> 2 tokenized_line = tokenizer.tokenize(line)

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_utils.py in tokenize(self, text, **kwargs)
    647
    648         added_tokens = list(self.added_tokens_encoder.keys()) + self.all_special_tokens
--> 649         tokenized_text = split_on_tokens(added_tokens, text)
    650         return tokenized_text
    651

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_utils.py in split_on_tokens(tok_list, text)
    644             return sum((self._tokenize(token, **kwargs) if token not \
    645                     in self.added_tokens_encoder and token not in self.all_special_tokens \
--> 646                     else [token] for token in tokenized_text), [])
    647
    648         added_tokens = list(self.added_tokens_encoder.keys()) + self.all_special_tokens

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_utils.py in <genexpr>(.0)
    644             return sum((self._tokenize(token, **kwargs) if token not \
    645                     in self.added_tokens_encoder and token not in self.all_special_tokens \
--> 646                     else [token] for token in tokenized_text), [])
    647
    648         added_tokens = list(self.added_tokens_encoder.keys()) + self.all_special_tokens

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_ctrl.py in _tokenize(self, text)
    141
    142         for token in text:
--> 143             split_tokens.extend([t for t in self.bpe(token).split(' ')])
    144         return split_tokens
    145

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_ctrl.py in bpe(self, token)
     94             return self.cache[token]
     95         word = tuple(token)
---> 96         word = tuple(list(word[:-1]) + [word[-1]+'</w>'])
     97         pairs = get_pairs(word)
     98

## Expected behavior

I expect the tokenizers to output identical tokenizations so that fine-tuning is consistent with pre-training.

I expect the tokenizer to handle double spaces.

## Environment

* OS: Linux
* Python version: 3.7.4
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:


Fix typo in documentation. toto -> to


Minor bug fixes on run_ner.py

Adding a dictionary entry outside of initialization requires an '=' instead of ':'


run_squad.py not running

## ❓ Questions & Help

When I try to run the script to fine-tune BERT on squad using the code from the examples:

```
python ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \
```
But my terminal just gets stuck on this stage

<img width="518" alt="image" src="https://user-images.githubusercontent.com/8890262/69446607-c39b3780-0d22-11ea-8bea-6135f05640da.png">

When I run the pytests, everything has passed


Truncating GPT2 past

## ❓ Questions & Help

I am using code like this to generate text:
```python
    while True:
        output_token, past = model.forward(output_token, past=past)
        output_token = output_token[:, -1, :]
        output_token = torch.multinomial(F.softmax(output_token, dim=-1), num_samples=1)
        out = torch.cat((out, output_token), dim=1)
```
The problem with this is that `past` keeps growing.
My solution is to check the size of past and truncate it like this:
```python
        if past[0].shape[-2] > max_past:
            past = [p[..., -max_past:, :] for p in past]
```
I don't think this is correct, can anyone enlighten me?


Any plan to include BART and T5?

# 🌟New model addition

## Model description

<!-- Important information -->

## Open Source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them)

## Additional context

<!-- Add any other context about the problem here. -->


How to perform common sense reasoning task with GPT-2?

Hello,

I am new to NLP so I have lots of questions.
I am interested in carrying out common sense reasoning task with GPT-2, for example, with Winograd Schema Challenge dataset.

 Q1. How should I tokenize the Winograd Schema Challenge dataset to process it with GPT-2 (with the double heads model, for instance)? Can someone please give me an example?

 Q2. Can GPT2DoubleHeadsModel be used to conduct common sense reasoning task with Winograd Schema Challenge dataset?

Thank you,


Some Questions about XLNet

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

1, XLNet is a model with relative position embeddings,  so  can either pad the inputs on the right or on the left ?

2, If i pad the inputs on the left, the length of input is 100 and the max length is 128, i don't consider the PAD token in NER task, i.e.  just use the hidden state of input token,  use  [-100:, :] over the output of model  ?
For length dimension, the last 100 are taken; for hidden state  dimension, the values of all dimensions are taken.

3, CLS token must be at end ?


XLNet is getting slower when enabling mems

Per API doc, using mems help to reduce inference time. However, I noticed that more time is needed when increasing mem_len. Do I misunderstand the usage of mem_len parameter?
Here is the testing code

```
import time
import torch
from transformers import XLNetTokenizer, XLNetLMHeadModel

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

text = """
    A horrible, messy split second presents
    itself to the heart-shaped version as Scott is moved. The upcoming movie benefits at
    the mental cost of ages 14 to 12. Nothing substantial is happened for almost 48 days.
    When that happens, we lose our heart. <eod> The quick brown fox jumps over the lazy dog. <mask>
"""

mems = None

input_ids = tokenizer.encode(text)
input_ids = torch.tensor(input_ids).unsqueeze(0)

epoch = 20

for men_len in [0, 16, 32, 64, 128]:
    model = XLNetLMHeadModel.from_pretrained('xlnet-base-cased', mem_len=men_len)

    start_dt = time.monotonic()
    for i in range(epoch):
        outputs = model(input_ids=input_ids, mems=mems)
        if men_len > 0:
            mems = outputs[1]
    end_dt = time.monotonic()

    print('Average Duration for men_len {}: {}'.format(men_len, round(end_dt-start_dt, 2)))
```

Output is
```
Average Duration for men_len 0: 2.49
Average Duration for men_len 16: 2.62
Average Duration for men_len 32: 2.67
Average Duration for men_len 64: 2.81
Average Duration for men_len 128: 3.28
```


Fix GPT2 docstring from #1906

Fixes #1906

Changes the GPT2 Tokenizer's docstrings to correctly explain the reason for `add_prefix_space` parameter


Bart Tokenizer treat symbols in a word as a new word.

## 🐛 Bug

<!-- Important information -->

The model I am using is Bart:

The problem arises when using:
* [ ] `tokenizer.encode`
* [ ] `tokenizer.decode`
The tasks I am working on is:
* [ ] Encode a string, then decode it back.

## To Reproduce

```
import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

test_string = 'text with percentage%'

# encode Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
input_ids = tokenizer.encode(test_string)
output = tokenizer.decode(input_ids)
print(output)
```

`>>> text with percentage %`

## Expected behavior

It should be `text with percentage%`, which treats the symbol in the word as one word.

## Environment

* OS: MacOS
* Python version: 3.7
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): Master


Passing inputs to TFGPT2LMHeadModel results in error: 'TensorSliceDataset' object has no attribute 'shape'

## 🐛 Bug

Model I am using (Bert, XLNet....): `TFGPT2LMHeadModel`

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X ] my own modified scripts: (give details)

```
import tensorflow as tf
from transformers import *

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = TFGPT2LMHeadModel.from_pretrained('gpt2')

raw_text = "Here comes the sun"
tokens = tokenizer.encode(raw_text, add_special_tokens=False)
inputs = tf.data.Dataset.from_tensor_slices( np.array(tokens) )
inputs = {'input_ids': inputs}
outputs = model(inputs)
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X ] my own task or dataset: (give details)
Trying to work out a stripped-down version of `run_generation.py` using `TFGPT2LMHeadModel` only.

## To Reproduce

Steps to reproduce the behavior: just run the code above, you should get the following error:

```
Traceback (most recent call last):
  File "./generate_text.py", line 47, in <module>
    out = sample_sequence(tokens, num_samples=num_samples)
  File "./generate_text.py", line 27, in sample_sequence
    outputs = model(inputs)
  File "/Users/Riccardo/development/ideal/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/Users/Riccardo/development/ideal/lib/python3.7/site-packages/transformers/modeling_tf_gpt2.py", line 490, in call
    transformer_outputs = self.transformer(inputs, **kwargs)
  File "/Users/Riccardo/development/ideal/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/Users/Riccardo/development/ideal/lib/python3.7/site-packages/transformers/modeling_tf_gpt2.py", line 257, in call
    position_ids = tf.range(past_length, shape_list(input_ids)[-1] + past_length, dtype=tf.int32)[tf.newaxis, :]
  File "/Users/Riccardo/development/ideal/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 475, in shape_list
    static = x.shape.as_list()
AttributeError: 'TensorSliceDataset' object has no attribute 'shape'
```

## Expected behavior

Still not sure!

## Environment

* OS: MacOsX 11.14.6 (Mojave)
* Python version: 3.7.5
* Tensorflow version: 2.0.0
* Tensorflow Transformers version (or branch): 2.1.1
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Training transformer XL from scratch with my own dataset

## ❓ Questions & Help
Is it possible to train a transformer XL model from scratch with my own dataset?
Just initialize the model with default params and compile & fit the model?
<!-- A clear and concise description of the question. -->


lm_fine-tuning on small dataset of 3 documents

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,

I am trying to use [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) on a sample dataset [here](https://github.com/dmlc/gluon-nlp/blob/master/scripts/bert/sample_text.txt).

I am running the script with following arguments but I get the exact identical pytorch_model.bin [440.5 MB] saved in the output_dir=op:
python run_lm_finetuning.py --train_data_file=sample_text.txt --output_dir=op --mlm --do_train --overwrite_output_dir --do_lower_case --save_steps=50

I was wondering if this dataset of 3 documents is too small to fine-tune on or if I can modify some arguments to get a domain-fine-tuned model.

Thanks!


Documentation error in GPT2Tokenizer

Documentation page:
https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2Tokenizer
Code place:
https://github.com/huggingface/transformers/blob/d7d36181fdefdabadc53adf51bed4a2680f5880a/transformers/tokenization_gpt2.py#L112-L113

This phrase:
> Otherwise, this tokenizer encode and decode method will not conserve the absence of a space at the beginning of a string: tokenizer.decode(tokenizer.encode(“Hello”)) = ” Hello”

is **NOT** correct.

Actually:
> tokenizer.decode(tokenizer.encode(“Hello”)) = ”Hello”

Try this example:
```
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
print("'" + tokenizer.decode(tokenizer.encode("Hello")) + "'")
```
Output:
```
'Hello'
```


run_summarization_finetuning.py

## ❓ Questions & Help
Hi
Does your code run_summarization_finetuning.py implement the abstractive summarization approach described in "Text summarization with pretrained encoders." by Liu, Yang, and Mirella Lapata?
Thanks
<!-- A clear and concise description of the question. -->


Typo in Documentation for GPT2LM Output "past"

I think the described output shape for "past" for GPT-2 is wrong.
In:
[https://github.com/huggingface/transformers/blob/0cdfcca24b5739e25b584c05b866baa19ea382ef/transformers/modeling_gpt2.py#L332](url)
it says the output shape of each key, value tensor in each self attention layer is
``(batch_size, num_heads, sequence_length, sequence_length)``
, but it should be
``(batch_size, num_heads, sequence_length, hidden_size / num_heads)``
 or
``(batch_size, num_heads, sequence_length, embed_size_per_head)``
.Also since the past tensor per layer always shows both key and value tensors it might even be clearer to write:
``(2, batch_size, num_heads, sequence_length, embed_size_per_head)``


Valohai integration

Huggingface / Transformers Valohai integration

Changes to existing Transformers code:
- Prints Valohai-styled logs (JSON)

Additional info:
- valohai.yaml has most (but not all) parameters used by run_glue.py
- Valohai execution downloads all glue datas by default (still pretty fast). Download script placed in `utils/download_glue_data.py`.
- Valohai execution only saves model/checkpoint at the end by default (adjust with Valohai UI)
- Valohai execution logs every 25 steps (adjust with Valohai UI)


Add CamemBERT models to modeling_auto

CamemBERT was in autoconfig but not in automodel, this PR aims to correct that. Also first PR here so please tell me if I missed some things :-)


Methods get_input_embeddings() and set_input_embeddings() appear in documentation but not available.

## ❓ Questions & Help
Hi, I'm trying to access the word embeddings layer of Bert multilingual, as I want to take out all tokens not belonging to spanish and add some tokens which are part of this language, with the objective of adapting BERT multilingual to spanish. The thing is, in your documentation you claim that there are 2 functions for this purpose: get_input_embeddings() and set_input_embeddings() (https://huggingface.co/transformers/model_doc/bert.html); in fact there is a link to the source code and they appear to be there. However, once I try to do this in version 2.1.1. (the one documentation refers to), none of this methods are part of BertModel class, which is astonishing to me. Please tell me what's wrong!! Did you remove these methods from current versions but have a unique documentation for all versions? Which versions can I find these methods in? Is the source code showed in the documentation different from the actual source code of the library?
<!-- A clear and concise description of the question. -->


Can GPT2DoubleHeadsModel be used for regular next token prediction task without adjusting its head?

Hello,

According to the HuggingFace Transformer's website (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel), **GPT2DoubleHeadsModel** is the GPT2 Model transformer with a language modelling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks.

Does this mean that we can use the **GPT2DoubleHeadsModel** for the regular language modelling task (i.e. next word prediction) without modifying its head? or would I need to adjust the head of **GPT2DoubleHeadsModel** if I want to do the next word prediction, since GPT2DoubleHeadsModel is for answering multiple-choice type questions only?

Thank you,


Classify entities - run_ner script

Hello all,
I am just wondering what extra input to the "BertForTokenClassification"  if I want to classify the entities TO (PER, LOC ..) . Note that the entities are given in advance. I used run_ner script but it extracts the entities and classify them (the extraction is not needed). I did not get how the script(or the input) can be modified for my task? any idea?


Is the usage of scheduler described in README correct?

https://github.com/huggingface/transformers/blame/master/README.md#L546

I think it's not here to do `scheduler.step` but in epoch loop.