:zip: #2106 tokenizer.tokenize speed improvement (3-8x) by caching added_tokens in a Set

in #2106, we see that adding tokens to tokenizer decreases progressively tokenization performance which is not really a surprise as you need to go through the list of tokens which grows. But it sounds that this increase is not linear.

By having a quick look at code, I've seen that:
- `added_tokens` list is built for every calls of tokenize,
- `all_special_tokens` is a python property that is reevaluated every time
- `split_on_tokens` is going 2x in both `all_special_tokens` and `added_tokens_encoder` lists which is `O(n)`.

I've tried to replace those by a simple cached Set of `added_tokens_encoder.keys() + all_special_tokens` that is reevaluated at each call of `add_tokens`. firstly, it avoids rebuilding the list at every call. Secondly, searching in a Set is `O(1)` in average and `O(n)` in worst case.

On RobertaTokenizer, the result is a significant speed improvement (testing on for 100.000 calls):
- for 0 added token, `tokenizer.tokenize` is >3x faster
- for 200 tokens, `tokenizer.tokenize` is >7x faster

Here are a few interesting plots.

### Execution time when adding more tokens between old code and new

We see here that old code is not linear and the execution time is impacted when more tokens are added.
New code seems to behave linearly (up to 200 at least)

### Rate of speed increase between old code and new

We see that new code is 3x faster by default and this advantage grows when adding more tokens (>7x for 200)

### Execution time between old code and new in a bar plot

Same as previous plot.

I know you're working on Rust tokenizers that will be much faster in theory. But until it's ready, what do you think about this basic correction (and maybe others) that already improves the speed drastically?

Don't hesitate to tell if you see that this modification would be very bad for other cases.


why I use run_squad.py for SQuAD2.0 have bad f1 score 43.638
The noanser_f1=0.0 it look like do not deal with not answerable situation
I don't have change anything
I just run like this

python3 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/train-v2.0.json \
  --predict_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/dev-v2.0.json \
  --per_gpu_train_batch_size 4 \
  --learning_rate 4e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /share/nas165/Wendy/transformers/examples/squad_debug_SQuAD_1213_bert/

Do anyone know how to do? Thanks a lot


merge new version


RobertaTokenizer token type issue

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Why the two middle `<\s>` are both assigned with token type 0?

https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/tokenization_roberta.py#L149

Could this one be better?
```python
return len(cls + token_ids_0 + sep) * [0] + len(sep + token_ids_1 + sep) * [1]
```


run_squad with roberta

Hi, @julien-c  @thomwolf  this PR is based on #1386 and #1984.

- This PR modified run_squad.py and models_roberta to support Roberta.

- This PR also made use of multiple processing to accelerate converting examples to features. (Converting examples to feature needed **15minus before and 34 seconds now** with 24 cpu cores' acceleration. The threads number is 1 by default which is the same as the original single processing's speed).

- The result of Roberta large on squad1.1:
`{'exact': 87.26584673604542, 'f1': 93.77663586186483, 'total': 10570, 'HasAns_exact': 87.26584673604542, 'HasAns_f1': 93.77663586186483, 'HasAns_total': 10570, 'best_exact': 87.26584673604542, 'best_exact_thresh': 0.0, 'best_f1': 93.77663586186483, 'best_f1_thresh': 0.0}`, which is sighltly lower than #1386 in a single run.
Parameters are `python ./examples/run_squad.py
    --model_type roberta
    --model_name_or_path roberta-large
    --do_train
    --do_eval
    --do_lower_case \
    --train_file data/squad1/train-v1.1.json
    --predict_file data/squad1/dev-v1.1.json
    --learning_rate 1.5e-5
    --num_train_epochs 2
    --max_seq_length 384
    --doc_stride 128
    --output_dir ./models_roberta/large_squad1
    --per_gpu_eval_batch_size=3
    --per_gpu_train_batch_size=3
    --save_steps 10000 --warmup_steps=500 --weight_decay=0.01`. Hope this gap can be improved by `add_prefix_space=true' . I will do this comparasion in the next days.

- The result of Roberta base is '{'exact': 80.65279091769158, 'f1': 88.57296806525736, 'total': 10570, 'HasAns_exact': 80.65279091769158, 'HasAns_f1': 88.57296806525736, 'HasAns_total': 10570, 'best_exact': 80.65279091769158, 'best_exact_thresh': 0.0, 'best_f1': 88.57296806525736, 'best_f1_thresh': 0.0}'. Roberta-base was also tested since it's more easy to be reproduced.

- The results of bert-base-uncased is `{'exact': 79.21475875118259, 'f1': 87.13734938098504, 'total': 10570, 'HasAns_exact': 79.21475875118259, 'HasAns_f1': 87.13734938098504, 'HasAns_total': 10570, 'best_exact': 79.21475875118259, 'best_exact_thresh': 0.0, 'best_f1': 87.13734938098504, 'best_f1_thresh': 0.0}'. This is tested for the multiple processing's influence on other models. This result is the same with bert-base-uncased result without multiple processing.

- Hope that someone else can help to reproduce my results. thank you! I will continue to find if three is some ways to improve the roberta-large's results.
- Squad1 model on google drive [roberta-large-finetuned-squad](https://drive.google.com/drive/folders/1BZJeOeri_cKGUG_cRI5OCmkWC5deQqcc?usp=sharing):


[cli] Upload is now compatible with folders


Small run_squad nit: eliminate trailing "_" in "best_predictions_.json" when no prefix

## 🐛 Bug

<!-- Important information -->
Prior convention for tf-based run_squad is to output best predictions in an nbest_predictions.json file. Now with the new convention of including a "prefix" in the generation of potentially many nbest files, in cases where there's not prefix, the name becomes nbest_predictions_.json (nothing after the "_"). Might be more backward compatible to include the rightmost "_" only when there's a prefix after it.

The same thing applies to predictions and null_odds, ...

Model I am using (Bert, XLNet....): albert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: SQuAD2
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. look at output of a successful run of run_squad.py
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch): v2.1.1, from source
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


BertForSequenceClassification() model TF to pytorch conversion

I added a script convert_bert_seqclass_tf_checkpoint_to_pytorch.py for the conversion a trained BertForSequenceClassification model from TF to pytorch.
I had to modify modeling_bert.py to support it, as well.


How to structure input data for training TFGPT2LMHeadModel using model.fit() in TF2.0?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am able to use the run_lm_finetuning.py script easily, but I wish to be able to use TF2.0 and call model.fit() on distilgpt2.

using the fine-tuning script as an example,  I structured my dataset as such:
```
#split text file by lines, tokenize lines, convert tokenized lines to integers
examples = []
with open(file_path, encoding="utf-8") as f:
    text = f.readlines()
for line in text:
    examples.append(tokenizer.encode(line))

#pad examples to appropriate length
pad_examples = tf.keras.preprocessing.sequence.pad_sequences(examples,
                                                             maxlen=256,padding='post',
                                                             truncating='post',
                                                             value=tokenizer.pad_token_id)

#create dataset, in the finetuning script, labels were just copies of the "examples" input array
dataset = tf.data.Dataset.from_tensor_slices((pad_examples, pad_examples))

BATCH_SIZE = 4
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
```

However, when I compile and run the script like this:
```
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=[loss, None, None, None, None, None, None], metrics=[metric])
model.fit(dataset, epochs=3)
```

my loss is NaN, and no training is happening.

I believe I am missing something fundamental about how GPT2 works with inputs/labels, but it seems from the torch script the labels and inputs are the same array and calling model.train() works just fine.

Any ideas would be greatly appreciated, as I have an existing TF2.0 architecture I am trying to connect to GPT-2 and being able to call model.fit() would be preferable. I have no issues fine-tuning BERT in TF2.0 with model.fit() as it is much clearer what the inputs and labels are in that case.


CUDA error at 'cublasSgemm' when using the pretrained BERT

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [v] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [v] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

My code:
```
torch.cuda.set_device(0)

        sequence_output, pooled_output = self.bert(input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
        sequence_output = self.dropout(sequence_output)
        pooled_output = self.dropout(pooled_output)

        sense_logits = self.sense_classifier(pooled_output)
        arg_logits = self.arg_classifier(sequence_output)
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 16.04
* Python version: 3.6.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? yes (1080)
* Distributed of parallel setup ? (no)
* Any other relevant information:

## Additional context

My error message is here:
When my model is on the training process, the error message occurs within some hours.

I'm sure my problem is exactly same with this link: https://github.com/huggingface/transformers/issues/1760

however, update pytorch and transformers doesn't help.

```
  File "training_finetuning.py", line 144, in train
    token_type_ids=b_token_type_ids, attention_mask=b_input_masks)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "../kaiser/src/modeling.py", line 53, in forward
    sequence_output, pooled_output = self.bert(input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 738, in forward
    encoder_attention_mask=encoder_extended_attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 384, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 355, in forward
    self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 309, in forward
    self_outputs = self.self(hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 230, in forward
    mixed_value_layer = self.value(hidden_states)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
```

<!-- Add any other context about the problem here. -->


using run_squad.py for predict and specifying config_name as path, config.json not found

## 🐛 Bug

<!-- Important information -->

The situation is that, when running a predict-only task and specifying 1) an explicit path for a fine-tuned albert model and 2) specifying a specific path to the corresponding config.json file, run_squad attempts to seek the config file in the location of the --output_dir.

It appears that, when specifying config_name as a full path, the specification is ignored and run_squad looks in the specified output_dir location for config.json. Since I am automatically generating several output dirs during the course of running my pipeline, it is not convenient nor sensible for me to also copy the config file to each result directory.

To be concrete, I am trying to specify that the config file should come from the training output model directory:

--config_name /home/.../albert_models/squad2/config.json
and that the eval/predict output should go to a result directory:

--output_dir /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0
When I do this I see the below.

Model I am using (Bert, XLNet....): albert, fine-tuned for SQuAD2 using albert xxl

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [2] an official GLUE/SQUaD task: SQuAD2
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. in transformers, run_squad.py, with arguments below

Also tried without --cache_dir, which also had the same effect.

The only thing that worked was to  use the model dir == the output dir, but that placed my outputs into the model dir, which is not acceptable (should be unaltered by predict-only tasks).

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
run_squad.py arguments:
```
 --model_type albert
--model_name_or_path  /home/.../albert_models/squad2
--cache_dir /home/.../transformers/cache_dir/v2.0-albert
--config_name  /home/.../albert_models/squad2/config.json
--do_eval
--do_lower_case
--predict_file /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0/questionnaire.json
--train_file None
--per_gpu_train_batch_size 2
--per_gpu_eval_batch_size 24
--learning_rate 3e-05
--num_train_epochs 2.0
'--max_seq_length 128
--doc_stride 128
--version_2_with_negative
--output_dir /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0/
```
Partial stack trace:
 ```
12/12/2019 19:18:24 - INFO - __main__ -   Evaluate the following checkpoints: ['/home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0'']                                                                                Traceback (most recent call last):                                                                                File "/home/.../transformers/transformers/configuration_utils.py", line 134, in from_pretrained                                 resolved_config_file = cached_path(config_file, cache_dir=cache_dir, force_download=force_download, proxies=proxies)
File "/home/.../transformers/transformers/file_utils.py", line 182, in cached_path raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0/config.json' not found                                                                                During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/.../transformers/transformers/configuration_utils.py", line 134, in from_pretrained
    resolved_config_file = cached_path(config_file, cache_dir=cache_dir, force_download=force_download, proxies=proxies)
  File "/home/.../transformers/transformers/file_utils.py", line 182, in cached_path
    raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0'/config.json not found
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
Expected that overriding the model_name_or_path with a direct path, and the config_name with another path (different from output_dir) would override the defaults.

## Environment

* OS: ubuntu 18.04.1
* Python version: 3.7.5
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 2.2.1, installed from source
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


How to do the further pretraining ?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello ! How could I futher Pre-train the BERT ( including the unsupervised masked language model
and next sentence prediction tasks ) **using my own corpus** ?   thank you very much !


Model parallelism + Adapters

Adding model parallelism for large T5 models and other models if needed.
Adding adapters (a generalization of #1289) at the same time.


[SMALL BREAKING CHANGE] Cleaning up configuration classes - Adding Model Cards

Clean up configuration.

Previously loading a JSON file in the configuration could be done either by `config = config_class(json_file)` or by `config = config_class.from_pretrained(json_file)`.

This was a historical artifact from the time configuration classes didn't use `from_pretrained()` method. This introduced complexity in logic to instantiate the classes which impacted PRs like #1548 and complexified the code to add new models.

In this  PR we remove the first path to favor using the standardized `config = config_class.from_pretrained(json_file)`.

cc @LysandreJik @mfuntowicz @julien-c


PreTrainedEncoderDecoder on tensorflow

## 🚀 Feature

Hi, would it be possible to create a tensorflow version of `PreTrainedEncoderDecoder`?
## Motivation

The main motivation is that I would like to use `PreTrainedEncoderDecoder` in TensorFlow. Yeah, I got it, PyTorch is better and I totally agree but unfortunately, I have to use TensorFlow.

## Additional context

Looking at the code it does not seem too hard to create a `TFPreTrainedEncoderDecoder`

Thank you guys


pad_to_max_length param is not supported in PreTrainedTokenizer.encode

## ❓ Questions & Help

Hello,

I've installed the current version of transformers package (2.2.1) through pip on Python 3.6.8rc1 on Windows 10 Pro (build 17763.678 if it is important). I am trying to get a sentence encoded and padded at the same time:

```python
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
temp = tokenizer.encode(text, add_special_tokens=True, max_length=MAX_LENGTH,
                        pad_to_max_length=True)
```
And I'm getting an error, that `pad_to_max_length` is unrecognized option. What am I missing?


Adding model type to config.json

## Feature
Add `model_type` to the *config.json* to define the model_type and make it independent from the name

## Motivation
Currently, the model type is automatically discovered from the name. So if it is a Bert model, the autoloader is choosing the right methods if the name contains `bert`. If not, an error would be thrown.
This is somehow cumbersome and error prone and it restricts the naming of the models. Why not just add this info as a attribute in the config.json?
Other suggestions are welcome!

## Info
I would happily start working at a PR if others agree as well.


[WIP] Add UniLM model

# Typical workflow for including a model

Here an overview of the general workflow:

- [x] add model/configuration/tokenization classes
- [x] add conversion scripts
- [x] add tests
- [x] finalize

Let's detail what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-paste or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-paste or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [x] copy-paste or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this script to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] add your models and tokenizer to `pipeline.py`
- [x] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation itself at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.


Low ROUGE scores for BertSum

Great work, very easy to pick up and play with. I downloaded the CNN/DM stories from the link provided and selected only the files that belong to the test set following See et al.'s dataset splits (https://github.com/abisee/cnn-dailymail/blob/master/url_lists/all_test.txt). Then I ran the model using the first command provided in the readme.

My question is, what is the expected ROUGE F1 scores on the test set? I expected something near what was presented in the paper for BertSumExtAbs, which is:
R-1: 0.4213
R-2: 0.1960
R-L: 0.3918
But the ROUGE scores I got were much lower:

****** ROUGE SCORES ******

** ROUGE 1
F1        >> 0.303
Precision >> 0.328
Recall    >> 0.288

** ROUGE 2
F1        >> 0.185
Precision >> 0.210
Recall    >> 0.172

** ROUGE L
F1        >> 0.335
Precision >> 0.356
Recall    >> 0.320


gpt-2 implement issue

Thanks for your good implementation some model in pytorch!
gpt-2 paper mentioned that they did few  modifications with original gpt, included "A modified initialization which accounts
for the accumulation on the residual path with model depth
is used. We scale the weights of residual layers at initialization by a factor of 1/
√
N where N is the number of
residual layers."
I assume that will help in training, it is crucial to reimplement the gpt2, so consider add this to repo?


How to find the  corresponding download models from Amazon?

## ❓ Questions & Help

As we know, the TRANSFORMER could easy auto-download models by the pretrain( ) function.
And the pre-trained BERT/RoBerta model are stored at the path of
./cach/.pytorch/.transformer/....

But,  all the name of the download models are like this:

d9fc1956a01fe24af529f239031a439661e7634e6e931eaad2393db3ae1eff03.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda.json

It's not readable and hard to distinguish which model is I wanted.

In another word,  if I want to find the pretrained  model of  'uncased_L-12_H-768_A-12', I can't finde which one is ?

Thanks for your answering.


End-Task Distillation with DistilBERT

## ❓ Questions & Help

The DistilBERT paper notes the IMDB and SQuAD results were obtained "with a second step of distillation during fine-tuning". What does this involve exactly and how can it be performed with the DistilBERT model in this repo?


Special Tokens are Split by BPE

## 🐛 Bug

When I load 'distilbert-base-uncased' DistilBertTokenizer (with do_basic_tokenize=False) and call tokenize() on a string that includes special tokens, the special tokens are broken up by BPE.

Model I am using (Bert, XLNet....): DistilBertForSequenceClassification

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Load pretrained DistilBertTokenizer
2. call tokenize on a string including special tokens

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased', do_lower_case=True, do_basic_tokenize=False)
print(tokenizer.special_tokens_map)
```
{'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}

```
text = '%s Hrabri (Brave) was the lead boat of the Hrabri-class submarines; built by the Vickers-Armstrong Naval Yard in the United Kingdom, for the Kingdom of Serbs, Croats and Slovenes (later Yugoslavia) %s' % (tokenizer.cls_token, tokenizer.sep_token)
print(text)
```
[CLS] Hrabri (Brave) was the lead boat of the Hrabri-class submarines; built by the Vickers-Armstrong Naval Yard in the United Kingdom, for the Kingdom of Serbs, Croats and Slovenes (later Yugoslavia) [SEP]

```
tokens = tokenizer.tokenize(text)
print(' '.join(tokens))
```
[ cl ##s ] hr ##ab ##ri ( brave ) was the lead boat of the hr ##ab ##ri - class submarines ; built by the vickers - armstrong naval yard in the united kingdom , for the kingdom of serbs , croats and slovene ##s ( later yugoslavia ) [ sep ]

## Expected behavior
```
tokens = tokenizer.tokenize(text)
print(' '.join(tokens))
```
[CLS] hr ##ab ##ri ( brave ) was the lead boat of the hr ##ab ##ri - class submarines ; built by the vickers - armstrong naval yard in the united kingdom , for the kingdom of serbs , croats and slovene ##s ( later yugoslavia ) [SEP]

## Environment

* OS: Windows
* Python version: 3.6.3
* PyTorch version: 0.4.1
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? no
* Distributed of parallel setup ? no
* Any other relevant information:

## Additional context


AlBERT UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

## ❓ Questions & Help

![image](https://user-images.githubusercontent.com/36267779/70731538-8584a880-1d0f-11ea-8516-7895c3a50c27.png)
![image](https://user-images.githubusercontent.com/36267779/70731555-8d444d00-1d0f-11ea-84a5-0df7b00ff820.png)

Hi! There is some problem while downloading any of the pre-trained AlBERT models, however, there weren't any problems a few days ago. Could you please tell me where can I download the AlBERT TensorFlow checkpoints (`albert_model.ckpt`) for running `convert_albert_original_tf_checkpoint_to_pytorch.py` script? Unfortunately, I wasn't able to find any for resolving this as in [2110](https://github.com/huggingface/transformers/issues/2110).
I'd really appreciate any help in resolving the issue.
Thanks a bunch in advance!

tensorflow | 2.0.0 |
torch | 1.3.1 |
transformers | 2.2.1 |
python | 3.7 |


BertAbs decoder_input_ids

## ❓ Questions & Help

What should the `decoder_input_ids` look like if we are fine-tuning the model on our own dataset?

I tried `[unused0] [unused2] summary_sent_toks [unused2] summary_sent_toks2 [unused1]` (looking at the paper) ... but I get shape errors because of line 150 in `modeling_bertabs.py`:

```
decoder_input_ids[:, :-1], encoder_hidden_states, dec_state
```

The `decoder_input_ids` shape I'm passing in in `(8, 512)` ... but the code above chops off the last column.


RoBERTa/GPT-2 tokenization: Why we call all_special_tokens for each token in split_all_tokens?

Is there a reason why the property function `all_special_tokens` is called in each iteration in `split_on_tokens()` when looping over all tokens?

When I initialize a new variable and call all_special_tokens only once in the tokenizer init, the tokenization is speed-up around 2~3 times for me. Maybe I am missing something :)


RoBERTa tokenization: Why do we call 'all_special_tokens' in each tokenize loop?

Is there a reason why the property function`all_special_tokens` is called in each iteration in `tokenize()` when looping over all tokens?

When I initialize a new variable and call `all_special_tokens` only once in the tokenizer init, the tokenization is speed-up around 2~3 times for me.


Is there a reason why the property function 'all_special_tokens' is called in each iteration in tokenize() when looping over all tokens?


:bug: #2120 in model.from_pretrained, PosixPath crashes at "albert" check

- `pretrained_model_name_or_path` is now stringified to allow the "albert" and "v2" checks with PosixPath (or any other path representation that isn't iterable).
- If `pretrained_model_name_or_path` is None, it gives string "None" which doesn't contain "albert" so it's OK.
- 2 x `str(pretrained_model_name_or_path)` doesn't impact performances as it's called only once per program and `and` operator will generally fail at left side.
- added a test to check non-regression


Fix encode plus

Fixing the tensor creation in encode_plus


Recommended way for creating distillBERT container and serving

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
As per documentation, I am supposed to load distilbert as below.

question_answering_model = torch.hub.load('huggingface/pytorch-transformers', 'modelForQuestionAnswering', 'distilbert-base-uncased-distilled-squad')
question_answering_tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer', 'distilbert-base-uncased-distilled-squad')

I'm using google Cloud Run. It bring up the container (and hence the model) only upon request. This causes download and load delay.

How to pre-download the model and serve? I am looking for dockerfile step where it could install the weights file and other files needed for the model. This way, I am hoping that my dynamic delays are reduced and I get inference must faster.

Please let me know if such a thing is possible.

thanks
Ishwar


doc: fix pretrained models table

Hi,

this PR fixes the pretrained models table, see #2145.


the docs pretrained models is missing

Hi,
your docs have the table with pretrained models missing, probably some formatting error, as the source code has the table
<img width="1280" alt="Screenshot 2019-12-11 at 17 31 01" src="https://user-images.githubusercontent.com/340180/70640463-4174a380-1c3c-11ea-9c6e-ca343ef46332.png">
https://huggingface.co/transformers/pretrained_models.html
Cheers,
Piotr


Allowing from_pretrained to load from url directly

Allowing `from_pretrained` to load from url directly.


Fix typo in examples/run_glue.py args declaration.

deay -> decay


master branch examples/run_squad.py: missing --predict_file argparse argument

## 🐛 Bug

<!-- Important information -->

Model I am using: albert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: (give details)
examples/run_squad/py: --predict_file not recognized
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: run_squad
* [ ] my own task or dataset: (give details)

## To Reproduce

Try running an eval only run with run_squad.py
Steps to reproduce the behavior:

1. provide all of the requisite arguments to run_squad.py
2. observe error `run_squad.py: error: unrecognized arguments: --predict_file`
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
A correct run
## Environment

* OS:
* Python version: 3.7.5
* PyTorch version: n/a
* PyTorch Transformers version (or branch): master
* Using GPU ? yes
* Distributed of parallel setup ?no
* Any other relevant information:

Quick inspection of examples/run_squad.py reveals missing declaration, present in, e.g., 1.2.0

## Additional context

<!-- Add any other context about the problem here. -->


Fine-tuning distilled GPT-2

## ❓ Questions & Help

To my understanding, examples/run_lm_finetuning.py can be used to fine-tune the model to new data. How do I fine-tune a distilled GPT-2? To be precise, I assume that I can use the entire code, but I just need to import the right module. I tried importing DistilGPT2Config, DistilGPT2LMHeadModel, DistilGPT2Tokenizer, but it doesn't work out.


return_tokens_mapped_to_origin not working

## 🐛 Bug

Model I am using: **Bert**
Language I am using the model on: **English**

## To Reproduce
Call `bertTokenizer.tokenize("text", return_tokens_mapped_to_origin=True)`
Result:

> TypeError: _tokenize() got an unexpected keyword argument 'return_tokens_mapped_to_origin'

## Expected behavior
The official documentation mentions a "return_tokens_mapped_to_origin" optional parameter that when set to True should return the index of each token in the initial given text.
https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=return_tokens_mapped_to_origin#transformers.PreTrainedTokenizer.tokenize

## Environment

* OS: macOS Mojave
* Python version: 3.7
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? No

## Additional context
In the source code this parameter is never used outside of the doc comment, neither in the base class nor in its implementations.


About Summarization

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Thank you very much for your wonderful work. I found that some new code for summarization has been added from "pretrained encoder" paper. However, I see only the evaluation part of the code. I want to ask if you will add the code for the training part. Thank you very much!


encode_plus not returning attention_mask and not padding

## 🐛 Bug

Tested on RoBERTa and BERT of the master branch, the [`encode_plus`](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.encode_plus) method of the tokenizer does not return an attention mask. The documentation states that by default an attention_mask is returned, but I only get back the input_ids and the token_type_ids. Even when explicitly specifying `return_attention_mask=True`, I don't get that back.

If these specific tokenizers (RoBERTa/BERT) don't support this functionality (which would seem odd), it might be useful to also put that in the documentation.

As a small note, there's also a typo in the documentation:

> return_attention_mask – (optional) Set to False to **avoir** returning attention mask (default True)

Finally, it seems that `pad_to_max_length` isn't padding my input (see the example below). I also tried `True` instead of an integer, hoping that it would automatically pad up to max seq length in the batch, but to no avail.

```python

from transformers import BertTokenizer

if __name__ == '__main__':
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

    orig_text = ['I like bananas.', 'Yesterday the mailman came by!', 'Do you enjoy cookies?']
    edit_text = ['Do you?', 'He delivered a mystery package.', 'My grandma just baked some!']

    # orig_sents and edit_text are lists of sentences
    for orig_sents, edit_sents in zip(orig_text, edit_text):
        orig_tokens = tokenizer.tokenize(orig_sents)
        edit_tokens = tokenizer.tokenize(edit_sents)

        seqs = tokenizer.encode_plus(orig_tokens,
                                     edit_tokens,
                                     return_attention_mask=True,
                                     return_tensors='pt',
                                     pad_to_max_length=120)
        print(seqs)
```

Output:

```
{'input_ids': tensor([[  101,  1045,  2066, 26191,  1012,   102,  2079,  2017,  1029,   102]]),
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 1, 1, 1, 1]])}
{'input_ids': tensor([[ 101, 7483, 1996, 5653, 2386, 2234, 2011,  999,  102, 2002, 5359, 1037, 6547, 7427, 1012,  102]]),
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]])}
{'input_ids': tensor([[  101,  2079,  2017,  5959, 16324,  1029,   102,  2026, 13055,  2074, 17776,  2070,   999,   102]]),
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]])}
```


Tokenization in C++

Is there any general strategy for tokenizing text in C++ in a way that's compatible with the existing pretrained `BertTokenizer` implementation?
I'm looking to use a finetuned BERT model in C++ for inference, and currently the only way seems to be to reproduce the `BertTokenizer` code manually (or modify it to be compatible with torchscript). Has anyone come up with a better solution than this?


is the tokenization broken for bert?

## 🐛 Bug

<!-- Important information -->

Model I am using is `bert-base-uncased`:

Language I am using the model on (English):

## To Reproduce

Steps to reproduce the behavior:

1. Just Ran the example from the docs

```
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM

# OPTIONAL: if you want to have more information on what's happening under the hood, activate the logger as follows
import logging
logging.basicConfig(level=logging.INFO)

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', force_download=True)

# Tokenize input
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
print(tokenized_text)
assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']

>INFO:transformers.tokenization_utils:loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /Users/1570137/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
```

The assertion fails and the `print(tokenized_text)` returns this actually,

`['[', 'cl', '##s', ']', 'who', 'was', 'jim', 'henson', '[MASK]', '[', 'sep', ']', 'jim', 'henson', 'was', 'a', 'puppet', '##eer', '[', 'sep', ']']`

## Extra Details

>! pip show transformers
```
Name: transformers
Version: 2.2.1
Summary: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch
Home-page: https://github.com/huggingface/transformers
Author: Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors
Author-email: thomas@huggingface.co
License: Apache
Location: /Users/1570137/anaconda3/envs/my_env/lib/python3.7/site-packages
Requires: sacremoses, numpy, tqdm, sentencepiece, regex, requests, boto3
Required-by: flair
```

I am on MacOS, No GPU.
Also is that behaviour expected?
Thanks.


Is there support for TensorflowJs?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I managed to save my tenforflow 2.0 model and I see keras .h5 and config.json files.

When I run the tensorflowjs converter it seems to run with no issues.

!tensorflowjs_converter --input_format=keras save/tf_model.h5 save/tfjs_model

I see the output as expected in the generated files. But, when I try to load them from Javascript, I get these errors:

models.ts:287 Uncaught (in promise) TypeError: Cannot read property 'model_config' of null
    at models.ts:287
    at common.ts:14
    at Object.next (common.ts:14)
    at a (common.ts:14)

I found some Github issues like this one https://github.com/tensorflow/tfjs/issues/931, that mention the issue is that the .h5 file only includes the weights and they provide a workaround which involves saving the model with the weights, but it is not clear to me how to do that with the HF library.

Is this something you support or is there a way to get the Keras model with the weights?


closes #1960 Add saving and resuming functionality for remaining examples

#1987 was merged in before I could update the other pytorch examples. This should also close #1960 once it's merged in.


Refactor functionality of run_squad and squad_utils into XXXForQuestionAnswering

## Request

Push some/most/all the functionality of the squad training scripts into the class ``XXXForQuestionAnswering``.

## Alt Request.

I'm guessing this is immediately objectionable, since ``XXXForQuestionAnswering`` is just the nice clean NN.
No messy string manipulation functions welcome here.

So perhaps I request a ``XXXQuestionAnswerHandler`` class.

# Context

Basically, as far as I can tell, there is quite a gulf between the input/output of ``XXXForQuestionAnswering`` and actually doing squad.
Currently, we attempt to straddle this gulf by a number of scripts, perhaps what's called glue code(?).
These require passing around of many arguments, so many that I can't keep track of them,
and a lot of conditionals to treat the idiosyncrasies of different models.

I think separating the ``XXXForQuestionAnswering`` from actually being able to do a squad like task is a cause of some problems.
If these models are really supposed to be used for question answering,
and not simply churning through a full training/eval squad-style json,
then these auxilliary scripts of answer selection, and answer cleaning should be fastened to the model firmly (like within some handler).

Squad has set the defacto format for CDQA,
and many of the steps in the scripts would be useful in wider applications.

## Context II

A massive thanks for refactoring the squad training code.
It is so much clearer than it was but I still think there's big room for improvement.

For me, using the previous incarnation of squad_run was ummm... constantly problematic (eg [like this](https://github.com/huggingface/transformers/issues/2038)).
Surely 97% cos I'm a noob - some of the issues I had unfathomably basic.
But the scripts were really not user friendly (now improved but still - see previous), and the current classes really don't give much away:

* "Hello, my dog is cute" is not a CDQA [?!](https://github.com/huggingface/transformers/blob/master/transformers/modeling_xlm.py#L869).

* The ``...Simple`` class, partly clarified by LysandreJik's [post](https://github.com/huggingface/transformers/issues/2038#issuecomment-564220238),
but doesn't explain why these are still lingering on the master branch without appearing in the official docs, while the tensor flow analogue only has a simple...
I mean its fine, but its pretty confusing if you're already confused.

* the documentation of the output being "well depends on the config" which although true is... not exactly useful to the uninitiated.

I got to the point of trying to refactor the training scripts myself, but made very little progress.
Very happy to see someone else has been on the case.

## Context III

An example functionality: allow pre-cache examples-to-feature - effectively do a sort of dry run.

Running ``squad_run.py`` on a GPU machine saw a lot of idle time where one CPU would take an hour to cache the examples suitable to the configuration of the finetuning.
Why not build a ``examples_to_features`` method within the class?
Then do this on your desktop before shipping if off for training _oven-ready_.
At the moment in the script ``squad_run.py``, the caching for the training is called from ``main`` (not ``train``) and caching the evaluation was done in ``evaluate``.
I don't follow this decision.
I tried to extract the caching function, but it was super hacky and would be addressed by the request.

## Invitation for comments

I'm sure this isn't a new idea, and hasn't happened cos its either too much work or a terrible idea.
I tried to see what others were doing but the scripts and specifics people have used to arrive at their claimed squad scores
do not seem as available as other resources ( ... XLM? ... Roberta? - Am I missing something?)

I'd be very happy to hear thoughts on this,
including responses that begin

> _"This is nonsense because ..."_

Thanks HF for your awesome opensource NLP lib


`bert-base-uncased` tokenizer broke around special tokens in v2.2.1

In `v2.2.1`, the `bert-base-uncased` tokenizer changed in a way that's probably not intentional:

```
Python 3.7.5 (default, Oct 25 2019, 10:52:18)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers.tokenization_auto import AutoTokenizer
To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.
>>> t = AutoTokenizer.from_pretrained("bert-base-uncased"); t.encode_plus(text='A, [MASK] AllenNLP sentence.')
{
    'input_ids': [101, 1037, 1010, 1031, 7308, 1033, 5297, 20554, 2361, 6251, 1012, 102],
    'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
}
```

In `v2.2.0`:
```
Python 3.7.5 (default, Oct 25 2019, 10:52:18)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers.tokenization_auto import AutoTokenizer
To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.
>>> t = AutoTokenizer.from_pretrained("bert-base-uncased"); t.encode_plus(text='A, [MASK] AllenNLP sentence.')
{
    'special_tokens_mask': [1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
    'input_ids': [101, 1037, 1010, 103, 5297, 20554, 2361, 6251, 1012, 102],
    'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
}
```
(indented the results for clarity)

The key difference is that in `v2.2.0`, it recognizes the `[MASK]` token as a special token and gives it token id `103`. In `v2.2.1`, this no longer happens. The behavior of `bert-base-cased` has not changed, so I don't think this is an intentional change.


[AB-219] Progress bar

## This PR:
- adds progress bars to tokenization


[BREAKING CHANGE] Setting all ignored index to the PyTorch standard

The CrossEntropy loss, as well as other losses, accept a value as an index they will ignore when computing the loss. This value was set to -1 in some cases, but left to the default value (-100) in other cases.

To stay consistent we're setting the value to be the default PyTorch one in all cases.

Includes a few documentation fixes.


Progress indicator improvements when downloading pre-trained models.

Downloading GPT2-XL can take a while.  If you're not expecting it, the current progress bar can be confusing.  It looks like this:

```
  4%|▉                       | 257561600/6431878936 [00:33<16:12, 6351328.14B/s]
```

With this change, the progress bar is much more readable:

```
Downloading:   3%|▋                         | 166M/6.43G [00:30<12:34, 8.31MB/s]
```

Also, by importing from `tqdm.auto` you will get a nice graphical progress bar if you're running in a jupyter notebook.  (Unless you're using jupyter lab and you don't have widgets set up properly, but that's it's own ball of wax.)


## In which directory the downloaded roberta-base models will be stored on linux server conda environment


Where is extract_features.py and run_classifier.py ?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello! I couldn't find the extract_features.py and run_classifier.py. Have they been renamed ?


Model2Model: RuntimeError: expected device cpu and dtype Float but got device cpu and dtype Bool

## ❓ Questions & Help

I'm going to try the new Model2Model feature:

```
import torch
import numpy as np
from transformers import Model2Model, BertTokenizer, BertModel

# device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
device = torch.device("cpu")

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = Model2Model.from_pretrained('bert-base-uncased').to(device)

source_ids = torch.tensor([tokenizer.encode("this is source sentence", add_special_tokens=True)],
                          dtype=torch.long).to(device)
target_ids = torch.tensor([tokenizer.encode("this is target sentence", add_special_tokens=True)],
                          dtype=torch.long).to(device)

model(source_ids, target_ids)
```

This is the output:
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-885d1d4b847f> in <module>
----> 1 model(source_ids, target_ids)

/users/tr.amirhj/anaconda3/envs/genv/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/users/tr.amirhj/anaconda3/envs/genv/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py in forward(self, encoder_input_ids, decoder_input_ids, **kwargs)
    229             "attention_mask", None
    230         )
--> 231         decoder_outputs = self.decoder(decoder_input_ids, **kwargs_decoder)
    232
    233         return decoder_outputs + encoder_outputs

/users/tr.amirhj/anaconda3/envs/genv/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/users/tr.amirhj/anaconda3/envs/genv/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, masked_lm_labels, encoder_hidden_states, encoder_attention_mask, lm_labels)
    871                             inputs_embeds=inputs_embeds,
    872                             encoder_hidden_states=encoder_hidden_states,
--> 873                             encoder_attention_mask=encoder_attention_mask)
    874
    875         sequence_output = outputs[0]

/users/tr.amirhj/anaconda3/envs/genv/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

/users/tr.amirhj/anaconda3/envs/genv/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    677                 seq_ids = torch.arange(seq_length, device=device)
    678                 causal_mask = seq_ids[None, None, :].repeat(batch_size, seq_length, 1) <= seq_ids[None, :, None]
--> 679                 extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
    680             else:
    681                 extended_attention_mask = attention_mask[:, None, None, :]

RuntimeError: expected device cpu and dtype Float but got device cpu and dtype Bool
```

If I missed something?


DistilmBERT training/distillation dataset

## ❓ Questions & Help

Thanks a lot for Distil**m**BERT (amongst everything else), is there any info on the dataset used in the distillation process?

Both the dataset itself or the process used to obtain it would be greatly appreciated!

Am I right to assume you used a similar (if not the same) data as the original [`multilingual-bert`](https://github.com/google-research/bert/blob/master/multilingual.md#details) with a processed dump of the biggest 104 wikipedia dumps?
Again, any pointer to the preprocessing steps would be great!


Is there a way to evaluate models during training in Multi-gpu setting

## ❓ Questions & Help

Hi all,
I always see comments in examples saying that "when single GPU otherwise metrics may not average well". So is this really something that shouldn't be done? I mean, is there a way to evaluate the model safely after each epoch in the multi-gpu training setting? Thanks.
<!-- A clear and concise description of the question. -->


Transformers for Tabular data extraction - e.g., wikitables

Hi Team,
Can you please let us know if Transformers can be used to extract information from tabular data. Example is - [https://demo.allennlp.org/wikitables-parser](https://demo.allennlp.org/wikitables-parser) . WikiTables is the dataset.

Example questions can be: show me all students who got marks greater than 40%

Wondering if BERT or any other SOTA transformer technology can be leveraged to solve this NLP problem


Remove misplaced summarization documentation

Documentation for the previous version of abstractive summarization is still present in the repository:

https://twitter.com/DavidMezzetti/status/1204123548966621184

This PR removes it.


"Write With Transformer" interface returning 502 on gpt2/xl model

## 🐛 Bug

The "Write With Transformer" interface is returning a `502` when the API calls the gpt2/xl model.

See: https://transformer.huggingface.co/doc/gpt2-xl

## To Reproduce

Steps to reproduce the behavior just using the API request:

```
curl 'https://transformer.huggingface.co/autocomplete/gpt2/xl' --data-binary '{"context":"See how a modern ","model_size":"gpt2/xl","top_p":0.9,"temperature":1,"max_time":1}' --compressed
```

## Expected behavior

Expecting autocomplete results, but getting this `502` response instead.
```
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
```


BertModel.from_pretrained() doesn't accept pathlib.PosixPath anymore

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): Japanese

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Prepare a directory with `config.json`, `pytorch_model.bin`.
2. Give the directory as a pathlib.PosixPath such like `bert_model = BertModel.from_pretrained(bert_path)`
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
TypeError: argument of type 'PosixPath' is not iterable

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
This used to load the model with 2.1.1, but started to cause an error since 2.2.0.
`bert_model = BertModel.from_pretrained(str(bert_path))` works.

## Environment

colab
* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch): 2.2.0 and later
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Finetune and generate text with BertForMaskedLM

## ❓ Questions & Help

I am trying to fine-tune and generate text using BertForMaskedLM. Although my script works I am not getting the output I am expecting. I am confused on what should I pass to BertForMaskedLM when training (attention mask, token types ids, etc) and how to generate text once the model is fine tuned. Any help is welcome, hereafter is my current code:

```
import torch
from torch.optim import Adam
from transformers import BertForMaskedLM, AutoTokenizer

if __name__ == "__main__":

    lr = 0.002
    epochs = 20
    model = BertForMaskedLM.from_pretrained("bert-base-uncased")
    tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
    optimizer = Adam(model.parameters(), lr=lr)

    dataset = ["this is the first sentence.",
               "this is the second, slightly longer, sentence.",
               "this is the third and last sentence."]

    # We precomputed this
    max_len = 12

    # Since we only have 3 sentences we fit all our dataset in a single batch.
    # The padded batch will look like:
    # [[101, 2023, 2003, 1996, 2034, 6251, 1012, 102, 0, 0, 0, 0],
    #  [101, 2023, 2003, 1996, 2117, 1010, 3621, 2936, 1010, 6251, 1012, 102],
    #  [101, 2023, 2003, 1996, 2353, 1998, 2197, 6251, 1012, 102, 0, 0]]
    padded_batch = []
    padding_id = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)
    for sentence in dataset:
        encoded_sentence = tokenizer.encode(sentence)
        padded_sentence = encoded_sentence + \
            [padding_id]*(max_len-len(encoded_sentence))

        padded_batch.append(padded_sentence)

    # The attention mask will have the same shape of the batch, with 0s for the
    # padded element and 1s for the non-padded ones
    # [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0],
    #  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
    #  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]]
    attention_mask = [[1 if t_id != padding_id else 0 for t_id in
                       tensor] for tensor in padded_batch]

    # The lm_labels will be the same as the padded batch but for the padded
    # elements which are replaced by -1
    # [[101, 2023, 2003, 1996, 2034, 6251, 1012, 102, -1, -1, -1, -1],
    #  [101, 2023, 2003, 1996, 2117, 1010, 3621, 2936, 1010, 6251, 1012, 102],
    #  [101, 2023, 2003, 1996, 2353, 1998, 2197, 6251, 1012, 102, -1, -1]]
    lm_labels = [[t_id if t_id != padding_id else -1 for t_id in tensor] for
                 tensor in padded_batch]

    # Converting the model input from list to tensor
    padded_batch = torch.tensor(padded_batch)
    attention_mask = torch.tensor(attention_mask)
    lm_labels = torch.tensor(lm_labels)

    # Since we only have one batch every epoch we do a single forward pass,
    # backprop and optimization step
    for i in range(epochs):
        loss, _ = model(input_ids=padded_batch, attention_mask=attention_mask,
                        lm_labels=lm_labels)
        print(loss.item())

        loss.backward()
        optimizer.step()
        model.zero_grad()

    # The model should now be trained and we want to generate the first three
    # words of a new sentence. Given the training data used we expect it to be
    # "this is the".

    # Initialize the model input with "[CLS] [MASK]" to generate word w_1, the
    # input for w_i is "[CLS] w_1 ... w_(i-1) [MASK]" where w_1 ... w_(i-1)
    # have been generated during previous steps.
    output = [101, 103]
    for i in range(4):
        generation_input = torch.tensor([output])
        pred = model(generation_input)[0]
        new_index = torch.argmax(pred, -1)
        output[-1] = new_index[:, -1].item()
        output.append(103)

    print(output)
    print(tokenizer.decode(output))

```


Could convert_pytorch_checkpoint_to_tf2.py convert any pytorch model to tf2?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


Encoder-decoders in Transformers

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. I'm trying to use the hybrid seq2seq model described in this [article](https://medium.com/huggingface/encoder-decoders-in-transformers-a-hybrid-pre-trained-architecture-for-seq2seq-af4d7bf14bb8). It is stated that the library is available from 2.2.0 version. I tried in both 2.2.0 and 2.2.1
I don't find the respective libraries working as expected.

```
from transformers import PreTrainedEncoderDecoder
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased','gpt2')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
encoder_input_ids=tokenizer.encode("Hi How are you")
ouput = model(torch.tensor( encoder_input_ids).unsqueeze(0),torch.tensor( encoder_input_ids).unsqueeze(0) )
```
and I get the following error:
```
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
```
I checked the code of modelling [gpt2](https://github.com/huggingface/transformers/blob/1d189304624db17749aee23fa2345f009cc48215/transformers/modeling_gpt2.py#L541) it doesn't take any input as encoder_hidden_states.

2. I also tried the another example from that article
![1_my6KF5Wa2AFa54tPnQp3Yw](https://user-images.githubusercontent.com/12907396/70492861-013de580-1ac3-11ea-9197-86ebae611f7e.png)

There is no decode method in Model2Model, but then do you mean decoder?
But then Bert using decoder I get the following error

```
TypeError: forward() got an unexpected keyword argument 'length'
```
As bert doesn't take the length as the input or any of the parameters shown in the example.

* OS: Windows
* Python version: 3.7.4
* PyTorch version:1.3.0
* PyTorch Transformers version (or branch):2.2.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Traceback (most recent call last):
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f912c2cc550>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /models.huggingface.co/bert/bert-base-chinese-vocab.txt (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f912c2cc550>: Failed to establish a new connection: [Errno 110] Connection timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 360, in _from_pretrained
    resolved_vocab_files[file_id] = cached_path(file_path, cache_dir=cache_dir, force_download=force_download, proxies=proxies, resume_download=resume_download)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/transformers/file_utils.py", line 180, in cached_path
    resume_download=resume_download)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/transformers/file_utils.py", line 327, in get_from_cache
    http_get(url, temp_file, proxies=proxies, resume_size=resume_size)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/transformers/file_utils.py", line 243, in http_get
    response = requests.get(url, stream=True, proxies=proxies, headers=headers)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /models.huggingface.co/bert/bert-base-chinese-vocab.txt (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f912c2cc550>: Failed to establish a new connection: [Errno 110] Connection timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bert.py", line 125, in <module>
    text_train, text_dev, text_test, label_train, label_dev, label_test = load_dataset('qa_dataset/Beauty_domain.txt', max_len = 60)
  File "bert.py", line 96, in load_dataset
    text_list = prepare_data(text_list)
  File "bert.py", line 54, in prepare_data
    tokenizer = BertTokenizer.from_pretrained("bert-base-chinese")
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 286, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/hgy/anaconda3/envs/pytorch-python3/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 372, in _from_pretrained
    raise EnvironmentError(msg)
OSError: Couldn't reach server at '{}' to download vocabulary files.


[WIP] Add MMBT Model to Transformers Repo

Implements the MMBT Model from Supervised Multimodal Bitransformers for Classifying Images and Text by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine (https://arxiv.org/abs/1909.02950) (https://github.com/facebookresearch/mmbt/)

Adds run_mmimdb.py to show example training run on MM-IMDb dataset (http://lisi1.unal.edu.co/mmimdb/)


Split models to multiple GPUs

I am willing to fine-tune GPT2-large which simply does not fit into GPU memory. I wanted to run the script `run_lm_finetuning.py` with GPT2-large having two Nvidia Tesla P100, but I suppose model splitting in not supported. Or am I wrong?


Running run_lm_finetuning.py within python

## Setup
* Model: roberta-base
* Language: english
* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Script inputs:
```
python run_lm_finetuning.py \
--output_dir=$OUTPUT_DIR \
--model_type=roberta \
--model_name_or_path=roberta_base \
--do_train \
--train_data_file=$TRAIN_FILE \
--do_eval \
--eval_data_file=$TEST_FILE \
--mlm \
--no_cuda
```
## ❓ Questions & Help
Is there a way to run the above within python? Said differently, if I want to call `run_lm_finetuning.py` from within one of my own python scripts using the above configurations, how would I best go about doing that?

Thanks in advance!


XLM model masked word prediction Double Language

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am trying to generate in-context word translations.

For instance, if the target language is french and "well" is the word to translate.
- I walked to the well. -> the translation for "well" should be "puit"
- I am doing well. -> the translation for "well" should be "bien"

I have a simple solution inspired from https://github.com/huggingface/transformers/issues/1842#issuecomment-555734728 and https://github.com/qiang2100/BERT-LS

I am basically concatenating a sentence clone, masking the second target word and changing its language inside the "langs" tensor.

The code is something like this :

```
from transformers import XLMTokenizer, XLMWithLMHeadModel
import torch

# model
model_string = "xlm-mlm-tlm-xnli15-1024"

# load tokenizer
tokenizer = XLMTokenizer.from_pretrained(model_string)

# encode sentence with a masked token in the middle
encoded_array = tokenizer.encode(
    "That is a well. That is a " + tokenizer.mask_token + ".")
sentence = torch.tensor([encoded_array])

# Identify the masked token position
masked_index = torch.where(sentence == tokenizer.mask_token_id)[1].tolist()[0]

# Load model
model = XLMWithLMHeadModel.from_pretrained(model_string)

# Load languages
language_id_from = tokenizer.lang2id['en']  # 0
language_id_to = tokenizer.lang2id['fr']  # 0
languages_array = [language_id_from] * len(encoded_array)
languages_array[masked_index] = language_id_to

langs = torch.tensor(languages_array)
langs = langs.view(1, -1)

# Get the five top answers
result = model(input_ids=sentence, langs=langs)
prediction_scores = result[0]

result = prediction_scores[:, masked_index].topk(20).indices
result = result.tolist()[0]
print(tokenizer.decode(result))

```

I've been getting some positive results, and not so positive results (with longer sentences). For example:

"I walked to the well" : well -> Puit is the 3rd result. [easily identifiable as correct!]
"I felt well." : well -> Bien is the 6th result. [easily identifiable as correct!]
"If the sentence is too convoluted, it won't translate well." -> The results are all parts of correct answers but wrong individually. Like "correct" and "ly" in french.  [very hard to piece together answer]
"After a hard day's work like yesterday, I really like to go jump in the well to cool down." -> all results are in English for some reason.  [no french at all!]
Sentences with the animal meaning of "bat" don't work at all. I don't think it knows a bat is an animal.

My questions.
- Should I be looking at something else than xlm; is there a better way of doing this already out there, or existing solution? I would rather avoid a cloud service like Yandex translate, as I will want to do this a lot!
- If this can be achieved with xlm, is there a way to force a particular language, and full words?


Could not run run_ner.py based on XLNET model

## ❓ Questions & Help

Hello everyone, when I try to use ELnet model for the NER task through run_ner.py, it shows the following problem:

__init__() got an unexpected keyword argument 'do_lower_case'

So is it some problem in the modeling_utils.py? Thanks for someone's response!


unable to load the downloaded BERT model  offline in local  machine . could not  find config.json and  Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] |

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I have downloaded the bert model [from the link in bert github page](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) offline but unable to load the model offline  .

from transformers import *
model = BertForMaskedLM.from_pretrained("/Users/Downloads/uncased_L-12_H-768_A-12/")

 Model name '/Users/Downloads/uncased_L-12_H-768_A-12/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed '/Users/Downloads/uncased_L-12_H-768_A-12/' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

below are the files present in /Users/Downloads/uncased_L-12_H-768_A-12/
bert_config.json
bert_model.ckpt.data-00000-of-00001
bert_model.ckpt.index
bert_model.ckpt.meta
vocab.txt
what should i do to load the downloaded model offline ?

since the error was saying config.json not found i changed the above 4 file names by removing the word bert from it.Below are the new file names

config.json
model.ckpt.data-00000-of-00001
model.ckpt.index
model.ckpt.meta
vocab.txt
now when i  load the downloaded model offline i get a different error

from transformers import *
model = BertForMaskedLM.from_pretrained("/Users/Downloads/uncased_L-12_H-768_A-12/")

Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory /Users/Downloads/uncased_L-12_H-768_A-12/ or `from_tf` set to False

python version:3.7
tensorflow version:1.12


Error in TFBertForSequenceClassification

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): Multi-lingual

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

I have fine-tuned a language model using `run_lm_finetuning.py`.

When trying to load it with TFBertForSequenceClassification however, it fails.

```
config = transformers.BertConfig.from_json_file('./bertlm_model/config.json')
model = transformers.TFBertForSequenceClassification.from_pretrained('./bertlm_model/', from_pt = True)
```

Showing the following error:

```
>>> model = transformers.TFBertForSequenceClassification.from_pretrained('./bertlm_model/', from_pt = True, config = config)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/myuser/.local/share/virtualenvs/BERTlm-giwpujkO/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 288, in from_pretrained
    return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file)
  File "/home/myuser/.local/share/virtualenvs/BERTlm-giwpujkO/lib/python3.7/site-packages/transformers/modeling_tf_pytorch_utils.py", line 82, in load_pytorch_checkpoint_in_tf2_model
    return load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys)
  File "/home/myuser/.local/share/virtualenvs/BERTlm-giwpujkO/lib/python3.7/site-packages/transformers/modeling_tf_pytorch_utils.py", line 145, in load_pytorch_weights_in_tf2_model
    assert name in pt_state_dict, "{} not found in PyTorch model".format(name)
AssertionError: classifier.weight not found in PyTorch model
```

If I try to run either `transformers.BertForSequenceClassification.from_pretrained('bertlm_model')` or `transformers.TFBertModel.from_pretrained('bertlm_model', from_pt = True)` all is fine!

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 18.04
* Python version: 3.7.5
* PyTorch version: 1.3.1
* Transformers version (or branch): Git repo master comit 0cb163865a4c761c226b151283309eedb2b1ca4d
* Using GPU: Yes
* Distributed of parallel setup ?
* Any other relevant information:


I am running bert fine tuning with cnnbase model but my project stops at loss.backward() without any prompt in cmd.

My aim is to make a five-category text classification

I am running transformers fine tuning bert  with `cnnbase` model but my program stops at `loss.backward()` without any prompt in `cmd`.

I debug find that the program stop at the loss.backward line without any error prompt

My program runs successfully in `rnn base` such as `lstm` and `rcnn`.

But when I am running some `cnnbase` model the strange bug  appears.

My cnn model code:
```
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers.modeling_bert import BertPreTrainedModel, BertModel
n_filters = 200
filter_sizes = [2,3,4]
class BertCNN(BertPreTrainedModel):
    def __init__(self, config):
        super(BertPreTrainedModel, self).__init__(config)
        self.num_filters = n_filters
        self.filter_sizes = filter_sizes
        self.bert = BertModel(config)
        for param in self.bert.parameters():
            param.requires_grad = True
        self.convs = nn.ModuleList(
            [nn.Conv2d(1, self.num_filters, (k, config.hidden_size))
                for k in self.filter_sizes])
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.fc_cnn = nn.Linear(self.num_filters *
                                len(self.filter_sizes), config.num_labels)

    def conv_and_pool(self, x, conv):
        x = F.relu(conv(x)).squeeze(3)
        x = F.max_pool1d(x, x.size(2)).squeeze(2)
        return x

    def forward(self, input_ids,
                attention_mask=None, token_type_ids=None, head_mask=None):
        outputs = self.bert(input_ids,
                            attention_mask=attention_mask,
                            token_type_ids=token_type_ids,
                            head_mask=head_mask)
        encoder_out, text_cls = outputs
        out = encoder_out.unsqueeze(1)
        out = torch.cat([self.conv_and_pool(out, conv)
                         for conv in self.convs], 1)
        out = self.dropout(out)
        out = self.fc_cnn(out)
        return out
```

My train code:
```
        for step, batch in enumerate(data):
            self.model.train()
            batch = tuple(t.to(self.device) for t in batch)
            input_ids, input_mask, segment_ids, label_ids = batch
            print("input_ids, input_mask, segment_ids, label_ids SIZE: \n")
            print(input_ids.size(), input_mask.size(),segment_ids.size(), label_ids.size())
            # torch.Size([2, 80]) torch.Size([2, 80]) torch.Size([2, 80]) torch.Size([2])
            logits = self.model(input_ids, segment_ids, input_mask)
            print("logits and label ids size: ",logits.size(), label_ids.size())
            # torch.Size([2, 5]) torch.Size([2])
            loss = self.criterion(output=logits, target=label_ids) #loss function:CrossEntropyLoss()
            if len(self.n_gpu) >= 2:
                loss = loss.mean()
            if self.gradient_accumulation_steps > 1:
                loss = loss / self.gradient_accumulation_steps
            if self.fp16:
                with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                    scaled_loss.backward()
                clip_grad_norm_(amp.master_params(self.optimizer), self.grad_clip)
            else:
                loss.backward() # I debug find that the program stop at this line without any error prompt
```
HELP~！~   、
I  posted my questions on various community platforms，stackoverflow、other github repositories.
No one replied to me.


create encoder attention mask from shape of hidden states

As noted by @efeiefei (#1770) we currently create masks on the encoder hidden states (when they're not provided) based on the shape of the inputs to the decoder. This is obviously wrong; sequences can be of different lengths. We now create the encoder attention mask based on the `batch_size` and `sequence_length` of the encoder hidden states.


RobertaTokenizer runs slowly after add _tokens

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I use RobertaTokenizer like this:
```python
tokenizer = RobertaTokenizer.from_pretrained(FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
tokenizer.add_tokens([x.strip() for x in open('add_tokens.txt').readlines()])
```
There are about 200 words in `add_tokens.txt`. I tested it on 300 sample datasets, and 250% more time after using `add_tokens.txt`. Is there any way to optimize it？


Some bug in using eval_all_checkpoints

when using --eval_all_checkpoints
checkpoints will find a pytorch_model.bin just under output_dir
when calling evaluate(args, model, tokenizer, prefix=global_step) will get a FileNotFoundError


Having trouble reproducing SQuAD 2.0 results using ALBERT v2 models

## ❓ Questions & Help

I tried to finetune ALBERT v2 models on SQuAD 2.0, but sometimes the loss doesn't decrease and performance on dev set is low. The problem may happen when using `albert-large-v2` and `albert-xlarge-v2` in my case. Any suggestions?
![TIM截图20191209111606](https://user-images.githubusercontent.com/14048129/70404527-54894700-1a75-11ea-8ec2-1471547e01a9.png)
![TIM截图20191209111551](https://user-images.githubusercontent.com/14048129/70404532-58b56480-1a75-11ea-9f60-17f4fc0b0200.png)
![TIM截图20191209111533](https://user-images.githubusercontent.com/14048129/70404534-5b17be80-1a75-11ea-88ea-a02a4f0dcb2e.png)


Is there any way to treat the whitespace characters same as other characters when tokenizing?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to work on clinical notes data, and the whitespaces in the notes may contain useful information (e.g. section separation). Is there any way to encode the whitespaces as well during tokenization?


How to pretrain BERT whole word masking (wwm) model?

## 🚀 Feature

Code to pretrain BERT whole word masking (wwm) model

## Motivation

WWM offers better performance, but the current codebase doesn't seem to support this feature.

## Additional context

Related i #1352


:bug: #2096 in tokenizer.decode, adds a space after special tokens for string format

This correction is cosmetic to correct the observed formatting issue.
No test was implemented because ideally composition of functions `encode.decode` should in theory return the original sentence. Yet there are some space strip (and lower-casing) in code so it's not certain to return exactly the original sentence with same spaces.


Unclear how to decode a model's output

## Unclear how to decode a model's output

Hello, after digging through the docs for about an hour it's still rather unclear to me how one is supposed to decode a model's output.

Using the following code:
```
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased')

input = torch.tensor(tokenizer.encode('Some text')).unsqueeze(0)
outputs = model(input)
lhs = outputs[0]
print(tokenizer.decode(lhs))
```

The lhs is always decoded as `[UNK]`

Is this just the expected result due to the model being untrained ? Is the decode functionality of the tokenizer being used in the wrong way ?

Searching for `decode` in the docs yields no code examples with it being used with a model's output.


which special token is used to predict the score in roberta?

In bert, we use the embedding of <cls> to predict the score, how about the roberta?


Understanding output of models and relation to token probability

## ❓ Questions & Help

So I understand that different models were trained on different objectives. An important one is a masked language modeling objective. I would assume, then, that the model outputs probabilities for each token as the final output. Is that true?

For models that have not been trained on MLM, is it still possible to get the model's given probability for that token? (I imagine that just taking the sigmoid is not exactly the probability of the model, right?)


about the special tokens

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
The question about roberta.
I konw that Bert use the embedding of token 'cls' to do predict, but when it comes to roberta, I dont know it clearly. Can you tell me which token's embedding is used to do predict in this project? Is it '<s>' ?


The added tokens do not work as expected

Here is a minimum example, where we add a special token [ENT]
```
from transformers import BertTokenizer
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased-vocab.txt')

bert_tokenizer.add_tokens(['[ENT]'])
print(len(tokenizer))

x = bert_tokenizer.encode("you are the [ENT] with [ENT] and [ENT]")
print(x)
bert_tokenizer.decode(x)
```
After decoding, we will end up having
```
you are the [ENT]with [ENT]with [ENT]
```
rather than
```
'you are the [ENT] with [ENT] with [ENT]'

```


Can't get gradients from TF TransformerXL model forward pass

## 🐛 Bug (Actually I'm not very sure if it's a bug or am I doing something wrong)

<!-- Important information -->

- Model I am using : Transformer-XL

- Language I am using the model on (English, Chinese....): Chinese

- The problem arise when using: my own modified scripts

- The tasks I am working on is: my own task or dataset

## To Reproduce

0. My current testing environment is a CPU machine and a Cloud GPU machine I rented.
1. Clone my repository(https://github.com/Morizeyao/Decoders-Chinese-TF2.0) and install requirements.
2. Copy contents from scripts folder to root folder.
3. Run prepare_data.sh
4. Run train_xl.sh
5. You will see the print result from line 107 of file train_transformer_xl.py shows that the gradients are all zero.
6. And the loss don't change during the training process.

## Expected behavior

I have tested TFGPT2 model from Transformers library, it worked fine (just run train_gpt2.py). Only TFTransfomerXL has this problem.

## Environment

* OS: macOS and Ubuntu
* Python version: 3.7
* PyTorch version: NA
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? No
* Distributed of parallel setup ? No

## And...

- Thank you guys for this awesome project!


How to save a model as a BertModel

## ❓ Questions & Help
I first fine-tuned a bert-base-uncased model on SST-2 dataset with run_glue.py. Then i want to use the output pytorch_model.bin to do a further fine-tuning on MNLI dataset. But if i directly use this pytorch_model.bin, an error will occur:

> RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
> 	size mismatch for classifier.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([3, 768]).
> 	size mismatch for classifier.bias: copying a param with shape torch.Size([2]) from checkpoint, the shape in current model is torch.Size([3]).

This error occurred because SST-2 has two classes but MNLI has three classes. Issue #1108 provide a solution by saving the BertModel without the classification head. But i wander if it‘s feasible for that the model class is chosen as BertForSequenceClassification at the beginning. How do i change the model class in saving step?


Remove pytest dependency.

This is a follow-up to PR #2055. This file was added between the moment I wrote #2055 and the moment in was merged.


When I use albertModel, it prints the following repeatedly.

```python
0 0
Layer index 0
0 1
Layer index 0
0 2
Layer index 0
0 3
Layer index 0
0 4
Layer index 0
0 5
Layer index 0
0 6
Layer index 0
0 7
Layer index 0
0 8
Layer index 0
0 9
Layer index 0
0 10
Layer index 0
0 11
Layer index 0

0 0
Layer index 0
0 1
Layer index 0
0 2
Layer index 0
0 3
Layer index 0
0 4
Layer index 0
0 5
Layer index 0
0 6
Layer index 0
0 7
Layer index 0
0 8
Layer index 0
0 9
Layer index 0
0 10
Layer index 0
0 11
Layer index 0

.......
```
Is this a mistake?


Error msg when running on the colab

## ❓ Questions & Help

![捕获](https://user-images.githubusercontent.com/43159433/70368968-05c68c00-1880-11ea-8d97-5189348a4eca.PNG)

Can anyone tell me where am I wrong or it's not my problem?I cloned whole the files from huggingface. Is it can be fixed? I would appreciate for any suggestion. Thank you.


AssertionError in official example

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

in the [official example](https://huggingface.co/transformers/quickstart.html)
the tokenizer result raise an assertionError

```
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']

```
gives

```
AssertionError                            Traceback (most recent call last)
<ipython-input-1-6533b2fb8252> in <module>
     16 masked_index = 8
     17 tokenized_text[masked_index] = '[MASK]'
---> 18 assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']
     19
     20 # Convert token to vocabulary indices

AssertionError:

```
And when i print out the tokenized_text, i find the sepcial tokens had been tokenized in a wrong way.
This may caused by the lower operation to the speicial tokens

```
print(tokenized_text)
['[', 'cl', '##s', ']', 'who', 'was', 'jim', 'henson', '[MASK]', '[', 'sep', ']', 'jim', 'henson', 'was', 'a', 'puppet', '##eer', '[', 'sep', ']']
```

## Expected behavior

no exception.

## Environment

* OS: Ubuntu 19.04 / Centos ?
* Python version: 3.7.5 / 3.6.4
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? maybe
* Distributed of parallel setup ? No
* Any other relevant information:

The official example is ok in Transformers version 2.1.1, after i update my Transformers it goes wrong

## Additional context


Use run_lm-finetuning on tpu

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Is it possible to use the script run_lm-finetuning on TPUs, if not, what do you recommend to fine-tune BERT language model on TPUs using the transformers library


Help with converting fine-tuned PT model to TF checkpoint

How do I convert PT model (.bin) to TF checkpoint successfully so that I can start serving using bert-as-a-service?

Below are the steps and errors:
Huggingface v2.2.1, Pytorch 1.2, TF 2.0

1. executed run_lm_finetuning.py to fine-tune an already finetuned model (clinicalBERT) on the target domain dataset. Successfully saved all the necessary files (.bin, config, vocab etc.)
2. To convert PT to TF, executed convert_pytorch_checkpoint_to_tf2.py  with    --tf_dump_path="/tf_test/" --model_type="bert" --pytorch_checkpoint_path="../pytorch_model.bin" --config_file='../config.json'
 **below was the error**
```
 Traceback (most recent call last):
File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 248, in
only_convert_finetuned_models=args.only_convert_finetuned_models)
File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 194, in convert_all_pt_checkpoints_to_tf
compare_with_pt_model=compare_with_pt_model)
File "/home/imagen/skc/bert/transformers-2.2.1/transformers/convert_pytorch_checkpoint_to_tf2.py", line 115, in convert_pt_checkpoint_to_tf
tf_model = load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path)
File "/home/imagen/skc/environments/.virtualenvs/lstm_dev_tf2x/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 82, in load_pytorch_checkpoint_in_tf2_model
return load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys)
File "/home/imagen/skc/environments/.virtualenvs/lstm_dev_tf2x/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 145, in load_pytorch_weights_in_tf2_model
assert name in pt_state_dict, "{} not found in PyTorch model".format(name)
**AssertionError: cls.seq_relationship.weight not found in PyTorch model**
```

3. I wanted to test PT to TF conversion, so I've pointed the script to original clinicalBERT model directory and it successfully converted. However, it was saved as .h5 model and not .ckpt
3.1 Ran below code to convert .h5 to save it as checkpoint - however, it seems not possible to save as checkpoint without creating the model architecture
 ran below code for saving as .ckpt in tf2.0

```
import tensorflow as tf
from keras.models import load_model
saver = tf.train.Checkpoint()
model = load_model("../converted_model-tf_model.h5", compile=False)
sess = tf.compat.v1.keras.backend.get_session()
save_path = saver.save("../converted_model-tf_model.ckpt")
```

So, in order to successfully use a fine-tuned model in bert-as-a-service

1. Was there anything I am doing incorrectly when fine-tuning a model? because, somehow the PT to TF conversion goes smoothly for clinicalBERT, but not for fine-tuned version of it (AssertionError: cls.seq_relationship.weight not found in PyTorch model)

2. How to save as checkpoint (.ckpt) instead of .h5 model? this is for bert-as-a-service? if this is not possible, please suggest alternatives (is creating architecture a necessary step?)
    #2069 - fwiw - I've used cleaned up version of the script

Thanks


How can I get similarity matching ?

## ❓ Questions & Help
Is there any way that can help me calculate the similarity between 2 questions ? Sometimes the questions is out of the scope of the data set questions. Using a simple similarity algorithm will always return the most similar even if it is not really correct.
It is the same thing as here :
https://github.com/deepmipt/dp_notebooks/blob/master/DP_BERT.ipynb

from deeppavlov import build_model, configs
model = build_model(configs.squad.squad_bert, download=True)
model(['DeepPavlov is a library for NLP and dialogue systems.'], ['What is DeepPavlov?'])

<!-- A clear and concise description of the question. -->


Hi,

The script examples/run_lm_finetuning.py skips evaluation on the validation dataset when run in distributed mode on multiple GPUs. The code includes this comment regarding this:
"Only evaluate when single GPU otherwise metrics may not average well"

I'd appreciate it if someone could explain this issue in a few words and a maybe suggest a way around it? Can I simply run the evaluation on a single GPU (e.g. only for local_rank==0)?

Thanks!


Write With Transformer: PPLM document is stuck

The Uber PPLM on Write With Transformer does not generate anything, regardless of the parameters. It simply sits there, loading, forever.