Question on AllenNLP vocabulary and huggingface BERT out of sync

Perhaps this should also be posted in the allennlp repo.
I'm currently trying to use a pretrained model (clinicalBERT) with a different set of vocabulary with huggingface's BertModel. Even though the code runs, I'm not 100% convinced that the vocabulary index to weight mappings are synced between the allennlp vocabulalry object and the BertModel. The tokenizer used for clinicalBERT is scispacy, not wordpiece. First of all, are there any examples in this repo or other places online that does this (has a stack of allennlp vocab + huggingface BERT, and loading a pretrained BERT model with vocab.txt?).

This is what I do:
` model = pytorch_transformers.BertModel.from_pretrained("my/path/to/clinicalBERT", output_hidden_states=True)`
Then, for the vocabulary, I do:
    ` vocab = Vocabulary(counter=None, max_vocab_size=max_v_sizes) `
    ` vocab.set_from_file(filename="path/to/clinicalBERT/vocab.txt", is_padded=1, namespace="scispacy")`
I then use resize_tokens on the model, and use a TokenIndexer from allennlp and index my instances with that vocabulary. Even at this point though, ` vocab.get_token_from_index(0, "scispacy")`
 returns ` @@PADDING@@` , and ` @@UNKNOWN@@`  is at index 101 for the vocab (even though allennlp's vocabulary to my understanding sets ` @@UNKNOWN@@`  to 1).
What should I do to realign the Vocabulary back to the vocab index -> weight embedding mapping in the model?


Why the codes of training BERT from scratch are deprecated

## ❓ Questions & Help
I'm wondering why the team removed the codes of training BERT from scratch, including pregenerate_training_data.py and finetune_on_pregenerated.py. They're very helpful and I still continue developing them to train the BERT as well as Roberta from scratch.


When to support Albert?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Do you have any plan to support Google's new model-ALBERT?


Downloading model in distributed mode

## 🐛 Bug

When running in distributed mode with `n` processes, a new model will be download `n` times. I don't think that's what you want. I found [this related issue](https://github.com/huggingface/transformers/issues/44) but that only fixed the race condition. Downloads still happen in parallel. Is there a way to only download the model once? Perhaps by passing a `local_rank` parameter and only downloading when `local_rank==0`?

Especially for large models this is not ideal as i. they take up a lot of space (multiplied by the number of processes) ii. downloading is extra slow because it happens multiple times in parallel, limiting bandwidth.

```bash
15-Oct 03:08:45 - [INFO]: https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin not found in cache or force_download set to True, downloading to /tmp/tmp0amm9x2s
15-Oct 03:08:45 - [INFO]: https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin not found in cache or force_download set to True, downloading to /tmp/tmp7wpg48uj
15-Oct 03:08:45 - [INFO]: https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin not found in cache or force_download set to True, downloading to /tmp/tmp89svv055
15-Oct 03:08:45 - [INFO]: https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin not found in cache or force_download set to True, downloading to /tmp/tmp7yk94f8s
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6552025106/6552025106 [03:57<00:00, 27631147.05B/s]
15-Oct 03:12:42 - [INFO]: copying /tmp/tmp89svv055 to cache at /home/bram/.cache/torch/transformers/c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6552025106/6552025106 [03:57<00:00, 27614197.65B/s]
15-Oct 03:12:43 - [INFO]: copying /tmp/tmp7wpg48uj to cache at /home/bram/.cache/torch/transformers/c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6552025106/6552025106 [03:57<00:00, 27605553.23B/s]
15-Oct 03:12:43 - [INFO]: copying /tmp/tmp0amm9x2s to cache at /home/bram/.cache/torch/transformers/c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6552025106/6552025106 [03:57<00:00, 27599668.53B/s]
15-Oct 03:12:43 - [INFO]: copying /tmp/tmp7yk94f8s to cache at /home/bram/.cache/torch/transformers/c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
```

An alternative would be to already 'touch' the file in the .cache _before_ downloading, and when it exists, not initiate a new download. (Taking into account sudden abortions.)


Changelog

## 🚀 Add changelog between versions

New versions are pushed to PyPi at a steady pace, but it's not evident to find the changes that new versions bring. Is there a changelog anywhere? Something similar to a HISTORY file would be nice. I think it would definitely contribute to better documentation!


Accuracy drop in finetuning roBERTa

## ❓ How to achieve GLUE leaderboard acc for QQP task trained with roBERTa?

I am trying to finetune roberta-base model for Quora Question Pair task. In the [GLUE Leaderboard](url) the accuracy claimed is F1 / Accuracy is 74.3/90.2

I am training with the following command to fine tune the roberta-base model,

> CUDA_VISIBLE_DEVICES=0 python run_glue.py --data_dir /home/arjun/datasets/quora_roberta --model_type roberta --model_name_or_path /home/arjun/transformers/models/roberta --task_name qqp --output_dir /home/arjun/transformers/output_models/run-2/ --do_train --do_eval --do_lower_case --logging_steps 250 --save_steps 5000

All other params are default params.

I get the accuracy after 3 epochs as (taken from the eval_results.txt file),
acc = 0.6329982984448578
acc_and_f1 = 0.3164991492224289
f1 = 0.0

Also I ran evaluation for all the checkpoints stored, for every checkpoint I got the same accuracy which is quite confusing.

I'm I doing something wrong here? How to reproduce the accuracy mentioned in GLUE leaderboard?

Thanks in advance.


Predefined token classification

Hello,
I am just wondering if "BertForTokenClassification"  can be modified to classify predefined tokens (just targeted tokens). E.g. in NER it identifies the entity and then classify it, but in my case I want to classify the targeted tokens only in a sentence with a predefined labels. I thought of adding the targeted token in the sentence to the end of the sentence to let the model knows this is the targeted token but I am sure it is not a good idea because I need to again add a label for the added token. (silly Q I know :)
Any idea on how to approach it? I feel that NER task "BertForTokenClassification"
model can be modified to achieve it but do not know how.
Thanks


Unable to import TF models

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: Quick tour TF 2.0 training and PyTorch interoperability from github homepage

## To Reproduce

Steps to reproduce the behavior:

1. Install libraries (update tensorflow to 2.0.0)
```

!pip install tensorflow-gpu
!pip install torch
!pip install transformers

```

2. Run example
```import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

# Load the TensorFlow model in PyTorch for inspection
model.save_pretrained('./save/')
pytorch_model = BertForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = "This research was consistent with his findings."
sentence_1 = "His findings were compatible with this research."
sentence_2 = "His findings were not compatible with this research."
inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')

pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")

```

3. Get error

```
      5 # Load dataset, tokenizer, model from pretrained model/vocabulary
      6 tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
----> 7 model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
      8 data = tensorflow_datasets.load('glue/mrpc')
      9

NameError: name 'TFBertForSequenceClassification' is not defined
```

## Environment

Google collab

I get the same error when trying to use any TF version of the transformers.


Fused optimizer and gradient clipper using apex

Significant (40ms / iter for XLNet squad finetuning) performance increase.

Also adds fused grad clipping, gives further ~30ms / iter saving in the same XLNet squad case.

Redefines the `AdamW` implementation such that the existing code will be used if apex's multi_tensor_apply code isn't available, and should drop-in speedup all existing scripts using `AdamW`. Also abstracts the gradient clipping (in order to keep run scripts concise and move apex-specific logic into `optimizations.py`)


Main and train for CTRL model

## 🚀 Feature

I have seen the CTRL model has been added to the repo but I don't see any script to run or train it. Is this going to be added soon?


/pytorch/aten/src/THC/THCTensorScatterGather.cu:100: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [319,0,0] Assertion `indexValue >= 0 && indexValue < src.sizes[dim]` failed.

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I have no idea about this error。


Force einsum to run in fp16

As noted in the comments, this will force `torch.einsum` to run in fp16 for the squad finetuning task (it should be valid for other tasks, but I haven't verified that) when run with `--fp16_opt_level="O1"` which is the default.

Otherwise, `torch.einsum` is treated as a "promote" operation by `apex.amp`, and if any argument is fp32, all arguments will be cast to fp32, and the answer will return in fp32. This will happen at any point when a parameter is used (XLNet in particular suffers here). Given all uses I've seen for einsum are to express gemm, batched-gemm and transpose, operations we'd normally consider to be safe in fp16, this should be a safe change. From a performance standpoint it allows TensorCore usage which can significantly boost achieved performance.

This change doesn't affect accuracy in my testing, and gives ~20-25% higher throughput on XLNet-based finetuning.


Fix import error in script to convert faisreq roberta checkpoints

Fix ImportError in `convert_roberta_original_pytorch_checkpoint_to_pytorch.py`, see #1459.


Run squad with all model lq


CalledProcessError

I'm running
```
python /content/transformers/examples/run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=TRAIN_FILE \
    --do_eval \
    --eval_data_file=TEST_FILE
```
in my [Colab notebook](https://colab.research.google.com/drive/1T3fUHHWPAgWKEEITOKZJFGvNp9332RW3) and it returns this:
```
10/14/2019 03:30:53 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, 16-bits training: False
10/14/2019 03:30:53 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /root/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
10/14/2019 03:30:53 - INFO - transformers.configuration_utils -   Model config {
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

10/14/2019 03:30:53 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /root/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
10/14/2019 03:30:53 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /root/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
10/14/2019 03:30:54 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at /root/.cache/torch/transformers/4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
10/14/2019 03:30:58 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=1024, cache_dir='', config_name='', device=device(type='cpu'), do_eval=True, do_lower_case=False, do_train=True, eval_all_checkpoints=False, eval_data_file='TEST_FILE', evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, local_rank=-1, logging_steps=50, max_grad_norm=1.0, max_steps=-1, mlm=False, mlm_probability=0.15, model_name_or_path='gpt2', model_type='gpt2', n_gpu=0, no_cuda=False, num_train_epochs=1.0, output_dir='output', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=50, save_total_limit=None, seed=42, server_ip='', server_port='', tokenizer_name='', train_data_file='TRAIN_FILE', warmup_steps=0, weight_decay=0.0)
Traceback (most recent call last):
  File "/content/transformers/examples/run_lm_finetuning.py", line 543, in <module>
    main()
  File "/content/transformers/examples/run_lm_finetuning.py", line 490, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/content/transformers/examples/run_lm_finetuning.py", line 102, in load_and_cache_examples
    dataset = TextDataset(tokenizer, file_path=args.eval_data_file if evaluate else args.train_data_file, block_size=args.block_size)
  File "/content/transformers/examples/run_lm_finetuning.py", line 67, in __init__
    assert os.path.isfile(file_path)
AssertionError
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-21-2156f3b9e4fc> in <module>()
----> 1 get_ipython().run_cell_magic('shell', '', 'cd /content/transformers\nexport TRAIN_FILE=/content/wikitext-103-raw/wiki.train.raw\nexport TEST_FILE=/content/wikitext-103-raw/wiki.test.raw\n \npython /content/transformers/examples/run_lm_finetuning.py \\\n    --output_dir=output \\\n    --model_type=gpt2 \\\n    --model_name_or_path=gpt2 \\\n    --do_train \\\n    --train_data_file=TRAIN_FILE \\\n    --do_eval \\\n    --eval_data_file=TEST_FILE')

2 frames
/usr/local/lib/python3.6/dist-packages/google/colab/_system_commands.py in check_returncode(self)
    136     if self.returncode:
    137       raise subprocess.CalledProcessError(
--> 138           returncode=self.returncode, cmd=self.args, output=self.output)
    139
    140   def _repr_pretty_(self, p, cycle):  # pylint:disable=unused-argument

CalledProcessError: Command 'cd /content/transformers
export TRAIN_FILE=/content/wikitext-103-raw/wiki.train.raw
export TEST_FILE=/content/wikitext-103-raw/wiki.test.raw

python /content/transformers/examples/run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=TRAIN_FILE \
    --do_eval \
    --eval_data_file=TEST_FILE' returned non-zero exit status 1.
```


remove leftover usage of DUMMY_INPUTS

Hey @thomwolf
This change https://github.com/huggingface/transformers/commit/da26bae61b8c1e741fdc6735d46c61b43f649561#diff-8ddce309e88e8eb5b4d02228fd8881daL28 removed the constant `DUMMY_INPUTS`, but one usage of that constant remains in the code.

So any call to `load_tf2_checkpoint_in_pytorch_model` is currently throwing: `NameError: name 'DUMMY_INPUTS' is not defined`

```
/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_pytorch_utils.py in load_tf2_checkpoint_in_pytorch_model(pt_model, tf_checkpoint_path, tf_inputs, allow_missing_keys)
    199
    200     if tf_inputs is None:
--> 201         tf_inputs = tf.constant(DUMMY_INPUTS)
    202
    203     if tf_inputs is not None:

NameError: name 'DUMMY_INPUTS' is not defined
```


Added performance enhancements (XLA, AMP) to examples

Summary of changes

- Minor enhancements to `run_tf_glue.py` (e.g. calculate train/val steps from number of train/val examples, standardize quotes etc.)
- Added option for mixed precision (Automatic Mixed Precision / AMP) to run models on Tensor Cores (NVIDIA Volta/Turing GPUs) and future hardware
- Added option for XLA, which uses the XLA compiler to reduce model runtime
- Options are toggled using `USE_XLA` or `USE_AMP`

Quick benchmarks from the script (no other modifications):

| GPU    | Mode | Time (2nd epoch) | Val Acc (3 runs) |
| --------- | -------- | ----------------------- | ----------------------|
| Titan V | FP32 | 41s | 0.8438/0.8281/0.8333 |
| Titan V | AMP | 26s | 0.8281/0.8568/0.8411 |
| V100    | FP32 | 35s | 0.8646/0.8359/0.8464 |
| V100    | AMP | 22s | 0.8646/0.8385/0.8411 |
| 1080 Ti | FP32 | 55s | - |

Mixed precision (AMP) reduces the training time considerably for the same hardware and hyper-parameters (same batch size was used).

>**Important Note**
>
>Unrelated to this PR, but restoring the PyTorch model for the TF2 saved model does not work. This does not work in the original, unmodified example script. [Here](https://github.com/huggingface/transformers/blob/master/transformers/modeling_tf_pytorch_utils.py#L201) is the offending line in the Transformers library that references a uninitialized variable. This is fixed by PR #1509

Feedback and comments welcome!

Related: Issue #1441


GPU Usage?

**Question**
> Note the query/issue might not have anything to do with the library as such, just looking for info as to why it will happen. Thanks for understanding.

- Why would the GPU show a usage[verified using `nvidia-smi`] of 420MB/32GB when i import `transformers`?

Note this only happens when i have `tensorflow-gpu 2.0` version in the same enviornment, otherwise it just works normally.

Pytorch v1.3.0
Transformers v2.1.1
tf-gpu v2.0.0
apex 0.1
GPU Tesla V100-SXM2-32GB
NVIDIA-SMI 410.79
Driver Version: 410.79
CUDA Version: 10.0
Python 3.7.3

Thanks.

(Sorry for a vague title and a quey)

Extra Outputs when i import transformers

```
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
2019-10-12 18:33:00.840558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-10-12 18:33:00.881658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:b5:00.0
2019-10-12 18:33:00.881762: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881816: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881854: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881892: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881931: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881967: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.882003: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.882015: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-10-12 18:33:00.882328: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-10-12 18:33:00.909789: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400000000 Hz
2019-10-12 18:33:00.916559: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b9ec5e0b90 executing computations on platform Host. Devices:
2019-10-12 18:33:00.916618: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-10-12 18:33:01.606656: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b9ec644220 executing computations on platform CUDA. Devices:
2019-10-12 18:33:01.606754: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-10-12 18:33:01.607078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-12 18:33:01.607115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]
>>>
```


Seq2Seq model with HugginFace

Hi
I am looking for a Seq2Seq model which is based on HuggingFace BERT model, I know fairseq has some implementation, but they are generally to me not very clean or easy to use, and I am looking for some good implementation based on HuggingFace work, thanks a lot for your help


Fixed the sample code in the title 'Quick tour'.

The variable pretrained_weights was fixed to 'bert-base-uncased' to be used in each model to experiment.  Otherwise, the last value of this variable in the previous loop was unintentionally effective in this loop which was causing throwing error.


Fine-tuning with run_squad.py, Transformers 2.1.1 & PyTorch 1.3.0 Data Parallel Error

## 🐛 Bug

Error message when fine-tuning BERT or XLNet on SQuAD1.1 or 2.0  with dual 1080Ti GPUs:

_"RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1"_

Model I am using:  BERT & XLNet

Language I am using the model on:  English

The problem arise when using:
* [X] my own modified scripts:  example script file below which ran successfully under previous PyTorch, PyTorch-Transformers, & Transformers versions.

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name) SQuAD 1.1 & 2.0

## To Reproduce

One shell script (there are others) that had worked before:

SQUAD_DIR=/media/dn/dssd/nlp/squad1.1

python ./run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file=${SQUAD_DIR}/train-v1.1.json \
  --predict_file=${SQUAD_DIR}/dev-v1.1.json \
  --per_gpu_eval_batch_size=8 \
  --per_gpu_train_batch_size=8 \
  --gradient_accumulation_steps=1 \
  --learning_rate=3e-5 \
  --num_train_epochs=2 \
  --max_seq_length=384 \
  --doc_stride=128 \
   --save_steps=2000 \
  --output_dir=./runs/bert_base_squad1_dp_ft_3 \

## Environment

* OS:  Ubuntu 18.04, Linux kernel 4.15.0-65-generic
* Python version:  3.7.4
* PyTorch version:  1.3.0
* Transformers version:  2.1.1 built from latest source
* Using GPU?   NVIDIA 1080Ti x 2
* Distributed or parallel setup?  Data Parallel
* Any other relevant information:  Have had many successful SQuAD fine-tuning runs on PyTorch 1.2.0 with Pytorch-Transformers 1.2.0, maybe even Transformers 2.0.0, and Apex 0.1.  New environment built with the latest versions (Pytorch 1.3.0, Transformers 2.1.1) spawns data parallel related error above


What is the best way to handle sequences > max_len for tasks like abstract summarization?

What is the best way to handle situations where a sequence in your dataset exceeds the max length defined for a model?

For example, if I'm working on an abstract summarization task with a Bert model having a `max_position_embeddings=512` and tokenizer with `max_len=512`, how should I handle documents where the tokens to evaluate exceed 512?

Is there a recommended practice for this situation?

Thanks


the working example code to use BertForQuestionAnswering

so we can use the pre-trained and fine-tuned on SQUAD Bert model to get an answer from a text and a question, similar to the way the CoreML model BERTSQUADFP16.mlmodel is used in the iOS example [Finding Answers to Questions in a Text Document](https://developer.apple.com/documentation/coreml/finding_answers_to_questions_in_a_text_document)


Issue with XLNet pretrained model

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

## To Reproduce

I'm trying to train the XLNet dropping the last layer, I get error stating that

below is my code :

```
class xlmodel(nn.Module):
    def __init__(self, xlnetModell):
        super(xlmodel, self).__init__()
        self.xlnetfeatures = nn.Sequential(*list(xlnetModel.children())[:-1])
        self.concat = nn.Linear(786, 200)
        self.predict = nn.Linear(200,2)

    def forward(self, xlinput_ids, xlattention_mask, labels, xltoken_type_ids) :
        inputs = { 'input_ids' : xlinput_ids , 'attention_mask' : xlattention_mask, 'token_type_ids' : xltoken_type_ids }
        xlnet_output = self.xlnetfeatures(**inputs)
        xl = nn.functional.relu((xlnet_output))
        output = self.predict(xl)
        return output

pretrained_weights = 'xlnet-base-cased'
 xlnetmodel = XLNetForSequenceClassification.from_pretrained(pretrained_weights, num_labels=2)
model(xlnetmodel)
    for _ in trange(num_train_epochs, desc="Epochs"):
        ep_tr_loss, nb_tr_steps, eval_accuracy = 0, 0, 0
        for step, batch in enumerate(train_data):
            model.train()
            batch = tuple(t.to(device) for t in batch)
            # model.
            inputs = {'xlinput_ids': batch[0], 'xlattention_mask': batch[1], 'labels': batch[3],
                      'xltoken_type_ids': batch[2] }
            optimizer.zero_grad()
            output = model(**inputs) -----> this is where error occurs
```
error stack :

```
    xlnet_output = self.xlnetfeatures(**inputs)
  File "E:\PycharmProjects\CommonSense\venv\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'input_ids'
Epochs:   0%|          | 0/10 [00:10<?, ?it/s]

```

I even tried not passing the inputs as dictionary but still i get this error.
I verified the input variable name in pytorch xlnet transformer it has input_ids
Any lead will be appreciated thanks.

But if I try to run xlnet as it is works fine.

## Environment

* OS: Windows 10
* Python version: 3.7
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): master
* Using GPU ? YEs
* Distributed of parallel setup ? No
* Any other relevant information:


How to load a different domain BERT-based pre-trained model?

I am trying to load the pre-trained model at pred/FinBERT-Pre2K_128MSL-500K  [FinBERT](https://github.com/psnonis/FinBERT)  and trying to run the basic task of SST-2 (sentiment classification) using run_glue.py (https://huggingface.co/transformers/examples.html#glue).

But I run into the following error:
OSError: Model name '/data/ftm/xgb_regr/FinBERT/pred/FinBERT-Pre2K_128MSL-250K' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed '/data/ftm/xgb_regr/FinBERT/pred/FinBERT-Pre2K_128MSL-250K' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

Also, since this seems to be trained using TF, I was wondering if I can use PyTorch to load it.

Thanks.


model.to(args.device) in run_glue.py taking around 10 minutes. Is this normal?

## ❓ Questions & Help

Currently line 484 of run_glue.py `model.to(args.device)` is taking close to 10 minutes to complete when loading the bert-base pretrained model. This seems like a long time compared to what I was seeing in pytorch-transformers.

My configuration:
    Tesla V100 - Driver 418.87.00
    Cuda toolkit 10.1
    PyTorch 1.3.0

The code I am running is:
`python example/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $(MY TASK) \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $(MY_DIR) \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=64   \
    --per_gpu_train_batch_size=64   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir $(MY_OUTDIR) \
    --overwrite_output_dir \
    --fp16`

Is this behavior expected or am I doing something wrong? Thanks!


Merge pull request #1 from huggingface/master

from 1.0->1.1


from 1.0->1.1


FR: Tokenizer function that can handle arbitrary number of sequences

## 🚀 Feature

Currently Tokenizers only support 1 or 2 sequences being added together, and them being concatenated with the appropriate SEP and CLS tokens for each model. My use case requires more sequences being added together, all separated by SEP tokens and having one CLS token at the start (or end for XLNet) of the entire sequence.

E.g., for BERT:
```
[CLS] This is my first sentence [SEP] This is my second sentence [SEP] And finally my third sentence [SEP]
```

Would it be possible to have a function supported for current and future models that can take a list of strings in and process them as outlined above.

I would be especially happy if there was an accompanying feature for token type ids, which would alternate between successive sequences.
Using the above example, this would mean:
```
0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0
```

Thanks!


Add new BERT models for German (cased and uncased)

Hi,

this PR adds new BERT models for German (both cased and uncased) from @dbmdz.

Details can be found in [this repository](https://github.com/dbmdz/german-bert).

Tasks:

* [x] Models are stored on S3, only permissions need to be adjusted by @julien-c


## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I tried a small chunk of code from the Readme.md

```
import torch
from transformers import *
MODELS = [(BertModel,       BertTokenizer,       'bert-base-uncased')]
for model_class, tokenizer_class, pretrained_weights in MODELS:
    # Load pretrained model/tokenizer
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights)
    input_ids = torch.tensor([tokenizer.encode("Here is some text to encode", add_special_tokens=True)])  # Add special tokens takes care of adding [CLS], [SEP], <s>... tokens in the right way for each model.
    with torch.no_grad():
        last_hidden_states = model(input_ids)[0]

```

It is giving me the following error

```
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-6528fe9b0472> in <module>
      3     tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
----> 4     model = model_class.from_pretrained(pretrained_weights)

~/.conda/envs/transformers/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    343
    344         if state_dict is None and not from_tf:
--> 345             state_dict = torch.load(resolved_archive_file, map_location='cpu')
    346
    347         missing_keys = []

~/.conda/envs/transformers/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    424         if sys.version_info >= (3, 0) and 'encoding' not in pickle_load_args.keys():
    425             pickle_load_args['encoding'] = 'utf-8'
--> 426         return _load(f, map_location, pickle_module, **pickle_load_args)
    427     finally:
    428         if new_fd:

~/.conda/envs/transformers/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    618     for key in deserialized_storage_keys:
    619         assert key in deserialized_objects
--> 620         deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
    621         if offset is not None:
    622             offset = f.tell()

RuntimeError: unexpected EOF, expected 7491165 more bytes. The file might be corrupted.
```

Haven't modified anything in the library.


Is encode_plus supposed to pad to max_length?

## ❓ Questions & Help

I am using AutoTokenizer and AutoModelForSequenceClassification and `encode_plus` to encode text. I am calling it like this:

`

            tokenizer = AutoTokenizer.from_pretrained(self.model_name)
            encoded_inputs = tokenizer.encode_plus(text,add_special_tokens=True,max_length=max_seq_length)
            input_ids = encoded_inputs["input_ids"]
            special_tokens_mask = encoded_inputs["special_tokens_mask"]
            token_type_ids = encoded_inputs["token_type_ids"]

            loggerinfo(logger, "len of encoded vals {} {} {}".format(len(input_ids),len(special_tokens_mask),len(token_type_ids)))
`

The output indicates that the encoded values are of different length. I expected them to all be = max_length which is 100 in this case.

Output:

> max seq len = 100
> len of encoded vals 39 39 39
> len of encoded vals 24 24 24
> len of encoded vals 11 11 11

Is that an incorrect expectation?


Excessively Long text_b Raises Unnecessary Warnings in `encode_plus`

In `encode_plus`, `convert_ids_to_tokens` is called before truncating to `max_len`. However, if either text_a or text_b are longer than `max_len`, `convert_ids_to_tokens` will raise a warning. Since sequences are truncated to the right length afterwards in `encode_plus`, this warning is unnecessary.


GLUE on TPU

This takes advantage of the pytorch 1.3 XLA implementation to fine-tune GLUE on a TPU.

MRPC fine-tuning in 3 epochs + evaluation takes a total of 6 minutes and 30 seconds.


convert int to str before adding to a str


Can you please share the pre-processed text dump of the bookcorpus and wikipediacorpus?

## ❓ Questions & Help
I am trying to train distilbert with different architecture. If you can share the text dump for the pre-training, it would be great. Thanks!
<!-- A clear and concise description of the question. -->


improve final answer extraction in utils_squad.py

Shouldn't ` get_final_text` use the specific optionally pre-trained tokenizer instead of generically using `BasicTokenizer` ?

[examples/utils_squad.py L911](https://github.com/huggingface/transformers/blob/6596e3d56626c921b3920e313866b7412633b91a/examples/utils_squad.py#L911)


Error while fine-tuning model for GPT2

## 🐛 Bug

<!-- Important information -->

Model I am using GPT2:

Language I am using the model on English:

The problem arise when using:
* [ ] the official example scripts: I run the run_lm_finetuning.py script from the examples folder

The tasks I am working on is:
* [ ] my own task or dataset: The WritingPrompts dataset.

## To Reproduce

Steps to reproduce the behavior:

1. Running the run_lm_generation.py on the WP dataset.
 python run_lm_finetuning.py --output_dir=output_ft_gpt2 --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_data_file=../stories_data/writingPrompts/train.wp_target --do_eval --eval_data_file=../stories_data/writingPrompts/test.wp_target --block_size=128 --save_total_limit=100

Error: Traceback (most recent call last):
  File "run_lm_finetuning.py", line 538, in <module>
    main()
  File "run_lm_finetuning.py", line 485, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 97, in load_and_cache_examples
    dataset = TextDataset(tokenizer, file_path=args.eval_data_file if evaluate else args.train_data_file, block_size=args.block_size)
  File "run_lm_finetuning.py", line 80, in __init__
    self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i:i+block_size]))
AttributeError: 'GPT2Tokenizer' object has no attribute 'build_inputs_with_special_tokens'


Create new


Integration of TF 2.0 models with other Keras modules

Add tests that TF 2.0 models can be integrated with other Keras modules.

Add more serialization tests for TF 2.0 and PyTorch models.

Fix TFSequenceSummary head and RoBERTa.


Does run_lm_finetuning.py finetune the entire BERT / Xlnet architecture

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
1. When finetuning on data without a task, i.e. **unsupervised finetuning** by running the **run_lm_finetuning.py** script, does the code finetune all the weight layers of the model or just finetunes by adding an extra layer over the top of the architecture?

2. The correct way to do this is to first finetune the top n layers for some epochs and then finetune all the layers. Is this how the finetuning takes place through run_lm_finetuning.py ?
I think answer is no.

I went through the run_lm_finetuning.py code and it seems that the entire model is getting finetuned from the start.

Still having doubt on this as I have little experience with pytorch.


Fixing CTRL tokenizer - Update error messages - XLM-MLM in run_generation

# CTRL tokenizer

We are trying to find a good full-python replacement for the fastBPE tokenizer originally used for CTRL.

We don't really want to depend on fastBPE, even though it's fast, because it's a cython package which means we may then have installation issues on specific platforms like Windows.

Current options are:
- test our own Bert whitespace tokenizer
- uses Moses which is already included (as sacredmoses) for XLM
- using a regex like GPT-2.

Currently favored option is sacredmoses.

cc @LysandreJik @keskarnitish @stefan-it

[UPDATE]: Updating this, upon deeper inspection, fastBPE tokenizer just basically [split on spaces only](https://github.com/glample/fastBPE/blob/master/fastBPE/fastBPE.hpp?fbclid=IwAR1Vp2WMLxDjpmBIpU6mkddeyxzi2vpvHOcm8fL4iaWL1m3tVbSfz-yZAcE#L652).

This tokenizer was used in CTRL which is confirmed by the fact that many vocabulary tokens in CTRL vocabulary contains end or start punctuation (see CTRL vocabulary [here](https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json)).

So the most logical solution is thus just to split on spaces which is also the easiest solution :-)

# Error messages

Improved error messages of `from_pretrained` when files are not found

# XLM MLM in run_generation

Add support for XLM MLM models in run_generation (though these models are not really intended for that anyway).


How can I get the transformers' parameters?

## ❓ Questions & Help
Hi, I am new to transformers. Does this library offer an interface to compute the total number of different model's parameters?


bert-large-uncased-whole-word-masking-finetuned-squad or BertForQuestionAnswering?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I'm trying to use the pre-trained model bert-large-uncased-whole-word-masking-finetuned-squad to get answer to a question from a text, and I'm able to run:
```
model = BertModel.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
model.eval()
```
but what should I do next? There's some example code using `BertForQuestionAnswering `:
```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
start_positions = torch.tensor([1])
end_positions = torch.tensor([3])
outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions)
loss, start_scores, end_scores = outputs[:2]
```
But when I try the code above, I get the following error:
```
I1009 23:26:51.743415 4495961408 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at /Users/ailabby/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
I1009 23:26:54.848274 4495961408 modeling_utils.py:405] Weights of BertForQuestionAnswering not initialized from pretrained model: ['qa_outputs.weight', 'qa_outputs.bias']
I1009 23:26:54.848431 4495961408 modeling_utils.py:408] Weights from pretrained model not used in BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-48-0738102265a4> in <module>
      5 end_positions = torch.tensor([3])
      6 outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions)
----> 7 loss, start_scores, end_scores = outputs[:2]

ValueError: not enough values to unpack (expected 3, got 2)
```

Should I use the pre-trained model bert-large-uncased-whole-word-masking-finetuned-squad or the BertForQuestionAnswering class, or both, to input a text and question and get an answer? Thanks for the help!


Much slower for inference, even when traced?

## ❓ Questions & Help

When running inference using BERT-large on a T4 GPU using bert-as-a-service, I could get well over 100/s on sentence pair classification. (I am aware that this utilized TF's graph freezing and pruning)

When running inference with Roberta-large on a T4 GPU using native pytorch and fairseq, I was able to get 70-80/s for inference on sentence pairs.

Even with using the torchscript JIT tracing, **I still am only able to get 17/s on a T4** using the transformers implementation of Bert-large, using a batch size of 8 (which fills most of the memory).

The training performance is similarly worse (about 40% - 100% longer even with apex vs no apex before).

One of the primary differences I can think of is that now I am padding all up to max-seq length, and it does increase performance a lot to decrease this. Is there a way to not pad in transformers? And just pass a list of pytorch tensors in that can be dynamically sized?

Should I try the tensorflow implementations?

Thank you!


RuntimeError: Error(s) in loading state_dict for BertModel:

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello, I need to use "py torch_model.bin" in a model. I used your "convert_bert_original_tf_checkpoint_to_pytorch.py" to generate bin file, but I used "model_bert.load_state_dict (torch.load (init_checkpoint, map_location='cpu') to load it in my code, except for the following mistakes:

Traceback (most recent call last):
File "train.py", line 579, in <module>
Model, model_bert, tokenizer, bert_config = get_models (args, BERT_PT_PATH)
File "train.py", line 157, in get_models
Args.no_pretraining)
File "train.py", line 125, in get_bert
Model_bert. load_state_dict (torch. load (init_checkpoint, map_location='cpu'))
File "/ home/ubuntu/anaconda3/lib/python 3.7/site-packages/torch/nn/modules/module.py", line 777, in load_state_dict
Self. class. Name, "n T. join (error_msgs)"
Runtime Error: Error (s) in loading state_dict for BertModel:
Missing key (s) in state_dict: "embeddings. word_embeddings. weight", "embeddings. position_embeddings. weight".
thanks


data loader for varying length input

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->