Implemented lazy line-by-line text data set loading for LM example script

See PR #3388. Master changed substantially, requiring relocation of code into previously untouched files etc. Instead, here is a new PR using the same code but refactored to fit in to the new more modular structure of the scripts in `examples`.


camembert-base-fquad

model card

Related to pretrained model cards

Model card for illuin release of camembert-base-fquad


Create model card

Related to pretrained model cards


Fast Tokenizers do not work when `return_offsets_mapping=True, return_tensors="pt"`

# 🐛 Bug

## Information

When I try the following code:

```python
from transformers import BertTokenizerFast
fast = BertTokenizerFast.from_pretrained("bert-base-cased")
fast.encode_plus("Hello I am tokenizing", return_offsets_mapping=True)
```

It works as intended.

However, if I try:

```python
from transformers import BertTokenizerFast
fast = BertTokenizerFast.from_pretrained("bert-base-cased")
fast.encode_plus("Hello I am tokenizing", return_offsets_mapping=True, return_tensors="pt")
```

It throws the following error:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-c6ee501a801c> in <module>
      1 from transformers import BertTokenizer, BertTokenizerFast
      2 fast = BertTokenizerFast.from_pretrained("bert-base-cased")
----> 3 fast.encode_plus("Hello I am tokenizing", return_offsets_mapping=True, return_tensors="pt")

~/.conda/envs/bertology/lib/python3.7/site-packages/transformers/tokenization_utils.py in encode_plus(self, text, text_pair, add_special_tokens, max_length, pad_to_max_length, stride, truncation_strategy, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, **kwargs)
   1972             return_offsets_mapping=return_offsets_mapping,
   1973             pad_to_max_length=pad_to_max_length,
-> 1974             **kwargs,
   1975         )
   1976

~/.conda/envs/bertology/lib/python3.7/site-packages/transformers/tokenization_utils.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, max_length, stride, truncation_strategy, pad_to_max_length, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, **kwargs)
   1926                 stack = tf.stack(stack, axis=0)
   1927             elif return_tensors == "pt":
-> 1928                 stack = torch.stack(stack, dim=0)
   1929             elif not return_tensors and len(stack) == 1:
   1930                 stack = stack[0]

TypeError: expected Tensor as element 0 in argument 0, but got list
```

## Environment info

- `transformers` version: 2.8.0
- Platform: Linux-4.4.0-174-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)


ALBERT with Masked Language Model Input Processing from SavedModel

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ALBERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
```
input_ids=tf.convert_to_tensor(input_ids,name='input_ids',dtype=tf.int32)
attention_mask=tf.convert_to_tensor(attention_mask,name='attention_mask',dtype=tf.int32)
position_ids=tf.convert_to_tensor(position,name='position_ids',dtype=tf.int32)
token_type_ids=tf.convert_to_tensor(type_emb,name='token_type_ids',dtype=tf.int32)
inputs={'input_ids': input_ids, 'position_ids': position_ids, 'token_type_ids': token_type_ids, 'attention_mask': attention_mask}
print(inputs)
outputs=model(inputs,training=False)
```
The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Set the inputs
2. Run the above script

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
ValueError: Could not find matching function to call loaded from the SavedModel. Got:
  Positional arguments (1 total):
    * {'input_ids': <tf.Tensor 'inputs_1:0' shape=(512,) dtype=int32>, 'position_ids': <tf.Tensor 'inputs_2:0' shape=(512,) dtype=int32>, 'token_type_ids': <tf.Tensor 'inputs_3:0' shape=(512,) dtype=int32>, 'attention_mask': <tf.Tensor 'inputs:0' shape=(512,) dtype=int32>}
  Keyword arguments: {'training': False}

Expected these arguments to match one of the following 4 option(s):

Option 1:
  Positional arguments (1 total):
    * {'position_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='position_ids'), 'token_type_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='token_type_ids'), 'attention_mask': TensorSpec(shape=(None, 512), dtype=tf.int32, name='attention_mask'), 'input_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_ids')}
  Keyword arguments: {'training': True}

Option 2:
  Positional arguments (1 total):
    * {'position_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/position_ids'), 'token_type_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/token_type_ids'), 'attention_mask': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/attention_mask'), 'input_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/input_ids')}
  Keyword arguments: {'training': True}

Option 3:
  Positional arguments (1 total):
    * {'input_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_ids'), 'position_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='position_ids'), 'token_type_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='token_type_ids'), 'attention_mask': TensorSpec(shape=(None, 512), dtype=tf.int32, name='attention_mask')}
  Keyword arguments: {'training': False}

Option 4:
  Positional arguments (1 total):
    * {'position_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/position_ids'), 'token_type_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/token_type_ids'), 'attention_mask': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/attention_mask'), 'input_ids': TensorSpec(shape=(None, 512), dtype=tf.int32, name='inputs/input_ids')}
  Keyword arguments: {'training': False}
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The model makes prediction on the inputs

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Ubuntu 16
- Python version: 3.7.7
- Tensorflow version (GPU?): 2.1.0 GPU
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Maybe it is a bug for Roberta vocab file.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [1 ] the official example scripts: (give details below)
* [1] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [1 ] my own task or dataset: (give details below)

Hi Huggingface team,

When I use Roberta, I found that there are so many words start with "Ġ" in the vocab file you provided. I just want to make sure whether it is a bug or not. Around 40,771 words start with "Ġ".

Roberta-base vocab file: https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json

Roberta-large vocab file:
https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-vocab.json

Best Regards,
Chen


🌟 Transformer Lite

# 🌟 New model addition

## Model description

Part of the abstract :

> In this paper, we present an efficient mobile NLP architecture, Lite Transformer to
facilitate deploying mobile NLP applications on edge devices. [...] brings consistent improvement over the vanilla transformer on three well-established language
tasks: machine translation, abstractive summarization, and language modeling. [...] Notably, Lite Transformer outperforms the AutoML-based Evolved
Transformer by 0.5 higher BLEU for the mobile NLP setting without the costly
architecture search that requires more than 250 GPU years

## Open source status

* [x] the model implementation is available: [Repository](https://github.com/mit-han-lab/lite-transformer)
* [x] the model weights are available: [In the README](https://github.com/mit-han-lab/lite-transformer#models)
* [x] who are the authors: @Michaelvll


TF BART ?

TensorFlow

Anything TensorFlow

# 🌟 New model addition

## Model description

BART is currently only available in Pytorch.

**Are you planning to release a TF version ?**


Adding --do_lower_case option

In SQuAD 1.1 using the uncased BERT model, **--do_lower_case** option must be used for initialization of a tokenizer.
Omitting this optional argument, the performance in development set would be lower around 8-9pt in both EM and F1 (In the official setting described in readme, the actual metrics were 72.09 (EM)/81.84 (F1) we confirmed).


How do I convert T5 checkpoint to pytorch bin to utilize fine-tuned model from finetune.py?

# ❓ Questions & Help

I've recently tried to fine-tune T5 using bart's [finetune.py](https://github.com/huggingface/transformers/blob/master/examples/summarization/bart/finetune.py) (supports T5 fine-tuning as well). The model is saved in a `ckpt` format when I wanted it as a pytorch `bin`, so I found this [page](https://huggingface.co/transformers/converting_tensorflow_models.html) in the docs on how to convert `ckpt` to `bin`. However, I don't see the T5 model in this page.

Is it possible to use the CLI to convert T5 models?

I also have questions around fine-tuning T5 but will discuss details in a separate issue.


Question about the output linear weight

I notice that the output linear weight for BertForMaskedLM is the same as input embedding. I don't read anywhere in the original paper that the output projection is tied with word embedding? Can you please explain the design choice here?

To reproduce:
```
m = transformers.BertForMaskedLM.from_pretrained('bert-base-uncased')
m.get_input_embeddings().weight - m.get_output_embeddings().weight
```


Toy size models versions for faster experimentation and testing

# 🚀 Feature request

To have a small(toy) in terms of number of parameters versions of models e.g.:
- `gpt2-toy`
- `bert-toy-uncased`
etc.
Alternative:  to have separate model type `toy-transformer`.

## Motivation

Working on the project and using transformers package our team noticed that it's difficult for some team members who had machines without GPU to experiment with code working with GPT2 model and to run tests for it. Also, it required significant resources on CI environment to run tests.

We've ended up by implementing a custom configuration for GPT2 model with a resulting model size ~10M parameters for local experimentation and testing.

I think it might be beneficial for the community to have tiny model versions in the package that weakens hardware requirements to start work and to run tests.
If it's unreasonable to implement tiny version for every model - having separate model type which implements same interfaces and have corresponding heads will also help to reach the same objective.

## Your contribution

If either of the options sounds reasonable I can participate in the implementation.


[DialoGPT] add dialogpt training tips


[model_cards] Add model card for Hindi-Bert

model card

Related to pretrained model cards

Trained with Google ELECTRA and Hindi Corpus (OSCAR CommonCrawl and latest Hindi Wikipedia)

README offers more details and links to notebooks showing pretraining and finetuning on movie reviews


Allow a more backward compatible behavior of max_len_single_sentence and max_len_sentences_pair

Since #3706, `max_len_single_sentence` and `max_len_sentences_pair` are now automatically setup in the base class.

This PR allows the user to try to setup `max_len_single_sentence` and `max_len_sentences_pair`. If the value if the same as the pre-computed, we display a deprecation warning (avoid breaking old code in this case). If the value is different we raise an explicit error message.

cc @HendrikStrobelt


[Generation] Generation should allow to start with empty prompt

This PR allows generation from an empty input.


RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1549287501208/work/aten/src/THC/THCTensorRandom.cu:35

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BertForSequenceClassification

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I am performing multi-class (50 classes) classification on the Project Gutenberg dataset. The max text length is 1620 so I set max length at 2048. I'm also padding the text with 0s.

```
MAX_LEN = 2048
def get_encodings(texts):
    token_ids = []
    attention_masks = []
    for text in texts:
        token_id = tokenizer.encode(text, add_special_tokens=True, max_length=2048)
        token_ids.append(token_id)
    return token_ids

def pad_encodings(encodings):
    return pad_sequences(encodings, maxlen=MAX_LEN, dtype="long",
                          value=0, truncating="post", padding="post")

def get_attention_masks(padded_encodings):
    attention_masks = []
    for encoding in padded_encodings:
        attention_mask = [int(token_id > 0) for token_id in encoding]
        attention_masks.append(attention_mask)
    return attention_masks

X_train = torch.tensor(train_encodings)
y_train = torch.tensor(train_df.author_id.values)
train_masks = torch.tensor(train_attention_masks)

X_test = torch.tensor(test_encodings)
y_test = torch.tensor(test_df.author_id.values)
test_masks = torch.tensor(test_attention_masks)

batch_size = 32

# Create the DataLoader for our training set.
train_data = TensorDataset(X_train, train_masks, y_train)
train_sampler = RandomSampler(train_data)
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

validation_data = TensorDataset(X_test, test_masks, y_test)
validation_sampler = SequentialSampler(validation_data)
validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)
```

My model is setup like so:

```
model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels = 50,
    output_attentions = False,
    output_hidden_states = False,
)
```

During training however, the line indicated below is throwing the error:

```
for step, batch in enumerate(train_dataloader):

        b_texts = batch[0].to(device)
        b_attention_masks = batch[1].to(device)
        b_authors = batch[2].to(device) <---- ERROR
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Ubuntu 16.04
- Python version: 1.0.0
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.1.0
- Using GPU in script?:  Yes. P4000
- Using distributed or parallel set-up in script?: No


Fix the typos

Fix the typos of "[0.0, 1.0[" --> "[0.0, 1.0]".


Language generation not possible with roberta?

Hi,

I have a small question.
I finetuned a roberta model and then I was about to do generation and I understood this was not possible on my model, but only in a smaller collection of model, where most of them are quite huge.
Why is that? Is there another way to generate language with roberta?

Thanks in a advance!

PS: the scripts I used were: `run_language_modeling.py` and `run_generation.py`


Model cards for KoELECTRA

Related to pretrained model cards

Hi:)

I've recently uploaded `KoELECTRA`, pretrained ELECTRA model for Korean:)
Thanks for supporting this great library and S3 storage:)


[Trainer] Add more schedules to trainer

Adds all available schedules to trainer and also add beta1 and beta2 for Adam to args.
@julien-c


fix output_dir / tokenizer_name confusion

Fixing #3950

It seems that `output_dir` has been used in place of `tokenizer_name` in `run_xnli.py`. I have corrected this typo.

I am not that familiar with code style in this repo, in a lot of instances I used:

`args.tokenizer_name if args.tokenizer_name else args.model_name_or_path`

Instead, I believe I could have set `args.tokenizer_name = args.tokenizer_name if args.tokenizer_name else args.model_name_or_path` somewhere at the start of the script, so that the code could be cleaner below.


Run Multiple Choice failure with ImportError: cannot import name 'AutoModelForMultipleChoice'

# 🐛 Bug

## Information
Run the example, it returns error as below. The version in April 17 doesn't have the problem.
python ./examples/run_multiple_choice.py \
> --task_name swag \
> --model_name_or_path roberta-base \
> --do_train \
> --do_eval \
> --data_dir $SWAG_DIR \
> --learning_rate 5e-5 \
> --num_train_epochs 3 \
> --max_seq_length 80 \
> --output_dir models_bert/swag_base \
> --per_gpu_eval_batch_size=16 \
> --per_gpu_train_batch_size=16 \
> --gradient_accumulation_steps 2 \
> --overwrite_output
Traceback (most recent call last):
  File "./examples/run_multiple_choice.py", line 26, in <module>
    from transformers import (
ImportError: cannot import name 'AutoModelForMultipleChoice'

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
Follow the exact steps in README.md
1. Download the transformer with git clone
2. git clone https://github.com/rowanz/swagaf.git
3. export SWAG_DIR=/path/to/swag_data_dir
python ./examples/run_multiple_choice.py \
--task_name swag \
--model_name_or_path roberta-base \
--do_train \
--do_eval \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir models_bert/swag_base \
--per_gpu_eval_batch_size=16 \
--per_gpu_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
***** Eval results *****
eval_acc = 0.8338998300509847
eval_loss = 0.44457291918821606

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: ubuntu 18.04
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0 GPU
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No.


Using the T5 model with huggingface's mask-fill pipeline

Does anyone know if it is possible to use the T5 model with hugging face's mask-fill pipeline? The below is how you can do it using the default model but i can't seem to figure out how to do is using the T5 model specifically?

```
from transformers import pipeline
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in ' + nlp_fill.tokenizer.mask_token)

```

Trying this for example raises the error "TypeError: must be str, not NoneType" because nlp_fill.tokenizer.mask_token is None.

```
nlp_fill = pipeline('fill-mask',model="t5-base", tokenizer="t5-base")
nlp_fill('Hugging Face is a French company based in ' + nlp_fill.tokenizer.mask_token)
```

Stack overflow [question](https://stackoverflow.com/questions/61408753/using-the-t5-model-with-huggingfaces-mask-fill-pipeline)


why the accuracy is very low when we test on sentences one by one of the words input?

# ❓ Questions & Help
hi, friends:
there is a problem , that we trained a good lm model on gpt2 and sequence len is 30,when we batch test in training with 30 seqlen, the accuracy can reach 90%. But when we test like this:
[1]->[2],[1,2]->[3],[1,2,3]-->[4], left is the input, and we hope it can predict the right, we found the accurcy is very low which is only 6%.


MLM Loss not decreasing when pretraining Bert from scratch

I want to pretrain bert base model from scratch with en-wikipedia dataset, since I havn't found a bookcorpus copy. The code I used was adapted from pytorch-pretrained-bert and Nvidia Megatron-LM. I can finetune BERT on SQuAD and get SOTA result. But for pretraining, the MLM loss stays around 7.3. I have no idea why this happens. Can someone offer some possible solutions? Great thanks!


Can't install transformers from sources using poetry

# 🐛 Bug

When installing transformers from sources using poetry

```bash
poetry add git+https://github.com/huggingface/transformers.git
```

The following exception is thrown:

```bash
[InvalidRequirement]
Invalid requirement, parse error at "'extra =='"
```

This is a problem on poetry side. The issue here is just to cross-link the problem.

Ref: https://github.com/python-poetry/poetry/issues/2326


Improve split on token

This patch makes the code shorter, also solves a bug that is suppressed already[1]

old code:
split_on_token(tok = '[MASK]', text='')
> ['[MASK]']

This is a bug, because split_on_token shouldn't add token when the input doesn't have the token

[1] 21451ec (handle string with only whitespaces as empty, 2019-12-06)


When I run `import transformers` , it reports an error. But I had no problem with pyrotch-transformers before. Is it a TensorRT problem?

# ❓ Questions & Help

when i run `import transformers`, i get this:

2020-04-26 17:02:59.210145: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
*** Error in `/conda-torch/bin/python': double free or corruption (!prev): 0x00007faa0eac23c0 ***

is there anything can fix it?


Add modelcard for Hate-speech-CNERG/dehatebert-mono-arabic model

Related to pretrained model cards


Fix t5 doc typos

Read through the docs for T5 at `docs/source/model_doc/t5.rst` and found some typos. Hope this helps!


xlnet large

If you train in batch 8 at max length = 512 with the xlnet-base-cased, you will learn well with changes in values such as Loss, Accuracy, Precision, Recall, and F1-score.

However, if you train with batch 2 at max length = 512 with a xlnet-large-cased, even if you run several times, there is almost no change, and precision is always 1.00000, and other values are only fixed.

Is it because the batch size is too small? Help.

[epoch1]
Tr Loss | Vld Acc | Vld Loss | Vld Prec | Vld Reca | Vld F1
0.39480 | 0.87500 | 0.39765 | 1.00000 | 0.87500 | 0.93333

[epoch2]
  Tr Loss | Vld Acc | Vld Loss | Vld Prec | Vld Reca | Vld F1
0.39772 | 0.87500 | 0.38215 | 1.00000 | 0.87500 | 0.93333


Fixed Style Inconsistency

This fixes a style inconsistency in BertForSequenceClassification's constructor where 'config' is referenced using both the parameter 'config' and 'self.config' on the same line.

This line is the only instance in modeling_bert.py where the config parameter is referenced using self.config in a constructor.


Added support for pathlib.Path objects instead of string paths in from_pretrained() (resolves #3962)

Resolves issue/feature request #3962

This only covers string paths in `from_pretrained` (so vocab_file paths for` __init__` for tokenizers aren't covered). This may or may not be a feature that is desired (since it's fairly easy for the client to convert a path-like to a string). With the changes, you would be able to use `from_pretrained` with path-likes:

```
from pathlib import Path
from transformers import AutoModel, AutoTokenizer

my_path = Path("path/to/model_dir/")
AutoTokenizer.from_pretrained(my_path)
AutoModel.from_pretrained(my_path)
```


Weights from pretrained model not used in GPT2LMHeadModel

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): gpt2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. trained gpt2 using run_language_modeling
2. using run_generation.py

`backend_1        | 04/25/2020 18:37:27 - INFO - transformers.modeling_utils -   Weights from pretrained model not u
sed in GPT2LMHeadModel: ['transformer.h.0.attn.masked_bias', 'transformer.h.1.attn.masked_bias', 'transformer.h.2.a
ttn.masked_bias', 'transformer.h.3.attn.masked_bias', 'transformer.h.4.attn.masked_bias', 'transformer.h.5.attn.mas
ked_bias', 'transformer.h.6.attn.masked_bias', 'transformer.h.7.attn.masked_bias', 'transformer.h.8.attn.masked_bia
s', 'transformer.h.9.attn.masked_bias', 'transformer.h.10.attn.masked_bias', 'transformer.h.11.attn.masked_bias']`


Pytorch 1.5.0

PyTorch 1.5.0 doesn't allow specifying a standard deviation of 0 in normal distributions. We were testing that our models initialized with a normal distribution with a mean and a standard deviation of 0 had all their parameters initialized to 0.

This allows us to verify that all weights in the model are indeed initialized according to the specified weights initializations.

We now specify a tiny value (1e10) and round to the 9th decimal so that all these values are set to 0.

closes #3947
closes #3872


Sized Fill-in-the-blank or Multi Mask filling with T5

In the Google T5 paper they mentioned :
 For example, with the input, “I love peanut butter and _4_ sandwiches,” the outputs looked like:
I love peanut butter and jelly, which is what makes good sandwiches.

How do I achieve Multi Mask filling with T5 in hugging face transformers?
Code samples please :)
![image](https://user-images.githubusercontent.com/1754080/80286349-98d96300-8748-11ea-9e68-573f4f3a8e61.png)


Problem with downloading the XLNetSequenceClassification pretrained xlnet-large-cased

 I've run this before and it was fine. However, this time, i keep encountering this runtime error. In fact, when i tried to initialize model = XLNetModel.from_pretrained('xlnet-large-cased') - it also gives me the 'negative dimension' error.

`model = XLNetForSequenceClassification.from_pretrained("xlnet-large-cased", num_labels = 2)
model.to(device)`
`---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-37-d6f698a3714b> in <module>()
----> 1 model = XLNetForSequenceClassification.from_pretrained("xlnet-large-cased", num_labels = 2)
      2 model.to(device)

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in __init__(self, num_embeddings, embedding_dim, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse, _weight)
     95         self.scale_grad_by_freq = scale_grad_by_freq
     96         if _weight is None:
---> 97             self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))
     98             self.reset_parameters()
     99         else:

RuntimeError: Trying to create tensor with negative dimension -1: [-1, 1024]`


Fix GLUE TPU script

Temporary fix until we refactor this script to work with the trainer.


override weights name

Optional override of the weights name, relevant for cases where torch.save() is patched like in https://github.com/allegroai/trains


Remove boto3 dependency

Downloading a model from `s3://` urls was not documented anywhere and I suspect it doesn't work with our `from_pretrained` methods anyways.

Removing boto3 also kills its transitive dependencies (some of which are slow to support new versions of Python), contributing to a leaner library:

```
boto3-1.12.46
botocore-1.15.46
docutils-0.15.2
jmespath-0.9.5
python-dateutil-2.8.1
s3transfer-0.3.3
six-1.14.0
urllib3-1.25.9
```


Proposal: saner num_labels in configs.

See https://github.com/guillaume-be/rust-bert/pull/21 for more context


Add CALBERT (Catalan ALBERT) base-uncased model card

Related to pretrained model cards

Hi there!

I just uploaded a new ALBERT model pretrained on 4.3 GB of Catalan text. Thank you for these fantastic libraries and platform to make this a joy!


Remove hard-coded pad token id in distilbert and albert

As the config adds `pad_token_id` attribute, `padding_idx` set the value of `config.pad_token_id` in BertEmbedding. ( PR #3793 )

But it seems that not only the config of `Bert`, but also that of `DistilBert` and `Albert` has `pad_token_id`. ([Distilbert config](https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-config.json), [Albert config](https://s3.amazonaws.com/models.huggingface.co/bert/albert-base-v1-config.json))

But in Embedding class of Distilbert and Albert, it seems that `padding_idx` is still hard-coded. So I've fixed those parts.


Error on dtype in modeling_bertabs.py file

# 🐛 Bug

Traceback (most recent call last):
  File "run_summarizationn.py", line 324, in <module>
    main()
  File "run_summarizationn.py", line 309, in main
    evaluate(args)
  File "run_summarizationn.py", line 84, in evaluate
    batch_data = predictor.translate_batch(batch)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 797, in translate_batch
    return self._fast_translate_batch(batch, self.max_length, min_length=self.min_length)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 844, in _fast_translate_batch
    dec_out, dec_states = self.model.decoder(decoder_input, src_features, dec_states, step=step)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 231, in forward
    step=step,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 328, in forward
    dec_mask = torch.gt(tgt_pad_mask + self.mask[:, : tgt_pad_mask.size(1), : tgt_pad_mask.size(1)], 0)
RuntimeError: expected device cpu and dtype Byte but got device cpu and dtype Bool

## Information

Model I am using (Bert, XLNet ...):bertabs

Language I am using the model on (English, Chinese ...):English

The problem arises when using:
* [ ] the official example scripts: (give details below) - Yes
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below) - my own task


run_language_modeling, RuntimeError: expected scalar type Half but found Float

# 🐛 Bug

## Information

CUDA: 10.1
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
Installed transformers from source.
Trying to train **gpt2-medium**

Command Line:
```
export TRAIN_FILE=dataset/wiki.train.raw
export TEST_FILE=dataset/wiki.test.raw

python3 examples/run_language_modeling.py --fp16 \
    --per_gpu_eval_batch_size 1 \
    --output_dir=output \
    --model_type=gpt2-medium \
    --model_name_or_path=gpt2-medium \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --overwrite_output_dir

```

```
Traceback (most recent call last):
  File "examples/run_language_modeling.py", line 284, in <module>
    main()
  File "examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 314, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 388, in _training_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 616, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 500, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 237, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 196, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 149, in _attn
    w = torch.where(mask, w, self.masked_bias)
RuntimeError: expected scalar type Half but found Float
```


`BertTokenizer.from_pretrained()` not working with native Python `pathlib` module

Consider this code that downloads models and tokenizers to disk and then uses `BertTokenizer.from_pretrained` to load the tokenizer from disk.

**ISSUE:** `BertTokenizer.from_pretrained()` does not seem to be compatible with Python's native [pathlib](https://docs.python.org/3/library/pathlib.html) module.

```
# -*- coding: utf-8 -*-
"""
Created on: 25-04-2020
Author: MacwanJ

ISSUE:
BertTokenizer.from_pretrained() is not compatible with pathlib constructs

"""

from pathlib import Path
import os
from transformers import BertModel, BertTokenizer

# enables proper path resolves when this script is run in terminal
# or in an interactive Python shell/notebook
try:
    file_location = Path(__file__).parent.resolve()
except NameError:
    file_location = Path.cwd().resolve()

PROJ_DIR = file_location.parent

#####################################################################
# DOWNLOAD MODELS & TOKENIZERS
#####################################################################

model_name = 'bert-base-uncased'

if not os.path.exists(PROJ_DIR/'models'/model_name):
    print(model_name,
          'folder does not exist. Creating folder now',
          'and proceeding to download and save model')
    os.makedirs(PROJ_DIR/'models'/model_name)
else:
    print(model_name,
          'folder already exists. Proceeding to download and save model')

tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertModel.from_pretrained(model_name)
print('Download complete. Proceeding to save to disk')

model.save_pretrained(PROJ_DIR/'models'/model_name)
tokenizer.save_pretrained(PROJ_DIR/'models'/model_name)
print('Model saving complete')

#####################################################################
# LOAD MODEL & TOKENIZER FROM DISK
#####################################################################
model_name = 'bert-base-uncased'

# Load pre-trained model tokenizer (vocabulary)
# !! DOES NOT WORK UNLESS I CONVERT THE PATHLIB OBJECT TO STRING!!
tokenizer = BertTokenizer.from_pretrained(PROJ_DIR/'models'/model_name)

# Load pre-trained model weights
model = BertModel.from_pretrained(PROJ_DIR/'models'/model_name,
                                  output_hidden_states=True)
```
Running:
`tokenizer = BertTokenizer.from_pretrained(PROJ_DIR/'models'/model_name)`
Yields this error:
```
Traceback (most recent call last):

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code

    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-94-eb0783e1626e>", line 1, in <module>

    tokenizer = BertTokenizer.from_pretrained(PROJ_DIR/'models'/model_name)

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\site-packages\transformers\tokenization_utils.py", line 393, in from_pretrained

    return cls._from_pretrained(*inputs, **kwargs)

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\site-packages\transformers\tokenization_utils.py", line 424, in _from_pretrained

    if os.path.isfile(pretrained_model_name_or_path) or is_remote_url(pretrained_model_name_or_path):

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\site-packages\transformers\file_utils.py", line 146, in is_remote_url

    parsed = urlparse(url_or_filename)

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\urllib\parse.py", line 367, in urlparse

    url, scheme, _coerce_result = _coerce_args(url, scheme)

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\urllib\parse.py", line 123, in _coerce_args

    return _decode_args(args) + (_encode_result,)

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\urllib\parse.py", line 107, in _decode_args

    return tuple(x.decode(encoding, errors) if x else '' for x in args)

  File "C:\Users\macwanj\AppData\Local\miniconda3\envs\bert\lib\urllib\parse.py", line 107, in <genexpr>

    return tuple(x.decode(encoding, errors) if x else '' for x in args)

AttributeError: 'WindowsPath' object has no attribute 'decode'

```
All the other functions and methods appear to be compatible with pathlib except for `BertTokenizer.from_pretrained()`

It would be good to ensure consistency with pathlib constructs across all functions because pathlib makes it easier to specify paths that work out of the box across operating systems.

- `transformers` version: 2.8.0
- Platform: Windows 10
- Python version: 3.7.0
- PyTorch version (GPU?):Non-GPU 1.3.1
- Tensorflow version (GPU?):2.1.0 non gpu
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


There are some warnings when I used AdamW and pytorch1.5.

# 🐛 Bug

## Information
pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of add_ is deprecated:
        add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
        add_(Tensor other, *, Number alpha)


Question regarding glue examples

hi,

I have used the run_glue.py for QQP dataset and the tested the saved model for a small test set. I get really high F1-score on dev set but get lower F1-score on the test set. My question is that, is the dev set used in glue examples is used to set parameters or is it a held out set?


I am trying to run the summarization example using BART and getting the following error.

```
tqdm_dict = {"loss": "{:.3f}".format(avg_loss), "lr": self.lr_scheduler.get_last_lr()[-1]}
AttributeError: 'LambdaLR' object has no attribute 'get_last_lr'
```

The error occurred at this [line](https://github.com/huggingface/transformers/blob/master/examples/transformer_base.py#L101). I am using PyTorch 1.3.


model.multiple_choice_head( ) function for Hugging Face GPT2 models

Hello,

I just have a quick question about the ```multiple_choice_head( )``` function for the Hugging Face GPT2 models.

when I execute the codes below, everything runs smoothly:

```python
# get pre-trained GPT2Model and GPT2DoubleHeadsModel
model_gpt2 = GPT2Model.from_pretrained('gpt2', output_hidden_states = True)
model_gpt2DoubleHeadsModel = GPT2DoubleHeadsModel.from_pretrained('gpt2')

# some parts of the codes are missing....

# trying to make a use of the gpt2DoubleHeadsModel.multiple_choice_head() function
hidden_states = model_gpt2(input_ids=input_ids, token_type_ids = token_type_ids)[2][1][:,:,:]
mc_logits = model_gpt2DoubleHeadsModel.multiple_choice_head(hidden_states).detach()
loss_fct = CrossEntropyLoss()
mc_loss = loss_fct(mc_logits.view(1,4), 1)
```
Here, I noticed that I do not need to specify the ```mc_token_ids``` when executing the line ```model_gpt2DoubleHeadsModel.multiple_choice_head(hidden_states)```.

My question is, does the ```multiple_choice_head()``` function from the line ```model_gpt2DoubleHeadsModel.multiple_choice_head(hidden_states)``` automatically take the
**last token** of my input text sequence as the ```cls_token``` (classification token) that is used by the ```mc_head``` to predict answers for the multiple-choice questions?

Thank you,


Allow the creation of "entity groups" for NerPipeline #3548

### This pull request applies the entity group transformation by setting the parameter: group=True.

This was done by reflecting the transformation inside NerPipeline. This is similar to a previously closed [PR](https://github.com/huggingface/transformers/pull/3607), which I closed because I accidentlly deleted my fork (apologies for my clumsiness).

Details of what I want to be able to do can be found in issue #3548.

cc @julien-c @mfuntowicz @petulla

Sample code:
```
# Install branch
# Make sure to restart runtime after installing if using Google Colab
!pip install -e git+git://github.com/enzoampil/transformers.git@add_index_to_ner_pipeline#egg=transformers

# Grouped NER
from transformers import pipeline
nlp = pipeline('ner', grouped_entities=True)
nlp("Enzo works at the Australian National University (AUN)")

# [{'entity_group': 'I-PER', 'score': 0.9968132972717285, 'word': 'Enzo'},
# {'entity_group': 'I-ORG', 'score': 0.9970400333404541, 'word': 'Australian National University'},
# {'entity_group': 'I-ORG', 'score': 0.9831967651844025, 'word': 'AUN'}]

# Ungrouped NER
nlp = pipeline('ner', grouped_entities=False)
nlp("Enzo works at the Australian National University (AUN)")

# [{'entity': 'I-PER', 'index': 1, 'score': 0.9983270168304443, 'word': 'En'},
# {'entity': 'I-PER', 'index': 2, 'score': 0.9952995777130127, 'word': '##zo'},
# {'entity': 'I-ORG', 'index': 6, 'score': 0.9984350204467773, 'word': 'Australian'},
# {'entity': 'I-ORG','index': 7, 'score': 0.9967807531356812, 'word': 'National'},
# {'entity': 'I-ORG', 'index': 8  'score': 0.9959043264389038, 'word': 'University'},
# {'entity': 'I-ORG', 'index': 10, 'score': 0.9900023937225342, 'word': 'AU'},
# {'entity': 'I-ORG', 'index': 11, 'score': 0.9763911366462708, 'word': '##N'}]
```

Tutorial on how to do Entity Grouping w/ NerPipeline [here](https://colab.research.google.com/drive/1CVLP0n3Q5t5qiWpode7jyhUNZpmLg0mS)

I'm very keen to get feedback for the above, so please let me know if I should change anything, or perform additional steps to bring its quality to an acceptable level.


Since about 1600 BST, 24 Apr 2020, loading ner pipeline gives RuntimeError:
```
from transformers import pipeline
nlp = pipeline("ner", ignore_labels=[])
```

> Traceback (most recent call last):
  File "test.py", line 2, in <module>
    nlp = pipeline("ner", ignore_labels=[])
  File "/Users/xxx/Github/xxx/xxx/.venv/lib/python3.6/site-packages/transformers/pipelines.py", line 1091, in pipeline
    model = model_class.from_pretrained(model, config=config, **model_kwargs)
  File "/Users/xxx/Github/xxx/xxx/.venv/lib/python3.6/site-packages/transformers/modeling_auto.py", line 1086, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/Users/xxx/Github/xxx/xxx/.venv/lib/python3.6/site-packages/transformers/modeling_utils.py", line 558, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for BertForTokenClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([9, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
        size mismatch for classifier.bias: copying a param with shape torch.Size([9]) from checkpoint, the shape in current model is torch.Size([2]).

- `transformers` version: 2.5.1
- Platform: macos 10.14.6
- Python version:  3.6.5
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.0.0
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Fix #3954 - GPT2 is not traceable

This PR replaces the `math.sqrt(...)` computation with `(...)**0.5` which allows the model to be correctly traced by `torch.jit.trace`.


GPT2 is not fully torch.jit.trace-able

# 🐛 Bug

## Information

I'm trying to use PyTorch's tracing on a pre-trained GPT2 model and running into the following warning emitted from torch.jit.trace:
```
/opt/miniconda3/envs/py3/lib/python3.6/site-packages/transformers/modeling_gpt2.py:144: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w = w / math.sqrt(v.size(-1))
```

Additionally, if I inspect the graph generated from the trace, I can see that the denominator in that division expression is a constant value, not one determined by the size of the tensor.

## To reproduce

The following snippet can be used to repro the problem (based on the example in the Quickstart guide)
```python
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2').eval()
example_input = torch.tensor([ tokenizer.encode("The Manhattan bridge")])
traced_model = torch.jit.trace(model, example_input)
```

## Environment info

- `transformers` version: 2.8.0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (False)


Fix BERT example code for NSP and Multiple Choice

After #3790 was merged I noticed that the examples for BERT NSP and Multiple Choice were wrong, too.

`token_type_id` wasn't being correctly set but that's necessary for the multi sequence tasks (NSP, Multiple Choice, Question Answering). So I changed it to use `encode_plus` / `batch_encode_plus` which takes care of that.

I'm correct in assuming that for Next Sentence Prediction the Linear Classifier isn't initialized with pretrained weights and has to be trained first, correct? I didn't find any mention of it in the code.


BertForSequenceClassification is not optimum

In the documentation about TFBertModel it says:

```
hidden_states (tuple(tf.Tensor), optional, returned when config.output_hidden_states=True):
tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

```
Hidden-states of the model at the output of each layer plus the initial embedding outputs.

But then in the code of BertForSequenceClassification:

```
        outputs = self.bert(inputs, **kwargs)

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output, training=kwargs.get("training", False))
        logits = self.classifier(pooled_output)

        outputs = (logits,) + outputs[2:]  # add hidden states and attention if they are here

        return outputs  # logits, (hidden_states), (attentions)
```
Wouldn't give better results using `pooled_output = outputs[0][:,0]` ?


run_xnli doesn't execute

# 🐛 Bug

## Information

(a related issue is this: #3950 )

I am trying to run the `run_xnli.py` script from the [Examples documentation](https://huggingface.co/transformers/examples.html#xnli), but I am getting an error. I am trying to run the script without training.

## To reproduce

Steps to reproduce the behavior:

I follow the [documentation](https://huggingface.co/transformers/examples.html#xnli)

My args are the following:

```
export XNLI_DIR=/path/to/xnli

python run_xnli.py \
  --model_type bert \
  --model_name_or_path bert-base-multilingual-uncased \
  --language de \
  --train_language en \
  --do_eval \
  --data_dir $XNLI_DIR \
  --per_gpu_train_batch_size 32 \
  --learning_rate 5e-5 \
  --num_train_epochs 1.0 \
  --max_seq_length 128 \
  --output_dir bert-base-multilingual-uncased \
  --save_steps -1
```

I get the following error:

```
Traceback (most recent call last):
  File "run_xnli.py", line 646, in <module>
    main()
  File "run_xnli.py", line 638, in main
    result = evaluate(args, model, tokenizer, prefix=prefix)
  File "run_xnli.py", line 290, in evaluate
    outputs = model(**inputs)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1139, in forward
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 2 is out of bounds.
```

## Expected behavior

I am not sure I am afraid. I am expecting some sort of accuracy measure as shown in the Example.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-74-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


In run_xnli.py, output_dir seems to be used in place of tokenizer_name

# 🐛 Bug

## Information

I am trying to run the `run_xnli` example as found in the documentation. Unfortunately, I get a strange error were the script thinks the `output_dir` argument contains a model name.

It seems that `output_dir` has been used in place of `tokenizer_name` in some instances, such as this: `tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)`

## To reproduce

Steps to reproduce the behavior:

1. Follow the example as found here: https://huggingface.co/transformers/examples.html#xnli

I get the following error:

```
Traceback (most recent call last):
  File "run_xnli.py", line 646, in <module>
    main()
  File "run_xnli.py", line 624, in main
    tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 868, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 971, in _from_pretrained
    list(cls.vocab_files_names.values()),
OSError: Model name '/tmp/debug_xnli/' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). We assumed '/tmp/debug_xnli/' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
```


After reading the tutorial I can use the BertModel to extract word embedding but how to use it extract the sentence embedding?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Add Type Hints to modeling_utils.py Closes #3911

Add Type Hints to methods in `modeling_utils.py`

Note: The coverage isn't 100%. Mostly skipped internal methods (and some I wasn't sure of).


Many tests fails with PyTorch 1.5.0

Several (56) tests fail on the newly released PyTorch 1.5.0. This is because the normal distribution can no longer accept a standard deviation of 0.

These errors should not reflect real-life usage, and, therefore, should not impact user experience too much.


ImportError: cannot import name 'DefaultDataCollator'

# 🐛 Bug

## Information

Using "from transformers import DefaultDataCollator" raises an ImportError on my system. Unsure if this is really a bug or if I'm doing something stupid. Any help would be appreciated.

Model I am using (Bert, XLNet ...): N/A

Language I am using the model on (English, Chinese ...): N/A

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. from transformers import DefaultDataCollator
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

For the import to go through cleanly.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Ubuntu 18.04
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0, No
- Tensorflow version (GPU?): -
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: -


BertConfig.to_json_file('config.json') saves  "num_labels"  as  "_num_labels" on Google Colab

# 🐛 Bug

There is a problem with config.to_json_file() on Google Colab.

## Information

The "num_labels" key pair is saved as "_num_labels" key in the config.json file produced by BertConfig.to_json_file('config.json').
When this file is subsequently read with BertConfig.from_json_file('config.json') the "_num_labels" is loaded along with a  "num_labels" key with default value 2.
This results in an improper configuration of a model TFBertForSequenceClassification.from_pretrained().

Model I am using (Bert, XLNet ...): TFBertForSequenceClassification

Language I am using the model on (English, Chinese ...): bert-base-german-cased

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

1.  modify the number of labels of the default bert-base-german-cased config and save to file

``` python
from transformers import BertConfig, TFBertForSequenceClassification

NUM_LABELS = 3
config = BertConfig.from_pretrained('bert-base-german-cased')
config.num_labels = NUM_LABELS
config.to_json_file('config.json')
```
This is what config.json looks like, notice the "_num_labels" key:

```json
{
  "_num_labels": 3,
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bad_words_ids": null,
  "bos_token_id": null,
  "decoder_start_token_id": null,
  "do_sample": false,
  "early_stopping": false,
  "eos_token_id": null,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "is_encoder_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "min_length": 0,
  "model_type": "bert",
  "no_repeat_ngram_size": 0,
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "prefix": null,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "task_specific_params": null,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 30000
}
```
2. read the saved config

``` python
config = BertConfig.from_json_file('config.json')
```
this is what the loaded config.json looks like:

```json
{
  "_num_labels": 3,
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bad_words_ids": null,
  "bos_token_id": null,
  "decoder_start_token_id": null,
  "do_sample": false,
  "early_stopping": false,
  "eos_token_id": null,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "is_encoder_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "min_length": 0,
  "model_type": "bert",
  "no_repeat_ngram_size": 0,
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "prefix": null,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "task_specific_params": null,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 30000
}
```

3. the loaded config improperly configures the number of labels of a model:

```python
model = TFBertForSequenceClassification.from_pretrained(
    './model/bert_de/weights.h5', config=config
)
```
```
>>> model  = TFBertForSequenceClassification.from_pretrained(
...     './model/bert_de/weights.h5', config=config
... )
2020-04-24 16:20:44.590400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/gontcharovd/anaconda3/envs/schwarz/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 274, in from_pretrained
    model.load_weights(resolved_archive_file, by_name=True)
  File "/home/gontcharovd/anaconda3/envs/schwarz/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 234, in load_weights
    return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
  File "/home/gontcharovd/anaconda3/envs/schwarz/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1220, in load_weights
    f, self.layers, skip_mismatch=skip_mismatch)
  File "/home/gontcharovd/anaconda3/envs/schwarz/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 777, in load_weights_from_hdf5_group_by_name
    str(weight_values[i].shape) + '.')
ValueError: Layer #2 (named "classifier"), weight <tf.Variable 'tf_bert_for_sequence_classification/classifier/kernel:0' shape=(768, 2) dtype=float32, numpy=
array([[ 0.00399719,  0.01253725],
       [ 0.00453608, -0.00098394],
       [ 0.01605183, -0.02316079],
       ...,
       [-0.0174976 ,  0.00032987],
       [-0.01292989,  0.00867058],
       [-0.02766422, -0.00422863]], dtype=float32)> has shape (768, 2), but the saved weight has shape (768, 3).
```

I load the model with saved weights after transfer learning starting from bert-base-german-cased.
The weights were saved by ModelCheckpoint:

```python
mc = ModelCheckpoint('./model/bert_de/weights.h5', monitor='val_loss', mode='min',
                     verbose=1, save_best_only=True, save_weights_only=True)
```

## Expected behavior

The correct number of labels = 3 should be read from the config.json file and  not the default 2.

```python
model_de = TFBertForSequenceClassification.from_pretrained(
    './model/bert_de/weights.h5', config=config_de
)
```

A hack for this problem is to specify the num_labels again after reading config.json:

```python
config = BertConfig.from_json_file('config.json')
config.num_labels = NUM_LABELS
mode = TFBertForSequenceClassification.from_pretrained(
    './model/bert_de/weights.h5', config=config
)
```

## Environment info
This happens on Google Colab. On my local machine I don't get the key "_num_labels".


Trainer: distributed eval

Tagging this as a Good First issue if anyone's interested.


How to input hidden state vectors from GPT2Model directly into mc_head of the GPT2DoubleHeads Model?

Hello,

I want to perform the following task:

**1.** _Feed in a text sequence to a GPT2Model (the GPT-2 model without output head);_

**2.** _Extract the hidden state vectors of the text sequence that are generated at each layer of the GPT2Model;_

**3.** _Input the hidden state vectors from each layer of the GPT2Model directly into the mc_head of the GPT2DoubleHeadsModel, and calculate the **mc_loss** that results from inputting the hidden state vectors._

I know that the code below can be used to input hidden state vectors into each individual layer of a HuggingFace GPT-2 model, but I am not sure if similar task can be done on the mc_head of the GPT2DoubleHeadsModel:

```python
model.transformer.h[i](embed)
```

My questions is:
 Is there anyway that I can input the hidden state vectors from each layer of the GPT2Model directly into the multiple choice head of the GPT2DoubleHeadsModel, and extract the **mc_loss** that results from it (i.e. is there any way that I can perform the task **3**)?

Thank you,


XLNetLMHeadModel: target mapping with num_predict > 1 and labels not working

Hi,

While trying to fine tune the XLNetLMHeadModel, I ran into the following error message:

`  File "[...]/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "[...]/lib/python3.8/site-packages/transformers/modeling_xlnet.py", line 1062, in forward
    loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1))
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
`

while calling the forward function like this:

               outputs = xlnet(input, # shape = (8, 50)

                                 attention_mask=attention_mask, # shape = (8, 50)

                                 perm_mask=perm_mask, # shape = (8, 50, 50)

                                 target_mapping=target_mapping, # shape = (8, 2, 50)

                                 token_type_ids=sequence_ids, # shape = (8, 50)

                                 labels=target) # shape = (8, 2)

According to the [docs](https://huggingface.co/transformers/model_doc/xlnet.html#transformers.XLNetLMHeadModel) the labels shape is expected to be (batch_size, num_predict). However using this shape causes the above mentioned error.

The error does not appear if we set num_predict to 1 and set labels.shape = torch.size([8])
The error also does not appear if we omit the labels and manually compute the loss


Decoding output sequences from TFGPT2Model

# ❓ Questions & Help
I asked this on SO without any luck.
https://stackoverflow.com/questions/61222878/how-can-you-decode-output-sequences-from-tfgpt2model

## Details
I'm trying to get generated text from the TFGPT2Model in the Transformers library. I can see the output tensor, but I'm not able to decode it. Is the tokenizer not compatible with the TF model for decoding?
```
import tensorflow as tf

from transformers import (
    TFGPT2Model,
    GPT2Tokenizer,
    GPT2Config,
)

model_name = "gpt2-medium"
config = GPT2Config.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
model = TFGPT2Model.from_pretrained(model_name, config=config)

input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute",
                 add_special_tokens=True))[None, :]  # Batch size 1
outputs = model(input_ids)
print(outputs[0])
result = tokenizer.decode(outputs[0])
print(result)
```
The output is:
```
$ python run_tf_gpt2.py
2020-04-16 23:43:11.753181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-04-16 23:43:11.777487: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-04-16 23:43:27.617982: W tensorflow/python/util/util.cc:319] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2020-04-16 23:43:27.693316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-04-16 23:43:27.824075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA n
ode, so returning NUMA node zero
...
...
2020-04-16 23:43:38.149860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10565 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:25:00.0, compute capability: 3.7)
2020-04-16 23:43:38.150217: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-16 23:43:38.150913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10565 MB memory) -> physical GPU (device: 2, name: Tesla K80, pci bus id: 0000:26:00.0, compute capability: 3.7)
2020-04-16 23:43:44.438587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
tf.Tensor(
[[[ 0.671073    0.60760975 -0.10744217 ... -0.51132596 -0.3369941
    0.23458953]
  [ 0.6403012   0.00396247  0.7443729  ...  0.2058892  -0.43869907
    0.2180479 ]
  [ 0.5131284  -0.35192695  0.12285632 ... -0.30060387 -1.0279727
    0.13515341]
  [ 0.3083361  -0.05588413  1.0543617  ... -0.11589152 -1.0487361
    0.05204075]
  [ 0.70787597 -0.40516227  0.4160383  ...  0.44217822 -0.34975922
    0.02535546]
  [-0.03940453 -0.1243843   0.40204537 ...  0.04586177 -0.48230025
    0.5768887 ]]], shape=(1, 6, 1024), dtype=float32)
Traceback (most recent call last):
  File "run_tf_gpt2.py", line 19, in <module>
    result = tokenizer.decode(outputs[0])
  File "/home/.../transformers/src/transformers/tokenization_utils.py", line 1605, in decode
    filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
  File "/home/.../transformers/src/transformers/tokenization_utils.py", line 1575, in convert_ids_to_tokens
    index = int(index)
  File "/home/.../venv/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 853, in __int__
    return int(self._numpy())
TypeError: only size-1 arrays can be converted to Python scalars
```


fix resize_token_embeddings to accept padding_idx for xlm-roberta models

When resize_token_embedding there is no option for padding_idx
and it leads to the wrong embedding for xlm-roberta models
which needs embedding to have padding_idx as 1 not 0.

I fixed issue by adding option for padding_idx and considered padding_idx as None (which is 0 as default) for compatibility for other types of transformers models.


Continue training args and tqdm in notebooks

Added a little more description to the metadata of training args for new scripts.
`--overwrite_output_dir` can be used to continue training from checkpoint if `--output_dir` points to a directory already having checkpoints.

This can help reduce the confusion when migrating from older script.

tqdm in colab prints every step in new line.
so updated to
`from tqdm.auto import tqdm`


[Benchmark] Parameter settings of XLM-R on NER tasks

# 🖥 Benchmarking `transformers`

## Benchmark

`XLM-RoBerta` on CoNLL 03 English NER dataset

## Set-up
`model_name_path`: xlm-roberta-base
`max_steps`: 3000 (roughly 6 epochs)
`warmup_steps`: 300
`save_steps`: 200
`learning_rate`: 5e-5
`per_gpu_train_batch_size`: 8
`n_gpu`: 3 (Titan X)

## Results
91.017 (averaged results of five runs)

## Issues
As reported in original [paper](https://arxiv.org/pdf/1911.02116.pdf), XLM-R can achieve 92.25 on this English NER dataset. Since they do not share any parameter settings for fine-tuning, I have to adopt the above ones and the F1 score is about 91.017. Has anybody ever obtained similar results to those in this paper with appropriate settings?


What should I do if I want to change the padding idx in pytorch bert(Huggingface)?

I find ,in RobertaModel class, the padding idx is limited to 1(is it true?),which is different from my tokenizer and data.So what should I do to change or just apply the model structure...

###
class RobertaEmbeddings(BertEmbeddings):
    """
    Same as BertEmbeddings with a tiny tweak for positional embeddings indexing.
    """

    def __init__(self, config):
        super().__init__(config)
        self.padding_idx = 1
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=self.padding_idx)
        self.position_embeddings = nn.Embedding(
            config.max_position_embeddings, config.hidden_size, padding_idx=self.padding_idx
        )

    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):
        if position_ids is None:
            if input_ids is not None:
                # Create the position ids from the input token ids. Any padded tokens remain padded.
                position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
            else:
                position_ids = self.create_position_ids_from_inputs_embeds(inputs_embeds)

        return super().forward(
            input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
        )


Pytorch 1.5 DataParallel

PyTorch

Anything PyTorch

# 🐛 Bug

## Information

Can't run forward in PyTorch 1.5.0, works fine in 1.4.0

Model I am using (Bert, XLNet ...): XLNet

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

Transformer + custom head + custom losses + differential learning rates, I don't think it matters.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

Custom news classification

## To reproduce

Steps to reproduce the behavior:

1. Install PyTorch 1.5.0
2. Run forward on xlnet
3.

```
  File "transformers/modeling_xlnet.py", line 761, in forward
    dtype_float = next(self.parameters()).dtype
StopIteration
```

## Expected behavior

Runs forward

## Environment info

- `transformers` version: 2.8.0
- Platform: Ubuntu 18.04
- Python version: Anaconda 3.7
- PyTorch version (GPU?): 1.5, Yes
- Tensorflow version (GPU?): N/A
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes


How to feeding hidden state vectors from one transformer directly into a layer of different transformer

Hello,

I want to perform the following task:

**1.** _Feed in a text sequence to a GPT2Model (the GPT-2 model without output head);_

**2.** _Extract the hidden state vectors of the text sequence that are generated at each layer of the GPT2Model;_

**3.** _Feed the hidden state vectors from each layer of the GPT2Model as an input to the GPT2DoubleHeadsModel (assuming that the n_embd of the GPT2Model and the GPT2DoubleHeadsModel are equal), and calculate the **mc_loss** that results from inputting the hidden state vectors. My GPT2DoubleHeadsModel would only be consisted of (1 layer) + (the output heads)._

I know that the code below can be used to feed in hidden state vectors as inputs to each individual layer of a HuggingFace GPT-2 model:

```python
model.transformer.h[i](embed)
```
But the code above will only give me the output of an individual layer, so it wouldn't give me the mc_loss that I am looking for (since to calculating the mc_loss from the hidden state vectors, I would first need to feed in the hidden state vectors into the 1st layer of the GPT2DoubleHeadsModel, and somehow feed in the output of the layer as an input to the mc_head (multiple choice output head) of the GPT2DoubleHeadsModel)

My questions are:
1. Is there any way to perform the task 3 with the Hugging Face GPT-2 models?
2. Is there anyway that I can access the GPT2DoubleHeadsModel's mc_head (multiple choice head), similar to the code that I wrote above?

Thank you,


[qol] example scripts: parse args from .args file or JSON

You can either:
- pass the path to a json file as the unique argument
- automatically load args from a .args file that's a sibling of the entrypoint script.

```json
{
  "model_name_or_path": "bert-base-cased",
  "task_name": "mnli",
  "data_dir": "./data/glue/MNLI",
  "output_dir": "./models/mnli"
}
```

```
--model_name_or_path distilroberta-base
--task_name mnli
```

I use the .args method myself, but I think both are ok.

Tagging @jplu and @stefan-it who have expressed interest in this.


Add ALBERT to the Tensorflow to Pytorch model conversion cli

This PR adds ALBERT to the `convert` command of `transformers-cli` to allow the conversion of pre-trained models from Tensorflow to Pytorch. The documentation is updated to show how to run the conversion.


[ci] Load pretrained models into the default (long-lived) cache

There's an inconsistency right now where:
- we load some models into CACHE_DIR
- and some models in the default cache
- and often, in both for the same models

When running the RUN_SLOW tests, this takes a lot of disk space, time, and bandwidth.

I'd rather always use the default cache


Loading a TF pretrained model into BertForSequenceClassification module

Hi, there might be something I am doing wrong, but I cannot figure out what that is, then any help would be welcome.

After downloading a TF checkpoint (containing model.index, model.data, model.meta, config.json and vocab.txt files), I used it to perform pretraining using some more text, more relevant to the task I would have ahead. Pretraining was performed using the API from BERT's official github. This generated other model.index, model.data and model.meta files. I am now trying to load them into the BertForSequenceClassification module, using the `from_pretrained` method. I figured I should include a config instance as well, so I used that config.json file that was attached to that very first one TF checkpoint I mentioned. But when passing the index file to `from_pretrained`, I get the error:

`AttributeError: 'BertForSequenceClassification' object has no attribute 'bias'`

Any help would be much appreciated.


How can i finetune an encode-decoder combination?

Hi, I want to finetune an encode-decoder model to train on a parallel dataset(something like translation) and I'm not sure what should I do. I read [this](https://medium.com/huggingface/encoder-decoders-in-transformers-a-hybrid-pre-trained-architecture-for-seq2seq-af4d7bf14bb8) blog post but It didn't help. It doesn't really matter for me which encode or decoder I choose, I just need the power of a pre-trained transformer.


Add support for LayerNorm before residual connection in transformer

Recent work has shown that placing the layer norm before the residual connection in a transformer leads to better gradient propagation and more stable training. For example, [Learning Deep Transformer Models for Machine Translation](https://arxiv.org/abs/1906.01787) shows that when stacking more layers, the Pre-LN transformer performs better than the Post-LN transformer. A more detailed discussion is found in [On Layer Normalization in the Transformer Architecture](https://openreview.net/pdf?id=B1x8anVFPr), which aids understanding despite being rejected from ICLR 2020 for lack of novelty.

_In my own experiments pretraining from scratch with TFAlbert, I have found pre-layer normalization to have greater training stability and improved performance than the default post-layer normalization._

**This is not a final PR, just opening the discussion. Some caveats:**
- If approved, I can extend this to all transformer models, not just ALBERT.
- Loading an existing pretrained model with `config.pre_layer_norm = True` leads to poor performance. A separate pretrained model would need to be made available with this configuration option. I would be happy to do so for ALBERT, but this would be outside my pay grade for all the models. Since the current focus of the repo is on finetuning rather than pretraining, I understand if this isn't a complexity tradeoff you want to make.

What do you think about adding this configuration option?


Fix TFAlbertForSequenceClassification classifier dropout probability.…

… It was set to config.hidden_dropout_prob, but should be config.classifier_dropout_prob.


❓ DistilBert test perplexity based on WikiText-2: ppl is too low?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
I tried to use the pre-trained model to fine-tune.  I got the ppl 6.12 of the DistilBert model, which is much lower than the GPT-2 model (ppl: 18.34, ref: https://paperswithcode.com/sota/language-modelling-on-wikitext-2).

Is the DistilBert model works much better than the GPT-2? Or is it just because the loss functions are different?

Here are the commands:
```
# 1) code:
git clone https://github.com/huggingface/transformers.git

# 2) Download dataset:
cd transformers/examples/
wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip
unzip wikitext-2-raw-v1.zip

# 3) Benchmart:
## distilbert:
export TRAIN_FILE=./wikitext-2-raw/wiki.train.raw
export TEST_FILE=./wikitext-2-raw/wiki.test.raw
CUDA_VISIBLE_DEVICES=6 python run_language_modeling.py \
    --output_dir=output_distilbert \
    --model_type=distilbert \
    --model_name_or_path=distilbert-base-uncased \
    --do_train \
    --per_gpu_train_batch_size 15 \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm
```
Result:
```
04/22/2020 13:25:33 - INFO - __main__ -   ***** Running evaluation  *****
04/22/2020 13:25:33 - INFO - __main__ -     Num examples = 535
04/22/2020 13:25:33 - INFO - __main__ -     Batch size = 4
Evaluating: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 134/134 [00:05<00:00, 24.02it/s]
04/22/2020 13:25:38 - INFO - __main__ -   ***** Eval results  *****
04/22/2020 13:25:38 - INFO - __main__ -     perplexity = tensor(6.1200)
```
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Remove 50k limits bug

This is a bug needs to be removed


Using the default trainer args

Hey guys!

Just looked through the transformer_base... why not also allow any of the trainer args to be used?

I imagine the best set up for you guys is:
- add default trainer args (since people can look into the lightning docs for details on this)
- add hf specific args (nlp related or whatever you guys need)
- add model specific args

But instead it looks like you only use a subset of the args
https://github.com/huggingface/transformers/blob/dd9d483d03962fea127f59661f3ae6156e7a91d2/examples/transformer_base.py#L275

Something like this should work:
```python
parser = ArgumentParser()

# enable all trainer args
parser = Trainer.add_argparse_args(parser)

# add the HF args
parser.add_argument(--some_hf_specific_thing, ...)

# add the model args
parser = GoodGAN.add_model_specific_args(parser)

# cook them all up :)
args = parser.parse_args()
```

Check here for mode details:
https://pytorch-lightning.readthedocs.io/en/stable/hyperparameters.html#multiple-lightning-modules

@srush @nateraw


Change uses of pow(x, 3) to pow(x, 3.0) to resolve #3873

This minor pull request fixes #3873 by changing the type of the exponent parameter for the _torch.pow()_ call in _gelu_new()_ from integer to float.


Feat/add model card

model card

Related to pretrained model cards

Add model card for sentiment control model.


LineByLineTextDataset limits the total number of examples to 50000 documents

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi,

While running run_language_modeling.py with argument --line_by_line, found out that it only takes first 50000 documents.

https://github.com/huggingface/transformers/blob/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d/src/transformers/data/datasets/language_modeling.py#L93

Please confirm is this a bug or intended behavior?


New run_language_modeling.py does not save vocab.txt and tokenizer_config.json

# 🐛 Bug
Hi,

While running new  run_language_modeling.py, at the end it only saves pytorch model and config.json and training arguments
Unlike before it also used to store vocab.txt and tokenizer_config.json

 I am not using any custom tokenizer, only using tokenizer provided by the model.

## Command Details I used

python run_language_modeling.py
--output_dir=bert-base-uncased-4
--overwrite_output_dir
--model_type=bert
--model_name_or_path="bert-base-uncased"
--do_train --train_data_file="./test.txt"
--per_gpu_train_batch_size 3
--mlm
--num_train_epochs 1

Model I am using (Bert, XLNet ...):
Bert-base-uncased

Language I am using the model on (English, Chinese ...):
Engilsh

## Expected behavior
It should also save vocab.txt and tokenizer_config.json


run_language_modeling.py line 251: checking if it is a directory

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi, I'm curious why you changed the code to check if model_args.model_name_or_path is a directory. I am trying to train OpenAI GPT in Google Colab, and using arguments as following:

!python /content/transformers/examples/run_language_modeling.py
--output_dir="/content/gdrive/My Drive/output"
--overwrite_output_dir
--model_type=openai-gpt
--model_name_or_path=openai-gpt
--do_train
--train_data_file="/content/gdrive/My Drive/train.txt"
--do_eval
--eval_data_file="/content/gdrive/My Drive/dev.txt"
--num_train_epochs 5
--learning_rate 1.5e-4
--save_steps 1
--save_total_limit 5

but this line causes following error:

Traceback (most recent call last):
File "/content/transformers/examples/run_language_modeling.py", line 283, in
main()
File "/content/transformers/examples/run_language_modeling.py", line 257, in main
trainer.train(model_path=model_path)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 363, in train
self._rotate_checkpoints()
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 458, in _rotate_checkpoints
checkpoints_sorted = self._sorted_checkpoints(use_mtime=use_mtime)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 443, in _sorted_checkpoints
regex_match = re.match(".*{}-([0-9]+)".format(checkpoint_prefix), path)
File "/usr/lib/python3.6/re.py", line 172, in match
return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object

It seems that openai-gpt is not a directory, thus making model_path variable equal to None. Is this due to my implementation or can this part be fixed?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


xlm-roberta (large/base) : run_language_modeling.py cannot starting training

Hi HuggingFace, thank you very much for your great contribution.

# ❓ Questions & Help

My problem is : run_language_modeling.py takes abnormally long time for `xlm-roberta-large & base` **_"before" start training_**
. It got stuck at the following step for 7 hours (so I gave up eventually) :

`transformers.data.datasets.language_modeling -   Creating features from dataset file at ./`

I have successfully running `gpt2-large`, `distilbert-base-multilingual-cased` using exactly the same command below (just change model) which start training within just 2-3 minutes. At first I thought that because of the big size of XLM-Roberta. However, as `gpt2-large` has similar size, is there somehow problem on finetuning with XLM-Roberta?  (So maybe a bug in the current version)

I also tried to rerun the same command in another machine, but got the same stuck (which is not the case for `gpt2-large`, `distilbert-base-multilingual-cased` )

**update** the same thing happen to `xlm-roberta-base`

## Command Details I used
Machine AWS p3.2xlarge (V100, 64GB Ram)
Training file size is around 60MB

!python transformers/examples/run_language_modeling.py \
    --model_type=xlm-roberta \
    --model_name_or_path=xlm-roberta-large \
    --do_train \
    --mlm \
    --per_gpu_train_batch_size=1 \
    --gradient_accumulation_steps=8 \
    --train_data_file={TRAIN_FILE} \
    --num_train_epochs=2 \
    --block_size=225 \
    --output_dir=output_lm  \
    --save_total_limit=1 \
    --save_steps=10000 \
    --cache_dir=output_lm \
    --overwrite_cache \
    --overwrite_output_dir


change scheduler.get_last_lr to get_lr to avoid bug

Learning rate scheduler function for getting last lr is `get_lr` instead of `get_last_lr`.


Create README.md

Related to pretrained model cards


feat: add logging through Weights & Biases

Hi,

I've been using hugging-face for a while and I've been loving it. I think it could become a central reference for NLP problems and the new `Trainer` class made it so much easier to add new functionality.

I would like to contribute in solving a few limitations I observed:
* some models don't have clear details on how they have been trained (hyper-parameters) and for which task, making it hard to reproduce their results (and even ensure those results were actually reached) -> ideally we would track & log their training process as well as config parameters used
* some new models keep on being requested/proposed without seeing their actual efficiency -> ideally we would be able to compare them to baselines
* I've seen issues where "better" models are offered to replace previous ones but there is no way to prove they are actually better -> maybe we can add a metrics table (linked to actual runs) to the model card
* it is hard to keep track of models and datasets
* overall I'd like to bring more traceability and transparency between users

This is why I propose to use [Weights & Biases](https://docs.wandb.com/) which is free for open source and will help solve those issues. Also it is already part of Pytorch-Lightning so it will be easy to integrate it with those scripts, if extra setup is required.

This PR brings :

* logging through Weights & Biases when `wandb` is installed
* logging of evaluation metrics at the end of training

See example run on distilbert-base-cased  -> [W&B run](https://app.wandb.ai/borisd13/transformers-examples/runs/3iuvnwam?workspace=user-borisd13)

![image](https://user-images.githubusercontent.com/715491/80064053-d3e75680-84fc-11ea-954a-9590185c65df.png)

If you navigate around, you will see that config parameters, metrics, gradients, computer resources, actual code (including git repo & command line used) and stdout have all been uploaded for full traceability and reproducibility. I can also easily upload trained model but maybe we could add an arg to make it optional in case users have limited connection (as you prefer)?

I can also easily compare bert-base-uncased and distilbert-base-uncased from my [project page](https://app.wandb.ai/borisd13/transformers-examples).
*Note: I didn't try any fine-tuning and just used same parameters on both in this example.*

![image](https://user-images.githubusercontent.com/715491/80064311-62f46e80-84fd-11ea-8e9a-d4a61af437b3.png)

This makes it easy to quickly see if a new model is actually interesting.

Finally, this integration let us create [cool reports](https://app.wandb.ai/stacey/keras_finetune/reports/Curriculum-Learning-in-Nature--Vmlldzo1MjcxNw) where graphs directly pool data and are interactive.

If you like it, then I'd love to add other features in later PR's:

* log trained model -> I'll just need to have access to path of relevant files so that I don't upload entire output directory
* add [W&B init args](https://docs.wandb.com/library/init) as parameters based on what can be useful for hugging-face
* log all command line args -> I need to access the parser so that I can also log parameters such as `model_path_or_name` and `max_seq_length` (for run_glue script) which will be convenient to create custom summary tables and compare models (or finetune)
* maybe log called script (unless `finetuning_taks` is enough?)
* in addition to metrics, track model size & inference speed which can help users choose the model they want
* use artifacts (W&B version control system) to track models, tokenizers & datasets -> central repository that will let us see easily that runs are based on same dataset or same pre-trained model + see all trained tasks associated to one pre-trained model
* log prediction samples (based on the task)
* create a central repo on W&B (entity: hugging-face) to have access to other's people runs
* integrate model card issue template so that we always add a link to a run as well as performance metrics
* add [sweeps](https://docs.wandb.com/sweeps) -> make it easy to optimize hyper-parameters
* we could add a hook to run automatically on all tasks any new model added (and link the run to the model cards)

Please let me know your thoughts and feel free to contact me or Weights & Biases directly to discuss this integration.


New run_language_modeling.py continuing trainng

I tried using new script for Language Modeling to train model from scratch.

I was training model with old script. When I tried continuing using new one, there is no option for `--should_continue`.

Does `--overwrite_output_dir` makes it train from scratch if i use same directory for `--model_name_or_path` and `--output_dir`?

So, I am continuing training with old script.