How to generate different suggestions with GPT2 or XLNet like Write With Transformers?

Hello,

I want to generate with run_generation more different suggestions of the next words, preferably with variable length and different terms or synonyms, like it is done in Write With Transformer.

Any suggestion or idea on how to achieve this?

Thanks


export to onnx issue

External

Using the library with external tools (onnx, tflite, ...)

Hi experts,

I got an error when running the onnx model after conversion. Can anyone please help to take a look?

code:
`torch.onnx.export(model,
    (input_ids, attention_mask, token_type_ids),
    "bert.onnx",
    input_names=['input_ids', 'attention_mask', 'token_type_ids'],
    export_params=True, verbose=True)`

`sess = rt.InferenceSession("bert.onnx")
inputs = {'input_ids': input_ids, 'attention_mask': attention_mask, 'token_type_ids': token_type_ids}
outputs = sess.run(None, inputs)
`

error:

Traceback (most recent call last):
  File "test.py", line 29, in <module>
    outputs = sess.run(None, inputs)
  File "/usr/local/lib/python3.6/dist-packages/onnxruntime/capi/session.py", line 142, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'' Status Message: indices element out of data bounds, idx=1 must be within the inclusive range [-1,0]


The prediction output is random

When I  use the official example scripts to predict a text sentence classification model,  I found that the output is different every time。

```
from transformers import BertTokenizer, BertForSequenceClassification
import torch
import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
model.eval()
input_ids = torch.tensor(tokenizer.encode("my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs= model(input_ids)
print(outputs)
```

fist result :(tensor([[-0.1939,  0.1449]], grad_fn=<AddmmBackward>),)
second result:(tensor([[-0.2425, -0.2737]], grad_fn=<AddmmBackward>),)
third result: (tensor([[ 0.0494, -0.7208]], grad_fn=<AddmmBackward>),)
......

I expected the outputs to be the same... am I doing this wrong?


Model download: tf-xlm-roberta-large "tf_model.h5" file missing

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): tf-xlm-roberta-large

The "tf_model.h5" file for tf-xlm-roberta-large appears to be missing as the following url from model hub is returning "NoSuchKey" errors: https://s3.amazonaws.com/models.huggingface.co/bert/jplu/tf-xlm-roberta-large/tf_model.h5

If intentional, is it being reupload soon?


why take the first hidden state for sequence classification (DistilBertForSequenceClassification)

Core: Modeling

Internals of the library; Models.

Usage

General questions about the library

In the last few layers of sequence classification [here][1], the first hidden state of the sequence length of the transformer output to be used for classification.

    hidden_state = distilbert_output[0]  # (bs, seq_len, dim) <-- transformer output
    pooled_output = hidden_state[:, 0]  # (bs, dim)           <-- first hidden state
    pooled_output = self.pre_classifier(pooled_output)  # (bs, dim)
    pooled_output = nn.ReLU()(pooled_output)  # (bs, dim)
    pooled_output = self.dropout(pooled_output)  # (bs, dim)
    logits = self.classifier(pooled_output)  # (bs, dim)

Is there any benefit to taking the first hidden state over the last, average, or even the use of a Flatten layer instead?

I've also asked this question on [Stack Overflow](https://stackoverflow.com/questions/60087613/why-take-the-first-hidden-state-for-sequence-classification-distilbertforsequen)

  [1]: https://github.com/huggingface/transformers/blob/33d3072e1c54bcd235447b98c6dea1b4cb71234c/src/transformers/modeling_distilbert.py#L634


Adapter-BERT is missing in transformers library?

Adapter BERT obtain comparable results to BERT on several NLP tasks while achieving parameter efficiency. ( https://github.com/google-research/adapter-bert ) @thomwolf
I think, it will be useful if adapter-bert is also included in the library.


Fix documentation in ProjectedAdaptiveLogSoftmax

The shape of outputs for forward in ProjectedAdaptiveLogSoftmax is flipped in the documentation: it should be log probabilities when `labels` is `None` and NLLs otherwise. This is what the code does, but the docstring has them flipped.


Add option to `cached_path` to automatically extract archives

Slight modification to `cached_path` so that zip and tar archives can be automatically extracted.
- archives are extracted in the same directory than the (possibly downloaded) archive in a created extraction directory named from the archive.
- automatic extraction is activated by setting `extract_compressed_file=True` when calling `cached_file`.
- the extraction directory is re-used t avoid extracting the archive again unless we set `force_extract=True`, in which case the cached extraction directory is removed and the archive is extracted again.

Currently not added to the `from_pretrained` methods. Probably better to have the user control this explicitly at this level (by first extracting the archive) => open to discussion though.

Also include a simple proposal to add TF/PT compatibility in hf_buckets (cc @julien-c)


[examples] rename run_lm_finetuning to run_language_modeling

And corresponding doc updates


Add albert-base-v3 to pretrained models?

# 🚀 Feature request

Albert v3 was recently released on TFHub [here](https://tfhub.dev/google/albert_base/3). Could you please add it to the list of available pretrained models [here](https://huggingface.co/transformers/pretrained_models.html)?

## Motivation

Would provide the community with the most up-to-date albert version.


Add contributors snapshot

powered by https://github.com/sourcerer-io/hall-of-fame


[docs] Add menu w/ links to other pages on hf.co


build: add poetry, an alternative to setup.py with dependency versions tracked

Hello,
I have written a `pyproject.toml` so your project can be setup using [poetry](https://github.com/python-poetry/poetry) .

That way requirements versions can easily be tracked for people wanting to use **poetry** (it is optional).

# Example

```bash
# first setup your virtual environment, then:
pip install poetry
poetry install  # this is equivalent to 'pip install .' but with versions tracked
poetry install --extras testing # pip install -e ".[testing]"
poetry install --extras examples # pip install -r examples/requirements.txt
poetry install --extras torch  # pip install -e ".[torch]"
poetry install --extras tf  # pip install -e ".[tf]"

# edit: updating dependencies to the latest possible:
poetry update

# adding new dependencies
poetry add MyPyModule
```

# Notes

This does not change any python code, i.e. everything still works :)


Loss is calculated on all tokens, including padding, in the LM fine-tuning example

# 🐛 Bug
The BERT fine-tuning example uses a special index to mark ignored locations for the loss function:
`loss_fct = CrossEntropyLoss(ignore_index=-1)`
While in the same example, the masking function that samples locations to be included or excluded uses a different index: -100 (which is the default ignored index for the cross-entropy loss function, if one is not supplied):
`labels[~masked_indices] = -100  # We only compute loss on masked tokens`

Model I am using (Bert, XLNet ...): All models.

Language I am using the model on (English, Chinese ...): All languages

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: LM Finetuning
* [ ] my own task or dataset: (give details below)

## Expected behavior
Loss should be computed only on the 15% (mlm_probability) of sampled tokens.

- `transformers` version: 2.3, 2.4


TFRoberta output with attention_mask changes in version 2.3.0 vs 2.4.1

Core: Modeling

Internals of the library; Models.

TensorFlow

Anything TensorFlow

Usage

General questions about the library

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): not relevant

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

```python
import tensorflow as tf
tf.get_logger().setLevel('CRITICAL')
import transformers
print(transformers.__version__)
from transformers import TFRobertaModel, RobertaConfig
from numpy.testing import assert_allclose

config = RobertaConfig()
model = TFRobertaModel(config)

input1 = tf.constant([[5, 3, 4, 8, 7, 1, 6]])
attention_mask1 = tf.constant([[1, 1, 1, 1, 1, 0, 1]])

out1, _ = model({'input_ids': input1, 'attention_mask': attention_mask1})

input2 = tf.constant([[5, 3, 4, 8, 7, 5, 6]])
attention_mask2 = tf.constant([[1, 1, 1, 1, 1, 0, 1]])

out2, _ = model({'input_ids': input2, 'attention_mask': attention_mask2})

assert_allclose(out1.numpy()[:, :5, :], out2.numpy()[:, :5, :])
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: I am using dummy token ids

## To reproduce

Steps to reproduce the behavior:

1. make a new virtualenv
2. install tensorflow
3. pip install transformers=2.3.0
4. save the script in test_mask.py
5. run python test_mask.py
6. repeat 1-5, but in 2 install the latest release: pip install transformers

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

In case of transformers==2.3.0, the test **passes**, giving the following output:

```
2.3.0
```

In case of transformers==2.4.1, the test **fails**:

```
2.4.1
Traceback (most recent call last):
  File "test_attention_mask.py", line 21, in <module>
    assert_allclose(out1.numpy()[:, :5, :], out2.numpy()[:, :5, :])
  File "/home/bartosz/.pyenv/versions/aphp-django/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/home/bartosz/.pyenv/versions/aphp-django/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 3840 / 3840 (100%)
Max absolute difference: 0.43364888
Max relative difference: 337.9916
 x: array([[[ 0.742064, -1.048889, -1.133795, ...,  1.208201, -0.110544,
         -1.556664],
        [-0.307906, -0.545374, -1.124657, ...,  0.067571, -0.857922,...
 y: array([[[ 0.718682, -0.995075, -1.105745, ...,  1.380688, -0.071943,
         -1.627201],
        [-0.390375, -0.534317, -1.113236, ...,  0.178188, -0.822041,...
```

## Expected behavior

In my understanding, the test should pass, because the only difference between inputs  `input1` and `input2` is oken with index 5 which is masked in both `attention_masks` (i.e. t, `input1[5]==1` while `input2[5]==5`). Note that I don't take the embedding for this token in the comparison of the outputs.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.4.1
- Platform: linux
- Python version: 3.7.4
- PyTorch version (GPU?): No
- Tensorflow version (GPU?): 2.1.0 (no GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Cannot reproduce SQUAD Example

I'm not able to reproduce the squad experimentation (via the example). I tried the command line;
python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

That gived very weird results

Then I read a little the forum and I tried:
python3 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --version_2_with_negative \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 10000 \
  --output_dir debug_squad/ \
  --overwrite_output_dir

(I also tried with v2 dataset it not works anymore). Can you give me some lead to reproduce the results given in the git readme, or the branch to do it ?


BERT decoder: Fix failure with the default attention mask.

PyTorch < 1.3 requires multiplication operands to be of the same type. This was violated when using default attention mask (i.e.., `attention_mask=None` in arguments) given BERT in the decoder mode.

In particular, this was breaking `Model2Mode`l and made a tutorial from quickstart.md failing.

A test is included, but here is a minimal snippet to reproduce:

```python
import torch
from transformers import BertModel
model = BertModel.from_pretrained("bert-base-uncased", is_decoder=True)
inputs = torch.LongTensor([[1, 2, 3]])
model(inputs)    # no `attention_mask` provided
```

On PyTorch 1.2 or older this was failing with
```
Traceback (most recent call last):
...
  File "/home/oleksiy.syvokon/transformers/src/transformers/modeling_bert.py", line 735, in forward
    extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
RuntimeError: expected device cpu and dtype Float but got device cpu and dtype Long
```


Multi-text files support for run_lm_finetuning

Ex: LM (Finetuning)

Related to language modeling fine-tuning

# 🚀 Feature request

Support multi-text files for run_lm_finetuning.

## Motivation

Currently, you support training from scratch but it only supports a single file. Usually, when we train from scratch we train a model using multi-text files, not a single text file.
It will be great to support multi-text file and maybe separate finetuning from training from scratch files.


Changed vocabulary save function. Variable name was inconsistent

Caused an error to be thrown when passing a file name instead of a directory.

UnboundLocalError: local variable 'vocab_file' referenced before assignment

Associated with issue #2753


Saving tokenizer vocabulary throws error when passing file name instead of directory.

# 🐛 Bug

## Information

Using transfo-xl-wt103

UnboundLocalError: local variable 'vocab_file' referenced before assignment

## To reproduce

Steps to reproduce the behavior:

```
from transformers import TransfoXLTokenizer, TFTransfoXLLMHeadModel

tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
tokenizer.save_vocabulary('vocab.txt')
```

## Pull Request

https://github.com/huggingface/transformers/pull/2754


Fix multi-gpu evaluation in run_glue.py example

Fix multi-gpu evaluation while training in `examples/run_glue.py`


Sentence pair classification

Usage

General questions about the library

# ❓ Questions & Help
Hi,

I want to do sentence pair classification on Quora Questions Dataset by fine-tuning BERT. I am new to this and do not know where to start? Can anyone let me know how do i get started with this?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


default output of BertModel.from_pretrained('bert-base-uncased')

General questions about the library

By default `output = BertModel.from_pretrained('bert-base-uncased')` is a 2-tuple where `output[0]` is the hidden states of the last layer, but how is `output[1]` computed? It doesn't seem to be average of the last layer hidden states vectors over multiple tokens. I am trying to leverage output as sentence embedding, not sure if I should use `output[1]`. Thank you!


Upgrade run_generation

The script `run_generation` has a few issues that I aim to fix in this PR:

- [x] The XLNet and XLM generations are broken (crash)
- [x] An end of sequence token is added to all sequences, even when models don't have that token, and results in weird end of sequences.
- [x] No way to generate multiple sequences at a time as it was possible before
- [x] The `length` parameter doesn't take into account the prompt length.
- [x]  The prompt is concatenated to the generated sequence, which results in concatenating the initial text for XLNet.
- [x] Actually implement languages for XLM


TFAlbertModelTest::test_pt_tf_model_equivalence  -> Fatal Python Error on Mac

PyTorch

Anything PyTorch

TensorFlow

Anything TensorFlow

Tests

Related to tests

Running the unit tests locally on mac, I get "Fatal Python error: Aborted"

To reproduce, try `pytest tests/test_modeling_tf_albert.py::TFAlbertModelTest::test_pt_tf_model_equivalence `

### Environment Info

- `transformers` version: 2.4.1
- Platform: Darwin-19.0.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.3.1 (False)
- Tensorflow version (GPU?): 2.0.0 (False)

### Traceback

```
tests/test_modeling_tf_albert.py Fatal Python error: Aborted

Current thread 0x0000000110d2adc0 (most recent call first):
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/functional.py", line 1372 in linear
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87 in forward
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541 in __call__
  File "/Users/shleifer/transformers_fork/src/transformers/modeling_albert.py", line 321 in forward
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541 in __call__
  File "/Users/shleifer/transformers_fork/src/transformers/modeling_albert.py", line 566 in forward
  File "/Users/shleifer/miniconda3/envs/nb/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541 in __call__
  File "/Users/shleifer/transformers_fork/tests/test_modeling_tf_common.py", line 111 in test_pt_tf_model_equivalence

```

https://github.com/huggingface/transformers/issues/2240 has a different error message from a similar test.

Thanks!


Arxiv README


Added CamembertForQuestionAnswering


Add BartModel

This ports BART, a "sequence-to-sequence model trained with denoising as pretraining objective." from https://github.com/pytorch/fairseq/tree/master/examples/bart
The decoder is left-to-right, the encoder is biredictional. As such, the code only uses a causal attention mask in the decoder.

### TODO:
- [x] conversion of pretrained weights
- [x] some unit testing
- [x] inference produces the same results as the fairseq version.
- [x] decide on signature/splitting of encoder, decoder arguments (see https://github.com/huggingface/transformers/blob/808bbd5a6abe5b26656ffd809ce0e753495c912a/src/transformers/modeling_encoder_decoder.py#L240

)
- [x] Docstrings
- [x] More comments for code readers

### Future PRs
- [ ] example with correct pretraining objective
- [ ] `BartForSummarization.from_pretrained('bart-large-cnn')`


Albert language model fine tuning not running run_lm_finetuning.py

PyTorch

Anything PyTorch

Usage

General questions about the library

# ❓ Questions & Help

## Information

Model I am using (Albert(all types)):

Language I am using the model on (English):

The problem arises when using:
* [ ] the official example scripts: (give details below)
the code returns memory allocation problems when run with any version from albert. i tried to reduce the sequence length and batch size to a minum setting, but the issue still arises. my setting and the minimized setting both run normally with bert or roberta, the issue arises only when i change the model to Albert.
an example:
`tcmalloc: large alloc 1951195136 bytes == 0x7f750f664000 @  0x7f76efbf8887 0x7f764c2a1b79 0x7f764c29fb0f 0x7f764c29fc33 0x7f764c26a155 0x7f764c26837e 0x7f764c26bbb1 0x7f764c2606df 0x50a8af 0x50c5b9 0x509d48 0x50aa7d 0x50c5b9 0x508245 0x509642 0x595311 0x5a067e 0x50d966 0x58efc9 0x4c9546 0x5886f4 0x58892e 0x551b81 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x50a080 0x50aa7d 0x50c5b9 0x508245`

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)
language model finetuning for albert
## To reproduce

Steps to reproduce the behavior:

1. in run_lm_finetuning add:
` from transformers import (AlbertConfig,
    AlbertForMaskedLM,
    AlbertTokenizer,
)`
2.add to MODEL_CLASSES dictionary:
`    "albert": (AlbertConfig, AlbertForMaskedLM, AlbertTokenizer),`
3. add file text.txt, a similar txt file to the wiki dataset that's mentioned in the docs.
4.run the finetuning script:
`python transformers/examples/run_lm_finetuning.py \
    --output_dir=output \
    --model_type=albert \
    --model_name_or_path=albert-base-v1 \
    --do_train \
    --train_data_file test.txt \
    --block_size 50 \
    --per_gpu_train_batch_size 2 \
    --max_steps 520000 \
    --weight_decay 0.01 \
    --logging_steps 5000 \
    --mlm`

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment

* OS: Google colab
* Python version: 3.7
* PyTorch version: 1.3.1
* `transformers` version (or branch): latest
* Using GPU ? yes
* Distributed or parallel setup ? no
* Any other relevant information:


PreTrainedEncoderDecoder keeps giving me the same next token

Hi, I am trying to use PreTrainedEncoderDecoder to train a seq2seq model. I have a working training code, but I'm not sure if I'm doing things correctly because the decoded token is always the same token during inference.

The data is paired input and target sentence pairs.

The dataset class looks like this:
```
class LineByLineLabelledTextDataset(Dataset):
    """Labelled text dataset where a line corresponds to a sample."""

    def __init__(self,
                 lines,
                 tokenizer,
                 sep="|||",
                 max_seqlen=512):
        self.lines = lines
        self.tokenizer = tokenizer
        self.sep = sep
        self.max_seqlen = max_seqlen

    def __len__(self):
        return len(self.lines)

    def __getitem__(self, i):
        splitted = self.lines[i].split(self.sep)
        input, target = splitted[0], splitted[1]

        # target += " [GEN_STOP]"

        input_dict = self.tokenizer.encode_plus(input,
                                                max_length=self.max_seqlen,
                                                pad_to_max_length=True)
        target_dict = self.tokenizer.encode_plus(target,
                                                 max_length=self.max_seqlen,
                                                 pad_to_max_length=True)

        return torch.tensor(input_dict["input_ids"]), torch.tensor(target_dict["input_ids"]), torch.tensor(input_dict["attention_mask"]), torch.tensor(target_dict["attention_mask"])

```

The training function for one step looks like this:
```
def train_batch(batch, model, optimizer, device, phase="train"):
    input_ids = batch[0].to(device)
    target_ids = batch[1].to(device)
    input_attention_mask = batch[2].to(device)
    target_attention_mask = batch[3].to(device)

    optimizer.zero_grad()

    with torch.set_grad_enabled(phase == "train"):
        outputs = model(input_ids, target_ids,
                        encoder_attention_mask=input_attention_mask,
                        decoder_attention_mask=target_attention_mask,
                        decoder_lm_labels=target_ids)
        lm_loss = outputs[0]
        loss = lm_loss

    loss.backward()
    optimizer.step()

    return loss
```

The decode function looks like this
```
def decode(encoder_input_text, model, tokenizer, max_length=20):
    model.eval()

    text = encoder_input_text
    generated_text = "[CLS]"

    while len(generated_text.split()) < max_length:
        encoder_input_ids = tokenizer.encode(text)
        encoder_input_tensor = torch.tensor([encoder_input_ids])
        print(f"encoder_input_tensor: {encoder_input_tensor}")

        decoder_input_ids = tokenizer.encode(generated_text, add_special_tokens=False)
        decoder_input_tensor = torch.tensor([decoder_input_ids])
        print(f"decoder_input_tensor: {decoder_input_tensor}")

        with torch.no_grad():
            outputs = model(encoder_input_ids=encoder_input_tensor, decoder_input_ids=decoder_input_tensor)
            predictions = outputs[0]

            predicted_index = torch.argmax(predictions[0, -1]).item()
            predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]

        generated_text += " " + predicted_token

        print(generated_text)
        print(len(generated_text.split()))

        if len(generated_text.split()) >= max_length:
            break

    return generated_text
```

I see the training loss goes down a bit during training. I don't know what I'm doing wrong.


do_lower_case strips accents!

# 🐛 Bug

When calling BertTokenizer with do_lower_case=True, the tokenizer gets rid of accents, which is a misleading behavior not indicated in the name of the parameter. We suggest that you create another parameter which indicates whether or not to strip accents, separated from do_lower_case! This also happens in AutoTokenizer. For some languages, like spanish, this is crucial (hacia is not the same as hacía). Moreover, it's set to True by default.
https://github.com/huggingface/transformers/blob/2184f87003c18ad8a172ecab9a821626522cf8e7/src/transformers/tokenization_bert.py#L346


XLM-Roberta mask filling error

# XLM-Roberta mask token filling error

Hi! I am trying to use XLM-Roberta for Masked LM task, but the error occurs when the model fills masked token in a test sentence.

**The code is:**

```
config.model_name = 'xlm-roberta-base'

tokenizer: tr.XLMRobertaTokenizer = tr.XLMRobertaTokenizer.from_pretrained(config.model_name)
model: tr.XLMRobertaForMaskedLM = tr.XLMRobertaForMaskedLM.from_pretrained(config.model_name)

input_ids = tokenizer.encode_plus("I want to <mask> New York!",
                                               max_length=config.max_length)['input_ids']
x = np.full((config.max_length), fill_value=tokenizer.pad_token_id)
attn = np.zeros_like(x)
for i, tok in enumerate(input_ids):
    x[i] = tok
    attn[i] = 1
x = torch.tensor(x).unsqueeze(0).to(device)
attn = torch.tensor(attn).unsqueeze(0).to(device)

outputs = model(x, attention_mask=attn, masked_lm_labels=x)
```
**The error is**

```
RuntimeError: cublas runtime error : library not initialized at ../aten/src/THC/THCGeneral.cpp:216
```

When I try Albert for similar task everything works fine, but the Roberta family doesn't.
Could you please help with this issue?


T5

    the code is here
    tokenizer = T5Tokenizer.from_pretrained('t5-small')
    model = T5WithLMHeadModel.from_pretrained('t5-small')
    input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
    print(tokenizer._tokenize("Hello, my dog is cute"))
    print(input_ids)
    print(input_ids.shape)
    outputs = model(input_ids=input_ids,attention_mask=torch.tensor([5.]))
    print(outputs[0].shape)

and the error is
extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
UnboundLocalError: local variable 'extended_attention_mask' referenced before assignment


Development Infrastructure for ML Projects

Hi guys, I am sorry to post an issue that is a bit outside the scope of this project. But I have been a consistent watcher of the transformers project and it is excellent on how I can collaborate and coordinate with people and develop something.
But I am a rookie when it comes to working with people and get going with a project and I have been assigned a task to create reliable and scalable infrastructure, where a team can have space for research, development, test, deploy.
I have been dabbling around with bitbucket pipelines and docker but it would be helpful to get your opinions on it.


Fix GPT2 config set to trainable

There's currently a bug in the GPT2 model which prevents it from being saved. This is caused by setting the trainable parameter to the GPT2 config, which cannot be packaged later in the save pipeline. Gotta love python...

Here is a simple script which you can use to reproduce this bug (and check the fix):
```
from transformers import (TFGPT2Model)

if __name__ == '__main__':
    _base_model = TFGPT2Model.from_pretrained("gpt2")
    print(base_model._layers[0].trainable)
```


Version 2.4.1 breaks run_lm_finetuning.py, version 2.3.0 runs fine

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)

Maksed language modeling

https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py

The tasks I am working on is:

* [ x] my own task or dataset: (give details below)

I am running on this dataset (though I doubt the issue is with the dataset, just use any text file)

https://drive.google.com/open?id=18oogYKR-VCQlFyUaYcGfgDiKTrFtkTHn

## To reproduce

Steps to reproduce the behavior:

```
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

python run_lm_finetuning.py --train_data_file train.raw --output_dir /output --model_type 'bert' --mlm --model_name_or_path 'bert-base-uncased' --do_train
```

without cuda for different error message

```
python run_lm_finetuning.py --train_data_file train.raw --output_dir /output --model_type 'bert' --mlm --model_name_or_path 'bert-base-uncased' --do_train --no_cuda
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Error message when using CUDA

```
Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/17 [00:00<?, ?it/s]/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [5,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [10,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [12,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [15,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [18,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [22,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [23,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [24,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [25,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [26,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [29,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [30,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=110 error=710 : device-side assert triggered
Traceback (most recent call last):
  File "HFpretrain.py", line 771, in <module>
    main()
  File "HFpretrain.py", line 721, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "HFpretrain.py", line 325, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 1019, in forward
    masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2021, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110
```

CPU error message

```
Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/17 [00:00<?, ?it/s]Traceback (most recent call last):
  File "HFpretrain.py", line 771, in <module>
    main()
  File "HFpretrain.py", line 721, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "HFpretrain.py", line 325, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 1019, in forward
    masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2021, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target -1 is out of bounds.
```

## Expected behavior

Should train as usual

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.4.1
- Platform: google colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): n/a
- Using GPU in script?: both cpu and gpu Tesla T4
- Using distributed or parallel set-up in script?: o


TensorFlow XLM doesn't accept NumPy arrays for the attention mask

Convert NumPy attention mask to a TensorFlow tensor so that the mask creation doesn't crash

closes #2729


test_attention_weights cleanup

No logic changes, just uses getattr to make code more readable.


pass langs parameter to certain XLM models

Adding an argument that specifies the language the SQuAD dataset is in so language-sensitive XLMs (e.g. `xlm-mlm-tlm-xnli15-1024`) don't default to language `0`.
Allows resolution of issue #1799 .


Save model wrapped in Keras

Hi all,

Sorry for my naive question but I am trying to save my keras model (<class 'tensorflow.python.keras.engine.training.Model'>) in which I use TFBertModel() function as an hidden layer. To do that I use the save() function provided by the tf.keras package.

But I got this error:
```python
---------------------------------------------------------------------------

NotImplementedError                       Traceback (most recent call last)

<ipython-input-13-3b315f7219da> in <module>()
----> 1 model.save('model_weights.h5')

8 frames

/tensorflow-2.1.0/python3.6/tensorflow_core/python/keras/engine/network.py in get_config(self)
    915   def get_config(self):
    916     if not self._is_graph_network:
--> 917       raise NotImplementedError
    918     return copy.deepcopy(get_network_config(self))
    919

NotImplementedError:
```

The error can be reproduce from my colab : https://colab.research.google.com/drive/18HYwffkXCylPqeA-8raL82vfwOjb-aLP

And another question is how should I call this model for prediction ?

Thx for your help!


Error for run_lm_finetuning.py (CUDA error: device-side assert triggered)

### Reporting Error
I updated transformers from 2.3.x to 2.4.1 today, and I'm facing a runtime error which is RuntimeError: CUDA error: device-side assert triggered.
I reviewed recent updates and found out the commits [Follow up 213] is causing the error.
Below are the changes from the commits:

- labels[~masked_indices] = -100 # We only compute loss on masked tokens
+ labels[~masked_indices] = -1 # We only compute loss on masked tokens

The changes are related to the calculation of masked language model loss, so the problem seems to occur when args.mlm is True. (If I change the value -1 to -100, it works fine)

Any suggestions?

### Sys Info
OS: Windows 10
Transformers: 2.4.1
PyTorch: 1.4.0
Tensorflow: 2.1.0

### Full Stack Trace
C:\Users\USER\Anaconda3\python.exe C:/Users/USER/PycharmProjects/Testing/huggingface/run_lm_finetuning.py --output_dir=output --model_type=roberta --model_name_or_path=roberta-base --do_train --train_data_file=../data/wikitext-2/wiki.train.raw --do_eval --eval_data_file=../data/wikitext-2/wiki.test.raw --evaluate_during_training --mlm --per_gpu_train_batch_size=1 --per_gpu_eval_batch_size=1
2020-02-04 10:46:01.194260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
02/04/2020 10:46:05 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
02/04/2020 10:46:05 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-config.json from cache at C:\Users\USER\.cache\torch\transformers\e1a2a406b5a05063c31f4dfdee7608986ba7c6393f7f79db5e69dcd197208534.a7ab0e5de2d8321d6d6a15b199110f2c99be72976b7d151423cb8d8c261a13b6
02/04/2020 10:46:05 - INFO - transformers.configuration_utils -   Model config RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "do_sample": false,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 1,
  "use_bfloat16": false,
  "vocab_size": 50265
}

02/04/2020 10:46:05 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json from cache at C:\Users\USER\.cache\torch\transformers\d0c5776499adc1ded22493fae699da0971c1ee4c2587111707a4d177d20257a2.ef00af9e673c7160b4d41cfda1f48c5f4cba57d5142754525572a846a1ab1b9b
02/04/2020 10:46:05 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-merges.txt from cache at C:\Users\USER\.cache\torch\transformers\b35e7cd126cd4229a746b5d5c29a749e8e84438b14bcdb575950584fe33207e8.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
02/04/2020 10:46:05 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin from cache at C:\Users\USER\.cache\torch\transformers\228756ed15b6d200d7cb45aaef08c087e2706f54cb912863d2efe07c89584eb7.49b88ba7ec2c26a7558dda98ca3884c3b80fa31cf43a1b1f23aef3ff81ba344e
02/04/2020 10:46:10 - INFO - transformers.modeling_utils -   Weights of RobertaForMaskedLM not initialized from pretrained model: ['lm_head.decoder.bias']
02/04/2020 10:46:12 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=510, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=True, do_train=True, eval_all_checkpoints=False, eval_data_file='../data/wikitext-2/wiki.test.raw', evaluate_during_training=True, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_steps=-1, mlm=True, mlm_probability=0.15, model_name_or_path='roberta-base', model_type='roberta', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='output', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=1, per_gpu_train_batch_size=1, save_steps=500, save_total_limit=None, seed=42, server_ip='', server_port='', should_continue=False, tokenizer_name=None, train_data_file='../data/wikitext-2/wiki.train.raw', warmup_steps=0, weight_decay=0.0)
02/04/2020 10:46:12 - INFO - __main__ -   Loading features from cached file ../data/wikitext-2\roberta_cached_lm_510_wiki.train.raw
02/04/2020 10:46:12 - INFO - __main__ -   ***** Running training *****
02/04/2020 10:46:12 - INFO - __main__ -     Num examples = 4740
02/04/2020 10:46:12 - INFO - __main__ -     Num Epochs = 1
02/04/2020 10:46:12 - INFO - __main__ -     Instantaneous batch size per GPU = 1
02/04/2020 10:46:12 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 1
02/04/2020 10:46:12 - INFO - __main__ -     Gradient Accumulation steps = 1
02/04/2020 10:46:12 - INFO - __main__ -     Total optimization steps = 4740
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
Iteration:   0%|          | 0/4740 [00:00<?, ?it/s]C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [2,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [9,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [10,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [11,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [12,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [15,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [16,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [17,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [18,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [19,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [20,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [21,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [22,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [23,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [24,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [25,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [26,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [28,0,0] Assertion `t >= 0 && t < n_classes` failed.
C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [30,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "C:/Users/USER/PycharmProjects/Testing/huggingface/run_lm_finetuning.py", line 790, in <module>
    main()
  File "C:/Users/USER/PycharmProjects/Testing/huggingface/run_lm_finetuning.py", line 740, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "C:/Users/USER/PycharmProjects/Testing/huggingface/run_lm_finetuning.py", line 356, in train
    loss.backward()
  File "C:\Users\USER\Anaconda3\lib\site-packages\torch\tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "C:\Users\USER\Anaconda3\lib\site-packages\torch\autograd\__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: device-side assert triggered

Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
Iteration:   0%|          | 0/4740 [00:00<?, ?it/s]

Process finished with exit code 1


Masked LM and TFBertForSequenceClassification

Hello,

Is it correct to say that fine-tuning a TFBertForSequenceClassification model is the same as fine-tuning BERT's mlm and in additional a classification layer at the same time?

Thanks!


QuickStart code error

In the model2model quickstart example, I'm getting an error here:

`outputs = model(question_tensor, answer_tensor, decoder_lm_labels=labels_tensor)`

With the following message:

`RuntimeError: The size of tensor a (8) must match the size of tensor b (768) at non-singleton dimension 3`

Any ideas?


Attention Mask for TFXLM Model doesn't work

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLM

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import *
tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-enfr-1024')
model = TFXLMModel.from_pretrained('xlm-mlm-enfr-1024')
text = "Good evening."
input_ids = tokenizer.encode(text, add_special_tokens=True)
last_hidden_states = model(np.array([input_ids]), attention_mask=np.ones_like(np.array(input_ids)))
```

Error output:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_xlm.py", line 589, in call
    outputs = self.transformer(inputs, **kwargs)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_xlm.py", line 348, in call
    mask, attn_mask = get_masks(slen, lengths, self.causal, padding_mask=attention_mask)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_xlm.py", line 88, in get_masks
    tf.debugging.assert_equal(shape_list(mask), [bs, slen])
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 546, in shape_list
    static = x.shape.as_list()
AttributeError: 'tuple' object has no attribute 'as_list'
```

Works fine if I attention mask is removed

## Expected behavior
`last_hidden_states` is a tuple of type `tf.Tensor`

## Environment info

- `transformers` version: 2.3.0
- Platform: OSX
- Python version: 3.6.5
- Tensorflow version (GPU?): 2.1.0(CPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


RuntimeError: expected dtype Float but got dtype Long - run_lm_finetuning.py

# 🐛 Bug

## Information

I'm using my Camembert-based language model on Italian language (built from scratch).

I'm trying to use [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) to fine-tune my language model on a dataset.

@julien-c suggested my to add  `--line_by_line`  to my launch script, beacuse without that flag, the program blocked on the tokenization of the training set. That advide let the program work. But after some hours, the program crashes with a strange Runtime Error: in the assignment of 10 % of random words to masks at [line 218](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L218) in the _mask_tokens()_ function:
```python
# 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random] #it crashes here
```

The error was this:
`RuntimeError: expected dtype Float but got dtype Long
`
It's a strange error, beacuse it crashes after several minutes or hours after the launch. The time of the proper functioning of program seems random: sometimes the script completes 3/4 epochs and then it crashes, sometimes it crashes before the end of the first epoch.

## To reproduce

I launched this:
```bash
python3 run_lm_finetuning.py \
--train_data_file /path/to/train.txt \
--eval_data_file /path/to/eval.txt \
--output_dir /path/to/output \
--mlm \
--do_train \
--do_eval \
--model_type camembert \
--model_name_or_path /path/to/my/model \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 8 \
--overwrite_output_dir \
--overwrite_cache \
--max_steps 500000 \
--block_size 128 \
--save_steps 50000 \
--eval_all_checkpoints \
--line_by_line
```

I got this error in the middle of 6th epoch:
```
File "run_lm_finetuning.py", line 801, in <module>51:22,  4.94it/s]
    main()
  File "run_lm_finetuning.py", line 750, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 342, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 222, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:  55%|█████▍    | 6/11 [20:26:33<17:02:07, 12265.60s/it]
Iteration:  69%|██████▊   | 33378/48603 [1:47:45<49:09,  5.16it/s]
```

I'm managing to run the code anyway, restarting the program using the flag `--model_name_or_path` and giving the last saved checkpoint rather then the original language model every time it crashes.

I printed `inputs[indices_random]` and `random_words[indices_random]` beacause are the two variables in the line in which the program crashes:

- The code crashes with this 2 variables:
```
inputs[indices_random] = tensor([1173.])
Random_words[indices_random] = tensor([4220])

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 797, in <module>
    main()
  File "run_lm_finetuning.py", line 747, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 349, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 229, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:  60%|██████    | 3/5 [14:31:21<9:40:54, 17427.18s/it]
```
- while before the crash the code enters the _mask_tokens()_ function corretly and prints lines like these:
```
inputs[indices_random] = tensor([19807,    78,    51,  1204])
Random_words[indices_random] = tensor([14538, 15381, 30255,  3778])
```

In my opinion the only difference is that **tensor([1173.])** in the crash example, contains a not integer value (there is the '.' at the end of the number, while all the other times not. Maybe with a cast of `inputs[indices_random]` it would work.

## Environment info

- `transformers` version: 2.4.1
- Platform: Linux-4.4.0-108-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


XLM Roberta token_type_ids bug with batch_encode_plus

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLM Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
import transformers

name = "xlm-roberta-base"
tokenizer = XLMRobertaTokenizer.from_pretrained(name)
model = XLMRobertaModel.from_pretrained(name)
x = tokenizer.batch_encode_plus(
    ["foo", "bar bar bar"], add_special_tokens=True, return_tensors="pt"
)
model(**x)
```

Output

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-33-3743974223ad> in <module>
      7     ["foo", "bar bar bar"], add_special_tokens=True, return_tensors="pt"
      8 )
----> 9 model(**x)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    797
    798         embedding_output = self.embeddings(
--> 799             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    800         )
    801         encoder_outputs = self.encoder(

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     62
     63         return super().forward(
---> 64             input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
     65         )
     66

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    189             inputs_embeds = self.word_embeddings(input_ids)
    190         position_embeddings = self.position_embeddings(position_ids)
--> 191         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    192
    193         embeddings = inputs_embeds + position_embeddings + token_type_embeddings

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: index out of range: Tried to access index 1 out of table with 0 rows. at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

## Expected behavior

No error

## Environment info

- `transformers` version: 2.4.1
- Platform: OS X
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (no GPU)
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


convert_tokens_to_ids(self, tokens)中的ids.append(self.vocab[token])，KeyError

# ❓ Questions & Help

## Details
<!-- Description of your issue -->
KeyError                                  Traceback (most recent call last)
<ipython-input-20-dfb0a32a8e67> in <module>()
     19 for mask_pos in mask_positions:
     20     candidates = options[num]
---> 21     candidates_ids = tokenizer.convert_tokens_to_ids(candidates)
     22     token_ids = tokenizer.convert_tokens_to_ids(tokenized_text)
     23     tokens_tensor = torch.tensor([token_ids])

~/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py in convert_tokens_to_ids(self, tokens)
    119         ids = []
    120         for token in tokens:
--> 121             ids.append(self.vocab[token])
    122         if len(ids) > self.max_len:
    123             logger.warning(

KeyError: 'persuading'

How to solve KeyError problems?Thank you.


add TinyBERT?

# 🌟 New model addition

## Model description
TinyBERT is a smaller version of the Base BERT model, it uses transformer distillation (a type of knowledge distillation) to transfer the plenty of knowledge encoded in a large “teacher” BERT to a small “student” TinyBERT. is empirically effective and achieves more than 96% the performance of teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT is also significantly better than state-of-the-art baselines on BERT distillation, with only ∼28% parameters and ∼31% inference time of them. Here I have feature request to add the pretrained weights of TinyBERT after general learning from https://github.com/huawei-noah/Pretrained-Language-Model and model for both TF2.0 and pytorch. I think the transformer distillation method should be introduced too.
https://arxiv.org/pdf/1909.10351.pdf

<!-- Important information -->

## Open source status

* [x] the model implementation is available: (give details)
https://github.com/huawei-noah/Pretrained-Language-Model, only pytorch is available in my knowledge at the moment
https://github.com/koursaros-ai/nboost
* [x] the model weights are available: (give details)
https://github.com/huawei-noah/Pretrained-Language-Model
* [x] who are the authors: (mention them, if possible by @gh-username)
https://github.com/huawei-noah/Pretrained-Language-Model  @jacobrxz https://github.com/jacobrxz


sequence labeling for sentences and not tokens

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I have sentences that belong to a paragraph. Each sentence has a label.
[s1,s2,s3,..], [l1,l2,l3,...]
I understand that I have to encode each sentence using an encoder e.g bert, and then use sequence labeling. Could you guide me on how I could do that, combining them?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
https://stackoverflow.com/questions/60048900/sequence-labeling-for-sentences-and-not-tokens


Improved testing

Adding some tests for some models that were not tested.


Bert and Roberta models cannot be converted to TFLite

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert / Roberta

Language I am using the model on (English, Chinese ...): N/A

The problem arises when using:
* [ ] the official example scripts: (give details below)
Sort of. Using the tflite conversion script provided here:
https://github.com/huggingface/tflite-android-transformers/blob/master/models_generation/distilbert.py

The tasks I am working on is: Converting models to tflite format

## To reproduce

Steps to reproduce the behavior:
I first tried the example script provided above to convert a distilbert model to tflite, and it worked fine. The GPT conversion also works great.

Next, I modified the above script to the following:
```
import tensorflow as tf
from transformers import TFRobertaModel

model = TFRobertaModel.from_pretrained('roberta-base')

input_spec = [tf.TensorSpec([1, 128], tf.int32), tf.TensorSpec([1, 128], tf.int32)]
model._set_inputs(input_spec, training=False)

print(model.inputs)
print(model.outputs)

converter = tf.lite.TFLiteConverter.from_keras_model(model)

# For conversion with hybrid quantization:
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.experimental_new_converter = True

tflite_model = converter.convert()
```
Note that the above can be replaced with "TFBertModel" and "bert-base-cased" with 3 input tensors with the same result below.

## Expected behavior

No errors, creates tflite model.

## Actual behavior
Error for both BERT and Roberta:
```
[<tf.Tensor 'input_1_11:0' shape=(None, 128) dtype=int32>, <tf.Tensor 'input_2_9:0' shape=(None, 128) dtype=int32>]
[<tf.Tensor 'tf_roberta_model_1/Identity:0' shape=(None, 128, 768) dtype=float32>, <tf.Tensor 'tf_roberta_model_1/Identity_1:0' shape=(None, 768) dtype=float32>]
---------------------------------------------------------------------------
ConverterError                            Traceback (most recent call last)
<ipython-input-15-1f63532e8b87> in <module>
     26 converter.experimental_new_converter = True
     27
---> 28 tflite_model = converter.convert()
     29
     30 open("distilbert-squad-384.tflite", "wb").write(tflite_model)

c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\python\lite.py in convert(self)
    444         input_tensors=input_tensors,
    445         output_tensors=output_tensors,
--> 446         **converter_kwargs)
    447
    448     if self._is_calibration_quantize():

c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\python\convert.py in toco_convert_impl(input_data, input_tensors, output_tensors, enable_mlir_converter, *args, **kwargs)
    447       input_data.SerializeToString(),
    448       debug_info_str=debug_info_str,
--> 449       enable_mlir_converter=enable_mlir_converter)
    450   return data
    451

c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\python\convert.py in toco_convert_protos(model_flags_str, toco_flags_str, input_data_str, debug_info_str, enable_mlir_converter)
    198       stdout = _try_convert_to_unicode(stdout)
    199       stderr = _try_convert_to_unicode(stderr)
--> 200       raise ConverterError("See console for info.\n%s\n%s\n" % (stdout, stderr))
    201   finally:
    202     # Must manually cleanup files.

ConverterError: See console for info.
2020-02-03 14:16:20.869205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2020-02-03 14:16:25.853657: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Cumsum
2020-02-03 14:16:25.854123: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.854715: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.855259: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.855869: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.856324: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.856863: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.857394: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.857914: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.858543: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.859107: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.859552: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.860084: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:26.060782: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 1517 operators, 2651 arrays (0 quantized)
2020-02-03 14:16:26.149298: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 1517 operators, 2651 arrays (0 quantized)
2020-02-03 14:16:26.149831: F tensorflow/lite/toco/graph_transformations/resolve_strided_slice_attributes.cc:95] Check failed: start_indices_size <= num_input_axes (4 vs. 2)StridedSlice op requires no more than 2 start indices
Fatal Python error: Aborted

Current thread 0x000013c4 (most recent call first):
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\toco\python\toco_from_protos.py", line 52 in execute
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\absl\app.py", line 250 in _run_main
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\absl\app.py", line 299 in run
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\python\platform\app.py", line 40 in run
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\toco\python\toco_from_protos.py", line 89 in main
  File "C:\drive\projects\ml-notebooks\pycharm-venv\Scripts\toco_from_protos.exe\__main__.py", line 9 in <module>
  File "C:\python\lib\runpy.py", line 85 in _run_code
  File "C:\python\lib\runpy.py", line 193 in _run_module_as_main
```

## Environment info

- `transformers` version:
- Platform: Windows 10
- Python version: 3.6.8
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): 2.0.0-dev20191002 (gpu=yes)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: N/A


Is transformers ovewriting tokenizer?

Hello. I haven't been able to use tokenizer since friday.
It seems that if I install transformers via pip it overwrites tokenizer installation with a version that doesn't work.

If I get a new instance and do that:

`pip install transformers`

When, I do that:
`pip install tokenizers`

I got the following msg:

> Requirement` already satisfied: tokenizers in /usr/local/lib/python3.7/site-packages (0.0.11)

And if I tried to import, I got this error msg:

> ImportError: cannot import name 'BertWordPieceTokenizer' from 'tokenizers'

I was wondering if it is a problem related to the new Transformers you released last Friday.


Add READMEs to Tensorflow versions of CamemBERT and XLM-RoBERTa

Add model cards.


Error when running run_lm_finetuning.py

I'm getting the following error when trying to finetune BERT for armenian language

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at C:\w\1\s\windows\pytorch\aten\src\THNN/generic/ClassNLLCriterion.c:97


DistilBertForMaskedLM is not passing ignore_index to loss fct nn.CrossEntropyLoss

# 🐛 Bug

I'm running `run_lm_finetuning.py` and got the error below:

```
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=110 error=710 : device-side assert triggered
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 795, in <module>
    main()
  File "run_lm_finetuning.py", line 745, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 349, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/tamvm/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tamvm/.local/lib/python3.6/site-packages/transformers/modeling_distilbert.py", line 550, in forward
    masked_lm_labels.view(-1))
  File "/home/tamvm/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tamvm/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/tamvm/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2016, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/tamvm/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1842, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
```

Looking through the code, I realized that `-100` is used as labels for indices that are not masked. However, `DistilBertForMaskedLM` is not passing `ignore_index=-100` to `nn.CrossEntropyLoss`, which makes loss function calculate loss on `-100` labels. and hence the error.

[https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_distilbert.py#L510](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_distilbert.py#L510)

## Information

Model I am using (Bert, XLNet ...): DistilBert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* run_lm_finetuning.py

The tasks I am working on is:
* Fine tune masked language model

## To reproduce

Steps to reproduce the behavior:

```bash
python run_lm_finetuning.py \
    --output_dir=finetune_output \
    --model_type=distilbert \
    --model_name_or_path=distilbert-base-multilingual-cased \
    --do_train \
    --train_data_file=./finetune_data/train.raw.txt \
    --do_eval \
    --eval_data_file=./finetune_data/val.raw.txt \
    --mlm \
    --block_size=128
```

## Expected behavior

Model should start training process without problem.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.3.0
- Platform: Ubuntu 18
- Python version: 3.6.9
- PyTorch version (GPU): 1.3.1


error while training distilbert multilingual model

hi,
I am trying to finetune distilbert multilingual cased model, but i am getting error while training the model:
while with the same code using distilbert uncased , there is no such error.
Can you please check if there is some problem with distilbert multilingual cased model?
error is:

ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 8 array(s), for inputs ['output_1', 'output_2', 'output_3', 'output_4', 'output_5', 'output_6', 'output_7', 'output_8'] but instead got the following list of 1 arrays: [<tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=int64>]


Added README.md to Swedish BERT models from National Library of Sweden

Following the lead of others these are not actual model cards but rather the README.md-files from https://github.com/Kungbib/swedish-bert-models


Optimize causal mask using torch.where

Instead of multiplying by 1.0 float mask, use torch.where with a bool mask for increased performance.


How to add Dense layer on top of TFBertForSequenceClassification model?

I am having a really hard time adding the dense layers on the top of this model. I have tried to add the layers of `TFBertForSequenceClassification` in a sequential model with some dense layers like this:

```
bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-cased", config=config)

model = keras.models.Sequential()
model.add(bert_model.layers[0])
model.add(keras.layers.Dense(10, activation='relu'))
model.add(keras.layers.Dense(1, activation='sigmoid'))
```
But when I fit the model using:
```
model.fit(
    [padded, attention_mask],
    [np.array(df[1][:2000])],
    epochs=100,
)
```

I am getting this error:
```AttributeError: 'list' object has no attribute 'shape'```

I have also tried to use the layers of the `TFBertForSequenceClassification` in `keras.models.Model() class`. But, again there is no way to get the input layer. For example using `bert_model.layers[0].input_shape` gives the following error:
```
   1571     """
   1572     if not self._inbound_nodes:
-> 1573       raise AttributeError('The layer has never been called '
   1574                            'and thus has no defined input shape.')
   1575     all_input_shapes = set(

AttributeError: The layer has never been called and thus has no defined input shape.
```

What is the right way to add layers on the top of this model.


# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

`Flaubert`

Language I am using the model on (English, Chinese ...):

`French`

The tasks I am working on is: **Fine-tune Flaubert on French-translated SQuAD**

The problem arises when using:

```
python3 ./examples/run_squad.py \
--model_type flaubert \
--model_name_or_path flaubert-base-uncased \
--do_train \
--do_eval \
--do_lower_case \
--train_file SQuAD-v1.1-train_fr_ss999_awstart2_net.json \
--predict_file SQuAD-v1.1-dev_fr_ss999_awstart2_net.json \
--learning_rate 3e-5 \
--num_train_epochs 2 \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir output \
--per_gpu_eval_batch_size=3 \
--per_gpu_train_batch_size=3
```

For some reason the downloaded weights from pre-trained model `flaubert-base-uncased` are not initialized for training:

```python-traceback
2/02/2020 15:10:53 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
02/02/2020 15:10:53 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/config.json from cache at /root/.cache/torch/transformers/d1cf66823bb82e0ef671e7bae75bf86161cbf8ca218f893bc0129599e6e40c2a.e40562626242ae71bf0ce9aa0832297b724c4859407a09771341048981bb3736
02/02/2020 15:10:53 - INFO - transformers.configuration_utils -   Model config FlaubertConfig {
  "amp": 1,
  "architectures": [
    "FlaubertWithLMHeadModel"
  ],
  "asm": false,
  "attention_dropout": 0.1,
  "bos_index": 0,
  "bos_token_id": 0,
  "bptt": 512,
  "causal": false,
  "clip_grad_norm": 5,
  "do_sample": false,
  "dropout": 0.1,
  "emb_dim": 768,
  "embed_init_std": 0.02209708691207961,
  "encoder_only": true,
  "end_n_top": 5,
  "eos_index": 1,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "fp16": true,
  "gelu_activation": true,
  "group_by_size": true,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "id2lang": {
    "0": "fr"
  },
  "init_std": 0.02,
  "is_decoder": false,
  "is_encoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "lang2id": {
    "fr": 0
  },
  "lang_id": 0,
  "langs": [
    "fr"
  ],
  "layer_norm_eps": 1e-12,
  "layerdrop": 0.0,
  "length_penalty": 1.0,
  "lg_sampling_factor": -1,
  "lgs": "fr",
  "mask_index": 5,
  "mask_token_id": 0,
  "max_batch_size": 0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "max_vocab": -1,
  "mlm_steps": [
    [
      "fr",
      null
    ]
  ],
  "model_type": "flaubert",
  "n_heads": 12,
  "n_langs": 1,
  "n_layers": 12,
  "num_beams": 1,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_index": 2,
  "pad_token_id": 0,
  "pre_norm": false,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "sample_alpha": 0,
  "share_inout_emb": true,
  "sinusoidal_embeddings": false,
  "start_n_top": 5,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "first",
  "summary_use_proj": true,
  "temperature": 1.0,
  "tokens_per_batch": -1,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "unk_index": 3,
  "use_bfloat16": false,
  "use_lang_emb": true,
  "vocab_size": 67542,
  "word_blank": 0,
  "word_dropout": 0,
  "word_keep": 0.1,
  "word_mask": 0.8,
  "word_mask_keep_rand": "0.8,0.1,0.1",
  "word_pred": 0.15,
  "word_rand": 0.1,
  "word_shuffle": 0
}

02/02/2020 15:10:53 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/vocab.json from cache at /root/.cache/torch/transformers/8f54ff51875f0422a9c265ab77515058f2655b901caa5f8ff19954c8a126a2fe.4dbbb80764d7ce5ea8639cef2ffdf2c6be3c491192c042bba9651d56b917d49c
02/02/2020 15:10:53 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/merges.txt from cache at /root/.cache/torch/transformers/42f0fe2cd5eebb0c450bd936b0104b27c21e33138b445e9c7124094e05df02f6.5e19e4f2e2e9e11ecde5cc44c2c65f0dc11671ff5dfcd0066699e64bbc7c5a8d
02/02/2020 15:10:53 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/pytorch_model.bin from cache at /root/.cache/torch/transformers/2931084022a5d35320c07628cb7de631bdefe38f0e87d5d48a9e04be799ce0ef.8a02ed26eb9bc391a8fd64b6acce3b2167eb7a01cd4365502dca3a5980918425
02/02/2020 15:11:00 - INFO - transformers.modeling_utils -   Weights of FlaubertForQuestionAnswering not initialized from pretrained model: ['qa_outputs.start_logits.dense.bias', 'qa_outputs.start_logits.dense.weight', 'qa_outputs.end_logits.dense_0.bias', 'qa_outputs.end_logits.dense_0.weight', 'qa_outputs.end_logits.LayerNorm.bias', 'qa_outputs.end_logits.LayerNorm.weight', 'qa_outputs.end_logits.dense_1.bias', 'qa_outputs.end_logits.dense_1.weight', 'qa_outputs.answer_class.dense_0.bias', 'qa_outputs.answer_class.dense_0.weight', 'qa_outputs.answer_class.dense_1.weight']
02/02/2020 15:11:00 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in FlaubertForQuestionAnswering: ['pred_layer.proj.bias', 'pred_layer.proj.weight']
02/02/2020 15:11:06 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir=None, device=device(type='cuda'), do_eval=True, do_lower_case=True, do_train=True, doc_stride=128, eval_all_checkpoints=False, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=3e-05, local_rank=-1, logging_steps=500, max_answer_length=30, max_grad_norm=1.0, max_query_length=64, max_seq_length=384, max_steps=-1, model_name_or_path='flaubert-base-uncased', model_type='flaubert', n_best_size=20, n_gpu=1, no_cuda=False, null_score_diff_threshold=0.0, num_train_epochs=2.0, output_dir='output', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=3, per_gpu_train_batch_size=3, predict_file='SQuAD-v1.1-dev_fr_ss999_awstart2_net.json', save_steps=500, seed=42, server_ip='', server_port='', threads=1, tokenizer_name='', train_file='SQuAD-v1.1-train_fr_ss999_awstart2_net.json', verbose_logging=False, version_2_with_negative=False, warmup_steps=0, weight_decay=0.0)
02/02/2020 15:11:06 - INFO - __main__ -   Creating features from dataset file at .
100%|██████████████████████████████████████████████████████████████████████████████████████| 442/442 [00:42<00:00, 10.49it/s]
convert squad examples to features:   4%|█▉                                             | 3457/84943 [02:18<37:30, 36.21it/s]
```

## To reproduce

Steps to reproduce the behavior:

Edit `run_squad.py` to support `flaubert`:

```python
MODEL_CLASSES = {
    "bert": (BertConfig, BertForQuestionAnswering, BertTokenizer),
    "roberta": (RobertaConfig, RobertaForQuestionAnswering, RobertaTokenizer),
    "xlnet": (XLNetConfig, XLNetForQuestionAnswering, XLNetTokenizer),
    "xlm": (XLMConfig, XLMForQuestionAnswering, XLMTokenizer),
    "distilbert": (DistilBertConfig, DistilBertForQuestionAnswering, DistilBertTokenizer),
    "albert": (AlbertConfig, AlbertForQuestionAnswering, AlbertTokenizer),
    "flaubert": (FlaubertConfig, FlaubertForQuestionAnswering, FlaubertTokenizer),
}
```

I had to do some other little edits. Then execute script:

```bash
python3 ./examples/run_squad.py \
--model_type flaubert \
--model_name_or_path flaubert-base-uncased \
--do_train \
--do_eval \
--do_lower_case \
--train_file SQuAD-v1.1-train_fr_ss999_awstart2_net.json \
--predict_file SQuAD-v1.1-dev_fr_ss999_awstart2_net.json \
--learning_rate 3e-5 \
--num_train_epochs 2 \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir output \
--per_gpu_eval_batch_size=3 \
--per_gpu_train_batch_size=3
```

Dataset available here: https://github.com/Alikabbadj/French-SQuAD

## Expected behavior

Load weights from pre-trained `flaubert-base-uncased` model to fine-tune on FR SQuAD train then use new trained weights to evaluate model on FR SQuAD dev.

## Environment info

- `transformers` version: `transformers==2.4.1`
- Platform: `Deep Learning AMI (Ubuntu 16.04) Version 26.0 `