TF2 version of Multilingual DistilBERT throws an exception  [TensorFlow 2]

## 🐛 Bug

I'm finding that several of the TensorFlow 2.0 Sequence Classification models don't seem to work.  Case in point:  `distilbert-base-uncased` works but `distilbert-base-multilingual-cased` does not.

My environment is:
* Platform Linux-4.15.0-65-generic-x86_64-with-Ubuntu-18.04-bionic
* Python 3.6.8 (default, Oct  7 2019, 12:59:55)
* [GCC 8.3.0]
* Tensorflow 2.0.0

Note that I am using v2.3.0 of `transformers` with patch [1efc208](https://github.com/huggingface/transformers/commit/1efc208ff386fb6df56302c8f6f9484ddf93b92a) applied to work around [this issue](https://github.com/huggingface/transformers/issues/2251).

However, problems with `distilbert-base-multilingual-cased` occur in v2.2.0, as well.

Here is code to reproduce the problem.

```
# define constants
MODEL_NAME = 'distilbert-base-multilingual-cased'  # DOES NOT WORK
# MODEL_NAME = 'distilbert-base-uncased'   # WORKS if uncommented

BATCH_SIZE=6
MAX_SEQ_LEN = 500

# imports and setup
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID";
os.environ["CUDA_VISIBLE_DEVICES"]="0";
import tensorflow as tf
from transformers import glue_convert_examples_to_features
from transformers import BertConfig, TFBertForSequenceClassification, BertTokenizer
from transformers import XLNetConfig, TFXLNetForSequenceClassification, XLNetTokenizer
from transformers import XLMConfig, TFXLMForSequenceClassification, XLMTokenizer
from transformers import RobertaConfig, TFRobertaForSequenceClassification, RobertaTokenizer
from transformers import DistilBertConfig, TFDistilBertForSequenceClassification, DistilBertTokenizer
from transformers import AlbertConfig, TFAlbertForSequenceClassification, AlbertTokenizer

TRANSFORMER_MODELS = {
    'bert':       (BertConfig, TFBertForSequenceClassification, BertTokenizer),
    'xlnet':      (XLNetConfig, TFXLNetForSequenceClassification, XLNetTokenizer),
    'xlm':        (XLMConfig, TFXLMForSequenceClassification, XLMTokenizer),
    'roberta':    (RobertaConfig, TFRobertaForSequenceClassification, RobertaTokenizer),
    'distilbert': (DistilBertConfig, TFDistilBertForSequenceClassification, DistilBertTokenizer),
    'albert':     (AlbertConfig, TFAlbertForSequenceClassification, AlbertTokenizer),
}

def classes_from_name(model_name):
    name = model_name.split('-')[0]
    return TRANSFORMER_MODELS[name]

# setup model and tokenizer
(config_class, model_class, tokenizer_class) = classes_from_name(MODEL_NAME)
tokenizer = tokenizer_class.from_pretrained(MODEL_NAME)
model = model_class.from_pretrained(MODEL_NAME)

# construct binary classification dataset
categories = ['alt.atheism', 'comp.graphics']
from sklearn.datasets import fetch_20newsgroups
train_b = fetch_20newsgroups(subset='train',
   categories=categories, shuffle=True, random_state=42)
test_b = fetch_20newsgroups(subset='test',
   categories=categories, shuffle=True, random_state=42)

print('size of training set: %s' % (len(train_b['data'])))
print('size of validation set: %s' % (len(test_b['data'])))
print('classes: %s' % (train_b.target_names))

x_train = train_b.data
y_train = train_b.target
x_test = test_b.data
y_test = test_b.target

train_csv = [(i, text, y_train[i]) for i, text in enumerate(x_train)]
valid_csv = [(i, text, y_test[i]) for i, text in enumerate(x_test)]

def convert_to_tfdataset(csv):
    def gen():
        for ex in csv:
            yield  {'idx': ex[0],
                     'sentence': ex[1],
                     'label': str(ex[2])}
    return tf.data.Dataset.from_generator(gen,
        {'idx': tf.int64,
          'sentence': tf.string,
          'label': tf.int64})
trn = convert_to_tfdataset(train_csv)
val = convert_to_tfdataset(valid_csv)

# preprocess datasets
train_dataset = glue_convert_examples_to_features(examples=trn, tokenizer=tokenizer
                                                  , max_length=MAX_SEQ_LEN, task='sst-2'
                                                  , label_list =['0', '1'])
valid_dataset = glue_convert_examples_to_features(examples=val, tokenizer=tokenizer
                                                  , max_length=MAX_SEQ_LEN, task='sst-2'
                                                  , label_list =['0', '1'])
train_dataset = train_dataset.shuffle(len(train_csv)).batch(BATCH_SIZE).repeat(-1)
valid_dataset = valid_dataset.batch(BATCH_SIZE)

# train model
opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=opt, loss=loss, metrics=[metric])
history = model.fit(train_dataset, epochs=1, steps_per_epoch=len(train_csv)//BATCH_SIZE,
                    validation_data=valid_dataset, validation_steps=len(valid_csv)//BATCH_SIZE)
```

The code above produces the following error:
```
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_utils.py in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
    529                        'Expected to see ' + str(len(names)) + ' array(s), '
    530                        'but instead got the following list of ' +
--> 531                        str(len(data)) + ' arrays: ' + str(data)[:200] + '...')
    532     elif len(names) > 1:
    533       raise ValueError('Error when checking model ' + exception_prefix +

ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 8 array(s), but instead got the following list of 1 arrays: [<tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=int64>]...
```

However, if you set MODEL_NAME to `distilbert-base-uncased`, everything works.

Other models that I've found do not work in TF2 include `xlnet-base-cased`.  To reproduce, set MODEL_NAME to  `xlnet-base-cased` in the code above.  The `xlnet-base-cased` model also throws an exception during the call to `model.fit`.


For Hugging Face transformer's hidden_states output, is the first hidden state tensor that is returned the out of the embeddings?

According to the Hugging Face Transformer documentation for the GPT2DoubleHeadsModel (under the 'output' section)

```
hidden_states: (optional, returned when config.output_hidden_states=True)
list of torch.FloatTensor (one for the output of each layer + the output of the embeddings)
```

So in this case, would the first hidden_states tensor (index of 0) that is returned be the output of the embeddings, or would the very last hidden_states tensor that is returned be the output of the embeddings?

I am confused about the order in which the hidden_states tensors are returned, because the documentation seem to indicate that the output of the embeddings is the last hidden_state tensor that is returned.

Thank you,


Fine-tuning pretrained BERT model using own dataset but with same training task

## ❓ Questions & Help

I would like to finetune a pretrained model using the same task as the original model was trained on, so this means that I want the model to predict masked words and do next sentence prediction. Is there anywhere some code snippet that achieves this or gives an idea on how I can implement this?


Update pipelines.py

Core: Pipeline

Internals of the library; Pipeline.

Modified QA pipeline to consider all features for each example before generating topk answers.
Current pipeline only takes one SquadExample, one SquadFeature, one start logit list, one end logit list to retrieve the answer, this is not correct as one SquadExample can produce multiple SquadFeatures.


Update QA pipeline


New SQuAD API for distillation script

The squad distillation script is still using methods from files that do not exist anymore (utils_squad and utils_squad_evaluate).

I updated the script to use the newer API.


Adding usage example with Tensorflow

Simple training and fine-tuning example of DistilBERT in a Colab.


## 🐛 Bug

<!-- Important information -->

Model I am using ROBERTa
ROBERTa model wrong padding for token_type_ids field if return_tensors=True.

Language I am using the model on English:

The problem arise when using:
* [ ] the official example scripts: (give details)
* [* ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ *] my own task or dataset: (give details)

## To Reproduce

Please run following code
```
from transformers import pipeline, AutoModel, AutoTokenizer
import torch

model_name = 'roberta-base'
tokenizer = AutoTokenizer.from_pretrained(model_name)
corpus = ['this is a test', 'this is another test example', 'one']

toks = tokenizer.batch_encode_plus(corpus, add_special_tokens=True, max_length=128)
print(toks)
encoded = model(**{k:v.cuda() for k, v in toks.items()}) #crash will be here
```

Steps to reproduce the behavior:

1. Run code.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-7ba1420d7b7f> in <module>
----> 1 encoded = model(**{k:v.cuda() for k, v in toks.items()})

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    733             head_mask = [None] * self.config.num_hidden_layers
    734
--> 735         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    736         encoder_outputs = self.encoder(embedding_output,
    737                                        attention_mask=extended_attention_mask,

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     68                                                       token_type_ids=token_type_ids,
     69                                                       position_ids=position_ids,
---> 70                                                       inputs_embeds=inputs_embeds)
     71
     72

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    188         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    189
--> 190         embeddings = inputs_embeds + position_embeddings + token_type_embeddings
    191         embeddings = self.LayerNorm(embeddings)
    192         embeddings = self.dropout(embeddings)

RuntimeError: CUDA error: device-side assert triggered
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
Crash happens because to convert lists to tensors it makes padding with value 1.
`padded_value = [v + [self.pad_token_id if key == 'input_ids' else 1] * (max_seq_len - len(v)) for v in padded_value]`
It's probably wrong strategy for BERT like models for field `token_type_ids` where 1 means next sentence token.

It might be wrong logic for attention_mask also because it should be 0 for non meaningful tokens. You should use return_attention_masks which is not enabled by default and also crashes on my machine.

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-ede112fd760a> in <module>
      1 corpus = ['this is a test', 'this is another test example', 'one']
----> 2 toks = tokenizer.batch_encode_plus(corpus, add_special_tokens=True, max_length=128, return_attention_masks=True, return_tensors='pt')
      3 toks

~/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, max_length, stride, truncation_strategy, return_tensors, return_input_lengths, return_attention_masks, **kwargs)
    971         if return_attention_masks:
    972             if is_tf_available():
--> 973                 batch_outputs['attention_mask'] = tf.abs(batch_outputs['attention_mask'] - 1)
    974             else:
    975                 batch_outputs['attention_mask'] = torch.abs(batch_outputs['attention_mask'] - 1)

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/util/dispatch.py in wrapper(*args, **kwargs)
    178     """Call target, and fall back on dispatchers if there is a TypeError."""
    179     try:
--> 180       return target(*args, **kwargs)
    181     except (TypeError, ValueError):
    182       # Note: convert_to_eager_tensor currently raises a ValueError, not a

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/ops/math_ops.py in abs(x, name)
    273   """
    274   with ops.name_scope(name, "Abs", [x]) as name:
--> 275     x = ops.convert_to_tensor(x, name="x")
    276     if x.dtype.is_complex:
    277       return gen_math_ops.complex_abs(x, Tout=x.dtype.real_dtype, name=name)

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py in convert_to_tensor(value, dtype, name, preferred_dtype, dtype_hint)
   1182   preferred_dtype = deprecation.deprecated_argument_lookup(
   1183       "dtype_hint", dtype_hint, "preferred_dtype", preferred_dtype)
-> 1184   return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
   1185
   1186

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py in convert_to_tensor_v2(value, dtype, dtype_hint, name)
   1240       name=name,
   1241       preferred_dtype=dtype_hint,
-> 1242       as_ref=False)
   1243
   1244

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx, accept_composite_tensors)
   1294
   1295     if ret is None:
-> 1296       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
   1297
   1298     if ret is NotImplemented:

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
    284                                          as_ref=False):
    285   _ = as_ref
--> 286   return constant(v, dtype=dtype, name=name)
    287
    288

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py in constant(value, dtype, shape, name)
    225   """
    226   return _constant_impl(value, dtype, shape, name, verify_shape=False,
--> 227                         allow_broadcast=True)
    228
    229

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py in _constant_impl(value, dtype, shape, name, verify_shape, allow_broadcast)
    233   ctx = context.context()
    234   if ctx.executing_eagerly():
--> 235     t = convert_to_eager_tensor(value, ctx, dtype)
    236     if shape is None:
    237       return t

~/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py in convert_to_eager_tensor(value, ctx, dtype)
     94       dtype = dtypes.as_dtype(dtype).as_datatype_enum
     95   ctx.ensure_initialized()
---> 96   return ops.EagerTensor(value, ctx.device_name, dtype)
     97
     98

ValueError: Attempt to convert a value (tensor([[-1, -1, -1, -1, -1, -1,  0],
        [-1, -1, -1, -1, -1, -1, -1],
        [-1, -1, -1,  0,  0,  0,  0]])) with an unsupported type (<class 'torch.Tensor'>) to a Tensor.
```

## Environment

* OS: Ubuntu 18.04
* Python version: Python 3.6.5 :: Anaconda, Inc.
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU ? Yes
* Distributed or parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Add XLM-RoBERTa model for TF2

Hello,

I have implemented the XLM-RoBERTa model handling for Tensorflow 2.


Installation of Transformers without Sacremoses

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi HuggingFace Team!

I was checking the dependencies of this library, and I found that sacremoses does not have an accepted licence type for my system. The setup.py file says that it's needed for XLM. If I don't plan on using XLM would I be able to modify the setup.py and remove the sacremoses requirement?

Thanks!
Zander


Remove redundant hidden states

The quickstart showcasing the usage of the Model2Model currently fails. This is due to a positional argument that should be a named argument.

As I understand it, the `encoder_hidden_states` are already present in the `kwargs_decoder` dictionary, there is therefore no need to pass it to the decoder forward call.

With the current quickstart example this crashes as the position of the `encoder_hidden_states` means it's passed as an `attention_mask`.

Please correct me if I'm wrong @rlouf @thomwolf


Add check for token_type_ids before tensorizing

Fix an issue where `prepare_for_model()` gives a `KeyError` when
`return_token_type_ids` is set to `False` and `return_tensors` is
enabled.


Error when running run_generation.py

I tried to run this code:
python ./examples/run_generation.py --model_type=gpt2 --length=20 --model_name_or_path=gpt2

However I am getting the error below:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\GPT2\\.cache\\torch\\transformers\\tmpy2recb0u' -> 'C:\\Users\\GPT2\\.cache\\torch\\transformers\\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./examples/run_generation.py", line 237, in <module>
    main()
  File "./examples/run_generation.py", line 200, in main
    tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path)
  File "C:\gpt2\venv\lib\site-packages\transformers\tokenization_utils.py", line 309, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "C:\gpt2\venv\lib\site-packages\transformers\tokenization_utils.py", line 415, in _from_pretrained
    raise EnvironmentError(msg)
OSError: Couldn't reach server at '{}' to download vocabulary files.

How do I get over this hump? Thanks


Evaluation not working on distilbert-base-uncased-distilled-squad

## 🐛 Bug

<!-- Important information -->

Model I am using DistilBert: distilbert-base-uncased-distilled-squad

Language I am using the model on English:

The problem arise when using:
* [x] the official example scripts: run_squad.py in examples

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQuAD 1.1 and SQuAD2.0 dev dataset

## To Reproduce

Steps to reproduce the behavior:

1. python run_squad.py --model_type distilbert     --model_name_or_path distilbert-base-uncased-distilled-squad     --do_eval     --do_lower_case     --predict_file $SQUAD_DIR/dev-v2.0.json --max_seq_length 384     --doc_stride 128     --output_dir ./distill_squad/     --per_gpu_eval_batch_size=4    --version_2_with_negative

2. python run_squad.py --model_type distilbert     --model_name_or_path distilbert-base-uncased-distilled-squad     --do_eval     --do_lower_case     --predict_file $SQUAD_DIR/dev-v1.1.json --max_seq_length 384     --doc_stride 128     --output_dir ./distill_squad/     --per_gpu_eval_batch_size=4

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Results on evaluation data
```python
{
"exact": 80.4177545691906,
"f1": 84.07154997729623,
"total": 11873,
"HasAns_exact": 76.73751686909581,
"HasAns_f1": 84.05558584352873,
"HasAns_total": 5928,
"NoAns_exact": 84.0874684608915,
"NoAns_f1": 84.0874684608915,
"NoAns_total": 5945
}
```
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: CentOS Linux
* Python version: 3.6.9
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU: yes
* Distributed or parallel setup: None
* Any other relevant information:

## Additional context
```code
01/08/2020 08:41:52 - INFO - __main__ -   ***** Running evaluation  *****
01/08/2020 08:41:52 - INFO - __main__ -     Num examples = 10833
01/08/2020 08:41:52 - INFO - __main__ -     Batch size = 4
Evaluating:   0%|                                                                                | 0/2709 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_squad.py", line 815, in <module>
    main()
  File "run_squad.py", line 804, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "run_squad.py", line 323, in evaluate
    outputs = model(**inputs)
  File "/media/data2/anaconda/envs/distill2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'token_type_ids'
```
<!-- Add any other context about the problem here. -->

I just changed the model to bert and model_name to bert-base-uncased. It is working fine. I think there is some problem with distilbert model. Can you please help me on this?


Tokenizer methods and padding

## ❓ Questions & Help

I wanted to know whether there was a perticular reason why the `get_special_tokens_mask` method of the tokenizer does not also return as mask over padding tokens, only <CLS> and <SEP> tokens, in the case where `already_has_special_tokens=True` ? I had to rewrite a custom function for my usecase, but it seemed off.

Also, I think there should be an additional `padding` kwarg in the method, which if provided would return a longer mask then the sum of lenfgths of `token_ids_0` and `token_ids_1`, in the case where `already_has_special_tokens=False`. The same should be true for `build_inputs_with_special_tokens` IMO. What do yoy think ?


Reproducibility problem with DistilBERT paper

## ❓ Questions & Help

We are currently working a follow-up to your work “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”. We’ve noticed some skeptical data in Table 1 of your paper. On MRPC, the reported averaged F1 and acc result is 90.2, which is even ~2% higher than BERT-base (teacher). We carefully reproduce your experiment with your code and pretrained checkpoint provided in huggingface/transformers on Github. Our reproduced result is 89.6/85.5, which means the averaged F1 and acc should be 87.55, which is very different from your reported result. With all due respect, we personally think you may have mistakenly report the F1 score instead of averaged F1 & acc. Another evidence is your previous blog (https://user-images.githubusercontent.com/16107619/64210993-c0ef1b80-ce72-11e9-806b-171313e8ae9e.png) and DistilRoBERTa, which has a much lower MRPC score of 86.6 (https://github.com/huggingface/transformers/tree/master/examples/distillation). We list your reported results and our reproduced results and reproduced results on GLUE dev set:

DistillBERT on GLUE Dev Set | CoLA | MNLI-m | MNLI-mm | MRPC | QNLI | QQP | RTE | SST-2 | STS-B
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
DistilBERT Blog | 42.5 | 81.6 | 81.1 | 85.35(88.3/82.4) | 85.5 | 89.15(87.7/90.6) | 60.0 | 92.7 | 84.75(84.5/85.0)
DistilBERT paper | 49.1 | 81.8 |   | 90.2 | 90.2 | 89.2 | 62.9 | 92.7 | 90.7
Our reproduced | 43.0 | - | - | 87.55(89.6/85.5) | 85.8 | - | - | - | 80.53(80.6/80.5)

According to our experiment, the result is actually very close to the previous results you reported on your blog. We are not able to reproduce results reported in your paper though we have tried some hyperparameter tuning. We will really appreciate it if you can confirm the result in your paper or send us the hyperparameters to reproduce the results.


## ❓ Questions & Help

I am receiving the error RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows.
What can I do to increase this source sentence length constraint?

<img width="1316" alt="Screen Shot 2020-01-08 at 2 01 49 pm" src="https://user-images.githubusercontent.com/44693666/71954228-8ed63780-321f-11ea-84ef-eac4519235c4.png">


Error occurs in XLMRobertaModel when token_type_ids is given.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLMRoberta

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* the official example scripts: (give details)
![image](https://user-images.githubusercontent.com/6512394/71953192-c2679080-3224-11ea-85f9-5b4e269e7956.png)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce
Steps to reproduce the behavior:

```
>>> import torch
>>> from transformers import XLMRobertaModel
>>> model = XLMRobertaModel.from_pretrained('xlm-roberta-base', cache_dir="cache_dir")
>>> input_ids = torch.tensor([[0, 164, 100231, 135758, 32, 2, 2, 157, 217, 164, 10869, 5, 2]])
>>> outputs = model(input_ids)
>>> outputs[0].size()
torch.Size([1, 13, 768])
>>>
>>> token_type_ids = torch.tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]])
>>> outputs = model(input_ids, token_type_ids=token_type_ids)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 735, in forward
    embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 70, in forward
    inputs_embeds=inputs_embeds)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 188, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 1 out of table with 0 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU ?: no
* Distributed or parallel setup ?: no
* Any other relevant information:


Update

本家のupdateを取り込みました。


porting XLM-Roberta to tensorflow 2.0

## ❓ Questions & Help

Yesterday I have ported XLM-Roberta from pytorch to tensorflow mainly following the instruction provided in [huggingface/from-tensorflow-to-pytorch](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28).
The final error is computed using the DUMMY_INPUT as input values for the large MaskedLM model and is evaluated on the final prediction_score output.
I compute the error as:
```python
    max_absolute_diff = np.amax(np.abs(tf_model_out.numpy() - pt_model_out.detach().numpy()))
```
and the final output is 0.00027179718; which is lower than the suggested 1e^-3 bound.

According to your indication the error seems to be acceptable, given that XLM-Roberta is a huge model. However, I have experienced some huge output difference when I do not specify the position_ids. That is, the position_ids computed by the TFRobertaEmbeddings seems to be correct since they correctly take in consideration the presence of some pad tokens using the ``create_position_ids_from_input_ids`` function. Instead the PyTorch RobertaEmbeddings doesn't.

Moreover, I'm also wandering if it is possible to merge the interface of the TF models with the PyTorch models. Not sure if it is worth it, but by using the __call__ and call function provided by TF2.0 it is possible to obtain an equivalent interface between the 2 frameworks.
For example:
```python
class TFXLMRobertaForMaskedLM(TFXLMRobertaPreTrainedModel):
    def __call__(self, input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, **kwargs):
        inputs = (input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds)
        return super(TFXLMRobertaForMaskedLM, self).__call__(inputs, **kwargs)

    def call(self, inputs, **kwargs):
        outputs = self.xlm_roberta(*inputs, **kwargs)

        sequence_output = outputs[0]
        prediction_scores = self.lm_head(sequence_output)
        outputs = (prediction_scores,) + outputs[2:]
        return outputs  # prediction_scores, (hidden_states), (attentions)
```
should be equivalent to the PyTorch implementation


loss_fct = CrossEntropyLoss(ignore_index=-1) for BERT/RoBERTa MaksedLM

## 🐛 Bug

<!-- Important information -->

The models I am using (Bert, RoBERTa....):

Language I am using the model on (English):

The problem arise when I tried to fine-tune the model using `MaskedLM` given the `masked_lm_labels`:

It seems that the model forward loop specifies that `loss_fct = CrossEntropyLoss(ignore_index=-1)` where the instructions previously stated masked ids are -100. This gives a "device-side assert triggered " error for GPU training and "Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97" for CPU training.

* [modeling_bert.py / modeling_roberta.py ] the official example scripts: for `RobertaForMaskedLM` / `BertForMaskedLM` we have `loss_fct = CrossEntropyLoss(ignore_index=-1)`
* [ ] my own modified scripts: set ignore_index = -100 or simply remove it

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. run the "run_lm_finetuning.py" file in the examples
2. It seems that if we use `pip install transformers` and get transformers 2.3.0  We would have this error. If installing from source code, the error is gone
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Linux 18.04.3
* Python version:  3.6
* PyTorch version:  1.3.1
* PyTorch Transformers version (or branch):  2.3.0
* Using GPU ? yes
* Distributed or parallel setup ? nope
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


is pytorch-pretrained-bert still being maintained in the future?

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

Details of the issue:

<!-- A clear and concise description of the migration issue. If you have code snippets, please provide it here as well. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed or parallel setup ?
* Any other relevant information:

## Checklist

- [ ] I have read the migration guide in the readme.
- [ ] I checked if a related official extension example runs on my machine.

## Additional context

<!-- Add any other context about the problem here. -->


DistilBertForSequenceClassification returning nans

## DistilBertForSequenceClassification returning NaNs
<!-- A clear and concise description of the question. -->
DistilBertForSequenceClassification using the distilbert-base-uncased is returning Nans for both the logits and loss. Has anyone encountered this issue?


Generating text with fine-tuned TFGPT2LMHeadModel in python.

I've finetuned GPT2 using the following script:
```
from transformers import GPT2Tokenizer, TFGPT2LMHeadModel
import tensorflow as tf

model = TFGPT2LMHeadModel.from_pretrained("distilgpt2")
tokenizer = GPT2Tokenizer.from_pretrained("distilgpt2")

file_path = 'text.txt'
with open(file_path, encoding="utf-8") as f:
    text = f.read()

tokenized_text = tokenizer.encode(text)

examples = []
block_size = 100
for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
    examples.append(tokenized_text[i:i + block_size])

inputs, labels = [], []
for ex in examples:
    inputs.append(ex[:-1])
    labels.append(ex[1:])

dataset = tf.data.Dataset.from_tensor_slices((inputs, labels))

BATCH_SIZE = 8
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=[loss, *[None] * model.config.n_layer], metrics=[metric])
model.fit(dataset, epochs=20)
```
This runs fine, and after 20 epochs I have an accuracy of ~0.59.
The problem comes when I tried to write my own text generation script:
```
def generate_text(model, tokenizer, start_string, num_generate):

        input_eval = tf.expand_dims(tokenizer.encode(start_string), 0)
        token_ids = []

        for i in range(num_generate):
            predictions = tf.squeeze(model.predict(input_eval)[0], 0)

            predicted_id = tf.random.categorical(predictions, 1)[-1, 0].numpy().item()

            input_eval = tf.expand_dims([predicted_id], 0)

            token_ids.append(predicted_id)

        return start_string + tokenizer.decode(token_ids)
```
I get output, but the output is of a sufficiently lower quality than when I train a model using "run_lm_finetuning.py" and generate text using "run_generation.py"
I looked into the example generation script, and it looks like there is simply a call to "model.generate(...)"

Where does this model.generate() method exist?


Fix typograpical errors

Fixed few typos.


Add CamemBERT model for TF2

Hello,

Here another contribution :) I have implemented the CamemBERT model handling for Tensorflow 2.

I now have the model on my disk, should I send it to you? Or will you generate it from your side? Or should I upload it on my account? As you wish :)

Best.

Julien.


Added repetition penalty to PPLM example

It was giving awful results, so I added repetition penalty which improved things.


update the config.is_decoder=True before initialize the decoder

Currently the PreTrainedEncoderDecoder class fails to initialize the "cross-attention layer" since it updates decoder.config.is_decoder = True after decoder initialization.


spelling correction


make test problem

## ❓ Questions & Help
Hello all,

I recently installed this library/module and wanted to run a test with: `make test`
However, things do not go right and I got the following message:

> python -m pytest -n auto --dist=loadfile -s -v ./tests/
> /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named pytest
> make: *** [test] Error 1

I'm using macOS X Catalina, with Python3.7 (3.7.5), pytest is installed.
(I have no clue why it returns an error on Py2.7 )

Thanks in advance


Fix misleading RoBERTa token type ids

RoBERTa does not actually make use of token type ids. When feeding the output of `encode_plus` used with a pair of sequences to the model directly, it crashes as it cannot handle token type ids that have a value of 1.

This fix returns a list of zeros as the token type ids instead.


How can I fine-tune XLM for sentence classification?

## ❓ Questions & Help

I am using the `XLMTokenizer` and `XLMForSequenceClassification` for fine-tuning the `xlm-mlm-en-2048` model to work on a sentence classification problem.
I am using the same configuration as the one that I have used for fine-tuning BERT.
Surprisingly, XLM seems not to be improving at all (The loss is decreasing but the accuracy isn't affected!).
Actually, the model has overfitted to always select the dominant class in the dataset!
```
EPOCH 0:
Iteration: 0. Loss: 1.0213737487792969. Accuracy: 66.17283950617283%
Iteration: 1000. Loss: 0.9081503748893738. Accuracy: 66.29629629629629%
EPOCH 1:
Iteration: 0. Loss: 0.6950288414955139. Accuracy: 66.29629629629629%
Iteration: 1000. Loss: 0.648954451084137. Accuracy: 66.29629629629629%
EPOCH 2:
Iteration: 0. Loss: 0.7168332934379578. Accuracy: 66.29629629629629%
Iteration: 1000. Loss: 0.38551628589630127. Accuracy: 66.29629629629629%
```

The function used to tokenize a sentence is:
```
def prepare_features(tokenizer, seq_1, max_seq_length = 100,
             zero_pad = True, include_CLS_token = True, include_SEP_token = True):
    ## Tokenzine Input
    tokens_a = tokenizer.tokenize(seq_1)

    ## Truncate
    if len(tokens_a) > max_seq_length - 2:
        tokens_a = tokens_a[0:(max_seq_length - 2)]
    ## Initialize Tokens
    tokens = []
    if include_CLS_token:
        tokens.append(tokenizer.cls_token)
    ## Add Tokens and separators
    for token in tokens_a:
        tokens.append(token)

    if include_SEP_token:
        tokens.append(tokenizer.sep_token)

    input_ids = tokenizer.convert_tokens_to_ids(tokens)
    ## Input Mask
    input_mask = [1] * len(input_ids)
    ## Zero-pad sequence length
    if zero_pad:
        while len(input_ids) < max_seq_length:
            input_ids.append(0)
            input_mask.append(0)
    return torch.tensor(input_ids).squeeze(0), input_mask
```

What do you advise me to do in order to investigate this strange result?

Thanks,
Amr


T5_INPUTS_DOCSTRING correct!?

Is this docstring even correct?

The [CLS] and [SEP] Tokens do not appear in the from google provided dictionaries for the pretrained "t5-base" model. If this is true could you then please provide a correct example of how to use the text generation feature of this almighty transformer? (BoolQ or QA) That would help me a lot.

Thanks!

```
T5_INPUTS_DOCSTRING = r"""
    Inputs:
        **input_ids**: ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
            Indices of input sequence tokens in the vocabulary.
            To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
            (a) For sequence pairs:
                ``tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]``
            (b) For single sequences:
                ``tokens:         [CLS] the dog is hairy . [SEP]``
            T5 is a model with relative position embeddings so you should be able to pad the inputs on
            the right or the left.
            Indices can be obtained using :class:`transformers.T5Tokenizer`.
            See :func:`transformers.PreTrainedTokenizer.encode` and
            :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details.
        **attention_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(batch_size, sequence_length)``:
            Mask to avoid performing attention on padding token indices.
            Mask values selected in ``[0, 1]``:
            ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
        **head_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(num_heads,)`` or ``(num_layers, num_heads)``:
            Mask to nullify selected heads of the self-attention modules.
            Mask values selected in ``[0, 1]``:
            ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**.
"""

```


It occurs error when python run_lm_finetuning.py

- Environment
> - python 3.6.9
> - torch 1.1.0
> - have installed transformers
- Command
> python run_lm_finetuning.py     --output_dir=output     --model_type=bert     --model_name_or_path=bert-base-uncased     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm
- Error
> Traceback (most recent call last):
  File "run_lm_finetuning.py", line 498, in <module>
    main()
  File "run_lm_finetuning.py", line 447, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 96, in load_and_cache_examples
    dataset = TextDataset(tokenizer, file_path=args.eval_data_file if evaluate else args.train_data_file, block_size=args.block_size)
  File "run_lm_finetuning.py", line 78, in __init__
    self.examples.append(tokenizer.add_special_tokens_single_sequence(tokenized_text[:block_size]))
**AttributeError: 'BertTokenizer' object has no attribute 'add_special_tokens_single_sequence'**
- Help
> How to deal it?


Padding part output in BERT NER task is not [PAD]?

## ❓ Questions & Help

Hello ,my friends.

I have a problem while I am trying to do a Chinese NER task.

For easy to understand I will use some English words instead.

Assume padding length =128, here is the sentence:

> [CLS] Marilyn Monroe is an famous actress. [SEP] [PAD]  [PAD]  ...  [PAD]

After I put it into `BertForTokenClassification` , I got a output like:

>[CLS] Marilyn Monroe is an famous actress.
>[CLS] B-PER I-PER O O O O O

That looks good ,but ,when output reached padding area it become strange:

>[SEP] [PAD] [PAD] [PAD] [PAD] [PAD] ...  [PAD]
>O O O O O  [CLS] [CLS] O ...O

it seems padding area including [SEP] token output randomly (they should be [SEP] and [PAD]), and most of them are 'O' with a littel [CLS] or [B-PER]\[I-PER].

I am confused about that.

I am sure taht:

- I already set `attention_masks` on padding area while training, attention_masks on [PAD] are 0.

- I already set `token_type_ids`, token_type_ids on [PAD] are 1.

and also in eval with `attention_mask` and `token_type_ids`.

So that's what I am depressed about, I have to cut output padding area before I solve this problem, I don't think it's a good idea.

Can someone help me? :(


ALBERT model does not work as expected

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,
I am new to Transformers. I tried the example for class AlbertForQuestionAnswering from huggingface.co. The results in each run are different and not correct. Please help.
Thanks,
Tuan Anh


Make doc regarding masked indices more clear

See this [issue](https://github.com/huggingface/transformers/issues/2418) for details, basically there used to be different ways of specifying masked indices (either -1 or -100), which was fixed by this [commit](https://github.com/huggingface/transformers/commit/418589244d263087f1d48655f621a65f2a5fcba6).

However the doc remains unclear, this PR fixes this.

I had an issue originally because I was using a version which did not incorporate the uniformisation yet.


Tokenize whole sentence vs. tokenize words in sentence then concat

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When using En-Fr XLMModel in transformers library,
I found that result from tokenizing a whole sentence is different from tokenizing words in the sentence and then concatenate.

My configuration is as below
**(XLMModel, XLMTokenizer, XLMConfig, 'xlm-mlm-enfr-1024')**

The result is as below
![image](https://user-images.githubusercontent.com/12389429/71879066-84a93000-3170-11ea-91fe-25227ebe478e.png)

The ultimate goal is to 'detokenize' tokenized sentence which is
['I', 'love', 'swim', '##ing'] -> ['I', 'love', 'swimming']
In order to do this, I have to know the raw token's index for each tokenized tokens.

It would be great if anyone can help with this problem.


convert tf ckpt to pytorch_model.bin, load back model(TFBertModel), will loss params

```
import os
pretrained_path = 'Models/chinese_L-12_H-768_A-12'
config_path = os.path.join(pretrained_path, 'bert_config.json')
checkpoint_path = os.path.join(pretrained_path, 'bert_model.ckpt.index')

config = BertConfig.from_pretrained(config_path)
model = BertForPreTraining.from_pretrained(checkpoint_path, from_tf=True, config=config)
model.save_pretrained('Models/chinese')
```
INFO:transformers.configuration_utils:Configuration saved in Models/chinese/config.json
INFO:transformers.modeling_utils:Model weights saved in Models/chinese/pytorch_model.bin

The load the save model:

```
# 加载模型
config = BertConfig.from_json_file("Models/chinese/config.json")
tfmodel = TFBertModel.from_pretrained('Models/chinese/',from_pt=True, config=config)
```
INFO:transformers.modeling_tf_utils:loading weights file Models/chinese/pytorch_model.bin
INFO:transformers.modeling_tf_pytorch_utils:Loading PyTorch weights from /home/work/Bert/Models/chinese/pytorch_model.bin
INFO:transformers.modeling_tf_pytorch_utils:PyTorch checkpoint contains 119,108,746 parameters
INFO:transformers.modeling_tf_pytorch_utils:Loaded 102,267,648 parameters in the TF 2.0 model.
INFO:transformers.modeling_tf_pytorch_utils:Weights or buffers not loaded from PyTorch model: {'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias'}

PyTorch checkpoint contains 119,108,746 parameters  To  Loaded 102,267,648 parameters in the TF 2.0 model.


[DistillBERT] tokenizer issue of multilingual-cased

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):  DistilBERT

Language I am using the model on (English, Chinese....): Korean

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

When I tokenize the korean by `transformers.DistilBertTokenizer` with the **bert-base-multilingual-cased** vocab. Every token in korean mapped to [UNK].

```python
from transformers import DistilBertTokenizer

ko_text = "CNP차앤박화장품 역시 국내 대표 ‘피부과 출신’ 화장품 브랜드다. CNP차앤박화장품‘프로폴리스 앰플 오일 인 크림’은 브랜드 베스트셀러인 프로폴리스 에너지 앰플에 오일을 함유해 보습 기능을 강화한 제품이다."

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')
print(tokenizer.tokenize(ko_text))
```
Results
['[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '.', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '[UNK]', '.']

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

However, when tokenizing the korean by  `transformers.BertTokenizer` with the **bert-base-multilingual-cased** vocab returns expected results.

```python
from transformers import BertTokenizer

ko_text = "CNP차앤박화장품 역시 국내 대표 ‘피부과 출신’ 화장품 브랜드다. CNP차앤박화장품‘프로폴리스 앰플 오일 인 크림’은 브랜드 베스트셀러인 프로폴리스 에너지 앰플에 오일을 함유해 보습 기능을 강화한 제품이다."

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
print(tokenizer.tokenize(ko_text))
```
Results
['CN', '##P', '##차', '##앤', '##박', '##화', '##장', '##품', '역시', '국', '##내', '대', '##표', '[UNK]', '피', '##부', '##과', '출', '##신', '[UNK]', '화', '##장', '##품', '브', '##랜드', '##다', '.', 'CN', '##P', '##차', '##앤', '##박', '##화', '##장', '##품', '[UNK]', '프로', '##폴', '##리스', '[UNK]', '오', '##일', '인', '크', '##림', '[UNK]', '은', '브', '##랜드', '베', '##스트', '##셀', '##러', '##인', '프로', '##폴', '##리스', '에', '##너', '##지', '[UNK]', '오', '##일', '##을', '함', '##유', '##해', '보', '##습', '기', '##능을', '강', '##화', '##한', '제', '##품', '##이다', '.']

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 18.04.3 LTS
* Python version: 3.6.8
* PyTorch version: 1.3.0+cu100
* PyTorch Transformers version (or branch): 2.2.2
* Using GPU ? : Not in this issue.
* Distributed or parallel setup ? No
* Any other relevant information: I use the docker image `horovod/horovod:0.18.2-tf2.0.0-torch1.3.0-mxnet1.5.0-py3.6-gpu`

## Additional context
I tested `transformers.DistilBertTokenizer` with the **bert-base-multilingual-cased** vocabs on english text. It returns expected results.

So, It seems that subclassing DistilBertTokenizer from BertTokenizer is the problem.... How can I solve this issue?

<!-- Add any other context about the problem here. -->


Is any possible for load local model ?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
For some reason(GFW), I need download pretrained model first then load it locally. But I read the source code where tell me below:

```
pretrained_model_name_or_path: either:
    - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
    - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
    - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
    - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
    - None if you are both providing the configuration and state dictionary (resp. with keyword arguments ``config`` and ``state_dict``)
```

I wanna download a pretrained model and load it locally with from_pretrained api, How can I do that?


[Albert] SentencePiece Error with AlbertTokenizer

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):Albert v2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: Sequence Classification

## To Reproduce

Steps to reproduce the behavior:

```
from transformers import AlbertTokenizer
from pyspark.sql import functions as F, types as T

tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")

# load data into spark
df = spark.read...

# df.columns =>  ["id", "text"]

# create a function to create the tokens from a supplied text column
def create_tokens(text=None, tokenizer=None):
    tokens = ["[CLS]"]
    tokens.extend(tokenizer.tokenize(text))
    tokens.append("[SEP]")
    return tokens

def create_tokens_udf = F.udf(lambda z: create_tokens(z, tokenizer=tokenizer), T.ArrayType(T.StringType()))

# apply the udf to the text
tokenized_df = df.withColumn("tokens", create_tokens_udf(F.column("text")))

# trigger the transformation
tokenized_df.cache().count()
```

The following traceback is observed (i've excluded the Py4J tracebacks for clarity):

```
File "<command-3594705570096092>", line 10, in create_tokens
  File "/databricks/python/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 302, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 438, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/tokenization_albert.py", line 90, in __init__
    self.sp_model.Load(vocab_file)
  File "/databricks/python/lib/python3.7/site-packages/sentencepiece.py", line 118, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)
RuntimeError: Internal: unk is not defined.

```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

The text gets tokenized as expected.

## Environment

* OS: Linux(?)
* Python version: 3.7.3
* PyTorch version: ?
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU N/A
* Distributed or parallel setup: Distributed (Databricks)
* Any other relevant information:

## Additional context

Thanks a million for this amazing library!


Bug Transformers 2.3.0 - ValueError: invalid literal for int() with base 10: 'pytorch'

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): PT-BR (Multilingual)

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

CUDA_VISIBLE_DEVICES=2,3 nohup python /home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py \
--output_dir=/home/lucasrodrigues/train/transformers/output/multi/250k/ \
--model_type=bert \
--model_name_or_path=/home/lucasrodrigues/train/transformers/model-multi-pytorch/ \
--tokenizer_name=/home/lucasrodrigues/train/transformers/model-multi-pytorch/ \
--config_name=/home/lucasrodrigues/train/transformers/model-multi-pytorch/ \
--block_size=510 \
--do_lower_case \
--train_data_file=/home/lucasrodrigues/datasets/nilc/initial/initial_corpus_train.txt \
--eval_data_file=/home/lucasrodrigues/datasets/nilc/initial/initial_corpus_train.txt \
--do_train \
--do_eval \
--evaluate_during_training \
--logging_steps=50 \
--save_steps=50 \
--per_gpu_train_batch_size=2 \
--per_gpu_eval_batch_size=2 \
--mlm \
> logs/transformers_multi_250k.txt &

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Traceback (most recent call last):
  File "/home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py", line 713, in <module>
    main()
  File "/home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py", line 663, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "/home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py", line 268, in train
    global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])
ValueError: invalid literal for int() with base 10: 'pytorch'

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu
* Python version: 3.6
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch):
* Using GPU ? 4x GeForce GTX 1080 Ti
* Distributed or parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

I can't execute the code, could anyone help?


Is there a way to reduce the vocabulary size?

## ❓ Questions & Help

For fine tuning task is it possible to reduce the vocabulary size?
does simply editing the vocab & config  file work?

<!-- A clear and concise description of the question. -->


Unclear documentation for indice masking

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): CamemBERT but this probly applies to all MLMs.

Language I am using the model on (English, Chinese....): French

The problem arise when using:
* [x] my own modified scripts, but I suspect that `https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py` is also impacted.

Basically, the masking procedure raises an assertion error device-side when I try to run something akin to:

```
model(labels, masked_lm_labels=labels)
```

I pinpointed the error to be due to the fact that making values to be ignored in the labels with value `-100` like [here in the `run_lm_finetuning.py` script](https://github.com/huggingface/transformers/blob/81d6841b4be25a164235975e5ebdcf99d7a26633/examples/run_lm_finetuning.py#L179) is problably deprecated. The documentation is unclear on the subject, as it says:

> **masked_lm_labels:** (optional) torch.LongTensor of shape (batch_size, sequence_length):
>
>     Labels for computing the masked language modeling loss.
>     Indices should be in [-1, 0, ..., config.vocab_size] (see input_ids docstring)
>     Tokens with indices set to -100 are ignored (masked), the loss is only computed
>     for the tokens with labels in [0, ..., config.vocab_size]

As you can see, information is contradictory: on one hand, they say values should be between [-1, vocab_size], but also say like in the script that tokens with values -100 are ignored. I tried, and using value -1 does indeed work.

The task I am working on is:

* [x] my own task or dataset: I am finetuning the CamemBERT pretrained model on a MLM task before reusing the model to a sentence classification one.

## To Reproduce

Steps to reproduce the behavior:

```
import torch
from transformers import CamembertForMaskedLM

model = CamembertForMaskedLM.from_pretrained(
    "camembert-base", cache_dir="models/pretrained_camembert"
)
inputs = torch.full((30, 1), 4).to(torch.long)
labels = inputs.clone()
labels[10] = -100
model(inputs, masked_lm_labels=labels)
```

This gives:

```
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97
```

If you run it on GPU a similar error is raised.

## Expected behavior

Should return a loss.

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6.9
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? Both do not work.
* Distributed of parallel setup ?
* Any other relevant information: Issue can be solved by replacing -1. As I said, I think at some point you switched to using -1 instead of -100 but did not propagate entirely the change to the doc and examples.


Albert to torchscript is not working

Trying to export torchscript module for AlbertForQuestionAnswering.
```
self.model = AlbertForQuestionAnswering.from_pretrained(self.model_dir)
script_model = torch.jit.script(self.model)
script_model.save("script_model.pt")
```

Getting following exception:
```
Python builtin <built-in function next> is currently not supported in Torchscript:
	at /usr/local/lib/python3.6/dist-packages/transformers/modeling_albert.py:523:67

	device = input_ids.device if input_ids is not None else inputs_embeds.device

	if attention_mask is None:
	attention_mask = torch.ones(input_shape, device=device)
	if token_type_ids is None:
	token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device)

	extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)
	extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility
	~~~~ <--- HERE extended_attention_mask=(1.0 - extended_attention_mask) * -10000.0 if head_mask is not None: if
		head_mask.dim()==1: head_mask=head_mask.unsqueeze(0).unsqueeze(0).unsqueeze(-1).unsqueeze(-1)
		head_mask=head_mask.expand(self.config.num_hidden_layers, -1, -1, -1, -1) elif head_mask.dim()==2:
		head_mask=head_mask.unsqueeze(1).unsqueeze(-1).unsqueeze(-1) # We can specify head_mask for each layer
		head_mask=head_mask.to(dtype=next(self.parameters()).dtype) # switch to fload if need + fp16 compatibility
		else: '__torch__.transformers.modeling_albert.___torch_mangle_15.AlbertModel.forward' is being compiled since it
		was called from '__torch__.transformers.modeling_albert.___torch_mangle_14.AlbertForQuestionAnswering.forward'
		at /usr/local/lib/python3.6/dist-packages/transformers/modeling_albert.py:767:8 def forward(self,
		input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None,
		start_positions=None, end_positions=None): outputs=self.albert( ~~~~~~~~~~~~~~~~~~~~~~... <--- HERE
		input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, position_ids=position_ids,
		head_mask=head_mask, inputs_embeds=inputs_embeds ) sequence_output=outputs[0]
```


Fixed answer structure for QAPipeline

Internals of the library; Pipeline.

Updated answers list in QuestionAnswering pipeline to handle multiple (question, context) pair with (top-k >1) solutions


greedy beam search generates same sequence N times

## ❓ Questions & Help

I am doing greedy beam search (without sampling to avoid randomness) using GPT-2. However, all the returned sequences are same. Why is that the case? Shouldn't it give N best and different sequences?

```python
model = GPT2LMHeadModel.from_pretrained('gpt2-medium').cuda()
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')

input_context = 'The dog'
input_ids = torch.tensor(tokenizer.encode(input_context)).unsqueeze(0).cuda()
outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3)
for i in range(3):
    print('Generated {}: {}'.format(i, tokenizer.decode(outputs[0][i], skip_special_tokens=True)))
```

The resulting output is:
```
Generated 0: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
Generated 1: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
Generated 2: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
````


Serializing XLMRobertaTokenizer

I am currently trying to use the XLMRobertaTokenizer in a multiprocessor setting. To do this, the XLMRobertaTokenizer needs to be serializable. Currently XLMRobertaTokenizer is not serializable while other tokenizers such as AlbertTokenizer are.

This PR adds the __getstate__ and __setstate__ methods to XLMRobertaTokenizer so that it can be serialized.


How to use transformers-cli serve , how to set up on the server side?

Internals of the library; Pipeline.

## ❓ Questions & Help

Hi, I would like to make the transformers based models running as a server on a remote machine, as the way bert-as-server did.
I suppose I could call the transformers-cli serve command on the server side, but I haven't find much clue on how to make it running on the client part.

BTW. I am trying to run the serve cmd with localhost like :
transformers-cli serve --task feature-extraction --model distilbert --config distilbert-base-uncased --tokenizer distilbert
and failed for ValueError: Can't find a vocabulary file at path  [cached dir file].
and I tried with transformers/src/transformers/__main__.py with same parameters and got same error.
Would you please give me a snippet on how to make the transformers-cli serve work on both sides?


Update Mish activation function to use torchscript JIT

This PR modifies the implementation of Mish to match that of the [fastai library](https://github.com/fastai/fastai_dev/blob/0f613ba3205990c83de9dba0c8798a9eec5452ce/dev/local/layers.py#L441). A discussion of the benefits of JIT for the Mish function can be found on the [fastai forums](https://forums.fast.ai/t/meet-mish-new-activation-function-possible-successor-to-relu/53299/587).


What is the difference between T5Model, T5WithLMHeadModel, T5PreTrainedModel?

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I notice that for T5 model, there are more choices(T5Model, T5WithLMHeadModel, T5PreTrainedModel) than BERT or GPT. What is the difference between these three? I think all three are pre-trained model. We do not use T5PreTrainedModel in our downstream task code. Besides, the difference between T5Model and T5WithLMHeadModel is that the latter contains one more linear layer at the end.   Am I right about these?


Typo in XLM moses pipeline.

The replacement on for the unicode punct replacement has a mistake at https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py#L477


Error in pipeline() when model left as None

Internals of the library; Pipeline.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Default models as per `SUPPORTED_TASKS` config in [pipeline.py](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py)

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (`pipeline.py`)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (`question-answering`, `ner`, `feature-extraction`, `sentiment-analysis`)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Install `transformers` 2.3.0
2. Run [example](https://github.com/huggingface/transformers#quick-tour-of-pipelines)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

```py
from transformers import pipeline
>>> nlp = pipeline('question-answering')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Continuum\anaconda3\envs\transformers\lib\site-packages\transformers\pipelines.py", line 860, in pipeline
    model = models[framework]
TypeError: string indices must be integers
>>> nlp = pipeline('question-answering', model='distilbert-base-uncased-distilled-squad', tokenizer='distilbert-base-uncased')
```

## Expected behavior

Leaving `model`/`tokenizer` args to `None` should not yield error.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: system='Windows', release='10', version='10.0.17134', machine='AMD64'
* Python version: 3.5.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Can't download models or model config

## ❓ Questions & Help
when I run fine-tune examples as  run_squad.py,it turns out error like this:
 E:\tensorflow_natural_question\transformers\examples>python run_squad.py --model_type bert --model_name_or_path bert-base-cased --do_train --do_eval --do_lower_case --train_file $SQUAD_DIR/train-v2.0.json --predict_file $SQUAD_DIR/dev-v2.0.json --per_gpu_train_batch_size 2 --learning_rate 3e-5 --num_train_epochs 1.0 --max_seq_length 384 --doc_stride 128 --output_dir /tmp/debug_squad/
2020-01-05 23:58:07.219057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
01/05/2020 23:58:08 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
01/05/2020 23:58:13 - INFO - filelock -   Lock 2170508660632 acquired on C:\Users\Administrator\.cache\torch\transformers\b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.d7a3af18ce3a2ab7c0f48f04dc8daff45ed9a3ed333b9e9a79d012a0dedf87a6.lock
01/05/2020 23:58:13 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json not found in cache or force_download set to True, downloading to C:\Users\Administrator\.cache\torch\transformers\tmptcqtrh98

win10+pytorch-gpu1.2.0+python3.7.3


[cli] Add support for T5 model conversion

I have added support for converting t5 model from CLI.


BERT's Embedding/Vocab Size in Code is Different from Provided Pretrained Config

## 🐛 A Subtle Bug

Hi, I really appreciate your work but I found a subtle problem here. Could you take a look of it?

- The model I am using is **BERT**.

- The language I am using the model on is **English**.

- The problem arises when using:

- The task I am working on is to simply initialize a BERT object with my own modifications to config, i.e., `BertConfig` class.

## To Reproduce

Steps to reproduce the behavior:

Simplely run this line:
```
BertModel.from_pretrained("bert-base-cased",config=BertConfig(output_hidden_states=True))
```
Then we have following error message:
`
File "D:\Anaconda3\envs\gnner\lib\site-packages\transformers\modeling_utils.py", line 486, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for BertModel:
	size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([28996, 768]) from checkpoint, the shape in current model is torch.Size([30522, 768]).
`

## Expected behavior

It should be good to run instead of reporting issues like that.

## Possible reason
The issue is because of `line 86` in `configuration_bert.py`, where the vocabulary size is **`30522`**. The default vocabulary size I believe should be consistent with that in the config file, i.e., `https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json`, where it's **`28996`**.

## Environment

* OS: Windows 10
* Python version: 3.7.3
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): latest pip package
* Using GPU ? It's independent of environments like this I believe.

<!-- Add any other context about the problem here. -->


weird resize during the initialization in the PreTrainedModel

Hi
I am using BertForMaskedLM in the run_lm_finetuning.py code. This module call the module of BertLMPredictionHead, in which there is a decoder layer which is of the size of hidden_size*vocab_size. I would like to change the dimension of this layer. when I change it, I realize that during the call to  elf.init_weights() inside the  BertForMaskedLM, the function resize the weights for the decoder layer. I cannot track where is this exactly happening, thanks for your advice on how I can resolve this issue.


Pretrained Model not available

## ❓ Questions & Help

01/05/2020 12:18:00 - INFO - root -   finetuned model not available - loading standard pretrained model
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   Model name '/opt/ml/code/pretrained_models/bert-base-uncased' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1). Assuming '/opt/ml/code/pretrained_models/bert-base-uncased' is a path or url to a directory containing tokenizer files.
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   Didn't find file /opt/ml/code/pretrained_models/bert-base-uncased/added_tokens.json. We won't load it.
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   Didn't find file /opt/ml/code/pretrained_models/bert-base-uncased/special_tokens_map.json. We won't load it.
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   Didn't find file /opt/ml/code/pretrained_models/bert-base-uncased/tokenizer_config.json. We won't load it.
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   loading file /opt/ml/code/pretrained_models/bert-base-uncased/vocab.txt
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   loading file None
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   loading file None
01/05/2020 12:18:00 - INFO - transformers.tokenization_utils -   loading file None


Add support for Albert and XLMRoberta for the Glue example