XLM Roberta token_type_ids bug with batch_encode_plus
Author: tamuhey
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLM Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
import transformers

name = "xlm-roberta-base"
tokenizer = XLMRobertaTokenizer.from_pretrained(name)
model = XLMRobertaModel.from_pretrained(name)
x = tokenizer.batch_encode_plus(
    ["foo", "bar bar bar"], add_special_tokens=True, return_tensors="pt"
)
model(**x)
```

<details><summary>Output</summary>
<p>

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-33-3743974223ad> in <module>
      7     ["foo", "bar bar bar"], add_special_tokens=True, return_tensors="pt"
      8 )
----> 9 model(**x)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    797 
    798         embedding_output = self.embeddings(
--> 799             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    800         )
    801         encoder_outputs = self.encoder(

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     62 
     63         return super().forward(
---> 64             input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
     65         )
     66 

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    189             inputs_embeds = self.word_embeddings(input_ids)
    190         position_embeddings = self.position_embeddings(position_ids)
--> 191         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    192 
    193         embeddings = inputs_embeds + position_embeddings + token_type_embeddings

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115 
    116     def extra_repr(self):

~/Library/Caches/pypoetry/virtualenvs/camphr-v19AnSgn-py3.7/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485 
   1486 

RuntimeError: index out of range: Tried to access index 1 out of table with 0 rows. at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```
</p>
</details>

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

No error

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.4.1
- Platform: OS X
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (no GPU)
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
# Note

- No error occurs for other models (e.g. `bert-base-cased`)
- I think the configuration of `xlm-roberta-base` is incorrect:

```
>>> cfg = XLMRobertaConfig.from_pretrained("xlm-roberta-base")
>>> cfg.type_vocab_size
1            # 2 is correct?
```

>     * I think the configuration of `xlm-roberta-base` is incorrect:
> 
> 
> ```
> >>> cfg = XLMRobertaConfig.from_pretrained("xlm-roberta-base")
> >>> cfg.type_vocab_size
> 1            # 2 is correct?
> ```
No the configuration is correct. The offical [XLM-RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/xlmr) doesn't have any token_type_ids:
```
...
    (decoder): RobertaEncoder(
      (sentence_encoder): TransformerSentenceEncoder(
        (embed_tokens): Embedding(250002, 1024, padding_idx=1)
        (embed_positions): LearnedPositionalEmbedding(514, 1024, padding_idx=1)
        (layers)
...
```
The problem here is that encode_plus produces model independent token_type_ids. I'm currently working on a fix (#2702). You can just replace the produced token_type_ids for now with:

`x = {key:value for (key,value) in x.items() if key != 'token_type_ids'}`


Hi! This will work once #3198 is merged. Please note, however, that the following:

```py
x = tokenizer.batch_encode_plus(
    ["foo", "bar bar bar"], add_special_tokens=True, return_tensors="pt"
)
```

will not work as your two sequences, "foo" and "bar bar bar", once tokenized, are not of equal length. To ensure this gets tokenized, you will need to pass `pad_to_max_length=True` to `batch_encode_plus`:

```py
x = tokenizer.batch_encode_plus(
    ["foo", "bar bar bar"], add_special_tokens=True, return_tensors="pt", pad_to_max_length=True
)
```