Longformer `token_type_ids` Vocabulary Size is 1 But Documentation States Otherwise
Author: HHousen
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2 and latest (4.0.0)
- Platform: Linux (Ubuntu)
- Python version: 3.6.9 and 3.8.6
- PyTorch version (GPU?): 3.7.0 Tesla P100-PCIE-16GB and Nvidia RTX 3090
- Tensorflow version (GPU?): None
- Using GPU in script?: Yes and No
- Using distributed or parallel set-up in script?: No

### Who can help
Possibly @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('allenai/longformer-base-4096')
model = AutoModel.from_pretrained('allenai/longformer-base-4096')

tokenizer_bert = AutoTokenizer.from_pretrained('bert-base-uncased')

inputs = tokenizer("How old are you?", "I'm 6 years old", return_tensors="pt", return_token_type_ids=True, return_attention_mask=True)
inputs_bert = tokenizer_bert("How old are you?", "I'm 6 years old", return_tensors="pt", return_token_type_ids=True, return_attention_mask=True)

print(inputs)
print(inputs_bert)
```

```python
inputs = {'input_ids': tensor([[   0, 6179,  793,   32,   47,  116,    2,    2,  100,  437,  231,  107,
          793,    2]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
inputs_bert = {'input_ids': tensor([[ 101, 2129, 2214, 2024, 2017, 1029,  102, 1045, 1005, 1049, 1020, 2086,
         2214,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
```

```python
inputs['token_type_ids'] = torch.tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]])
model.forward(**inputs)
```

Stack Trace:

```
---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

<ipython-input-65-905d6a7d6135> in <module>()
----> 1 model.forward(**inputs)

6 frames

/usr/local/lib/python3.6/dist-packages/transformers/modeling_longformer.py in forward(self, input_ids, attention_mask, global_attention_mask, token_type_ids, position_ids, inputs_embeds, output_attentions, output_hidden_states)
    995 
    996         embedding_output = self.embeddings(
--> 997             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    998         )
    999 

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     66 
     67         return super().forward(
---> 68             input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
     69         )
     70 

/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    178             inputs_embeds = self.word_embeddings(input_ids)
    179         position_embeddings = self.position_embeddings(position_ids)
--> 180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    181 
    182         embeddings = inputs_embeds + position_embeddings + token_type_embeddings

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127 
    128     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853 
   1854 

IndexError: index out of range in self
```

## Expected behavior

The longformer documentation should be updated and state that the longformer does not support `token_type_ids` like RoBERTa. The `token_type_ids` [vocabulary size is 1](https://huggingface.co/allenai/longformer-base-4096/raw/main/config.json) (compared to [2 for BERT](https://huggingface.co/bert-base-uncased/raw/main/config.json)) for `allenai/longformer-base-4096`, which means `0` is the only valid input for `token_type_ids`. However, [the documentation](https://huggingface.co/transformers/model_doc/longformer.html#transformers.LongformerModel.forward) says `token_type_ids` can be selected in `[0, 1]` for the longformer. The documentation should to specify that the longformer doesn't support `token_type_ids`. For instance, the [RoBERTa documentation](https://huggingface.co/transformers/model_doc/roberta.html) states "RoBERTa doesn’t have `token_type_ids`, you don’t need to indicate which token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or `
`)." Should a similar message be added for the longformer since it is based on RoBERTa?


Comments:
It also might be a good idea to catch this error somewhere before `IndexError: index out of range in self` because that is not descriptive and makes debugging difficult.

You're right @HHousen - thanks for the note! Do you want to open a PR to fix the docs? That would be awesome :-) Otherwise, I can do it as well 

We are using longformer and we are passing (input_ids, attention_mask , global_attention_mask ,token_type_ids) as input. if we are passing token_type_ids as 0's we are not having any issues but when we try to pass token_type_ids as 1's, or 0's afor segments within the sequence it is throwing following error.
**IndexError: index out of range in self**
**C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: block: [52,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.**

i have created separate issue for **index out of range in self ** (https://github.com/huggingface/transformers/issues/9162) while using  token_type_ids, from the above comment by @HHousen should i remove token_type_ids as parameter while passing it to model ?

@yuvarajvc Correct. The Longformer doesn't support `token_type_ids`, so you should not pass them to the model.