DistilBERT does not support token type ids, but the tokenizers produce them
Author: dirkgr
Pull request: no
Labels: name: wontfix

```Python
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("distilbert-base-uncased-distilled-squad")
>>> tokenized = tokenizer.encode_plus("I ate a clock yesterday.", "It was very time consuming.")
>>> tokenized
{'input_ids': [101, 1045, 8823, 1037, 5119, 7483, 1012, 102, 2009, 2001, 2200, 2051, 15077, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

>>> model = transformers.AutoModel.from_pretrained("distilbert-base-uncased-distilled-squad")
>>> model(**tokenized)
TypeError: forward() got an unexpected keyword argument 'token_type_ids'
```

In contrast, RoBERTa also does not support token type ids, but its forward method still takes the parameter, and its tokenizer produces type ids that are all zero.

Comments:
RoBERTa accepts token typen ids because RoBERTa is basically the same architecture as BERT. (The "innovation" lies in how it's pretrained, not architectural changes.) It's literally nothing more than this:

https://github.com/huggingface/transformers/blob/ddb6f9476b58ed9bf4433622ca9aa49932929bc0/src/transformers/modeling_roberta.py#L149-L169

Distilbert's changes are more intricate.

Looking at your example, I agree that it'd be nice that all forward methods have the same signature for easier use of the `AutoModel`s.

T5 does something different, it just accepts `**kwargs`. That would solve the issue that there is now for Distilbert, but it has some adverse, non-pythonic side effects (imo): less readability, no IDE autocomplete, default values need to be set inside the method rather than declaration (in `pop`). I'm not a big fan of this.

It's best that the maintainers make a suggestion on how to continue with this.

> RoBERTa accepts token typen ids because RoBERTa is basically the same architecture as BERT. (The "innovation" lies in how it's pretrained, not architectural changes.)

The fairseq roBERTa doesn't accepts token typ ids and doesn't even has a layer for those:
```
TransformerSentenceEncoder(
  (embed_tokens): Embedding(50265, 768, padding_idx=1)
  (embed_positions): LearnedPositionalEmbedding(514, 768, padding_idx=1)
```
The huggingface implementation of RoBERTa accepts token typ ids because RobertaModel inherits from BertModel and the layer is inherited by RobertaEmbeddings from BertEmbeddings:
```
RobertaEmbeddings(
  (word_embeddings): Embedding(50265, 768, padding_idx=1)
  (position_embeddings): Embedding(514, 768, padding_idx=1)
  (token_type_embeddings): Embedding(1, 768)
  (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  (dropout): Dropout(p=0.1, inplace=False)
)
```
The huggingface RoBERTa It is still along with the fairseq implementation due to the dictionary size of the token_type_embeddings layer. It only accepts one value (e.g. 0) while Bert accepts two values (e.g. 0 and 1).  

Back to the original topic. Call [encode_plus](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.encode_plus) with return_token_type_ids=False and you won't get them.

Yes, I was talking about the transformers implementation where Roberta is subclassing the Bertmodel.

Of course it's possible to just change the argument when encoding, but you'd want a unified approach so that you can just use automodel/autotokenizer, encode your input, and feed the encoded inputs to the forward method *for any input to automodel without having to change anything else*. In that respect this is more a usability question.

As an alternative to unifying the signature of all models, Distilbert's Tokenizer can be changed to not return the token type ids. 

Okay, in case the OP is looking for a generic solution I think it is cleaner to get the parameters from the model itself by calling `model.forward.__code__.co_varnames`. This will return a tuple of parameters names and can be used with a dictionary comprehension like below:
```
from transformers import DistilBertModel
tokenized = {'input_ids': [101, 1045, 8823, 1037, 5119, 7483, 1012, 102, 2009, 2001, 2200, 2051, 15077, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

model = DistilBertModel.from_pretrained('distilbert-base-uncased-distilled-squad')

tokenized = {key:value for (key,value) in tokenized.items() if key in model.forward.__code__.co_varnames}

model(**tokenized)
```

That's cool, but it would be a lot better if this were streamlined on the library-side rather than users having to implement this themselves. Options are, as far as I can see:

- make sure the signature for all models' forward models are the same, with None-values where unexpected values occur
- ensure that tokenizers only return the features that their respective models use

The second one seems like the way to go imo.

On the fact that our RoBERTa implem takes (inoperant by default) `token_type_ids`, maybe we should actually remove them from the implem. If you want to train some, you can always subclass RoBERTa and add them back (but I'm not 100% sure a lot of people use them). Thoughts?

> On the fact that our RoBERTa implem takes (inoperant by default) `token_type_ids`, maybe we should actually remove them from the implem. If you want to train some, you can always subclass RoBERTa and add them back (but I'm not 100% sure a lot of people use them). Thoughts?

Agreed. Sticking close to original implementations or particularly their descriptions in paper (i.e. no token_type_ids in this case) seems a good idea. Users who read the paper or saw examples in other implementations would expect that. As you say, if required it's not that hard to add them again.

On top of that, a one-on-one relationship between the output of `encode_plus` and the input of the corresponding model seems a neat improvement, too, so that the issue of OP doesn't ever occur. What this means is that using an `AutoModel` and `AutoTokenizer` can always be used like this without running into type errors.

```python
encoded = tokenizer.encode_plus(...)
out = model(**encoded )
```

As I mentioned before, I am not a big fan of how this is done in T5, which accepts anything in its forward (`**kwargs`). It is easier to implement, but it has many drawbacks in terms of usability and perhaps maintenance.

I use type ids. I just recently built a model that relies on them. I even monkey-patched a bigger embedding matrix into RoBERTa to get the ability back. But maybe a cleaner implementation would be if `forward()` took another tensor of shape `(batch, tokens, hidden_size)` that just gets added to the word piece embedding.

Either way though, it's more important to me that the output of the tokenizer matches the input of the model.

@julien-c : I'm not sure how you handle that, but I would like to work on both issues (RoBERTa token_type embedding layer and encode_plus should only output model related tokens). Can you please assign this issue to me? Should I create a separate issue regarding RoBERTa? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This is an old issue, but I thought I'd try to ask here.

As of `transformers>=4.18`, the `**kwargs` argument was removed from the `call` methods of all models.  Thus, an error occurs if you supply `token_type_ids` to `TFDistilbertForSequenceClassification.call` method.

What is the recommended way to programmatically determine whether or not a model accepts the `token_type_id` parameter in the latest version of **transformers**?

You can just inspect its signature with the `inspect` module.

Thanks, Sylvain.

For anyone stumbling on this issue, the recommended solution continues to be to simply examine the signature of the `call` method (for TensorFlow models) or the `forward` method (in PyTorch models) with the `inspect` module or (as shown above in 2020) with something like this:
```python
from transformers import TFAutoModelForSequenceClassification
model = TFAutoModelForSequenceClassification.from_pretrained('distilbert-base-uncased')
uses_token_type_ids = ("token_type_ids" in model.call.__code__.co_varnames)
print(uses_token_type_ids)
# prints False
```