RoBERTa has a token_type layer (just a cosmetic issue)
Author: cronoik
Pull request: no
Labels: name: wontfix

name: Core: Modeling

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): RoBERTa

Language I am using the model on (English, Chinese ...): Language-independent

The cosmetic problem:
The fairseq RoBERTa doesn't has a token_type layer:
```
TransformerSentenceEncoder(
  (embed_tokens): Embedding(50265, 768, padding_idx=1)
  (embed_positions): LearnedPositionalEmbedding(514, 768, padding_idx=1)
```
The huggingface implementation of RoBERTa accepts token typ ids because RobertaModel inherits from BertModel and the layer is inherited by RobertaEmbeddings from BertEmbeddings:
```
RobertaEmbeddings(
  (word_embeddings): Embedding(50265, 768, padding_idx=1)
  (position_embeddings): Embedding(514, 768, padding_idx=1)
  (token_type_embeddings): Embedding(1, 768)
  (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  (dropout): Dropout(p=0.1, inplace=False)
)
```

@julien-c wrote in #2702:
> On the fact that our RoBERTa implem(tation) takes (inoperant by default) `token_type_ids`, maybe we should actually remove them from the implem. If you want to train some, you can always subclass RoBERTa and add them back (but I'm not 100% sure a lot of people use them). Thoughts?

## Expected behavior
I think the huggingface models should be as close to original as possible and therefore RoBERTA should not have a token_type_embeddings layer and not accept token_type_ids. I know this is just a cosmetic issue, but I think it causes some confusion. I would like to use this issue to collect some opinions.

When there are no others opinions, I would like to work on this. This also affects #2727




Comments:
This is something that we're looking at with @LysandreJik and @thomwolf  – In the meantime, feel free to open a draft PR.

As discussed in the other issues, it would be great if a lot of care is taken in maximising the compatibility between a tokenizer and its corresponding model, as I discussed in https://github.com/huggingface/transformers/issues/2702#issuecomment-581480669. In other words, the tokenizer encode methods should only return those values that are accepted by its model's forward method.

RoBERTa has only one possible token type id (`0`), but the embedding for that still needs to be there. That embedding is added to all word piece embeddings, and that's how it was trained. If you suddenly stop doing that, the model will stop working.

You could just add that embedding to all word piece embeddings and store the model under a new name to achieve the same effect. But you can't just take it away.

If you take it away, you have to retrain the whole thing.

In my opinion you dont have to retrain it as the weights of this layer are zero. Have a look at the example below:
```from transformers.modeling_roberta import RobertaForSequenceClassification
model = RobertaForSequenceClassification.from_pretrained('roberta-base')
print(model.state_dict()['roberta.embeddings.token_type_embeddings.weight'])
##Output truncated by me:
##tensor([[0., 0., 0., 0., 0., .....0., 0., 0., 0.]])
```
So what happens when we remove this layer:
```
##Defining our own roberta class without the token_type_embeddings layer
from torch import nn
from transformers.modeling_bert import BertLayerNorm, BertModel, BertPreTrainedModel
from transformers.modeling_roberta import RobertaClassificationHead, RobertaConfig, ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP, create_position_ids_from_input_ids, CrossEntropyLoss
class MyBertEmbeddings(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=0)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        #self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)

        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
        # any TensorFlow checkpoint file
        self.LayerNorm = BertLayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):
        if input_ids is not None:
            input_shape = input_ids.size()
        else:
            input_shape = inputs_embeds.size()[:-1]

        seq_length = input_shape[1]
        device = input_ids.device if input_ids is not None else inputs_embeds.device
        if position_ids is None:
            position_ids = torch.arange(seq_length, dtype=torch.long, device=device)
            position_ids = position_ids.unsqueeze(0).expand(input_shape)
        if token_type_ids is None:
            token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device)

        if inputs_embeds is None:
            inputs_embeds = self.word_embeddings(input_ids)
        position_embeddings = self.position_embeddings(position_ids)
        #token_type_embeddings = self.token_type_embeddings(token_type_ids)

        embeddings = inputs_embeds + position_embeddings #+ token_type_embeddings
        embeddings = self.LayerNorm(embeddings)
        embeddings = self.dropout(embeddings)
        return embeddings

class MyRobertaEmbeddings(MyBertEmbeddings):

    def __init__(self, config):
        super().__init__(config)
        self.padding_idx = 1
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=self.padding_idx)
        self.position_embeddings = nn.Embedding(
            config.max_position_embeddings, config.hidden_size, padding_idx=self.padding_idx
        )

    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):
        if position_ids is None:
            if input_ids is not None:
                # Create the position ids from the input token ids. Any padded tokens remain padded.
                position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
            else:
                position_ids = self.create_position_ids_from_inputs_embeds(inputs_embeds)

        return super().forward(
            input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
        )

    def create_position_ids_from_inputs_embeds(self, inputs_embeds):
        input_shape = inputs_embeds.size()[:-1]
        sequence_length = input_shape[1]

        position_ids = torch.arange(
            self.padding_idx + 1, sequence_length + self.padding_idx + 1, dtype=torch.long, device=inputs_embeds.device
        )
        return position_ids.unsqueeze(0).expand(input_shape)

class MyRobertaModel(BertModel):
    config_class = RobertaConfig
    pretrained_model_archive_map = ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
    base_model_prefix = "roberta"

    def __init__(self, config):
        super().__init__(config)

        self.embeddings = MyRobertaEmbeddings(config)
        self.init_weights()

    def get_input_embeddings(self):
        return self.embeddings.word_embeddings

    def set_input_embeddings(self, value):
        self.embeddings.word_embeddings = value

class MyRobertaForSequenceClassification(BertPreTrainedModel):
    config_class = RobertaConfig
    pretrained_model_archive_map = ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
    base_model_prefix = "roberta"

    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.roberta = MyRobertaModel(config)
        self.classifier = RobertaClassificationHead(config)

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
    ):
        outputs = self.roberta(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
        )
        sequence_output = outputs[0]
        logits = self.classifier(sequence_output)

        outputs = (logits,) + outputs[2:]
        if labels is not None:
            if self.num_labels == 1:
                #  We are doing regression
                loss_fct = MSELoss()
                loss = loss_fct(logits.view(-1), labels.view(-1))
            else:
                loss_fct = CrossEntropyLoss()
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            outputs = (loss,) + outputs

        return outputs  # (loss), logits, (hidden_states), (attentions)
```

```
mymodel = MyRobertaForSequenceClassification.from_pretrained('roberta-base')

##We need to set the weights of the randomly initialized layers to the same values
import torch
for name, param in mymodel.named_parameters():
  if not(torch.all(param.data.eq(model.state_dict()[name]))):
    print('{} is not identical'.format(name))
    param.data = model.state_dict()[name]
##Output:
##classifier.dense.weight is not identical
##classifier.dense.bias is not identical
##classifier.out_proj.weight is not identical
##classifier.out_proj.bias is not identical
```
Now we can compare mymodel with model:
```
from transformers import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')

for m in [mymodel, model]:
  input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
  labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
  outputs = m(input_ids, labels=labels)
  print(outputs)
##(tensor(0.8658, grad_fn=<NllLossBackward>), tensor([[ 0.0927, -0.2271]], grad_fn=<AddmmBackward>))
##(tensor(0.8658, grad_fn=<NllLossBackward>), tensor([[ 0.0927, -0.2271]], grad_fn=<AddmmBackward>))
```
and see that the output is the same.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.