Is the GPT-2 forward too different from Bert or RoBerta?
Author: celsofranssa
Pull request: no
Labels: name: wontfix

I am using some Transformers (Bert, RoBerta, etc.) in my project.
When including the `GPT-2` as shown below:

```python
from transformers import GPT2Tokenizer, GPT2Model
import torch

# inits model and tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model = GPT2Model.from_pretrained('gpt2')

# inputs
inputs = tokenizer.encode(text="Hello, my dog is cute", max_length=12, padding="max_length",
                                         truncation=True)
#[15496, 11, 616, 3290, 318, 13779, 50257, 50257, 50257, 50257, 50257, 50257]

# input_ids tensor and attention mask
features = torch.tensor([inputs])
# tensor([[15496,    11,   616,  3290,   318, 13779, 50257, 50257, 50257, 50257,
#        50257, 50257]])
attention_mask = (features < 50257).int()
# tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0]], dtype=torch.int32)

# outputs
outputs = model(
    input_ids=features, 
    attention_mask=attention_mask
)

```

I have had the following error:

```python
IndexError                                Traceback (most recent call last)
<ipython-input-31-763ba5835cf8> in <module>()
----> 1 outputs = model(input_ids=features, attention_mask=attention_mask)

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, use_cache, output_attentions, output_hidden_states, return_dict)
    679 
    680         if inputs_embeds is None:
--> 681             inputs_embeds = self.wte(input_ids)
    682         position_embeds = self.wpe(position_ids)
    683         hidden_states = inputs_embeds + position_embeds

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127 
    128     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853 
   1854 

IndexError: index out of range in self
```

Comments:
the `IndexError` is from the padding token being added. GPT2 doesn't have a padding token. You should be able to manually set that input_id to 0 (or any other valid input id) and then rely on the attention mask to ignore those positions. 

> the `IndexError` is from the padding token being added. GPT2 doesn't have a padding token. You should be able to manually set that input_id to 0 (or any other valid input id) and then rely on the attention mask to ignore those positions.

But I had already added the `PAD` token which receives the `token_id = 50257`:
```
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
```

that tells the tokenizer to add a token and creates the new token id, but it doesn't modify the embedding layer of the model. the new token id is still invalid for the embedding layer of GPT-2 (which does not include a pad token). the reason it works for roberta and bert is b/c they were trained with pad tokens and therefore have entries in their embedding layers for that token.  you want to do something like, 

`inputs[inputs==50257] == 0`

Hi! In the [documentation of `add_special_tokens`](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=resize_token_embeddings#transformers.tokenization_utils_base.SpecialTokensMixin.add_special_tokens), you'll see in the sample code the following line:

```py
model.resize_token_embeddings(len(tokenizer))
```

as @galtay mentions, you need to resize the embedding layer when adding tokens to the tokenizer, otherwise the model will not know that its embedding matrix has been resized.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.