Why the functions "add_special_tokens()" and "resize_token_embeddings()" hurt the performance of 'gpt2' and 'gpt2-medium' but not 'gpt2-large' and 'gpt2-xl' ?
Author: MingfengXue
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

## Details
When I use add_special_tokens and resize_token_embeddings to expand the vocabulary, the LM loss would become very large in gpt2 and gpt2-medium models (loaded by from_pretrained('gpt2') and from_pretrained('gpt2-medium)). But it don't happen when I load the gpt2-large and gpt2-xl models (also loaded by from_pretrained).  Why?

Environment Info: 
python 3.7.7
Linux 16.04
transformers 3.3.1
pytorch 1.6.0

Codes and results:
'''
import torch
from transformers import GPT2Tokenizer
from transformers import GPT2LMHeadModel
device = torch.device('cuda:3')
input_sentence = 'who win this game?'

gpt2tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
gpt2model = GPT2LMHeadModel.from_pretrained('gpt2', return_dict=True)
gpt2model.to(device)
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(5.0102, device='cuda:3', grad_fn=<NllLossBackward>)

gpt2tokenizer.add_special_tokens({'additional_special_tokens': ['[first]', '[second]']})
gpt2model.resize_token_embeddings(len(gpt2tokenizer))
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(77.1513, device='cuda:3', grad_fn=<NllLossBackward>)

gpt2tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')
gpt2model = GPT2LMHeadModel.from_pretrained('gpt2-large', return_dict=True)
gpt2model.to(device)
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(5.1567, device='cuda:3', grad_fn=<NllLossBackward>)

gpt2tokenizer.add_special_tokens({'additional_special_tokens': ['[first]', '[second]']})
gpt2model.resize_token_embeddings(len(gpt2tokenizer))
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(5.1568, device='cuda:3', grad_fn=<NllLossBackward>)
'''

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This behavior is caused by the fact that GPT2 is a model that ties the weights of the token embeddings and the language model head. For some settings of the pretrained weights, a new randomly initialized row of the language model head weight matrix might effectively unilaterally assign very high likelihood to the new word. This is what's happening for some pretrained models (but not others). A workaround is, after resizing the embeddings, set the new token embeddings to be the original unk token embedding, for example:

    model.transformer.wte.weight[-1] = model.transformer.wte.weight[-2]

Alternatively, if you want to break the symmetry created by initializing with the exact same embedding, you can set the new embedding to be the average of all other embeddings, or add some noise when copying the existing unk token embedding. But just naively copying like above fixes the problem for GPT-2 (small).

@eric-mitchell You are right. Now I know how to solve my problem. Thanks for that.

Exactly same code with @eric-mitchell 's solution gives me error like:
`model.transformer.wte.weight[-1] = model.transformer.wte.weight[-2]`

> "RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation." 

As far as i know, in-place operation is the operation that changes directly the content. But here I can't find any in-place operation. Does anyone know why this happens?

@jucho2725 I've just found a possible solution [here](https://stackoverflow.com/questions/49161652/how-to-get-around-in-place-operation-error-if-index-leaf-variable-for-gradient-u) 

@Loreb92 Thanks for the help. How I solve the issue is the same as there. 
Here is a code that I use. Hope this helps someone.

```
tokenizer = GPT2Tokenizer.from_pretrained(model_args.model_name_or_path)
tokenizer.pad_token = tokenizer.eos_token # gpt2 does not have pad token at first.

special_tokens_dict = {
         "additional_special_tokens": ['[ABC]', '[DEF]', '[GHI]'],
}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))
unk_tok_emb = model.transformer.wte.weight.data[tokenizer.unk_token_id, :] 
for i in range(num_added_toks):
        model.transformer.wte.weight.data[-(i+1), :] = unk_tok_emb
```