text: 
  
  TITLE
  Why the functions "add_special_tokens()" and "resize_token_embeddings()" hurt the performance of 'gpt2' and 'gpt2-medium' but not 'gpt2-large' and 'gpt2-xl' ?
  
  COMMENTS
  6
  
  REACTIONS
  +1: 2
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # ❓ Questions & Help
  
  ## Details
  When I use add_special_tokens and resize_token_embeddings to expand the vocabulary, the LM loss would become very large in gpt2 and gpt2-medium models (loaded by from_pretrained('gpt2') and from_pretrained('gpt2-medium)). But it don't happen when I load the gpt2-large and gpt2-xl models (also loaded by from_pretrained).  Why?
  
  Environment Info: 
  python 3.7.7
  Linux 16.04
  transformers 3.3.1
  pytorch 1.6.0
  
  Codes and results:
  '''
  import torch
  from transformers import GPT2Tokenizer
  from transformers import GPT2LMHeadModel
  device = torch.device('cuda:3')
  input_sentence = 'who win this game?'
  
  gpt2tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
  gpt2model = GPT2LMHeadModel.from_pretrained('gpt2', return_dict=True)
  gpt2model.to(device)
  input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
  outputs = gpt2model(**input, labels=input['input_ids'])
  outputs.loss
  
  tensor(5.0102, device='cuda:3', grad_fn=<NllLossBackward>)
  
  gpt2tokenizer.add_special_tokens({'additional_special_tokens': ['[first]', '[second]']})
  gpt2model.resize_token_embeddings(len(gpt2tokenizer))
  input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
  outputs = gpt2model(**input, labels=input['input_ids'])
  outputs.loss
  
  tensor(77.1513, device='cuda:3', grad_fn=<NllLossBackward>)
  
  gpt2tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')
  gpt2model = GPT2LMHeadModel.from_pretrained('gpt2-large', return_dict=True)
  gpt2model.to(device)
  input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
  outputs = gpt2model(**input, labels=input['input_ids'])
  outputs.loss
  
  tensor(5.1567, device='cuda:3', grad_fn=<NllLossBackward>)
  
  gpt2tokenizer.add_special_tokens({'additional_special_tokens': ['[first]', '[second]']})
  gpt2model.resize_token_embeddings(len(gpt2tokenizer))
  input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
  outputs = gpt2model(**input, labels=input['input_ids'])
  outputs.loss
  
  tensor(5.1568, device='cuda:3', grad_fn=<NllLossBackward>)
  '''