GPT2 (pre-trained not fine-tuned) only generates additional special tokens 
Author: al3xpapangelis
Pull request: no

## Environment info
- `transformers` version: 3.5.0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.6.3
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten

## Information

Model I am using (GPT2 / DistilGPT2):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

I'm using GPT2 or DistilGPT2 on MetalWOZ and the issue I'm having is when I add special tokens (even bos, eos, etc) and prompt the model, it only generates those special tokens - no other token. For example, if I add the tokens <USER> and <SYSTEM> and prompt the model with:

"I want a pepperoni pizza with mushroom"

I get:

"I want a pepperoni pizza with mushroom <USER> <USER> <USER> <SYSTEM> <USER> <USER> <USER> <SYSTEM> <USER> <USER>"

## To reproduce

Steps to reproduce the behavior:

1. Add special tokens to a GPT2 model (example below with distilgpt2 but I get the same behavior with gpt2)
2. Resize embeddings
3. Prompt model

```
import torch
import torch.nn.functional as F
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('distilgpt2')
tokenizer.add_special_tokens(
	{'additional_special_tokens': ['<USER>', '<SYSTEM>']}
)

model = GPT2LMHeadModel.from_pretrained('distilgpt2')
model.resize_token_embeddings(len(tokenizer))
inp_tok_ids = tokenizer.encode('I want a pepperoni pizza with mushroom')
inp_tensor = torch.LongTensor(inp_tok_ids).unsqueeze(0)
model.eval()

with torch.no_grad():
	for i in range(10):
		outputs = model(inp_tensor)
		logits = outputs[0][:, -1, :]
		probs = F.softmax(logits, dim=-1)
		next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
		inp_tensor = torch.cat([inp_tensor, next_token.unsqueeze(-1)], dim=-1)

print(tokenizer.decode(inp_tensor[0]))
```

## Expected behavior

I would expect a mix of the new special tokens and other tokens.


Comments:
@g-karthik 

Was able to reproduce this as well.

From the observation, I suspect the random weights being initialized for the added tokens in the final `Linear` and/or `SequenceSummary` head are such that no matter what hidden state is sent in, a special token gets the highest final scalar score. Haven't dived in to check how the random initialization is done, but if it's done from a standard unit Gaussian, I would imagine this cannot happen at every single time-step.

You should fine-tune your model on a dataset containing your added tokens, otherwise the model will very probably generate gibberish.

@LysandreJik The model is only generating special tokens. It does not generate any of the original tokens in the pre-trained model's vocabulary. I'd understand if there were special tokens generated occasionally, but that's not the case.

I understand. By adding new tokens, you're resizing the token embedding layer with some *randomly initialized values*. These values can be of an entirely different dimension to the ones currently initialized in your token embedding layer, which can lead to these tokens being overly generated.

As I said before: you should fine-tune your model on a dataset containing your added tokens, otherwise the model will very probably generate gibberish.

I'm aware that fine-tuning on a dataset containing the added tokens will bring those "entirely different dimension" (as you call them) values back to the "same dimension". But that's besides the point here. We're talking about expected behavior.

> By adding new tokens, you're resizing the token embedding layer with some randomly initialized values

It's not just the token embedding layer that'll get resized though, right? There's the final `Linear` that outputs the distribution over the vocabulary as well that would have to be adjusted to account for the new vocabulary size.

> These values can be of an entirely different dimension to the ones currently initialized in your token embedding layer

Can you please point me to the code that does this "entirely different dimension" random initialization for the added tokens? If the argument you're making is that a pre-trained model *should* generate *only-special-tokens* gibberish if special tokens were added to its vocabulary and it were resized accordingly, then I disagree with it. I would expect a mix of both, and if the random initialization can be altered to ensure the pre-trained model's behavior with and without added special tokens is *mostly* similar, that would be the best outcome for consistency. That's why I brought up the point about random initialization from a unit Gaussian earlier in this issue.

> It's not just the token embedding layer that'll get resized though, right? There's the final Linear that outputs the distribution over the vocabulary as well that would have to be adjusted to account for the new vocabulary size.

Yes, you are right. However, the embedding layer and output linear layer are shared. They have the same values, as they are tied. Resizing the embedding layer resizes the output layer at the same time.
This is done in the [_tie_or_clone_weights method](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L571), which is called by the [init_weights method](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py#L705).

> Can you please point me to the code that does this "entirely different dimension" random initialization for the added tokens?

[Here it is](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py#L341-L351).

> If the argument you're making is that a pre-trained model should generate only-special-tokens gibberish if special tokens were added to its vocabulary and it were resized accordingly, then I disagree with it.

I am not making this argument. The argument I'm making is that if you're adding *any* token to an embedding matrix **without** training the model afterwards, you will obtain gibberish.

> The argument I'm making is that if you're adding any token to an embedding matrix without training the model afterwards, you will obtain gibberish.

What @al3xpapangelis and I are saying is that a pre-trained model's behavior *before* and *after* adding *any* new token, with no *further* training done beyond the original pre-training, should be the same *on average* for the same input. We're talking about expected behavior vs. actual behavior, and what can be done to make them the same. Your argument makes sense to me, but only in the scenario where the model's not been trained originally at all.

Thanks for the pointers! Looks like the random initialization for `Linear` is from a 0-centered, 0.02 std. deviation Gaussian. I'll do some analysis to see how vectors from this distribution "vary" on average from a pre-trained embedding for a regular token.

Hey @g-karthik and hey @al3xpapangelis (cool to see you here again :-) ), 

The reason for this behavior is mainly because `lm_head` is tied to the word_embedding matrix and therefore the softmax over the output logit vectors seems to give very high values to the randomly init tokens. So this seems to suggest the distribution of trained logit vectors is very different from the rnd init ones.

I'd also suggest to play around with changing the init scheme for new tokens or just setting the newly added tokens manually to some better value:

```python
model.lm_head.weight[:, -2] = # good init vector for <USER>
model.lm_head.weight[:, -1] = # good init vector for <SYSTEM>
```

If you do this for example, the tokens won't be generated.

```python
import torch
import torch.nn.functional as F
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('distilgpt2')
tokenizer.add_special_tokens(
	{'additional_special_tokens': ['<USER>', '<SYSTEM>']}
)

model = GPT2LMHeadModel.from_pretrained('distilgpt2')
model.resize_token_embeddings(len(tokenizer))
inp_tok_ids = tokenizer.encode('I want a pepperoni pizza with mushroom')
inp_tensor = torch.LongTensor(inp_tok_ids).unsqueeze(0)
model.eval()

model.lm_head.weight[-2, :] = (torch.zeros((768,)) - 10000.0) 
model.lm_head.weight[-1, :] = (torch.zeros((768,)) - 10000.0) 

with torch.no_grad():
	for i in range(10):
		outputs = model(inp_tensor)
		logits = outputs[0][:, -1, :]
		probs = F.softmax(logits, dim=-1)
		next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
		inp_tensor = torch.cat([inp_tensor, next_token.unsqueeze(-1)], dim=-1)

print(tokenizer.decode(inp_tensor[0]))
```

@patrickvonplaten yes, I was thinking I'll try and estimate the mean and covariance of the set of values in GPT-2's pre-trained embeddings (across all of its 4 model sizes), assuming a Gaussian distribution. And then update the random initialization's mean and std. dev. accordingly in the model's [`_init_weights()`](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py#L346). That way, the random initialization comes from a distribution that's effectively "similar" to that of the pre-trained vectors, and hence decoding sequences would result in a mixture of the original tokens and added tokens.

Thanks @LysandreJik and @patrickvonplaten! I like @g-karthik suggestion, it would be nice for this bevahiour to happen automatically

i found the same problem while i was trying to fine tune gpt2 , and here is how i have solved it: 

```
# special tokens are defined
add_spe_tokens = ['
']

special_tokens_dict = {'eos_token': '
', 
                       'bos_token': '
', 
                       'pad_token': '
', 
                       'additional_special_tokens':add_spe_tokens}

# the new token is added to the tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
print(f"bos_token_id: {tokenizer.bos_token_id}")
print(f"eos_token_id: {tokenizer.eos_token_id}")
print(f"pad_token_id: {tokenizer.pad_token_id}")


# model configuration to which we add the special tokens
config = AutoConfig.from_pretrained('gpt2', 
                                    bos_token_id=tokenizer.bos_token_id,
                                    eos_token_id=tokenizer.eos_token_id,
                                    pad_token_id=tokenizer.pad_token_id,
                                    output_hidden_states=False)

# we load the pre-trained model with custom settings
model = GPT2LMHeadModel.from_pretrained('gpt2', config=config)

# model embeding resizing
print("before : ", model.lm_head.weight.shape)
model.resize_token_embeddings(len(tokenizer))

new_weights = torch.cat([model.lm_head.weight[:-4, :], torch.zeros(4, model.lm_head.weight.shape[1]) -10000])
model.lm_head.weight = torch.nn.Parameter(new_weights) 


print("after : ", model.lm_head.weight.shape)
```

```
input_text = "Once upon a time,"

input_ids = tokenizer.encode(input_text, return_tensors="pt")
print(input_ids)
output = model.generate(input_ids, max_length=50, num_return_sequences=1)

print(output)
# Decode and print the generated text
# generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
# print(generated_text)
```

but i am not sure if this will ruin the pre-trained weights while fine tuning or not