🔴 🚨  Resizing tokens embeddings: initialize from old embeddings' normal distribution.
Author: abuelnasr0
Pull request: yes

# What does this PR do?

This PR initializes new tokens embeddings from a normal distribution with the old embeddings' mean and covariance. as described in this article https://nlp.stanford.edu/~johnhew/vocab-expansion.html. Thanks to this article, you can now add new tokens to your model without affecting its generation accuracy. 

Fixes #32948 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

cc: @ArthurZucker @LysandreJik 



Comments:
This gist shows the before and after GPT2 model generation after adding new tokens: https://colab.research.google.com/gist/abuelnasr0/7cc8decff72ccdbcf5073ad8259ee360/embedding_resize.ipynb

Nice! Thanks @abuelnasr0. cc @Rocketknight1 in case you have some bandwidth to do a first review?

Yes, I'm happy to take it, it seems like a great PR!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33325). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @LysandreJik 
And thanks @Rocketknight1 for the review and the comments. I have added a test case to check if the new embeddings' mean is close to the original embeddings' mean, I think that will reduce the effect of any outlier values.
Feel free to let me know if there is anything else I should add.

@ArthurZucker Sorry for the delay. last week was very busy for me.
I have added the flag to `resize_token_embeddings`,  `_get_resized_embeddings`, and `_get_resized_lm_head`. 
`_get_resized_lm_head` is very important for models that have untied weights. This should have been added from the beginning but I don't know how I didn't notice that.

Bias for linear lm_heads is not included in the article. but I have came to the conclusion that initializing with zero is the best solution.
here is the proof given the article:

$p_{\theta'}(w_i \mid w_{1:i-1}) = \frac{\exp(h_{i-1}^{\top} e_{w_i} + b_{w_i})}{Z + \exp(h_{i-1}^\top e_{n+1} +  b_{n+1} ) }$
$p_{\theta'}(w_i \mid w_{1:i-1}) = \frac{\exp(h_{i-1}^{\top} e_{w_i} + b_{w_i})}{Z + \exp(h_{i-1}^\top e_{n+1} ) exp( b_{n+1})}$

If the new bias equals zero, then exp(0) = 1, and then the proof in the article will remain the same.


@abuelnasr0 I haven't dived into the math, but that seems counter-intuitive to me. Most models do not have a bias on the output head, but if they do, I would guess that the bias is usually large and negative for most tokens (I can't test this, though, because I can't find any examples of models with bias on the output head!)

If the average bias is large and negative, then initializing new tokens with the mean embedding but a zero bias will mean that logits for the new token will be very large, right? 

UPDATE: I finally found an example of an old model with a bias on the lm_head (`Salesforce/ctrl`). The mean bias was `-0.05`, which is a much smaller magnitude than I expected. This probably isn't a big problem - a zero embedding is fine!

@Rocketknight1 That's right. 
I didn't consider that the new value of `Z` has also bias in it.

After considering the new value of `Z`, the new bias should be initialized with the old bias mean. I have tried to be more focused this time. I hope I am not wrong again :sweat_smile:

this link contains a proof that should be replaced with "Averaging bounds the KL-divergence" part in the article: https://imgur.com/a/ZZQ3Mwk

I have initialized the new bias like this:
```python
new_lm_head.bias.data.normal_(mean=bias_mean, std=bias_std * 1e-5)
```
I have multiplied the std by 1e-5 just because the article initializes the embeddings by multiplying the covariance by 1e-5. I don't really know why, maybe to make the new embeddings alot closer to the mean.

The only thing left is to fix the failing test which are related (example torch for example) 🤗 

@ArthurZucker Thanks for your review.
Regarding tests, all tests have passed after rebasing. 

All good for me! 🤗 

Congrats on the PR @abuelnasr0, and thank you!

@Rocketknight1 Thank you for the reviews and the help.