[WIP] add noisy-average new word embed init
Author: john-hewitt
Pull request: yes

# What does this PR do?

This WIP PR is intended to start a discussion around the default initialization of new word embeddings when the vocabulary of a pretrained LM is expanded.

A few existing github issues (e.g., #8472, #8039) bring up the issue that after adding new words to the `gpt2` model ("vocabulary expansion") (and before any finetuning), the model now _only_ generates the new words.

In a [blog post](https://nlp.stanford.edu//~johnhew//vocab-expansion.html), characterize when this happens (it doesn't happen for all models) -- it's due to the initialization of the new words' embeddings to small random noise. This can change the pretrained LM's distribution to a very large extent. In the blog post I demonstrate under what conditions this is the case.

I briefly also show that this "put all the probability on only the new words" behavior can also harm adaptation performance finetuning on a domain. I speculate it's because the first few gradient steps change the whole model just to remove probability from the new words!

I also show that averaging all existing embeddings to initialize the new words' embeddings guarantees that you avoid this behavior, by guaranteeing that you deviate very little from the pre-vocabulary expansion LM.

In this WIP PR, I demonstrate what this might look like on the PyTorch side, suggesting (1) giving users an explicit choice of option for how to initialize the embeddings, and (2) making "average all the existing embeddings + add noise" the default.

If you try out the following code under this commit, you'll see the differences in generation:

```
import transformers
tok = transformers.GPT2Tokenizer.from_pretrained('gpt2')
tok.add_tokens(['Aragorn', 'Frodo', 'Lothlorien'])
sent = 'I love dogs because they'
```

This replicates the current default behavior:
```
model = transformers.AutoModelForCausalLM.from_pretrained('gpt2')
model.resize_token_embeddings(len(tok), init_strategy='as_pretrained')
tok.decode(model.generate(**tok(sent, return_tensors='pt'), do_sample=True)[0])
> "'I love dogs because they Lothlorien Lothlorien Lothlorien..."
```

And this is under the "average the embeddings + add noise" behavior:
```
model = transformers.AutoModelForCausalLM.from_pretrained('gpt2')
model.resize_token_embeddings(len(tok), init_strategy='avg_emb')
tok.decode(model.generate(**tok(sent, return_tensors='pt'), do_sample=True)[0])
> "'I love dogs because they are amazing fun and not boring."
```

Naturally, we expect or intend to finetune these embeddings anyway, but I hope my blog post demonstrates that regardless, the choice of initialization can matter.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ WIP] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ todo if we move forward] Did you write any new necessary tests?


## Who can review?

Here's a few people who may be interested: @patrickvonplaten  @n1t0



Comments:
Thanks for the pull request @john-hewitt - I'm inclined to add such a feature. I think it's a good addition and necessary since we've seen issues about this before. 

What do you think @LysandreJik @sgugger ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

any updates on this?