Making Roformer models compatible with pre-trained Roformer v2 models
Author: sijunhe
Pull request: yes

# What does this PR do?

RoFormer v2 is a more recent and lightweight version of RoFormer. The differences between the two models are:
- RoFormer v2 removed the bias term from all the attention modules
- RoFormer v2 used a simple RMS norm instead of LayerNorm

Currently, loading a pre-trained RoFormer v2 model such as [this one](https://huggingface.co/junnyu/roformer_v2_chinese_char_base) with RoFormer will raise a lot of "newly initialized but not found in checkpoint" warning. This PR ensures that the redundant weights are not created when a RoFormer V2 config is provided.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@JunnYu who contributed RoFormer to HF
@patrickvonplaten who reviewed RoFormer before

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18084). All of your documentation changes will be reflected on that endpoint.

Thanks a lot for your PR, @sijunhe!

In this situation, we would favor adding a new model architecture instead of editing the existing one to add support for a newer model. 

The gist of it is that:
- If the changes are not part of the original codebase, original paper, or original pretrained weights
- If the initial checkpoints cannot be loaded in the architecture that will be enabled

then a new model architecture is warranted. You can read about this aspect of our philosophy [here](https://huggingface.co/blog/transformers-design-philosophy).

We have a tool to allow you to generate a model exactly as RoFormer to add your contribution here: [add-new-model-like](https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command). In this situation, the arguments that you add here likely don't need to exist: I suppose the rotary operations will always be there in v2, and the `rms_norm` will always be used as well. Is that correct?

Thanks!

Hi @LysandreJik. Thanks for sending over HF's philosophy. It was a good read!

I'd agree with you. But in this case, the difference between RoFormer and RoFormer V2 is so small and I am not sure if it's worth a whole new model and tests. Technically, it's possible to load RoFormer V2 weights with the current RoFormer class and we would just have some redundant weights that would be randomly initialized. 

I think a counter example here is the BERT model and its 3 different kinds of position embeddings (absolute, relative_key and relative_key_query). And in this case, the architectural difference between RoFormer and RoFormer V2 is much smaller than BERT.

Good point regarding BERT, but that's actually a mistake from our part when the philosophy was still evolving :sweat_smile:. Same with GPT-2 and some arguments for it to scale better.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.