Add RoPE dimensions to LLaMA config
Author: xunkai55
Pull request: yes

# What does this PR do?

This PR enables partial RoPE that can be applied to Llama variants / siblings.

It's discussed that only applying a half of positional encodings could lead to slight improvements on model performance. Example: https://github.com/lucidrains/x-transformers/issues/40

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] (Not Applicable?) Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker  @younesbelkada

Comments:
cc @gante @ArthurZucker 

I also would like to get your advise here -

The CI "repository consistency" fails because this PR upgrades RoPE implementation. However, some models copied RoPE from Llama and marked the functions as copied (example: https://github.com/huggingface/transformers/blob/main/src/transformers/models/falcon/modeling_falcon.py#L82)

I would like to also update all these implementations as long as you agree.

Thanks!

@xunkai55 If these changes are approved for the Llama implementations then we would want to have the same updates applied to copied code. It's easier for reviewers when the diff is smaller, so I would wait until the initial proposal has been reviewed and then apply to the rest of the code base (this can easily be done by running `make fix-copies`)

Hi @ArthurZucker , thanks for the pointer! I updated the code to better align with existing persimmon implementation. Please take a look.

Thanks!

Hey! What I meant is that the porting of ChatGLM should be straightforward if we use the Persimmon architecture rather than  the LlamaArchitecture (we would need to convert the q, k, v layer into qkv matrix and simply allow the bias at a config level like it was done [here](https://github.com/huggingface/transformers/pull/26302)). The thing is changing llama in such a way (adding partial rotary) goes against our philosophy, while the change in persimmon for bias is alright. 
So changes would be similar to #26302 but for Persimmon. WDYT? 

Thank you Authur for the quick review!

Persimmon and Llama are both important and brilliant works. However, we prefer to port models to Llama here, as the community has (much) more open projects supporting Llama rather than Persimmon.

I'm not 100% following here on why #26302 met our philosophy but this change doesn't - I feel both changes involve a new field in `LlamaConfig` and enable some variants.

Sure, our philosophy is 1 model 1 file, and in that regard adding a bias does not change the logic of the code, but adding partial rotation changes the logic and adds code that is completely unrelated to Llama. 

In terms of readability, someone who just wants to see the Llama model will have an additional burden and code that is unrelated. Moreover a lot of people would then want to add custom changes and it's hard to draw the line, which for us is changing the foward logic with if else and adding such changes, specifically when we have a PersimmonModel which has the **exact** same API as Llama so no reason for it not to be supported outside.

So I don't think there is a strong enough incentive to change Llama 🤗 

Understood, and that makes sense. Closing this PR then. Thank you Arthur for the suggestions!