Move instantiation of RoPE from MistralAttention to MistralModel
Author: Speymanhs
Pull request: yes

# What does this PR do?

- Fixes the scope for defining the Rotary Positional Embedding in Mistral:  instantiation of RoPE is moved from `MistralAttention` to `MistralModel `
- Each `MistralAttention` instance can use the same instance of RoPE as RoPE follows the same rule for applying positional embedding for every layer.

# What is the impact of this PR?
- **When loading the original Mistral model from pre-trained weights, this reduces the VRAM required for loading the model from 14.9 GB to 14.3 GB**
- **For those working on Deep Learning/LLM Architecture, this can be extremely important. For example, by reducing the number of heads for custom applications like training student models, like in building a baby Mistral with a single head and 3 hidden_layers (using exactly similar config aside from ```num_hiddel_layers``` set to 1 and ```num_hidden_layers``` set to 3: ```teacher = MistralForCausalLM(MistralConfig(num_attention_heads=1, num_hidden_layers=10))```), the difference in VRAM requirement becomes even more significant between the two version (9.78GB versus 5.68GB)**

# How did I test?
To make sure these changes are consistent with other files and the model still works, I ran the following line with ```MistralForCausalLM```once imported after my changes and once imported using the original definition:
```
# TEACHER_DIR used below is the directory where the weights of the mistral model are stored.
teacher = MistralForCausalLM.from_pretrained(TEACHER_DIR, torch_dtype=torch.bfloat16).eval().to('cuda')
```

Then I tested the output of both models to make sure my changes do not cause any problem with the model performance:
```
prompts = [
"The capital of France is", 
"The largest ocean on the Earth is", 
"2 + 2 = ", 
"2 * 1 = ",
"The concept of a cat following a mouse is ",]
model = teacher
tokenizer = AutoTokenizer.from_pretrained(TEACHER_DIR)
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)
outputs = model.generate(**inputs, max_length=128, do_sample=True, temperature=0.01, top_p=0.9)
for i in range(len(prompts)):
    print(outputs[i])
```

The results are both exactly the same as below :

```
The capital of France is a city that is full of history, culture, and art. It is a city that has been around for centuries and has seen many changes over the years. The city is home to some of the most famous landmarks in the world, including the Eiffel Tower, the Louvre Museum, and Notre Dame Cathedral.

The largest ocean on the Earth is the Pacific Ocean. It is the largest ocean in the world.

2 + 2 = 4
The above equation is a simple one. It is a simple equation that is easy to understand.

2 * 1 = 2
2 * 2 = 4
2 * 3 = 6
2 * 4 = 8
2 * 5 = 10
2 * 6 = 12

The concept of a cat following a mouse is 100% true. Cats are natural hunters and they will follow their prey wherever it goes. This is because cats have a strong instinct to hunt and they will do whatever it takes to catch their prey.
```

By increasing the temperature still both models are producing sane arguments but their output is obviously different as the temperature introduces randomness to the predicted tokens in different runs.

**Note: I took a quick look and I figured these changes seem to be applicable to Llama2 implementation as well!**

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker and @younesbelkada


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Ah, right. Could you please elaborate more on why for precisions reasons this should not be cached?

As more context, I previously worked with llama2 from the llama repository [here ](https://github.com/meta-llama/llama/blob/main/llama/model.py) and they precomputed the cosine and sine frequencies in the model initialisation (under `freqs_cis`) and then passed this to each Attention layer as an argument during forward pass but the transformer library version instantiates the class in each layer.

😅 see #29285 for more details.

Right, thank you for this!

So I think this could be maybe of use for those caring about saving up to the last byte but comes at a cost of precision loss!