All `meta-llama/Llama-2-*-hf` models have incorrect `max_position_embeddings`
Author: zijian-hu
Pull request: no

### System Info


- `transformers` version: 4.31.0
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.34
- Python version: 3.8.13
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The hugging face version of Llama 2 have `max_position_embeddings` set to `2048` instead of `4096` in the config file. I am unsure if it's just an incorrect setting or if the models need to be converted again.

See the below links for detail:
- [`meta-llama/Llama-2-7b-hf` config.json](https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/config.json#L13)
- [`meta-llama/Llama-2-7b-chat-hf` config.json](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/main/config.json#L13)
- [`meta-llama/Llama-2-13b-hf` config.json](https://huggingface.co/meta-llama/Llama-2-13b-hf/blob/main/config.json#L13)
- [`meta-llama/Llama-2-13b-chat-hf` config.json](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf/blob/main/config.json#L12)
- [`meta-llama/Llama-2-70b-hf` config.json](https://huggingface.co/meta-llama/Llama-2-70b-hf/blob/main/config.json#L13)
- [`meta-llama/Llama-2-70b-chat-hf` config.json](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf/blob/main/config.json#L12)

If helpful, someone posted an issue in [Meta's official repo](https://github.com/facebookresearch/llama/issues/359).

### Expected behavior

According to Meta, Llama 2 has a context length `4096`. This should be the `max_position_embeddings` value in all the `config.json` files. If the model checkpoints on hugging face are not correctly converted, they should be converted again using the correct configuration.

Comments:
Hey! 👋🏻 
Indeed the paper does mention that the default `max_position_embeddings` is `4096`, and no, the models ~don't have~ should not have  to be reconverted: 
```
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", max_position_embeddings=4096)
```
worked out of the box for me. The config should probably be updated, the previous choice is explained by the fact that in all the demonstrations [`example_chat_completion`](https://github.com/facebookresearch/llama/blob/main/example_chat_completion.py) and [`example_text_completion`](https://github.com/facebookresearch/llama/blob/main/example_text_completion.py) the `max_position_embeddings` was lowered (on purpose it seems?). 

I also noticed [this](https://huggingface.co/daryl149/llama-2-7b-chat-hf/commit/d8654a4f69178a0c9260cf730241ebac2e72b923) which is why I am investigating! Github issue on the original repo: [#359 ](https://github.com/facebookresearch/llama/issues/359#issuecomment-1640876808)

The `max_position_embeddings` only affects the ROPE embedding layer, which is computed on the fly. 

Okay, so currently we have[ a safeguard](https://github.com/ArthurZucker/transformers/blob/050c4a48f77e42b9d5cd87fccaac955950799acc/src/transformers/models/llama/modeling_llama.py#L118-L120): 
```python
        if seq_len > self.max_seq_len_cached:
            self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)
```
which is not the best in terms of compute, but produces the correct values for now 😉 
This also explains why changing for the actual value is wrong: the cos sin will not be re-computed! 

I'll open fixes in transformers (make inv_freq non persistent) and will push updated checkpoints

Thank you so much for the update! I just took a look at the code; this safeguard is already part of the `transformers v4.31.0` release.

It would be great if you could let me know the correct way to use Llama 2 if we want to maintain the advertised `4096` context length without degrading the performance. Should we just pass `max_position_embeddings=4096` as mentioned earlier? Or should we use `max_position_embeddings=2048` until the problem is fully resolved?
```python
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", max_position_embeddings=4096)
```

I updated the configuration file of all the models to make sure you don't need to do anything! 
The performance degradation was not reported again, so If it comes up, I will make sure to adresse it! 

Can confirm that just updating the config file works fine.

Updated all configs online, closing as it is fixed! 🤗