why max_position_embeddings = 2048  for llama2 
Author: ckfgihub
Pull request: no
Labels: name: Good First Issue

### System Info

when I use under cmd to convert llama2 7B model and  max_position_embeddings = 2048,but  llama2 this max_position_embeddings = 4096.

```
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

{
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 2048,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.34.1",
  "use_cache": true,
  "vocab_size": 32000
}



https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py


```
    if base > 10000.0:
        max_position_embeddings = 16384
    else:
        max_position_embeddings = 2048
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

when I use under cmd to convert llama2 7B model and  max_position_embeddings = 2048,but  llama2 this max_position_embeddings = 4096.

```
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

{
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 2048,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.34.1",
  "use_cache": true,
  "vocab_size": 32000
}



https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py


```
    if base > 10000.0:
        max_position_embeddings = 16384
    else:
        max_position_embeddings = 2048
```

### Expected behavior

when I use under cmd to convert llama2 7B model and  max_position_embeddings = 2048,but  llama2 this max_position_embeddings = 4096.

```
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

{
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 2048,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 10000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.34.1",
  "use_cache": true,
  "vocab_size": 32000
}



https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py


```
    if base > 10000.0:
        max_position_embeddings = 16384
    else:
        max_position_embeddings = 2048
```

Comments:
Hi @ckfgihub, thanks for raising this issue! 

The conversion script was originally written for the original Llama, which I suspect is why we have this discrepancy. cc @ArthurZucker to confirm. If so, it'll be a simple case of updating the condition in the script. Happy to review a PR for anyone who would like to fix this.  

Yes, Llama1 one had 2048. I don't think that this is an issue as you should only need to update this in the config. Feel free to open a PR for a fix! 

Hi @ArthurZucker , I can take this one!

Hey folks, just to confirm based on this [PR](https://github.com/huggingface/transformers/pull/28754) is this issue done and fixed? If not is there work I can hop into to help out? Thanks

It's done let's close it!