Performance discrepancy between main branch and transformers 4.31.0 for WizardLM/WizardCoder-Python-34B-V1.0
Author: loubnabnl
Pull request: no

### System Info

- `transformers` version: 4.33.0.dev0
- Platform: Linux-5.15.0-1023-aws-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:no

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

For [WizardLM/WizardCoder-Python-34B-V1.0](https://huggingface.co/WizardLM/WizardCoder-Python-34B-V1.0), using `transformers` from main seems to give lower quality generations compared to transformers 4.31.0. I was initially comparing to generations from [vllm](https://github.com/vllm-project/vllm) which seemed better, turned out downgrading transformers fixes the discrepancy.

Below is an example prompt (from an evaluation benchmark) where it doesn't enclose solution in ` ```python` backticks and give an explanation of the code when using main (happens also for other prompts and the code completions are of lower quality since evaluation score drops from 70% to 60%).


```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# using generate
prompt = 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n\n### Instruction:\nCreate a Python script for this problem:\nfrom typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n    """ Check if in given list of numbers, are any two numbers closer to each other than\n    given threshold.\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n    False\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n    True\n    """\n\n\n### Response:'

device="cuda"
tokenizer = AutoTokenizer.from_pretrained("WizardLM/WizardCoder-Python-34B-V1.0")
model = AutoModelForCausalLM.from_pretrained("WizardLM/WizardCoder-Python-34B-V1.0", torch_dtype=torch.float16).to(device)
inputs = tokenizer.encode(prompt, return_tensors="pt")
inputs = inputs.to(device)
outputs = model.generate(inputs, do_sample=False, max_length=2048)
print(tokenizer.decode(outputs[0]))
```

Result from main branch fo `transformers`
```
Here's the Python script for the given problem:


from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    for i in range(len(numbers)):
        for j in range(i+1, len(numbers)):
            if abs(numbers[i] - numbers[j]) < threshold:
                return True
    return False


# Testing the function
print(has_close_elements([1.0, 2.0, 3.0], 0.5)) # False
print(has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)) # True 

```
After downgrading to transformers 4.31.0
````
Here's the Python script for the given problem:

```python
from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    for i in range(len(numbers)):
        for j in range(i+1, len(numbers)):
            if abs(numbers[i] - numbers[j]) < threshold:
                return True
    return False
```

The `has_close_elements` function takes two arguments: a list of numbers (`numbers`) and a threshold value (`threshold`). It returns `True` if there are any two numbers in the list that are closer to each other than the given threshold, and `False` otherwise.

The function uses a nested loop to compare each pair of numbers in the list. If the absolute difference between two numbers is less than the threshold, the function returns `True`. If no such pair is found, the function returns `False`. 

````
This second generation is the same output as when using `vllm` for generation.
The text encodings match, so probably not a tokenization issue. Let me know if I should test other transformers versions

### Expected behavior

Get same high quality generations wit main branch as with transformers 4.31.0

Comments:
Thanks for reporting! Looking at the model it is a Llama architecture, not sure what exactly changed in the llama modeling code between the previous release. cc @ArthurZucker who has a better idea than me regarding Llama modeling code
Maybe 015f8e110d270a0ad42de4ae5b98198d69eb1964 is the culprit but really not sure. Would you be able to quickly test the generation quality with and without that commit? 🙏 

Lots of things could be at play here, `generation_config` and `tokenization`  would be my main culprits, as the `Llama` code updates did not change much. I'll investigate, thanks for reporting! 

I tried generation with and without the CodeLLaMa PR, and indeed they're good before but worse after it.

Ah then the `rope_theta` is the only culprit here! If you look at the configuration it is set to `  "rope_theta": 1000000,` 

Meaning the config is wrong on the wizard side, it should be `10_000`

Ah maybe they used 10_000 in training, I thought it had the correct value since they fine-tuned CodeLlama

I can confirm that this fixes the generations when using main, thanks a lot @ArthurZucker & @younesbelkada !