Standard Generation and Prompt Lookup Decoding (PLD) generations are not consistent
Author: shwetha0312
Pull request: no
Labels: name: Generation

### System Info

- `transformers` version: 4.40.1
- Platform: Linux-5.10.192-183.736.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.12
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.3
- Accelerate version: 0.29.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, A100 machine with one 80 GB GPU
- Using distributed or parallel set-up in script?: No

### Who can help?

Settings:
Greedy decoding, precision dtype - float32
PLD is generating inconsistent outputs compared to standard decoding with the above settings. I tried with a different model as well (llama2), the results are still inconsistent.

However this is not the case when I use an older version of transformers (for the same settings). I tested the same code with the following system info and the generations are equal:

- `transformers` version: 4.37.1
- Platform: Linux-5.10.192-183.736.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.19
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.21.0
- Accelerate config:    not found 
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, A100 machine with one 80 GB GPU
- Using distributed or parallel set-up in script?: No

Related issue raised in the PLD code repository: https://github.com/apoorvumang/prompt-lookup-decoding/issues/5

@gante 



### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import transformers
import sys,torch
import tqdm,pdb
from fastchat.model import get_conversation_template
from transformers import AutoModelForCausalLM, AutoTokenizer,set_seed

print('Currently using transformers version:',transformers.__version__) 
device = "cuda" # the device to load the model onto
model_name="lmsys/vicuna-7b-v1.3"
model = AutoModelForCausalLM.from_pretrained(model_name,torch_dtype=torch.float32)
tokenizer = AutoTokenizer.from_pretrained(model_name)
model.to(device)

question="Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook?"
conv = get_conversation_template("vicuna")
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
ip_ids = inputs.input_ids


modes=['pld','normal']
outputs={mode:[] for mode in modes}


# ****
generated_ids = model.generate(ip_ids,
                       max_new_tokens=1000,
                       do_sample=False,
                       return_dict_in_generate=True)

outputs['normal'].append(tokenizer.batch_decode(generated_ids['sequences'])[0])

# ****
generated_ids = model.generate(ip_ids,
                       max_new_tokens=1000,
                       do_sample=False,
                       return_dict_in_generate=True,
                       prompt_lookup_num_tokens=10,
                       # max_matching_ngram_size=1,
                      output_hidden_states=True)
outputs['pld'].append(tokenizer.batch_decode(generated_ids['sequences'])[0])
print('Are outputs same?',outputs['pld']==outputs['normal'])
# print(outputs)
```
I have attached the generated outputs in the text file.
[output.txt](https://github.com/huggingface/transformers/files/15090016/output.txt)

Please let me know if I have overlooked some detail..


### Expected behavior

I expect both of the outputs to be the same. 

Comments:
The provided script work fine in the latest version, should have been fixed recently. You can upgrade transformers with `pip install transformers==4.41.0.dev0` to get the correct generation 🤗 

I get `ERROR: No matching distribution found for transformers==4.41.0.dev0` when I try to install

```
ERROR: Could not find a version that satisfies the requirement transformers==4.41.0.dev0 (from versions: 0.1, 2.0.0, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.6.0, 2.7.0, 2.8.0, 2.9.0, 2.9.1, 2.10.0, 2.11.0, 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0, 3.3.1, 3.4.0, 3.5.0, 3.5.1, 4.0.0rc1, 4.0.0, 4.0.1, 4.1.0, 4.1.1, 4.2.0, 4.2.1, 4.2.2, 4.3.0rc1, 4.3.0, 4.3.1, 4.3.2, 4.3.3, 4.4.0, 4.4.1, 4.4.2, 4.5.0, 4.5.1, 4.6.0, 4.6.1, 4.7.0, 4.8.0, 4.8.1, 4.8.2, 4.9.0, 4.9.1, 4.9.2, 4.10.0, 4.10.1, 4.10.2, 4.10.3, 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.12.0, 4.12.1, 4.12.2, 4.12.3, 4.12.4, 4.12.5, 4.13.0, 4.14.1, 4.15.0, 4.16.0, 4.16.1, 4.16.2, 4.17.0, 4.18.0, 4.19.0, 4.19.1, 4.19.2, 4.19.3, 4.19.4, 4.20.0, 4.20.1, 4.21.0, 4.21.1, 4.21.2, 4.21.3, 4.22.0, 4.22.1, 4.22.2, 4.23.0, 4.23.1, 4.24.0, 4.25.1, 4.26.0, 4.26.1, 4.27.0, 4.27.1, 4.27.2, 4.27.3, 4.27.4, 4.28.0, 4.28.1, 4.29.0, 4.29.1, 4.29.2, 4.30.0, 4.30.1, 4.30.2, 4.31.0, 4.32.0, 4.32.1, 4.33.0, 4.33.1, 4.33.2, 4.33.3, 4.34.0, 4.34.1, 4.35.0, 4.35.1, 4.35.2, 4.36.0, 4.36.1, 4.36.2, 4.37.0, 4.37.1, 4.37.2, 4.38.0, 4.38.1, 4.38.2, 4.39.0, 4.39.1, 4.39.2, 4.39.3, 4.40.0, 4.40.1)
ERROR: No matching distribution found for transformers==4.41.0.dev0
```

Is there something I need to enable to install that version? @zucchini-nlp 

The latest version I'm able to install is 4.40.1 and I'm getting the same issue as @shwetha0312 

```
{'pld': ["
 A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook? ASSISTANT: If I have just overtaken the second person, then overt,ed the second person, then my current position, my current current position position, then, then my current position position, where, person, position, person, where person person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person person person person, person person, person, person, person, person, person, person, person, person person person person person person, person, person, person, person, person, person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person person"],
 'normal': ["
 A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook? ASSISTANT: If I have just overtaken the second person, then I am currently in second place. The person I just overtook would be in third place.
"]}
```

@zucchini-nlp @shwetha0312  I was able to install the latest version with `!pip install --upgrade git+https://github.com/huggingface/transformers.git` and issue seems to be resolved, thanks!

It was probably related to #30042 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.