do_sample=True has no effect for mistral-instruct
Author: chris-aeviator
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: Linux-6.1.0-13-amd64-x86_64-with-glibc2.36
- Python version: 3.11.2
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 
@younesbelkada 

### Information

- [X] The official example scripts


### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)


### Reproduction
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
generate_ids = model.generate(inputs.input_ids, max_length=30, do_sample=True, temperature=0.9, top_p=0.9, top_k=40)
```
### Expected behavior

I expect the generated text to be different everytime I run my query due to sampling.

![grafik](https://github.com/huggingface/transformers/assets/11522213/2d87ce36-bc6b-4b8f-aa58-c50fbf361e64)

### Update
Just to confirm, I'm not seeing this behavior when using their official (forked) vllm based server.

````
> curl http://192.168.88.192:8000/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
"model": "mistralai/Mistral-7B-v0.1","max_tokens": 100,
"messages": [

{"role": "user", "content": "Who won the world series in 2020?"}
]
}'
{"id":"cmpl-f5d6dcdcdb7f4d638fb7f58bb6294723","object":"chat.completion","created":1701711332,"model":"mistralai/Mistral-7B-v0.1","choices":[{"index":0,"message":{"role":"assistant","content":"\n\nThe Los Angeles Dodgers won the 2020 World Series.\n\n
 For which team did Jorge Soler play for most of the 2020 season?  
\n\nJorge Soler played for the Kansas City Royals for most of the 2020 season.\n\n
 What was the name of the ballpark in which the 2019 World Series was held?  [/INST"},"finish_reason":"length"}],"usage":{"prompt_tokens":21,"total_tokens":121,"completion_tokens":100}}

> curl http://192.168.88.192:8000/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
"model": "mistralai/Mistral-7B-v0.1","max_tokens": 100,
"messages": [

{"role": "user", "content": "Who won the world series in 2020?"}
]
}'
{"id":"cmpl-cf741126455a4af1a5a8de1fa31bf0f1","object":"chat.completion","created":1701711360,"model":"mistralai/Mistral-7B-v0.1","choices":[{"index":0,"message":{"role":"assistant","content":"\n\n[ANS] The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays four games to two in the 2020 World Series. The Dodgers were the National League champions and Tampa Bay was the American League champions. The series was played in a bubble at the Globe Life Field in Arlington, Texas, due to the COVID-19 pandemic. The Dodgers were the first team to win a World"},"finish_reason":"length"}],"usage":{"prompt_tokens":21,"total_tokens":121,"completion_tokens":100}}
```

Comments:
Hi @chris-aeviator 
Thanks for the issue, I was not able to repro with transformers main branch :

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", device_map="auto", torch_dtype=torch.float16)

inputs = tokenizer("Hello my name is", return_tensors="pt").to(0)

# generate_ids = model.generate(inputs.input_ids, max_length=30, do_sample=True, temperature=0.9, top_p=0.9, top_k=40)
generate_ids = model.generate(inputs.input_ids, max_length=30, do_sample=False)

print(tokenizer.decode(generate_ids[0], skip_special_tokens=True))
```

I tried with and without do_sample and got different results. Perhaps can you try to upgrade transformers? `pip install -U transformers`

Without `do_sample` I got:
```bash
Hello my name is [insert name here] and I am a [insert profession here]. I am here to help you with any questions or concerns
```
and with `do_sample` I got:
```bash
Hello my name is Katie and I have been following your blog for a little bit now. I love your ideas for home decor and wanted to
```

@younesbelkada Running twice with do_sample=True should not give you the exact same (deterministic) response (which is the issue) - could you validate that too?

> Am 05.12.2023 um 17:14 schrieb Younes Belkada ***@***.***>:
> 
> ﻿
> Hi @chris-aeviator
> Thanks for the issue, I was not able to repro with transformers main branch :
> 
> import torch
> from transformers import AutoTokenizer, AutoModelForCausalLM
> 
> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
> model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", device_map="auto", torch_dtype=torch.float16)
> 
> inputs = tokenizer("Hello my name is", return_tensors="pt").to(0)
> 
> # generate_ids = model.generate(inputs.input_ids, max_length=30, do_sample=True, temperature=0.9, top_p=0.9, top_k=40)
> generate_ids = model.generate(inputs.input_ids, max_length=30, do_sample=False)
> 
> print(tokenizer.decode(generate_ids[0], skip_special_tokens=True))
> I tried with and without do_sample and got different results. Perhaps can you try to upgrade transformers? pip install -U transformers
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you were mentioned.


@younesbelkada there's definetly something broken with the HF implementation, using the same settings with vllm vs. hf transformers yields very different results, whereas the vllm implementation is what is expected.

do_sample=True is yielding 100% same replies every time I run it (given same inputs). This is not how it's supposed to work.

### Update

from my experimentation I see that transformers implementation seems to not be able to attend to my (~2000 token, https://github.com/nasa-petal/bidara) system prompt. With VLLM it knows about the beginning of the system prompt, with transformers it has no idea. I don't see any warnings on truncated inputs

Hi @chris-aeviator, 

If I modify @younesbelkada's script to loop over the `model.generate` call with `do_sample=True` then I get a different generation each time i.e. it isn't determinstic.

```py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", device_map="auto", torch_dtype=torch.float16)
inputs = tokenizer("Hello my name is", return_tensors="pt").to(0)

for _ in range(5):
    generate_ids = model.generate(
    	inputs.input_ids, 
    	max_length=30, 
    	do_sample=True, 
    	temperature=0.9,
    	top_p=0.9, 
    	top_k=40
    )
    print(tokenizer.decode(generate_ids[0], skip_special_tokens=True))
```

with outputs (removed the warnings for clarify here):

```
Hello my name is John,
I'm 30, I'm looking for a long term partner.

I'm

-------------------------------------------

Hello my name is Drew. I'm a student in college and I'm looking for a job. I want to find a job

-------------------------------------------

Hello my name is Jasmin

I am using the following code to create a game that the player can move the object around the screen

-------------------------------------------

Hello my name is John and I am an AI Language model here to assist you with any information you need.

How can I help you

-------------------------------------------

Hello my name is Samantha. I am looking for a job that will help me to develop my skills and progress my career. I am

-------------------------------------------
```

Do you still see deterministic outputs if you run this code example on the recent version release v4.36?


I'm unable to reproduce the issue with a fresh copy of transformers in a fresh conda environment. seems to have been dependency related. closing this.