top-p sampling gives different results even after fixing all random seeds
Author: jasonppy
Pull request: no
Labels: name: bug

name: Generation

### System Info

python: 3.11.9
transformers: 4.43.3
torch: 2.4.0+cu121

### Who can help?

@ArthurZucker @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When using pipeline to generate text using llama3.1 8B, even if I have fixed all the random seeds, with the same prompt, the output will be different everytime. If I set do_sample=False, output will be the same each time.

I understand that do_sample does top_p sampling (or top_k) and therefore there are randomness, but since I have fixed the seed, shouldn't they be the same?

below is the script to reproduce:

```python
import os, random, numpy as np
import transformers
import torch
cache_dir = "some_dir"
model_size="8B"
model_id = f"meta-llama/Meta-Llama-3.1-{model_size}-Instruct"
def seed_everything(seed=1):
    os.environ['PYTHONHASHSEED'] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    # torch.use_deterministic_algorithms(True)
seed_everything(1)
pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16, "cache_dir": cache_dir},
    device_map="auto",
    # do_sample=False
)
message = [
    {"role": "system", "content": "You are a helpful assistant that generate random sentences."},
    {"role": "user", "content": "please generate a random sentence."}
]
for _ in range(5):
    outputs = pipeline(
                message,
                max_new_tokens = 2048
            )
    print(outputs[0]["generated_text"][-1]['content'])

```

### Expected behavior

when you fixed the random seed, with the same prompt, each generation should give the same results.

Comments:
According to the page, the default configuration for do_sample is set to True. This means that if you want to achieve consistent results across repeated inferences, you should pass do_sample=False. 

https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/generation_config.json

Yes, but setting do_sample = False will lead to greedy sampling which I would like to avoid.
Theoretically, if all random seeds are fixed, top_p sampling should produce the same samples if the prompt is the same

cc @gante @zucchini-nlp 

Seems like you need to set the seed before every generation call to get the identical results. The below works for me
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed

model_id = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id)


inputs = tokenizer("Hello there", return_tensors="pt").to("cuda")

for i in range(3):
    set_seed(1)
    outputs = model.generate(**inputs, do_sample=True, top_p=100, max_new_tokens=20)
    response = tokenizer.batch_decode(outputs)[0]
    print(response)
```

Setting a fixed seed value only ensures that you get the same results when running this script multiple times, rather than getting the same value every time you call a random function within the same script. Otherwise, all random functions would become ineffective after setting the seed.
<img width="516" alt="image" src="https://github.com/user-attachments/assets/620ac08a-8797-4f52-a7ca-fcc4ee7c7a89">