transformers.set_seed seems to do nothing
Author: ghost
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante, @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. install transformes with support for alpaca model
2. run this code with one seed
3. run this code with any other seed
4. see that the results are the same
```
from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM, set_seed
from torch import float16, compile, no_grad


set_seed(621)

# Enhances prompt
def enhance_prompt(prompt, input=None):
    if input:
        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Input:
{input}

### Response:"""
    else:
        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:"""

# Gets response from Alpaca
def get_response(prompt):
    with no_grad():
        outputs = alpaca.generate(input_ids=tokenizer(prompt, return_tensors="pt").input_ids.to("cuda"), generation_config=generation_config, return_dict_in_generate=True, output_scores=True)
    outputs = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
    return outputs.split("### Response:")[1]


# Sets up Alpaca
tokenizer = LlamaTokenizer.from_pretrained("chainyo/alpaca-lora-7b")
alpaca = LlamaForCausalLM.from_pretrained("chainyo/alpaca-lora-7b", load_in_8bit=True, torch_dtype=float16, device_map="auto")
generation_config = GenerationConfig(temperature=0.2, top_p=0.75, top_k=40, num_beams=4, max_new_tokens=64)
alpaca.eval()
compile(alpaca)

# Gets output from Alpaca
prompt = enhance_prompt("Write a simple poem about flowers.")
out = get_response(prompt)

# Prints Alpaca's output
print(out)
```

### Expected behavior

Model will output two different answers but now it gives the same every seed I try.

Comments:
@mojejmenojehonza 👋

Two notes:
1. You should pass `do_sample=True` in your generation config or in your `.generate()` call. Most models have it off by default, causing the generation to be deterministic (and ignoring parameters like `temperature`, `top_k`, etc).
2. With `temperature=0.2`, the relative weight of the most likely logits is massively increased, making generation almost deterministic. Even if there are no bugs in your script, it's far from guaranteed that two different seeds produce different outputs with such low temperature :)

@gante
Thanks worked like a charm :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@AliHaider20 this is the `transformers` repo, not openai

My issue is the same. Transformer seed is not working for me

@AliHaider20 do you mind opening a new issue with your specific code and problem so that we may take a look? Thank you.

I think I'll remove the comment cause that's the problem with OpenAI, and it can't be solved. If anyone gets to this page, just for reference:

[Model level controls for consistent outputs - seed and system_fingerprint](https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter#model-level-controls-for-consistent-outputs---seed-and-system_fingerprint)
[seed](https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter#seed)
If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend.

[system_fingerprint](https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter#system_fingerprint)
This fingerprint represents the backend configuration that the model runs with. It can be used in conjunction with the seed request parameter to understand when backend changes have been made that might impact determinism.This is the indicator on whether users should expect "almost always the same result".

> My issue is the same. Transformer seed is not working for me

They mean the `transformers` package, not transformers in general!