model.generate temperature parameter is completely ineffective
Author: AndreaSottana
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@gante @sg

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hello,
I am trying to generate text using different models and different temperature parameters. I have noticed, however, that while changing hyperparameters such as `num_beams` affects the output text, changing the `temperature` parameter doesn't seem to do anything, and setting a temperature to 0.0 or 1.0 (very different) always leads to the same output. This has been observed across multiple different language models. I suspect this might be a bug such that the set temperature is not shown to the model. In order to reproduce, run the example below (feel free to try with different text samples to convince yourself it's not a one-off occurrence)

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, AutoConfig
from accelerate import init_empty_weights, infer_auto_device_map
import torch

tokenizer = AutoTokenizer.from_pretrained('bigscience/T0pp')  # feel free to try a different model
config = AutoConfig.from_pretrained('bigscience/T0pp')

max_memory={i: "24GiB" for i in range(torch.cuda.device_count())}
with init_empty_weights():
    model = AutoModelForSeq2SeqLM.from_config(config)
    device_map = infer_auto_device_map(model, no_split_module_classes=['T5Block'])
    print(device_map)
    device_map['lm_head'] = 0

model = AutoModelForSeq2SeqLM.from_pretrained('bigscience/T0pp', device_map=device_map, load_in_8bit=True, max_memory=max_memory)
text = "Complete the following story: Once upon a time there was a "
input_ids = tokenizer.encode(text, return_tensors='pt').to(0)
for temp in [0.0, 1.0]:
    beam_outputs = model.generate(
        input_ids, 
        max_length=512,
        num_beams=5,
        no_repeat_ngram_size=4,
        temperature=temp,
        num_return_sequences=1, 
        early_stopping=True,
    )
    print(tokenizer.decode(beam_outputs[0], skip_special_tokens=True))
```

### Expected behavior

I would expect the two printed outputs to be different. I understand that occasionally they might be the same, but I've tried with over 1,000 different inputs, and the generated outputs with `temperature=0` and `temperature=1` are ALWAYS the same which means there is something wrong

Comments:
Hey @AndreaSottana 👋 

I would recommend reading our [blog post on how to generate](https://huggingface.co/blog/how-to-generate). 

TL;DR -- there are several generation modes, and not all `.generate()` parameters are active for a given generation mode. In particular, the popular `temperature`, `top_p`, and `top_k` are only active when `do_sample=True` is also passed. Some tasks benefit from `do_sample=True`, while others do not. Popular tools like `ChatGPT` operate with sampling.

We are aware that our `.generate()` has too many options and too little checks/examples, we are working on it 🤗 

Hi @gante 
Thank you very much for clarifying, I wasn't aware that some parameters were not effective when `do_sample=False`. Closing the issue for now

> Hey [@AndreaSottana](https://github.com/AndreaSottana) 👋
> 
> I would recommend reading our [blog post on how to generate](https://huggingface.co/blog/how-to-generate).
> 
> TL;DR -- there are several generation modes, and not all `.generate()` parameters are active for a given generation mode. In particular, the popular `temperature`, `top_p`, and `top_k` are only active when `do_sample=True` is also passed. Some tasks benefit from `do_sample=True`, while others do not. Popular tools like `ChatGPT` operate with sampling.
> 
> We are aware that our `.generate()` has too many options and too little checks/examples, we are working on it 🤗

This was a great answer. I was just wondering how to do temperature with this function, and figured when I set it to about `0`, that the outputs would be deterministic. They were almost deterministic, but I had forgotten about `top_k`, which you mention directly after `temperature`. Setting `top_k` gives me the same outputs as setting `temperature=0.0001`, which is what I needed to confirm it was working (i.e., the probability is all centered over the most likely token). Or maybe that's better proven via `top-p`...