model generate with different batch size but get different results
Author: Alwin4Zhang
Pull request: no

### System Info

I'm using MT5ForConditionalGeneration by transformers to generate summaries,but when I use the arguments below,I will get different results when using beam search + do_sample + top_k + top_p。Only use beam search or do_sample can not cause this phenomenon。But why？



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
self.model = MT5ForConditionalGeneration.from_pretrained(model_dir)
results = conditional_generation_summarizer.batch_summarize(
            documents=temp,
            # max_length=MAX_LENGTH,
            # bad_words=bad_words,
            max_new_tokens=MAX_LENGTH,
            num_beams=6,
            # num_beam_groups=3,
            # temperature=4.0,
            # diversity_penalty=2.0,
            # max_new_tokens=30,
            no_repeat_ngram_size=2,
            do_sample=True,
            top_k=20,
            # top_k=1,
            top_p=0.9,
            repetition_penalty=4.0,
            length_penalty=20.0,
            early_stopping=True,
            # num_return_sequences=6
        )

result = self.model.generate(
            input_ids,
            # attention_mask=attention_mask,
            decoder_start_token_id=self.tokenizer.cls_token_id,
            eos_token_id=self.tokenizer.sep_token_id,
            # max_length=max_length,
            # early_stopping=True,
            # num_beams=num_beams,
            **kwargs
        )
```
<img width="526" alt="Pasted Graphic 20" src="https://user-images.githubusercontent.com/33918902/234769066-ee29b84f-90ca-46a4-942c-fef7adcdf1ba.png">
<img width="501" alt="Pasted Graphic 21" src="https://user-images.githubusercontent.com/33918902/234769088-348c19e6-d5db-42b9-b7ee-4a4c1b3690a6.png">

### Expected behavior

I think different batch_size should not do effect on generation.

Comments:
cc @gante

> cc @gante

thks to reply.I find some things about this stranger phenomenon.When doing generation,the second time it started to be different.The pic below is the beam_new_tokens results when using `batch size = 1` and `batch_size = 2`.

<img width="1975" alt="image" src="https://user-images.githubusercontent.com/33918902/235030665-5fde95a5-f22c-436b-afcc-9e9953d95b3e.png">


debug line by line,I find that something about the torch.multinomial，after this operator，results begin to be different

Hey @Alwin4Zhang 

When you use `do_sample=True`, the results will be different every time you call `generate` :) Read [our blog post on text generation](https://huggingface.co/blog/how-to-generate) and [our docs](https://huggingface.co/docs/transformers/generation_strategies) for further information.

> Hey @Alwin4Zhang
> 
> When you use `do_sample=True`, the results will be different every time you call `generate` :) Read [our blog post on text generation](https://huggingface.co/blog/how-to-generate) and [our docs](https://huggingface.co/docs/transformers/generation_strategies) for further information.

Thks 4 reply.It's weird that it's the same when I only use `do_sample=True` + `top_k` + `top_p` with different batch_size,I'm sure that manual_seed is the same.But add `beam_search` arg,the strange things above will happen.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@gante Facing the same issue even when `do_sample = False`

[Colab Notebook replicating this issue](https://colab.research.google.com/drive/1et5wYV25Bv8miAx9T8ijJ4trpTV2QPGh?usp=sharing)

Hey @varadhbhatnagar 👋 

To be more specific, batching is not entirely innocuous on the output. The final result depends on the order of operations with FP16 (and other precisions), and batching changes the order of operations, meaning that the model outputs will see tiny fluctuations. There is nothing we can do to remedy this effect other than increasing the precision (e.g. to FP32).

These fluctuations often cause no change in the model output with `do_sample = False` -- unless the two most likely tokens have very similar probabilities. This may happen with some frequency when you're using a model with out of distributions inputs, such as using a code model with a non-code input (as seen in your colab) :)