google / flan-t5-xxl introduces different result to inference API
Author: YJYJLee
Pull request: no

### System Info

- `transformers` version: 4.37.0.dev0
- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.10
- Python version: 3.8.18
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.0.dev20240104+cu121 (True)
- Tensorflow version (GPU?): 2.13.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

In inference API [webpage](https://huggingface.co/google/flan-t5-xxl) of flan-t5-xxl, the result is produced properly as below.
<img width="584" alt="Screenshot 2024-01-08 at 10 37 43 PM" src="https://github.com/huggingface/transformers/assets/28900943/73816a9b-5ba4-4aac-a61e-9341027b712f">

However, when I try to run from the source code, the result is somehow not generated to the end.

```
from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch
dtype = torch.float16
device = torch.device("cuda:0")

model_name = "google/flan-t5-xxl"
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name, torch_dtype=dtype).to(device)

input = tokenizer(['translate English to German: The main local beer is "Number One", it is not a complex beer, but pleasant and refreshing. The other local beer is  called "Manta".'], return_tensors="pt", padding=True)
output = model.generate(**input.to(device), num_beams=5)
tokenizer.decode(output[0], skip_special_tokens=True)
```

It produces 
```
'Das Hauptlokale Bier ist "Number One", es ist nicht ein'
```

### Expected behavior

I suspect both inference API and python code should generate the same output, but running python code does not complete the generation.

I also tried with other inputs, but they all have the same problem

Comments:
Hi,

This is because the inference API uses greedy decoding by default, whereas you are using beam search (as you are providing `num_beams=5` to the generate method). One would need to use the same generation parameters as [the ones used by the inference API](https://huggingface.co/docs/api-inference/detailed_parameters#text-generation-task) to compare apples-to-apples.

> Hi,
> 
> This is because the inference API uses greedy decoding by default, whereas you are using beam search (as you are providing `num_beams=5` to the generate method). One would need to use the same generation parameters as [the ones used by the inference API](https://huggingface.co/docs/api-inference/detailed_parameters#text-generation-task) to compare apples-to-apples.

@NielsRogge Thanks! I understand the difference in output by using different decoding strategy, but the fact that beam search is not generating output sequence to the end seems like another problem.

Is it a bug in beam search implementation?

@NielsRogge 

Also, I tried the greedy decoding by removing `num_beams=5`, but the output is still not generated to the end

```
Die Hauptbier der Insel ist "Number One", es ist nicht ein
```

You can increase `max_new_tokens`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.