T5 working on cpu but not gpu
Author: mystsec
Pull request: no

### System Info

transformers 4.16.2
ubuntu 22.04
python 3.10.6
gpu = amd radeon vii
torch 2.0.1 + rocm 5.4.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am playing around with summarization, and the following code works fine when `device = torch.device("cpu")`, but when I try on cuda I get the error below.
```
model = T5ForConditionalGeneration.from_pretrained("t5-large")
device = torch.device("cuda")
model = model.to(device)
tokenizer = T5Tokenizer.from_pretrained("t5-large")
inputs = tokenizer.encode("summarize: " + text, return_tensors="pt", max_length=512, truncation=True).to(device)
outputs = model.generate(
    inputs,
    max_length=150,
    min_length=40,
    length_penalty=2.0,
    num_beams=4,
    early_stopping=True)
print(tokenizer.decode(outputs[0]))
```

```
Traceback (most recent call last):
  File "/home/user/testing/summary.py", line 81, in <module>
    outputs = model.generate(
  File "/home/user/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/generation_utils.py", line 1234, in generate
    return self.beam_search(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/generation_utils.py", line 2026, in beam_search
    beam_outputs = beam_scorer.process(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/generation_beam_search.py", line 257, in process
    input_ids[batch_beam_idx].clone(),
IndexError: index -18014394218708992 is out of bounds for dimension 0 with size 4
```

### Expected behavior

When running on cpu, the code runs without errors and prints the output. I am trying to get the same results with gpu.

Comments:
Hi @mystsec, thanks for raising this issue! 

Version 4.16.2 is over a year old and since then there have been a lot of updates to our generation code. I'm able to run the example provide on the most recent release of transformers - v4.28.1



@amyeroberts I updated transformers to 4.28.1, and now I get the following warning + similar error to earlier:

```
/home/user/.local/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5.py:163: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on t5-large automatically truncating your input to 512 when padding/encoding.
- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
  warnings.warn(
Traceback (most recent call last):
  File "/home/user/testing/summary.py", line 81, in <module>
    outputs = model.generate(
  File "/home/user/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 1524, in generate
    return self.beam_search(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 2897, in beam_search
    sequence_outputs = beam_scorer.finalize(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/generation/beam_search.py", line 360, in finalize
    decoded: torch.LongTensor = input_ids.new(batch_size * self.num_beam_hyps_to_keep, sent_max_len)
RuntimeError: Trying to create tensor with negative dimension -36028792732385279: [1, -36028792732385279]
```

@mystsec Could you try doing a fresh install of transformers in your environment? I'm unable to replicate the error with transformers 4.28.1 on both cpu and gpu.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.