transformers 4.44.2 doesn't work with torch.compile and torch.export on T5 generate()
Author: yiming0416
Pull request: no
Labels: name: WIP

name: bug

name: Generation

name: Compilation

### System Info

- `transformers` version: 4.44.2
- Platform: Linux-5.19.0-0_fbk12_hardened_11583_g0bef9520ca2b-x86_64-with-glibc2.34
- Python version: 3.10.14
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.0a0+git33ba952 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA PG509-210

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The following code breaks:

```python
import torch
import transformers
from transformers import GenerationConfig
from transformers import AutoConfig

def generate_inputs_for_model(model_cls, model):
    eval_context = torch.randint(0, model.config.vocab_size, (4, 2048)).to("cuda")
    return {"input_ids": eval_context}

config = AutoConfig.from_pretrained("t5-small")
model_cls = getattr(transformers, "AutoModelForSeq2SeqLM")
model = model_cls.from_config(config).to("cuda")
example_inputs = generate_inputs_for_model(model_cls, model)
example_inputs = (example_inputs["input_ids"],)

generation_config = GenerationConfig(
    max_new_tokens=256,
    pad_token_id=0,
    eos_token_id=None,
    do_sample=False,
    num_beams=1,
    use_cache=True,
)

class GenerationWrapper(torch.nn.Module):
    def __init__(self, model, generation_config):
        super().__init__()
        self.model = model
        self.generation_config = generation_config

    def forward(self, inputs):
        return self.model.generate(inputs, self.generation_config)

model = GenerationWrapper(model, generation_config)
# torch.compile repro
model_opt = torch.compile(model)
output = model_opt(*example_inputs)
# torch.export repro
torch.export.export(model, args=example_inputs, strict=False)
```

With the following error:
```
ValueError: `decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.
```
If I manually add `decoder_start_token_id=0` to the GenerationConfig. Then both compile and export work, although very slow.


### Expected behavior

Expected generate to work like before without manually specifying `decoder_start_token_id` or `bos_token_id` in the `GenerationConfig`.

Comments:
Thanks for the issue! cc @ArthurZucker 

#33221 seems like it is required quite a lot. 
cc @gante let's fix the `generate` issues! 

cc @zucchini-nlp, who will be converting BART and T5 to be `compile`-compatible (using `EncoderDecoderCache`, like we did on `Whisper`)

The T5 model is compile compatible now from https://github.com/huggingface/transformers/pull/34089, so closing as resolved