How to enable grad_fn when calling the generate() method of a T5 model
Author: wmmxk
Pull request: no

I was trying to attribute a prediction by a T5 model to the words of an input by gradient method. For a ```T5_model```, I call ```T5_model(**inputs)``` when training, and call ```T5_model.generate(**inputs)``` when doing inference. In training, the grad_fn is enabled for the loss but not in inference.

So how do I enable grad_fn when calling the ```generate()``` method to generate a prediction? So that I can get the gradient of the prediction with respect to each word of the input sentence.

Comments:
In fact, in the implementation file of T5, [modeling_t5.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_t5.py), I did not see ```with torch.no_grad:```. So I was wondering how ```grad_fn``` is disabled in the ```generate()``` method.

Although I think it has a low chance to work, I tried to add ```torch.set_grad_enabled(True)``` before calling the ```generate()``` method, but the ```grad_fn``` is still not enabled.

It turned out there is a decorator before the [generate() method](https://github.com/huggingface/transformers/blob/a4fc0c80b11e14aaf6a9ec7c6fa5e6dab54261e4/src/transformers/generation_utils.py#L110). I was looking for it in the body of the method.

> It turned out there is a decorator before the [generate() method](https://github.com/huggingface/transformers/blob/a4fc0c80b11e14aaf6a9ec7c6fa5e6dab54261e4/src/transformers/generation_utils.py#L110). I was looking for it in the body of the method.

Did you solve this problem? I run into the same problem and need a solution. Thanks!

> > It turned out there is a decorator before the [generate() method](https://github.com/huggingface/transformers/blob/a4fc0c80b11e14aaf6a9ec7c6fa5e6dab54261e4/src/transformers/generation_utils.py#L110). I was looking for it in the body of the method.
> 
> Did you solve this problem? I run into the same problem and need a solution. Thanks!

Did you guys solve the problem? 

I commented this line `@torch.no_grad()` in the transformers [this file](https://github.com/huggingface/transformers/blob/a4fc0c80b11e14aaf6a9ec7c6fa5e6dab54261e4/src/transformers/generation_utils.py#L110), and added `torch.set_grad_enabled(True)` inside `def generate(XXX)`. 

But I am still not sure if it works. 

![image](https://github.com/huggingface/transformers/assets/31528604/2584f6ee-9657-4079-92b4-a69e9e7690cb)
![image](https://github.com/huggingface/transformers/assets/31528604/96e85143-71c5-4477-bfb6-014acb25c591)

Please note that I only edited these two lines. 

I’ve also been working on enabling gradient propagation while using model.generate(). I came across a related discussion here: https://discuss.huggingface.co/t/how-to-output-loss-from-model-generate/16999/12. 

In that thread, it was mentioned that there is a slight difference in the logits when comparing outputs with and without the no_grad decorator, even after temporarily removing it. I was wondering if anyone has found a reasonable workaround for this issue or an alternative approach that works reliably.

Any insights or suggestions would be greatly appreciated!