Improved documentation/blog post around `generate()`
Author: puzzler10
Pull request: no

Hi there. 

I have seen a bit of confusion in the community around the beam search procedure and the returned classes, particularly the `scores` attribute. This seemed to be the motivation behind the recent [PR](https://github.com/huggingface/transformers/pull/14654) that aims to improve generation to make things easier for the user.

I am finding however that I still have a lot of questions that aren’t answered by the current documentation. I was thinking an updated blog post about `generate()` (as mused about [here](https://github.com/huggingface/transformers/pull/14654#issuecomment-1020503207)) would certainly be very useful to myself and to other users.  

Here are my main questions about the process that I’m hoping could be covered:

* what exactly are `scores`, and how do they differ from log probabilities? What was changed in the pull request about the way they are calculated? 
* how are `scores`different from the output of the new function `model.compute_transition_beam_scores()`? 
* how do I interpret the new returned  `beam_indicies`?
* what exactly are `sequence_scores`? What’s the motivation behind their calculation (e.g why is `length_penalty` involved?)

In addition, I have some questions about behaviours I have noticed: 

* when I sum up the transition scores from  `model.compute_transition_beam_scores()`, I find that the ordering of the returned sequences isn’t the same as that of `sequence_scores` , which means that the most probable sequences aren’t those being returned first - why is this? 
* like some other people I am using gradients of the generate function by removing the `no_grad` decorator from the generate function. I notice when I do this I can get gradients when calling `model.compute_transition_beam_scores()` but not with the `sequence_scores` output - why is this? 

Finally I would like to thank @patrickvonplaten for all the hard work he has put into the beam search process, and to give my apologies if this post comes off a little entitled.


Comments:
Hey @puzzler10, 

That's great feedback - thanks a lot :-) It helps me a lot to find good content for an updated version of the `generate()` blog post and some documentation. I hope to be able to finish such a blog post in ~1 month! 

Maybe we can use this issue as some kind of feedback what people would like to understand better with regards to `generate()` - I'll post this on Twitter if you don't mind. 

To answer some of your questions:

> When I sum up the transition scores from model.compute_transition_beam_scores(), I find that the ordering of the returned sequences isn’t the same as that of sequence_scores , which means that the most probable sequences aren’t those being returned first - why is this?

This sounds like a bug to me then or a bad design. Let me look into it.

> Like some other people I am using gradients of the generate function by removing the no_grad decorator from the generate function. I notice when I do this I can get gradients when calling model.compute_transition_beam_scores() but not with the sequence_scores output - why is this?

I've never used `generate()` to compute gradients - do lots of people use this now?


Thanks for looking into this! 

> Maybe we can use this issue as some kind of feedback what people would like to understand better with regards to generate() - I'll post this on Twitter if you don't mind.

No worries from me.

> I've never used generate() to compute gradients - do lots of people use this now?

It's something I've seen a bit through github issues and forum posts (e.g. [1](https://github.com/huggingface/transformers/issues/6951), [2](https://github.com/huggingface/transformers/issues/3720)). My particular use case is that I use the gradients to fine-tune a transformer to maximise some particular reward function via a policy gradient algorithm.  To get the gradients on I use this work-around to create a modified version of `generate`:

```
from undecorated import undecorated
from types import MethodType

generate_with_grad = undecorated(model.generate)
model.generate_with_grad = MethodType(generate_with_grad, model)
```

which seems to work well for me. 


@patrickvonplaten are there any news on this or the blogpost? I'm trying to use the scores and also find myself with the same questions about the differences between the scores `model.compute_transition_beam_scores()`, `sequence_scores` and `scores`

Got caught up in another project :-/ But I'm thinking about how to structure the new blogpost

Hey! I wouldn't mind helping out planning out or kick-starting the blogpost but I do have some questions regarding the internals and how the compute_transition_probabilities relate to beam sequence scores!

Cool! It'd already be very helpful if you could drop your questions here - I'll try to answer them in the blog post then :-)

Linking some more feedback here https://github.com/huggingface/transformers/issues/10012#issuecomment-1060219687

> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
> 
> Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

unstale

> undecorated

It seems not to work now.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sadly I don't think I'll find the time in the near future to write another blog post here. I'll add a tutorial to the docs, but probably won't manage to rewrite the whole blog post. Linking the best resource there is so far: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Thanks for looking into this!
> 
> > Maybe we can use this issue as some kind of feedback what people would like to understand better with regards to generate() - I'll post this on Twitter if you don't mind.
> 
> No worries from me.
> 
> > I've never used generate() to compute gradients - do lots of people use this now?
> 
> It's something I've seen a bit through github issues and forum posts (e.g. [1](https://github.com/huggingface/transformers/issues/6951), [2](https://github.com/huggingface/transformers/issues/3720)). My particular use case is that I use the gradients to fine-tune a transformer to maximise some particular reward function via a policy gradient algorithm. To get the gradients on I use this work-around to create a modified version of `generate`:
> 
> ```
> from undecorated import undecorated
> from types import MethodType
> 
> generate_with_grad = undecorated(model.generate)
> model.generate_with_grad = MethodType(generate_with_grad, model)
> ```
> 
> which seems to work well for me.

Hi Tom @puzzler10, thanks for the discussion that helped me a lot! I have a following question that, when we are trying to optimize a reward function via policy gradient, one could run into issues that gradients can not be backproped due to in-place operations like [this thread](https://github.com/huggingface/transformers/issues/13954). Do you have some elegant solutions/tips to implement policy gradient with huggingface code base?

> Thanks for looking into this!
> 
> > Maybe we can use this issue as some kind of feedback what people would like to understand better with regards to generate() - I'll post this on Twitter if you don't mind.
> 
> No worries from me.
> 
> > I've never used generate() to compute gradients - do lots of people use this now?
> 
> It's something I've seen a bit through github issues and forum posts (e.g. [1](https://github.com/huggingface/transformers/issues/6951), [2](https://github.com/huggingface/transformers/issues/3720)). My particular use case is that I use the gradients to fine-tune a transformer to maximise some particular reward function via a policy gradient algorithm. To get the gradients on I use this work-around to create a modified version of `generate`:
> 
> ```
> from undecorated import undecorated
> from types import MethodType
> 
> generate_with_grad = undecorated(model.generate)
> model.generate_with_grad = MethodType(generate_with_grad, model)
> ```
> 
> which seems to work well for me.

Hey! You said to fine-tune the transformer with policy gradient, but as you know language models are auto-regressive, how did you generate N words and then passed them to the RL model, such that the gradient flow doesn't break? I mean it is end-to-end only if you pass only one token, but it that case the model would hardly converge, how did you do that?  Thanks in advance.

> > undecorated
> 
> It seems not to work now.

True. However, there is a workaround based on the implementation of `undecorated` module.

```
from types import MethodType

generate_with_grad = model.generate.__closure__[1].cell_contents
model.generate_with_grad = MethodType(generate_with_grad, model)
```

Below is the specification of my environment:
- python 3.11.3 
- transformers 4.29.2   
- pytorch 2.0.1