Add flags to return scores, hidden states and / or attention weights in GenerationMixin
Author: SBrandeis
Pull request: yes
Labels: name: Core: Modeling

name: Ex: Generation

# What does this PR do?


Add flags and logic to return attention scores, hidden states, and/or logits when using a model in generation mode.

Fixes #9121 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Issue: #9121 
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten 

=> Check out forum post for more detail: https://discuss.huggingface.co/t/announcement-generationoutputs-scores-attentions-and-hidden-states-now-available-as-outputs-to-generate/3094

## UPDATE:

This PR is almost ready for merge. Here are the last things to take care of:

- [x] @patrickvonplaten, @SBrandeis, @LysandreJik, @sgugger - check if naming of return arguments is ok.
- [x] @patrickvonplaten add a correct `_check_outputs()` function for special models (Reformer, XLNet, XLM, TransfoXL, ...)
- [x] @SBrandeis We could think about a simple code snippet to advertise the new feature. Maybe use "Beam Search" for translation and show the different "sequence_scores" (probs) for 3 different translations.

## Future PR:

- [ ] Change the output in all notebooks, examples, tests to `return_dict_in_generate=True`.
- [ ] Even further in the future PR: Think about a way to deprecate the default usage of `return_dict_in_generate=False`.
- [ ] @patrickvonplaten Discuss with @sgugger if model documentation is ok. @sgugger - I think `generate()` might deserve its own "main" doc page maybe in "main classes" under "Models". I would add a small description at the top, then add all the generate functions and the new "Generation Outputs" there. Do you think this makes sense or would you rather keep "generate" as a subsection under "Models"? IMO, "generate" could be more visible in the docs.
- [ ] @patrickvonplaten, @LysandreJik, @sgugger - this PR adds a lot more generate testing to many models. Since `generate` quite an expensive method, it might be worth checking here that this PR does yield a significant slow down of the tests. Simon and I tried to make the generate test as short and light-weight as possible, but it might still be significant.


Comments:
@patrickvonplaten ready for a second review
I guess the next steps are to implement GreedySearchOutput for TF and to update the pipelines using generation?

Think it should now be relatively straight-forward to implement the outputs for the other generate methods. Seems like circle ci is in holiday...let's check back on Monday again

When will this PR be merged? Now I have a model which need the score of generated sequences.

Hi @sgugger  and @LysandreJik, thansk for the review ! I made the suggested changes to the documentation.

Thanks for your work on this @SBrandeis, great job!

Nice work, am I correct that this only works for the PyTorch models?

When would this be added to a release version? 

@SBrandeis How would one go about converting these scores into probabilities? Specifically, the probability of the word when it was generated? Seems like we would have had to softmax during generation and store that value rather than the raw score? Does that mean I am still left to role something custom to do that or am I missing something?

Hey @mshuffett, we will do a release later today, so this will be included in the release.

Regarding how to turn the scores into probabilities, please see this discussion on the forum: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175

We don't want to add this inside `generate()` because there are all kinds of probs one could calculate and we want to keep it as "barebone" as possible for better maintenance.