Refactor output handling in generate for cleaner decoding methods
Author: manueldeprada
Pull request: yes

Each decoding method has a common block of output handling boilerplate that worsens readability:

```
output_attentions = generation_config.output_attentions
output_hidden_states = generation_config.output_hidden_states
output_scores = generation_config.output_scores
output_logits = generation_config.output_logits
return_dict_in_generate = generation_config.return_dict_in_generate

# init attention / hidden states / scores tuples
scores = () if (return_dict_in_generate and output_scores) else None
raw_logits = () if (return_dict_in_generate and output_logits) else None
decoder_attentions = () if (return_dict_in_generate and output_attentions) else None
cross_attentions = () if (return_dict_in_generate and output_attentions) else None
decoder_hidden_states = () if (return_dict_in_generate and output_hidden_states) else None

# if model is an encoder-decoder, retrieve encoder attention weights and hidden states
if return_dict_in_generate and self.config.is_encoder_decoder:
    encoder_attentions = model_kwargs["encoder_outputs"].get("attentions") if output_attentions else None
    encoder_hidden_states = (
        model_kwargs["encoder_outputs"].get("hidden_states") if output_hidden_states else None
    )

...

while not finished:
    # Store scores, attentions and hidden_states when required
    if return_dict_in_generate:
        if output_scores:
            scores += (next_token_scores,)
        if output_logits:
            raw_logits += (next_token_logits,)
        if output_attentions:
            decoder_attentions += (
                (outputs.decoder_attentions,) if self.config.is_encoder_decoder else (outputs.attentions,)
            )
            if self.config.is_encoder_decoder:
                cross_attentions += (outputs.cross_attentions,)

        if output_hidden_states:
            decoder_hidden_states += (
                (outputs.decoder_hidden_states,)
                if self.config.is_encoder_decoder
                else (outputs.hidden_states,)
            )

...
if return_dict_in_generate:
    if self.config.is_encoder_decoder:
        return XXXEncoderDecoderOutput(
            sequences=input_ids,
            scores=scores,
            logits=raw_logits,
            encoder_attentions=encoder_attentions,
            encoder_hidden_states=encoder_hidden_states,
            decoder_attentions=decoder_attentions,
            cross_attentions=cross_attentions,
            decoder_hidden_states=decoder_hidden_states,
            past_key_values=model_kwargs.get("past_key_values"),
        )
    else:
        return XXXDecoderOnlyOutput(
            sequences=input_ids,
            scores=scores,
            logits=raw_logits,
            attentions=decoder_attentions,
            hidden_states=decoder_hidden_states,
            past_key_values=model_kwargs.get("past_key_values"),
        )
else:
    return input_ids
```


This PR takes that boilerplate to reusable generate helpers



TODO: add generalization so that users can say output_x and x from forward gets forwarded.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante this PR is more of a RFC to see what you think than a full PR. If you agree this simplifies generate, I will put more work to make it clean for assisted gen and make tests happy!

related: https://github.com/huggingface/transformers/issues/39834

@gante This is ready for review, left some comments in the code. The `_accumulate` method in this PR does not hardcode arg names (attentions, hidden states, etc) but rather iterate on a general dict. This is less readable but prepares the ground for custom `output_xxx` like `image_hidden_states` in a future PR (as suggested in #39834).

I would suggest merging this PR as-is, and then I can make a second PR that enables custom `output_xxx`. For that second PR, we need to agree what is the best interface for users to specify which extra args from model output they want. A big caveat also to be discussed is that assisted generation needs to use `_split_model_outputs` which might not work for special model outputs, so we might just not support extra outputs in assisted generation. WDYT?

hey @manueldeprada , how is this going? Do you plan to finish it or I can take over, since returning arbitrary outputs from model is important for new multimodal models