Request: Providing gen_kwargs for Seq2SeqTrainer
Author: atyshka
Pull request: no

### Feature request

As far as I am aware, there is currently no way to customize the generation behavior when generation is run as part of the training loop. `Seq2SeqTrainer.predict()` takes a `gen_kwargs` argument to pass along to `generate()`, but there is no way to pass `gen_kwargs` to `predict()` when predict is called automatically as part of the training loop.

### Motivation

Customizing generation may be necessary, doing this in the training loop as opposed to after training would be ideal.

### Your contribution

I'm happy to contribute, the main question is how this could be elegantly integrated into the current Trainer API.

Comments:
May be of interest to @gante @sgugger 

You can customize any generation kwarg by passing them to the model configuration (soon there will be a generation configuration that will make this clearer).

Ah I wasn't aware of that. It doesn't have all the generation parameters though, in my case I wanted to use prefix_allowed_tokens_fn, which model config doesn't seem to have. It's easily solved with a trainer subclass for me though, and it's probably a specific enough use case that the API doesn't need to be modified. Since the most common params can be used that way, I'll consider this resolved.

@atyshka I have the same issue using `prefix_allowed_tokens_fn` for Seq2SeqTrainer. Can you share how you solved it through trainer subclass?

Hi what is the status of this. I think the `gen_kwargs` is not respected when you call the predict() function on the trainer. 
It is not respected even when appropriate keys are set in the generation config 

This is critical for so many models. 

Hi @abhinav-kashyap-asus 👋 Would you be able to provide an example? 🤗 

The `predict` function in the `Seq2Seq` trainer calls the `predict` function from the `Trainer` class  `return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)` 
It does not pass on the `gen_kwargs` that we can send to the Seq2Seq predict function. Why is this the case??



The `prediction_step` that is defined within the `Seq2Seq` class is called from the parent correct??
If you check the function call in the `Trainer` class, the `gen_kwargs` is not passed on to this function as well. 


This makes me wonder whether the `generate` function of the model is getting the appropriate parameters. 
When I try to use the trainer for running whisper models,  I see that whisper generates tokens in language other than the one that I specify. For example consider the "hindi" example in the official repo, I see that my examples are generated in Urdu! 

The generation kwargs are not passed around because we have a `generation_config` trainer argument, whose contents are stored in the corresponding model attribute ([here](https://github.com/huggingface/transformers/blob/c662c78c71e693234725d70a6291398b33042444/src/transformers/trainer_seq2seq.py#L74)). As such, in the prediction step, `generate` is parameterized by `generation_config` :)

Thanks for the response @gante 
How do you set the generation_config for speculative decoding?

`generation_config.gen_kwargs = {"assistant_model": assistant_model}` will throw an error while saving the model because the `assistant_model` cannot be serialized. 

@abhinav-kashyap-asus speculative decoding is not yet compatible with training, unless you manipulate the code yourself :)