generation_config ignored in evaluate of Seq2SeqTrainer
Author: zouharvi
Pull request: no

### System Info

- `transformers` version: 4.33.0.dev0
- Platform: Linux-5.10.184-174.730.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.9
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.13.0 (True)


### Who can help?

@muellerz @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Pass modified generation config to `Seq2SeqTrainingArguments`. In this example we want to make a NLLB model predict into a certain language.

```python3
generation_config = copy.deepcopy(model.generation_config)
generation_config.forced_bos_token_id = tokenizer.lang_code_to_id["deu_Latn"]

training_args = Seq2SeqTrainingArguments(
    ...,
    generation_config=generation_config
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
)

trainer.train()
```

### Expected behavior

The generation config gets ignored somewhere along the way and the model generation does not have access to them. They are definitely not present in kwargs of the [`evaluate(...)` call](https://github.com/huggingface/transformers/blob/1fa2d89a9bb98a15e9720190e07d272a42f03d28/src/transformers/trainer_seq2seq.py#L112). When I pass the argument to `evaluate` directly, it works well (and generates in the desired language).

The parent class is calling evaluate from [here](https://github.com/huggingface/transformers/blob/1fa2d89a9bb98a15e9720190e07d272a42f03d28/src/transformers/trainer.py#L2254) but without the generation arguments. Those arguments are stored _on the model_ [here](https://github.com/huggingface/transformers/blob/1fa2d89a9bb98a15e9720190e07d272a42f03d28/src/transformers/trainer_seq2seq.py#L73) but NLLB overwrites them in `generate`.

Maybe I'm just not using the TrainingArguments/generation config correctly but there seems to be some clash between the model's configuration and `gen_kwargs`. Related PR is [this one](https://github.com/huggingface/transformers/pull/22323).

Comments:
cc @gante who assisted the PR for this support

A temporary solution in this case is to mark the generation config as not from the model because that's what's making it revert [here in generate](https://github.com/huggingface/transformers/blob/a4dd53d88e4852f023332d284ff07a01afcd5681/src/transformers/generation/utils.py#L1418):

```python3
generation_config = copy.deepcopy(model.generation_config)
generation_config.forced_bos_token_id = tokenizer.lang_code_to_id[lang2nllb]
generation_config._from_model_config = False
```

That seems unidiomatic though (accessing "private" field). What is the intended way of using all the model's defaults except for one attribute?

Hey @zouharvi 👋 Thank you for opening this issue.

Yeah, situations like yours could happen: in a nutshell, when `_from_model_config` is `True` (which can happen under a myriad of situations), the legacy mode is triggered and `model.config` takes precedence.

I believe I have found a better way to avoid false triggers of this legacy mode, if everything goes well I'll open a PR that fixes this! 

Would you be able to share a complete reproducer, so I can check on my end? Otherwise, I might ask you to be my guinea pig :D 

@zouharvi I believe the PR above fixes it (and all related issues), but I'd like to ask you to confirm it on your particular script after it gets merged :)

@gante Thanks for the PR! It seems to cover my use-case perfectly and I'll definitely test it out.

@zouharvi it was merged. If you have the chance to test it out, let us know whether it works 🤗 I'm very confident it works, but nothing like double-checking.

I was curious how you detect the change of state -- hashing is cool. Seems to work well, thanks for the fix. 🙂