Why is generation_config.json has a higher priority ?.
Author: vchagari
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@sanchit-gandhi @Narsil @gante: 
I am using WhisperForConditionalGeneration api to load an open-ai whisper model and do fine-tuning with the custom data. I see that the trainer is saving generation_config.json and config.json for every check point. 
Config.json have the configuration i set for training, but where as generation_config.json has config imported from the open-ai base model. Pls correct me if that is wrong. Which config will be used during training eval?. 
I have seen PR/Latest code where the generation_config.json has higher priority over config.json/model config that user sets explicitly, i am wondering what is that so?, wouldn't it supposed to be the config that user sets is a higher priority ? and for other parameters which user didn't set can be imported from the default config. Please enlighten me if i am assuming something wrong. 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Use WhisperForConditionalGeneration.from_pretrained and load any open-ai model to load.
2. Set some model config parameters like (forced_decoder_ids and so on) explicitly.
3. Train the model, not sure which config trainer gonna use while running train eval.

### Expected behavior

User set parameters needs to be higher priority than the base mode default parameters. 

Comments:
Hey @vchagari 👋 Following our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗  

Since this is your first issue with us, I'm going to answer your question :) 

In the past, the model config held both model parameters (like number of layers) and generate parameterization (like forcing tokens at generate time). That is suboptimal, as you may e.g. wish to have several generation configurations for the same model. As such, we have decided to separate the two files. 

We are currently in a transition phase where we accept both formats. However, I'd recommend setting all generate parameterization in the generation config file, as our long-term plan is to focus on this format. 

Indeed, using the `generation_config` is now advised for all generated related parameters. You can use this simple rule of thumb when deciding whether you need to update the `config` or `generation_config`. The Whisper fine-tuning blog post is outdated in this regard, as is the case with blog post / Colabs over time. Referring to the docs is the best way of staying up-to-date with the best library practices

Thank you @gante @sanchit-gandhi for your response and patience. I will use GenerationConfig for the configs that it supports. I have two questions, could you please address them. 

Q1:For other configs that GenerationConfig doesn't have like dropout and so on, what is the best way of setting those?. 
Noticed, setting configurations like dropouts and so on via model object that "WhisperForConditionalGeneration.from_pretrained" returns has no effect during training. 
Using "WhisperConfig" Class to set those parameters and then pass the config to WhisperForConditionalGeneration like below is the right way?.
WhisperForConditionalGeneration.from_pretrained(model, config=config) 

Q2: Passing GenerationConfig to Seq2SeqTrainingArguments is the correct way?, will this config will be used in training eval process as well?. 



What is the correct way of passing model related config (dropout and so on)



Q1: dropout is not used at generate time (well, unless you want to approximate a distribution of probabilities at inference time, which is an advanced use case), but rather at train time. Train time configuration stays in the model config :)

Q2: yes, that is the intended workflow 💪 

Thank you for clarifying @gante , appreciate it. 

@sanchit-gandhi: We can setup the model config like dropout using various ways, especially for training. 
1. Using the returned model object by "WhisperForConditionalGeneration.from_pretrained"
2.  Using "WhisperConfig" Class to set the parameters and then pass the config to WhisperForConditionalGeneration like below
WhisperForConditionalGeneration.from_pretrained(model, config=config)

I would like to know which is the best way to follow?, please let me know. 





Either way is fine :) After you've set the config attribute (like dropout) using either method 1 or 2, you'll have exactly the same object, so the two are equivalent and there is no 'better' method.

It's down to personal choice which you prefer. For me, I prefer doing one call to `.from_pretrained`, so tend to use method 1. But you might prefer the syntax for method 2, in which case there's nothing stopping you from using that.

I had to write a script to delete the generation_config.json after saving a model from HF. This way, I got back control over the generation config used by generate function. It took me days to figure out why a new fine-tuned model was performing worse... I don't think this prioritization makes sense intuitively.

Hey @samuelazran - sorry to hear you had a difficult experience here. Happy to discuss how we could improve / make it clearer what the behaviour is.

Note that you can pass any arguments used by the generation config directly to the `.generate` method at test time, e.g.
```python
model.generate(input_features, max_new_tokens=128, return_timestamps=True, langauge="fr")
```
So you have full control over the generation parameters, with the priority of:
1. Arguments to `.generate` - pass these directly when you call `.generate` as done above
2. Generation config - accessed with `model.generation_config`
3. Model config - accessed with `model.config`

Could you elaborate on what you mean by losing control over the generation config? What kind of unexpected behaviour did you encounter? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @samuelazran - were you able to control the generation properties using the control flow outlined above? Do let us know if there's anything that is still unclear, or if you have suggestions for how to improve the API! We're all ears!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.