Different generations during test time and validation time
Author: karthsu
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.22.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes 2x 4090 24GB

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to fine tune a model on a summarization task where I am trying to convert a Domain specific language into a summary and I get different quality of outputs for the validation and the test phase. For example, when I pass in the same input during the validation and testing phases I get two very different results:

**Validation Phase output:** 

I used the tititanic dataset and anding only those records where the passenger' a parents.

**Test phase output:** 

I used the Titanic dataset, retaining only those records where the passenger had two children.

As you can see the quality of these outputs are vastly different. And just to be clear, what I mean by the validation phase is getting the prediction text via a `compute_metrics` function during training. And by testing time, I mean outputs generated by using the `model.generate()` function after the training loop is complete using the final model or any of its checkpoints during the intermediate stages. 

### Expected behavior

I want to understand what is going on here and why there are vastly different results during the two phases. Finally, it would be helpful if someone could point out how to bring some uniformity in these generations in terms of quality

Comments:
You should use the [forums](https://discuss.huggingface.co/) for questions like this as we keep issues for clear bugs in the library and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> You should use the [forums](https://discuss.huggingface.co/) for questions like this as we keep issues for clear bugs in the library and feature requests only.

@sgugger 
This may be a possible bug in the code. I had used QLoRA to train the model and printed out the generations during the validation phase in the `compute_metrics` function. There is may be something going on with the quantization of the model during the validation phase that may not be handled properly that may lead to these suboptimal generations. When i fuse the checkpoint LoRA weights back into the model during inference phase and produce generations using the `generate` function, the outputs are not showing signs of any of these problems.