Allow `load_best_model_at_end=True` to work when `save_steps < eval_steps` and best model is saved
Author: KrisTHL181
Pull request: no
Labels: name: Feature request

### Feature request

Allow load_best_model_at_end=True to work even when save_steps is not a round multiple of eval_steps, and optionally preserve the best model even when reaching save_total_limit. 

This change would remove the current restriction that enforces save_steps to be a multiple of eval_steps when load_best_model_at_end=True. Additionally, it proposes an optional flag to prevent deletion of the best model when the total number of saved checkpoints exceeds the limit. 

No specific paper is associated with this feature. This is a usability improvement based on common user workflows and constraints. 

### Motivation

Users with limited disk space (e.g., Colab users) often want to:
- Save more frequently (e.g., `save_steps=100`) to avoid losing progress
- Evaluate less frequently (e.g., `eval_steps=200`) to save compute
- Still be able to load the best model at the end using `load_best_model_at_end=True`

Currently, this is not possible unless `save_steps` is a multiple of `eval_steps`, which is unnecessarily restrictive. The restriction could be lifted by simply ensuring that the best model is saved at least once during training, regardless of the save/eval frequency ratio.

Additionally, users may want to keep the best model even when reaching the `save_total_limit`, which currently may cause the best model to be deleted.

This request is related to the discussion in Hugging Face Transformers GitHub issues, where users have reported frustration over this limitation.


### Your contribution

Although I currently lack the resources to submit a PR myself, I'm happy to support the discussion and help refine the proposal. I believe contributions go beyond code — asking questions, sharing feedback, and helping others in the community are also valuable ways to contribute. 
I encourage others who are interested in this feature to join the discussion or take up the implementation. I'm also happy to test or provide input if someone decides to work on it. 
In the meantime, I’ll continue to support the project by spreading the word and showing appreciation for the library’s impact. 

Comments:
This is a really good point — I’ve run into this myself on Colab where I wanted to save often (so I don’t lose progress) but evaluate less to save compute, and it’s annoying that load_best_model_at_end breaks if save_steps and eval_steps don’t align.

From what I can see, the trainer already tracks the best_model_checkpoint, so it feels doable to just make sure that checkpoint is explicitly saved regardless of the save/eval step mismatch, and also not delete it when save_total_limit kicks in.

If no one’s picked this up yet, I’d be interested to take a stab at it — just let me know if there are any gotchas or tricky edge cases you’d recommend looking out for.

Just following up — is this issue open for me to pick up? Happy to start once confirmed!

Yes, it’s absolutely open! Please feel free to start—happy to share details or help if needed.

KrisTHL181
________________________________
From: Dev ***@***.***>
Sent: Friday, July 18, 2025 10:28:05 AM
To: huggingface/transformers ***@***.***>
Cc: Kris ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/transformers] Allow `load_best_model_at_end=True` to work when `save_steps < eval_steps` and best model is saved (Issue #39476)

[https://avatars.githubusercontent.com/u/147872111?s=20&v=4]devesssi left a comment (huggingface/transformers#39476)<https://github.com/huggingface/transformers/issues/39476#issuecomment-3086455669>

Just following up — is this issue open for me to pick up? Happy to start once confirmed!

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/39476#issuecomment-3086455669>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWHDDMVNQK3XD7XL7DSB7B33JBLTLAVCNFSM6AAAAACBX46JB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAOBWGQ2TKNRWHE>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


thanks for approval!

cc @sunmarc

Happy to have a PR submitted for that ! 

Thanks for clarifying and confirming — I’ve just started digging into this and it’s not too late on my end.  

I couldn’t find the PR for this in the repo (apologies if I missed it), but if one already exists, could you please share the link? I’d like to go through the implementation and see if there’s anything I can help with or learn from.  

Looking forward to contributing where I can!

@SunMarc I've verified that this PR implements the issue I reported (#39560). Could you please review it when you have a moment? Thanks! :hugs: