Add argument to set number of eval steps in Trainer
Author: brianhill11
Pull request: no
Labels: name: Good First Issue

name: trainer

name: Feature request

### Feature request

I would like to add an argument to the [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) class that allows for setting the number of eval steps (batches) for the evaluation procedure during training. 

The current behavior is to fully iterate through the entire dataset provided by the `eval_dataset` argument. However, when using a very large evaluation dataset, the evaluation process can take a long time. Especially for debugging, it would be helpful to be able to explicitly specify how many batches you would like to use as part of the evaluation procedure. 

### Motivation

Currently, I am using dataset streaming to train and evaluate models (motivated by large dataset size). My `eval_strategy` argument is set to `"steps"` since I'm using the streaming dataset, and so the Trainer runs the evaluation procedure every `eval_steps` number of steps. However, because there is no way to control how many steps are run during the evaluation procedure, the Trainer iterates through the entire eval dataset, which can be very time-consuming for a large dataset. Ideally, we could specify how many evaluation steps to run inside the evaluation procedure. 

Note that the argument I would like to add is different than the current `eval_steps` argument -- the current `eval_steps` argument specifies "Number of update steps _between_ two evaluations" but what I would like to specify is the number of steps _within_ the evaluations. 

### Your contribution

I would be happy to contribute code for this if this is deemed a relevant feature request and there isn't existing functionality that I'm unaware of. 

Comments:
cc @muellerzr @SunMarc 

Hi @brianhill11, thanks for raising this issue ! I'm open to having this feature as this could indeed become handy for users who wants to quickly debug without having to split their dataset by themselves. I don't think we have it in accelerate yet cc @muellerz. Note that for streaming dataset, you can use `take()` to split your dataset ! See documentation [here](https://huggingface.co/docs/datasets/en/stream#split-dataset). If you are up for the challenge, feel free to open a PR ! 

hey, I'd be happy to work on this.

> hey, I'd be happy to work on this.

@sajadn -- it would be great if you would be willing to take this on! Please let me know if you would like to, otherwise I will work on this when I have some spare time. 

Hi, thanks for making this feature request!

I would also be interested in having this implemented for quick debugging.

Hello 
I took a look at this issue, I have worked with the Transformers library enough to fix it. Can I work on it? 

@soghomon-b yes, please do! 

Starting a PR, still need testing and documentation update

Hi! I’d like to work on this issue as my first contribution for Hacktoberfest. Let me know if there are any specific guidelines to follow!



@SomaxSoma  Hi! 
I’ve already implemented this and opened a PR, but I ran into issues with the built-in tests. They’re not all passing on my setup, and I suspect it might be related to environment/dependency problems. If you’re interested, it would be great if you could check the PR and help get the tests passing. Thanks a lot!


Hi, I’d love to work on this issue as my first contribution to Transformers.  
Could I take this up? I’ll add the new argument to Trainer and update docs/tests accordingly.  
Thanks!


Hi 👋  
I’d like to work on adding this argument to the `Trainer` class.  
Can you confirm if there’s a preferred name for the new parameter (e.g., `max_eval_steps`)?  
I’ll send a draft PR soon — thanks!


shooing the code agents away from this issue