evaluate_during_training after each epoch
Author: dianags
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Hello! I just migrated from `pytorch_pretrained_bert` to `transformers` (3.1.0) and I am having problems understanding how to make the model evaluate at the end of each epoch. For this purpose, it is recommended to use `--evaluate_during_training` as mentioned in the issue [#4617](https://github.com/huggingface/transformers/issues/4617). Looking at `Trainer` source code the condition to run an evaluation is the following:
```python
if self.args.evaluate_during_training and self.global_step % self.args.eval_steps == 0:
                        self.evaluate()
```

I am not quite following the logic of why the evaluation depends on the number of steps and not on the number of epochs. I would appreciate if someone helps me on this.

In the same issue mentioned above (#4617) someone said that you should set `--save_steps` to `number_of_samples/batch_size`. If this is true (currently testing if this works on my dataset), shouldn't it be mentioned in the documentation?
 
Thank you!

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hi there. When using `Trainer`, the evaluation loop is run every `args.eval_steps` (which is more consistent across datasets than an evaluation at the end of each epoch since with a small or a large dataset you would get evaluations that don't have the same meaning). If you want it run at the end of each epoch, you can indeed set it to `number_of_samples/batch_size`.

I'm not quite sure how this is surprising when the documentation clearly says it runs every `eval_steps` though.

> Hi there. When using `Trainer`, the evaluation loop is run every `args.eval_steps` (which is more consistent across datasets than an evaluation at the end of each epoch since with a small or a large dataset you would get evaluations that don't have the same meaning). 

Hi @sgugger . I see! I've been focused on making the `run_multiple_choice.py` work on my dataset using the default parameters. When running the example to get results on SWAG with 3 epochs, I did see the output of the evaluation loop after each epoch. I didn't understand why the same didn't happen with my dataset. Didn't see the problem from your perspective (be more consistent across datasets) but now that you mention it, it makes sense. Thank you for the explanation!

> If you want it run at the end of each epoch, you can indeed set it to `number_of_samples/batch_size`. I'm not quite sure how this is surprising when the documentation clearly says it runs every `eval_steps` though.

In the `--help` I do see that:
--evaluate_during_training
                        Run evaluation during training at each logging step.
  --eval_steps EVAL_STEPS
                        Run an evaluation every X steps.

Guess I got confused because I expected the `--evaluate_during_training` doc to refer to `eval_steps` instead of logging step. It wasn't that straightforward for me that what I need to do is to set `number_of_samples/batch_size`, that's what I meant it would be nice to find in the documentation. It is all clear now though! Thank you again :)



Note that we can add support for an evaluation strategy that is at the end of each epoch instead of every n steps. That could make things even clearer :-)

I'll look into it when I have some time.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.