A version of Trainer that calculates the eval metrics (e.g. accuracy) on the training set as well
Author: quantitative-technologies
Pull request: no

# 🚀 Feature request

The `Trainer` class only calculates accuracy (assuming that is one of the eval metrics) on the evaluation dataset. However, I would like these metrics to be computed on the training set as well. 

Note that `TensorFlow` automatically calculates accuracy on the training set. 

## Motivation

Training set accuracy is *crucial* for some applications/investigations in ML. 

## Your contribution

I want to make sure I am not reinventing the wheel before subclassing `Trainer` myself?


Comments:
cc @sgugger 

This feature has been asked for a lot, so we will implement it when we have some spare time. Note that in the meantime, you can run `trainer.evaluate(train_dataset)` to get those metrics whenever you like.

True, but normally the training set gets randomly shuffled. So for precision work one has to ensure that the batches are identical for training and evaluation, at least if the metric isn't additive.  

The metric is computed on the whole predictions/labels at once in the `Trainer`. So the shuffling is irrelevant for this.

OK, I think I see: First of all the evaluation dataloader does not shuffle. Secondly, even though evaluation feeds the data to the model in batches, the results are first collated together before the metrics are computed on the whole predictions/labels at once.

Still, if instead the metrics where computed on each batch during training (this is the requested feature if I understood correctly), and a metric was not additive, then it might have a different result than when evaluated on the whole training set at once. 

I am not suggesting to implement computing a metric on each batch during training as this is not something that works for NLP where most metric (F1 score, BLEU, ROUGE etc.) are not additive. What we can add is a flag that makes evaluation on the training set on top of the validation set.

Ah, I see. But then you will not have the TensorFlow behaviour were you can monitor the training set accuracy in "real time" as the epoch progresses :)

Since that behavior is actually harmful for non-additive metrics, I would rather not have it :-) 

The most obvious way to get the training set metrics for each epoch is to save the model at each epoch, and then do an evaluation afterwards.

On the other hand, it seemed more convenient to instead make a hook that calculates the training metrics at the end of each epoch:

```
class EvaluateTrainingDatasetCallback(TrainerCallback):
    """
    A :class:`~transformers.TrainerCallback` that evaluates metrics on the training dataat the end of each epoch.
    """

    def __init__(self, trainer) -> None:
        super().__init__()
        # Abuse of callback mechanism, but not enough is passed to the hooks to avoid this
        self._trainer = trainer

    def on_epoch_end(self, args, state, control, **kwargs):
        if control.should_evaluate:
            # Needed to deal with side effects of accessing the trainer
            control_copy = deepcopy(control)
            self._trainer.evaluate(eval_dataset=self._trainer.train_dataset, metric_key_prefix="train")
            return control_copy
```

However, I found that the callback was affecting the results after the first epoch (so different train and eval metrics than without the callback). 

I thought that calling `Trainer.evaluate` did not affect the random generator. Any insight into why the callback is affecting the state?

I'm not sure, since you use `evaluate` there shouldn't be any difference.

Perhaps I wasn't clear: Without the callback, first training is done on all 3 epochs, and then *afterwards* the training dataset metrics are calculated by loading the respective checkpoints and calling `evaluate`. 

However, the callback above calls `evaluate` immediately after training each epoch, respectively. On the first epoch both methods agree, but on *subsequent* epochs I am getting different results, on both the training and validation sets, as a result of using the callback. So it seems that the callback is having "side-effects", which I don't believe it should. 

Here is an example without the callback
![Screenshot from 2021-10-03 13-49-43 fine-tuning 1](https://user-images.githubusercontent.com/29150871/135744777-659e26df-e549-4b4c-9b96-1a6a013c186a.png)

And with the callback, causing side-effects:
![image](https://user-images.githubusercontent.com/29150871/135744823-357ae2d4-98f3-4274-ac21-6ed1e98bed3c.png)



What I means is, I have no idea where this comes from since there is no randomness called during the evaluation.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.