When to Stop Training your Deep Learning Model | Towards Data Science

One of the first decisions to be made when training deep neural networks is to select the epoch in which to stop. And it is not an easy one.
If the training is stopped before the optimal time, the model will not have had time to learn the most important features from the training set and therefore provide poorly fitted solutions for both the training and the test sets.
On the other hand, if the training is stopped after the optimal epoch, high performance will be achieved in the training set, but when looking at the test results, the model would not generalize correctly, as it has over-adjusted to the training set.
Fortunately, there is an easy and widely used method to avoid these two issues. It is known as early stopping and most deep learning frameworks provide an out-of-the-box interface for using it.
Early stopping should be used almost universally.
— Ian Goodfellow et al., Deep Learning.
The idea behind this approach lies in the comparison between the training set and the test set or, for more reliable results, between the training set and the validation set.
By analyzing the error or loss graph in both the training and validation splits as the epochs pass, we will see that at first both decrease. However, the validation error will start to flatten out or increase at a certain time while the training error will continue to decrease.
The purpose of a validation set is to give us an idea of how our model behaves on data on which it has not been trained. Therefore, the epoch when the validation error starts to increase is precisely when the model is overfitting to the training set and does not generalize new data correctly. This is when we need to stop our training.
Although validation loss is a widely used metric when applying early stopping to your model, it is not always the most relevant.
Sometimes it may be better to select a metric that represents the performance of the model based on the use case. It can, for example, be accuracy, precision, recall, or a combination of metrics.
Now we know which metrics to observe and that we should stop when the validation metrics no longer improve along with the training metrics or start to perform worse.
However, it may happen that the observed metric has multiple ups and downs (due to the stochasticity of the neural networks), which means that stopping the first time the validation metric gets worse might not be the best solution (although it is the simplest one).
Some more complex triggers to stop training are as follows (note that they can also be applied in combination):
The observed metric has not improved over a given number of epochs (known as the early stopping patience).
The observed metric has not improved more than a given minimum change to be considered an improvement (known as the min_delta).
Note that when early stopping patience is used, the trigger will be executed when the monitored accuracy has not improved over the best result obtained in the last X epochs. If it improves but it does not exceed the best result, it would not count.
Keras API offers a callback to use on model.fit() to stop training when a monitored metric has stopped improving.
The metric argument receives the name of the metric you want to observe. In the case of referring to a validation metric (more realistic results as it approximates how your model would behave in production), the name must be preceded by the prefix “val_”. Note that the validation set must be passed to model.fit() in order to include validation metrics.
In addition, you can specify patience and min_delta as introduced in the previous section. Finally, the mode argument specifies whether the desired result should go from high to low (min mode) or from low to high (max mode). For instance, when monitoring accuracy the mode should be max.
callback = tf.keras.callbacks.EarlyStopping(    monitor="val_loss",    min_delta=0,    patience=0,    verbose=0,    mode="auto",    baseline=None,    restore_best_weights=False,)
This callback also offers a parameter restore_best_weights to restore the resulting model with the model weights obtained at the best-performing epoch.
To use the early stopping callback, we just need to pass the object to the callbacks argument in model.fit(), as follows:
model.fit(train_images, train_labels, epochs=10, batch_size=1, callbacks=[callback], verbose=0)
Depending on your use case, the early stop callback may not be sufficient. For example, in the case of having a model with multiple outputs and wanting to monitor a set of metrics instead of just one. Or a combined metric such as F-beta score given precision and recall.
To solve this, you can find a custom callback that I developed based on the available implementation in the Keras documentation and a StackOverflow issue. It receives a metrics names argument where instead of a single metric you can specify a set of them. To combine the metrics, the harmonic mean is calculated as commonly used for ratios.
At the end of the code, you can find an example of how to create an early stopping callback with the validation f1-score as the monitored metric (i.e. harmonic mean between validation precision and recall).
As most of the time, the answer to this question depends highly on the model and dataset you are using.
From my experience, I would recommend first running a training stage with a high number of epochs and visually observing from the results when the monitored metric starts to decrease and at what rate. After that, I would run the following experiments with a few additional epochs until it started to saturate, along with the patience as the number of epochs where it decreased by a significant percentage, also keeping in mind that if the results are very stochastic the patience should be greater.
In this article, one of the most common challenges when developing machine learning models has been addressed: when to stop the training. The widely used solution known as early stopping has been presented in detail, explaining more advanced techniques such as complex early stopping triggers to obtain optimal results and how to monitor multiple metrics by implementing a custom callback.
If you want to discover more posts like this one, you can find me at: