Activate Early Stopping in Boosting Algorithms to Mitigate Overfitting | by Rukshan Pramoditha | Towards Data Science

In Part 7, I’ve mentioned that overfitting can easily happen in boosting algorithms. Overfitting is one of the main drawbacks of boosting techniques.
Early stopping is a special technique that can be used to mitigate overfitting in boosting algorithms. It is used during the training phase of the algorithm.
In boosting, a new tree is added in each boosting round (iteration). The new tree attempts to correct the errors made by the tress in previous rounds.
As the name suggests, early stopping stops the training process of the algorithm somewhere before the specified boosting rounds given by n_estimators. The stopping point can be monitored by plotting the learning curve.
We calculate both training and validation scores (usually, RMSE for regression and Log Loss for classification) by providing training and validation data and then plot them against the number of boosting rounds. There is a point where the validation score does not further improve and begins to get worse while the training score still continues to improve. This is where overfitting starts and model training should be stopped.
It is so important to identify the value of that point in terms of iterations or boosting rounds. Before that point, the validation score improves at each iteration. At that point, the boosted model generalizes well on unseen data. After that point, the model tends to overfit.
By default, early stopping is not activated by the boosting algorithm itself. To activate early stopping in boosting algorithms like XGBoost, LightGBM and CatBoost, we should specify an integer value in the argument called early_stopping_rounds which is available in the fit() method or train() function of boosting models.
.fit(early_stopping_rounds=int)#OR.train(early_stopping_rounds=int)
The integer value represents the boosting round where the model generalizes well on unseen data. It can be found by monitoring the learning curve as explained earlier.
CatBoost is a powerful boosting algorithm. It has two nice features that are not available in other boosting algorithms. It can handle categorical features directly. The other one is that we can easily generate the learning curve by setting the plot argument to True in the fit() method or train() function. This is the reason why we use CatBoost here to implement early stopping.
Now, we build a CatBoost classification model on the heart_disease dataset and monitor the learning process by creating the learning curve. In addition to training data, we should also provide validation (test) data to the eval_set argument in the fit() method. Here, we use “Logloss” as the evaluation metric for our model. The lower the log loss value, the less the predicted probabilities deviate from actual values. Therefore, a lower value for log loss is better.
The point where the model generalizes well on unseen data is marked with the green dot. The X coordinate can be obtained by running:
cb.get_best_iteration()
This returns 42 meaning that the model generalizes well on unseen data at the 42nd iteration. After that, the model begins to overfit. We can provide this value for the early_stopping_rounds argument to activate early stopping.
cb.fit(X_test, y_test,       early_stopping_rounds=42)
When you run the model again, it will stop training at the 42nd boosting round (iteration) although we’ve specified 550 boosting rounds in n_estimators!
We’ve discussed how we can implement early stopping specific to the CatBoost algorithm that has built-in features to monitor the learning process. However, we can also use early stopping with other boosting algorithms such as AdaBoost, Gradient Boosting, XGBoost and LightGBM. Here are some specific ideas to apply early stopping to those algorithms.
It is easy to create the learning curve against boosting rounds for CatBoost with its built-in plot argument. For other algorithms, you need to create the learning curve manually.
XGBoost, LightGBM and CatBoost have an argument called early_stopping_rounds in the fit() method or train() function. You can specify an integer value that is less than the value in n_estimators to activate early stopping. There is not such a built-in argument in AdaBoost and Gradient Boosting. To activate early stopping for AdaBoost and Gradient Boosting, you can monitor the learning process and find the point (if any) where the model begins to overfit and set that value in the n_estimators argument. Then, you need to re-execute the algorithm with the new n_estimators value.
Not all boosting models have an opportunity for early stopping. You can easily find this by looking at the learning curve. The model with the following learning curve does not have an opportunity for early stopping even after 5000 boosting rounds!
For boosting algorithms like XGBoost and LightGBM, there are two APIs: Scikit-learn API and Learning API. The Scikit-learn API uses the common fit() method for training and the Learning API uses its own train() function for training. For example,
import lightgbmlgbm = lightgbm.LGBMRegressor()#Training: Scikit-learn APIlgbm.fit(X_train, y_train)#Training: Learning APIlightgbm.train(params=...,               train_set=...,               valid_sets=...)
The easiest and recommended one is the Scikit-learn API. There may be different names for the same parameter in these APIs and algorithms.
The outputs may vary significantly due to the stochastic nature of the train/test data splitting process. I recommend you shuffle your data before splitting and use as much data as possible for the training set. I also recommend you run the algorithm multiple times at different random_state values and see the output.
This is the end of today’s post. I always do my best to write and organize my contents in a way that you can get the most out of them. There should be many ways to further improve my contents and you may have great ideas for that. So,
Please let me know if you’ve any feedback.
Meanwhile, you can sign up for a membership through the following link to get full access to every story I write and I will receive a portion of your membership fee.
rukshanpramoditha.medium.com
Also, never miss a story from me again. Subscribe to get stories like this:
https://rukshanpramoditha.medium.com/subscribe
Thank you so much for your continuous support! See you in the next story. Happy learning to everyone!
Special credit goes to Christian Riedl on Pixabay, who provides me with a nice cover image for this post.