Selecting Optimal Parameters for XGBoost Model Training | by Andrej Baranovskij | Towards Data Science

There is always a bit of luck involved when selecting parameters for Machine Learning model training. Lately, I work with gradient boosted trees and XGBoost in particular. We are using XGBoost in the enterprise to automate repetitive human tasks. While training ML models with XGBoost, I created a pattern to choose parameters, which helps me to build new models quicker. I will share it in this post, hopefully you will find it useful too.
I’m using Pima Indians Diabetes Database for the training, CSV data can be downloaded from here.
This is the Python code which runs XGBoost training step and builds a model. Training is executed by passing pairs of train/test data, this helps to evaluate training quality ad-hoc during model construction:
%%time

model = xgb.XGBClassifier(max_depth=12,
                        subsample=0.33,
                        objective='binary:logistic',
                        n_estimators=300,
                        learning_rate = 0.01)
eval_set = [(train_X, train_Y), (test_X, test_Y)]
model.fit(train_X, train_Y.values.ravel(), early_stopping_rounds=15, eval_metric=["error", "logloss"], eval_set=eval_set, verbose=True)

[0]	validation_0-error:0.231518	validation_0-logloss:0.688982	validation_1-error:0.30315	validation_1-logloss:0.689593
Multiple eval metrics have been passed: 'validation_1-logloss' will be used for early stopping.

Will train until validation_1-logloss hasn't improved in 15 rounds.
[1]	validation_0-error:0.206226	validation_0-logloss:0.685218	validation_1-error:0.216535	validation_1-logloss:0.686122
[2]	validation_0-error:0.196498	validation_0-logloss:0.681505	validation_1-error:0.220472	validation_1-logloss:0.682881
[3]	validation_0-error:0.196498	validation_0-logloss:0.67797	validation_1-error:0.220472	validation_1-logloss:0.679601
[4]	validation_0-error:0.180934	validation_0-logloss:0.674278	validation_1-error:0.208661	validation_1-logloss:0.676067
[5]	validation_0-error:0.177043	validation_0-logloss:0.670627	validation_1-error:0.212598	validation_1-logloss:0.673761
[6]	validation_0-error:0.175097	validation_0-logloss:0.667069	validation_1-error:0.216535	validation_1-logloss:0.671441
[7]	validation_0-error:0.18677	validation_0-logloss:0.663582	validation_1-error:0.212598	validation_1-logloss:0.668586
[8]	validation_0-error:0.180934	validation_0-logloss:0.660353	validation_1-error:0.23622	validation_1-logloss:0.665983
[9]	validation_0-error:0.161479	validation_0-logloss:0.656739	validation_1-error:0.228346	validation_1-logloss:0.662987
[10]	validation_0-error:0.167315	validation_0-logloss:0.653582	validation_1-error:0.228346	validation_1-logloss:0.660091
[259]	validation_0-error:0.122568	validation_0-logloss:0.34313	validation_1-error:0.220472	validation_1-logloss:0.475866
[260]	validation_0-error:0.124514	validation_0-logloss:0.34261	validation_1-error:0.220472	validation_1-logloss:0.476068
[261]	validation_0-error:0.120623	validation_0-logloss:0.342156	validation_1-error:0.216535	validation_1-logloss:0.476165
[262]	validation_0-error:0.120623	validation_0-logloss:0.341714	validation_1-error:0.216535	validation_1-logloss:0.476143
[263]	validation_0-error:0.124514	validation_0-logloss:0.341209	validation_1-error:0.216535	validation_1-logloss:0.476063
[264]	validation_0-error:0.120623	validation_0-logloss:0.340779	validation_1-error:0.220472	validation_1-logloss:0.47595
[265]	validation_0-error:0.120623	validation_0-logloss:0.340297	validation_1-error:0.212598	validation_1-logloss:0.475858
[266]	validation_0-error:0.120623	validation_0-logloss:0.339908	validation_1-error:0.212598	validation_1-logloss:0.476057
[267]	validation_0-error:0.120623	validation_0-logloss:0.339312	validation_1-error:0.220472	validation_1-logloss:0.476228
[268]	validation_0-error:0.120623	validation_0-logloss:0.338874	validation_1-error:0.216535	validation_1-logloss:0.476266
[269]	validation_0-error:0.120623	validation_0-logloss:0.338543	validation_1-error:0.216535	validation_1-logloss:0.476202
[270]	validation_0-error:0.120623	validation_0-logloss:0.33821	validation_1-error:0.216535	validation_1-logloss:0.47607
[271]	validation_0-error:0.120623	validation_0-logloss:0.337716	validation_1-error:0.212598	validation_1-logloss:0.476229
[272]	validation_0-error:0.118677	validation_0-logloss:0.337295	validation_1-error:0.212598	validation_1-logloss:0.47612
[273]	validation_0-error:0.118677	validation_0-logloss:0.336927	validation_1-error:0.212598	validation_1-logloss:0.476152
[274]	validation_0-error:0.118677	validation_0-logloss:0.33651	validation_1-error:0.212598	validation_1-logloss:0.476127
[275]	validation_0-error:0.120623	validation_0-logloss:0.336017	validation_1-error:0.216535	validation_1-logloss:0.476117
[276]	validation_0-error:0.120623	validation_0-logloss:0.335497	validation_1-error:0.212598	validation_1-logloss:0.476063
[277]	validation_0-error:0.116732	validation_0-logloss:0.335159	validation_1-error:0.216535	validation_1-logloss:0.476113
[278]	validation_0-error:0.114786	validation_0-logloss:0.334812	validation_1-error:0.216535	validation_1-logloss:0.476143
[279]	validation_0-error:0.114786	validation_0-logloss:0.334481	validation_1-error:0.216535	validation_1-logloss:0.476163
[280]	validation_0-error:0.116732	validation_0-logloss:0.333843	validation_1-error:0.216535	validation_1-logloss:0.476359
Stopping. Best iteration:
[265]	validation_0-error:0.120623	validation_0-logloss:0.340297	validation_1-error:0.212598	validation_1-logloss:0.475858

CPU times: user 690 ms, sys: 310 ms, total: 1 s
Wall time: 799 ms

 

Key parameters in XGBoost(the ones which would affect model quality greatly), assuming you already selected max_depth (more complex classification task, deeper the tree), subsample (equal to evaluation data percentage), objective (classification algorithm):
n_estimators — the number of runs XGBoost will try to learn
learning_rate — learning speed
early_stopping_rounds — overfitting prevention, stop early if no improvement in learning
When model.fit is executed with verbose=True, you will see each training run evaluation quality printed out. At the end of the log, you should see which iteration was selected as the best one. It might be the number of training rounds is not enough to detect the best iteration, then XGBoost will select the last iteration to build the model.
With matpotlib library we can plot training results for each run (from XGBoost output). This helps to understand if iteration which was chosen to build the model was the best one possible. Here we are using sklearn library to evaluate model accuracy and then plotting training results with matpotlib:
# make predictions for test data
y_pred = model.predict(test_X)
predictions = [round(value) for value in y_pred]

# evaluate predictions
accuracy = accuracy_score(test_Y, predictions)
print("Accuracy: %.2f%%" % (accuracy * 100.0))

Accuracy: 78.74%

# retrieve performance metrics
results = model.evals_result()
epochs = len(results['validation_0']['error'])
x_axis = range(0, epochs)
# plot log loss
fig, ax = pyplot.subplots()
ax.plot(x_axis, results['validation_0']['logloss'], label='Train')
ax.plot(x_axis, results['validation_1']['logloss'], label='Test')
ax.legend()
pyplot.ylabel('Log Loss')
pyplot.title('XGBoost Log Loss')
pyplot.show()
# plot classification error
fig, ax = pyplot.subplots()
ax.plot(x_axis, results['validation_0']['error'], label='Train')
ax.plot(x_axis, results['validation_1']['error'], label='Test')
ax.legend()
pyplot.ylabel('Classification Error')
pyplot.title('XGBoost Classification Error')
pyplot.show()

 

Let’s describe my approach to select parameters (n_estimators, learning_rate, early_stopping_rounds) for XGBoost training.
Step 1. Start with what you feel works best based on your experience or what makes sense
n_estimators = 300
learning_rate = 0.01
early_stopping_rounds = 10
Results:
Stop iteration = 237
Accuracy = 78.35%
Results plot:
With the first attempt, we already get good results for Pima Indians Diabetes dataset. Training was stopped at iteration 237. Classification error plot shows a lower error rate around iteration 237. This means learning rate 0.01 is suitable for this dataset and early stopping of 10 iterations (if the result doesn’t improve in the next 10 iterations) works.
Step 2. Experiment with learning rate, try to set a smaller learning rate parameter and increase number of learning iterations
n_estimators = 500
learning_rate = 0.001
early_stopping_rounds = 10
Results:
Stop iteration = didn’t stop, spent all 500 iterations
Accuracy = 77.56%
Results plot:
Smaller learning rate wasn’t working for this dataset. Classification error almost doesn’t change and XGBoost log loss doesn’t stabilize even with 500 iterations.
Step 3. Try to increase the learning rate.
n_estimators = 300
learning_rate = 0.1
early_stopping_rounds = 10
Results:
Stop iteration = 27
Accuracy = 76.77%
Results plot:
With increased learning rate, the algorithm learns quicker, it stops already at iteration Nr. 27. XGBoost log loss error is stabilizing, but the overall classification accuracy is not ideal.
Step 4. Select optimal learning rate from the first step and increase early stopping (to give the algorithm more chances to find a better result).
n_estimators = 300
learning_rate = 0.01
early_stopping_rounds = 15
Results:
Stop iteration = 265
Accuracy = 78.74%
Results plot:
A slightly better result is produced with 78.74% accuracy — this is visible in the classification error plot.
Resources:
Jupyter notebook on GitHub
Blog post — Jupyter Notebook — Forget CSV, fetch data from DB with Python
Blog post — Avoid Overfitting By Early Stopping With XGBoost In Python