CatBoost: Cross-Validated Bayesian Hyperparameter Tuning
By: Peijin Chen
Tags: Data Science, Artificial Intelligence, Python, Machine Learning, Bayesian Optimization

We know that, in order to train a model that generalizes well to unseen data, that we cannot overfit the model to the data. Given a real-life data set, there is no way to tell how well a model will generalize, just because you never know what kind of real life data might be thrown at the model in the future. New data might not conform well to the distributions of existing data.

So you use cross-validation — divide the data into folds (let’s say 5, for now), and train on all but one of the folds (4 of them) and then test your model on the 5th — in other words, always leave one out as the validation data. So you’re actually training 5 models, each of which have different performance scores on their respective validation sets. Since these models might have different weights, in order to consolidate you have to find some “average” of the three models, something that would do well no matter what validation data it’s facing.

Of course, before you do that you have to have some loss function to train the model with, and then evaluate the model using some other error measure. Let’s say we have a binary classification problem, so that the target class is {0,1}, and we decide to use the Matthews Correlation Coefficient, which is a balanced measure of classifier performance. It takes into account all 4 “quadrants” of the confusion matrix — true and false positives and negatives.

CatBoost, like most decision-tree based learners, needs some hyperparameter tuning. There are plenty of hyperparameter optimization libraries in Python, but for this I am using bayesian-optimization. From their documentation is this explanation of how the whole thing works:

Bayesian optimization works by constructing a posterior distribution of functions (gaussian process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not, as seen in the picture below.

For the Matthews Correlation Coefficient, you can just use the implementation from matthews_corrcoef from sklearn.metrics.

Take a gander at the code first. In this case X1 is the data matrix of features, and Y1 is the binary class targets:

You’re probably familiar with the StratifiedKFold from sklearn, which divides the data into folds, while trying to keep the ratio of classes about the same in each fold. You can see here we have 5 folds, and, importantly, that we are appending the Matthews score into a list (so there will be 5 scores) and then finding the mean (np.mean(), where np is numpy). Therefore, the idea is that we are finding values of the hyperparameters that, in theory, guarantee a certain level of generalizability. In addition, we can use l2_leaf_reg, and model_size_reg, num_bins, num_leaves and other such hyperparameters to reduce overfitting. The pbounds (parameter bounds) dictionary you see in the code above contains ranges for the hyperparameters, and of course, YMMV, but I have found that these ranged are a good place to start. Sometimes, you have to run the algorithm a few times, first with wider ranges, and then narrowing it down. This is because too large a band and the algorithm might not converge in the number of iterations you’ve allotted.

You can see that I did int(depth) and int(num_leaves), for example, because the BHO process will come up with floats, which then have to converted into integers, because you can’t have 2.33 leaves or 44.55 estimators (trees). I want to use the best model, so the use_best_model parameter is set to True, which means that when you call .fit, you have to add the eval_set = (valx1, valy1) bit at the end.

Just to reiterate, the point of this that I wanted to create a model that is, to the best of our knowledge, generalizable — but generalizable specifically in terms of its performance on the Matthews Correlation Coefficient. You’ll find that using the ROC-AUC, F1, accuracy, or other such metrics will result in different hyperparameter values.