5 Cute Features of CatBoost | Towards Data Science

We’ve already discussed 5 boosting algorithms: AdaBoost, Gradient Boosting, XGBoost, LightGBM and CatBoost.
Out of them, CatBoost is so special because of its special features that other boosting algorithms don’t have.
Generally, there are two main drawbacks of boosting algorithms:
Overfitting can easily happen in boosting algorithms because they are tree-based algorithms.
It is not easy to parallelize the training process of boosting algorithms because new trees are created one after another sequentially.
Keeping these drawbacks in mind, CatBoost was designed to build more generalized models at lightning-fast speed for large datasets. It has built-in parameters to reduce overfitting and do parallel learning, plus some other unique features! Let’s explore them.
Both pip and conda installations of CatBoost provide GPU support for CatBoost.
pip install catboost#ORconda install -c conda-forge catboost
The following parameter settings in the CatBoostClassifier() and CatBoostRegressor() classes can be used to get GPU support during training.
task_type: Possible values are “CPU” and “GPU”. The value type is string. The default value is “CPU”. To get GPU support during the training, simply use “GPU” as the value for task_type.
devices: This parameter defines the IDs for GPU devices in your computer. The value type is string. You can use numbers such as ‘0’, or ‘1’ to define a single GPU device. To define multiple GPU devices at once, you can use a range of numbers as ‘0:1:2’.
To find how many GPUs you have and the correct ID for GPU, in Windows, go to Task Manager and then click on the Performance tab.
To allow CatBoost to use my NVIDIA GPU, I should set task_type=“GPU” and devices=‘1’. Here is an example with the CatBoostClassifier() class.
from catboost import CatBoostClassifiermodel = CatBoostClassifier(iterations=100,                           task_type="GPU",                           devices='1')
Implementing CatBoost with GPU is efficient with larger datasets that have millions of objects and hundreds of features! According to the official documentation, you can get up to 40x speed-ups with powerful GPUs for such large datasets. If your dataset is not large enough, you will not see much difference in training time for CPU and GPU.
Usually, we have to encode categorical features (if any) in the dataset before we use them in algorithms. But with CatBoost, you don’t need to do encoding as the algorithm automatically encodes those categorical features into numeric values using one-hot encoding.
So, if your data has categorical features, you need to convert the entire dataset into CatBoost’s special Pool datatype by using the Pool() class. If your data has only numerical features, I recommend you not to convert the data into the Pool datatype as this may slow down the training process. So, convert only categorical data.
The following parameter settings in the Pool() class can be used to include categorical features directly with CatBoost.
data: The feature matrix defined by X. Possible types are Pandas DataFrames or 2D Numpy arrays.
label: The target or label column defined by y. Possible types are Pandas Series or 1D Numpy arrays.
cat_features: A list of categorical feature names in the dataset.
Here is an example:
from catboost import CatBoostRegressor, Poolmodel = CatBoostRegressor()pool_data = Pool(X_train,                 y_train,                                   cat_features=['F1', 'F2', ...])model.fit(pool_data)
A more detailed guide written by me can be found here.
The following parameters in both CatBoostClassifier() and CatBoostRegressor() classes and their fit() method can be used to reduce the overfitting of the model.
early_stopping_rounds: This is available in the fit() method of both CatBoostClassifier() and CatBoostRegressor() classes. The default value is False that does not activate early stopping. We can use an integer to stop the learning process early to reduce overfitting. A detailed guide written by me can be found here.
l2_leaf_reg or reg_lambda: This is available in both CatBoostClassifier() and CatBoostRegressor() classes. This adds the coefficient of the L2 regularization term to the cost function. The default value is 3. The type is float. Increasing the value prevents overfitting.
depth: This parameter defines the depth of the tree. This is similar to the max_depth parameter found in decision trees. The default value is 6. The type is integer. Decreasing the value prevents overfitting. Good values are 2 and 3.
The learning curve is extremely important to monitor the learning process of a model. Both train and validation scores are plotted against the number of boosting rounds defined by n_estimators or iterations.
In CatBoost, we can easily generate the learning curve by setting the plot argument to True in the fit() method of both CatBoostClassifier() and CatBoostRegressor() classes. The following image shows such a learning curve created by CatBoost.
This learning curve has two use cases:
To find whether there is an opportunity for early stopping
To find the optimal value for the early_stopping_rounds argument
A detailed guide about this can be found by reading this article written by me.
This is another great feature of CatBoost. You don’t need to do parameter tuning in CatBoost to obtain great results. So, you can save time! According to the official documentation, the tuned and default parameter values provide almost the same performance scores!
What is your favorite CatBoost feature(s)? Please mention in the comment section.
To sum up: CatBoost provides a special approach to implement gradient boosting on decision trees. With its special features, it is easy to increase learning speed for large datasets with millions of data points. It also provides useful features to reduce overfitting. Finally, you can save time on parameter tuning since no tuning is needed in CatBoost.
Should you consider installing CatBoost? Yes, definitely! CatBoost is an open-source library that everyone can use! In addition to that, it has great documentation available on its official website. Have a look at it!
Explore boosting algorithms in ML with their Python implementations!
Find the solutions for the worst problem that most data scientists face: The Problem of Overfitting!
This is the end of today’s post. I always do my best to write and organize my content in a way that you can get the most out of them. There should be many ways to further improve my content and you may have great ideas for that. So,
Please let me know if you’ve any feedback.
Meanwhile, you can sign up for a membership to get full access to every story I write and I will receive a portion of your membership fee.
Thank you so much for your continuous support! See you in the next story. Happy learning to everyone!
Special credit goes to Yerlin Matu on Unsplash, who provides me with a nice cover image for this post.