Stop Using XGBoost. Use CatBoost Instead | Towards Data Science

IntroductionXGBoost DisadvantageousCatBoost BenefitsSummaryReferences
Introduction
XGBoost Disadvantageous
CatBoost Benefits
Summary
References
I want to first say that XGBoost is an incredibly powerful machine learning algorithm that has proven to win countless data science competitions, as well as most likely be at the forefront of most professional use cases in data science. For that reason, this algorithm has been in the spotlight for the past few years, which is a good thing. With that being said, it means that its disadvantages have also been in the spotlight, therefore, creating the motivation to make a similar algorithm that excels where XGBoost does not. Let’s discuss XGBoost and its competitor, CatBoost, and highlight why you would want to use this emerging algorthim, whether you are in school, or working as a professional data scientist and/or machine learning engineer. Keep on reading below to learn more about the disadvantages of XGBoost as well as the benefits of CatBoost.
This algorithm [3] is decision tree-based, utilizing optimized distributed gradient boosting.
Numerical Values Only
XGBoost is great — with numeric values. This point means that it cannot handle categorical features unless they are transformed for the model to work. Most data scientists will use one-hot-encoding, which will transpose the feature values as columns themselves, and put a 0 or 1 as the value. This is great again, but can lead to another problem.
Sparse Dataset
What I mean by sparse, here, is that because you can not have a single column for a categorical feature, you will have as many columns as you have unique values within that categorical feature. For example, if there are 100 unique values, you will have 100 more columns appended to your dataset, full of zeros and ones. Of course, this can lead to another problem.
Longer Training
As you have more and more categorical features transformed into numeric features, this might mean that your model will take longer to run with a bigger dataset.
Deployment
As you want to deploy your model and make inferences or predictions on new data, this data will have to be in the same structure as your training data. This problem means that you will have to transform your prediction data, which can lead to more code, more room for errors, and just more of a hassle.
Parameters
XGBoost is very powerful, but can be overwhelming with parameters, as well as tuning them. It can take some time to understand all the params as well as tune them.
As you can see, most of these disadvantages relate to categorical features. So, you can see why there is a similar model that shares a lot of the same benefits as XGBoost, but focuses on fixing the category problems of XGBoost. Hence, the name CatBoost — categorical boosting.
This algorithm [5] is also a high-perming algorithm based on gradient boosting no decision trees, with a focus on handling categorical features, which have often been neglected in the past. Below, are the benefits of CatBoost:
Handles Categorical Features
Do not need to transform categorical features into numeric features
Meaning you do not have to preprocess your data (as much — depending on your use case)
Your dataset can remain in its original format, as the original columns, without having to make one-hot-encoded columns, which is easier/simpler, and more conducive to deployment, as well as explaining your features to stakeholders
Easy Parameters
Utilizing the default parameters from CatBoost will most likely lead to the best result
Meaning, less time spent on tuning params
Accuracy Improvements
While XGBoost tends to have the best accuracy out of most algorithms, CatBoost has been proven on severe popular datasets to beat it
Both algorithms are great, but if you have categorical features, then CatBoost is a great alternative.
This article is one-sided, as I am highlighting the benefits of one algorithm while discussing the disadvantages of another. However, that is why CatBoost is what it is. But, we can not have CatBoost without XGBoost, so to summarize, both algorithms and respective libraries are powerful, beneficial, and ultimately it is up to you to decide which one to use. There are other factors that can come up, like documentation. XGBoost has much more documentation as well as examples online, and this is where CatBoost struggled, but over time, I can see CatBoost improving in this area. Perhaps, another algorithm will emerge that is even better than CatBoost.
To summarize, here are the main reasons to use CatBoost:
* Ease of using categorical features* Implementation of dataset processing* Improved accuracy
I hope you found my article both interesting and useful. Please feel free to comment down below if you agree with these comparisons — why or why not? Do you prefer XGBoost or CatBoost? If you do, feel free to comment on the reasons why you prefer one over the other. What disadvantegoues are there to CatBoost? Do you know of an even newer machine Learning algorithm and library that could best CatBoost out in terms of simplicity, ease of use, and accuracy? Thank you for reading my article!
Please feel free to check out my profile and other articles, Matt Przybyla, as well as reach out to me on LinkedIn.
[1] Photo by Pacto Visual on Unsplash, (2016)
[2] Photo by Sebastian Unrau on Unsplash, (2015)
[3] xgboost developers, XGBoost Documentation, (2020)
[4] Photo by Ludemeula Fernandes on Unsplash, (2017)
[5] Yandex, CatBoost Documentation, (2021)