Introduction to XGBoost
By: Daksh Mehta
Tags: Gradient Boosting, Regression, AI, Xgboost, Machine Learning

What is XGBoost?

It is a decision tree based ensemble learning model that makes use of gradient boosting. Another widely used algorithm these days are artificial neural networks (or ANNs). However, ANNs are used in problems where the data is unstructured such as while performing text recognition/image pattern recognition/etc. When we are dealing with structured data, Decision Tree based algorithms are considered the best algorithm these days.

XGBoost was developed by Tianqi Chen and can be used to solve regression as well as classification problems.

Boosting

Boosting works on the principle of an ensemble and combines various weak learners to make an improved and more accurate prediction.(A weak learner is one which is slightly better than random guessing,i.e., whose prediction accuracy is greater than 0.5)

It is a technique where the outcomes of the next model are based on the misclassifications made by the previous model. The correctly predicted outcomes are given a lower weight and the incorrect ones are given a higher weight so that the next weak learner tries to rectify the mistakes made by the previous one.

Let’s take a look at an illustration to help us understand it better:

These 4 boxes represent 4 different classifiers trying to classify + and - classes respectively.

Box 1: It creates a split at D1 and classifies anything to the left of it as ‘+’ and to the right as ‘-’ and as a result it misclassifies 3 points as ‘+’.

Box 2: Classifier 2 gives more weight to the three misclassified points by the previous learner and hence creates a split at D2 but ends up misclassifying 3 ‘-’ points.

Box 3: Classifier 3 gives more weight to the three ‘-’ points misclassified earlier and splits at D3 but it still misclassifies three points as shown in the illustration.

Box 4: Classifier 4 is a weighted combination of all the three previous ones and as a result does a great job classifying all the points.

This is the basic idea behind boosting algorithms. Making several weak learners and each subsequent learner making use of the misclassifications made by the previous one and in the end combining all of them to make a prediction. XGBoost’s default base learners are tree ensembles. It is a set of classification and regression trees(CART).

Why to go with XGBoost when other Algos are available?

The two main reasons to use XGBoost are as follows:

Execution Speed Model Performance

As a matter of fact these two reasons are also the main goals of the XGBoost project, to provide faster execution with better performance.

Benchmark performance of XGBoost

So how does XGBoost perform so well?

Advantages over normal gradient boosting

Parallelization: It offers parallelization of tree construction so that all CPU cores are used while training thus reducing training time.

2. Hardware Optimization: The algorithm makes efficient use of hardware resources.

3. Cross-validation: The algorithm comes with built-in cross validation.

4. Sparsity Awareness: XGBoost naturally admits sparse features for inputs by automatically ‘learning’ best missing value depending on training loss and handles different types of sparsity patterns in the data more efficiently.

5. Regularization: It can make use of both L1(lasso) and L2(ridge) regularization to prevent overfitting.

6. Tree Pruning: In normal GBM framework the stopping criteria for tree splitting is greedy and is dependent upon loss criterion at point of split. In comparison XGBoost uses max depth and starts pruning trees backwards. The result is significant improvement in computational performance.

Can XGBoost be used for every problem?

The toughest decision for a data scientist when he or she comes across a problem is that which model should be used to tackle the problem. They must test all possible algorithms and see which performs the best and gives the best results only then we can decide a winner. Howevever Hyperparameter tuning is in fact as important as choosing the right model if not more. Furthermore we also have to keep in mind the computational complexity of each algorithm. Keeping all these factors in mind only we can select an algorithm. There is no single algorithm which can take care of all problems since each problem is different in nature.