XGBoost: Order Does Matter. | by Bitya Neuhof | Aug, 2021 | Medium | Towards Data Science

You probably ask yourself — why would I use feature importance to find related features in my data? It is much simpler, for example, to look at the Pearson correlation between pairs of variables.
That’s right.
Take a look at the (Pearson) correlation matrix:
There is no question about x1 and x4 having a high correlation, but what about x3 and x4? What does a correlation of 0.37 mean?
Data Scientists use machine learning models, such as XGBoost, to map the features (X) to the target variable (Y). Ideally, we would like the mapping to be as similar as possible to the true generator function of the paired data(X, Y). We use labeled data and several success metrics to measure how good a given learned mapping is compared to the true one.
XGBoost is a high-performance gradient boosting ensemble of decision trees, widely used for classification and regression tasks on tabular data. Like other decision tree algorithms, it consists of splits — iterative selections of the features that best separate the data into two groups. The algorithm assigns a score for each feature on each iteration and selects the optimal split based on that score (to read more about XGBoost, I recommend [1]). However, what happens if two features have the same score at a given level in the model training process? Which one will be preferred by the algorithm? It turns out that in some XGBoost implementations, the preferred feature will be the first one (related to the insertion order of the features); however, in other implementations, one of the two features is selected randomly.
Before we continue, I would like to say a few words about the randomness of XGBoost. In the process of building an ensemble of trees, some decisions might be random: sampling from the data, selecting sub-groups of features for each tree, etc. Running XGBoost with default parameters and no parallel computing yields a completely deterministic set of trees. If you change the value of the parameter subsample to be less than 1, you will get random behavior and will need to set a seed to make it reproducible (with the random_state parameter). In this post, I use subsample=1 to avoid randomness, so we can assume the results are not random.
Here is a simple example:
I created a simple data set with two features, x1 and x2, which are highly correlated (Pearson correlation coefficient of 0.96), and generated the target (the true one) as a function of x1 only.
num_samples = 1000x1 = (np.random.randn(num_samples)       + 2 * np.random.randn(num_samples)       + 3 * np.random.randn(num_samples) + 4)x2 = x1 * 0.5 + np.power(x1, 2) * 0.06 + 0.07df_simple = pd.DataFrame({'x1': x1, 'x2': x2})df_simple['target'] = df_simple['x1'] * 0.08
Training an XGboost model with default parameters and looking at the feature importance values (I used the Gain feature importance type. To read more about XGBoost types of feature importance, I recommend [2]), we can see that x1 is the most important feature. Great!
simple_model = xgb.XGBRegressor()simple_model.fit(X_train, y_train)...x1 importance: 0.295x2 importance: 0.0
Now, we will train an XGBoost model with the same parameters, changing only the feature's insertion order. x2 got almost all of the importance.
Not so great...
simple_model_reverse = xgb.XGBRegressor()simple_model_reverse.fit(X_train[['x2', 'x1']], y_train)...x1 importance: 0.03x2 importance: 0.118
Clearly, a correlation of 0.96 is very high. Starting at the beginning, we shouldn’t have included both features.
Back to our question about the correlation of 0.37, here is another, yet pretty simple, example:
The data set consists of 4 features, where x3 is a noisy transformation of x2, x4 is a non-linear combination of x1, x2, and x3, and the target is a function of x1 and x3 only.
x1 = np.random.randn(num_samples)x2 = np.random.uniform(0, 1, num_samples)x3 = x2 * 2 - 0.4345 + np.random.randn(num_samples)x4 = [x1[i] if x1[i] < 0.1 else       x2[np.random.choice(num_samples)]       + x3[i] for i in range(num_samples)]df = pd.DataFrame({'x1': x1, 'x2': x2, 'x3': x3, 'x4': x4})df['target'] = (0.2 * df['x1']                + 0.67 * np.sqrt(np.power((df['x1'] - df['x3']), 2))               + df['x1'] * df['x3'])
To simulate the problem, I re-built an XGBoost model for each possible permutation of the 4 features (24 different permutations) with the same default parameters. You can see in the figure below that the MSE is consistent. But, in contrast to the model’s performance consistency, feature importance orderings did change.
In 75% of the permutations, x4 is the most important feature, followed by x1 or x3, but in the other 25% of the permutations, x1 is the most important feature. There are two problems here:
The order is inconsistent.The target is an arithmetic expression of x1 and x3 only! x4 was not part of the equation that generated the true target.
The order is inconsistent.
The target is an arithmetic expression of x1 and x3 only! x4 was not part of the equation that generated the true target.
Different features ordering yields a different mapping between features and the target variable. The reason might be complex indirect relations between variables. But why should I care?
It might not be correct to consider the feature importance as a good approximation of the contribution of each feature to the true target. It is important to remember that it only reflects the contribution of each feature to the predictions made by the model. Sometimes this is just what we need. But, in other cases, we would like to know whether the feature importance values explain the model or the data ([3]).
A feature might not be related (linearly or in another way) to another feature. Yet, during the training process, at some subspace of the features space, it might get the same score as the other feature and be chosen to split the data. If two features can be used by the model interchangeably, it means that they are somehow related, maybe through a confounding feature.
Pay attention to features order. If you are not sure, try different orderings. Use your domain knowledge and statistics, like Pearson correlation or interaction plots, to select an ordering. Criticize the output of the feature importance. Again, use your domain knowledge to understand if another order might be equally reasonable.
[1] XGBoost Tutorials — Introduction to Boosted Trees
[2] Interpretable Machine Learning with XGBoost by Scott Lundberg
[3] Chen, H., Janizek, J. D., Lundberg, S., & Lee, S. I., True to the Model or True to the Data? (2020), arXiv preprint arXiv:2006.16234.‏
[4] Correlation in XGboost by Vishesh Gupta
[5] Feature importance results sensitive to feature order