Keep Calm and Stack Up— Implement Stacking Regression in Python using mlxtend | by Dehao Zhang | Towards Data Science

If you have ever combined multiple ML models to boost your score up on the leaderboard on some Kaggle competition, you know what this is about. In fact, many winning solutions of these Kaggle competitions use ensemble models instead of just a single fine-tuned model.
The intuition behind ensemble models is quite simple: combining different ML algorithms effectively can reduce the risk of an unfortunate selection of one poor model.
In this post, I will discuss Stacking, a popular ensemble method and how to implement a simple 2-layer stacking regression model in Python using the mlxtend library. The sample task that I have chosen is Airbnb pricing prediction.
What is stacking and how does it work?How does stacking model perform compared with the base models?What are the caveats of stacking?What can we do next?
What is stacking and how does it work?
How does stacking model perform compared with the base models?
What are the caveats of stacking?
What can we do next?
Stacking, or Stacked Generalization, is a meta-learning algorithm that learns how to combine the predictions of each base algorithm in the best way [1].
In simple terms, here is how you build a 2-layer stacking model. The first layer, and some people refer to this as the base layer, includes the base models. In the context of classification, you can think of the base models as any classifier that you can use to make prediction as Neural Network, SVM, Decision Tree, etc. Similar logic also applies to regression. After we train these models using the training set, the second layer builds the meta-model by taking the predictions from those models and the expected outputs on the out-of-sample data. You can think of the simplest case of this meta-model is to average out the predictions from base models.
A common approach to avoid over-fitting is to perform cross-validation on these base models, and then use the out-of-fold predictions and the outputs to build the meta-model.
Check out David Wolphet’s paper for more technical details on how stacking works.
I have chosen the Airbnb pricing prediction task from Kaggle (link) to be the example here because it is a relatively simple dataset with small sample size and set of features. The task is to predict the apartment rental listing price on Airbnb based on information such as number of bedrooms, type of rental (whole apartment or private room or shared room), etc.
To see the full Python code, check out my Kaggle kernel.
Without further ado, let’s get to the details!
For the purpose of this post, I won’t discuss the preprocessing steps, but please refer to the Kaggle kernel for full detail.
On a high level, I examined the distribution of each feature, removed outliers, create dummy variables for the categorical ‘room_type’ (since there are only three categories), and standardize the features.
Train/Test Split
I set 70% of the dataset to be training set and the remaining 30% would be used as test set.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)
Base Models
Let’s first build some base models. Here I have chosen Xgboost, Lasso Regression, and KNN as the three base models. In principle, stacking works best when the models are in a diverse range and their prediction errors are as uncorrelated as possible, so that individual model’s weaknesses and biases are offset by the strengths of others[2].
Performance Metric
In terms of performance metric to evaluate models, I choose to use Mean Absolute Error (MAE), which measures how far the predicted values are away from the measured values.
Performance of the Base Models
Let’s quickly examine the performance of the three base models (all default hyperparameters):
Model: XGBRegressor, MAE: 35.80629559761084Model: Lasso, MAE: 35.0691542225316Model: KNeighborsRegressor, MAE: 38.35917874396135
The MAEs are between 35–38, meaning the predicted prices are about $35-$38 away from the true prices on average.
Build a Stacking Model Using mlxtend
Now, we can build on top of the base models for the meta-learner. Alternatively, we can do a shortcut using ‘StackingCVRegressor’ from the mlxtend library (documentation):
from mlxtend.regressor import StackingCVRegressorstack = StackingCVRegressor(regressors=(XGBRegressor(),                             Lasso(), KNeighborsRegressor()),                            meta_regressor=Lasso(), cv=10,                            use_features_in_secondary=True,                            store_train_meta_features=True,                            shuffle=False,                            random_state=1)
The same three based models are used, and for the meta-learner, I chose to use Lasso. We could also try other regression models as meta-learners, and then build another meta-learner on top (3-layer stacking!) Note that in this configuration, I choose to also use the raw training data as inputs to the meta-model, since it can provide more context to the meta-model.
Now let’s fit our training data into the model and check the performance on the test set:
stack.fit(X_train, y_train)X_test.columns = ['f0', 'f1', 'f2', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', 'f10', 'f11'] # Xgboost internally renames the featurespred = stack.predict(X_test)score = mean_absolute_error(y_test, pred)print('Model: {0}, MAE: {1}'.format(type(stack).__name__, score))--------------------------------------------------------------------Model: StackingCVRegressor, MAE: 34.01755981865483
We see that the new MAE is about 34, decreased by 1 compared with the result from the best base model, Lasso.
Although the difference might seem trivial, small gains like this can actually make a huge difference on the leaderboard ranking.
If you want to look into ensemble neural networks, check out this paper.
Despite the capabilities of stacking, here are a few caveats to keep in mind:
Effective use of stacking requires a bit of trials and errors sometimes, and there is no guarantee that stacking would result in an improved performance in all cases.Stacking can be computationally expensive, especially when the number of stacking layer is high.The interpretability of the model decreases as the number of stacking layer increases. This is not a big problem for Kaggle competition but for business cases, understanding the importance of features and its incremental impact on the outcome variable might be more important.
Effective use of stacking requires a bit of trials and errors sometimes, and there is no guarantee that stacking would result in an improved performance in all cases.
Stacking can be computationally expensive, especially when the number of stacking layer is high.
The interpretability of the model decreases as the number of stacking layer increases. This is not a big problem for Kaggle competition but for business cases, understanding the importance of features and its incremental impact on the outcome variable might be more important.
Here are a few things that we can try as next steps:
Hyperparameter tuning — There are a few hyperparameters that we can tune in the selected models to further optimize the performance. ‘GridSearchCV’ can be a helpful tool in this process.Keep stacking — Try to build additional stacking layer(s) and make a plot of the performance metric vs the number of layers to examine if diminishing marginal return holds true (It should!).
Hyperparameter tuning — There are a few hyperparameters that we can tune in the selected models to further optimize the performance. ‘GridSearchCV’ can be a helpful tool in this process.
Keep stacking — Try to build additional stacking layer(s) and make a plot of the performance metric vs the number of layers to examine if diminishing marginal return holds true (It should!).
Let’s quickly recap.
We implemented a simple 2-layer stacking regression model in Python using the mlxtend library, compared its test MAE with the ones of three base models and observed an improvement.
I hope you enjoyed this blog post and please share any thoughts that you may have :)
Check out my other post on dimensionality reduction using t-SNE:
towardsdatascience.com
[1] https://machinelearningmastery.com/stacking-ensemble-machine-learning-with-python/[2] http://support.sas.com/resources/papers/proceedings17/SAS0437-2017.pdf