Tackling Kaggle’s Mercedes-Benz Greener Manufacturing Competition with Python
By: Borna Ahmadzadeh
Tags: AI, Kaggle, Data Science, Machine Learning, Artificial Intelligence

Photo by Marcel Strauß on Unsplash

Introduction

In this part (which is the last article in this series), we’ll build our final model, a reasonably accurate ensemble that will place in the top 10% of the Kaggle private leaderboard. You can find the notebook for this tutorial here.

Without further ado, let’s get coding (in Colab)!

Feature Selection Using OMP

We’ve already seen how well orthogonal matching pursuit performs on our dataset. One reason could be that, like many tabular datasets, most of the columns are extremely noisy and/or irrelevant to the dependent variable, so only a handful of features are actually needed. That is very true with this dataset, as we’ve already seen how, using only 5 binary features, we can make predictions almost as accurate as a model trained on the entire dataset, which consists of a whopping 376 features. Therefore, a model like orthogonal matching pursuit, which uses a fraction of the original features, should and does score relatively high on this dataset.

It would thus not be unreasonable to train other (tree-based) models on the features orthogonal matching pursuit uses (i.e. ones with nonzero coefficients), which should give us better results because remember, at the end of the day, OMP is just a linear model. At first, I was skeptical of this idea because we’ve already seen how feature selection using a Random Forest didn’t help at all, so why should this be any different? Well, first of all, it is possible our original Random Forest couldn’t see the relation between certain columns and the dependent variable whereas orthogonal matching pursuit did, so we threw those away during feature selection. But perhaps more importantly is that matching pursuit is using a one-hot encoded version of our data, which means if all that matters when it comes to ‘X0’ is, say, whether it’s 3, 7, or something else (for example, if it’s 3, then test times rise, if it’s 7, test times are shorter, and if it’s anything else, it has no effect on the dependent value), then it can easily multiply the dummy variable for ‘X0’ = 3 and 7 by the right coefficients. In theory, Random Forests should be able to take care of such instances but in practice, if doing so requires too many splits and/or there are better splits available, that might not be the case (had we tried feature selection using Random Forests on a one-hot encoded version of our data, we might’ve been able to achieve what we’re about to see without the need to use orthogonal matching pursuit. Yet another mistake on my behalf).

Let’s give this a shot:

orth = OrthogonalMatchingPursuit(n_nonzero_coefs=20).fit(X_one, y) # np.where() returns the indices of nonzero elements # orth.coef_ is just the coefficients multiplied by the features, # so if the ith coefficient isn't zero, it means that feature # is being used to_keep = list(X_one.columns[np.where(orth.coef_)[0]]) train_keep = train_one[to_keep+[dep_var]].copy() X_keep = train_keep.drop(dep_var, axis=1) test_keep = test_one[to_keep].copy() display_df(train_keep)

Output:

Like we speculated, only certain categories in ‘X0’, ‘X2’, and ‘X5’ matter and the rest are trivial (attentive readers might notice some similarities between the binary variables we have here and the ones used to create our magical feature). There’s one more step left before we can train tree-based models on this dataset — and get surprisingly good results.

Dummy Variables to Categorical Columns

Linear models, as you already know, require the dummification of categorical features. However, one-hot encoding could hurt tree-based models’ performance and hence, we need to convert the dummy variables in our dataset into single categorical columns. In part 4, we did something similar while we were creating our super feature, and we can use the same approach here as well, albeit on select columns rather than the entire dataset:

def dummy_to_cat(col, df): dummy_names = [i for i in df.columns if i.startswith(col+'_')] dummies = df[dummy_names] # Create a Series of tuples where the ith # tuple is a tuple of the dummy variables # in row i cat = dummies.apply(tuple, axis=1) # Label encoding (tuples >- ints) cat = cat.astype('category').cat.codes df.drop(dummy_names, axis=1, inplace=True) df[col] = cat return df df = train_keep.drop(dep_var, axis=1).append(test_keep) # Doing this to 'X2's dummy variables isn't # needed, since there's only 1 such variable for col in ['X0', 'X2', 'X5']: df = dummy_to_cat(col, df) X_keep = df[:len(train_raw)] train_keep = X_keep.copy() train_keep[dep_var] = y test_keep = df[len(train_raw):] display_df(df)

Output:

Neat! Let’s test out a Random Forest to see whether what we’ve done so far can help tree-based models as well:

rf_cross_val = partial(print_cross_val, RandomForestRegressor, train_keep, n_jobs=-1, n_estimators=40) rf_cross_val()

Output: Cross validation scores: [0.609 0.614 0.533 0.599 ...]

Cross validation average score: 0.561

Cross validation standard deviation: 0.060 rf_cross_val(min_samples_leaf=3)

Output: Cross validation scores: [0.537 0.615 0.595 0.599 ...]

Cross validation average score: 0.569

Cross validation standard deviation: 0.059 rf_cross_val(min_samples_leaf=5)

Output: Cross validation scores: [0.638 0.633 0.518 0.466 ...]

Cross validation average score: 0.571

Cross validation standard deviation: 0.059 rf_cross_val(min_samples_leaf=10)

Output: Cross validation scores: [0.447 0.613 0.554 0.479 ...]

Cross validation average score: 0.561

Cross validation standard deviation: 0.063 rf_cross_val(min_samples_leaf=5, max_features=0.5)

Output: Cross validation scores: [0.560 0.570 0.505 0.590 ...]

Cross validation average score: 0.567

Cross validation standard deviation: 0.052

The best score we get is 0.571, which is slightly worse than orthogonal matching pursuit but quite a bit better than anything we were able to attain with Random Forests in past articles. It’s important to keep in mind that previously, we sometimes got a score over 0.57, but that was mostly due to the randomness involved in train-test splits. Right now, out of every 4 or 5 runs of cross-validation (with k = 100), we get an R2 of more than 0.57, which suggests this Random Forest is likely better than the previous ones (in other words, if we set k to a large enough value, this Random Forest would score higher than 0.57, whereas previous iterations would score lower). We’re definitely on the right track!

LightGBM

Random Forests are nice for quick experimentation since they’re fast, intuitive, and easy to tune. Unfortunately, though, gradient boosting machines generally outperform them by a margin that is often decisive in Kaggle competitions, making them a favourite amongst contestants (it also doesn’t hurt that they’re quite fast and heavily optimized). Hence, if we’d like to have a shot at competing, we need try them out as well.

There are numerous available implementations of GBMs for Python, with the two most popular ones being XGBoost and Microsoft’s LightGBM. Both models yield roughly the same accuracy more often than not, but the tiebreaker is their execution time. LightGBM is often 3–10X faster than XGBoost, which is vital in our case because it enables us to increase k while doing cross-validation, thereby giving us a better picture of our model’s performance and allowing for more rigorous hyperparameter tuning. LightGBM is Scikit-learn compatible, so it has a gentle learning curve for those familiar with the Scikit-learn:

from lightgbm import LGBMRegressor

Before we go any further, remember that in the first article, we dropped the ID column since dealing with IDs can sometimes be frustrating and, at first, I couldn’t see how the ID of a car could be related to its test time. Just to be sure, though, I tried feeding it to a Random Forest (alongside the rest of the data) and even after extensive tuning, it didn’t budge our score whatsoever.

However, it did help LightGBM and XGBoost, so we’ll bring it back:

train_keep = train_keep.join(train_id) X_keep = X_keep.join(train_id) test_keep.reset_index(drop=True, inplace=True) test_keep = test_keep.join(test_id)

Again, LGBM is Scikit-learn compatible, so the process of training & validation remains pretty much the same. Please be aware that, unlike Random Forests, gradient boosting machines have a real chance of overfitting if there are too many trees and hence, we can’t begin with, say, 40 trees for validation and add more trees for our final model. Instead, we’ll have to start with a relatively high number of trees right off the bat:

# Good initial parameters lgbm_cross_val = partial(print_cross_val, LGBMRegressor, train_keep, k=1000, n_jobs=-1, n_estimators=400, learning_rate=0.01, max_depth=3, min_child_samples=1) lgbm_cross_val()

Output: Cross validation scores: [0.636 0.466 0.475 0.516 ...]

Cross validation average score: 0.573

Cross validation standard deviation: 0.055 lgbm_cross_val(max_depth=4)

Output: Cross validation scores: [0.618 0.604 0.628 0.589 ...]

Cross validation average score: 0.571

Cross validation standard deviation: 0.056 lgbm_cross_val(learning_rate=0.03)

Output: Cross validation scores: [0.606 0.614 0.597 0.476 ...]

Cross validation average score: 0.572

Cross validation standard deviation: 0.055 lgbm_cross_val(min_child_samples=3)

Output: Cross validation scores: [0.595 0.554 0.488 0.597 ...]

Cross validation average score: 0.576

Cross validation standard deviation: 0.054 lgbm_cross_val(min_child_samples=5) Output: Cross validation scores: [0.529 0.449 0.644 0.571 ...]

Cross validation average score: 0.573

Cross validation standard deviation: 0.054

With min_child_samples=3, we get an R2 of 0.576, which is absolutely astounding, considering we’re not using anything from articles 1, 2, 3, and 4 (in fact, one could only read this part and still fully understand what we’re talking about) and yet, we’re outperforming every single model we’ve come up with so far. We’re almost ready to submit to Kaggle.

XGBoost

Although, at their core, XGBoost and LGBM are the same algorithm, their details do differ, so ensembling them should generally help, particularly in Kaggle competitions since every last decimal point counts. Unsurprisingly, XGBoost is nearly identical to LightGBM in terms of hyperparameters and syntax, so everything remains pretty much the same:

xgb_cross_val = partial(print_cross_val, XGBRegressor, train_keep, k=200, silent=True, n_jobs=-1, n_estimators=400, max_depth=3, learning_rate=0.01, colsample_bytree=1) xgb_cross_val()

Output: Cross validation scores: [0.633 0.589 0.593 0.444 ...]

Cross validation average score: 0.550

Cross validation standard deviation: 0.059 xgb_cross_val(max_depth=4)

Output: Cross validation scores: [0.538 0.455 0.585 0.417 ...]

Cross validation average score: 0.552

Cross validation standard deviation: 0.058 xgb_cross_val(max_depth=5)

Output: Cross validation scores: [0.573 0.574 0.561 0.615 ...]

Cross validation average score: 0.550

Cross validation standard deviation: 0.059 xgb_cross_val(max_depth=4, learning_rate=0.03)

Output: Cross validation scores: [0.480 0.567 0.583 0.631 ...]

Cross validation average score: 0.566

Cross validation standard deviation: 0.054 xgb_cross_val(max_depth=4, learning_rate=0.1)

Output: Cross validation scores: [0.504 0.584 0.548 0.526 ...]

Cross validation average score: 0.533

Cross validation standard deviation: 0.058 xgb_cross_val(max_depth=4, learning_rate=0.03, colsample_bytree=0.5)

Output: Cross validation scores: [0.588 0.650 0.580 0.508 ...]

Cross validation average score: 0.570

Cross validation standard deviation: 0.057 xgb_cross_val(max_depth=4, learning_rate=0.03, colsample_bytree=0.3)

Output: Cross validation scores: [0.620 0.468 0.590 0.578 ...]

Cross validation average score: 0.557

Cross validation standard deviation: 0.055

An R2 of 0.570, which seems underwhelming at first but as we’ll see, our XGBoost can provide a nice boost for our LGBM.

Ensembling

Ensembling is the final step in virtually all winning solutions on Kaggle, no matter the domain or topic of the dataset. In short, it is basically when one leverages the power of several models to get a slightly better, bigger model. There are numerous ensemble techniques, some very simple, others fairly complex. The method we’ll be using today belongs to the first category, and it is known as weighted average ensembling, which is, as you’ve probably guessed, when you take the weighted average of the predictions of multiple models. Choosing what weights to assign to each model can be a tedious task if there are too many models, so one usually uses a linear model to automatically figure out the optimal values. However, there are only 2 models in our case, making the process of finding good weights manually rather straightforward (and also faster).

Scikit-learn learn has a class named VotingRegressor(), which does exactly what we need: It takes in a bunch of Scikit-learn compatible models, alongside the weights for each one, and returns a model with Scikit-learn functionalities, such as fit(), predict(), etc.

Let’s first create the base models:

lgbm = LGBMRegressor(n_jobs=-1, n_estimators=400, learning_rate=0.01, max_depth=3, min_child_samples=3) xgb = XGBRegressor(silent=True, n_jobs=-1, n_estimators=400, max_depth=4, learning_rate=0.03, colsample_bytree=0.5) estimators = [('lgbm', lgbm), ('xgb', xgb)]

And now we can proceed as usual ([0.9, 0.1], [0.7, 0.3], …, [0.1, 0.9] are good weights):

ens_cross_val = partial(print_cross_val, VotingRegressor, train_keep, estimators=estimators, n_jobs=-1) ens_cross_val(weights=[0.9, 0.1])

Output: Cross validation scores: [0.493 0.570 0.501 0.507 ...]

Cross validation average score: 0.578

Cross validation standard deviation: 0.058 ens_cross_val(weights=[0.7, 0.3])

Output: Cross validation scores: [0.457 0.597 0.618 0.622 ...]

Cross validation average score: 0.568

Cross validation standard deviation: 0.059

As we would expect, assigning too much weight to XGBoost’s predictions isn’t a great idea, since its accuracy is well below that of our LGBM. Despite that, a tiny amount of XGBoost does help a bit, resulting in a model with a promising R2 of 0.578!

P.S: Please bear in mind that in order for an ensemble model to be as effective as possible, the base models need to be as uncorrelated as possible: Maybe one is a linear model, another is a Random Forest, perhaps there’s an LGBM trained on a modified version of the dataset, and so forth. Thus, it’d seem like having a linear model or a Random Forest in our ensemble should help, especially because both score better than XGBoost and almost as good as LGBM. However, I tried adding various models into the mix and for some reason, it didn’t improve our score one bit.

Submitting to Kaggle

We’re now ready to submit to Kaggle. Were we competing for first place, our model would have to be a lot better and, let’s be honest, we didn’t really put in that much work into it. But our aim is top 10%, and our model is likely that good, given that our secret sauce was OMP (which, I looked on Kaggle and no one was using), without which we wouldn’t have been able to get an R2 of 0.578

ens = VotingRegressor(estimators, n_jobs=-1, weights=[0.9, 0.1]).fit(X_keep, y) test_pred = ens.predict(test_keep) test_pred = pd.DataFrame({'ID': test_id, dep_var: test_pred}) test_pred.to_csv('submission.csv', index=False) # Submit to Kaggle !kaggle competitions submit -c mercedes-benz-greener-manufacturing -f submission.csv -m ""

Heading to the competition’s submission page gives:

Our R2 on the private test set is 0.55183, which puts us at 376th (<10%) on the leaderboard! In a competition with a moderately high number of participants (over 3800), that is definitely a great starting point, and we should be proud of how far we’ve come with comparatively little work. But we must not get too cocky, as there are quite a few ticks we could use to place even higher up the leaderboard. Some that come to mind include:

More hyperparameter tuning: There are quite a few hyperparameters in XGBoost and LGBM we didn’t cover in this article. Tuning them might increase our score by as much as a thousandth. Higher k: It is possible that due to the fact that we get such high STDs during cross-validation (implying we can sometimes get values that are too optimistic / pessimistic), some of the hyperparameter values we chose are not actually optimal. Setting k to a higher value would lead to more robust values for our hyperparameters, especially since we’re using Monte Carlo cross-validation. Classifier for detecting outliers: You should know by now that this dataset is riddled with anomalies, and being able to detect which rows are potentially outliers could be game-changing. Thus, it’d be interesting to see whether a model can actually do that and if so, how could we use it to modify our model’s predictions? Less clumsy constant column removal: Before we fed our data to orthogonal matching pursuit, we removed all binary columns which had fewer than 40 instances of zero or one, which could really damage our model’s performance. A more careful cleaning may have possibly led to different columns chosen by orthogonal matching pursuit, which could’ve in turn made our final model better.

If you’re interested in this competition, you should give these a shot and see how well they perform. But even without these, know that we have a model that earns us a bronze medal in a Kaggle competition!

Conclusion

In this part, we used orthogonal matching pursuit to not only get rid of redundant binary columns, but to also reduce down categorical columns with cardinalities ranging from 25 to 45 to ones with fewer than 10 categories. We then loosely tuned an LGBM (R2 = 0.576) and an XGBoost (R2 = 0.570), two popular gradient boosting model which dominate Kaggle tabular competitions. Finally, we used a weighted average ensemble to combine the two models, which put us in the top 10% of Kaggle’s private leaderboard, a pretty good ranking given the number of contenders!

This was officially the last part of this series, and there’ll be no more articles after this one. I hope you learned at least 1 thing from this series and enjoyed reading it as much as I did writing it!

Please, if you have any questions or feedback at all, feel welcome to post them in the comments below and as always, thank you for reading!

Part 1: https://medium.com/python-in-plain-english/mercedes-benz-greener-manufacturing-part-1-basic-data-pre-processing-a32d17803064

Part 2: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-3ddff72d0187

Part 3: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-1ca6b030bf58

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-b5220f479a44

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-7b203e886f8d

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-a004659e02c4

Part 4: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-82dd27e53757

Part 5: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-e31198ecafae

Twitter: https://twitter.com/bobmcdear

GitHub: https://github.com/bobmcdear