Tackling Kaggle’s Mercedes-Benz Greener Manufacturing Competition with Python
By: Borna Ahmadzadeh
Tags: AI, Kaggle, Data Science, Machine Learning, Artificial Intelligence

Introduction

In this part, we’re going to try to come up with just one feature (well, technically 5, as we’ll see) that can be used to predict the y-value. Although we won’t improve of our score (which makes everything we’re going to do useless since this is a Kaggle competition), we’ll build an insanely simple model which, in a real-world scenario, can be very useful, for it’s very simple and uses so few features. You can find the notebook for this tutorial here.

Without further ado, let’s get coding (in Colab)!

From Infinity to 8

With the original dataset, there were 368 binary features and 8 categorical ones, each with around 25 classes so in theory, there would be about 10¹²² (the number of protons you could fit in the universe) possible rows! If all these columns captured vital data, there’d be no way we could compress them into one single feature without sacrificing performance.

However, that is hardly ever true when it comes to tabular data and as we’ve already seen, there are many duplicate and redundant features in the original dataset, idly consuming precious memory and making the training of our models notably longer. Currently, we know 5 binary features, alongside ‘X5’, are all the features needed for a model as good as one trained on all the columns. But it’s possible we could remove even more features without causing a decline in our score, so let’s try just that.

First, let’s set a baseline we can later compare our scores with:

print_cross_val(RandomForestRegressor, train, **rf_args)

Output: Cross validation scores: [0.564 0.614 0.458 0.511 ...]

Cross validation average score: 0.566

Cross validation standard deviation: 0.050

As we already know from previous articles, our Random Forest places high importance on each of the features in ‘train’, implying they’re all predictive of the dependent variable and hence, removing them would cause a drop in our score, right? Not necessarily. It’s possible that there’s a high correlation between each one of our features and the dependent variable but the value of one (or multiple) of the features can be deduced by looking at the rest of the columns. So, in short, all the columns in the dataset could be used to make accurate predictions but that doesn’t indicate they’re all needed.

With that in mind, let’s make an interesting hypothesis: Every feature in the dataset can be deduced by solely looking at the other features, even though it might be correlated no single feature.

In order to test this hypothesis, we can simply train a Random Forest on all but on of our column for all the columns in the dataset and see how that affects the model’s R2.

for feat in X.columns: print_cross_val(RandomForestRegressor, train.drop(feat, axis=1), **rf_args)

Output: (Too big)

Removing some of the features, such as ‘X119’, made our model a lot worse, contrary to what we thought. However, our hypothesis does indeed hold true for others, like ‘X5’, even though our model initially ranked it as an important feature. Again, this doesn’t necessarily mean that ‘X5’ isn’t correlated with the dependent variable; it merely means ‘X5’s role could be replaced by some of the other features.

This result by itself is very interesting and requires more analysis (one could, for instance, try the exact same thing as above but drop two features instead of one to see the importance of feature interactions. Another thing you could try would be recreating a feature using the other ones) but what I’d like to direct your attention to for now is ‘X5’. Since removing it from the dataset doesn’t really cause our R2 to go down, we might as well permanently drop it:

x5 = train['X5'] train.drop('X5', axis=1, inplace=True) X.drop('X5', axis=1, inplace=True)

Now here’s the interesting part: We’ve got only 5 binary features left and therefore, there are only 2⁵ = 32 possible rows we could have in our dataset. In reality, there’d be even fewer rows because our features are most likely not independent and there’s a relatively low number of instances in our dataset anyway.

Pandas has a function that returns all the different rows in a DataFrame, so let’s call it:

X.drop_duplicates()

Output:

Only 8 rows! Now, with big datasets, there’d be so many combinations and possibilities that great generalization (something Random Forests excel at) would be needed for a model to perform decently on data it hasn’t seen before. However, that’s not the case here and we could probably get away with predicting the average dependent value of a particular row for all instances of that row in the validation set (in a sense, we’re looking for a model that grossly overfits).

To do so, we could easily create a model that uses pd.merge or something similar under the hood but what’d be even simpler is a decision tree that creates a leaf node for each possible row, practically resulting in the same model:

from sklearn.tree import DecisionTreeRegressor print_cross_val(DecisionTreeRegressor, train, k=2000)

Output: Cross validation scores: [0.435 0.611 0.633 ... ]

Cross validation average score: 0.565

Cross validation standard deviation: 0.055

Seldom do decision trees perform as good as Random Forests on a task, but, as we can see, there are certainly exceptions to every general rule-of-thumb (this is only because even with bagging, there wouldn’t be much variation between a Random Forest’s decision trees, rendering bootstrapping redundant).

We can go one step further and compact all these features into just one feature, ultimately reducing down a dataset with 376 features into just one column. But we need to make sure we’re using the same encoding for both the training and test set, so we’ll do everything globally.

test = test[X.columns] df = X.append(test) # A Series where the ith row is # a tuple of the form (X[i][X314], X[i][X315], ...) comb = df.apply(tuple, axis=1) df['comb'] = comb df.drop(X.columns, axis=1, inplace=True) # Map all the tuples to an int df['comb'] = df['comb'].astype('category') df['comb'] = df['comb'].cat.codes+1 train = df.iloc[:len(train)] train[dep_var] = y X = df.iloc[:len(train)] test = df.iloc[len(train):] display_df(df)

Output:

Exactly what we wanted! Let’s also look at some of the basic characteristics of this column:

df['comb'].nunique() df['comb'].value_counts() df['comb'].value_counts().plot.bar()

Output: 9 4 2611

8 2401

1 1518

7 1302

2 348

6 216

5 12

3 9

9 1

There are 9 distinct rows (8 in the training set + 1 new in the test set), the top two most common rows appear around 2500 times each, and the bottom three are almost non-existent.

Let’s train a decision tree on this new dataset:

print_cross_val(DecisionTreeRegressor, train, k=2000)

Output: Cross validation scores: [0.469 0.652 0.521 ...]

Cross validation average score: 0.565

Cross validation standard deviation: 0.055

Our R2 remains roughly the same, as is expected, but we now have just one column instead of 5. You should definitely experiment with something similar as it can both be a lot of fun (for me, it was honestly very satisfying & entertaining seeing how small this dataset could become) and might lead you to a finding which could help your model’s performance.

Conclusion

In this part, we first realized that even though a column could be correlated with the dependent value, that doesn’t mean it’s needed for a good performance, as was the case with some of the features. With ‘X5’ out of the way, we were left with only 5 binary features, basically allowing a simple decision tree to make as good as predictions as a Random Forest. Lastly, we assigned a value to each distinct row which let us replace the 5 binary columns with just one categorical column with a cardinality of 9 (a “super” feature, if you will).

In the next part, we’ll build a pretty decent ensemble model and use it to make final predictions on the test set and submit them to Kaggle.

Please, if you have any questions or feedback at all, feel welcome to post them in the comments below and as always, thank you for reading!

Part 1: https://medium.com/python-in-plain-english/mercedes-benz-greener-manufacturing-part-1-basic-data-pre-processing-a32d17803064

Part 2: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-3ddff72d0187

Part 3: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-1ca6b030bf58

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-b5220f479a44

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-7b203e886f8d

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-a004659e02c4

Part 5: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-e31198ecafae

Part 5 (final part): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-ecbd2714d952

Twitter: https://twitter.com/bobmcdear

GitHub: https://github.com/bobmcdear