Tackling Kaggle’s Mercedes-Benz Greener Manufacturing Competition with Python
By: Borna Ahmadzadeh
Tags: Machine Learning, Artificial Intelligence, AI, Data Science, Data Visualization

Photo by Markus Spiske on Unsplash

Introduction

In this part, we’ll perform exploratory data analysis (EDA) on our data, which is a crucial part of most machine learning problems. Although we might not end up increasing our score, we will draw invaluable insights from our data, which is often one of the primary objectives of real-world machine learning. We are going to use some of the traditional EDA techniques, but we’ll also touch on a few underused ones as well. You can find the notebook for this tutorial here.

Without further ado, let’s get coding (in Colab)!

Ceteris Paribus

From the previous article, we have a vague idea of what partial dependence is, the problem it tries to solve, and how it does so. However, quite a few of the details were left out, so this article will be devoted to filling those empty spots.

First, let’s talk about what ‘with all other things being equal’ means. Suppose we have a smaller version of our dataset, which could look something like this:

+------+------+------+----+-------+

| X314 | X119 | X127 | X5 | y |

+------+------+------+----+-------+

| 0 | 0 | 1 | a | 99 |

| 0 | 1 | 0 | b | 97 |

| 1 | 1 | 1 | c | 102 |

| 1 | 0 | 0 | a | 97.5 |

| 0 | 0 | 0 | b | 96.5 |

| 1 | 0 | 1 | c | 102.9 |

+------+------+------+----+-------+

And we’d like to know how the dependent variable reacts to different values of ‘X5’. In order to do so, we must modify our dataset so that all variables but ‘X5’ remain the same. How would we do that? Well, it’s as simple as replacing all instances of ‘X5’ with the class we’d like to know the average test time of. So we’d have 3 (number of classes in ‘X5’) different versions of our dataset and in each one, ‘X5’ is a constant:

Dataset A: +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | y |

+------+------+------+----+-------+

| 0 | 0 | 1 | a | 99 |

| 0 | 1 | 0 | a | 97 |

| 1 | 1 | 1 | a | 102 |

| 1 | 0 | 0 | a | 97.5 |

| 0 | 0 | 0 | a | 96.5 |

| 1 | 0 | 1 | a | 102.9 |

+------+------+------+----+-------+ Dataset B: +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | y |

+------+------+------+----+-------+

| 0 | 0 | 1 | b | 99 |

| 0 | 1 | 0 | b | 97 |

| 1 | 1 | 1 | b | 102 |

| 1 | 0 | 0 | b | 97.5 |

| 0 | 0 | 0 | b | 96.5 |

| 1 | 0 | 1 | b | 102.9 |

+------+------+------+----+-------+ Dataset C: +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | y |

+------+------+------+----+-------+

| 0 | 0 | 1 | c | 99 |

| 0 | 1 | 0 | c | 97 |

| 1 | 1 | 1 | c | 102 |

| 1 | 0 | 0 | c | 97.5 |

| 0 | 0 | 0 | c | 96.5 |

| 1 | 0 | 1 | c | 102.9 |

+------+------+------+----+-------+

There are now 3 variations of our dataset, where the only difference between them is ‘X5’ (that is, all other things are equal). All that’s left is simply taking the average of the dependent values of each of the datasets above. But there’s obviously a huge problem we face: The dependent values are for the original dataset, not the modified ones. For example, in the first row, ‘y’ = 99 is the test time for ‘X5’ = ‘a’, not ‘b’ or ‘c’. Thus, we need a way to find the dependent value for a row not present in our dataset. Fortunately, we have the tools to do exactly that.

A “Partial” Solution

In part 2 of this series, we built a slightly tuned Random Forest and guess what it can do? Estimate the test time of a vehicle, which is just what we need. We can simply use our model to predict the dependent values of datasets A, B, and C (the y-values shown here are most likely not consistent and make no sense at all. But that’s besides the point):

Dataset A: +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | y |

+------+------+------+----+-------+

| 0 | 0 | 1 | a | 100 |

| 0 | 1 | 0 | a | 98 |

| 1 | 1 | 1 | a | 105 |

| 1 | 0 | 0 | a | 97 |

| 0 | 0 | 0 | a | 95 |

| 1 | 0 | 1 | a | 102.9 |

+------+------+------+----+-------+ Dataset B: +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | y |

+------+------+------+----+-------+

| 0 | 0 | 1 | b | 90 |

| 0 | 1 | 0 | b | 99 |

| 1 | 1 | 1 | b | 105 |

| 1 | 0 | 0 | b | 97 |

| 0 | 0 | 0 | b | 96 |

| 1 | 0 | 1 | b | 102 |

+------+------+------+----+-------+ Dataset C: +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | y |

+------+------+------+----+-------+

| 0 | 0 | 1 | c | 90 |

| 0 | 1 | 0 | c | 105 |

| 1 | 1 | 1 | c | 100 |

| 1 | 0 | 0 | c | 97.1 |

| 0 | 0 | 0 | c | 96.5 |

| 1 | 0 | 1 | c | 102.1 |

+------+------+------+----+-------+

Now we can take the average of the dependent values to get a reasonably close estimate for the average test time of vehicles with a specific ‘X5’.

This method, like any other, has its own disadvantages: For starters, it’s only as good as our model. Therefore, if our model’s not very accurate, the results we get aren’t very reliable and might even be worse than taking the average of the original y-values.

Another issue that arises with the use of partial dependence is that not all categories are compatible. For instance, let’s go back to the example of ‘X5’ and how its value can correspond to the level of climate consciousness of the owner. And we’ll also add a made-up column, called ‘X1000’, that includes 26 categories (‘a’, ‘b’, …, ‘z’) , relating to the type of AC used in a car. Now, assume ‘a’ is a type of AC which is cheap but comes at the expense of being inefficient relative to the amount of gas used and therefore damaging to the climate. But on the other end of the alphabet, we have ‘z’, a costly but climate-smart option for customers feeling guilty about themselves and their carbon footprint.

In that case, if someone chooses ‘X5’ = ‘ag’ (which, remember, is the fuel-efficient type of tire), they’re most likely not going to choose ‘a’ as their AC because why order a cheeseburger combo with a diet Coke?

However, for every data point in our dataset, partial dependence pairs all classes in ‘X5’ with it, even if the new rows are erroneous. This issue can be illustrated as follows:

Original: +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | X1000 |

+------+------+------+----+-------+

| 0 | 0 | 1 | ag | y |

| 1 | 1 | 1 | ag | z |

| 1 | 0 | 0 | aa | a |

| 0 | 0 | 0 | aa | b |

+------+------+------+----+-------+ Partial dependence for 'X5' = 'aa' +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | X1000 |

+------+------+------+----+-------+

| 0 | 0 | 1 | aa | y |

| 1 | 1 | 1 | aa | z |

| 1 | 0 | 0 | aa | a |

| 0 | 0 | 0 | aa | b |

+------+------+------+----+-------+ Partial dependence for 'X5' = 'ag' +------+------+------+----+-------+

| X314 | X119 | X127 | X5 | X1000 |

+------+------+------+----+-------+

| 0 | 0 | 1 | ag | y |

| 1 | 1 | 1 | ag | z |

| 1 | 0 | 0 | ag | a |

| 0 | 0 | 0 | ag | b |

+------+------+------+----+-------+

Please note that, in the first example, climate-smart ‘X5’ options (‘ag’) go with climate-smart ‘X1000’ options (‘y’ and ‘z’) and carbon-emitting ‘X5’ options (‘aa’) go with carbon-emitting ‘X1000’ options (‘a’ and ‘b’), as is expected from customers with different views on climate-related issues. But in the second and third examples, climate-smart ‘X5’ options go with carbon-emitting ‘X1000’ options and carbon-emitting ‘X5’ options go with climate-smart ‘X1000’ options, which is the opposite of what is expected from (both green and non-green) customers.

This could potentially be a problem because:

Our model can’t make accurate predictions for rows which aren’t from the same distribution as our training set or Even if our model is nearly perfect when it comes to generalization, some rows might not even make theoretical sense and hence, we shouldn’t include them when doing partial dependence. Such an example would be a car made in the 1800s equipped with a turbocharger, etc.

And lastly, perhaps partial dependence’s biggest problem in real-world machine learning is the amount of time it could take. For a dataset with a couple hundred thousand rows and a few hundred features with high cardinalities, performing partial dependence would be infeasible, especially if our model takes long for inference. A typical workaround is using only a small subset of the rows we’re given or doing partial dependence only for the features we really care about (the former shouldn’t change the results drastically as long as our mini-dataset is representative of the actual one).

Despite all its imperfections, however, partial dependence is still an extremely powerful tool which enables you to gain insights into your dataset which traditional EDA methods simply can’t and these insights could then be turned into better business decisions which could maximize profit and impact. There are numerous such examples, and here we’ll go through one together.

Bulldozer Auction

A while ago, I wrote a series about a Kaggle competition where the goal was to successfully predict the auction price of a bulldozer given various features such as its size, the ID of the auctioneer, and a lot more, with most being technical terms not everyone (including me) understands.

In one of the later articles, we realized the column containing the year the bulldozers were made (‘YearMade’) is very important to our model’s performance, which is probably no surprise to bulldozer professionals. However, we didn’t know how ‘YearMade’ affects the sale prices of the heavy equipment: Do they increase monotonically? Maybe plotting ‘YearMade’ against the dependent variable be all over the place? Or perhaps as ‘YearMade’ increases the sale price decreases? Logically, the first scenario should be the case but we need a way to prove that. You can probably see where this is going…

The first thing that jumps to mind is taking the average of all sale prices for all years and seeing what that gives us. If our hypothesis is indeed true, we should get an increasing line/curve, right? But it turns out if we do that, there’d be a dip around ‘YearMade’ = 2000, which would mean bulldozers made in the early 1990s sell for more than the ones made in the late 1990s, contrary to our speculation.

Picture from the ‘Introduction to Machine Learning for Coders’ course

The organizers of the auctions might then decide since bulldozers made in the late 1990s sell for less than the ones made in the early 1990s, the former isn’t worth their time so they’ll stop auctioning it.

By now, you should be suspicious of conclusions drawn by taking averages and resort to using partial dependence instead. If you do so for this bulldozer dataset, the result will be very different from the above graph and confirm our initial hypothesis.

Picture from the ‘Introduction to Machine Learning for Coders’ course

Please note that the yellowish line is what we should be looking at (just ignore the blue ones for now) and the y-axis is the log of the dependent variable. As we can see, ‘YearMade’ and the y-axis have an almost linear relationship which means in reality, sale prices grow exponentially with respect to the year our bulldozers were made.

There are a few possible explanations for the inconstistency between the two plots: Recession, difference between the quality of the bulldozers made during various time intervals, etc. Whichever the case, however, we can be certain if two bulldozers are identical in all ways but the year they were made, the older one will have a lower sale price. But if the auction organizers make their decisions based on the initial plot, they wouldn’t know that and would lose big sums of money by not auctioning heavy equipment made in certain years.

Conclusion

In this part, we saw how partial dependence works behind the scenes and frankly, it’s not very complicated: In order to calculate the true average of the dependent value for a category c (or a continuous value) in a feature F, it sets F to c for all rows in our dataset (or a subset of it, if it’s too big) and uses a predictive model to figure the dependent values for these modified rows. It then takes the averages of the predictions made by our model and that’s basically what we’re looking for (there are a few other steps involved but that’s the general idea).

Admittedly, partial dependence does come with its own particular challenges, the major ones being the issue of some categories not being compatible with each other (a car made in the 1800s with autopilot) and the fact that it can be time- and resource- consuming sometimes. Nevertheless, it’s still a great tool to have in your toolbox and can aid you in making business decisions and drawing meaningful insights from your dataset.

In the next part, we’ll look at how to implement partial dependence in Python using a powerful library called PDPBox, which comes with beautiful visualizations and several other useful related tools.

To be continued…

Please, if you have any questions or feedback at all, feel welcome to post them in the comments below and as always, thank you for reading!

Part 1: https://medium.com/python-in-plain-english/mercedes-benz-greener-manufacturing-part-1-basic-data-pre-processing-a32d17803064

Part 2: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-3ddff72d0187

Part 3: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-1ca6b030bf58

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-b5220f479a44

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-a004659e02c4

Part 4: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-82dd27e53757

Part 5: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-e31198ecafae

Part 5 (final part): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-ecbd2714d952

Twitter: https://twitter.com/bobmcdear

GitHub: https://github.com/bobmcdear