Tackling Kaggle’s Mercedes-Benz Greener Manufacturing Competition with Python
By: Borna Ahmadzadeh
Tags: Artificial Intelligence, Machine Learning, Data Visualization, Data Science, AI

Photo by MARCIN CZERNIAWSKI on Unsplash

Introduction

In this part, we’ll perform exploratory data analysis (EDA) on our data, which is a crucial part of most machine learning problems. Although we might not end up increasing our score, we will draw invaluable insights from our data, which is often one of the primary objectives of real-world machine learning. We are going to use some of the traditional EDA techniques, but we’ll also touch on a few underused ones as well. You can find the notebook for this tutorial here.

Without further ado, let’s get coding (in Colab)!

PDPbox

Having gone thoroughly over what partial dependence is and how it works, we can now jump straight to implementing it in Python.

Although writing it from scratch isn’t terribly tough, it’s still a tedious task so we’ll be using a library called PDPbox which has all the functionalities we’ll need and more (Scikit-learn too has partial dependence, but it’s not as easily customizable as PDPbox, in my opinion).

Colab doesn’t come pre-installed with this library, so we’ll have to do it ourselves:

!pip install pdpbox

And import it:

from pdpbox import pdp

The rest is awfully easy, requiring no more than a few lines of code:

# We need to pass in a model trained # on our dataset to PDPbox rf.fit(X, y) # Calculate partial dependence pdp_isolate = pdp.pdp_isolate(rf, X, list(X), 'X127') # Plot it fig, axs = pdp.pdp_plot(pdp_isolate, 'X127', center=False) # Only display the area around dependent variable = 100 axs['pdp_ax'].set_ylim([95, 105]) Output:

pdp_isolate takes in a number of arguments: A Scikit-learn-compatible predictive model, the dataset that’s going to be used for calculating partial dependence (one can pass in a small portion of a dataset if the original one is too big), a list of names of features the model was trained on, and finally, the actual feature we’re investigating (in case of a one-hot encoded feature, we can pass in a list of the names of the resulting binary columns). There also around 10 optional arguments, most of them relating to how the results are going to be plotted.

pdp_plot requires only 2 arguments: A PDPIsolate object (the output we get from pdp_isolate), plus the name of the feature we’re working on and once again, there are a bunch of optional parameters we could pass in. Setting ‘center’ to false means the y-axis represents the actual dependent values and not a scaled version, or, in case of a binary classification task, the average of the predicted probabilities for each row. I strongly advise you take a look at the documentation, as it showcases various uses of these parameters, and play around with them on the dataset of your choice.

Why did we choose ‘X127’? Well, if we take a look at the average value for ‘X127’ = 0 and 1 (no partial dependence), we’ll get 107 and 94 respectively, which is a pretty big difference, right?

However, using partial dependence, we can clearly observe that there’s virtually no difference between the two categories. The exact numerical values calculated by partial dependence can be accessed through pdp_isolate:

pdp_isolate.pdp

Output: array([101.36611244, 100.62125509])

Less than 1! Again, these values aren’t perfect, but we can be very certain the effects of ‘X127’ (and only ‘X127’) on test times of vehicles is almost negligible, contrary to what we saw by taking a vanilla mean. There are many possible explanations, some of which we went through in the previous articles.

Going through the other binary features, the results we get by partial dependence don’t differ significantly from what we’ve already seen, so we won’t go through them.

PDPbox can also be used for categorical features (‘X5’), using the exact same lines of code. But there’s a nuisance we need to deal with first: The x-ticks are going to be ints and not the actual names of the categories, which can be annoying since we’d need to see what class each integer corresponds to every time we’d like to make sense of our original dataset. An easy way to solve this issue would be to simply change the Axes’ x-ticks to the classes they’re referring to. For instance, if ‘u’ in the original dataset equates 27 in the cleaned dataset, we’d need to replace x = 27 in the PDP (partial dependence plot) with ‘u’.

To do so, we first need to know what number is what category, which is quite straightforward: We can iterate through ‘train’ and ‘train_raw’, see what ‘X5’ is in the two DataFrames, and store both values in a dictionary, with the value in ‘train’ being the key and the one in ‘train_raw’ being the value. This is a sloppy trick we can use to achieve our goal and there are almost certainly better methods. However, this just came to mind and is very simple to code (had we shuffled the datasets, we wouldn’t have been able to do this so you should be very cautious when doing something similar):

from collections import defaultdict raw_x5, x5 = train_raw['X5'], train['X5'] int2cat = defaultdict(lambda: 'X') for i in range(len(train_raw)): int2cat[x5[i]] = raw_x5[i]

A defaultdict works quite similar to a normal dictionary in Python, but instead of raising a KeyError, it returns a default value set by us.

Now, we can simply use ‘int2cat’ to modify the x-ticks:

feat = 'X5' # cust_grid_point tells pdp to do partial dependence # for every value in x5_classes, because pdp thinks # X5 is a numerical column and so uses a slightly # different method, which we don't want. x5_classes = x5.unique() pdp_isolate = pdp.pdp_isolate(rf, X, list(X), feat, cust_grid_points=x5_classes) fig, axs = pdp.pdp_plot(pdp_isolate, feat, center=False) axs['pdp_ax'].set_ylim([95, 105]) # The labels for the x-axis x_ticks = [int2cat[i] for i in range(1, x5.max()+1)] # Before we set the labels to text, # we must specify we'd like to see all # x-ticks axs['pdp_ax'].set_xticks(range(1, x5.max()+1)) # Int x-ticks to categories axs['pdp_ax'].set_xticklabels(x_ticks)



Output:

Compared to the stats we found in the previous couple of articles, there’s a few dissimilarities that stand out: For starters, ‘ag’ had the highest mean amongst the common categories (ones which appeared at least 20 times), with ’n’, ‘ab’, and ‘c’ being close contenders. With partial dependence, however, that’s not true anymore: As we can see, ’n’ has the highest mean, and ‘o’ and ‘m’ take 2nd and 3rd respectively. However, ‘ag’ still has one of the higher averages, but just not as big as it was before. The evidence in favour of performing PD just keeps piling up.

Two-way Partial Dependence

So far, we’ve only looked at one-way partial dependence plots, meaning we know how the dependent variable reacts to changes in just one column. But the exact same concept can be used to find out how the dependent variable changes with respect to n features (in practice, n is rarely more than 2 due to the perception problems that arise with more dimensions). For instance, say we’d like to see how vehicles’ test times relate to ‘X315’ and ‘X127’. Previously, for either feature, we’d create 2 datasets, one where that feature was always set to 0 and one where it was always 1. But now, we’d have 4 datasets, where our two features’ categories are paired in all possible combinations:

Both features are always 0. Both features are always 1. ‘X315’ is always 1 and ‘X127’ is always 0. ‘X315’ is always 0 and ‘X127’ is always 1.

Or, if we wanted to look into the interaction between ‘X5’ and ‘X314’, we’d have 58 (there are 29 classes in ‘X5’) datasets, each one corresponding to a pair of an ‘X5’ class and either 0 or 1. The same dis/advantages that one-way partial dependence has are also present here, so we won’t go take the time to go through them.

Since the idea behind this is pretty much the same as one-way PD, the syntax will also look very familiar, with only a few small differences:

cols = list(X) # Go through all possible pairs of feature for i in range(len(cols)-1): for j in range(i+1, len(cols)): feats = [cols[i], cols[j]] # Just do binary features if 'X5' in feats: continue pdp_interact = pdp.pdp_interact(rf, X, cols, feats, n_jobs=-1) fig, axes = pdp.pdp_interact_plot(pdp_interact, feats, plot_type='grid')

Output: (Too big)

As you can see, the gist is very similar, and we have only changed some of the names and added a plot_type=’grid’, which tells pdp to shape the plot like a rectangular grid, with each cell representing the average dependent value for a combination of our two features (for numerical columns, this should be set to ‘contour’, which is the default value).

Were this series centred around EDA, we would/should have devoted a great deal of our time to interpreting these results (in addition to everything else we’ve seen so far), come up with various explanations relating to different anomalies & patterns, and tried to dis/prove these hypotheses. However, the goal of this series is chiefly model building and feature selection/engineering, which is common for most Kaggle competitions. The main reason I decided to add an EDA section was to provide a gentle introduction to various fundamental methods and techniques such as PD, so you would have the basic understanding needed to enter the vast world of data analysis, which can also sometimes aid us in creating more accurate models.

There is a lot regarding partial dependence we didn’t get to go through, and a great starting point would be playing around with PDPbox. I honestly learned so much by just modifying tiny bits of its source code and trying out a wide variety of arguments, so I definitely suggest you do the same as well.

Next, you could try writing the basics of PD from scratch, and test it out against PDPbox. There are a few bits and pieces here and there we didn’t discuss, but nothing too major or mysterious.

Lastly, I recommend checking out accumuluated local effects (ALE) plots (less mathy version here), a modified version of partial dependence that takes into account the dependency between features, a key weakness of PD we went through in depth. There’s also the individual conditional expectation (ICE) plot, which is partial dependence but instead of plotting merely the average dependent value for each class in a feature, we’re plotting the dependent value of every individual row in our dataset for every single category in a feature. ICE can be quite handy at times, especially when taking the average can be misleading and deceiving (the ICE paper gives numerous such examples).

Oh, and most importantly, please practice, practice, and practice!

Conclusion

In this part, we explored PDPbox, a neat little Python library that provides many partial dependence-related tools & functionalities and is accompanied by very elegant and beautiful plots and charts. We saw some of its various uses, such as simple one- and two-way PDPs, but there’s still a lot of it left uncovered and I encourage you to spend some time experimenting with it.

In the next part, we’ll get back to model building and use a surprising fact to create a couple different models, which are both simpler than our Random Forest yet score as good (or maybe even better)!

Please, if you have any questions or feedback at all, feel welcome to post them in the comments below and as always, thank you for reading!

Part 1: https://medium.com/python-in-plain-english/mercedes-benz-greener-manufacturing-part-1-basic-data-pre-processing-a32d17803064

Part 2: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-3ddff72d0187

Part 3: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-1ca6b030bf58

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-b5220f479a44

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-7b203e886f8d

Part 4: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-82dd27e53757

Part 5: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-e31198ecafae

Part 5 (final part): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-ecbd2714d952

Twitter: https://twitter.com/bobmcdear

GitHub: https://github.com/bobmcdear