Tackling Kaggle’s Mercedes-Benz Greener Manufacturing Competition with Python
By: Borna Ahmadzadeh
Tags: Artificial Intelligence, Regression, Machine Learning, Data Science, AI

Introduction

In this part (which is the last article in this series), we’ll build our final model, a reasonably accurate ensemble that will place in the top 10% of the Kaggle private leaderboard. You can find the notebook for this tutorial here.

Without further ado, let’s get coding (in Colab)!

Data Cleaning

Initially, my plan was to use a very small part of the dataset to create the final model, considering the fact the our Random Forest reckoned most of the original features useless. However, after giving it more thought, I came to the conclusion that it’s possible for non-tree-based models to derive meaning from the (according to our Random Forest) redundant features. Thus, for some models, we’ll use the entire dataset (minus parts that are really dispensable).

But unfortunately, up until now, every single time we came up with an idea, we made permanent changes to our cleaned dataset, so we only have two datasets: The raw dataset and the one we did feature selection on (that now has only one feature). Therefore, we must repeat all the steps we did in the first part of this series, as well as a couple new things (this is my fault; we should have saved the cleaned dataset and used a copy thereafter. Lesson learned).

First and foremost, we need to label encode our categorical features (already did this in the first article):

df = train_raw.drop(dep_var, axis=1).append(test_raw) df[cat_names] = df[cat_names].astype('category') for i in cat_names: df[i] = df[i].cat.codes

We are also going to remove constant and duplicate columns. The reason we didn’t do so previously is that our Random Forest obviously would’ve placed low importance on the former and so during feature selection, we’d have gotten rid of them (and some other columns which weren’t necessarily constant). After that, we took care of corresponding features by using Spearman’s rank correlation coefficient and manually inspecting suspicious columns. However, we could simply go through all the features and drop the constant/duplicate ones. Albeit less efficient, this method has the benefit of us not having to drop a feature just because our Random Forest says so.

# Loop through all the features for col in list(df): # Disregard the categorical variables if col in cat_names: continue # The first if says if there are fewer than 40 ones # in a feature (meaning almost the entire column is zeros), # then treat it as a constant column # But it's possible a column only has ones instead of zeros, # and ^1 turns zeros to ones and ones to zeros. So the second # if means if there are fewer than 40 zeros in a column, # treat it as a constant one. if df[col].sum()<40 or (df[col]^1).sum()<40: df.drop(col, axis=1, inplace=True) to_drop = set() # Go through every pair of features, # starting from column[1] since column[0] # is the dependent variable for i in range(1, df.shape[1]-1): for j in range(i+1, df.shape[1]): # We only care about binary features if list(df)[i] in cat_names or list(df)[j] in cat_names: continue col1 = df.iloc[:, i] col2 = df.iloc[:, j] # The first if says if the two features are # rarely the same (which means when one is zero the other # is one and vice versa), then drop one. The second one means # if they're often the same, then drop one (~ turns falses to # trues and trues to falses). if ((col1 == col2).sum() <= 40) or (~(col1 == col2)).sum() <= 40: to_drop.add(list(df)[i]) df.drop(list(to_drop), axis=1, inplace=True)

And let’s make sure everything looks sensible:

# 'X118' and 'X119' were duplicates, # so only one of them should've remained # ^ is XOR (exclusive or) in Python # It returns a 1 if and only if one of # the inputs is true and the other one if false ('X118' in df.columns) ^ ('X119' in df.columns) display_df(df)

Output: True (Too big)

We now have a fairly “clean” version of our dataset, with no duplicates or constant features. There are other things we could’ve done (with the most major one being examining and dealing with outliers) but, as it turns out, this is enough to earn us a pretty good spot on Kaggle’s leaderboard.

One-hot Encoding

Random Forests have been our go-to machine learning model in this series due to the fact that they require little to no pre-processing, are easy and fast to train & validate, are extremely robust to outliers, and a lot more. Despite that, it’d be very ignorant of us not to try other algorithms as well. Ergo, one of our primary aims shall be to test out other models, decide whether they could be of use, and if so, ensemble them for even better performance.

Decision trees accept categorical features as they are, meaning no special encoding (other than turning strings to ints) is needed. However, we’ll be using linear models as well and hence, we need to normalize our data and one-hot encode our categorical features. The former isn’t necessary in our case because we don’t have any continuous variables but we do have categorical features, so we must turn them into dummy variables. To do so, we could either use pandas’ .get_dummies() or Scikit-learn’s OneHotEncoder(). A big advantage the latter has is that it remembers the categories it has seen, so it can easily transform the test as well without us needing to do anything extra (this is very helpful when there are classes in the training set that aren’t in the test set or vice versa, which is often the case) but since we’re doing everything globally, that’s not as important and for just one DataFrame, pandas is quite a bit easier to use (not to mention the fact that we’d only like to encode certain categories), so that’s what we’ll stick to.

def one_hot_encode(df, drop_first=False): df_one = df.copy() df_one.reset_index(drop=True, inplace=True) # Go through all the categorical features for cat_name in cat_names: # One-hot encoded version of the current column one_hot = pd.get_dummies(df_one[cat_name], prefix=cat_name, drop_first=drop_first).reset_index(drop=True) # Drop the original column and replace it with the dummy variables df_one.drop(cat_name, axis=1, inplace=True) df_one = df_one.join(one_hot) return df_one

Let’s check it out:

df_one = one_hot_encode(df) display_df(df_one)

Output: (Too big)

As can be seen, the original categorical columns have been replaced by dummy variables but the rest of the dataset is left untouched.

‘df’ and ‘df_one’ contain both the training and test set, so let’s split them into two separate DataFrames:

X = pd.DataFrame(df[:len(train_raw)]) train = X.copy() train[dep_var] = y test = pd.DataFrame(df[len(train_raw):]) X_one = pd.DataFrame(df_one[:len(train_raw)]) train_one = X_one.copy() train_one[dep_var] = y test_one = pd.DataFrame(df_one[len(train_raw):])

We’re now ready to move onto modeling.

Orthogonal Matching Pursuit

While trying to increase the cross-validation score, I stumbled upon a roadblock: The usual machine learning models I was familiar with simply couldn’t do the job. XGBoost, LightGBM, Extra Trees, Lasso, Ridge, nothing. Ensembling seemed to help, but not enough as to make me feel confident enough to submit the final predictions to Kaggle. Frustrated, I headed to Scikit-learn’s linear models’ documentation and decided to look through all the available models and try out the ones I hadn’t before and thought might give good results. That’s where I encountered orthogonal matching pursuit, a linear model with a strict number of nonzero coefficients. Given the fact that there was only one hyperparameter to be tuned, I was able to quickly assess it using cross-validation and, spoiler alert, it did really well.

Let’s import it:

from sklearn.linear_model import OrthogonalMatchingPursuit

And set a baseline (by default, n_nonzero_coefs is set to 0.1*the number of features):

print_cross_val(OrthogonalMatchingPursuit, train_one, k=1000, n_nonzero_coefs=40)

Output:

Cross validation scores: [0.556 0.520 0.632 0.489 ...]

Cross validation average score: 0.561

Cross validation standard deviation: 0.054

Not quite as good as our best score, but that’s merely because we haven’t tuned n_nonzero_coefs yet. Since we’ve already done lots of tuning, I won’t put the code here but you should keep reducing it by 5 until the R2 starts to get worse. 25 and 20 gave very similar results and consistently outperformed other values but 20 had a slight edge when it came to the standard deviation, so that’s what we’ll use from now on:

print_cross_val(OrthogonalMatchingPursuit, train_one, k=1000, n_nonzero_coefs=20)

Output: Cross validation scores: [0.468 0.582 0.610 0.538 ...]

Cross validation average score: 0.574

Cross validation standard deviation: 0.05346815398190671

Boom! As you can see, our score is significantly higher than anything we’ve come up with so far and all it required was a few minutes of data cleaning and the right linear model. This indicates that we are definitely on the right track and with a bit more work, can increase our score even further!

P.S: After spending quite some time reading the literature and playing around with it, I still can’t quite comprehend how orthogonal matching pursuit actually works so it’s not my place to try to explain it. But the Scikit-learn user guide does a good job of outlining the basics so I advise you read it to understand what’s going on under the hood.

Conclusion

In this part, we did some data cleaning and removed duplicate & constant features by brute-forcing and using a simple for loop instead of training a Random Forest, calculating feature importances, yada yada. But since we’ll be using linear models, we had to one-hot encode our categorical variables, which can be done easily by using pd.get_dummies(). This method is suitable for when you don’t need to worry about one-hot encoding future DataFrames with possibly different classes but if that’s the case, using pandas can be a headache and Scikit-Learn’s OneHotEncoder() is preferred. Finally, we got introduced to orthogonal matching pursuit, a linear model with a limited number of nonzero coefficients (i.e. it only uses a small subset of features) that fits our dataset nicely.

In the next part, we’ll use the model we created today for feature selection, which will help other models quite a bit.

To be continued…

Please, if you have any questions or feedback at all, feel welcome to post them in the comments below and as always, thank you for reading!

Part 1: https://medium.com/python-in-plain-english/mercedes-benz-greener-manufacturing-part-1-basic-data-pre-processing-a32d17803064

Part 2: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-3ddff72d0187

Part 3: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-1ca6b030bf58

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-b5220f479a44

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-7b203e886f8d

Part 3 (continued): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-a004659e02c4

Part 4: https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-82dd27e53757

Part 5 (final part): https://medium.com/python-in-plain-english/tackling-kaggles-mercedes-benz-greener-manufacturing-competition-with-python-ecbd2714d952

Twitter: https://twitter.com/bobmcdear

GitHub: https://github.com/bobmcdear