How fast can a Mercedes Benz car be tested and released? — A Kaggle Competition
By: Jeru Luke
Tags: Random Forest, Kaggle, Competition, Data Science, Machine Learning

Each row has a unique ID, a collection of categorical variables(X0 — X8) and binary/identity variables(X10 — X378) as well as the output ‘y’.

Upon analysis, we came across some columns:

that showed non-variance (same values throughout the column) that were highly skewed (few values occurring most of the time (> 95%)) that were exact duplicates that were linearly related to certain other columns (positively or negatively)

We did not come across any columns with missing values. Hence there was no need for imputation.

Data Preprocessing

The main idea behind preprocessing is to counter the aspects we came across in the data exploration phase. So what did we do? Before asking that let us ask ‘Why do we do it?’

Reduces dimensionality of the problem in hand. Reduces the possibility of over-fitting (the model tends to learn all available features that are redundant, duplicate or unnecessary ). Lesser number of features mean simpler models, lesser memory, reduced training time and faster results.

Now we get back to the question ‘What did we do?’

We removed columns showing no variability, meaning columns that contained only one value throughout. We also removed columns whose values occurred in more than 95% in the entire column. We scanned the data set for duplicate occurrences and retained only one column in each case. Columns that showed high correlation were also removed to reduce over-fitting.

Feature Engineering

A feature is a useful piece of information for prediction. Hence, better features lead to better results.

Actually, features(numerical ones) available in the raw dataset can be used for prediction right away. You however, won’t achieve the desired level of accuracy needed. What if we could create more features and transform them into something the algorithm can understand? By doing so, we are making things simpler for the model to learn by highlighting the underlying problem.

Impact Encoding

Since the data contained categorical variables, our first job was to transform them into a form understandable by the algorithm. We performed impact encoding, where each categorical variable is replaced by the ‘mean’ of the occurrence of that variable in that column. Hence a variable with more ‘mean’ value will have a higher numerical value.

There are many ways of converting categorical variables to numerical ones. This article provides more insights to do so.

Creating new features in categorical variables

We were able to create new features having categorical variables. Here are the first five rows of the train set:

First five rows of the dataframe

Columns X0, X1 and X2 had something in common. They had some variables having two letters (ex: az, ab, bc, etc.) So we thought of creating a new column called X0_club, X1_club and X2_club. Each of these columns would have variables after clubbing those staring with the same letter. For instance: [aa, ab, ac, ……, az] would be clubbed as [a] and [ba, bb, bc, ……, bz] would be clubbed as [b] and so on. The following image shows the dataframe containing the newly created columns:

Notice the three additional columns in the end

New features can be created for numerical variables as well. But since there were plenty of columns already we didn’t create more (big mistake).

Note: A keypoint to remember is that the steps performed on the training dataset MUST be performed on the test dataset as well. So while doing data preprocessing and feature engineering make sure these steps are done for the test data simultaneously.

Building and training the model

Since our problem is one involving regression, we decided to use Random Forest , which is one among the many ensemble methods available. There are many other algorithms to choose from as well.

We used the RandomForestRegressor function available in the sklearn library to perform regression. The different parameters of this function (number of trees, depth of trees, minimum sample to split and number of features) were decided by running a for loop and was validated using cross validation.

Another approach

On visualizing the output variable against the number of observations, we saw that it was highly skewed having a number of outliers as well. We decided to transform this using Box Cox transformation. It transforms a skewed distribution to a normal distribution. We made use of the corresponding function available in scipy for this purpose. However, training the model on the transformed result did not yield an improvement in the validation accuracy

What we could have done?

Here are some techniques/approaches we could have used but did not:

Create new features using those variables that are of high importance from the existing data. The importance of a variable can be found out by observing which variable is most dependent on in determining the outcome. Cross validation can help you out in this case. We thought of performing basic logical operations (AND, XOR and OR), among the binary variables in order to increase the number of features. But since we had a system with 8GB RAM and given the large number of binary variables (~300), it hanged several times :(. In case this step works for you, care must be taken to ensure duplicate and correlated columns are removed. Use AdaBoost algorithm. Though it is sometimes sensitive to noise and outliers, in most cases it outperforms the other ensemble methods. Care must be taken though, as there is a high tendency to overfit. We could have also used the method of regularization in cohesion with regression. The Lasso function in the sklearn library can help you with that.

Ending Note

So now you want the code huh? Well all the relevant code is available on THIS GitHub repository.

Do you have a better way to solve this problem? Or do you any other comments that you would like to share? Please feel free to comment. By the way, this is my first post on Medium.

I am currently participating in another Kaggle competition — Instacart Market Basket Analysis. I will surely write about my experience once I am done.