Mercedes-Benz Greener Manufacturing Problem
Tags: Ensemble, Regression, Kaggle, Machine Learning

B usiness Problem:

Since the first automobile, the Benz Patent Motor Car in 1886, Mercedes-Benz has stood for important automotive innovations. These include, for example, the passenger safety cell with crumple zone, the airbag and intelligent assistance systems. Mercedes-Benz applies for nearly 2000 patents per year, making the brand the European leader among premium car makers. Daimler’s Mercedes-Benz cars are leaders in the premium car industry. With a huge selection of features and options, customers can choose the customized Mercedes-Benz of their dreams. .

To ensure the safety and reliability of each and every unique car configuration before they hit the road, Daimler’s engineers have developed a robust testing system. But, optimizing the speed of their testing system for so many possible feature combinations is complex and time-consuming without a powerful algorithmic approach. As one of the world’s biggest manufacturers of premium cars, safety and efficiency are paramount on Daimler’s production lines.

In this competition, Daimler is challenging to tackle the curse of dimensionality and reduce the time that cars spend on the test bench. Competitors will work with a dataset representing different permutations of Mercedes-Benz car features to predict the time it takes to pass testing. Winning algorithms will contribute to speedier testing, resulting in lower carbon dioxide emissions without reducing Daimler’s standards.

Project Overview:

In automobiles industries there will be a Testing department which tests the safety and reliability of cars that come out of the production. These tests play a crucial part in manufacturing process. Mercedes Benz industry manufactures huge number of cars every day and they will be sent to testing as final stage in production. Every possible feature combination of vehicles must undergo through the test bench to ensure that each vehicle is robust for daily use. More time spent on the test bench will result in more CO2 emissions and increase of costs to company.

Business Use Case:

In this we are trying to optimize the time spent on testing for every vehicle that comes to test bench. Using this optimization will result in speedier testing and decrease in CO2 emissions due to testing procedure for Mercedes Benz.

Mapping the problem to a Machine Learning model:

The Mercedes Greener dataset is comprised of 369 features which are (categorical and numeric)are anonymized. So conventional set of rules don’t work, we can use ML algorithms that has ability to apply complex calculations to large number of features which helps in optimizing the time spent on test bench without reducing Daimler standards.

Here we are trying to predict a continuous target variable which makes this a Regression Problem.

This problem can be solved using Classical ML techniques where we are trying to predict Time taken by a vehicle on a test bench which is a continuous target variable based on vehicle features which are independent. This type of problem is Supervised Learning where model can learn from labelled data.

Performance Metrics:

Mercedes Benz industry suggested to use R2 (Coefficient of Determination) to evaluate the model performance.

Why R² (Coefficient of Determination):

R² is given by the formula 1-(SSE/TSS).

SS regression= Sum of Squared errors (sum of the squares of the differences between the observed values and the expected values)

SS total= Total sum of squares (Total sum of squares, the sum of the squares of the differences between the observed values and the mean of the observed values.)

The R2 is usually between 0 and 1. It is the simply the fraction of response captured by the model. If R2 values is 1 then our model explains all the variability of the target data around its mean. R2 value can be more easily interpretable and measures goodness of fit in capturing variance of the data. RMSE is measure of deviation of model predictions from actual values. RMSE doesn't have any range. If we look at RMSE only it doesn't add how much variability our model has. R2 is most common measure used in model fitting procedure for regression and forecasting.

Data Source:

Mercedes Benz industry posed the problem on Kaggle and data is provided in two CSV files (train, test). Data can be downloaded using below link.

(https://www.kaggle.com/c/mercedes-benz-greener-manufacturing/data)

Existing Approaches:

In this blog he used the dataset to explain how stacking models perform. He dropped the ID column and columns which have zero variance from the dataset and used Label Encoder to encode categorical features. He trained XG boost model and Random forest model on the data. These two models are used as baseline for future models. He created a stacking model using this data which gave him best performance out of all models.

Since this dataset challenges the curse of dimensionality. He used PCA, TSVD, ICA dimensionality reduction techniques to create new features and add them to our original dataset. He trained XG boost, Random forest models on the new dataset. In this he focused on showing how stacking models perform over ensemble models.

2. https://medium.com/analytics-vidhya/mercedes-benz-greener-manufacturing-74a932ae0693

In this he removed the columns which has low variance. used a threshold of 150 seconds on target variable. He used Label Encoding for encoding Categorical features. He used PCA as dimensionality reduction to create new features. He added the 2-way and 3-way interaction features based on first place winner solution.

He used linear regression as the baseline model and went to ensemble models. He also trained a stacking model on the dataset which gave him best results of all models.

3. https://medium.com/@williamkoehrsen/capstone-project-mercedes-benz-greener-manufacturing-competition-4798153e2476

In this blog he mentioned a model known as Weighted average model which was discussed in Kaggle. Since most top performing models are of stacked and XG boost model. He tried to find which models complemented each other used these to create weighted average model.

He dropped the ID column, outliers and transformed the data using PCA. He used XG boost model and Stacked model where output of Linear regression with Elastic net goes to Extra trees regressor and predicts output based on output from linear regression. The final model is average of the Stacked model and XG boost model.

4. https://medium.com/analytics-vidhya/mercedes-benz-greener-manufacturing-kaggle-competition-1c25c89e012

In this blog he used Label, Frequency, Mean encoding for categorical variables. He transformed the dataset using PCA and used Lasso model to find which type of encoding gives best results.

Using encoding selected above, he removed low variance features, features which have correlation between each other, duplicate features. He used Pearson correlation to find which variables are highly correlated to target. He created 2 way and 3 way interaction features out of those and added to the dataset.

Using Select K best on the above dataset he reduced the dimensionality of the dataset. He trained Lasso, RF, Extra trees, XG boost and Stacked model. Stacked model gave him the best result.

Improvements to existing approaches:

In all the above approaches the models which performed well were the model with interaction features, and dimensionality reduction features. Since the highest score is 0.5550 there are certain things in solutions which might not be very helpful in real world scenario. The following steps I have tried to on existing approaches:

I have tried to reduce Overfitting since most models from above approaches seems to be overfitting. I didn’t remove low variance features as some blogs removed low variance features which might be cause of overfitting. Some kagglers created models without removing outliers. But I have used a threshold on target variable and removed all points which are greater than threshold. Some kagglers got good scores without using categorical variables. I have used the categorical variables in my dataset. I have used PCA and TSVD as dimensionality reduction techniques. I have implemented the paper(Thresholding of singluar values (https://arxiv.org/pdf/1305.5870.pdf)) to select optimum rank for TSVD.

Exploratory Data Analysis:

Data overview:

Mercedes Benz industry posed the problem on Kaggle and data is provided in two CSV files (train.csv, test.csv) The dataset consists of 377 features (x0, x1, x2,…., x386) which are anonymized and each represent a custom feature in Mercedes car like 4WD, suspension or heads up display which together define a car model. The dataset has two types of features. Categorical features, Numerical Binary features. The ID column in data represent the unique configuration vehicle Id. The target variable y represents the time a vehicle takes to pass testing in seconds. There are 8 Categorical features and 369 Numerical features in the train.csv

Eda is divided into three parts:

Target Variable (Analysis and Visualization) Categorical Variables (Analysis and Visualization) Numerical Variables (Analysis and Visualization)

Target Variable:

Analysis:

This code is used to generate descriptive statistics that summarize the central tendency, dispersion.

We can see for target variable ‘y’ mean is 100.669 and 75% value is 109. So there are 25% values in dataset which have ‘y’ greater than 109.Maximum of value of ‘y’ in dataset is 265.

Visualization:

Scatter plot:

Above is scatter plot made between series(with length of dataset) and sorted values of Y from train data frame. We can see that most points lies between 80 and 140. There are some points which are far away and these are outliers.

PDF, CDF plots:

CDF plot of y

From above PDF plot we can see two peaked curves and distribution is similar to Gaussian. We can see that the values are centered around almost 100 and most values lie between 80 and 120. Since the curve is highly skewed to right shows that some values are above 200. From the CDF plot we can see that atleast 95% values lie below 150.

Categorical Variables:

X0:

Bar plot:

The above plot is a bar plot which represents how many values each class has in our dataset. We can see that there are 3 classes with lowest value than others.

Scatter plot:

The above plot is a scatter plot between Categories of X0 and distribution of y for each category. We can see that there is an point in class y of X0 which is far away from other points. This can be considered as an outlier. There is only one point in classes of ac,g,ab.

X1:

Bar plot:

The above plot is a bar plot which represents how many values each class of X1 has in our dataset. Since v class has highest count, this feature is used common for most of the cars. We can see that there are 3 classes (q,d,ab) with lowest value than others.

Scatter plot:

The above plot is a scatter plot between Categories of X1 and distribution of y for each category. We can see that there is an point in class r of X1 which is far away from other points. This can be considered as an outlier. Less number of points belongs to classes of d,q,ab.

X2:

Bar plot:

The above plot is a bar plot which represents how many values each class of X2 has in our dataset. as class has highest count, that means this feature is most commonly used in cars. We can see that there are more than 5 classes with lowest value than others which implies these features might be rare in cars.

Scatter plot:

The above plot is a scatter plot between Categories of X2 and distribution of y for each category. We can see that there is an point in class ai of X2 which is far away from other points. This can be considered as an outlier. There is only one point in classes of af, l, aa, c, o, ar which might be rare features.

X3:

Bar plot:

The above plot is a bar plot which represents how many values each class of X3 has in our dataset. X3 has only 7 features and class a has highest count, that means this feature is most commonly used in cars. Since b has low count in dataset, we can think it as a rare feature used in cars.

Scatter plot:

The above plot is a scatter plot between Categories of X3 and distribution of y for each category. We can see that there is an point in class f of X3 which is far away from other points. This can be considered as an outlier.

Violin plot:

The above plot is a violin plot between Categories of X3 and distribution of y for each category, Data is distributed uniformly and has overlapping over classes. We can see that most values are around 80 and 130.

X4:

The above plot is a bar plot which represents how many values each class of X4 has in our dataset. Class d has highest count, that means this feature is most commonly used in cars.

Scatter plot:

The above plot is a scatter plot between Categories of X4 and distribution of y for each category. We can see that there is an point in class d of X4 which is far away from other points. This can be considered as an outlier. There are only 4 values in our training data which has b,c,d classes of X4. These features are rarer. This means that the variance of X4 is very low. We can remove this column from the dataset.

Box plot:

The above plot is box plot Categories of X4 and y for each category. We can see that class d has most values between 75 and 140. Test time for class b is around 115,class c is 130 and class a is between 90 and 110.

X5:

Bar plot:

The above plot is a bar plot which represents how many values each class of X5 has in our dataset. No of values for top classes are almost equal. This means that amount of data in our train for most classes is equal. There are 4 classes which has lowest amount of points.

Box plot:

The above plot is box plot Categories of X5 and y for each category. We can see that for most classes values have range of 75 and 110. From the above plot there is a point in class ag which can be considered as an outlier. Class u has test time around 130, class y and g has test time around 90. Class x and h might have test times between 75 and 90.

X6:

Bar plot:

The above plot is a bar plot which represents how many values each class of X6 has in our dataset. X3 has only 12 features and class g and class j has same no of points. Class k,c,b,f,e has lowest amount of points compared to other classes.

Violin plot:

The above plot is a violin plot between Categories of X6 and distribution of y for each category, Data is distributed uniformly and has overlapping over classes. We can see that most values are around 80 and 130. There is an outlier in class l.

X8:

Bar plot:

The above plot is a bar plot which represents how many values each class of X8 has in our dataset. Observing bar plots of other categorical variables, bar plot of X8 has enough data for each class. We can see that amount of data for classes are almost equal.

Violin plot:

The above plot is a violin plot between Categories of X8 and distribution of y for each category,Data is distributed uniformly and has overlapping over classes. We can see that for most classes values have range of 75 and 110. From the above plot there is a point in class t which can be considered as an outlier.

Numerical variables:

ID:

From above plot we can see that there is a slight decrease of Y with respect to ID. It might be that cars later inspected took less time in test bench. Even though in plot it shows that ID has importance in predicting model, but in reality using ID doesnt make sense. We will train our best model with ID and without ID. From the dataset we can see that ID has been randomly sampled into train and test sets.

Analysis:

We can see that there are 12 features which have same value i.e. the variance for this columns is zero. So we can remove these columns from the dataset. The rest 356 features have only 1 or 0 values.

Multivariate analysis:

I am doing Recursive Feature Elimination (RFE) to get top features.

The above 15 features are top features i got after recursive feature elimination.

The above plot is a correlation plot of the top 15 numerical features found using Recursive feature elimination (linear regression as model). From above plot we can see that X29 and X76 are highly correlated and X17 ,X382 are highly correlated as well. So there are some features in our dataset which are multicollinear. We need to remove these features.

Conclusion for Exploratory Data Analysis:

Removing feature ID and X4 from the dataset since X4 has lowest variance. Removing constant values columns from the dataset. Removing points which have target test time greater than 150 from the dataset. Remove multicollinear features.

Interaction Features:

I have used Random forest to get top features from numerical columns using feature importance to create interaction features.

From above feature importance's we can take the top features and create interaction features.

First Cut approach:

Categorical variables:

We have 7 categorical variables. I have used below three types of encoding to see which performs better.

Label Encoding, Mean Encoding, Generalized Linear Mixed Model(GLMM) encoding.

Numerical Binary variables:

For the numerical features I standardized the data and then used PCA and TSVD to create new features.

For PCA I have used Elbow method to get optimum number of components.

For TSVD I have implemented the below paper to get optimum rank for creating TSVD features.

Interaction Features:

From the Top features we got from random forest I have used combinations from itertools library to generate two way and three way interaction features.

I have created a data frame consisting of above interaction features and took features which have correlation to ‘y’ greater than 0.68. Below are the features I’ve got.

After Featurization I’ve got 4 types of datasets for each encoding and a total of 12 datasets. They are as follows:

Label Encoding features Label Encoding + Interaction features Label Encoding + Interaction + PCA features Label Encoding + Interaction + TSVD features Mean Encoding features Mean Encoding + Interaction features Mean Encoding + Interaction + PCA features Mean Encoding + Interaction + TSVD features GLMM Encoding features GLMM Encoding + Interaction features GLMM Encoding + Interaction + PCA features GLMM Encoding + Interaction + TSVD features

Models:

The idea I had in implementing models was, first Iwill try with base line models for each dataset and then try to improve results using complex models.

I have created two function in python which will do GridSearchCV and RandomizedSearchCV and plot the Train, CV scores for any model passed. Below is the code.

The below function plots features importance horizontal bar plot for top 20 features.

Linear Regression:

I am using Linear Regression as the baseline model. The scores from linear regression will act as a baseline score for further reference and improvement. For hyper parameter tuning I have used grid search cv.

Random Forests:

Random Forest is an ensemble technique known as Bagging with bootstrap sampling. This model reduces variance while training to prevent overfitting. Random forests are highly interpretable and we can get feature importance's easily.

For hyperparameter tuning of Random forest model, I have used RandomSearchCV.

Extra Trees Regressor:

This is similar to Random forests we have Extra trees regressor which is also known as Extremely Randomized trees. Extra trees reduces the variance further than random forests. This model performed well with the datasets created.

Hyper parameter tuning is done using Random Search CV.

XG Boost:

XG boost is an ensemble technique known as boosting. It reduces bias during training. XG boost is popular algorithm used in Kaggle competitions.

Stacked Model:

In most Kaggle competitions Stacked models are used to increase score to move up in the leaderboard. I have combined the Random forest, Extra trees, XG boost into a stacked model with ridge as meta regressor with regularizations taken as 0 so that it doesn’t impact the output from stacked models.

The above code snippets are for dataset (Label encoding ). For Mean, GLMM encoding and datasets with Interaction, PCA, TSVD features added visit my GitHub Repository.

Model Comparison:

Datasets -1 (Encoding of variables):

Datasets -2 (Encoding + Interaction features):

Datasets -3 (Encoding + Interaction + PCA features):

Datasets -4 (Encoding + Interaction + TSVD features):

Conclusions:

From the above scores the Best performance was provided by Extra Trees Regressor on Mean Encoding with Interaction and PCA features added with a score of 0.55258 on private leaderboard on Kagggle. As we can see that the second best model was Extra trees regressor on Label Encoding with Interaction features added. It got score of 0.55241 on private leaderboard of kaggle. We can see that the Mean encoding worked better than Label encoding and GLMM encoding did not work well. From the plots we can see that Interaction features has high importance in all the plots and X314,X315,X47 combination has top feature importance no matter which encoding we used. The PCA components performed better with Mean encoding rather than Label and GLMM. The best score on Kaggle 0.55258 give me a position of 102 on private leader board and in Top 10% scores.

My Score

Kaggle Leaderboard

Future Work:

In this I have used PCA and TSVD to create new features. We can apply other dimensionality reduction techniques like ICA, AE. We can implement Neural networks which might give a good score and a different approach.

References:

GitHub Repository: https://github.com/bnspriyatham/Mercedes-Greener-Manufacturing-Problem

LinkedIn Profile: https://www.linkedin.com/in/priyatham-bhaskara-21a670ab/