Kaggle CareerCon 2019 competition report (#121 of 1449, top 9%)
By: Albert Sanchez Lafuente
Tags: Random Forest, Machine Learning, Data Science, Kaggle, Sklearn

This has been my first Kaggle competition and I’m very happy with the result I got. I devoted many hours to analyzing data, writing code and generating my data science template which helped me achieve a great result. During the last days of the competition I was around position number 500, however when the test set was changed from the public to the private one I jumped a lot of positions to the #121 which meant being at the top 9% of the competition ranking.

I started writing some reports during the competition and posting them on medium (1), (2). Those were a very primitive solution which didn’t get a good score. After the second post, I started working on my data science template and applied it to this competition. Only doing this gave me a jump in accuracy from 0.39 to 0.65. That was my most important jump through all the competition and I could do it in just a few days work. From then on I kept working on improving my solution but never getting more than a 1% jump in accuracy at each step.

The most important steps I did that allowed me to increase my accuracy were adding a lot of feature engineering and doing feature selection.

The competition

The objective of the competition was to predict on which kind of surface a robot was moving (wood, concrete, etc.). To predict this they gave data of the movement of the robots, including orientation given in quaternions (x, y, z, w), angular velocity (x, y, z) and linear acceleration (x, y, z). They were given as time series data divided in sets of 128 measurements.

Visualisation of the time series data

It is very interesting to follow the kernels published in Kaggle from other competitors and right after the competition ended, the top performers show how they got their results. I’m a little bit disappointed with the final results because one of the main factors that gave people the victory was realizing the dataset which was divided in chunks of 128 measurements, those chunks of measurements were part of bigger recording sessions also available inside the dataset. Basically you just have to compare the different chunks between each other and see which ones belong to the same big chunk.

I decided I wouldn’t implement this feature in my code since this was my first competition and my objective was to learn data science, not winning a competition. This kind of feature wouldn’t give any relevant result in a real-life project so it wouldn’t really teach me anything useful for my projects.

The code

After this introduction, let’s dive into the code and used techniques. I implemented it using Google Colab. I will only show the important parts of the project, I won’t get into detail about how I loaded the dataset or how I pre-processed the data to make sure I had a clean dataset. I will only focus on the most important parts which made the difference in the result.

On a first data exploration, the most insightful thing was the correlation matrix:

#Correlation analysis

sns.heatmap(df_train.corr(), annot=True, fmt='.2f')

Correlation matrix for the dataset

As you can see some variables were highly correlated like orientation_W and orientation_X. During the final steps of the competition this lead to deleting some of those highly correlated features, boosting the model accuracy.

Another important step was feature engineering. I added many features based on the original time series data, while I lost the time series information, I got a set of new features based on that which were even more informative. The features included the mean of the time series, the maximum value, the minimum value and many, many more. One important set of features was the Fourier Transform, adding the frequency domain transformation to the time series data.

For cross validation I used a very simple strategy of dividing the training data given by the competition into a training and validation set. For that I used 70% of the data for training and 30% for validation purposes, this was done while stratifying both sets (having the same percentage of each category in both sets).

The algorithm

Then came the model selection phase. I tested many models including linear regression, Random Forest, XGBoost, LightGBM. The one that gave me the best results on the first run was random forest, so I chose to work with it. Looking back on it, I regret not working also with the XGBoost algorithm since I believe with some hyper-parameter tuning it could have given great results.

Using Grid Search I found the best values for the hyper-parameters for a random forest with this dataset. Those values were:

n_estimators=5000

min_samples_leaf=1

max_features=0.3

They were found using grid search, testing different values. This is the list of values that were tested:

n_estimators:[100,500,1000,2500,5000,10000]

min_samples_leaf:[1, 3, 5, 10, 15, 20]

max_features:['auto', 0.1, 0.3, 0.5]

Feature selection

After training the random forest model, I took the least important features out of the model. This gave me an important jump in accuracy.

important_features = feature_importances[feature_importances['importance']>0.001].index.values

X_train = X_train[important_features]

Using the dataFrame only with the important features, I retrained the random forest and got an accuracy boost.

Submission

After having a working model I wrote a function to generate the submission file.

testpreds = rfc.predict(df_test)

testpreds = le.inverse_transform(testpreds) dfres = pd.DataFrame()

dfres['series_id'] = df_test.index

dfres['surface'] = testpreds dfresf = dfres.groupby('series_id').agg(lambda x:x.value_counts().index[0]) dfresf.to_csv('out.csv')

All was left then was to upload the submission to Kaggle and enjoy the emotion and nervousness while waiting for the evaluation.

Conclusion

The solution outlined in this post, was not the first solution I got for the competition. It involved a lot of iteration and testing of different ideas, many of them good and many of them not.

Overall it was a very interesting challenge and a great opportunity to participate in a data science project with a defined objective and a great learning experience. Totally recommended to follow a Kaggle competition from the beginning and fight to get in the first positions of the ranking.