What I learnt from my first Kaggle competition
By: Rachel Adams
Tags: Competition, Videogames, Experience, Learning

As part of the Datern summer internship programme 2021, I took part in my first Kaggle competition. This was a closed competition for our cohort aiming to predict global video game sales.

There are several things I’ve learnt about Kaggle and how I prefer to use it:

Using Jupyter notebooks in Kaggle makes it really easy to manage the code and it makes importing the data and standard modules easier as there is default code for this in blank workbooks!

The inbuilt version control is really useful and allowed me to structure my thought process by giving titles when I save after every code block or two, although I did find I prefer the quick save option rather than the default save and run all code. Partly because I had just run everything and this used extra computer resource, but also I received errors when saving if there were errors in the code I was still looking in to.

There is no reason not to submit your first output as the allowed 20 submissions a day gives a great opportunity to test the model and compare the metrics to one you have calculated.

The training and test data is already given split, with the test data not having the variable being predicted. For a supervised learning problem, this is different to having one data set which you split into training and test so the test data has actuals. This means the predictions can be compared to calculate quantities such as mean square error (MSE) and the residuals. As far as I can see at the moment, the best way to check your model as you go along is to split the given training data set into a second train and test data set, check the assumptions and errors are within an allowable limit and then retrain the model with the original training data set and see what numbers Kaggle gives e.g. MSE depending on the competition.

The difference between public and private leaderboard is the data the model is tested on. The public leaderboard is calculated on a fixed combination of test and training data, but the private leaderboard the final results are calculated on is a different combination of the data. This is to ensure robustness of models and to encourage competitors to avoid overfitting.

As this was my first project predicting a continuous variable, having only attempted a classification problem before, I decided to use linear regression due to its simplicity to implement. Checking there exists a linear relationship and there was no collinearity between the variables gave me a good understanding of the 13 variables on the data set.

When exploring the data, the main thing that stood out to me was there was lots of missing data. Due to initially being unsure how to approach this I only used column with at least 90% of data for my initial model. Obviously I need to remove the ID number and name of video game as these are identifiers which exist for every model.

Following some research, see references below, I decided the best thing to do would be to impute values for the missing data rather than simply ignore them. From the scikit learn python package, I tried simple impute, iteration and KNNto transform the data. Initially, using a trial and error approach simple impute was the most accurate according to the public leaderboard. Once further columns were added, detailed below, the KNN with a k value of 15 gave better results. To investigate this further I would have liked to do a plot of MSE against k value if I had split the training data to be able to calculate such measures.

Another aspect of the data I had to learn new ways of working with was the categorical variables. Some of the variables had over 100 values so I was hesitant to use them, but using a simple label encoder, this was able to handle all of these values.

Once I had a working model, I decided to try and improve this by seeing if the other columns were good predictors. There were some columns which were related e.g. a user rating and the number of ratings, similarly for critics. Due to the relationship between these, I decided to only use one to avoid introducing bias in to my model. I also decided not use developer since there were 1000s who only produce 1 or 2 games so it is unlikely a new game would bear any relation to the previous developers.

Having built the model, the tests for the final assumptions: autocorrelation and homoscedasticity, in fact suggested this linear regression is not the best model to use as the assumptions are violated. A future project would be to investigate other potential models that may suit the data better.

In summary I have learnt a lot from my first Kaggle competition, shown by the length of this article, as well as enjoying the problem solving nature and competing against myself to improve the accuracy of my model.

References: