How to get started on Kaggle Competitions
By: Rafael Bastos
Tags: Kaggle, Data Science, Machine Learning, Data Analysis, Tutorial

1. Understand the Data

The first step when you face a new data set is to take some time to know the data. In Kaggle competitions, you’ll come across something like the sample below.

Kaggle competition landing page

On the competition’s page, you can check the project description on Overview and you’ll find useful information about the data set on the tab Data. In Kaggle competitions, it’s common to have the training and test sets provided in separate files. On the same tab, there’s usually a summary of the features you’ll be working with and some basic statistics. It’s crucial to understand which problem needs to be addressed and the data set we have at hand.

You can use the Kaggle notebooks to execute your projects, as they are similar to Jupyter Notebooks.

2. Import the necessary libraries and data set

2.1. Libraries

The libraries used in this project are the following.

2.2. Data set

The next step is to read the data set into a pandas DataFrame and obtain target vector y, which will be the column SalePrice , and predictors X, which, for now, will be the remaining columns.

To get an overview of the data, let’s check the first rows and the size of the data set.

Sample of the first samples of the DataFrame

Checking the size of X and y

We have 1,460 rows and 79 columns. Later on, we’ll check these columns to verify which of them will be meaningful to the model.

In the next step, we’ll split the data into training and validation sets.

3. Training and validation data

It is crucial to break our data into a set for training the model and another one to validate the results. It’s worth mentioning that we should never use the test data here. Our test set stays untouched until we are satisfied with our model’s performance.

What we’re going to do is taking the predictors X and target vector y and breaking them into training and validation sets. For that, we’ll use scikit-learn’s train_test_split .

4. Analyze and prepare the data

Now, we start analyzing the data by checking some information about the features.

Checking X info

From the summary above, we can observe that some columns have missing values. Let’s take a closer look.

4.1. Missing Values

Checking for missing values

Some features have missing values counting for the majority of their entries. Checking the competition page, we find more details about the values for each feature, which will help us handle missing data.

For instance, in the columns PoolQC , MiscFeature , Alley , Fence , and FireplaceQu , the missing values mean that the house doesn't count with that specific feature, so, we'll fill the missing values with "NA". All the null values in columns starting with Garage and Bsmt are related to houses that don't have a garage or basement, respectively. We'll fill those and the remaining null values with "NA" or the mean value, considering if the features are categorical or numerical.

4.2. Preprocessing the categorical variables

Most machine learning models only work with numerical variables. Therefore, if we feed the model with categorical variables without preprocessing them first, we’ll get an error.

There are several ways to deal with categorical values. Here, we’ll use One-Hot Encoding, which will create new columns indicating the presence or absence of each value in the original data.

One issue of One-Hot Encoding is dealing with variables with numerous unique categories since it will create a new column for each unique category. Thus, this project will only include categorical variables with no more than 15 unique values.

4.3. Create a pipeline

Pipelines are a great way to keep the data modeling and preprocessing more organized and easier to understand. Creating a pipeline, we’ll handle the missing values and the preprocessing covered in the previous two steps.

As defined above, numerical missing entries will be filled with the mean value while missing categorical variables will be filled with “NA”. Furthermore, categorical columns will also be preprocessed with One-Hot Encoding.

We are using SimpleImputer to fill in missing values and ColumnTransformer will help us to apply the numerical and categorical preprocessors in a single transformer.

5. Define a model

Now that we have bundled our preprocessors in a pipeline, we can define a model. In this article, we are working with XGBoost, one of the most effective machine learning algorithms, that presents great results in many Kaggle competitions. As a metric of evaluation, we are using the Mean Absolute Error.

Mean Absolute Error of the validation set

6. Cross-validation

Using Cross-Validation can yield better results. Instead of simply using the training and test sets, cross-validation will run our model on different subsets of the data to get multiple measures of model quality.

We’ll use the cross-validator KFold in its default setup to split the training data into 5 folds. Then, each fold will be used once as validation while the remaining folds will form the training set. After that, cross-validate will evaluate the metrics. In this case, we’re using the Mean Absolute Error.

Mean Absolute Error after cross-validation

With cross-validation we could improve our score, reducing the error. In the next step, we’ll try to further improve the model, optimizing some hyperparameters.

7. Hyperparameter tuning

XGBoost in its default setup usually yields great results, but it also has plenty of hyperparameters that can be optimized to improve the model. Here, we’ll use a method called GridSearchCV which will search over specified parameter values and return the best ones. Once again, we’ll utilize the pipeline and the cross-validator KFold defined above.

GridSearchCV will perform an exhaustive search over parameters, which can demand a lot of computational power and take a lot of time to be finished. We can speed up the process a little bit by setting the parameter n_jobs to -1 , which means that the machine will use all processors on the task.

The optimal parameter values were:

n_estimators: 400

max_depth: 3

min_child_weight: 0.0001

learning_rate: 0.1

8. Generate test predictions

After tuning some hyperparameters, it’s time to go over the modeling process again to make predictions on the test set. We’ll define our final model based on the optimized values provided by GridSearchCV.

9. Submit your results

We’re almost there! The machine learning modeling is done, but we still need to submit our results to have our score recorded.

This step is quite simple. We need to create a .csv file containing the predictions. This file consists of a DataFrame with two columns. In this case, one column for "Id" and the other one for the test predictions on the target feature.

10. Join the competition

Finally, we just need to join the competition. Please follow the steps below, according to Kaggle’s instructions.

Start by accessing the competition page and clicking on Join Competition .

. In your Kaggle notebook, click on the blue Save Version button in the top right corner of the window.

A pop-up window will show up. Select the option Save and Run All and then click on the blue Save button.

and then click on the blue Save button. A new pop-up shows up in the bottom left corner while your notebook is running. When it stops running, click on the number to the right of the Save Version button. You should click on the ellipsis (…) to the right of the most recent notebook version, and select Open in Viewer . This brings you into view mode of the same page.

button. You should click on the to the right of the most recent notebook version, and select . This brings you into view mode of the same page. Now, click on the Output tab on the right of the screen. Then, click on the blue Submit button to submit your results to the leaderboard.

After submitting, you can check your score and position on the leaderboard.