How to Enter Your First Kaggle Competition | by Rebecca Vickery | Towards Data Science

Kaggle is probably the most well-known machine learning competition website. A Kaggle competition consists of a dataset made available from the website with a problem to solve with machine, deep learning or some other data science technique. Once you have developed a solution you upload your predictions back onto the site and dependant on the success of your solution will earn a place on a leaderboard for the competition, and possibly even a cash prize.
Kaggle can be a great place to hone your machine learning and data science skills, benchmark yourself against others and learn new techniques. In the following article, I am going to provide a walkthrough of how to enter a Kaggle competition for the first time. In this article we will;
Develop a model to predict if a tweet is about a genuine disaster.
Use the model to make predictions for the test data set supplied by Kaggle.
Make the first submission and get placed on the Kaggle leaderboard.
One of the latest competitions on the website provides a data set containing tweets together with a label which tells us if they are really about a disaster or not. This competition has a leaderboard with nearly 3,000 entries and a top cash prize of $10,000. The data and competition outline can be viewed here.
If you don’t already have a Kaggle account you can create one for free here.
If you select “download all” from the competition page you will get a zip file containing three CSV files.
The first contains a set of features and their corresponding target label for training purposes. This data set consists of the following attributes:
Id: a numerical identifier for the tweet. This will be important when we upload our predictions to the leaderboard.
Keyword: a keyword from the tweet which may in some cases be missing.
Location: the location the tweet was sent from. This may also not be present.
Text: the full text of the tweet.
Target: this is the label we are trying to predict. This will be 1 if the tweet is really about a disaster and 0 if not.
Let’s read in these files and understand a little more about them. You will notice in the below code that I have included a set_option command. Pandas set_options allow you to control how the format in which the results of a dataframe are displayed. I have included this command here to ensure that the full contents of the text column are displayed which makes my results and analyses easier to view.
import pandas as pdpd.set_option('display.max_colwidth', -1)train_data = pd.read_csv('train.csv')train_data.head()
The second data set contains only the features and for this data set, we will predict the target label and use the results to gain a place on the leaderboard.
test_data = pd.read_csv('test.csv')test_data.head()
The third gives us an example of how our submission file should look. This file will contain the id column from the test.csv file and the target we predict with our model. Once we have created this file we will submit it to the website and obtain a position on the leaderboard.
sample_submission = pd.read_csv('sample_submission.csv')sample_submission.head()
With any machine learning task before we can train a model, we have to perform some data cleaning and preprocessing. This is particularly important when working with text data.
To keep things simple for our first model, and because there is a lot of missing data in these columns, we are going to drop the location and keyword features, and only use the actual text from the tweet for training. We are also going to drop the id column as this won’t be useful for training the model.
train_data = train_data.drop(['keyword', 'location', 'id'], axis=1)train_data.head()
Our data set now looks like this.
Text, in particular tweets, can often contain lots of special characters that are not necessarily going to be meaningful to a machine learning algorithm. The first step I am therefore taking is to remove these. I am also making all words lower case.
import redef  clean_text(df, text_field):    df[text_field] = df[text_field].str.lower()    df[text_field] = df[text_field].apply(lambda elem: re.sub(r"(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)|^rt|http.+?", "", elem))      return dfdata_clean = clean_text(train_data, "text")data_clean.head()
Another useful text cleaning process is the removal of stop words. Stop words are very commonly used words that generally convey little meaning. In English these include words such as “the”, “it” and “as”. If we leave these words in the text they will create a lot of noise which would make it more difficult for the algorithm to learn.
NLTK is a collection of python libraries and tools for processing text data, the full documentation can be accessed here. In addition to its processing tools, NLTK also has a large collection of text corpora and lexical resources, including one containing all stop words in a variety of languages. We are going to use this library to remove the stop words from our data sets.
The NLTK library can be installed via pip. Once installed you need to import the library corpus and then download the stopwords file.
import nltk.corpusnltk.download('stopwords')
Once this step is completed you can read in the stop words and use it to remove them from the tweets.
from nltk.corpus import stopwordsstop = stopwords.words('english')data_clean['text'] = data_clean['text'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))data_clean.head()
Once clean the data needs further preprocessing to prepare it for use in a machine learning algorithm.
All machine learning algorithms use mathematical computations to map patterns in the features, in our case text or words, and the target variable. Before a machine learning model can be trained the text must, therefore, be turned into a numerical representation so that these calculations can be performed.
There are a number of methods for this type of preprocessing but I am going to be using two from the scikit-learn library for this example.
The first step in this process is to split the data into tokens, or individual words, count the frequency in which each word appears in the text, and then represent these counts as a sparse matrix. The CountVectoriser function achieves this.
The next step is to apply a weighting to the word counts produced by CountVectoriser. The goal of applying this weighting is to scale down the impact of very frequently occurring words in the text, so that less frequently occurring, and perhaps more informative words are deemed important during the model training. TfidTransformer can perform this function.
Let’s put all this preprocessing together with model fitting in a scikit-learn pipeline and see how the model performs. For the first attempt, I am using a linear support vector machine classifier (SGDClassifier) as this is generally regarded as one of the best text classification algorithms.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(data_clean['text'],data_clean['target'],random_state = 0)from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.pipeline import Pipelinefrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.feature_extraction.text import TfidfTransformerfrom sklearn.linear_model import SGDClassifierpipeline_sgd = Pipeline([    ('vect', CountVectorizer()),    ('tfidf',  TfidfTransformer()),    ('nb', SGDClassifier()),])model = pipeline_sgd.fit(X_train, y_train)
Let’s use the trained model to predict on our reserved test data and see how the model performs.
from sklearn.metrics import classification_reporty_predict = model.predict(X_test)print(classification_report(y_test, y_predict))
For a first attempt, the model performs reasonably well.
Let’s now see how the model performs on the competition test data set and how we rank on the leaderboard.
First, we need to clean the text in the test file and use the model to make predictions. The code below takes a copy of the test data and performs the same cleaning we applied to the training data. The output is shown below the code.
submission_test_clean = test_data.copy()submission_test_clean = clean_text(submission_test_clean, "text")submission_test_clean['text'] = submission_test_clean['text'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))submission_test_clean = submission_test_clean['text']submission_test_clean.head()
Next, we use the model to create predictions.
submission_test_pred = model.predict(submission_test_clean)
To create a submission we then need to construct a dataframe containing just the id from the test set and our prediction.
id_col = test_data['id']submission_df_1 = pd.DataFrame({                  "id": id_col,                   "target": submission_test_pred})submission_df_1.head()
Finally, we save this as a CSV file. It is important to include index=False otherwise, the index will be saved a column in the file and your submission will be rejected.
submission_df_1.to_csv('submission_1.csv', index=False)
Once we have the CSV file we can return to the competition page and select the submit predictions button. This will open a form where you can upload the CSV file. It is a good idea to add some notes about the approach so that you have a record of the attempts you have previously submitted.
Once you have submitted the file you will see this screen.
Now we have a successful submission!
This model gave me a score of 0.78 on the leaderboard and a ranking of 2,371. There is clearly some room for improvement but now I have a benchmark for future submissions.
In this post, I have given an overview of how to make your first submission to a Kaggle competition. There are a lot of further steps you could take to improve on this score. This includes better cleansing of the text, different preprocessing approaches, trying other machine learning algorithms, hyperparameter tuning of the model and much more.
Thanks for reading!