How I used a Pipeline() to solve the Kaggle Disaster Tweet competition Question
Tags: Python, Kaggle, Data Science, Machine Learning, Programming

A pipeline is used to chain multiple procedures together, thereby facilitating the machine learning process. Chains can be extremely useful because the machine learning steps must undergo a certain sequence of events, which are accomplished by the pipeline.

The functions that can be put into a pipeline are transformers or estimators. Transformers can be encoders, scalers or vectorizer/transformers. The estimators are the model that is used to predict a certain the outcome of the program, which could be a regression or a classification.

The pipeline must fit and then transform the data when it is being processed, such as being put in an encoder. The pipeline must fit and then predict on the data after it has been processed and then put into a machine learning model, such as, say, a logistic regressor.

A pipeline shortens the steps that a person must take to reach a certain outcome. For instance, if data needs to be encoded, scaled, and predicted upon, rather than writing separate lines of code to encode, scale and predict, the pipeline only requires that one set of code be written rather than three. The convenience of pipelines are that one only has to call fit and predict once on the data to fit a whole sequence of estimators.

Pipelines are very safe because they help avoid leaking statistics from test data into the trained model in cross-validation by ensuring that the same samples are used to train the transformers and predictors. All estimators in a pipeline, except the last one, must be transformers, thereby having a transform method. The last estimator in the pipeline may be any type of function in the sklearn library, such as a transformer, classifier, or regressor.

In order to show how versatile and efficient a pipeline is, I have used the Kaggle Disaster Tweet competition question for this purpose. The link to the files that must be downloaded for this competition can be found at the below link:- https://www.kaggle.com/c/nlp-getting-started/data

Extracts from the competition question are states as follows:-

“Twitter has become an important communication channel in times of emergency. The ubiquitousness of smartphones enables people to announce an emergency they’re observing in real-time. Because of this, more agencies are interested in programatically monitoring Twitter (i.e. disaster relief organizations and news agencies).

But, it’s not always clear whether a person’s words are actually announcing a disaster.

In this competition, you’re challenged to build a machine learning model that predicts which Tweets are about real disasters and which one’s aren’t.

Disclaimer: The dataset for this competition contains text that may be considered profane, vulgar, or offensive.

This dataset was created by the company figure-eight and originally shared on their ‘Data For Everyone’ website here.”

The first thing that I did was to import the libraries and the datasets needed to predict upon this problem:-

I then read the datasets and checked for any null values in the datasets:-

I decided to replace any null values with the word “unknown”, although the columns that had missing values were not used as part of the prediction:-

I imported a HTML parser and created a new column, “tidy_tweet”, in both the train and test file. All of the natural language processing has occurred in this newly created column:-

As part of the text processing, I removed the twitter handles, which had been anonymised so were not necessary. I also changed all of the text to lowercase to ease the text processing process:-

I removed the rare words from the processed text because they did not add any meaning to the analysis. I also removed words less than two characters because they would add little meaning to the analysis as well.

After processing the text, I tokenized all of the words:-

I then stemmed the words, which is to keep only the root of the word and discard the rest. After stemming, I removed the stop words from the text, which had the effect of further reducing the number of words that would need to be analysed:-

Finally, I removed the hashtags from the tweets because they are not necessary for any analysis:-

After the text was preprocessed, I defined the X, y and X_test variables. X is train.tidy_tweet, X_test is test.tidy_tweet, and y is train.target.

After the dependant and independent variables had been defined, I split the train dataset up for training and validating:-

I then created the pipeline, which streamlined the process of vectorizing the predicting upon the validation and test data. I used CountVectorizer and TfidfTransormer, which combined do the same thing that TfidfVectorizer did, which I mentioned in my earlier post and can be found in the below link:-

https://medium.com/python-in-plain-english/how-i-used-tfidfvectorizer-to-solve-a-tagging-problem-f76496c642bb

I defined the LinearSVC model and then fit these three functions into the pipeline:-

I predicted on the validation set and obtained an accuracy of 77.03%:-

When I predicted on the test dataset I obtained an accuracy of 78.36%, which was a 5.516% improvement on my previous submission.

The code for this competition question can be found in its entirety on the Kaggle website, the link being found below:- https://www.kaggle.com/tracyporter/identify-the-disaster-pipeline-linearsvc?scriptVersionId=45973839