Use Pre-trained Word Embedding to detect real disaster tweets
By: Zeineb Ghrib
Tags: Word Embeddings, Kaggle, Kears, NLP, Deep Learning

Use Pre-trained Word Embedding to detect real disaster tweets

In this post we will go through the overall text classification pipeline, and especially the data pre-processing steps, we will be using a Glove pre-trained word embedding.

Textual features processing is a little bit more tricky than linear or categorical features. In fact, machine learning algorithms are more about scalars and vectors rather than characters or words. So we have to convert the text input into scalars, and the keystone 🗝 element consists in how to find out the best representation of the input words. This is the main idea behind Natural Language Processing

We will use a dataset from a Kaggle competition called Real or Not? NLP with Disaster Tweets. The task consists in predicting whether or not a given tweet is about a real disaster. To address this text classification task we will use word embedding transformation followed by a recurrent deep learning model. Other less sophisticated solutions, but still efficient, are also possible such as combining tf-idf encoding and a naive Bayes classifier (check out my last post).

Also I will include some handy Python code that can be reproduced in other NLP tasks. The overall source code is accessible in this kaggle notebook.

Introduction :

Models such as LSTM or CNN are more efficient to capture the words order and the semantic relationship between them, which usually is critical to the text’s meaning : a sample from our dataset that is labelled as a real disaster:

'#RockyFire Update => California Hwy. 20 closed in both directions due to Lake County fire - #CAfire #wildfires'

It is obvious that the words order was important in the example above.

In the other hand, we need to convert input text to a machine readable format. It exists many technics such as

one-hot encoding : each sequence text input is represented in d- dimensional space where d is size of the dataset vocabulary. Each term would get 1 if it is present in the document 0 otherwise. with a large corpus, the vocabulary would be about tens of thousands of tokens, making the one-hot vectors very sparse and inefficient.

: each sequence text input is represented in d- dimensional space where d is size of the dataset vocabulary. Each term would get 1 if it is present in the document 0 otherwise. with a large corpus, the vocabulary would be about tens of thousands of tokens, making the one-hot vectors very sparse and inefficient. TF-IDF encoding : words are mapped to numerics generated using tf-idf metric. The platform has integrated fast algorithms making it possible to keep all uni-grams and bi-grams tf-idf encoding without having to apply dimension reducing

: words are mapped to numerics generated using tf-idf metric. The platform has integrated fast algorithms making it possible to keep uni-grams and bi-grams tf-idf encoding without having to apply dimension reducing Word embedding transformation : words are projected to a dense vector space, where semantic distance between words are preserved: (see Figure below):

What is pre-trained Word Embeddings?

An embedding is a dense vector that represents a word (or a symbol). By default, the embedding vectors are randomly initialized, then will gradually be improved during the training phase, with the gradient descent algorithm at each back-propagation step, so that similar words or words in the same lexical field or with common stem … will end up close in terms of distance in the new vector space; (see figure below):

by Zeineb Ghrib

Pre-trained word embedding is an example of Transfer Learning. The main idea behind it is to use public embeddings that are already trained on large datasets. Specifically, instead of initializing our neural network weights randomly, we will set these pre trained embeddings as initialization weights. This trick helps to accelerate training and boost the performance of NLP models.