Automated NLP with Prevision.io (Part1 : Naive Bayes Classifier)
By: Zeineb Ghrib
Tags: Automl, Naive Bayes Classifier, Scikit Learn, Benchmark, Previsionio

In this post we will expose some new features integrated in Prevision.io platform that covers Natural Language Processing.

Textual features is usually more tricky and harder to process than the linear/categorical features. In fact we need to transform texts into machine readable format which requires a lot of pre-processing.

The main challenges in text based processing are the following:

Data cleaning : in fact it not about how to find linear outliers or to handle missing values.. texts may contain useless symbols that have to be removed : urls, html tags, emoticons, punctuation, spelling errors which require a lot of work😱

It is not about how to transform words to numerics as much as which is the best way to do it; because you can just map each word to a unique index and that’s it 🙈: however it would not be an efficient representation, and the ML-algorithms would end up with disappointing performances.

to do it; because you can just map each word to a unique index and that’s it 🙈: however it would not be an efficient representation, and the ML-algorithms would end up with disappointing performances. Unlike categorical features a textual feature can quickly reach a huge amount of vocabulary; so an encoding such as one-hot-encoding would be very sparse and inefficient 🐌

Words context is a meaningful information that needs to be captured: example

“he does not eat sea food” => It is not about about not eating food but rather not eating sea food

Introduction

Fortunately, Prevision auto ml makes it easy for us, all the mentioned transformations above are automatically operated, and also you have the choice to pick up the operations you want to apply. The textual feature engineering supported by the platform are the following:

Statistical-analysis based transformation (TF-IDF) : words are mapped to numerics generated using tf-idf metric. The platform has integrated fast algorithms making it possible to keep all uni-grams and bi-grams tf-idf encoding without having to apply dimension reducing

(TF-IDF) : words are mapped to numerics generated using tf-idf metric. The platform has integrated fast algorithms making it possible to keep uni-grams and bi-grams tf-idf encoding without having to apply dimension reducing Word embedding transformation : words are projected to a dense vector space, where semantic distance between words are preserved: Prevision trains a word2vec algorithm on the actual input corpus, to generate their corresponding vectors.

: words are projected to a dense vector space, where semantic distance between words are preserved: Prevision trains a word2vec algorithm on the actual input corpus, to generate their corresponding vectors. Sentence embedding : Prevision has integrated BERT-based transformers, as a pre-trained contextual model, that captures words relationships in a bidirectional way. BERT transformer makes it possible to generate more efficient vectors than word Embedding algorithms, it has a linguistic “representation” of its own. To make a text classification, we can use these vector representations as input to basic classifiers to make text classification.

In this article we will dig deeper in the very first feature NLP operator followed by a naive Bayes classifier. First we will test it out with a python code on a kaggle dataset called Real or Not? NLP with Disaster Tweets then we will launch an auto ml use case with prevision within the same dataset.

1- DIY method (Do It Yourself):

Step1 : Text cleaning : 🧹

Regardless of the EDA step that can bring out the uncleaned elements and help us to customize the cleaning code, we can apply some basic data cleaning that are recurrent in tweeters such as removing punctuation, html tags urls and emojis, spelling correction,..

Below a python code that can be be reproduced in other similar use cases 😉

Step 2: Tokenization:

Split the initial text input into small subtext using 2-gram representation : the unique unigrams + bigrams tokens define the dataset vocabulary.

Example :

“ oh my god there is another earthquake”

=> The uni-gram representation are ['oh', 'my', 'god', 'there', 'is', 'another', 'earthquake'] , the bi-grams representation would be ['oh my', 'my god', 'god there', 'there is'....]

Step 3 🤖: TF-IDF

Once we have converted our text samples into sequences of words, we need to turn these sequences into numerical vectors. One very common text encoding is the tf-idf : term-frequency inverse document-frequency.

The idea behind this metric is to take into account the occurrence frequency of a given token within a tweet while penalizing common tokens that occur very frequently in all tweets.

Step 4 🧺: Feature Selection OR dimension Reducing

Once the text converted into unigrams and bigrams, the resulted amount of tokens will certainly be very large and might slow down the training of the model later. We can either apply a dimension reducing (using SVD or PCA), or choose to drop less important tokens using f_classif or chi2 metrics.

In our example we will use sklearn.feature_selection.SelectKBest to select top 10K tokens, and we will use f_classif to compute tokens’ importance.

Python code from here

Step 5 : Naive Bayes Classifier

Theory🤓:

The algorithm is based on the Bayes theorem while assuming the naive assumption that each token in the corpus is independent of all the others.

Bayes’ theorem :

Bayes Thorem

Where :

y : the target variable

x_i : i’th token (i from 1 to n=10K tokens)

Using the “naive” assumption, it can be simplified as follows:

Bayes theorem : simplified

As the P(x1,…,xn) is a constant term, we can use the following classification rule:

We will test 2 implementations of naive bayes classifier : Gaussian Naive Bayes and Bernoulli Naive Bayes (here an exhaustive list of scikit learn naive bayes classifiers). These algos differ by how they compute the likelihood P(xi/y)

Gaussian based Naive Bayes : it assumes that the likelihood is Gaussian:

Gaussian likelihood

Bernoulli based Naive Bayes:

It will calculate the proportion of how many tweets that include the term by the total of all tweets.

We will test a stratified cross validation performances: each cv-fold keeps the same proportion as the original dataset

Lets checkout the results:

Performances results are almost similar, Bernoulli based classifier has a better precision but smaller recall, whereas the Gaussian classifier has balanced performances.