NLP in Python- Vectorizing
By: Divya Raghunathan
Tags: Machine Learning, Python, NLP, Data Science, Data

NLP in Python- Vectorizing

Common vectorizing techniques employed in a typical NLP machine learning model pipeline using the real of fake news dataset from Kaggle.

Photo by Roman Kraft from Unsplash

In this article, we will learn about vectorizing and different vectorizing techniques employed in an NLP model. Then, we will apply these concepts to the context of a problem.

We will work with a dataset that classifies news as fake or real. The dataset is available on Kaggle, the link to the dataset is below:

The initial step involved in a typical machine learning text pipeline is data cleaning. This step is covered in detailed in a previous article, linked below:

Dataset after data cleaning

The raw news titles were transformed into a cleaned format containing only the essential information (last column of the above picture). The next step is to further transform the cleaned text into a form that the machine learning model can understand. This process is known as Vectorizing. In our context, each news title is converted to a numerical vector representative of that particular title. There are many vectorization techniques, but in this article, we will focus on the three widely used vectorization techniques- Count vectorization, N-Grams, TF-IDF, and their implementation in Python.

Count vectorization

As discussed above, vectorization is the process of converting text to numerical entries in a matrix form. In the count vectorization technique, a document term matrix is generated where each cell is the count corresponding to the news title indicating the number of times a word appears in a document, also known as the term frequency. The document term matrix is a set of dummy variables that indicates if a particular word appears in the document. A column is dedicated to each word in the corpus. The count is directly proportionate to the correlation of the category of the news title. This means, if a particular word appears many times in fake news titles or real news titles, then the particular word has a high predictive power of determining if the news title is fake or real.

def clean_title(text):

text = "".join([word.lower() for word in text if word not in string.punctuation])

title = re.split('\W+', text)

text = [ps.stem(word) for word in title if word not in nltk.corpus.stopwords.words('english')]

return text

count_vectorize = CountVectorizer(analyzer=clean_title)

vectorized = count_vectorize.fit_transform(news['title'])

Dissecting the above code, the function “clean_title”- joins the lowercase news titles without punctuation. Then, the text is split on any non-word character. Finally, the non-stop words are stemmed and presented as a list. A detailed description of the cleaning process is given in this article.

Next, we have made use of the “CountVectorizer” package available in the sklearn library under sklearn.feature_extraction.text. The default values and the definition are available in the scikit-learn — Count Vectorizer documentation. In the above code, we have instantiated Count Vectorizer and defined one parameter — analyzer. The other parameters are its default values. The analyzer parameter calls for a string and we have passed a function, that takes in raw text and returns a cleaned string.

The shape of the document term matrix is 44898,15824. There are 44898 news titles and 15824 unique words in all the titles.

A subset of the 15824 unique words in the news title

The vectorizer produces a sparse matrix output, as shown in the picture. Only the locations of the non-zero values will be stored to save space. So, an output of the vectorization will look something like this:

<20x158 sparse matrix of type '<class 'numpy.int64'>'

with 206 stored elements in Compressed Sparse Row format>

but, converting the above to an array form yields the below result:

As shown in the picture, most of the cells contain a 0 value, this is known as a sparse matrix. Many vectorized outputs would look similar to this, as naturally many titles wouldn’t contain a particular word.

2. N-Grams

Similar to the count vectorization technique, in the N-Gram method, a document term matrix is generated and each cell represents the count. The difference in the N-grams method is that the count represents the combination of adjacent words of length n in the title. Count vectorization is N-Gram where n=1. For example, “I love this article” has four words and n=4.

if n=2, i.e bigram, then the columns would be — [“I love”, “love this”, ‘this article”]

if n=3, i.e trigram, then the columns would be — [“I love this”, ”love this article”]

if n=4,i.e four-gram, then the column would be -[‘“I love this article”]

The n value is chosen based on performance.

For the python code, the cleaning process is performed similarly to the count vectorization technique, but the words are not in a tokenized list form. The tokenized words are joined to form a string, so the adjacent words can be gathered to effectively perform N-Grams.

The cleaned title text is shown below:

The remaining vectorization technique is the same as the Count Vectorization method we did above.

The trade-off is between the number of N values. Choosing a smaller N value, may not be sufficient enough to provide the most useful information. Whereas choosing a high N value, will yield a huge matrix with loads of features. N-gram may be powerful, but it needs a little more care.

3. Term Frequency-Inverse Document Frequency (TF-IDF)

Similar to the count vectorization method, in the TF-IDF method, a document term matrix is generated and each column represents a single unique word. The difference in the TF-IDF method is that each cell doesn’t indicate the term frequency, but the cell value represents a weighting that highlights the importance of that particular word to the document.

TF-IDF formula: