Text Processing Techniques on Twitter data | by Ramya Vidiyala | Towards Data Science

There is a huge amount of data in text format. Analyzing data in the text format is the most complex task for a machine as it’s difficult for a machine to understand the semantics behind the text. To serve this purpose we process text data into a machine-understandable format.
Text Processing is simply converting the data in text format to numerical values (or vectors ) so that we can give these vectors as input to the machine and analyze the data using the concepts of algebra.
However, when we perform such transformation, there could be data loss. The key is to maintain an equilibrium between conversion and retaining the data.
Before processing the text, preprocessing of text is essential. To have a quick glance at what text preprocessing is, check this article.
medium.com
In this article, we will discuss various techniques to perform text processing.
Before getting started, let us understand a few commonly used terms.
Each text data point is called a Document
The entire set of documents is called Corpus
Text processing can be done using the following techniques,
Bag of WordsTF-IDFWord2Vec
Bag of Words
TF-IDF
Word2Vec
Now let us begin exploring each technique in detail.
Bag of Words does a simple transformation of the document to vector by using a dictionary of unique words. This is done in just two steps,
Create a dictionary of all the unique words in the data corpus in a vector form. Let the number of unique words in the corpus be, ‘d’. So each word is a dimension and hence this dictionary vector is a d-dimension vector.
For each document, say, ri we create a vector, say, vi. Now, this vi which has d-dimensions can be constructed in two ways:
For each document, the vi is constructed in accordance with the dictionary such that each word in the dictionary is reproduced by the number of times that word is present in the document.For each document, the vi is constructed in accordance with the dictionary such that each word in the dictionary is reproduced to,
For each document, the vi is constructed in accordance with the dictionary such that each word in the dictionary is reproduced by the number of times that word is present in the document.
For each document, the vi is constructed in accordance with the dictionary such that each word in the dictionary is reproduced to,
1 if the word is present in the document or
0 if the word is not present in the document
This type is known as Binary Bag of Words.
Now, we have vectors for each document and a dictionary that has a set of unique words of the data corpus. These vectors can be analyzed by
plotting in d-dimension space or
calculating distance between vectors to get the similarity (the closer the vectors are, more similar they are)
Problem: Bag of Words does not consider the semantic of the words. Means words that have the same semantic as tasty, delicious are classified into 2 different words.Solution: Preprocessing the data in using techniques like Stemming and Lemmatization.
Problem: Bag of Words does not retain the sequential information of the document which means, ‘not good’ which is clearly not- good. Bag of words classifies it as ‘not’ and ‘good’, which is clearly not good. Solution: Instead of creating vectors where each cell is a word, we can create vectors where each cell has 2 words ( called bi-grams) or 3 words (called tri-grams). Using these n-grams, sequential information can be retained. However, when we use n-grams instead of uni-grams, the dimensionality of features increases.
There are three elements here — word, document, corpus. Term Frequency — Inverse Document Frequency, TF-IDF, in short, uses the relationship between these to convert text data into vectors.
Term Frequency says about the relationship between a word and a document. Whereas, Inverse Document Frequency says about the relationship between a word and the corpus.
Term frequency is the probability of the word wj in the document ri. And is calculated as,
If Term Frequency of a word in a review is high implies, the word is frequent in that review. If Term Frequency of a word in a review is low implies, the word is rare in that review.
Inverse Document Frequency says how frequently the word is in the entire corpus. And is calculated as,
If Inverse Document Frequency is low, implies the word is frequent in the corpus.If Inverse Document Frequency is high, implies the word is rare in the corpus.
The reason to use log instead of the simple inverse ratio is scaling. Term Frequency which is a probability ranges between 0 and 1. When we simply take this inverse ratio it will be a huge value, thus the entire TF-IDF value will be biased to IDF. This is one of the simple and highly accepted reasons for using the log in IDF term.
TF-IDF of a word in review is TF(word, review) *IDF(word, document corpus).
Now in the vector form of each document, we have this TF-IDF of the word. Converting a document into a vector, using TF-IDF values is called TF-IDF vectorization.
TF-IDF vectorization gives high importance to words which are
frequent in a document (from TF)
rare in the corpus (from IDF)
In Bag of Words and TF-IDF, we convert sentences into vectors. But in Word2Vec, we convert word into a vector. Hence the name, word2vec!
Word2Vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space. Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located close to one another in the space.
Each word is closer in the space to the word which has the same semantic(meaning like woman, girl)It retains the relationship between words (vector of man to woman is parallel to the vector of king to queen)
Each word is closer in the space to the word which has the same semantic(meaning like woman, girl)
It retains the relationship between words (vector of man to woman is parallel to the vector of king to queen)
In this article, we will be using Customer Support on the Twitter dataset from Kaggle.
www.kaggle.com
The Customer Support on the Twitter dataset is a large, modern corpus of tweets and replies to aid innovation in natural language understanding and conversational models, and for the study of modern customer support practices and impact. This dataset offers a large corpus of modern English (mostly) conversations between consumers and customer support agents on Twitter.
Let us use the preprocessed data and perform each text preprocessing on it. Let us look at the preprocessed data
data["preprocessed_text"]
Sckit learns provides us with many libraries. It makes us so easy to implement any functionality in a single line.
from sklearn.feature_extraction.text import CountVectorizer
Sckit learn provides CountVectorizer to perform Bag of Words.
bow=CountVectorizer( min_df=2, max_features=1000)bow.fit(data['preprocessed_text'])bow_df=bow.transform(data['preprocessed_text']).toarray()
Creating an object of count vectorizer and fitting, transforming with the preprocessed data. bow_df is a sparse vector which contains more number of zeros than ones.
from sklearn.feature_extraction.text import TfidfVectorizer
Sckit learn provides TFIDfVectorizer to perform TF-IDF Vectorization.
tfidf = TfidfVectorizer( min_df=2, max_features=1000)tfidf.fit(data['preprocessed_text'])tfidf_df=bow.transform(data['preprocessed_text']).toarray()
tfidf_df contains the tf-idf values of each word in a document in a 1000 dimensional vector.
Google provides a huge list of vectors trained on humongous data of google news. To use these vectors, we need to import Gensim.
import gensimtokenize=data['preprocessed_text'].apply(lambda x: x.split())w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)w2vec_model.train(tokenize,total_examples= len(data['preprocessed_text']),epochs=20)
Output : (18237, 22200) data points. This means 18237 data points of text is now converted to a 22200-dimension vector each.
There is no obvious answer to this question: it really depends on the application.
For example, the Bag of Words is commonly used for document classification applications where the occurrence of each word is used as a feature for training a classifier.
TF-IDF is used by search engines like Google, as a ranking factor for content.
When the application is about understanding the context of words or, detecting the similarity of the words or, translating given documents into another language, which requires a lot of information on the document, Word2vec comes into the picture.
Thanks for the read. I am going to write more beginner-friendly posts in the future. Follow me up on Medium to be informed about them. I welcome feedback and can be reached out on Twitter ramya_vidiyala and LinkedIn RamyaVidiyala. Happy learning!