Analysis and Visualization of Unstructured Text | Towards Data Science

Stuck behind the paywall? Read this article with my friend link here.
What will you do when I ask you to explain textual data? What steps will you take to build the textual visualization story? Here, I am not going to explain how you create a visualization story.
But this article will help you to get the required information to build the visualization story and explain the textual data.
Insights from textual data will help us to discover the connection between the articles. It will detect trends and patterns. Analysis of textual data will set aside the noise and uncovers previously unknown information.
This analysis process is also known as Exploratory Text Analysis (ETA). With the help of K-means, Tf-IDF, word frequency, etc. method, we will analyze these textual data. Also, ETA is useful in the data cleaning process.
We also visualize the results in graphs, word clouds, and plots using Matplotlib, seaborn, and Plotly libraries.
Before analyzing the Textual Data, complete these pre-processing tasks.
There is a lot of unstructured text data available for analysis. You can get data from the below sources.
1. Twitter text dataset from Kaggle.
2. Reddit and twitter dataset using API.
3. Scrape articles from a website using Beautifulsoup and Requests python library.
I am going to use Reuters’ article available in SGML format. For analysis purposes, I will fetch the date, title, and article body from data files using the Beautifulsoup library.
Use the below code to fetch the data from all data files and store the output in a single CSV file.
1. You can also use the Regex and OS library to combine or loop all the data files.
2. Each article’s body starts with <Reuters>, so use find_all(‘reuters’).
3. You can also use the pickle module to save data, instead of CSV.
In this section, we remove noise such as null values, punctuation, numbers, etc. from the textual data. First, we remove the rows which contain null values in the text column. Then we deal with other column’s null values.
import pandas as pd import rearticles_data = pd.read_csv(‘articles_data.csv’) print(articles_data.apply(lambda x: sum(x.isnull()))) articles_nonNull = articles_data.dropna(subset=[‘text’]) articles_nonNull.reset_index(inplace=True)def clean_text(text):‘’’Make text lowercase, remove text in square brackets,remove \n,remove punctuation and remove words containing numbers.’’’    text = str(text).lower()    text = re.sub(‘<.*?>+’, ‘’, text)    text = re.sub(‘[%s]’ % re.escape(string.punctuation), ‘’, text)    text = re.sub(‘\n’, ‘’, text)    text = re.sub(‘\w*\d\w*’, ‘’, text)    return textarticles_nonNull[‘text_clean’]=articles_nonNull[‘text’]\                                  .apply(lambda x:clean_text(x))
· When we remove null values present in the text column, then the null values in other columns also vanishes.
· We have used re method to remove noise in textual data.
Steps taken in the data cleaning process may increase or decrease as per the textual data. So, study your text data carefully and frame your clean_text() method accordingly.
As the pre-processing task is complete, we are going to move on to analyzing textual data.
We know that the length of all the articles is not the same. So, we will consider those articles whose length equal to or more than one paragraph. As per the study, the average length of a sentence is 15–20 words. And there should be four sentences in a paragraph.
articles_nonNull[‘word_length’] = articles_nonNull[‘text’].apply(lambda x: len(str(x).split())) print(articles_nonNull.describe())articles_word_limit = articles_nonNull[articles_nonNull[‘word_length’] > 60]plt.figure(figsize=(12,6)) p1=sns.kdeplot(articles_word_limit[‘word_length’], shade=True, color=”r”).set_title(‘Kernel Distribution of Number Of words’)
· I have removed those articles whose length is less than 60 words.
· Word_length distribution is right-skewed.
· Most of the articles have around 150 words.
· Reuters articles that contain fact or stock information have fewer words.
In this section, we count the words present in the articles and analyze the result. We analyze the word count based on the N-gram method. N-gram is the occurrence of words based on its N value.
We will remove the stopwords from the textual data. Because stopwords are noise and not have much use in the analysis.
Let’s plot unigram words in a bar graph and word cloud for Unigram words.
Share, trade, and stock are some of the most frequent words and based on the stock market and financial sector articles.
So, we can say that most Reuters articles belong to the finance and Stock category.
Let’s plot the bar graph and word cloud for Bigram words.
article_bigrams = defaultdict(int)for tweet in articles_word_limit[‘temp_list_stopw’]:    for word in generate_ngrams(tweet, n_gram=2):        article_bigrams[word] += 1df_article_bigrams=pd.DataFrame(sorted(article_bigrams.items(),                                key=lambda x: x[1])[::-1])N=50# bar graph of top 50 bigram wordsfig, axes = plt.subplots(figsize=(18, 50), dpi=100)plt.tight_layout()sns.barplot(y=df_article_bigrams[0].values[:N],            x=df_article_bigrams[1].values[:N],             color=’red’)axes.spines[‘right’].set_visible(False)axes.set_xlabel(‘’)axes.set_ylabel(‘’)axes.tick_params(axis=’x’, labelsize=13)axes.tick_params(axis=’y’, labelsize=13)axes.set_title(f’Top {N} most common Bigrams in Reuters Articles’,               fontsize=15)plt.show()#Word cloudwc = WordCloud(width=2000, height=1000, collocations=False,               background_color=”white”,               color_func=col_func,               max_words=200,               random_state=np.random.randint(1,8))\               .generate_from_frequencies(article_bigrams)fig, ax = plt.subplots(figsize=(20,10))ax.imshow(wc, interpolation=’bilinear’)ax.axis(“off”)ax.set_title(‘Trigram Words of Reuters Articles’, pad=24,             fontdict=fd)plt.show()
Bigram gives more information and context on text than unigram. Like, share loss bigram shows that most people lost money in stock.
Let’s plot bar graph and word cloud for Trigrma words.
article_trigrams = defaultdict(int)for tweet in articles_word_limit[‘temp_list_stopw’]:    for word in generate_ngrams(tweet, n_gram=3):        article_trigrams[word] += 1df_article_trigrams = pd.DataFrame(sorted(article_trigrams.items(),                                   key=lambda x: x[1])[::-1])N=50# bar graph of top 50 trigram wordsfig, axes = plt.subplots(figsize=(18, 50), dpi=100)plt.tight_layout()sns.barplot(y=df_article_trigrams[0].values[:N],            x=df_article_trigrams[1].values[:N],             color=’red’)axes.spines[‘right’].set_visible(False)axes.set_xlabel(‘’)axes.set_ylabel(‘’)axes.tick_params(axis=’x’, labelsize=13)axes.tick_params(axis=’y’, labelsize=13)axes.set_title(f’Top {N} most common Trigrams in Reuters articles’,               fontsize=15)plt.show()# word cloudwc = WordCloud(width=2000, height=1000, collocations=False,background_color=”white”,color_func=col_func,max_words=200,random_state=np.random.randint(1,8)).generate_from_frequencies(article_trigrams)fig, ax = plt.subplots(figsize=(20,10))ax.imshow(wc, interpolation=’bilinear’)ax.axis(“off”)ax.set_title(‘Trigrams Words of Reuters Articles’, pad=24,             fontdict=fd)plt.show()
Most trigrams are similar to bigrams. Trigrams words help us understand the text more than bigram. But not able to provide more information. So we end this section here.
NER is a process to extract specific information from textual data. With the help of NER, we extract location, person name, date, quantity, and organization entity from the text. Learn more about NER here. We use the Spacy python library for this job.
· From this graph, you can say that most articles contain news from the US, Japan, Canada, London, and China.
· High mention of the US represents the focus of Reuters business in the US.
· The person variable implies who are the famous people in 1987. This information helps us to know about those people.
· The organization variable contains the most mention organization around the world.
We are going to find unique words in the articles using TF-IDF. Term Frequency (TF) is a count of words per article. Inverse Document Frequency (IDF) measures the importance of words while considering all the mention articles.
TF-IDF scores are high for those words which have a high count in one article and rare or not present in others.
Let’s calculate the TF-IDF score and find unique words.
from sklearn.feature_extraction.text import TfidfVectorizertfidf_vectorizer = TfidfVectorizer(use_idf=True)tfidf_vectorizer_vectors=tfidf_vectorizer.fit_transform(articles_word_limit[‘text_clean’])tfidf = tfidf_vectorizer_vectors.todense()tfidf[tfidf == 0] = np.nan#Use nanmean of numpy which will ignore nan while calculating meanmeans = np.nanmean(tfidf, axis=0)# convert it into a dictionary for later lookupMeans_words = dict(zip(tfidf_vectorizer.get_feature_names(),                       means.tolist()[0]))unique_words=sorted(means_words.items(),                    key=lambda x: x[1],                    reverse=True)print(unique_words)
K-Means is an unsupervised Machine learning algorithm. It helps us to collect the same type of articles in one group. We can decide the number of groups or clusters by initializing k value. Know more about K-Means and how we select k value here. For reference, I choose k=4 to form four groups.
from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.cluster import KMeansfrom sklearn.metrics import adjusted_rand_scorevectorizer = TfidfVectorizer(stop_words=’english’,use_idf=True)X = vectorizer.fit_transform(articles_word_limit[‘text_clean’])k = 4model = KMeans(n_clusters=k, init=’k-means++’,               max_iter=100, n_init=1)model.fit(X)order_centroids = model.cluster_centers_.argsort()[:, ::-1]terms = vectorizer.get_feature_names()clusters = model.labels_.tolist()articles_word_limit.index = clustersfor i in range(k):    print(“Cluster %d words:” % i, end=’’)for title in articles_word_limit.ix[i                    [[‘text_clean’,’index’]].values.tolist():    print(‘ %s,’ % title, end=’’)
It helps us in sorting the articles in different groups such as sports, money, finance, and more. The accuracy of K-Means is generally low. So, this is useful in low-level analysis.
NER and K-Means are my favorite method for analysis. Others might like N-gram and Unique words methods. In this article, I have covered the most famous and unheard method for text visualization and analysis. All these methods in this article are unique and help you in visualization and analysis.
I hope this article will help you to discover unknowns in your textual data.
Other Articles by Author
First step in EDA : Descriptive Statistic AnalysisAutomate Sentiment Analysis Process for Reddit Post: TextBlob and VADERDiscover the Sentiment of Reddit Subgroup using RoBERTa Model
First step in EDA : Descriptive Statistic Analysis
Automate Sentiment Analysis Process for Reddit Post: TextBlob and VADER
Discover the Sentiment of Reddit Subgroup using RoBERTa Model