Analysis and Visualization of Unstructured Text
By: Manmohan Singh
Tags: Data Visualization, Matplotlib, Python Programming, Exploratory Data Analysis, Data Analysis

What will you do when I ask you to explain textual data? What steps will you take to build the textual visualization story? Here, I am not going to explain how you create a visualization story.

But this article will help you to get the required information to build the visualization story and explain the textual data.

Insights from textual data will help us to discover the connection between the articles. It will detect trends and patterns. Analysis of textual data will set aside the noise and uncovers previously unknown information.

This analysis process is also known as Exploratory Text Analysis (ETA). With the help of K-means, Tf-IDF, word frequency, etc. method, we will analyze these textual data. Also, ETA is useful in the data cleaning process.

We also visualize the results in graphs, word clouds, and plots using Matplotlib, seaborn, and Plotly libraries.

Before analyzing the Textual Data, complete these pre-processing tasks.

Retrieve data from Data Source

There is a lot of unstructured text data available for analysis. You can get data from the below sources.

1. Twitter text dataset from Kaggle.

2. Reddit and twitter dataset using API.

3. Scrape articles from a website using Beautifulsoup and Requests python library.

I am going to use Reuters’ article available in SGML format. For analysis purposes, I will fetch the date, title, and article body from data files using the Beautifulsoup library.

Use the below code to fetch the data from all data files and store the output in a single CSV file.

1. You can also use the Regex and OS library to combine or loop all the data files.

2. Each article’s body starts with <Reuters>, so use find_all(‘reuters’).

3. You can also use the pickle module to save data, instead of CSV.

Data Cleaning Process

In this section, we remove noise such as null values, punctuation, numbers, etc. from the textual data. First, we remove the rows which contain null values in the text column. Then we deal with other column’s null values.

import pandas as pd import re articles_data = pd.read_csv(‘articles_data.csv’) print(articles_data.apply(lambda x: sum(x.isnull()))) articles_nonNull = articles_data.dropna(subset=[‘text’]) articles_nonNull.reset_index(inplace=True) def clean_text(text): ‘’’Make text lowercase, remove text in square brackets,remove

,remove punctuation and remove words containing numbers.’’’ text = str(text).lower()

text = re.sub(‘<.*?>+’, ‘’, text)

text = re.sub(‘[%s]’ % re.escape(string.punctuation), ‘’, text)

text = re.sub(‘

’, ‘’, text)

text = re.sub(‘\w*\d\w*’, ‘’, text)

return text articles_nonNull[‘text_clean’]=articles_nonNull[‘text’]\

.apply(lambda x:clean_text(x))

· When we remove null values present in the text column, then the null values in other columns also vanishes.

· We have used re method to remove noise in textual data.

Steps taken in the data cleaning process may increase or decrease as per the textual data. So, study your text data carefully and frame your clean_text() method accordingly.

As the pre-processing task is complete, we are going to move on to analyzing textual data.

Let’s start with our analysis.

1. Length of Reuters Articles

We know that the length of all the articles is not the same. So, we will consider those articles whose length equal to or more than one paragraph. As per the study, the average length of a sentence is 15–20 words. And there should be four sentences in a paragraph.

articles_nonNull[‘word_length’] = articles_nonNull[‘text’].apply(lambda x: len(str(x).split())) print(articles_nonNull.describe()) articles_word_limit = articles_nonNull[articles_nonNull[‘word_length’] > 60] plt.figure(figsize=(12,6)) p1=sns.kdeplot(articles_word_limit[‘word_length’], shade=True, color=”r”).set_title(‘Kernel Distribution of Number Of words’)

Source : Image by Author

· I have removed those articles whose length is less than 60 words.

· Word_length distribution is right-skewed.

· Most of the articles have around 150 words.

· Reuters articles that contain fact or stock information have fewer words.

2. Frequent words in Reuters Articles

In this section, we count the words present in the articles and analyze the result. We analyze the word count based on the N-gram method. N-gram is the occurrence of words based on its N value.

We will remove the stopwords from the textual data. Because stopwords are noise and not have much use in the analysis.

1. Most Frequent Unigram Words (N=1)

Let’s plot unigram words in a bar graph and word cloud for Unigram words.