A Complete Exploratory Data Analysis and Visualization for Text Data
By: Susan Li
Tags: NLP, Data Science, Plotly, Visualization, Python

It worked!

Univariate visualization with Plotly

Single-variable or univariate visualization is the simplest type of visualization which consists of observations on only a single characteristic or attribute. Univariate visualization includes histogram, bar plots and line charts.

The distribution of review sentiment polarity score

df['polarity'].iplot(

kind='hist',

bins=50,

xTitle='polarity',

linecolor='black',

yTitle='count',

title='Sentiment Polarity Distribution')

Figure 4

Vast majority of the sentiment polarity scores are greater than zero, means most of them are pretty positive.

The distribution of review ratings

df['Rating'].iplot(

kind='hist',

xTitle='rating',

linecolor='black',

yTitle='count',

title='Review Rating Distribution')

Figure 5

The ratings are in align with the polarity score, that is, most of the ratings are pretty high at 4 or 5 ranges.

The distribution of reviewers age

df['Age'].iplot(

kind='hist',

bins=50,

xTitle='age',

linecolor='black',

yTitle='count',

title='Reviewers Age Distribution')

Figure 6

Most reviewers are in their 30s to 40s.

The distribution review text lengths

df['review_len'].iplot(

kind='hist',

bins=100,

xTitle='review length',

linecolor='black',

yTitle='count',

title='Review Text Length Distribution')

Figure 7

The distribution of review word count

df['word_count'].iplot(

kind='hist',

bins=100,

xTitle='word count',

linecolor='black',

yTitle='count',

title='Review Text Word Count Distribution')

Figure 8

There were quite number of people like to leave long reviews.

For categorical features, we simply use bar chart to present the frequency.

The distribution of division

df.groupby('Division Name').count()['Clothing ID'].iplot(kind='bar', yTitle='Count', linecolor='black', opacity=0.8,

title='Bar chart of Division Name', xTitle='Division Name')

Figure 9

General division has the most number of reviews, and Initmates division has the least number of reviews.

The distribution of department

df.groupby('Department Name').count()['Clothing ID'].sort_values(ascending=False).iplot(kind='bar', yTitle='Count', linecolor='black', opacity=0.8,

title='Bar chart of Department Name', xTitle='Department Name')

Figure 10

When comes to department, Tops department has the most reviews and Trend department has the least number of reviews.

The distribution of class

df.groupby('Class Name').count()['Clothing ID'].sort_values(ascending=False).iplot(kind='bar', yTitle='Count', linecolor='black', opacity=0.8,

title='Bar chart of Class Name', xTitle='Class Name')

Figure 11

Now we come to “Review Text” feature, before explore this feature, we need to extract N-Gram features. N-grams are used to describe the number of words used as observation points, e.g., unigram means singly-worded, bigram means 2-worded phrase, and trigram means 3-worded phrase. In order to do this, we use scikit-learn’s CountVectorizer function.

First, it would be interesting to compare unigrams before and after removing stop words.

The distribution of top unigrams before removing stop words

top_unigram.py

Figure 12

The distribution of top unigrams after removing stop words

top_unigram_no_stopwords.py

Figure 13

Second, we want to compare bigrams before and after removing stop words.

The distribution of top bigrams before removing stop words

top_bigram.py

Figure 14

The distribution of top bigrams after removing stop words

top_bigram_no_stopwords.py

Figure 15

Last, we compare trigrams before and after removing stop words.

The distribution of Top trigrams before removing stop words

top_trigram.py

Figure 16

The distribution of Top trigrams after removing stop words

top_trigram_no_stopwords.py

Figure 17

Part-Of-Speech Tagging (POS) is a process of assigning parts of speech to each word, such as noun, verb, adjective, etc

We use a simple TextBlob API to dive into POS of our “Review Text” feature in our data set, and visualize these tags.

The distribution of top part-of-speech tags of review corpus

POS.py

Figure 18

Box plot is used to compare the sentiment polarity score, rating, review text lengths of each department or division of the e-commerce store.

What do the departments tell about Sentiment polarity

department_polarity.py

Figure 19

The highest sentiment polarity score was achieved by all of the six departments except Trend department, and the lowest sentiment polarity score was collected by Tops department. And the Trend department has the lowest median polarity score. If you remember, the Trend department has the least number of reviews. This explains why it does not have as wide variety of score distribution as the other departments.

What do the departments tell about rating

rating_division.py

Figure 20

Except Trend department, all the other departments’ median rating were 5. Overall, the ratings are high and sentiment are positive in this review data set.

Review length by department

length_department.py

Figure 21

The median review length of Tops & Intimate departments are relative lower than those of the other departments.

Bivariate visualization with Plotly

Bivariate visualization is a type of visualization that consists two features at a time. It describes association or relationship between two features.

Distribution of sentiment polarity score by recommendations

polarity_recommendation.py

Figure 22

It is obvious that reviews have higher polarity score are more likely to be recommended.

Distribution of ratings by recommendations

rating_recommendation.py

Figure 23

Recommended reviews have higher ratings than those of not recommended ones.

Distribution of review lengths by recommendations

review_length_recommend.py

Figure 24

Recommended reviews tend to be lengthier than those of not recommended reviews.

2D Density jointplot of sentiment polarity vs. rating

sentiment_polarity_rating.py

Figure 24

2D Density jointplot of age and sentiment polarity

age_polarity.py

Figure 25

There were few people are very positive or very negative. People who give neutral to positive reviews are more likely to be in their 30s. Probably people at these age are likely to be more active.

Finding characteristic terms and their associations

Sometimes we want to analyzes words used by different categories and outputs some notable term associations. We will use scattertext and spaCy libraries to accomplish these.

First, we need to turn the data frame into a Scattertext Corpus. To look for differences in department name, set the category_col parameter to 'Department Names' , and use the review present in the Review Text column, to analyze by setting the text col parameter. Finally, pass a spaCy model in to the nlp argument and call build() to construct the corpus.

Following are the terms that differentiate the review text from a general English corpus.