NLP Part 3 | Exploratory Data Analysis of Text Data | by Kamil Mysiak | Towards Data Science

This is a continuation of a three part series on NLP using python. Feel free to check out my other articles. (Part 1, Part 2)
Let’s get a better understanding of our newly cleansed dataset.Exploratory Data Analysis (EDA) is the process by which the data analyst becomes acquainted with their data to drive intuition and begin to formulate testable hypotheses. This process typically makes use of descriptive statistics and visualizations.
Let’s begin, as always, by importing the necessary libraries and opening our dataset.
import pandas as pdimport numpy as npimport nltkimport pickleimport pyLDAvis.sklearnfrom collections import Counterfrom textblob import TextBlobfrom nltk.tokenize import word_tokenizefrom nltk.probability import FreqDistfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.decomposition import LatentDirichletAllocation, NMFfrom wordcloud import WordCloud, ImageColorGeneratorimport matplotlib.pyplot as pltimport seaborn as sns%matplotlib inlinepd.options.mode.chained_assignment = Nonepd.set_option('display.max_colwidth', 100)with open('indeed_scrape_clean.pkl', 'rb') as pickle_file:    df = pickle.load(pickle_file)
If you recall from our previous tutorial, we went through a series of pre-processing steps to clean and prepare our data for analysis. Our final dataset contains numerous columns but the last column “lemmatized”, contained our final cleansed list of words. We are going to overwrite our existing dataframe because we are only interested in the “rating” and “lemmatized” columns.
df = df[['rating', 'lemmatized']]df.head()
Sentiment analysis is the process of determining the writer’s attitude or opinion ranging from -1 (negative attitude) to 1 (positive attitude). We’ll be using the TextBlob library to analyze sentiment. TextBlob’s Sentiment() function requires a string but our “lemmatized” column is currently a list. Let’s convert the list into a string.
df['lemma_str'] = [' '.join(map(str,l)) for l in df['lemmatized']]df.head()
Now we can pass the “lemma_str” column into the Sentiment() function to calculate sentiment. Since we have the “rating” column, we can validate how well the sentiment analysis was able to determine the writer’s attitude. That said, we do see obvious errors as rating #5 has a rating of 5but a fairly low sentiment.
df['sentiment'] = df['lemma_str'].apply(lambda x: TextBlob(x).sentiment.polarity)df.head()
When comparing a histogram of our sentiment, we can see that the vast majority of our derived sentiment rating is overwhelmingly positive. When we compare this against the “ratings” column, we can see a similar pattern emerge. Not only do we feel comfortable in the accuracy of the sentiment analysis but we can see that the overall employee attitude about the company is very positive. It is no wonder Google regularly makes the Forbes’s Best Places to Work list.
plt.figure(figsize=(50,30))plt.margins(0.02)plt.xlabel('Sentiment', fontsize=50)plt.xticks(fontsize=40)plt.ylabel('Frequency', fontsize=50)plt.yticks(fontsize=40)plt.hist(df['sentiment'], bins=50)plt.title('Sentiment Distribution', fontsize=60)plt.show()
x_rating = df.rating.value_counts()y_rating = x_rating.sort_index()plt.figure(figsize=(50,30))sns.barplot(x_rating.index, x_rating.values, alpha=0.8)plt.title("Rating Distribution", fontsize=50)plt.ylabel('Frequency', fontsize=50)plt.yticks(fontsize=40)plt.xlabel('Employee Ratings', fontsize=50)plt.xticks(fontsize=40)
plt.figure(figsize=(30,10))plt.title('Percentage of Ratings', fontsize=20)df.rating.value_counts().plot(kind='pie', labels=['Rating5', 'Rating4', 'Rating3', 'Rating2', 'Rating1'],                              wedgeprops=dict(width=.7), autopct='%1.0f%%', startangle= -20,                               textprops={'fontsize': 15})
polarity_avg = df.groupby('rating')['sentiment'].mean().plot(kind='bar', figsize=(50,30))plt.xlabel('Rating', fontsize=45)plt.ylabel('Average Sentiment', fontsize=45)plt.xticks(fontsize=40)plt.yticks(fontsize=40)plt.title('Average Sentiment per Rating Distribution', fontsize=50)plt.show()
Let’s create two additional features of “word_count” to determine the number of words per review and “review_len” to determine the number of letters per review.
df['word_count'] = df['lemmatized'].apply(lambda x: len(str(x).split()))df['review_len'] = df['lemma_str'].astype(str).apply(len)
Although the differences are not significantly large it seems the longest reviews based on the count of letters and words seem to be negative and neutral. It seems disgruntled employees typically provide significantly more detail in their reviews. This result is not uncommon as humans have a tendency to complain in detail but praise in brief. This can be further confirmed by examining the correlation matrix below. Both ratings and sentiment have a negative correlation with “review_len” and “word_count”. This would explain the inverse relationship as the count of letters and words per review increases the overall rating and sentiment decreases. However, once again the correlation is rather small nevertheless negative.
letter_avg = df.groupby('rating')['review_len'].mean().plot(kind='bar', figsize=(50,30))plt.xlabel('Rating', fontsize=35)plt.ylabel('Count of Letters in Rating', fontsize=35)plt.xticks(fontsize=40)plt.yticks(fontsize=40)plt.title('Average Number of Letters per Rating Distribution', fontsize=40)plt.show()
word_avg = df.groupby('rating')['word_count'].mean().plot(kind='bar', figsize=(50,30))plt.xlabel('Rating', fontsize=35)plt.ylabel('Count of Words in Rating', fontsize=35)plt.xticks(fontsize=40)plt.yticks(fontsize=40)plt.title('Average Number of Words per Rating Distribution', fontsize=40)plt.show()
correlation = df[['rating','sentiment', 'review_len', 'word_count']].corr()mask = np.zeros_like(correlation, dtype=np.bool)mask[np.triu_indices_from(mask)] = Trueplt.figure(figsize=(50,30))plt.xticks(fontsize=40)plt.yticks(fontsize=40)sns.heatmap(correlation, cmap='coolwarm', annot=True, annot_kws={"size": 40}, linewidths=10, vmin=-1.5, mask=mask)
Let’s take an in-depth look at the actual reviews themselves. What are the most common words? What are the most common words by rating? Answers to these questions will provide further insights into the opinions of Google’s employees.
NLTK has a great library named “FreqDist” which allows us to determine the count of the most common terms in our corpus. First, we need to convert our individual lists of tokenized reviews into a comprehensive list of iterable tokens which stores all the reviews together. Finally, we pass FreqDist() the “allwords” object and apply the “most_common(100)” function to obtain the 100 most common words.
words = df['lemmatized']allwords = []for wordlist in words:    allwords += wordlistprint(allwords)
mostcommon = FreqDist(allwords).most_common(100)wordcloud = WordCloud(width=1600, height=800, background_color='white').generate(str(mostcommon))fig = plt.figure(figsize=(30,10), facecolor='white')plt.imshow(wordcloud, interpolation="bilinear")plt.axis('off')plt.title('Top 100 Most Common Words', fontsize=100)plt.tight_layout(pad=0)plt.show()
mostcommon_small = FreqDist(allwords).most_common(25)x, y = zip(*mostcommon_small)plt.figure(figsize=(50,30))plt.margins(0.02)plt.bar(x, y)plt.xlabel('Words', fontsize=50)plt.ylabel('Frequency of Words', fontsize=50)plt.yticks(fontsize=40)plt.xticks(rotation=60, fontsize=40)plt.title('Frequency of 25 Most Common Words', fontsize=60)plt.show()
The results of the term frequency analysis certainly supports the overall positive sentiment of the reviews. Terms such as “Great”, “Work”, “People”, “Team”, “Company” point to a positive company environment where employees enjoy working together.
Based on the fact terms such as “work”, “Google”, “job” and “company” have such a high frequency in the corpus it might be a good idea to remove them (ie. add them to our stopwords) prior to analysis.
That said, a company can always improve therefore, let’s examine the most common words for each review rating.
It seems the most common words for reviews where the rating = 1 had something to do with the “Management”, “Manager”, “People”. We have to be careful when interpreting these results as there are only (2%) reviews with a rating of 1 based on our pie chart printed above.
group_by = df.groupby('rating')['lemma_str'].apply(lambda x: Counter(' '.join(x).split()).most_common(25))group_by_0 = group_by.iloc[0]words0 = list(zip(*group_by_0))[0]freq0 = list(zip(*group_by_0))[1]plt.figure(figsize=(50,30))plt.bar(words0, freq0)plt.xlabel('Words', fontsize=50)plt.ylabel('Frequency of Words', fontsize=50)plt.yticks(fontsize=40)plt.xticks(rotation=60, fontsize=40)plt.title('Frequency of 25 Most Common Words for Rating=1', fontsize=60)plt.show()
Reviews with a rating of 2 had a common theme of “manager”, “management”. Once again the rating distribution is very skewed but this does give us some clues on ways to improve the organization.
group_by_1 = group_by.iloc[1]words1 = list(zip(*group_by_1))[0]freq1 = list(zip(*group_by_1))[1]plt.figure(figsize=(50,30))plt.bar(words1, freq1)plt.xlabel('Words', fontsize=50)plt.ylabel('Frequency of Words', fontsize=50)plt.yticks(fontsize=40)plt.xticks(rotation=60, fontsize=40)plt.title('Frequency of 25 Most Common Words for Rating=2', fontsize=60)plt.show()
It is difficult to derive accurate insights from a “neutral” rating as the employee didn’t have anything overly positive or negative to say about the company. That said, it is interesting that “Management” has once again crept into the top 10 words. So far roughly 14% of the employees had a negative or neutral (didn’t have anything good or bad to say) about the management at Google. Words like “work” and “Google” seem to be skewing the distribution for all ratings, it would be a good idea to remove these words from future analysis.
Notice the “...”, we have some more data processing to perform. 😞
group_by_2 = group_by.iloc[2]words2 = list(zip(*group_by_2))[0]freq2 = list(zip(*group_by_2))[1]plt.figure(figsize=(50,30))plt.bar(words2, freq2)plt.xlabel('Words', fontsize=50)plt.ylabel('Frequency of Words', fontsize=50)plt.yticks(fontsize=40)plt.xticks(rotation=60, fontsize=40)plt.title('Frequency of 25 Most Common Words for Rating=3', fontsize=60)plt.show()
Rating of 4 and 5 had very similar terms as it seems employees enjoy their work, the people with whom they work, and value the environment/culture at Google. For example, “design”, “learning opportunities”, “people”, “time” and “team” all made an appearance. That said, we don’t see “Management” or “Manager” anywhere in the top 10 words. This is very insightful as it helps to validate the results from ratings 1, 2, and 3. Last but not least, these word frequencies (ie. ratings 4 & 5) have been derived from a very large number of reviews which only adds to the validity of these results; management is certainly an area of improvement.
group_by_3 = group_by.iloc[3]words3 = list(zip(*group_by_3))[0]freq3 = list(zip(*group_by_3))[1]plt.figure(figsize=(50,30))plt.bar(words3, freq3)plt.xlabel('Words', fontsize=50)plt.ylabel('Frequency of Words', fontsize=50)plt.yticks(fontsize=40)plt.xticks(rotation=60, fontsize=40)plt.title('Frequency of 25 Most Common Words for Rating=4', fontsize=60)plt.show()
group_by_4 = group_by.iloc[4]words4 = list(zip(*group_by_4))[0]freq4 = list(zip(*group_by_4))[1]plt.figure(figsize=(50,30))plt.bar(words4, freq4)plt.xlabel('Words', fontsize=50)plt.ylabel('Frequency of Words', fontsize=50)plt.yticks(fontsize=40)plt.xticks(rotation=60, fontsize=40)plt.title('Frequency of 25 Most Common Words for Rating=5', fontsize=60)plt.show()
Finally, let’s apply a few topic modeling algorithms to help derive specific topics or themes for our reviews. Before we have determined the topics for each rating we have to perform one additional processing step. Right now our data/words are still readable to us human beings whereas computers only understand numbers. We need to convert our text into numbers or vectors.
The CountVectorizer method of vectorizing tokens transposes all the words/tokens into features and then provides a count of occurrence of each word. The result is called a document term matrix, which you can see below.
First, we create the vectorizer object. Max_df=0.9 will remove words that appear in more than 90% of the reviews. Min_df=25 will remove words that appear in less than 25 reviews. Next, we create the spare matrix as the result of fit_transform(). Finally, we create a list of all the words/features. The result is our document term matrix. Each row represents individual employee reviews and counts of how many times each word/feature occurs in each review.
tf_vectorizer = CountVectorizer(max_df=0.9, min_df=25, max_features=5000)tf = tf_vectorizer.fit_transform(df['lemma_str'].values.astype('U'))tf_feature_names = tf_vectorizer.get_feature_names()doc_term_matrix = pd.DataFrame(tf.toarray(), columns=list(tf_feature_names))doc_term_matrix
Now that we have prepared our data for topic modeling, we’ll be using the Latent Dirichlet Allocation (LDA) approach to determine the topics present in our corpus. In our model, we are going to produce 10 individual topics (ie. n_components). Once the model is created let’s create a function to display the identified topics. Each topic will consist of 10 words. The function will have three required parameters; the LDA model, feature names from the document term matrix, and the number of words per topic.
lda_model = LatentDirichletAllocation(n_components=10, learning_method='online', max_iter=500, random_state=0).fit(tf)no_top_words = 10def display_topics(model, feature_names, no_top_words):    for topic_idx, topic in enumerate(model.components_):        print("Topic %d:" % (topic_idx))        print(" ".join([feature_names[i]                          for i in topic.argsort()[:-no_top_words - 1:-1]]))              display_topics(lda_model, tf_feature_names, no_top_words)
It certainly takes a little imagination to determine the topics produced by the LDA model. Since we know that “work”, “google” and “job” are very common works we can almost ignore them.
Topic 0: Good Design Processes
Topic 1: Great Work Environment
Topic 2: Flexible Work Hours
Topic 3: Skill Building
Topic 4: Difficult but Enjoyable Work
Topic 5: Great Company/Job
Topic 6: Care about Employees
Topic 7: Great Contractor Pay
Topic 8: Customer Service
Topic 9: ?
pyLDAvis is an interactive LDA visualization python library. Each circle represents a unique topic, the size of the circle represents the importance of the topic and finally, the distance between each circle represents how similar the topics are to each other. Selecting a topic/circle will reveal a horizontal bar chart displaying the 30 most relevant words for the topic along with the frequency of each word appearing in the topic and the overall corpus.
The relevance metric helps to distinguish words which are distinct/exclusive to the topic (λλ closer to 0.0) and words which have a high probability of being included in the selected topic (λλ closer to 1.0).
pyLDAvis.enable_notebook()panel = pyLDAvis.sklearn.prepare(lda_model, tf, tf_vectorizer, mds='tsne')panel
LDA isn’t the only approach to topic modeling. Let’s try another method named the Non-Negative Matrix Factorization (NMF) approach and see if our topics can be slightly more defined. Instead of using the simple CountVectorizer method to vectorize our words/tokens, we’ll use the TF-IDF (Term Frequency — Inverse Document Frequency) method. The TF-IDF method helps to bring down the weight/impact of high-frequency words (ie. “work”, “Google” and ‘job” in our case).
Much like the CountVectorizer method we first create the vectorizer object. Max_df=0.9 will remove words that appear in more than 90% of the reviews. Min_df=25 will remove words that appear in less than 25 reviews. Next, we create the spare matrix as the result of fit_transform(). Finally, we create a list of all the words/features.
tfidf_vectorizer = TfidfVectorizer(max_df=0.90, min_df =25, max_features=5000, use_idf=True)tfidf = tfidf_vectorizer.fit_transform(df['lemma_str'])tfidf_feature_names = tfidf_vectorizer.get_feature_names()doc_term_matrix_tfidf = pd.DataFrame(tfidf.toarray(), columns=list(tfidf_feature_names))doc_term_matrix_tfidf
nmf = NMF(n_components=10, random_state=0, alpha=.1, init='nndsvd').fit(tfidf)display_topics(nmf, tfidf_feature_names, no_top_words)
The topics produced via NMF seem to be much more distinct compared to LDA.
Topic 0: Fun Work Culture
Topic 1: Design Process
Topic 2: Enjoyable Job
Topic 3:
Topic 4: Great Experience
Topic 5: Perks
Topic 6: Learning Opportunities
Topic 7: Great Company/Job
Topic 8: Contractor Employee Experience
Topic 9: Management
Let’s add both the LDA and NMF topics into our dataframe for further analysis. Let’s also remap the integer topics into our subjectively derived topic labels.
nmf_topic_values = nmf.transform(tfidf)df['nmf_topics'] = nmf_topic_values.argmax(axis=1)lda_topic_values = lda_model.transform(tf)df['lda_topics'] = lda_topic_values.argmax(axis=1)lda_remap = {0: 'Good Design Processes', 1: 'Great Work Environment', 2: 'Flexible Work Hours', 3: 'Skill Building', 4: 'Difficult but Enjoyable Work', 5: 'Great Company/Job', 6: 'Care about Employees', 7: 'Great Contractor Pay', 8: 'Customer Service', 9: 'Unknown1'}df['lda_topics'] = df['lda_topics'].map(lda_remap)nmf_remap = {0: 'Fun Work Culture', 1: 'Design Process', 2: 'Enjoyable Job', 3: 'Difficult but Enjoyable Work',              4: 'Great Experience', 5: 'Perks', 6: 'Learning Opportunities', 7: 'Great Company/Job',              8: 'Contractor Employee Experience', 9: 'Management'}df['nmf_topics'] = df['nmf_topics'].map(nmf_remap)
Examining the frequency of topics produced by NMF we can see that the first 5 topics show up at a relatively similar frequency. Keep in mind these are the topics across all reviews (positive, neutral, and negative) and if you recall our dataset is negatively skewed as the majority of the reviews are positive.
nmf_x = df['nmf_topics'].value_counts()nmf_y = nmf_x.sort_index()plt.figure(figsize=(50,30))sns.barplot(nmf_x, nmf_y.index)plt.title("NMF Topic Distribution", fontsize=50)plt.ylabel('Review Topics', fontsize=50)plt.yticks(fontsize=40)plt.xlabel('Frequency', fontsize=50)plt.xticks(fontsize=40)
Let’s split our data and examine the topics for the negative reviews based on ratings of 1 and 2. It is interesting to see that despite the negative ratings the employees still overwhelmingly enjoyed their work, culture, and the company overall. It is very difficult to obtain an accurate perspective on the topics for negative reviews due to the skewness of our dataset (ie. the relatively small amount of negative reviews).
df_low_ratings = df.loc[(df['rating']==1) | (df['rating']==2)]nmf_low_x = df_low_ratings['nmf_topics'].value_counts()nmf_low_y = nmf_low_x.sort_index()plt.figure(figsize=(50,30))sns.barplot(nmf_low_x, nmf_low_y.index)plt.title("NMF Topic Distribution for Low Ratings (1 & 2)", fontsize=50)plt.ylabel('Frequency', fontsize=50)plt.yticks(fontsize=40)plt.xlabel('Review Topics', fontsize=50)plt.xticks(fontsize=40)
Since we have many more positive reviews the topics derived via NMF will be much more accurate. It seems contractor employees make up many of the reviews. Employees find an efficient design process, work which is difficult but enjoyable, and an overall happy sentiment towards Google.
df_high_ratings = df.loc[(df['rating']==4) | (df['rating']==5)]nmf_high_x = df_high_ratings['nmf_topics'].value_counts()nmf_high_y = nmf_high_x.sort_index()plt.figure(figsize=(50,30))sns.barplot(nmf_high_x, nmf_high_y.index)plt.title("NMF Topic Distribution for High Ratings (3 & 4)", fontsize=50)plt.ylabel('Frequency', fontsize=50)plt.yticks(fontsize=40)plt.xlabel('Review Topics', fontsize=50)plt.xticks(fontsize=40)
Based on the results obtained it seems Google’s employees are overwhelmingly happy working at Google. We see a negatively skewed distribution where 84% of employees had given Google a rating of 4 or 5 (out of a 1–5 Likert scale). A sentiment analysis confirmed these results even when employees who gave Google a rating between 2 and 3 had a positive average sentiment score.
As we dug a bit deeper into the data an interesting discovery was made which would need to be validated with additional data. When we observed the term/word frequencies per rating it seemed that terms around “managers/management” seemed to present themselves for ratings 1, 2, and 3. As mentioned earlier, the data was skewed as the majority of ratings were positive but it was interesting to see that employees who had a negative or a neutral rating seemed to mention “management” often.
On the other hand, employees who rated Google between 4 and 5 seemed to be using words such as “great”, “work”, “job”, “design”, “company”, “good”, “culture”, “people”, etc. These results were slightly confirmed with an examination of topics/themes in our corpus. An NMF analysis of topics determined that employees who rated Google with a 4 or 5 were eager to discuss the difficult but enjoyable work, great culture, design process. It was interesting to also see the absence of any mention of terms like “manager” and “management” which is also telling and helps to validate our previous insight.
Google continues to be a preferred employer of choice for many, as 84% of reviews were positive. That said, we identified a potential area of improvement which stemmed from Google’s managers and/or management techniques.
Can you think of any other EDA methods and/or strategies we could have explored? Posted them in the comments below. Also, if you have any constructive feedback or see a mistake I have made please call me out 😃