Exploratory text analysis in Python | by Zolzaya Luvsandorj | Towards Data Science

Why do we do exploratory data analysis before we build a model? I would say ‘to understand the data better so that we preprocess the data in a suitable way and choose an appropriate modelling technique’. This necessity to understand data is still relevant when working with text data. This post is the first of the three sequential posts on steps to build a sentiment classifier. In this post, we will look at one way to conduct exploratory data analysis on text, or exploratory text analysis for brevity.
Before we dive in, let’s take a step back and look at the bigger picture first. CRISP-DM methodology outlines the process flow for a successful data science project. In the diagram below, 2–4th stages of data science project are shown. In data understanding stage, exploratory data analysis is one of the key tasks.
When working on a data science project, it is not unusual to be going back and forth between stages rather than linearly progressing. This is because ideas and questions come up in subsequent stages and you want to go back a stage or two to try out the idea or find the answer to the question. The pink arrows are not in the official CRISP-DM, however, I think these are often necessary. In fact, we will be doing a bit of data preparation in this post for the purpose of exploratory text analysis. For those who are interested to learn more about CRISP-DM, this is a nice short introduction and this resource provides a more detailed explanation.
This post assumes that the reader (👀 yes, you!) has access to and is familiar with Python including installing packages, defining functions and other basic tasks. If you are new to Python, this is a good place to get started.
I have tested the scripts in Python 3.7.1 in Jupyter Notebook.
Let’s make sure you have the following libraries installed before we start:◼️ Data manipulation/analysis: numpy, pandas◼️ Data partitioning: sklearn◼️ Text preprocessing/analysis: nltk◼️ Visualisation: matplotlib, seaborn
Once you have nltk installed, please make sure you have downloaded ‘punkt’, ‘stopwords’ and ‘wordnet’ from nltk with the script below:
import nltknltk.download('punkt') # for sent_tokenizenltk.download('stopwords') nltk.download('wordnet') # for WordNetLemmatizer
If you have already downloaded, running this will notify you of so.
Now, we are ready to import all the packages:
# Setting random seedseed = 123# Data manipulation/analysisimport numpy as npimport pandas as pd# Data partitioningfrom sklearn.model_selection import train_test_split# Text preprocessing/analysisimport refrom nltk import word_tokenize, sent_tokenize, FreqDistfrom nltk.util import ngramsfrom nltk.corpus import stopwordsfrom nltk.stem import WordNetLemmatizerfrom nltk.tokenize import RegexpTokenizer# Visualisationimport matplotlib.pyplot as pltimport seaborn as snssns.set(style="whitegrid", context='talk',         palette=['#D44D5C', '#43AA8B'])
Takeaways or notes will be accompanied with 🍀 when exploring.
We will use IMDB movie reviews dataset. You can download the dataset here and save it in your working directory. Once saved, let’s import it to Python:
sample = pd.read_csv('IMDB Dataset.csv')print(f"{sample.shape[0]} rows and {sample.shape[1]} columns")sample.head()
From looking at the head of the data, we can instantly see that there are html tags in the second record.
🍀 Inspect html tags further and see how prevalent they are.
Let’s look at the split between sentiments:
sample['sentiment'].value_counts()
Sentiment is evenly split in the sample data. Before we start exploratory text analysis, let’s first partition the data into two sets: train and test. We will set aside 5000 cases for testing:
# Split data into train & testX_train, X_test, y_train, y_test = train_test_split(sample['review'], sample['sentiment'], test_size=5000, random_state=seed,                                                     stratify=sample['sentiment'])# Append sentiment back using indicestrain = pd.concat([X_train, y_train], axis=1)test = pd.concat([X_test, y_test], axis=1)# Check dimensionsprint(f"Train: {train.shape[0]} rows and {train.shape[1]} columns")print(f"{train['sentiment'].value_counts()}\n")print(f"Test: {test.shape[0]} rows and {test.shape[1]} columns")print(test['sentiment'].value_counts())
By specifying the stratify argument, we ensured that sentiment is evenly split in both sets.
We will be using train for the exploratory text analysis in this post. Once we explore the training dataset, it may be useful to check the key characteristics of test set separately. Ideally, both sets should be representative of the underlying population. Let’s inspect the head of the training dataset:
train.head()
Alright, we are all set to explore! ✨
One way to guide exploratory data analysis is to write down questions that you are interested in and answer them with the data. Finding answers to questions often leads to other questions that you may want to explore. Here are some example questions we could answer:
📋 2.1. Warm-up:* How many strings are there?* What are the most common strings?* How do the shortest strings look like?* How do the longest strings look like?
📋 2.2. Tokens:
💡 Token is a sequence of characters. Tokens are often loosely referred to as words.💡 Tokenisation is a process of splitting a document into tokens and sometimes also throwing away certain characters such as punctuation. Example: Tokenisation turns ‘This movie was awesome’ into 4 tokens: [‘This’, ‘movie’, ‘was’, ‘awesome’]
* How many tokens are there?* How many unique tokens are there?* What is the average number of characters per token?
📋 2.3. Stop words:
💡 Stop words are common words which provide little to no value to the meaning of the text. Example: the, a, and.
* What are the most common stop words?* What other words occur so often such that could be added to stop words?
📋 2.4. Common word n-grams:
💡 Word n-grams are all combinations of adjacent n words in text data. Example: Bigrams in ‘This movie was awesome’ are [‘This movie’, ‘movie was’, ‘was awesome’]
* What are the most common tokens?* What are the most common bigrams?* What are the most common trigrams?* What are the most common fourgrams?
📋 2.5. Documents:
💡 Document is a text record. Example: each movie review is a document.💡 Corpus is a collection of documents. Simply put, text data is a corpus. Example: we can refer training data as training corpus.
* What is the average number of sentences per document?* What is the average number of tokens per document?* What is the average number of characters per document?* What is the average number of stop words per document?* How do the answers differ by sentiment?
Now, it’s time to find answers to these questions! 😃
Let’s merge all the reviews into one string and then split it into sub-strings (here onward referred to as strings) on white space. This ensures the corpus is minimally altered (e.g. keeping punctuation as it is) for this warm-up exploration:
# Prepare training corpus into one giant stringtrain_string = " ".join(X_train.values)print(f"***** Extract of train_string ***** \n{train_string[:101]}", "\n")# Split train_corpus by white spacesplits = train_string.split()  print(f"***** Extract of splits ***** \n{splits[:18]}\n")
✏️ 2.1.1. How many strings are there?
print(f"Number of strings: {len(splits)}")print(f"Number of unique strings: {len(set(splits))}")
There are over 10 million strings in the training corpus with around 410 thousand unique strings. This gives us initial ballpark figures. We will get a view on how these numbers look for tokens after we tokenise properly.
✏️ 2.1.2. What are the most common strings?
Let’s prepare frequency distribution for each string to answer the question:
freq_splits = FreqDist(splits)print(f"***** 10 most common strings ***** \n{freq_splits.most_common(10)}", "\n")
It is not surprising to see that most common strings are stop words. We will explore stop words further later in section 2.3. Stop words.
🍀 Remove stop words before looking at common tokens and n-grams.
✏️ 2.1.3. How do the shortest strings look like?
Let’s define short string as strings with a length of less than 4 characters and check their frequency:
short = set(s for s in splits if len(s)<4)short = [(s, freq_splits[s]) for s in short]short.sort(key=lambda x:x[1], reverse=True)short
Many short strings appear to be stop words but there are also numbers and other short words.
🍀 There are numbers in different forms: 3, 2nd, 70s, 90% — we need to decide whether to drop or keep them.
🍀 There is case variation: ‘the’, ‘The’, ‘THE’ - these need to be normalised.
Because we haven’t tokenised yet, some strings currently contain punctuation that is attached to a word. As a result, otherwise same words are considered as different as shown in this example:
🍀 Discarding punctuation will be helpful to further normalise words.
✏️ 2.1.4. How do the longest strings look like?
Let’s define long strings as 16+ characters long and repeat the process.
long = set(s for s in splits if len(s)>15)long = [(s, freq_splits[s]) for s in long]long.sort(key=lambda x:x[1], reverse=True)long
The frequency of long string looks much lower than short strings which is not too surprising. Long strings look quite interesting and there are a few key takeaways:
🍀 There is American and British spelling of the same word: ‘characterization’ vs ‘characterisation’. After a quick check using the script below, American spelling looks more dominant for this word. Quantifying how prevalent the two spelling in the entire training corpus is a bit tricky.
print(f"characterisation: {sum([c for s, c in long if re.match(r'characterisation*', s.lower())])} strings")print(f"characterization: {sum([c for s, c in long if re.match(r'characterization*', s.lower())])} strings")
🍀 There are hyphenated words: ‘thought-provoking’, ‘post-apocalyptic’, ‘under-appreciated’ and ‘film--specially’ (this one with double hyphens). If we tokenise on white space or punctuation, these strings will be split into separate words. For most cases, this will conserve the gist of the sentence. If we keep hyphenated words as they are, they won’t be as common and consequently removed as rare words.
🍀 There are words combined with other punctuation (some due to lack of space): ‘actors/actresses’, ‘Mission:Impossible’, “actors(HA!HA!HA!)...they’re”, ‘different:actually,Bullock’. It would be good to separate these cases into separate words when tokenising. So probably tokenising based on white space or punctuation is a good idea.
🍀 There are websites and email addresses: ‘/>www.ResidentHazard.com’, ‘http://www.PetitionOnline.com/gh1215/petition.html’, ‘IAMASEAL2@YAHOO.COM’’. However, there doesn’t seem to be many.
🍀 There are outlaw words that repeats the same character more than twice: “Booooring.......Don’t”, ‘NOOOOOOIIIISE!),’. If you know the correct term for these elongated words, I would love to find out. Until then, we will refer them as ‘outlaw words’. These outlaw cases seem to appear very rarely.
There are other interesting findings we could add to these but these are good starters for now.
It’s important to understand whether these cases that we just explored are prevalent enough to justify additional preprocessing steps (i.e. longer run time). It’s useful to experiment to see if adding an additional preprocessing step improves model performance. We will do a bit of this in the third post.
✏️ 2.1.5. Following up questions emerged so far
We have answered all 4 warm up questions! While looking for answers, we have gathered even more questions. Before we jump to the next set of predefined questions, let’s quickly follow up some of the questions that have came up on the go.
◼️ How frequent is html tags? This question emerged when we were looking at head of the sample data in section 1. Data. The example html tag: ‘<br /><br />’ would have been split into three strings: ‘<br’, ‘/><br’ and ‘/>’ when we split the data based on white space. On side note, the <br> tag seems to be used to break lines. Let’s roughly gauge how prevalent the html tags are.
All follow up questions are similar since we are going to assess the frequency of a particular type of string. To avoid repeating ourselves, let’s make a function.
def summarise(pattern, strings, freq):    """Summarise strings matching a pattern."""    # Find matches    compiled_pattern = re.compile(pattern)    matches = [s for s in strings if compiled_pattern.search(s)]        # Print volume and proportion of matches    print("{} strings, that is {:.2%} of total".format(len(matches), len(matches)/ len(strings)))        # Create list of tuples containing matches and their frequency    output = [(s, freq[s]) for s in set(matches)]    output.sort(key=lambda x:x[1], reverse=True)        return output# Find strings possibly containing html tagsummarise(r"/?>?w*<|/>", splits, freq_splits)
If we scroll through the output, there aren’t many html tags other than the <br> tag and a few cases of <i> tag.
🍀 If we remove punctuation when tokenising, ‘/><br’ and ‘<br’ will become ‘br’ and we could probably add ‘br’ to stop words.
◼ ️How frequent are numbers? We found some instances of numbers from section 2.1.3. Short strings. Let’s see how frequent they are with the following script:
summarise(r"\d", splits, freq_splits)
Strings containing numbers are infrequent. In the context of movie reviews, it’s hard to intuitively make sense of how numbers would be useful. 10/10 could be an indication of positive sentiment but what can we infer from numbers such as 4, 80’s and 20th? We will drop numbers when tokenising.
Depending on the timeline of a project, you may not have enough time to try all the interesting ideas. In that case, it’s handy to keep a list of nice-to-try items which you could experiment with when you have time. We will add the following tasks to that list: 1) Keep and convert numbers to text2) Create a feature indicating whether a review contains numbers or not
◼ ️How frequent are hyphenated words? We saw hyphenated words when inspecting long strings in section 2.1.4.Let’s see how frequent they are:
summarise(r"\w+-+\w+", splits, freq_splits)
Roughly less than 1% of strings contained hyphenated words. From glancing through the hyphenated words, it makes more sense to separate them out to keep the data simple. For example: We should tokenise ‘camera-work’ to 2 tokens: [‘camera’, ‘work’] instead of 1 token: [‘camera-work’]. We could add ‘keep hyphenated words as they are’ to the list of nice-to-try items.
◼ ️How frequent are words combined by other punctuation? Much like the previous question, we saw these cases during long string exploration. Let’s see how frequent they are:
summarise(r"\w+[_!&/)(<\|}{\[\]]\w+", splits, freq_splits)
Not too frequent, these ones definitely need to be separated.
◼ ️How frequent are outlaw words? We saw elongated words like ‘NOOOOOOIIIISE!),’ earlier. Let’s see how prevalent they are:
def find_outlaw(word):    """Find words that contain a same character 3+ times in a row."""    is_outlaw = False    for i, letter in enumerate(word):        if i > 1:            if word[i] == word[i-1] == word[i-2] and word[i].isalpha():                is_outlaw = True                break    return is_outlawoutlaws = [s for s in splits if find_outlaw(s)]print("{} strings, that is {:.2%} of total".format(len(outlaws), len(outlaws)/ len(splits)))outlaw_freq = [(s, freq_splits[s]) for s in set(outlaws)]outlaw_freq.sort(key=lambda x:x[1], reverse=True)outlaw_freq
These are not worth correcting for because there too few cases.
That was the last follow up question! We have learned a little bit about the data. Hope you are feeling warmed up. 💦 You may have noticed how easily we could keep expanding our questions and keep exploring? In the interest of time, we will stop this section here and try to keep the next sections as succinct as possible. Otherwise, this post will be over a few hours long. 💤
Let’s answer these two questions at one go:
✏️ 2.2.1. How many tokens are there?✏️ 2.2.2. How many unique tokens are there?
We have to tokenise the data first. Recalling from earlier exploration, it seemed best to drop punctuation and numbers when tokenising. With this in mind, let’s tokenise the text into alphabetic tokens:
tokeniser = RegexpTokenizer("[A-Za-z]+")tokens = tokeniser.tokenize(train_string)print(tokens[:20], "\n")
Now we have tokenised, we can answer the first two questions:
print(f"Number of tokens: {len(tokens)}")print(f"Number of unique tokens: {len(set(tokens))}")
There are over 10 million tokens in the training data with around 122 thousand unique tokens. Currently, ‘Watch’, ‘watch’ and ‘watching’ are counted as different tokens. Hmm, wouldn’t it be great if we could normalise them to ‘watch’ and count them as one unique token? If we normalise, the number of unique tokens would be lower. Let’s quickly do two things: convert all tokens into lowercase and lemmatise them:
lemmatiser = WordNetLemmatizer()tokens_norm = [lemmatiser.lemmatize(t.lower(), "v") for t in tokens]print(f"Number of unique tokens: {len(set(tokens_norm))}")
Great, the number of unique tokens dropped by about 30%.
📌 Exercise: If you are keen and have time, instead of combining both steps like above, try separating out and see how the number of unique tokens change at each step. For instance, you could first convert tokens to lowercase, and check the number, and then lemmatise and check the number again. If you change the order of these two operations, is the final number of unique tokens different than 82562? Why would that be?
👂 Psst, I will show another way to lemmatise in the next post when preprocessing the text for the model.
✏️ 2.2.3. What is the average number of characters per token?
Let’s find out the average token length and inspect its the distribution:
# Create list of token lengths for each tokentoken_length = [len(t) for t in tokens]# Average number of characters per tokenprint(f"Average number of characters per token: {round(np.mean(token_length),4)}")# Plot distributionplt.figure(figsize=(12, 12))sns.countplot(y=token_length)plt.title("Counts of token length", size=20);
There are a few tokens that is very long but also very rare. Let’s look at the exact counts for those longer than 10 characters:
pd.DataFrame(data=token_length, columns=['length']).query("length>10").value_counts()
17+ characters long words are infrequent. Let’s inspect some of them:
[t for t in tokens if len(t)>=20]
Interesting, some are valid long words whereas some are long because they lack white space or outlaw words (i.e. elongated).
🍀 When preprocessing, we should make sure that very rare tokens like these ones are dropped such that they won’t create separate columns when vectorising tokens into a matrix.
✏️ 2.3.1. What are the most frequent stop words?
Let’s first inspect all stop words:
stop_words = stopwords.words("english")print(f"There are {len(stop_words)} stopwords.\n")print(stop_words)
At the time of writing this post, there are 179 stop words. The list of stop words could extend in future. It appears that we could extend the stop words to include a few more. In fact, I have proposed on Github to add generic stop words from below to the list of English stop words in nltk. Let’s also make sure to add a custom stop word ‘br’ to the list:
stop_words.extend(["cannot", "could", "done", "let", "may" "mayn",  "might", "must", "need", "ought", "oughtn", "shall", "would", "br"])print(f"There are {len(stop_words)} stopwords.\n")
Now, let’s check what are the most common stop words:
freq_stopwords = [(sw, tokens_norm.count(sw)) for sw in stop_words]freq_stopwords.sort(key=lambda x: x[1], reverse=True)freq_stopwords[:10]
The frequency is really high (duh, I mean they are stop words, of course they will be frequent 😈), especially for ‘be’ and ‘the’. Wouldn’t it be interesting to find out what proportion of tokens are stop words? Let’s quickly check:
n_stopwords = len([t for t in tokens_norm if t in stop_words])print(f"{n_stopwords} tokens are stop words.")print(f"That is {round(100*n_stopwords/len(tokens_norm),2)}%.")
About half of the tokens are stop words. 💭
✏️2.3.2. What other words occur so often such that could be added to stop words?
We will answer this question when we look at common tokens soon.
It’s time to find out common n-grams. Let’s answer all four questions together:
✏️ 2.4.1–4. What are the most common tokens, bigrams, trigrams and fourgrams?
Firstly, let’s remove stop words:
tokens_clean = [t for t in tokens_norm if t not in stop_words]print(f"Number of tokens: {len(tokens_clean)}")
This is the remaining 49% of tokens. Now, we can inspect the common tokens (i.e. unigrams), bigrams, trigrams and fourgrams:
def preprocess_text(text):    """Preprocess text into normalised tokens."""    # Tokenise words into alphabetic tokens    tokeniser = RegexpTokenizer(r'[A-Za-z]{2,}')    tokens = tokeniser.tokenize(text)        # Lowercase and lemmatise     lemmatiser = WordNetLemmatizer()    lemmas = [lemmatiser.lemmatize(token.lower(), pos='v') for token in tokens]        # Remove stopwords    keywords= [lemma for lemma in lemmas if lemma not in stop_words]    return keywordsdef get_frequent_ngram(corpus, ngram, n=20):    """Find most common n n-grams tokens."""    # Preprocess each document    documents = [preprocess_text(document) for document in corpus]        # Find ngrams per document    n_grams = [list(ngrams(document, ngram)) for document in documents]        # Find frequency of ngrams    n_grams_flattened = [item for sublist in n_grams for item in sublist]    freq_dist = FreqDist(n_grams_flattened)    top_freq = freq_dist.most_common(n)    return pd.DataFrame(top_freq, columns=["ngram", "count"])# Get frequent ngrams for all 4for i in range(1,5):    mapping = {1:"uni", 2:"bi", 3:"tri", 4:"four"}    plt.figure(figsize=(12,10))    sns.barplot(x="count", y="ngram", data=get_frequent_ngram(train['review'], i))    plt.title(f"Most common {mapping[i]}grams");
The word ‘film’ and ‘movie’ looks quite frequent compared to the other frequent words. The answer to question 2.3.2. is to potentially add ‘movie’, and ‘film’ to stop words. It’s interesting to see frequent bigrams, trigrams and fourgrams. As we increase n, the frequency drops as expected. Bigrams may be potentially useful but trigrams and fourgrams are not frequent enough relative to the token frequency.
Let’s answer these questions together:
✏️ 2.5.1. What is the average number of sentences per document?✏️ 2.5.2. What is the average number of tokens per document?✏️ 2.5.3. What is the average number of characters per document?✏️ 2.5.4. What is the average number of stop words per document?✏️ 2.5.5. How do answers to these questions differ by sentiment?
Firstly, we have to prepare the data:
# tokeniser = RegexpTokenizer("[A-Za-z]+")train["n_sentences"] = train["review"].apply(sent_tokenize).apply(len)train["tokens"] = train["review"].apply(tokeniser.tokenize)train["n_tokens"] = train["tokens"].apply(len)train["n_characters"] = train["review"].apply(len)train["n_stopwords"] = train["tokens"].apply(lambda tokens: len([t for t in tokens if t in stop_words]))train["p_stopwords"] = train["n_stopwords"]/train["n_tokens"]# Inspect headcolumns = ['sentiment', 'n_sentences', 'n_tokens', 'n_characters', 'n_stopwords', 'p_stopwords']train[columns].head()
Let’s check the descriptive statistics of the variables of interest:
train.describe()
We have the answers to the first four questions in this table. Now, let’s see if it differs by sentiment. If they differ significantly, we could use the variable as a feature to the model:
num_vars = train.select_dtypes(np.number).columnstrain.groupby("sentiment")[num_vars].agg(["mean", "median"])
From looking at central tendency, it doesn’t appear to differ by sentiment substantially. Just to be sure, let’s look at the distribution:
def plot_distribution(df, var, hue):    """Plot overlayed histogram and density plot per sentiment."""    fig, ax = plt.subplots(nrows=1, ncols=2, figsize=[16,4])        # Histogram    sns.histplot(data=df, x=var, hue=hue, bins=30, kde=False, ax=ax[0])    ax[0].set_title(f"Histogram for {var}")        # Density plot    sns.kdeplot(data=df, x=var, hue=hue, shade=True, ax=ax[1])    ax[1].set_title(f"Density plot for {var}");    # Plot for all numerical variablesfor var in num_vars:    plot_distribution(train, var, 'sentiment')
The distribution of variables seem pretty similar between sentiments. It’s unlikely that they will be useful as features, but we could always experiment. Maybe we could add this to a list of nice-to-try items?
Before we wrap up, let’s look at one last thing - whether common words differ by sentiment. Let’s prepare the data for each sentiment:
pos_documents = [preprocess_text(document) for document in train.loc[train['sentiment']=='positive', 'review']]pos_tokens = [item for sublist in pos_documents for item in sublist]pos_freq = FreqDist(pos_tokens)pos_common = [word for word, frequency in pos_freq.most_common(20)]print(f"***** 20 frequent tokens in positive reviews: *****\n{pos_common}\n")neg_documents = [preprocess_text(document) for document in train.loc[train['sentiment']=='negative', 'review']]neg_tokens = [item for sublist in neg_documents for item in sublist]neg_freq = FreqDist(neg_tokens)neg_common = [word for word, frequency in neg_freq.most_common(20)]print(f"***** 20 frequent tokens in negative reviews: *****\n{neg_common}\n")common = set(neg_common).union(pos_common)print(f"***** Their union: *****\n{common}\n")
The 3 most common tokens in both sentiments are ‘film’, ‘movie’ and ‘one’. Let’s look at their frequencies:
# Create a dataframe containing the common tokens and their frequencycommon_freq = pd.DataFrame(index=common, columns=["neg", "pos"])for token in common:    common_freq.loc[token, "pos"] = pos_freq[token]    common_freq.loc[token, "neg"] = neg_freq[token]common_freq.sort_values(by="pos", inplace=True)# Add ranks and rank differencecommon_freq['pos_rank'] = common_freq['pos'].rank()common_freq['neg_rank'] = common_freq['neg'].rank()common_freq['rank_diff'] = common_freq['neg_rank'] - common_freq['pos_rank']common_freq.sort_values(by='rank_diff', inplace=True)common_freq.head()
Now, time to visualise:
fig, ax =plt.subplots(1, 2, figsize=(16, 10))sns.barplot(x="pos", y=common_freq.index, data = common_freq, ax=ax[0])sns.barplot(x="neg", y=common_freq.index, data = common_freq, ax=ax[1])fig.suptitle('Top tokens and their frequency by sentiment');
Hmm, it’s interesting to see ‘film’ is more frequent than ‘movie’ in positive reviews. In negative reviews, it’s flipped. Maybe they shouldn’t be added to stop words after all despite their frequency. Let’s look at the chart again, but excluding these two common words:
rest = common_freq.index.drop(['film', 'movie'])fig, ax =plt.subplots(1, 2, figsize=(16, 10))sns.barplot(x="pos", y=rest, data = common_freq.loc[rest], ax=ax[0])sns.barplot(x="neg", y=rest, data = common_freq.loc[rest], ax=ax[1])fig.suptitle('Top tokens and their frequency by sentiment');
It’s intuitive to see that the word ‘great’, ‘well’ and ‘love’ are more frequent in positive reviews whereas ‘even’ and ‘bad’ are more frequent in negative reviews.
There are still many things to explore, but it’s time to wrap up! 🕛
Well done to you for making this far! 😎 Let’s summarise the key points:◼️ Remove punctuation and numbers when tokenising◼️ Normalise text (lowercase, lemmatise, etc)◼️ Enrich stop words with ‘br’ and other missing auxillary verbs◼️ Remove rare words
A list of nice-to-try items: ◼️ Convert British spelling to American spelling (or vice versa)◼️ Keep numbers and convert them to words◼️ Keep hyphenated words as they are when tokenising◼️ Include bigrams◼️ Add numerical features such as number of sentences, tokens, characters and stop words
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my post. Exploratory data analysis is an open-ended and subjective task. You may have noticed that we had to make many small choices when exploring and preprocessing. I hope this post gave you a taste of how to structure the analysis and showed example questions you could think about during the process. Having done some exploratory analysis, we are one step closer to building a model. In the next post, we will prepare the data for the model. Here are links to the other two posts of the series:◼️ Preprocessing text in Python◼️ Sentiment classification in Python
Here are links to the my other NLP-related posts:◼️ Simple wordcloud in Python(Below lists a series of posts on Introduction to NLP)◼️ Part 1: Preprocessing text in Python◼️ Part 2: Difference between lemmatisation and stemming◼️ Part 3: TF-IDF explained◼️ Part 4: Supervised text classification model in Python◼️ Part 5A: Unsupervised topic model in Python (sklearn)◼️ Part 5B: Unsupervised topic model in Python (gensim)
Bye for now 🏃💨