Analyzing Product Reviews With Natural Language Processing Toolkit (NLTK)
By: Emily Hsiao
Tags: Data Analysis, User Insight, Product Management, Mozilla, Mobile Apps

#2: Natural Language Toolkit (NLTK)

Step 1: Environment Setting

I ran my project on Jupyter notebook, a handy tool that allows me to write and run snippets of Python code on a web interface. Please follow the helpful links below.

Install Python 3 on MacOS: https://installpython3.com/mac/ (If you have trouble downloading XCode, just download Command Line Tool for XCode manually here.)

In your console, type the following commands to install notebook, numpy and nltk.

$ pip3 install notebook $ pip3 install numpy $ pip3 install nltk

Step 2: Create Your Project

$ jupyter notebook

Launch Jupyter notebook. And your browser should launch something like this:

(Optional) Click New -> Folder to create a folder to start with.

After that, move into the folder, click New -> Python 3 to create a new notebook. And you can get started!

Step 3: Import Libraries

For this project, we will focus primarily on cleaning data, removing stop words, and ranking keywords to create a word cloud. Press Shift + Enter to run the code.

import numpy as np

import nltk

import re from nltk.corpus import stopwords

from wordcloud import WordCloud

Step 4: Access & Read Raw Data

In a new code block, save your raw data under the folder you created earlier and assign the path to a String.

PATH = “./raw_data.txt”

Depend on your raw data format, you can specify data type, delimiter, and so on. Because my raw data (reviews) has lots of comma, I recommend saving it into a tab-separated .txt file. Run this block of code.

raw_data = np.genfromtxt(PATH, dtype = str, delimiter = “\t”)

This is how my raw data look like: a version code, a rating, a title (could be empty), and a multi-line comment.

Now, you can check if your data has been successfully read. Run the code below and you should get something like (a,b). a shows how many line you have and b indicates how many columns there are in the raw data.

raw_data.shape

Step 5: Clean the Data

First, remove the emojis. The code below lists out emojis in their unicode:

emoji_pattern = re.compile(“[“

u”\U0001F600-\U0001F64F” # emoticons

u”\U0001F300-\U0001F5FF” # symbols & pictographs

u”\U0001F680-\U0001F6FF” # transport & map symbols

u”\U0001F1E0-\U0001F1FF” # flags (iOS)

“]+”, flags=re.UNICODE) # source: stackoverflow

Then based on your data characteristics, convert all the words to lowercase and clean the irrelevant words and symbols too. I assigned the cleaned data into an array, tokenized_reviews, for our convenience to iterate through.

tokenized_reviews = []

for i, string in enumerate(raw_data[:, 3]):

raw_data[i, 3] = emoji_pattern.sub(‘’, string)

raw_data[i, 3] = raw_data[i, 3].lower().replace(“‘s “, ‘ ‘).strip()

raw_data[i, 3] = re.sub(“[\.,?!`？’;:)(&]”, ‘’, raw_data[i, 3])



tokenized_reviews.append(nltk.word_tokenize(raw_data[i, 3]))

Now run the block and you should see the preview box below showing your raw data without all the distractions like emoji unicodes and punctuations.

(Note: Sometimes the punctuations seem so similar but they are actually different in char code, so you can paste them to this tool to check.)

Step 6: Test the Famous NLTK

Go to the first block and download the wordlists. Don’t forget to run this block again after you do so. Stopwords are words like “you”, “i’m”, “do”, which do not carry too much semantic meanings.

nltk.download(‘stopwords’) stopwords = stopwords.words(‘english’)

Now, go to a new block and let it do its magic (removing the stopwords):

for i, review in enumerate(tokenized_reviews):

tokenized_reviews[i] = [word for word in review if word not in stopwords]

You can again check the preview screen to see what it did to the raw data. You can also add more words to the stopword list on the first block:

stopwords.extend([‘firefox’, ‘browser’,’focus’, ‘app’])

Again, don’t forget to run the first block again after you add new words. This step allow you to customize the result you care about. For example, as I was trying to understand what users felt about the focus browser, words related to product name itself are less relevant to me.

You can also adopt some NLP models at this step to: 1) convert the verbs from different tenses to their base form. 2) analyze the sentiment of the words, whether they are positive, negative, or neutral. 3) other functionalities that might help you learn about your user insights.

Step 7: Visualize the Reviews- Create a Word Cloud

This handy function provides a primary understanding for us. Since I did not apply any specifications to my data, the word cloud simply displays the frequencies of words appeared in reviews.