Introducing TextBlob. A Python library for processing textual... | by Susan Li | Towards Data Science

As an NLP library for Python, TextBlob has been around for a while, after hearing many good things about it such as part-of-speech tagging and sentiment analysis, I decided to give it a try, therefore, this is the first time I am using TextBlob to perform natural language processing tasks.
The Yelp dataset is a subset of its businesses, reviews, and user data for use in personal, educational, and academic purposes. Available as JSON files, we will only use yelp_academic_dataset_review.json and yelp_academic_dataset_user.json and can be downloaded from here.
The data sets are in JSON format, to be able to read in pandas data frame, we load JSON data first, then normalize semi-structured JSON data into a flat table, then use to_parquet to write the table to the binary parquet format. Later when we need it, we load a parquet object from the file path, returning a pandas Data Frame.
The following process gives us two data tables, user and review.
We merge user table and review table, use suffix to deal with the same column names, and remove zero stars.
user_review = (review.merge(user, on='user_id', how='left', suffixes=['', '_user']).drop('user_id', axis=1))user_review = user_review[user_review.stars > 0]
x=user_review['stars'].value_counts()x=x.sort_index()plt.figure(figsize=(10,6))ax= sns.barplot(x.index, x.values, alpha=0.8)plt.title("Star Rating Distribution")plt.ylabel('count')plt.xlabel('Star Ratings')rects = ax.patcheslabels = x.valuesfor rect, label in zip(rects, labels):    height = rect.get_height()    ax.text(rect.get_x() + rect.get_width()/2, height + 5, label, ha='center', va='bottom')plt.show();
Good to know that most of the reviews star ratings are pretty high, and not many terrible reviews. Obvious, there’s an incentive for businesses to solicit as many good reviews as possible.
fig, axes = plt.subplots(ncols=2, figsize=(14, 4))user_review.year.value_counts().sort_index().plot.bar(title='Reviews per Year', ax=axes[0]);sns.lineplot(x='year', y='stars', data=user_review, ax=axes[1])axes[1].set_title('Stars per year');
user_review.member_yrs.value_counts()
Yelp was founded in 2004, there were over 4,000 people have been Yelp members since then in our data.
Let’s have a look a sample review:
review_sample = user_review.text.sample(1).iloc[0]print(review_sample)
Let’s check the polarity of this sample review. Polarity ranges from -1 (most negative) to 1 (most positive).
TextBlob(review_sample).sentiment
The above review has a polarity of about -0.06, meaning it is slightly negative, and a subjectivity of about 0.56, meaning it is fairly subjective.
To proceed faster, we will sample 1 million reviews from our current data, and add a new column for polarity.
sample_reviews = user_review[['stars', 'text']].sample(1000000)def detect_polarity(text):    return TextBlob(text).sentiment.polaritysample_reviews['polarity'] = sample_reviews.text.apply(detect_polarity)sample_reviews.head()
First several rows look good, stars and polarity are in line with each other, meaning the higher the star, the higher the polarity, as it should be.
num_bins = 50plt.figure(figsize=(10,6))n, bins, patches = plt.hist(sample_reviews.polarity, num_bins, facecolor='blue', alpha=0.5)plt.xlabel('Polarity')plt.ylabel('Count')plt.title('Histogram of polarity')plt.show();
Most polarity scores are above zero, meaning most of the reviews are positive sentiment in the data, this is in line with the star rating distribution we discovered earlier.
plt.figure(figsize=(10,6))sns.boxenplot(x='stars', y='polarity', data=sample_reviews)plt.show();
In general, this is as good as we’d expect. Let’s investigate deeper and see whether we can find anything interesting or outlier.
Reviews that have the lowest polarity:
sample_reviews[sample_reviews.polarity == -1].text.head()
Reviews that have the lowest star ratings:
sample_reviews[sample_reviews.stars == 1].text.head()
They all look as we expect what the negative reviews are.
Reviews that have lowest polarity (most negative sentiment) but with a 5-star:
sample_reviews[(sample_reviews.stars == 5) & (sample_reviews.polarity == -1)].head(10)
Reviews that have the highest polarity (most positive sentiment) but with a 1-star:
sample_reviews[(sample_reviews.stars == 1) & (sample_reviews.polarity == 1)].head(10)
Both tables look weird. Apparently, some polarity does not agree with its associate rating. Why is that?
After digging a bit more, it turns out, TextBlob goes along finding words and phrases it can assign polarity and subjectivity to, and it averages them all together for longer text, such as our Yelp reviews.
Want to learn how TextBlob calculates polarity and subjectivity, this article from Aaron Schumacher has given a simple yet clear explanation.
I enjoyed learning and playing with TextBlob. I realized that TextBlob can be used to accomplish many other NLP tasks such as part-of-speech tagging, noun phrase extraction, classification, translation, and more, and we will get our hands dirty on them later.
Jupyter notebook can be found on Github. Enjoy the rest of the week!
References:
TextBlob documentation
Book: Hands-on Machine Learning for Algorithmic Trading