K-Means Clustering Chardonnay Reviews using Scikit-Learn & NLTK | by Eric Kleppen | Towards Data Science

Chardonnay is the most popular white wine in the world. The grape is easy to grow, and it’s known for being complex and adaptable to different techniques and flavors. For example, last night I tried a sweet, unoaked Chardonnay. How popular is Chardonnay? As described in the 2018 California Grape Acreage Report, of the 176,092 total acreage dedicated to white wine, Chardonnay consumed over half of it at 93,148 acres! The next highest variety is French Colombard, consuming 18,246 acres; measly by comparison. That is a lot of grapes!
Since I’m familiar with the wine reviews data set available on kaggle, I decided to load up a notebook and analyze the Chardonnays.
Can clustering help us identify relationships between the description and the rating?
In this article, I show how to use Scikit-Learn and the Natural Language Tool Kit to process, analyze and cluster the Chardonnay data. By using these techniques, I hope to see if there is a difference in themes or topics used in the wine reviews with above average ratings compared to the reviews of the wines with average or below ratings.
Since this article combines several techniques to analyze the Chardonnay reviews, we have quite a few things to import. Additionally, I have done some preliminary cleaning on the dataset to remove duplicates and nulls and stored it in an SQLite database:
#import dependenciesimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsimport refrom sklearn import preprocessing, decomposition, model_selection, metrics, pipelinefrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizerfrom sklearn.cluster import KMeansimport nltkfrom nltk.stem.wordnet import WordNetLemmatizerfrom nltk.corpus import stopwordsimport sqlite3from sqlite3 import Error#force output to display the full descriptionpd.set_option('display.max_colwidth', -1)#connect to database fileconn = sqlite3.connect('db\wine_data.sqlite')c = conn.cursor()#create dataframe from sql querydf = pd.read_sql("Select country, description, rating, price, title, variety from wine_data where variety = 'Chardonnay'", conn)#display the top 3 rowsdf.head(3)
Right from the get-go, I know I want to add a couple features: First, I’ll add word count column to store the number of words in each description. Second, I’ll add a binary field to signify whether or not the wine’s rating is above average. Before I can add those features, I’ll need do a bit of analysis to find the average rating.
Analyzing the word counts can help you decide whether or not you want to reduce the dataset. For example, looking at the data we see the minimum word count for a wine review is 3 words. Looking at reviews that have less than 15 words, the average rating is 82 in from a range of 80–100. This tells me that short reviews might be associated with lower ratings. Likewise, when I look at the other end of the distribution, I notice longer reviews might be associated with higher ratings.
Short reviews might be associated with lower ratings. Longer reviews might be associated with higher ratings.
#add a column for the word countdf['word_count'] = df['description'].apply(lambda x: len(str(x).split(" ")))print("Word Count Median: " + str(df['word_count'].median()))print(df['word_count'].describe())x = df['word_count']n_bins = 95plt.hist(x, bins=n_bins)plt.xlabel('Number of Words in Description')plt.ylabel('Frequency')plt.show()
#word counts less than 15wc15 = df.loc[df['word_count'] < 15]print(wc15.rating.median())print(wc15.rating.describe())#word counts greater than 70wc70 = df.loc[df['word_count'] > 70]print(wc70.rating.median())print(wc70.rating.describe())#plot the countsplt.figure(figsize=(14,4))sns.countplot(x ='rating', data = wc70).set_title("Rating Counts")
The data can be visualized to help us see the relationship between word count and rating.
Since I want to see if the descriptions can be used to differentiate if the wine has a higher than average rating, I have to find the average rating. Pandas describe function makes finding the stats simple. Looking at the data, we get an average rating of 88. Adding a binary value column to the data frame is easy using a list comprehension. If the rating is greater than 88, our new column will be 1. If the rating is 88 or less, the value will be 0 since the wine is not better than average.
If the rating is greater than 88, our new column will be 1. If the rating is 88 or less, the value will be 0 since the wine is not better than average.
print("Number of Unique Ratings: " + str(len(df['rating'].unique())))print("Rating Median: " + str(df['rating'].median()))print(df['rating'].describe())plt.figure(figsize=(14,4))sns.countplot(x='rating', data=df).set_title("Rating Counts")plt.show()#add column to flag records with rating greater than 88df['above_avg'] = [1 if rating > 88 else 0 for rating in df['rating']]
I perform several NLP techniques before clustering the data like removing stop words, punctuation and special characters, and normalizing the text. After the text has been processed, I am going to vectorize the text using the Scikit-Learn tf-idf vectorizer.
Before clustering, I want to remove stop words. Stop words are common words like “the” and “of.” Removing them from the descriptions allows the more relevant frequent words to stand out. I look at word frequency to gauge whether additional words should be added to the stopwords list. Additionally, I clean the descriptions using Regular Expressions to remove punctuation, tags and special characters, and then I lemmatize the words which reduces the word to a root form while maintaining that it is a real word. Lemmatisation is a technique for normalizing the text:
#create a list of stop wordsstop_words = set(stopwords.words("english"))#show how many words are in the list of stop wordsprint(len(stop_words))#179#construct a new list to store the cleaned textclean_desc = []for w in range(len(df.description)):    desc = df['description'][w].lower()        #remove punctuation    desc = re.sub('[^a-zA-Z]', ' ', desc)        #remove tags    desc = re.sub("&lt;/?.*?&gt;"," &lt;&gt; ",desc)        #remove special characters and digits    desc = re.sub("(\\d|\\W)+"," ",desc)        split_text = desc.split()        #Lemmatisation    lem = WordNetLemmatizer()    split_text = [lem.lemmatize(word) for word in split_text if not word in stop_words and len(word) >2]     split_text = " ".join(split_text)    clean_desc.append(split_text)
The TfidfVectorizer transforms the text into a vector space. To simplify the concept, imagine you have two sentences:
The dog is whiteThe cat is black
Converting the sentences to a vector space model would transform them in such a way that looks at the words in all sentences, and then represents the words in the sentence with a number. For example, if the word is in the sentence, it is a 1. If the word is not in the sentence, it is represented with a 0:
The dog cat is white blackThe dog is white = [1,1,0,1,1,0]The cat is black = [1,0,1,1,0,1]
TF-IDF stands for term frequency-inverse document frequency. It is a method for weighting the word value instead of simply counting it. It is used to determine how important a word is to a text within a collection documents. This functionality is useful for information retrieval like search engines and text mining. The TfidfVectorizer in Scikit-Learn converts a collection of raw documents to a matrix of TF-IDF features. It returns the matrix using the fit_transform method.
#TF-IDF vectorizertfv = TfidfVectorizer(stop_words = stop_words, ngram_range = (1,1))#transformvec_text = tfv.fit_transform(clean_desc)#returns a list of words.words = tfv.get_feature_names()
K-means clustering is a popular unsupervised learning algorithm that can be used to extract topics by grouping similar reviews together and producing a list of common words. I am going to try dividing the data into 21 clusters (n_clusters = 21) to see if I can discover themes that are common among high ratings and themes that are common among low ratings. Scikit-Learn makes it easy to apply k-means.
#setup kmeans clusteringkmeans = KMeans(n_clusters = 21, n_init = 17, n_jobs = -1, tol = 0.01, max_iter = 200)#fit the data kmeans.fit(vec_text)#this loop transforms the numbers back into wordscommon_words = kmeans.cluster_centers_.argsort()[:,-1:-11:-1]for num, centroid in enumerate(common_words):    print(str(num) + ' : ' + ', '.join(words[word] for word in centroid))
Using heat-maps, I can see how the ratings are clustered. I can also see if above average ratings clustered together compared to average or below ratings.
#add the cluster label to the data framedf['cluster'] = kmeans.labels_clusters = df.groupby(['cluster', 'rating']).size()fig, ax1 = plt.subplots(figsize = (26, 15))sns.heatmap(clusters.unstack(level = 'rating'), ax = ax1, cmap = 'Reds')ax1.set_xlabel('rating').set_size(18)ax1.set_ylabel('cluster').set_size(18)clusters = df.groupby(['cluster', 'above_avg']).size()fig2, ax2 = plt.subplots(figsize = (30, 15))sns.heatmap(clusters.unstack(level = 'above_avg'), ax = ax2, cmap="Reds")ax2.set_xlabel('Above Average Rating').set_size(18)ax2.set_ylabel('Cluster').set_size(18)
I can see the distributions by splitting apart the data frame and graphing the cluster counts!
#create dataframe of reviews not above averagenot_above = df.loc[df['above_avg'] == 0]not_above.describe()#create data frame of reviews above averageabove_avg = df.loc[df['above_avg'] == 1]above_avg.describe()#plot the countsplt.figure(figsize=(14,4))sns.countplot(x='cluster', data=not_above).set_title("Rating Counts")plt.show()plt.figure(figsize=(14,4))sns.countplot(x='cluster', data=above_avg).set_title("Rating Counts")plt.show()
Looking at the visualizations, they show above average ratings were clustered more heavily in 5, 6, and 12. That means the words in those clusters were commonly used in the wine reviews given above average ratings. A deeper look at the words within the cluster is needed since it is difficult to distinguish significant differences when only looking at top 10 words. For example, words like apple, aroma, and flavor appear in several of the groupings making it hard to understand distinct themes. Additionally, there are different algorithms that might perform better at topic modeling.
Latent Dirichlet Allocation (LDA) is another popular unsupervised learning algorithm for topic modeling that can outperform K-means. Hopefully I’ll get a chance to explore LDA and compare my results.
You can download my notebook from my github repo:
github.com
If you enjoyed this, follow me on Medium for more
Get FULL ACCESS and help support my content by subscribing
Let’s connect on LinkedIn
Analyze Data using Python? Check out my website
— Eric Kleppen