Why Do You Like Me?
By: Henrik Robert Kram
Tags: Machine Learning, Data Science, Topic Modeling, Text Mining, Marketing Intelligence

Lifting The Curtain: What is Topic Modelling?

In all brevity & simplicity: What topic modelling does is clustering co-occuring words together based on the complete set of texts (a so-called corpus). The result of this statistical model does not just provide you with clusters of words, which you can interpret to form topics, but as well enables you to describe each and every text as a mixture of the topics. However, we saw that enriching the topic modelling with relevant covariates enhanced the quality of the topics. That is why we performed a sentiment analysis on all reviews scraped and integrated the sentiment polarity into our topic modelling algorithm. This essentially means that we performed a so-called structured topic modelling approach². In this context, we previously validated three different German sentiment dictionaries (GPC, SentiWS and NRC) on a random subsample of our text data base and found the GPC lexicon to have the highest congruence with our judgement. With respect to our topic modelling approach, we fine-tuned the hyperparameters of the algorithm in a systematic manner by assessing the score of each parameter setting over five different quality metrics (For the nerds: Semantic Coherence, Held-Out Likelihood, Exclusivity, Residuals and Lower Bound).

Six of the fifteen identified topics for otto.de along with their most exclusive (FREX) and probable (PROB) words per topic [Source: Image by Author]

All in all, we identified 15 topics for otto.de where we were able to recognize different overarching elements of OTTO’s service. The topics identified were ranging from evaluations about the delivery management to critical assessments of the price levels at the online shop. See the above table for a subset of the topics and their respective most characteristic words. For VINOS, we identified 11 different topics where most of these focused on different aspects of the product delivery highlighting that this aspect is of particular importance for VINOS’ clients. To get the big picture on how these topics were describing and summarising the complete sets of reviews we had for each company, we further on performed dimensionality reduction on the data. In this line, a t-SNE reduced dimensional space (see picture below) shows that all reviews were cleanly clustered into inherently different aspects. In other words: Where we previously we had solely superficial statistics to describe how much our customers like our products, we now identified the over-arching topics. However, we have to go a step further because, after all, we want to know why our customers like us.

All 23,820 reviews of the otto.de corpus visualized on a (t-SNE) reduced dimensional space, coloured according to their topic assignment [Source: Image by Author]

Determining the Why: Impact of Topics on Overall Satisfaction

After having identified several latent topics in the customer reviews, a logistic regression model was executed to determine single topic effects on customer satisfaction. In this project, the review’s star rating was assumed to be a proxy for the satisfaction of customers and serves as the dependent variable. A high star rating, respectively five or four stars, indicates that customers are generally more satisfied. Contrarily, customers that assigned the company with a low star rating (one or two stars) were considered to be overall dissatisfied. Hence, star ratings have been recoded into a binary variable enabling our logistic regression to report the exact estimates on how much each driver leads to satisfaction.

Effects (x-axis) of 7 of the 15 previously identified OTTO topics on customer satisfaction levels as measured with the star ratings. Note that the line on the 1.0 mark separates negative (left side) from positive effects (right side) [Source: Image by Author]

For both companies we were able to identify significant effects of the topics on the overall satisfaction levels. Take, for instance, the image above where seven topics of OTTO’s data set are displayed. Here, you can clearly see that three topics lead to high satisfaction drivers. In particular, OTTO’s exceptional usability on its Point-of-Purchase touchpoints seems to be a strong driver of customer satisfaction. On the other hand, although overall star rating statistics indicate that OTTO’s customers are quite happy, we were nevertheless able to identify a strong driver of dissatiscation: A sloppy communication when it comes down to notifying customers of failed payments, warning notices and refunds. Summarizing all the results for OTTO and VINOS, we learned a couple of things:

Powered by ML: Companies are very much capable of identifying topics which are over-arching the complete chatter and noise ongoing at social media platforms

Combined with supervised ML it is possible to determine the effect of topics on customer satisfaction

With these effects, companies cannot just unveil the what but as well the why behind customer reviews

Interestingly, as well for companies with a near-to-perfect average rating our report shows that the methods still can accurately identify dissatisfaction drivers

¹ Wickham, Robert (2020): “Easily Harvest (Scrape) Web Pages” https://cran.r-project.org/web/packages/rvest/rvest.pdf

² Roberts, Margaret; Stewart, Brandon M; Tingley, Dustin (2020): “stm: RPackage for Structural Topic Models” Journal of Statistical Software