Using Data Science to Predict Negative Customer Reviews | by Chuck Utterback | Towards Data Science

If you could confidently predict that a customer was going to leave a negative review, what would your business do differently?
Proactively intervene to improve customer experience and hopefully divert them from leaving bad public review scores?
Learn from important features in your predictive model to fix the root causes of poor customer experiences?
The answer is “yes” to both, with careful consideration of the cost-benefit tradeoff and the company’s ability to implement corrective actions.
“Negative reviews have convinced 94 percent of consumers to avoid a business” — Review Trackers
Brazil’s leading e-commerce marketplace for small businesses is Olist. Olist Store enables merchants from all over Brazil to sell and ship products to customers using Olist logistics partners. After delivery, customers receive an email satisfaction survey using a scale of 1 (unhappy) to 5 (pleased).
Olist has kindly released anonymized customer review and order details for over 100K transactions from 2017–2018. This popular data set has been downloaded by data scientists over 76K times.
Let’s build a predictive model for “bad customer reviews” as defined by receiving a score of either 1 or 2 on the scale of 5. In our large sample of delivered orders, 13.2% of review scores were bad (negative).
We will solve for a binary target variable: review_bad = 1.
Our goal is to find a supervised machine learning classification model that can predict bad customer reviews with the best precision.
High precision indicates that when our model predicts a bad review, it’s usually correct. This minimizes company risk of taking incorrect action on predicted negative customer reviews.
Olist provided data from 8 relational tables which had excellent referential integrity. The diagram below shows a familiar data model with unique order IDs facilitating joins to multiple items and payments per order. Master data tables include products, customers and sellers. Customer reviews are linked to an overall order that can include multiple products or sellers on an order.
Raw data across the 8 tables includes 52 distinct columns. Predictive features I developed can be grouped by these domains.
In the interest of brevity, I’m not including the full code used on this project. For full details, please refer to my Jupyter notebook on GitHub.
Data transformations that were used to prepare predictive features:
Filtered out records from 2016 and non-delivered orders (less than 1%)
Filled in just a handful of records where nulls were present
Aggregated metrics on orders, payments, products and sellers as depicted
Calculated days interval metrics using lifecycle dates from the orders
Estimated delivery distance in kilometers between seller and customer zip codes using median latitude and longitude in a haversine function
Created one-hot encoding variables (flags) for payment methods, customer states, day-of-week and time-of-day buckets (morning, afternoon, etc.)
Purged multiple reviews for any order, keeping only the final rating; this affected less than 1% of orders
During data analysis, I usually export the final merged data frame from Jupyter notebook out to Tableau for faster visual inspection. Let’s take a look at the most interesting views for insight.
Delivered Days Late: Olist provides customers with an estimated delivery date at time of order which averages 24 expected processing/transit days. The actual delivery performance matters greatly as shown above. For orders that were delivered late (negative variances to estimated date), customer bad reviews averaged 55% versus 10% on non-late deliveries. The greater the number of days late, the higher proportion of bad reviews.
Seller-Product Quality: The scatter plot depicts a linear trend between late deliveries and bad reviews with each observation being a seller’s average performance across all customers. Late orders don’t explain the whole story as many sellers with above average delivery performance (circled) but are seeing higher bad review rates.
Variety in Products: Using the tree map, we can see variation in bad review rates by product category. Drilling into some individual products we can see some “dogs” with higher than 50% bad reviews where customer expectations are clearly not met. The average bad review rate for a product is a significant feature in the final model.
Geographic Breakdown: There are 26 states and 1 federal district in Brazil. We can definitely see variations in bad reviews amongst the states.
Sáo Paulo — representing 42% of orders in our data, this state significantly out-performed on each metric.
Rio de Janeiro — this nearby state, representing 13% of orders, significantly under-performed on each metric.
Both states are each home to one of the two largest cities in Brazil, with high population density and relative proximity to each other. Rio de Janeiro should be a target for deeper digging into delivery performance and product mix. Why do customers on average order products with a high index of bad reviews relative to average? (a 148 product index means the products purchased on average had 48% worse bad review rates than the average product in Brazil overall).
Running Pearson correlations against our review_bad target variable in Python largely confirms associations we’ve seen in prior plots.
When looking at single feature correlations, we see moderate positive correlations on all forms of delivery intervals (is_late, act_delivery_days, days_late, and days_variance).
We also see positive 0.23 and 0.17 correlations on products and sellers overall bad review rates, respectively.
Lastly we see some other minor influences on order/seller item counts, freight/order values, and delivery distance. The only 2 states of 27 with correlations greater than 0.03 are indeed Rio de Janeiro (+0.07) and Sáo Paulo (-0.06), supporting our state map graphic above.
Seven supervised learning classification models were optimized on the “precision” metric but looking for a reasonable “recall” (proportion of bad reviews our model correctly predicts). For each model I used grid search on 80% training data with stratified 5-fold cross-validation to explore a variety of parameter combinations.
Models compared included Random Forest, AdaBoost, Gradient Boost, XG Boost, Logistic Regression, SGD Classifier and Gaussian Naive Bayes.
Relevant features were scaled using StandardScaler so I had consistent training features as input to each model (even though most models did not require scaling).
The resulting “best model” for each classifier was then run on previously unseen 20% scaled test data with comparative model results shown above.
I’ve recommended the Gradient Boost model which had the following performance: Precision of 84%, Recall of 29%, Accuracy 88%, AUC 64%. Parameters from the best estimator were:
'learning_rate': 0.01, 'max_depth': 8, 'min_samples_leaf': 25, 'n_estimators': 100
Using this model with the standard 0.50 probability decision threshold results in predicting about 1/3rd of bad reviews (recall) while only being wrong 1 out of 6 times (precision, false positives). This would be a start to take action, and we would want to continue to refine the model with experience and broader data.
The most important features in the Gradient Boost model included:
Strong: Delivered Days Late, Days Variance, Product Bad Review %, Count of Items on Order
Moderate: Count of Sellers on Order, Seller Bad Review %
Weak: Actual Delivery Days, Total Orders for Product, Total Orders for Seller, Freight Amount on Order, Delivery Distance (km)
In summary, bad reviews are most sensitive to late deliveries, products/seller histories, and coordination of multiple sellers/products. Olist should try to fix the root causes for poor delivery performance. Olist might also try a randomized cost-benefit trial to see if proactive communications, acknowledgment or customer concessions might prevent bad reviews from being posted.
Ultimately, a negative review is a gift. It shows that your customers care enough about your brand to take the time to leave you feedback. Even the worst reviews can turn into positive experiences, if handled correctly. — Emily Heaslip
A few observations on data quality and assumptions inherent in this project.
Sample vs. Population: The sample of 100K orders, each with an associated review rating, is great for building a model but not representative of the population. Given only 5–10% of customers write reviews, and not every time, we’d have some work to translate our model to generalize against the full population of Olist orders.
Need Additional Features: While 84% precision predicting 29% of bad reviews is a start, we need to do better. Additional data I’d recommend that Olist acquire for further feature development: full database of orders, customer returns, customer web logs, customer service interactions, customer personas, demographics, brand/product/seller social sentiment, and data from sellers.
Marketing Funnel: Olist also released a marketing funnel dataset on Kaggle which contains 2 upstream tables from the seller qualification process. This data was unusable when joined to the order schema since it contained only sellers signed-up between 2017–2018, representing a small fraction of actual orders during this period (most were by established sellers).