Yelp Dataset — Classifying Users as ‘Harsh’ or ‘Lenient’
By: Mabast Hussein
Tags: Yelp, Hypothesis Testing, Review, Python, Data Science

Classifying Popular Users

We have shown that the reviews of popular users are in fact significant. Now we will classify these users as ‘harsh’ or ‘lenient’ based on a defined criteria.

For each review of business j from user i, a score is assigned

where M is the total number of restaurants user i has reviewed, rᵢⱼ is the rating user i gave to restaurant j and < rⱼ > is the average rating of restaurant j.

sᵢⱼ < 0 indicates a harsh score and sᵢⱼ > 0 indicates a lenient score. Therefore, the average of all the scores of each user can be taken and users can be classified using this criteria. However, the significance of these results need to be tested. We instead carry out a two-tailed t-test at the 5% significance level to obtain a p-value pᵢ, where the hypotheses are

We will refer to sᵢ as the average rating deviation for user i. The t-test assumes that the sᵢ follow a normal distribution which is a decent assumption based on figure below. Also, when carrying out a Kolmogorov-Smirnov test, a p-value of 0.027 is returned meaning the sᵢ can be described by a normal distribution at the 2.5% significance level.

Multiple Hypothesis Test

Had we been testing the score for only one user and obtained pᵢ ≤ 5%, then we reject the null hypothesis and classify the user based on their score. However, we are testing many users on the same data leading to the scientific phenomena of P-Hacking — the manipulation of data to obtain a p-value of statistical significance (see this article for more detail). This in turn increases the number of false positives in our data — incorrectly significant results. To control the number of false positives occurring we can correct for this using either the Family Wise Error Rate (FWER) or the False Discovery Rate (FDR).

The FWER is the probability of at least one false discovery hence we will use Bonferroni’s method to reduce the number of false positives. To achieve this, the null hypothesis is rejected when pᵢ ≤ α/Ν, where α is the 5% significance interval. It can be seen that when we have large N (large number of tests), then this condition becomes more stringent and we end up rejecting true positives. This will be displayed in the results.

To counter the problem of rejecting true positives, we can instead use the less restrictive FDR criterion to control the type 1 errors. For this we follow the BH(q) (Benjamini-Hochberg procedure). The procedure is as follows

Sort the p-values in ascending order Find the largest p-value such that pᵢ ≤ q*i/Ν Reject p-values 1,…,i

These different correction methods will classify users differently.

Exploring User Attributes

To finish off with we will explore some of the attributes of these users. This will be based on how other users react to all of the reviews a popular user has given which, in turn, is given by the reactions; ‘cute’, ‘funny’, ‘plain’, ‘writer’, ‘hot’ and ‘cool’. For these attributes, the values are all converted to probabilities i.e. what is the probability that user i will gain a ‘cool’ reaction on their review. For example, if a user has a total of Rᵗᵒᵗᵃˡ reactions to their reviews, of which Rᶜᵒᵒˡ are ‘cool’ reactions, then this user has a value Pᶜᵒᵒˡ = Rᶜᵒᵒˡ/Rᵗᵒᵗᵃˡ. The sum of P for all the reactions should equal to one. The correlation of each of these attributes to the average rating deviation of each user, sᵢ, will be calculated by carrying out a regression analysis.

The final attribute will be the gender of the popular user. The Yelp dataset doesn’t include gender, instead the genderize package available on Python is used to guess the gender based on the users’ name. The analysis for this will be based on the first four moments.