Effect Size. Moving beyond the p-value | by Aren Carpenter | Towards Data Science

In the sciences, we deal with p-values and statistical tests constantly. We hope to see a p-value < 0.05 to declare that we’ve been successful in our efforts, but this fervor for incredibly low p-values means that we don’t ask another very important question. “Does it even matter?”
Let’s take a case where we are looking at two sample means. Simply, the p-value is the probability that sample means are the “same” or come from the same underlying distribution. This is the null hypothesis (Ho: mean1 = mean2), whereas the alternative hypothesis (Ha: mean1 ≠ mean2) represents the two sample means not being equal.
Dive deeper into hypothesis testing here. This post assumes some base knowledge.
After conducting a two-sample t-test for difference of means we can calculate a p-value to either reject or fail to reject our null hypothesis. If we get a p-value of less than 0.05, then we can reject the null and conclude that the means are different. But that’s all we can say!
Let’s get back to our initial question, “Does it even matter?” Even if we know that the means are different, we don’t know if that difference is significant. For a quick semantic point, I don’t mean that the difference is not statistically significant (that’s what our p-value gives us), but significant in the colloquial sense. Does it matter that one mean is different from the other, and by how much?
It is possible to generate very significant p-values for effect sizes with little practical importance and vice versa.
This is incredibly important in data science due to the size of our datasets.
With enough observations, even tiny differences in parameters become statistically significant, even if these differences are not important. And for those not intimately familiar with the scale of the data, there is no common framework for interpreting these differences.
For a deeper examination of Statistical Power and Power Analysis, Machine Learning Mastery has an excellent guide.
Looking at the raw differences between means is one simple, quick method of assessing the effect size, but often requires some previous knowledge or sense of the data if it’s not an intuitive measure like height or weight. For example, with a population of male and female weights, perhaps men weigh 15 kg more than women on average. That is much more intuitive than (drawing from my previous research) a 2x increase in Investigations per Entry between control and experimental groups of Diamondback terrapins.
Building on just raw values, you can also calculate an overlap threshold where the respective population distributions cross each other.
Using the AUC (Area Under the Curve), you can then calculate the amount of overlap in the distributions and count the observations who are on the “wrong side”. This measure is also called the Misclassification Rate.
Another unstandardized measure is the probability of Superiority. Basically, taken random samples from each group, what is the probability that one is larger than the other. It’s an easy method to calculate, but as a random measure it won’t be as precise as other methods.
As unstandardized measures of effect size, both overlap and superiority have the benefits of being comparable between studies because they are raw probabilities and they provide raw values to get an intuitive sense of the differences between the groups.
One way to rationalize the differences between the means is to standardize the effect size by dividing by some standardizer (usually standard deviation). The most common method is Cohen’s d.
Simply, it’s used to represent the magnitude of differences between two groups on a given variable. We standardize the difference of means by dividing by the pooled standard deviation of all groups. You can also use the standard deviation from either of the groups alone if there is a normative comparison to be made (say you had a control group and an experimental group).
d = effect size (difference of means) / pooled standard deviation
In code, Cohen’s d for two groups can be calculated as:
def cohen_d(group1, group2):     """          Calculate Cohen's d for two groups.         Group1/Group2: Pandas Series or Numpy array         Return d: float64     """     diff = group1.mean() - group2.mean()     n1, n2 = len(group1), len(group2)     var1, var2 = group1.var(), group2.var()     pooled_var = (n1 * var1 + n2 * var2) / (n1 + n2)     d = diff / np.sqrt(pooled_var) # sqrt of var is stdev     return d # d is ultimately a ratio
Once we have some value for d, how do we interpret it? Remember that d is a ratio and is thus unit-less. One benefit of this is that we can directly compare d values from studies with different units/parameters, for example new trial drugs with different mechanisms of action, to see their relative effect size.
As a rule of thumb:
Small Effect = 0.2
Medium Effect = 0.5
Large Effect = 0.8
Here’s a great tool for getting an intuitive sense of Cohen’s d.
Effect size is very often overlooked when calculating various statistical tests on data, but it’s very important to know if the differences between our groups actually matter! In data science, our datasets can be very large which makes it easier to find statistically significant differences between groups, even if those differences aren’t practically significant. I’d recommend starting with an unstandardized method to get a sense of the raw difference between your groups, before calculating Cohen’s d to see how much that difference matters compared to other experiments!
I’m always looking to connect and explore other projects!
LinkedIn | Medium | GitHub