Design of experiment basics: if you build them, they will come
By: Anatoly Zaytsev
Tags: Statistics, Getting Started, Ab Testing, Editors Pick, Data Science

There are already thousands of sources on the web explaining p-value and concepts related to statistical significance testing. Why write another one? I see over and over again statistical tests used as a “black box” to obtain p-value and apply the “golden rule”: if p-value is less than 0.05, then results are significant. Very often this is done blindly without actually understanding what p-value actually shows, what it doesn’t show and how to properly design and interpret an experiment. My goal for this post is not to provide a recipe for how to solve a specific problem, but rather describe the basic concepts associated with hypothesis testing and how we arrive at them.

I believe that any concepts can be learned and understood only when: (1) these concepts are intuitively linked to other concepts we already very familiar with and know very well (2) when we try these new concepts in practice.

In this blog post, I’ll go over main principles of statistical significance testing and experiment design using only basic probability theory. We will not use any statistical tests, instead, we will build a statistical significance testing process from the probability basics, step by step: from the basics to the concepts such as statistical tests, p-value, significance level, minimal detectable effect, type I/II errors, power of the test and design of experiment fundamentals.

To illustrate all concepts in practice, I’ll use a coin toss example throughout the post and Monte Carlo simulations applied to a problem: how to test whether a coin is fair or biased? This is a famous problem frequently asked by top tech companies in their job interviews. It is also famous enough that it even has its own Wikipedia page. Coin toss experiments and binomial distribution is a safe starting point that anyone understands. Let’s dive in.

Simple coin example and p-value

Suppose we have a fair coin with probability of heads P(heads) = 0.5. Let’s say we flip it 20 times and we get 13 heads and 7 tails. Should we conclude that the coin is weighted or biased? Is there a measure to estimate how unlikely this outcome is? To address the questions above, we need to calculate the probability of our outcome, which is observing 13 heads out of 20 coin tosses. It is given by binomial distribution (eq. 1 and the only equation we will need to use in this post):

Equation 1. Probability of observing heads H number of times in Ntosses independent trials, given probability of heads P(heads) = p.

Plugging the numbers into the equation, we can calculate the probability of our observed outcome as Bin(H = 13, Ntosses = 20, p = 0.5) = 0.074. Just the probability of the outcome is not sufficient to evaluate how unlikely is our outcome. For example, the probability to draw any specific random card from 52 cards deck is about 0.019 but all possible outcomes are equally likely. We cannot answer how unlikely is a particular observation unless we consider all possible outcomes and their corresponding probabilities.

One way to calculate how unlikely is particular observation is to calculate the total probability of seeing any equally likely or a less likely outcome. Going back to our coin example, we need to calculate the probabilities for all possible outcomes for 20 coin flips (Figure 1). Next, we calculate what the chances are of obtaining an equally or less likely result (red area in the probability distribution on Figure 1), which gives us ~0.19:

Figure 1. Probability distribution for all possible outcomes of 20 coin flips assuming the coin is fair and P(heads) = 0.5. Function is represented by Eq. 1 with parameters Bin(H, Ntosses = 20, p = 0.5). The area in red corresponds to the total probability of observing an equally or less likely outcome and is called p-value.

The value we just calculated is called p-value. More formal definition: given an observation (in our case the results of a coin tosses series), and a basic assumption about the system’s property (also called null hypothesis, in our case we assume the coin is fair), p-value is the total probability of seeing equally or less likely observation. For example in card deck example p-value of observing any particular card is 1, assuming deck is shuffled.

Let’s sum it up. Using only basic probability theory we just derived a statistical test for a coin. This is essentially what all statistical tests do: given observations and basic assumptions about the system (assuming there is no effect), provide the p-value — probability of obtaining an equally likely or less likely result than was actually observed.

Significance level, type I error

Now that we know what p-value is, and we’ve derived a statistical test for coin fairness, let’s do a computational experiment. Let’s imagine that we have 1000 fair coins and we flip each coin Ntosses= 1000 times. After completing the series of coin tosses, we will calculate p-value for each coin using the method above. Next, we plot all 1000 obtained p-values as a histogram:

Figure 2. Distribution of p-values obtained for 1000 fair coins. The calculated p-values form a uniform distribution (it’s exactly uniform for continuous values, in our coin example — close to uniform).

As you can see from the histogram, p-value of the experiment when the null hypothesis is true (all of tested 1000 coins are actually fair) is essentially a random variable that describes the unlikeness of the observations. In the example above when the coin was fair p-value resulted in a close to uniform distribution.

Let’s say in the future we want to test whether an unknown coin is fair or not. We can select a threshold for p-value, below which we will reject the assumption that the coin is fair. But such a threshold comes with a tradeoff: since p-value for fair coin is distributed uniformly, selecting 0.05 as threshold will result in a 5% chance of concluding that the fair coin is biased (Figure 2, red area). In our simulated example with 1000 fair coins we will falsely conclude that 56 coins are biased (approximately 5% of the total coins tested).

The selected threshold for a p-value is called a significance level, usually denoted as 𝛂. Now it should be clear that 𝛂 corresponds to how often we will falsely reject the null hypothesis and conclude that we have an effect when our obtained observations are actually due to random chance. This type of error is called type I error.

Minimal Detectable Effect (MDE) and type II error

Let’s consider a problem now where we have a bag of coins but we don’t know which coins are fair P(heads) = 0.5 or biased P(heads)≠ 0.5. How can we approach this problem from the statistics point of view? Let’s design an experiment to test coin fairness.

First, we need to define what we will call a biased coin. We need to set a threshold of how different the probability should be from 0.5 that we will consider a coin biased. For illustration purposes, let’s say we will only consider a coin biased if the P(heads)> 0.6 or P(heads) < 0.4, in other words, if the heads probability differs from the fair coin by 20% or more (this value is called minimal detectable effect or MDE). To test the coins, let’s toss each coin Ntosses = 100 times, and count the total number of heads. We can plot the expected distribution of the total number of heads out of 100 tosses for fair coin and for a biased coin at selected minimal detectable level of 20% (Figure 3). Note, there are two bells since a biased coin can be biased in both ways.

Figure 3. Probability distributions for total number of heads out of 100 coin tosses for fair (top) and biased (bottom) coins. Filled area in red corresponds to type I error rate; filled area in blue corresponds to type II error rate.

Next we need to select a significance level, 𝛂 — what is the acceptable chance of calling a fair coin biased? For illustration purposes in this example we will go with 𝛂 =0.1. Now we have sufficient information to identify the range for the total number of heads out of the 100 tosses we need to obtain in order to conclude that the coin is fair, or to reject the null hypothesis that the coin is fair and conclude that the coin is biased. The area outside these boundaries for the fair coin probability distribution should be equal to 0.1 (Figure 3, red area at the top graph), which gives us a range [42, 58] (boundaries included). If after 100 tosses the coin will have less than 42 or more than 58 heads (in other words with p-value less than 𝛂) we will reject the null hypothesis and conclude that the coin is biased.

Very simple, right? However, would we be able to correctly identify all truly biased coins using this approach? What are the chances that a biased coin after 100 trials will actually result in a total number of heads between 42 and 58? To answer this question, we need to look at the expected probability distribution for biased coins (Figure 3, bottom graph). The area in blue between 42 and 58 is 0.38, which means that in 38% of cases a biased coin will be actually called fair. This is called type II error and its probability is usually denoted as 𝛃. Let’s illustrate all these concepts in an actual simulation.

Power of the experiment, False Discovery Rate

Let’s start with a bag of 1000 coins where 500 of coins are fair and 500 coins are biased (at minimal detectable level 20%). Suppose that we don’t know which coins are fair and biased, and we want to perform our statistical test to find it out. To test, let’s now toss each coin 100 times, calculate p-value, compare it to our chosen significance level 𝛂 = 0.1, and make a decision whether that coin is biased or not. The results of such an experiment is shown in Figure 4.

Figure 4. Results of the stochastic simulation of fairness test for 1000 coins. Each coin was flipped Ntosses = 100 times and obtained p-value was compared to significance level 𝛂 = 0.1 to conclude if coin is biased or fair.

Only in 62% of cases (1- 𝛃) we were able to correctly conclude that an actual biased coin is biased. This value is called power of the test. We can also note that from all 310 + 56 = 366 identified biased coins, only 310 were actually biased and 56 were in fact fair coins. The fraction of number of incorrectly discovered biased coins to the number of total discovered biased coins is called false discovery rate and is 15% in our example above. We can increase the power of the test if we obtain more data as illustrated on Figure 5.

Figure 5. Illustration on how the number of coin tosses Ntosses affects the power of the statistical test. In each case the minimal detectable effect was chosen MDE = 10% and significance level was 𝛂 =0.05.

It’s very important to understand that getting statistically insignificant results doesn’t mean that we don’t have an effect, or in other words, it doesn’t prove our null hypothesis. Our test might simply has low power and insufficient data to reject the null hypothesis.

Another common mistake very often seeing in A/B testing is to run the experiment until p-value becomes lower than significance level instead of waiting for the pre-determined in advance number of observations (unless you know what you are doing and performing sequential or Bayesian testing). Such “as you go” approach dramatically increases type I error rate as well as false discovery rate. To illustrate this, we can repeat computational experiment from Figure 4 where we need to test for bias 1000 coins, but this time we will check significance “as we go”, calculating p-value after each flip. We stop the experiment if one of two conditions are met: p-value is less than 𝛂 (and conclude that coin is biased) or maximum number of flips Ntosses is reached (conclude that coin is fair). Type I error rate and false discovery rate as a function of Ntosses are shown on Figure 6 (red data).

Figure 6. Same experiment as in Figure 4 but p-values were calculated “as we go” (data in red). Chose significance level 𝛂 = 0.1.

If testing performed only after collecting pre-determined number of observations, type I error rate stays constant and is close to chosen significance level 𝛂 regardless of observation length (Figure 6, blue line). This is very different when we calculate p-value “as we go”, when type I error rate increases steeply as we continue to observe. This simulation demonstrates the common pitfall when we draw a conclusion about A/B test as it progresses.

Summarizing it all together for statistical experiment design

In practice, we very often start from defining the desired significance level 𝛂, minimal detectable effect, and power (how important for us not to miss the effect). For example, during the design of A/B tests each experiment typically starts from the questions:

1) How small are the effects we want to be able to reliably detect (this will give us the minimal detectable effect)?

2) What should the chance be that we will implement in production the change that actually has no effect? This is significance level 𝛂 or type I error rate.

3) What should the chance be that we will not detect an actually useful change? This gives us type II error rate 𝛃.

Figure 7. For any hypothesis testing experiment four values on the picture are always related to each other. This means that for a given test and basic assumption about the tested system (null hypothesis) if given any three values out from four you can always calculate the fourth.

Having these three values allows us to calculate the required number of observations for the test (Figure 7). Very importantly, we need to collect all required observations before stopping the experiment and making conclusions, otherwise we can potentially increase false discovery rate (Figure 6). Alternatively, we can use collected data (known observations number), fixed significance level, and power, to calculate the smallest real effect size we can detect at our selected power. An important universal concept for any statistical test is that if we know three out of the four values: significance level, power, number of observations and minimal detectable effect, we can calculate the fourth value.

Conclusion

In conclusion, I want to stress again that this post was not a recipe for any particular statistical test or practical application. My goal was to describe the basic concepts associated with hypothesis testing and how we arrive at them. Essentially, this is the way how I would teach hypothesis testing to myself. After understanding the fundamentals, you can read many practical guides but with the intuition behind the concepts rather than using them as a “black box”. And remember:

“If you need statistics to prove something, it probably isn’t true.”

(quote found on wpi.edu).

link to Github

LinkedIn: https://www.linkedin.com/in/anatoly-zaytsev/

Twitter: https://twitter.com/zaytsev_anatoly