Power Analysis Made Easy. So easy, you can do it by hand. | by Rezwan Hoppe-Islam | Towards Data Science

You’ve redesigned your site’s landing page and now you need to A/B test it. How long should you run the test for? This article will demonstrate how to answer this question using a sample size calculation (“power analysis”).
Statistics can be a daunting subject to learn, and it doesn’t have to be. The aim of these guides is to make statistical tools more accessible to people. Make sure that you’re comfortable with the concept of the sampling distribution (see here if you need a refresher) and the calculation of the standard error for a proportion metric (see here). Enjoy!
You want to A/B test a redesign of your landing page. The current landing page sees an average of 200 visitors per day and has a click-through-rate (CTR) of 5%. You believe that the redesign will deliver a CTR of at least 7%. Once we allocate traffic evenly, the current landing page (Control) should see 100 visitors per day, and the redesign (Test) should also see 100 visitors per day:
How many days should you run the test for in order to validate that the Test indeed outperforms the Control?
Let’s say we plan to run our A/B test for just 7 days at which point we would have collected 700 visitors per group. Let’s compute the sampling distribution for the null hypothesis (no difference in population CTRs) and for the alternative hypothesis (a true difference in population CTRs).
Step 1: Calculate the pooled sample proportion (p).
Step 2: Use p to calculate the standard error (se).
Step 3: Compute the sampling distributions.
H0 (null hypothesis): a normal distribution with mean = 0, standard deviation = 0.013.
HA (alternative hypothesis): a normal distribution with mean = 0.02, standard deviation = 0.013.
Let’s plot these distributions:
At a significance level (α) of 0.05, the critical value is 0.0249. This means:
If H0 is true, then the decision to reject it would be incorrect (a Type 1 error). This would occur at a rate of 5 out of 100 experiments (α).
If HA is indeed true, then the decision to reject H0 would be correct. This would occur at a rate of 35 out of 100 experiments (1-β). This rate is also referred to as the power of a test.
If H0 is indeed true, then the decision not to reject it would be correct. This would occur at a rate of 95 out of 100 experiments (1-α).
If HA is true, then the decision not to reject H0 would be incorrect (a Type 2 error). This would occur at a rate of 65 out of 100 experiments (β).
To summarize:
Our decision to run the A/B test for only 7 days would result in us unable to detect the uplift from the Test group in 65 out of 100 experiments. This error rate is too high! Typically, we’d want to bring this value down to 20% which is equivalent to a power of 80%. We can achieve this by increasing our sample size.
Recap: By running the A/B test for 7 days and, therefore, collecting a sample size of 700 visitors per group, the power of our experiment is only 35%. To reach a power of 80%, we’d need to increase our sample size, which means letting the A/B test run longer.
What sample size would we need to obtain a test with 80% power?
Let’s transform our sampling distribution under the null hypothesis to a standard normal distribution to make the calculations more straightforward:
The critical value at 0.05 significance is ~1.96 which you can obtain from a look-up table.
If we now consider the sampling distribution given the alternative hypothesis, then we want the area under the curve between -1.96 and 1.96 to equal 20% (for 80% power). Therefore, the critical value has to be a distance of ~0.84 away from the mean (also available from a look-up table).
So the total standardized difference between means must be 1.96 + 0.84 = 2.8.
This may be easier to understand graphically:
We also know that the real difference between means is 0.02, therefore, the standardized difference equals 0.02 / se. So we can construct an equality and solve for n:
So we’d need to collect 2,211 observations per group. At a rate of 100 visitors per day per group, we’d need to run the test for at least 23 days to obtain an experiment with 80% power. With this sample size, if the redesign page indeed delivers at least 7% CTR, then we’d detect this uplift in 80 out of 100 experiments.
Recap: Referring to the standard normal distribution, we were able to determine that the standardized difference between means should be ~2.8 for 80% power. We were then able to express this in terms of the standard error and then solve for n. We calculated n = 2,211 observations.
By this point, you should feel fairly comfortable with the overall framework of a sample size calculation (power analysis). Now let’s see how we can reduce all the above-mentioned steps into just a few lines of code. In R we would write:
Which produces:
[1] "absolute difference (Test — Control)"0.02[1] "pooled sample proportion"0.06[1] "standardized distance for requested power"2.80158521811297[1] "solve for n"2213.38408508644