CXL Institute: Growth Marketing Mini Degree Review #9️⃣
By: Evelina Califano
Tags: Google Analytics, Statistical Analysis, Beginners Guide, Growth Marketing

Stats for Growth Marketing

Photo credits to Rishi.

The larger the sample size, the higher your power (as we defined in review number 7). The higher the power, the better.

When you consider conducting an experiment the first thing to set is to decide on the appropriate sample size. This is because whenever testing, your aim is to make sure you have enough power to meaningfully expose a difference in conversion rates.

There are several factors you need to take into account when calculating your sample size:

How large is the difference to detect Level of confidence between 90% and 95% Power and Variability (closest to 50% at a higher variability; should be 80%) A conversion rate of control 5%

As for the sample size variables we need to take into account:

Control group’s expected conversion rate Minimum relative change in conversion you want to detect (Lift) Confidence level

Be aware of the Sampling Error!

We can define the sampling error as an error that occurs when the sample is not representative of its population. The sample is not close to the mean.

According to our instructor, when calculating power of 0.80 is considered the best standard practice to follow on. When thinking like this, it means that there is an 80% chance that if there is an effect (from your testing) you will be able to detect it. It is pivotal to keep this in mind because you will be guided by this percentage throughout your experiments.

Our instructor, Labay is able to explain difficult concepts in a way that makes it easier to digest for less-educated students. I have enjoyed grasping the concepts of power and the factors to acknowledge before running experiments.

Optimization should always be hypothesis-driven.

However, validity is also a problem. This is because the more variants you have, the higher probability of one becoming a winner when it actually is not, because we define it as false positive.

Our confidence level is between 90 and 95% and we accept a 5% of probability of type I error, false positive. It is important to test with only one variant. This is because the number increases quickly when we add more variants and the risk of having false-positive as well increases.

The rule is to have no more than 3 variants plus the control should be run in one test.

That is why it is extremely important to consider your KPIs to improve the accuracy of your experiments, as well as keeping a vigilant eye on the power and the number of variants included in the test.

You will need to select and prioritise your main KPI (it cannot be both click rates AND conversion rates).

MACRO KPIs: Answer how much

MICRO KPIs: Answer why (clicks, views, etc.)

Another suggested rule is whenever there are some uncertainties in some metrics, the longer the test should be run in order to verify the meaningful effects.

Bayesian vs Frequentist Statistics

According to Labay, the two model of implementing statistics calculation is merely a difference in points of views, that he defines as the difference of philosophy applied to statistics.

If in a Bayesian point of view, a hypothesis is assigned to probability, conversely in a Frequentist point of view, a test is run without a hypothesis being assigned to probability.

The Frequentist “school” involves the ability to make predictions on the underlying truths of the experiments using only the data collected in the current experiment, whilst a Bayesian “school” supports the inclusion of previous knowledge (of data collected in other experiments) when predicting the outcome of the upcoming experiment.