Reviewing A/B Testing Course by Google on Udacity
By: Suyash Maheshwari
Tags: Technology, Google, Data Science, Business, Testing

Reviewing A/B Testing Course by Google on Udacity

Read to find out how A/B tests are performed at Google.

A/B tests are online experiments which are used to test potential improvements to a website or mobile application. This experiment requires two groups — control group and experiment group. Users in the control group are shown the existing website whereas users in the experiment group are shown the changed version of the website.The results are then compared and the data is analyzed to see if the change is worth launching. The A/B testing course by Google in association with Udacity explains the various metrics that need to be considered for analysis , how to leverage the power of statistics to evaluate the results and finally whether the change must be launched or not. The course does not delve deeper into the statistical concepts instead explains the business application of these tests.

A/B tests are used extensively by various companies. For example, Amazon used it to test user recommendation feature , Google uses it so extensively that they had once used it to test 41 different shades of blue in UI! Albeit the term is recent the method has been in practice since a very long period of time. It has been used by farmers to see what method yields the best crop or by doctors for clinical trials to check the effectiveness of a drug.

Credit : Unsplash

Policy and Ethics :

Before running A/B tests one must consider a few things.

Risk : What risk is the participant undertaking while participating in the test. Minimal risk is defined as the probability and magnitude of harm that a participant would encounter in normal daily life.If the risk exceeds this then the participant must be informed about it. Benefit : It is important to be able to state what the benefit would be from completing the study. Alternatives : what the other alternative services that a user might have, and what the switching costs might be, in terms of time, money, information, etc. Data sensitivity : It refers to the data being gathered , whether it would reveal the identity of the person , security measures taken to safeguard the data and the implications it might have on a person if the data becomes public.

Choosing and Characterizing Metrics :

The metrics can be divided into two categories. Invariant metrics are the ones that should stay the same across both the groups. For instance , the number of people in both the groups ,distribution based on demographics. Evaluation metrics are the ones that you will use to measure the effectiveness of your experiment. For example, if you are testing for a change in the number of users who click on “start button”, click through probability (number of unique visitors who clicked / total number of unique visitors) could be your evaluation metric. Sanity check is performed to check that the invariant metrics you have chosen are correct. A/A tests can be performed to check this.

Sensitivity and robustness must also be considered. The metric should neither be too sensitive nor too robust. Mean can be too sensitive to outliers whereas a median may be too robust to capture the change. 90th or 99th percentiles are considered as good metrics to notice the change.To find a good balance between sensitivity and robustness one can perform A/A tests , use data from previous experiments or perform retrospective analysis of logs.

Designing an experiment :

Unit of diversion is used to define an individual person in the experiment.

User id : If a person has logged in to the account , we can use user id as unit of diversion to track the activities.

: If a person has logged in to the account , we can use user id as unit of diversion to track the activities. Cookie : It is a small piece of data sent from a website and stored on the user’s computer by the user’s web browser while the user is browsing.Cookies are browser specific and can also be cleared by the user.

: It is a small piece of data sent from a website and stored on the user’s computer by the user’s web browser while the user is browsing.Cookies are browser specific and can also be cleared by the user. Event based diversion : Any action such as reloading a page can be considered as an event. These are mainly used for non user visible changes. For example, latency change.

Population : You must also consider the population on which you will run the experiment, whether it would be the entire population , people from a specific region or people from a specific sector.

Size : The size of the population is also an important factor. It is influenced by parameters like significance level (alpha), sensitivity(1 — beta) and whether the unit of analysis is equal to unit of diversion.

Duration and Exposure : It refers to the time period for which you want to run the experiment. Also , it is very important to determine when to run the experiment. For example , on weekends or weekdays , holiday season or non holiday season. Generally, it is good to balance between the two so that you understand the trend and seasonality effect better. Exposure is basically the fraction of population to which you want to expose the new feature.

Analyzing results :

Sanity check : After you have the results of the experiment with you , the first thing you have to do is check whether your invariant metrics are the same across both the groups, like whether the population distribution was done correctly.

Single metric : To check if a single invariant metric is within the acceptable range you will have to calculate the pooled standard error and multiply it with Z- score (1.96 for 95% CI) and find the margin of error. Then find the lower and upper bounds and check if the difference in values for a particular metric is within the range.

You should also perform tests on evaluation metrics to check if the result is both statistically and practically significant. For a result to be statistically significant the range of the difference in values must not contain 0 and for a result to be practically significant the range should not contain the practical significance boundary. Also, to double check the result a sign test may be performed. If the two tests do not agree with each other we might have to look deeper into the data as it might be due to Simpson’s paradox(individual subgroups are showing stable results but their aggregation is causing the problem).

Multiple metrics : If you are considering multiple metrics at a time , it is possible that you might be seeing one of the metric as significant because of a false positive. In order to deal with this you may use bootstrapping or bonferroni correction.

After the results have been analyzed you have to answer a few questions like is the change significant ? Do I understand the effect of this change? Is the change worth launching after reviewing other business factors? Based on this you may either launch the change , perform some more tests or may not launch the change. You may need to perform A/A tests in pre and post periods for sanity checking and to see the effects of changes on users.

Conclusion :

A/B testing course by Google on Udacity is a must for anyone who wants to understand the process of A/B testing. The project at the end can further help you in understanding the concepts. This is just an overview of the course.