Package: pyabtest
PyabtestA simple tool to calculate P-value after conducting an A/B experimentA/B experiment & Hypothesis testingNormally we run an A/B experiment to see whether a new model brings some improvement in the production metrics. After running the experiment for a fixed time period, we use hypothesis testing to scientifically come to a conclusion whether to accept the new feature or not. Usually, hypothesis testing has following components:Null hypothesis: New model does not bring any improvementAlternative hypothesis: New model does bring some improvementThis tool will be useful to calculate P-value to check whether we can reject the null hypothesis or not.InstallationUse the package managerpipto install pyabtestpipinstallpyabtestUsageFollowing functionalities are exposed in this package1. Test for Sample Ratio Mismatch (SRM)This is a test to check whether we have created audience for control vs test in a truly random manner. If there is an SRM, we should discard the A/B test results as control and variant have different type of audience. For example, we can pass following numbers in control vs test to check for SRM.Number of male vs Number of femaleNumber of users of age < 40 vs Number of users of age >= 40Number of active users vs Number of inactive usersNumber of english speaking users vs Number of non-english speaking usersNumber of mobile users vs Number of desktop usersInput: Control group 1 size, Control group 2 size, Variant group 1 size, Variant group 2 sizeOutput: P-value, Alpha, Decision>>>importpyabtest>>>pyabtest.test_for_sample_ratio_mismatch(control_group1_size=1000,control_group2_size=2000,variant_group1_size=1010,variant_group2_size=1990,alpha=0.05){'P-value':0.78445,'Alpha value (significance level)':0.05,'Decision':"Don't discard A/B test results"}Test used: Chi-squared Test2. Test for Binary MetricThis test can be used when when the result/action/feedback is binary & we want to see if variant observations are coming from a differant population when compared to control. For example, this test can be used in the following situations:Clicks vs No clicksCart vs No cartOrder vs No orderNumber of zero search results vs Number of non-zero serach resultsNumber of successful sessions vs Number of non-successful sessionsNumber of positive reviews vs Number of negative reviewsNumber of converted users vs Number of non-converted usersInput: No. of success in Control, No. of failures in Control, No. of success in variant, No. of failures in variantOutput: P-value, Alpha, Decision>>>importpyabtest>>>pyabtest.test_for_binary_metric(control_success=50,control_failures=1000,variant_success=40,variant_failures=900,alpha=0.05){'P-value':0.58718,'Alpha value (significance level)':0.05,'Decision':'Do not reject null hypothesis'}Test used: Chi-squared Test3. Test for Numeric MetricThis test can be used for any generic numeric metric (Count or Fraction). We can use this test even if the observations do not follow a normal distribution. In general, this test does not assume anything about the distribution as it is a non-parametric test. Example metrics include:Number of clicks per unique userNumber of carts per unique userNumber of orders per unique userClicks/Views per unique userOrders/Views per unique userOrders/Session per unique userRevenue per unique userSession time per unique userOrder value per unique userSuccessful sessions per unique userInput: Control array (Ex: Array containing no. of clicks for each user in control, order does not matter), Variant array (Ex: Array containing no. of clicks for each user in variant, order does not matter)Output: P-value, Alpha, Decision>>>importpyabtest>>>fromnumpyimportrandom>>>random.seed(12)>>>pyabtest.test_for_numeric_metric(control_observations=random.randint(100,size=(20)),variant_observations=random.randint(100,size=(20)),test_type="ttest",alpha=0.05){'P-value':0.18443,'Alpha value (significance level)':0.05,'Decision':'Do not reject null hypothesis'}Test used by default: Welch's t-test. Pass test_type="bootstrap" if you want to use Bootstrap test instead of Welch's t-test.LicenseMITReferencesHypothesis testingChi-squared testWelch's t-testBootstrap test1,Bootstrap test2AuthorRama BadrinathEmail:ramab1988@gmail.com