Package: twiser
The Twiser package implements a Python library for variance reduction in A/B tests using pre-experiment covariates supporting publication[1].
These functions extend the idea of using pre-experiment data for variance reduction previously proposed in publication[2].InstallationOnlyPython>=3.7is officially supported, but older versions of Python likely work as well.The package is installed with:pipinstalltwiserSeeGitHub,PyPI, andRead the Docs.Example UsageA full demo notebook of the package is given indemo/survey_loan.ipynb.
Here is a snippet of the different methods from the notebook:Setup a predictor as a control variateFirst, we need to define a regression model.
We can use anything that fits the sklearn idiom offitandpredictmethods.
This predictor is used to take then x darray of treatment unit covariatesx_covariatesand predict the treatment outcomesn-length outcome arrayx.
Likewise, it makes predictions from them x darray of control unit covariatesy_covariatesto the controlm-length outcome arrayy.predictor=RandomForestRegressor(criterion="squared_error",random_state=0)Basic z-testFirst, we apply the basic two-sample z-test included in Twiser.
This works basically the same asscipy.stats.ttest_ind.estimate,(lb,ub),pval=twiser.ztest(x,y,alpha=0.05)show_output(estimate,(lb,ub),pval)ATE estimate: 0.80 in (-0.14, 1.75), CI width of 1.89, p = 0.0954Variance reduction with held out dataNext, we apply variance reduction where the predictor was trained on a
held out 30% of the data. This is the easiest to show validity, but some
of the added power is lost because not all data is used in the test.estimate,(lb,ub),pval=twiser.ztest_held_out_train(x,x_covariates,y,y_covariates,alpha=0.05,train_frac=0.3,predictor=predictor,random=np.random.RandomState(123),)show_output(estimate,(lb,ub),pval)ATE estimate: 1.40 in (0.20, 2.59), CI width of 2.39, p = 0.0217*Variance reduction with cross validationTo be more statistically efficient we train and predict using 10-fold
cross validation. Here, no data is wasted. As we can see it is a more
significant result.estimate,(lb,ub),pval=twiser.ztest_cross_val_train(x,x_covariates,y,y_covariates,alpha=0.05,k_fold=10,predictor=predictor,random=np.random.RandomState(123),)show_output(estimate,(lb,ub),pval)ATE estimate: 1.38 in (0.51, 2.25), CI width of 1.74, p = 0.0019*Variance reduction in-sampleIn the literature it is popular to train the predictor in the same
sample as the test. This often gives the most power. However, any
overfitting in the predictor can also invalidate the results.estimate,(lb,ub),pval=twiser.ztest_in_sample_train(x,x_covariates,y,y_covariates,alpha=0.05,predictor=predictor,random=np.random.RandomState(123),)show_output(estimate,(lb,ub),pval)ATE estimate: 0.86 in (0.24, 1.49), CI width of 1.24, p = 0.0065*Other interfacesIt is also possible to call these methods using raw control predictions instead of training the predictor in the Twiser method.
It also supports a sufficient statistics interface for working with large datasets.
See thedocumentationfor details.SupportCreate anew issue.LinksThesourceis hosted on GitHub.Thedocumentationis hosted at Read the Docs.Installable fromPyPI.References[1]R. Turner, U. Pavalanathan, S. Webb, N. Hammerla, B. Cohn, and A. Fu. Isotonic regression
adjustment for variance reduction. In CODE@MIT, 2021.[2]A. Deng, Y. Xu, R. Kohavi, and T. Walker. Improving the sensitivity of online controlled
experiments by utilizing pre-experiment data. In Proceedings of the Sixth ACM International
Conference on Web Search and Data Mining, pages 123–132, 2013.[3]A. Poyarkov, A. Drutsa, A. Khalyavin, G. Gusev, and P. Serdyukov. Boosted decision tree
regression adjustment for variance reduction in online controlled experiments. In Proceedings of
the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages
235–244, 2016.[4]I. Barr. Reducing the variance of A/B tests using prior information. Degenerate State, Jun
2018.LicenseThis project is licensed under the Apache 2 License - see the LICENSE file for details.