Package: tw-experimentation
TW Experimentation: A library for automated A/B testingTW Experimentationis a library to design experiments, check data, run statistical tests and make decisionsTable of ContentsTW Experimentation: A library for automated A/B testing and causal inferenceWhat can this do for you?1. Designing experiments2. Evaluating resultsInstallationQuick StartNotebooksStreamlit web appFor DevelopersTestingWhat can this do for you?The experimentation library can help you with:Sample Size Calculatorlink- here you can estimate how many observations you need to perform an experiment on the given parameters.Integrity checks + Evaluationlink- here you can check important plots to understand your data better and then run statistical evaluation. Our library will automatically detect if it is a binary metric or continuous and then apply specific method as well as multiple hypothesis correction if needed.Evaluation (Bayesian A/B testing)link- here you can apply bayesian evaluation on your dataset1. Designing experimentsBy usingTW Experimentationyou can design your experiments, choose sample size, evaluate the experiment.2. Evaluating resultsYou can use various statistical tests for the metrics provided. (Frequentist / Bayesian)
For this goal you can use jupyter notebooks or streamlit app with user-friendly interface.
Using this repo you can:Run frequentist evaluationRun bayesian evaluationApply multiple hypothesis correctionRemove outliersCheck different plotsRun segmentation to check evaluation on specific segments as well as find unusual segments using our another tool: wise-pizzaInstallationYou can easily install this repo using "pip"pip install tw-experimentationThen you can just use all functionality.For running streamlit app please just open the terminal and run:run_tw_experimentation_streamlitYou can install the package  via the dependency manager poetry after cloning/git pull/download it as a zip from this repository.To do so, clone the repository by runninggit clone git@github.com:transferwise/tw-experimentation.gitfrom terminal.
To set up poetry, runmake set-up-poetry-macfor mac (or linux) andmake set-up-poetry-windowsfor windows.
Then, runmake run-streamlit-poetryfrom the root of the package folder.Alernative:TW Experimentation requires the following libraries to work which you can find in the .yml file. To install requirements please make sure you have installed the package manager Anaconda and then run the following commands in the terminal:conda env create -n <my_env> -f envs/environment.yml
conda activate <my_env>If you are using Windows, please do these additional steps:pick a jaxlib-0.3.7 wheel from herehttps://whls.blob.core.windows.net/unstable/index.htmland install it manually (pip install <wheel_url>)Install jax==0.3.7If you have any problems with jax on Mac, please do the following:pip uninstall jax jaxlib
conda install -c conda-forge jaxlib
conda install -c conda-forge jaxQuick StartMake sure you have followed the installation instructions.NotebooksYou can use the jupyter notebooks1_pre_experiment.ipynbor2_integrity_checks + evaluation.ipynbfor experiments design and evaluation.
The tw experimentation package can be used for different things, for example for analyzing results:df=pd.read_csv('experiment.csv')ed=ExperimentDataset(data=df,variant="T",targets=['conversion','revenue'],date='trigger_dates',pre_experiment_cols=None,n_variants=2,)ed.preprocess_dataset(remove_outliers=True)This code will generate the data model for experiment analysisAnd then you can run evaluationevaluation=FrequentistEvaluation(experiment_dataset)evaluation.start()Streamlit web appFor running streamlit app please just open the terminal and run after "pip install" section:run_tw_experimentation_streamlitOr:Open terminal and navigate to the repository.
Then navigate to the folder./tw_experimentation/streamlit.Now run the commandstreamlit run Main.pyand the app should open in your browser.Then you can use or test dataset from data/test_data.csvTip on navigation:ls- show files in current directorypwd- print current directory addresscd- change directory, e.g.cd ./tw_experimentation/streamlitFor DevelopersTestingWe usePyTestfor testing. If you want to contribute code, make sure that the tests in tests/ run without errors.