Package: autotst
AutoML Two-Sample Testautotstis a Python package for easy-to-use two-sample testing and distribution shift detection.Given two datasetssample_Pandsample_Qdrawn from distributions $P$ and $Q$, the
goal is to estimate a $p$-value for the null hypothesis $P=Q$.autotstachieves this by learning a witness function and taking its mean discrepancy as a test statistic
(see References).The package provides functionalities to prepare the data, an interface to train an ML model, and methods
to evaluate $p$-values and interpret results.By default, autotst uses the Tabular Predictor ofAutoGluon, but it is easy
to wrap and use your own favorite ML framework (see below).The full documentation of the package can be foundhere.InstallationRequires at least Python 3.7. Since the installation also installs AutoGluon, it can take a few moments.pip install autotstHow to useautotstWe provide worked out examples in the 'Example' directory. In the following assume thatsample_Pandsample_Qare twonumpyarrays containing samples from $P$ and $Q$.Default Usage:The easiest way to compute a $p$-value is to use the default settingsimportautotsttst=autotst.AutoTST(sample_P,sample_Q)p_value=tst.p_value()You would then reject the null hypothesis ifp_valueis smaller or equal to your significance level.Customizing the testing pipelineWe highly recommend to use the pipeline step by step, which would look like this:importautotstfromautotst.modelimportAutoGluonTabularPredictortst=autotst.AutoTST(sample_P,sample_Q,split_ratio=0.5,model=AutoGluonTabularPredictor)tst.split_data()tst.fit_witness(time_limit=60)# time limit adjustable to your needs (in seconds)p_value=tst.p_value_evaluate(permutations=10000)# control number of permutations in the estimationThis allows you to change the time limit for fitting the witness function and you can also pass other
arguments to the fit model (seeAutoGluondocumentation).Since the permutations are very cheap, the default number of permutations is relatively high and should work for most
use-cases. If your significance level is, say, smaller than 1/1000, consider increasing it further.Customizing the machine learning modelIf you have good domain knowledge about your problem and believe that a specific ML framework will work well,
it is easy to wrap your model.
Therefore, simply inherit from the classModeland wrap the methods
(see our implementation inmodel.py).You can then run the test simply by importing your model and initializing the test accordingly.importautotsttst=autotst.AutoTST(sample_P,sample_Q,model=YourCustomModel)......etc.We also provide a wrapper forAutoGluonImagePredictor. However, it seems that this should not be used
with small datasets and small training times.ReferencesIf you use this package, please cite this paper:Jonas M. Kübler, Vincent Stimper, Simon Buchholz, Krikamol Muandet, Bernhard Schölkopf: "AutoML Two-Sample Test",arXiv 2206.08843(2022)Bibtex:@misc{kubler2022autotst,
  doi = {10.48550/ARXIV.2206.08843},
  url = {https://arxiv.org/abs/2206.08843},
  author = {Kübler, Jonas M. and Stimper, Vincent and Buchholz, Simon and Muandet, Krikamol and Schölkopf, Bernhard},  
  title = {AutoML Two-Sample Test},
  publisher = {arXiv},
  year = {2022},
}