Package: bmdal-reg
Deep Batch Active Learning for RegressionThis repository contains code accompanying our paper"A Framework and Benchmark for Deep Batch Active Learning for Regression". It can be used for the following purposes:Apply various pool-based Batch Mode Deep Active Learning (BMDAL) algorithms for regression to custom neural networks (NNs) or kernel methodsUse our NN for tabular regression through a simple scikit-learn style interfaceDownload large tabular regression data sets from our benchmarkCompare BMDAL algorithms using our benchmarkIf you use this code for research purposes, plese citeour paper.Implemented methodsThis repository contains an efficient implementation of our framework for building BMDAL algorithms for NN regression, which includesan implementation of the greedyCore-Setmethod for regressionan implementation ofBAITfor regressionan implementation of variants ofACS-FWfor regressionan adaptation ofBADGEfor regressionan adaptation ofBatchBALDfor regressionan adaptation ofBALDfor regressionan implementation of our own LCMD-TP method with sketched gradient featuresVersionsThe commit corresponding toversion 1of our arXiv paper is taggedarxiv_v1and also archived with the corresponding data atDaRUS.The commit corresponding toversion 2of our arXiv paper is taggedarxiv_v2and also archived with the corresponding data atDaRUS.The commit corresponding toversion 3of our arXiv paper is taggedarxiv_v3and also archived with the corresponding data atDaRUS.
Results from versions 1 and 2 are run with slightly different options, hence they should not be mixed though the numbers (except for the runtimes) are very similar. Changes in version 2 are listed below. Changes in version 3 are minimal and produce almost the same numbers.LicenseThis source code is licensed under the Apache 2.0 license. However, the implementation of the acs-rf-hyper kernel transformation inbmdal/features.pyis adapted from the source code athttps://github.com/rpinsler/active-bayesian-coresets, which comes with its own (non-commercial) license. Please respect this license when using the acs-rf-hyper transformation directly frombmdal/features.pyor indirectly through the interface provided atbmdal/algorithms.py.InstallationThis code has been tested with Python 3.9.2 but may be compatible with versions down to Python 3.6.Through pipFor running our NN and the active learning methods, apipinstallation is sufficient. The library can be installed viapip3 install bmdal_regWhen using our benchmarking code through apipinstallation, the paths where experiment data and plots are saved can be modified through changing the corresponding path variables ofbmdal_reg.custom_paths.CustomPathsbefore running the benchmark.ManuallyFor certain purposes, especially trying new methods and running the benchmark, it might be helpful or necessary to modify the code. For this, the code can be manually installed via cloning theGitHub repositoryand then following the instructions below:The following packages (available throughpip) need to be installed:General:torch,numpy,dillFor running experiments withrun_experiments.py:psutilFor plotting the experiment results:matplotlib,seaborn,scipyFor downloading the data sets withdownload_data.py:pandas,openml,mat4pyIf you want to install PyTorch with GPU support, please follow the instructionson the PyTorch website. The following command installs the versions of the libraries we used for running the benchmark, which however come with security warnings in the meantime:pip3 install -r requirements_original.txtAlternatively, the following command installs current versions of the packages:pip3 install torch numpy dill psutil matplotlib seaborn pandas openml mat4py scipyClone the repository (or download the files from the repository) and change to its folder:git clone git@github.com:dholzmueller/bmdal_reg.git
cd bmdal_regThen, copy the filebmdal_reg/custom_paths.py.defaulttobmdal_reg/custom_paths.pyviacp bmdal_reg/custom_paths.py.default bmdal_reg/custom_paths.pyand, if you want to, adjust the paths incustom_paths.pyto specify the folders in which you want to save data and results.Downloading dataIf you want to use the benchmark data sets, you need to download and preprocess them. We do not provide preprocessed versions of the data sets to avoid copyright issues, but you can download and preprocess the data sets usingpython3 download_data.pyNote that this may take a while. This depends of course on your download speed. The preprocessing is mostly fast, but for the (large) methane data set, it took around five minutes and 25 GB of RAM for us. If you cannot download/process the data due to limited RAM, please contact the main developer (see below).UsageDepending on your use case, some of the following introductory Jupyter notebooks may be helpful:examples/benchmark.ipynbshows how to download or reproduce our experimental results, how to benchmark other methods, and how to evaluate the results.examples/using_bmdal.ipynbshows how to apply our BMDAL framework to your use-case.examples/framework_details.ipynbexplains how our BMDAL framework is implemented, which may be relevant for advanced usage.New:examples/nn_interface.ipynbshows how our NN configuration can be used (without active learning) through a simple scikit-learn style interface.Besides these notebooks, you can also take a look at the code directly. The more important parts of our code are documented with docstrings.Code structureThe code is structured as follows:Library code is contained in thebmdal_regfolder, while directly executable files are contained in the top-level folder.Thebmdal_reg/bmdalfolder contains the implementation of all BMDAL methods, with its main interface inbmdal/algorithms.py.Thebmdal_reg/evaluationfolder contains code for analyzing and plotting generated data, which is called fromrun_evaluation.py.Theexamplesfolder contains Jupyter Notebooks for instructive purposes as mentioned above.The filedownload_data.pyallows for downloading the data,run_experiments.pyallows for starting the experiments,test_single_task.pyallows for testing a configuration on a data set, andrename_algs.pycontains some functionality for adjusting experiment data in case of mistakes.The filecheck_task_learnability.pyhas been used to check the reduction in RMSE on different data sets when going from 256 to 4352 random samples. We used this to sort out the data sets where the reduction in RMSE was too small, since these data sets are unlikely to make a substantial difference in the benchmark results.The filesbmdal_reg/data.py,bmdal_reg/layers.py,bmdal_reg/models.py,bmdal_reg/task_execution.py,bmdal_reg/train.pyandbmdal_reg/utils.pyimplement parts of data loading, training, and parallel execution.Updates to the second version of the benchmarkAdded the BAIT selection method with variants BAIT-F and BAIT-FB.For the normalization of input data, mean and standard deviations for the features are now computed on training and pool set instead of only on the initial training set.More precise runtime measurement through CUDA synchronize (only applied in one of the 20 splits where only one process is run per GPU).Now, 64-bit floating point computations are used for computations involving posterior transformations. This can sometimes cause RAM overflows when using parallel execution, though.We use $\sigma^2 = 10^{-6}$ instead of $\sigma^2 = 10^{-4}$ now, which still works well due to the change to 64-bit floats.The computation of the last-layer kernel does not require the full backward pass now since the earlier layers setrequires_grad=Falsefor the computation.Fixed a discrepancy between the implementation of selection methods and the corresponding paper pseudocode: Previously, some selection methods could re-select already selected samples in case of numerical issues, which triggered a code filling up the batch with random samples. Now, selecting already selected samples is explicitly prevented.Changed the interface ofrun_experiments.pyto be based on lists instead of callbacks.Updates to the third version of the benchmarkWe fixed a bug where MaxDist, LCMD and KMeansPP selected the first point incorrectly when applied in TP mode (i.e., when usingsel_with_train=True). This only led to negligible differences at the batch sizes used in our benchmark.We fixed a bug where MaxDet and BAIT could select a training point for the batch, which would then cause the rest of the batch to be filled up randomly instead. This bug should occur rarely since training points usually have very low uncertainty, and it did not affect our benchmark results.We fixed a problem where adding jitter if needed for the Cholesky decomposition was only performed for CPU computations. This should also not affect the benchmark results because the benchmark was run with regularization and 64-bit floats.ContributorsDavid Holzmüller(main developer)Viktor Zaverkin(contributed to the evaluation code)If you would like to contribute to the code or would be interested in additional features, please contact David Holzmüller.