Package: pyrelational
PyRelationALPyRelationAL is an open source Python library for the rapid and reliable construction of active learning (AL) pipelines and strategies. The toolkit offers a modular design for a flexible workflow that enables active learning with as little change to your models and datasets as possible. The package is primarily aimed at researchers so that they can rapidly reimplement, adapt, and create novel active learning strategies. For more information on how we achieve this you can consult the sections below, our comprehensive docs, or our paper. PyRelationAL is principally designed with PyTorch workflows in mind (for Bayesian inference approximation with neural networks) but is designed from the start to be agnostic to the user's choice of ML framework for model implementation.Detailed in theoverviewsection below, PyRelationAL is centered around 5 modules for the development of AL pipelines and strategies. Allowing the user to freely focus on different aspects of the active learning cycle whilst ensuring other components adhere to a consistent API.DataManager: Data management in AL pipelines.ModelManager: Framework agnostic wrappers for ML models to work with PyRelationAL.Strategy: Module for developing active learning strategies.Oracle: Interfaces for different oracles and labelling tools.Pipeline: Facilitate the communication between different PyRelationAL modules to run an active learning cycle.Uncertainty: PyRelationAL also offers special wrappers for PyTorch modules that enable Bayesian inference approximation for deep active learning.Furthermore, the package comes with a growing number ofbenchmark datasets and default AL tasksbased on literature with associated public licenses to help researchers test their AL strategies and build on a common set of benchmarks.One of our main incentives for making this library is to get more people interested in research and development of AL. hence we have made primers, tutorials, and examples available on our website for newcomers (and experienced AL practitioners alike). Experienced users can refer to our numerous examples to get started on creating custom pipelines and strategies in their AL projects.Quick installpipinstallpyrelationalThePyRelationALpackageExample# Active Learning packagefrompyrelational.data_managersimportDataManagerfrompyrelational.model_managersimportModelManagerfrompyrelational.strategies.classificationimportLeastConfidenceStrategyfrompyrelational.oraclesimportBenchmarkOraclefrompyrelational.pipelineimportPipeline# Instantiate data-loaders, models, trainers the usual Pytorch/PytorchLightning way# In most cases, no change is needed to current workflow to incorporate# active learningdata_manager=DataManager(dataset,train_indices,validation_indices,test_indices,labelled_indices)# Create a ModelManager that will handle model instantiation, training and evaluationmodel=ModelManager(ModelConstructor,model_config,trainer_config,**kwargs)# Use the various implemented active learning strategies or define your ownstrategy=LeastConfidenceStrategy()# Interface with various dataset annotation tools or use an oracle for Benchmarkingoracle=BenchmarkOracle()# Bring it all together within a Pipeline that manages the active learning cyclepipeline=Pipeline(data_manager,model,strategy,oracle)# Use the pipeline to run active learning cycles and log performance datato_annotate=pipeline.step(num_annotate=5)pipeline.run(num_annotate=10)print(pipeline)OverviewThePyRelationALpackage decomposes the active learning workflow into five main components: 1) aDataManager, 2) aModelManager, 3) an ALStrategy, 4) anOracleand 5) aPipelinethat runs the show.TheDataManager(pyrelational.data_managers.DataManager) wraps around a PyTorch Dataset object and handles dataloader instantiation as well as tracking and updating of labelled and unlabelled sample pools. The Dataset must have
specific properties for compatibility: 1) The labels must be contained within an attribute (by default PyRelationAL assumes that this is 'y', but the user can change this) 2) The__getitem__method returns a tuple of tensors, where the first element is a tensor of featuresTheModelManager(pyrelational.model_managers.ModelManager) wraps a user defined ML model (e.g. PyTorch Module, Flax module, or scikit-learn estimator) and primarily handles instantiation, training, testing, as well as uncertainty quantification (e.g. ensembling, MC-dropout) if relevant. It enables the use of ML models implemented using different ML frameworks (for example seeexamples/demo/model_gaussianprocesses.pyorexamples/demo/scikit_estimator.py) with PyRelationAL workflows.The ALStrategy(pyrelational.strategies.Strategy) defines an active learning strategy. We like to typically think of strategies being compositions of aninformativeness measureand aquery selection algorithmthat selects observations based on the perceived informativeness. Together they compute the utility of a query or set of queries for a batch active mode strategy. We define various classic strategies for classification, regression, and task-agnostic scenarios based on the informativeness measures defined inpyrelational.informativeness. The flexible nature of theStrategyallows for the construction of strategies from simple serial uncertainty sampling approaches to complex agents that leverage several informativeness measures, state and learning based query selection algorithms, with query batch building bandits under uncertainty from noisy oracles. Users can implement their own strategies by overriding the.__call__()method. Look atexamples/demo/model_badge.pyfor an example.TheOracle(pyrelational.oracles.Oracle) is an entity which provides annotations to observations, as suggested by an active learning strategy. In PyRelationAL, the oracle is an interface to whatever annotation tool is being used (e.g. LabelStudio or a bespoke lab-in-the-loop setup). For benchmarking active learning strategies this is not necessary, and we provide aBenchmarkOraclefor this purpose.ThePipeline(pyrelational.pipeline.Pipeline) arbitrates the active learning cycle and the communication between itsDataManager,ModelManager,Strategy,Oraclecomponents. It also logs various data for the evaluation of different active learning strategies such as the performance of the model at each iteration.In addition to the main modules above we offer tools foruncertainty estimation. In recognition of the growing use of deep learning models we offer a suite of methods for Bayesian inference approximation to quantify uncertainty coming from the functional model such as MCDropout and ensembles of models (which may be used to also define query by committee and query by disagreement strategies).Finally we to help test and benchmark strategies we offerBenchmark datasetsandAL task configurations. We offer an API to a selection of datasets used previously in AL literature and offer each in several AL task configurations, such as cold and warm initialisations, for pain free benchmarking. For more details see our paper and documentation.In the next section we briefly outline currently available strategies, informativeness measures, uncertainty estimation methods and some planned modules.List of included strategies and uncertainty estimation methods (constantly growing!)Uncertainty EstimationMCDropoutEnsemble of models (a.k.a. commitee)Informativeness measures included in the libraryRegression (N.B. PyRelationAL currently only supports single scalar regression tasks)Mean prediction scoreLeast confidence scoreExpected improvement scoreThompson sampling scoreUpper confidence bound (UCB) scoreBALDClassification (N.B. PyRelationAL does not support multi-label classification at the moment)Least confidenceMargin confidenceEntropy based confidenceRatio based confidenceBALDModel agnostic and diversity sampling based approachesRepresentative samplingDiversity sampling (with user defined relative distance functions)Random acquisitionBADGEQuickstart & examplesTheexamples/folder contains multiple scripts and notebooks demonstrating how to use PyRelationAL effectively.The diverse examples scripts and notebooks aim to showcase how to use pyrelational in various scenario. Specifically,examples with regressionlightning_diversity_regression.pylightning_mixed_regression.pymcdropout_uncertainty_regression.pymodel_gaussianprocesses.pymodel_badge.pyexamples with classification tasksensemble_uncertainty_classification.pylightning_diversity_classification.pylightning_representative_classification.pymcdropout_uncertainty_classification.pyscikit_estimator.pyexamples with task-agnostic acquisitionlightning_diversity_classification.pylightning_representative_classification.pylightning_diversity_regression.pymodel_badge.pyexamples showcasing different uncertainty estimatorensemble_uncertainty_classification.pymcdropout_uncertainty_classification.pymodel_gaussianprocesses.pymodel_badge.pyexamples on how to create a custom acquisition strategymodel_badge.pylightning_mixed_regression.pyexamples using different ML frameworksmodel_gaussianprocesses.pyscikit_estimator.pyContributing to PyRelationALWe welcome contributions to PyRelationAL, please see and adhere to theCONTRIBUTING.mdandCODE_OF_CONDUCT.mdguidelines.Prerequisites and setupFor those just using the package, installation only requires standard ML packages and PyTorch. More advanced users or those wishing to contribute should start with a new virtual environment (miniconda environment recommended) and install standard learning packages and numerical tools.pipinstall-rrequirements.txtIf you wish to contribute to the code, runpre-commit installafter the above step.Organisation of repositorypyrelationalfolder contains the source code for the PyRelationAL package. It contains the main sub-packages for active learning strategies, various informativeness measures, and methods for estimating posterior uncertainties.examplesfolder contains various example scripts and notebooks detailing how the package can be used to construct novel strategies, work with different ML frameworks, and use your own datatestsfolder contains unit tests for pyrelational packagedocsfolder contains documentation and assets for docsBuilding the docsMake sure you havesphinxandsphinx-rtd-themepackages installed (pip install sphinx sphinx_rtd_themewill install this).To generate the docs,cdinto thedocs/directory and runmake html. This will generate the docs
atdocs/_build/html/index.html.Citing our work@article{pyrelational,
  title={PyRelationAL: A Library for Active Learning Research and Development},
  author={Scherer, Paul and Gaudelet, Thomas and Pouplin, Alison and Soman, Jyothish and Edwards, Lindsay and Taylor-King, Jake P and others},
  journal={arXiv preprint arXiv:2205.11117},
  year={2022}
}