Package: betty-ml
An automatic differentiation library for generalized meta-learning and multilevel optimizationDocs|Tutorials|Examples|Paper|Citation|CASLpipinstallbetty-mlUpdate[Sep 22 2023]"SAMA: Making Scalable Meta Learning Practical" got accepted atNeurIPS 2023![Jan 21 2023]Betty got accepted as anotable-top-5% (oral)paper atICLR 2023![Jan 12 2023]We releaseBetty v0.2with new distributed training support for meta-learning! Currently
available features are:Distributed Data Parallel(DDP)ZeRO Redundancy Optimizer(ZeRO)(experimental)Fully Sharded Data Parallel(FSDP)You can now easily scale up meta-learning (or even meta-meta-learning) with one-liner change!Example:Meta-Weight-Net with RoBERTaTutorial:linkIntroductionBetty is aPyTorchlibrary for generalized meta-learning (GML)
and multilevel optimization (MLO) that allows asimpleandmodularprogramming interface for a number oflarge-scaleapplications includingmeta-learning,hyperparameter optimization,neural architecture search,data reweighting, and many more.With Betty, users simply need to do two things to implement any GML/MLO programs:Define each level's optimization problem using theProblemclass.Define the hierarchical problem structure using theEngineclass.Quick StartProblemBasicsEach level problem can be defined with seven components: (1) module, (2) optimizer, (3)
data loader, (4) loss function, (5) problem configuration, (6) name, and (7) other
optional components (e.g.  learning rate scheduler). The loss function (4) can be
defined via thetraining_stepmethod, while all other components can be provided
through the class constructor. For example, an image classification problem can be
defined as follows:frombetty.problemsimportImplicitProblemfrombetty.configsimportConfig# set up module, optimizer, data loader (i.e. (1)-(3))cls_module,cls_optimizer,cls_data_loader=setup_classification()classClassifier(ImplicitProblem):# set up loss functiondeftraining_step(self,batch):inputs,labels=batchoutputs=self.module(inputs)loss=F.cross_entropy(outputs,labels)returnloss# set up problem configurationcls_config=Config(type='darts',unroll_steps=1,log_step=100)# Classifier problem class instantiationcls_prob=Classifier(name='classifier',module=cls_module,optimizer=cls_optimizer,train_data_loader=cls_data_loader,config=cls_config)Interactions between problemsIn GML/MLO, each problem will often need to access modules from other problems to
define its loss function. This can be achieved by using thenameattribute as
follows:classHPO(ImplicitProblem):deftraining_step(self,batch):# set up hyperparameter optimization loss...# HPO problem class instantiationhpo_prob=HPO(name='hpo',module=...)classClassifier(ImplicitProblem):deftraining_step(self,batch):inputs,labels=batchoutputs=self.module(inputs)loss=F.cross_entropy(outputs,labels)"""accessing weight decay hyperparameter from anotherproblem HPO can be achieved by its name 'hpo'"""weight_decay=self.hpo()reg_loss=weight_decay*sum([p.norm().pow(2)forpinself.module.parameters()])returnloss+reg_losscls_prob=Classifier(name='classifier',module=...)EngineBasicsTheEngineclass handles the hierarchical dependencies between problems. In GML/MLO,
there are two types of dependencies: upper-to-lower (u2l) and lower-to-upper (l2u).
Both types of dependencies can be defined with a Python dictionary, where the key is
the starting node and the value is the list of destination nodes.frombettyimportEnginefrombetty.configsimportEngineConfig# set up all involved problemsproblems=[cls_prob,hpo_prob]# set up upper-to-lower and lower-to-upper dependenciesu2l={hpo_prob:[cls_prob]}l2u={cls_prob:[hpo_prob]}dependencies={'u2l':u2l,'l2u':l2u}# set up Engine configurationengine_config=EngineConfig(train_iters=10000,valid_step=100)# instantiate Engine classengine=Engine(problems=problems,dependencies=dependencies,config=engine_config)# execute multilevel optimizationengine.run()SinceEnginemanages the whole GML/MLO program, you can also perform a global validation
stage within it. All problems that comprise the GML/MLO program can again be accessed with
their names.classHPOEngine(Engine):# set up global validation@torch.no_grad()defvalidation(self):loss=0forinputs,labelsintest_loader:outputs=self.classifer(inputs)loss+=F.cross_entropy(outputs,targets)# Returned dict will be automatically logged after each validationreturn{'loss':loss}...engine=HPOEngine(problems=problems,dependencies=dependencies,config=engine_config)engine.run()Once we define all optimization problems and the hierarchical dependencies between them
with, respectively, theProblemclass and theEngineclass, all complicated internal
mechanisms of GML/MLO such as gradient calculation and optimization execution order will
be handled by Betty. For more details and advanced features, users can check out ourDocumentationandTutorials.Happy multilevel optimization programming!ApplicationsWe provide reference implementations of several GML/MLO applications, including:Hyperparameter OptimizationNeural Architecture SearchData ReweightingDomain Adaptation for Pretraining & Finetuning(Implicit) Model-Agnostic Meta-LearningWhile each of the above examples traditionally has a distinct implementation style, note
that our implementations share the same code structure thanks to Betty. More examples
are on the way!FeaturesGradient Approximation MethodsImplicit DifferentiationFinite Difference (or T1-T2) (DARTS: Differentiable Architecture Search)Neumann Series (Optimizing Millions of Hyperparameters by Implicit Differentiation)Conjugate Gradient (Meta-Learning with Implicit Gradients)Iterative DifferentiationReverse-mode Automatic Differentiation (Model-Agnostic Meta-Learning (MAML))TrainingGradient accumulationFP16/BF16 trainingDistributed data-parallel trainingGradient clippingLogging(PyTorch) TensorBoardwandbContributingWe welcome contributions from the community! Please see ourcontributing
guidelinesfor details on how to contribute to Betty.CitationIf you use Betty in your research, please citeour
paperwith the following Bibtex entry.@inproceedings{
choe2023betty,
title={Betty: An Automatic Differentiation Library for Multilevel Optimization},
author={Sang Keun Choe and Willie Neiswanger and Pengtao Xie and Eric Xing},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=LV_MeMS38Q9}
}LicenseBetty is licensed under theApache 2.0 License.