Package: hpman
hpman (超参侠): The uncompromising hyperparameter manager.English |简体中文hpmanis a hyperparameter manager (HPM) library that truly make sense.
It enables a Distributed-Centralized HPM experience in deep learning
experiment. You can define hyperparameters anywhere, but manage them as a
whole.hpman is intended to be used as a basic building blocks for downstream tools, such as
command line interface, IDE integration, experiment management system, etc.hpman supports Python version greater equal than 3.5.hpman (超参侠): The uncompromising hyperparameter manager.Story / BackgroundCentralized HPMHowever ...Distributed HPMDistributed-Centralized HPMInstallationUsageExamplesFeaturesDesign principlesAribitrary ImportsDefine HyperparametersStatic ParsingRuntime Value Getter/SetterHintsNested HyperparametersContributingLicenseStory / Backgroundlib.py:# File: lib.pyfromhpman.mimport_defadd():return_("a",0)+_("b",0)defmult():return_("a")*_("b")main.py:#!/usr/bin/env python3importosimportargparsefromhpman.mimport_importlibdefmain():basedir=os.path.dirname(os.path.realpath(__file__))_.parse_file(basedir)parser=argparse.ArgumentParser()parser.add_argument("-a",default=_.get_value("a"),type=int)parser.add_argument("-b",default=_.get_value("b"),type=int)args=parser.parse_args()_.set_value("a",args.a)_.set_value("b",args.b)print("a ={}".format(_.get_value("a")))print("b ={}".format(_.get_value("b")))print("lib.add() ={}".format(lib.add()))print("lib.mult() ={}".format(lib.mult()))if__name__=="__main__":main()Results:$./main.pya=0b=0lib.add()=0lib.mult()=0$./main.py-a2-b3a=2b=3lib.add()=5lib.mult()=6The core library is designed as a backend for hyperparameter data manipulation,
rather than an end-to-end solution.It is highly recommend to start with a
better frontend:CLI frontend:hpargparseInstallationpython3 -m pip install hpmanStoryManaging ever-changing hyperparameters is a pain in the a**.
From the practice of performing enormous amount of deep learning experiments,
we found two existing hyperparameter managing patterns of the utmost
prevalence.Centralized HPMWe call the first type "centralized HPM". It follows the way of
configuration management in traditional software, regardless of using a python
file or json or yaml or whatever that can store some key-value mapping (may
remind you ofsettings.ini,nginx.conf,config.yamletc.):# File: config.pyBATCH_SIZE=256NUM_EPOCH=120LEARNING_RATE=1e-1WEIGHT_DECAY=4e-5OPTIMIZER='SGD'LR_DECAY_EPOCHS=[30,60,90]HIDDEN_CHANNELS=128NUM_LAYERS=5INPUT_CHANNELS=784OUTPUT_CHANNELS=10# File: model.pyfromtorchimportnnimportconfigdefbuild_model():returnnn.Sequence([nn.Sequence(nn.Linear(config.INPUT_CHANNELS,config.HIDDEN_CHANNELS),nn.BatchNorm1d(config.HIDDEN_CHANNELS),nn.ReLU())]+[nn.Sequence(nn.Linear(config.HIDDEN_CHANNELS,config.HIDDEN_CHANNELS),nn.BatchNorm1d(config.HIDDEN_CHANNELS),nn.ReLU())foriinrange(config.NUM_LAYERS-1)]+[nn.Linear(config.HIDDEN_CHANNELS,config.OUTPUT_CHANNELS)])This way of managing hyperparameters is widely seen in machine learning
libraries, e.g., xgboost, whose hyperparameters are fairly stable compare than
that in deep learning research.However ...However, it is quite common for researchers to add some hyperparameters at
their inspiration (e.g., suddenly come up with a "Temperature" parameter in
softmax.). They found pleasure in tweaking the hyperparameters, but quickly
abandon it if the experiment goes wrong.  These acts are calledNon-Recurring
Engineering (NRE).In these cases, the "centralized HPM" reveals obvious drawbacks:Whenever you need to introduce a new hyperparameter, you must kind of
"declare" it in the configuration file, while using it in some
deeply-nested easy-to-forget files.Whenever you need to abandon an existing hyperparameter, you must not only
remove all the appearances of that hyperparameter in some deeply-nested
easy-to-forget files, but also remove it in the centralized configuration
file.There's a "Heisenberg uncertainty principle" on hyperparameters: you cannot
know both what and where the hyperparameters are at the same time.
The context around where the hyperparameter are used conveys valuable
information of the precise usecase of that hyperparameter.
You can either look it up in the code, or in the centralized config file.These drawbacks essentially requires the user to maintain a distributed data
structure, which not only induces great mental burden doing experiments,
but also be error-prone to bugs.Distributed HPMSo researchers come to another solution: forget about config files; define and
use whatever hyperparameters whenever you need, anywhere in the project. We
call this "Distributed HPM".  However, this is hardly called "management"; it
is more like anarchism: no management is the best management. This makes add a
hyperparameter cheap: let yourself free and do whatever you want.Let it go, let it gofromtorchimportnndefbuild_model():hidden_channels=128# <-- hyperparameterreturnnn.Sequence([nn.Sequence(nn.Linear(784,hidden_channels),# <-- hyperparameternn.BatchNorm1d(hidden_channels),nn.ReLU())]+[nn.Sequence(nn.Linear(hidden_channels,hidden_channels),nn.BatchNorm1d(hidden_channels),nn.ReLU())foriinrange(4)# <-- hyperparameter]+[nn.Linear(hidden_channels,10)# <-- hyperparameter])However, barbaric growth of hyperparameters of different names in different
places without governance would soon run into a disaster in knowledge sharing,
communication, reproduction, and engineering. Nobody knows what happened, when
did it happen, and nobody knows how to know easily. You know nothing, unless
you read and diff through all the source codes.You know nothing, Jon Snow.咱也不知道，咱也不敢问呀Distributed-Centralized HPMNow we have two ways of managing hyperparameters: one is good for engineering
but inconvenient for researchers, another one is convenient for researchers,
but bad for engineering.We are uncompromising. We did not want to make a decision between these two
choices; we want the best of both worlds.Only children make choices, adults want them all.小孩子才做选择，大人全都要After some trial and error, we came up with a design like this:main.py#!/usr/bin/env python3fromhpman.mimport_importhpargparseimportargparsedeffunc():weight_decay=_("weight_decay",1e-5)print("weight decay is{}".format(weight_decay))defmain():parser=argparse.ArgumentParser()_.parse_file(__file__)hpargparse.bind(parser,_)parser.parse_args()func()if__name__=="__main__":main()and you can:$./main.py
weightdecayis1e-05
$./main.py--weight-decay1e-4
weightdecayis0.0001
$./main.py--weight-decay1e-4--hp-list
weight_decay:0.0001
$./main.py--weight-decay1e-4--hp-listdetail
Allhyperparameters:['weight_decay']Details:
+--------------+--------+---------+--------------------------------------------------------------+|name|type|value|details|+==============+========+=========+==============================================================+|weight_decay|float|0.0001|occurrence[0]:|||||./main.py:10|||||5:|||||6:importargparse|||||7:|||||8:|||||9:deffunc():|||||==>10:weight_decay=_("weight_decay",1e-5)|||||11:print("weight decay is {}".format(weight_decay))|||||12:|||||13:|||||14:defmain():|||||15:parser=argparse.ArgumentParser()|+--------------+--------+---------+--------------------------------------------------------------+
$./main.py-h
usage:main.py[-h][--weight-decayWEIGHT_DECAY][--hp-saveHP_SAVE][--hp-loadHP_LOAD][--hp-list[{detail,yaml}]][--hp-serial-format{auto,yaml,pickle}][--hp-exit]optionalarguments:-h,--helpshowthishelpmessageandexit--weight-decayWEIGHT_DECAY--hp-saveHP_SAVESavehyperparameterstoafile.Thehyperparametersaresavedafterprocessingofallotheroptions--hp-loadHP_LOADLoadhyperparametersfromafile.Thehyperparametersareloadedbeforeanyotheroptionsareprocessed--hp-list[{detail,yaml}]Listallavailablehyperparameters.If`--hp-listdetail`isspecified,averbosetablewillbeprint--hp-serial-format{auto,yaml,pickle}Formatofthesavedconfigfile.Defaultstoauto.Canbesettooverrideautofiletypededuction.--hp-exitprocessallhpargparseactionsandquit(Example taken fromhpargparse)We are now bothdistributed(write anywhere) andcentralized(manage them as a whole).Our design is inspired by theunderscore
functioncommonly used ingettextin software
translation. We deem "hyperparameters" as slots of text to be translated,
while different hyperparameter values correspond to different "language" of the
same text.We achieve the above things by parsing your source code statically and extract
where and how you are defining your hyperparameters. It follows the thoughts
ofCode as Data.Also, expression evaluation in hpman is quite safe as we are usingast.literal_eval.FeaturesDesign principlesLow runtime overhead.Values of hyperparameter can be any type.Arbitrary ImportsHyperparameter managers are the most important objects of hpman. We are
usingfrom hpman.m import _throughout the tutorial, as well as recommend
using underscore ("_", courtesy ofgettext) as the name of imports in
practice, but you can actually use anything name you want.Thehpman.mmodule is configured to allow arbitrary imports. Whatever you
import will always be an object of hyperparameter manager and works the same as
"_":fromhpman.mimport_,hpm,hp,ddd,abc,hello
ddd('a',1)abc('a',2)_('hello',3)Hyperparameter managers imported by different names work independently and work
in parallel. Imports of the same name are cached in the sense that, imports of
the same name in the same process will return always the same object.There are caveats:Assignment of these imported objects to variables will not work in static
parsing (will be addressed later), but works at runtime (if you skipped
parsing stage). e.g.:# XXX: BAD EXAMPLEfromhpman.mimport_hello=_# this breaks the rulehello('a',1)# <-- hpman will not be aware of this 'a' hyperparameter.Variables share the same name withhpman.mimports will be statically
parsed by hpman, but will not work as expected at runtime. e.g.:deffunc(*args,**kargs):pass_=func_("a",1)# <-- hpman can do nothing with "_" at runtimefromhpman.mimport_print(_.parse_file(__file__).get_values())# Will output "{'a': 1}", which is a "false positive" of hyperparameter# occurrence.Define HyperparametersThe most basic (and the most frequently used) function of hpman is to define a
hyperparameter.fromhpman.mimport_deftraining_loop():# training settingsbatch_size=_('batch_size',128)# first use of `num_layer` is recommend to come with default valueprint('num_layers ={}'.format(_('num_layers',50)))# use it directly without storing the valuesif_('use_resnet',True):# second use of `num_layer` should not provide default valueforiinrange(_('num_layers')):passThere are a few caveats:Among all the occurrence of the same hyperparameter,one and only oneoccurrence should come with a default value. Nonetheless, which one has the
default value does not matter (you can surely first use, then define the
default value in later occurrence).The name of the hyperparameter must be aliteral string.The value of the hyperparameter can be an arbitrary object (variable,
lambda, string, whatever), but it is highly recommended to use onlyliteral values, which is precisely defined by whatast.literal_evalfunction accepts. It not only makes the serialization of hyperparameters in
downstream frameworks (such as hpargparse) easier but also improves the
interoperability of hyperparameter settings among different programming
languages and frameworks. The readability of dumped hyperparameters will be
better as well.Static ParsingWe employ static parsing to retrieve information on where and how you are using
the hyperparameters in your source codes. It is employed by_.parse_fileand_.parse_source._.parse_fileaccepts file paths, directory names, or a list of both. It
internally calls_.parse_source._.parse_sourceaccepts only a piece of source code string.Examples:_.parse_file(__file__)_.parse_file('main.py')_.parse_file('library_dir')_.parse_file(['main.py','library_dir'])_.parse_source('_("a", 1)')Parsing is done using theastmodule provided in the python standard library.
We match all function calls with required syntax to detect proper calls to
hyperparameter manager.Runtime Value Getter/SetterValue of a hyperparameter can be retrieved by two ways in runtime:use__call__syntax:_('varname')use dedicated function:_.get_value('varname')A dict of all hyperparameters can be retrieved by_.get_values()Setting a hyperparameter can only be done with_.set_value('varname',value)HintsHintsis intended to provide a mechanism for extending hpman.It provides an interface to store and retrieve arbitrary information provided
at hyperparameter definition.
Downstream libraries and frameworks could utilize this provided information to
better serve its purpose.For example, say we would like to create an argparse interface for setting
hyperparameters from the command line, a user could write something like_('optimizer','adam',choices=['adam','sgd'])in their codebase, and the entry point of the program, we could
retrieve this information and provide better argparse options:# File: hints_example.pyfromhpman.mimport_fromhpman.hpm_dbimportLimportargparse_('optimizer','adam',choices=['adam','sgd'])if__name__=='__main__':parser=argparse.ArgumentParser()_.parse_file(__file__)occurrences=_.db.select(lambdarow:row.name=='optimizer')oc=[ocforocinoccurrencesifoc['hints']isnotNone][0]choices=oc['hints']['choices']value=oc['value']parser.add_argument('--optimizer',default=value,choices=choices)args=parser.parse_args()print('optimizer:{}'.format(args.optimizer))usecase is as follows:$python3hints_example.py
optimizer:adam
$python3hints_example.py-h
usage:hints_example.py[-h][--optimizer{adam,sgd}]optionalarguments:-h,--helpshowthishelpmessageandexit--optimizer{adam,sgd}$python3hints_example.py--optimizersgd
optimizer:sgd
$python3hints_example.py--optimizerrmsprop
usage:hints_example.py[-h][--optimizer{adam,sgd}]hints_example.py:error:argument--optimizer:invalidchoice:'rmsprop'(choosefrom'adam','sgd')The example can be found atexamples/02-hintsNested Hyperparameters当超参数数量增多时，会带来管理压力。我们经常将超参数分为若干族，使用相同的前缀方便管理。你可以批量操作同一族的超参数。如将超参数导出成如下结构的yaml，提高了可读性。也可以直接导入树状结构的yaml。discriminator:in_channels:3spectral:truenorm:'instance'activation:'leaky_relu'residual:trueinput_size:[512,512]Notice:一个key不能同时指向一棵树和一个值，你可以通过set_value和set_tree分别指明超参数的类型是value还是tree。当你通过下划线函数定义默认值时，会被视为是value。
所以如下代码_('a',{'b':1})# 被视为name='a'的超参数，默认值为{'b': 1}。此时a是value。_('a.b')# 被视为超参树a中的b，此时a是tree。在运行时会抛出异常：KeyError:'`a.b` not found'在静态解析时会抛出异常：hpman.primitives.ImpossibleTree:node`a`hasisbothaleafandatree.缺点 and 兼容性破坏：你不能使用两个超参数，一个是另一个的前缀 (split by '.')。因为tree的name允许为空，所以你仍然可以在超参数的name中使用.，包括以.开头，以.结尾，或连续的.都是合法的。Like_(".hpman is a good...man.").Best PracticesIt is advised thatDO use hpman whenglobal hyperparameters are needed(e.g., config.{py,yml,json}). hpman can substitute a global config file theoretically.DO NOT use hpman in python libraries share among projects, unless you fully aware what the consequences are.DevelopmentInstall requirements:python3-mpipinstall-rrequirements.dev.txtActivate git commit templategit config commit.template .git-commit-template.txtInstall pre-commit hookpre-commitinstallTo format your source codemakeformatTo check the coding stylemakestyle-checkTo run the testsmaketestCAVEATThis project is still in its early stage. API may subject to radical changes
(until version 1.0.0).ContributingLicenseMIT© MEGVII Research