Package: exp
Experimentdesign,deployment, andoptimizationEXP is a python experiment management toolset created to simplify two simple use cases: design and deploy experiments in the form of python modules/files.An experiment is a series of runs of a given configurable module for a specified set of parameters. This tool covers one of the most prevalent experiment deployment scenarios: testing a set of parameters in parallel in a local machine or homogeneous cluster. EXP also supportsglobal optimizationusinggaussian processesor other surrogate models such asrandom forests. This can be used for instance as a tool forhyperoparameter tuningfor machine learning models.Featuresparameter space designbased on configuration files (TOMLformat);parallel experiment deploymentusingmultiprocessingprocesses;CUDA gpu workersone parallel process per available GPUs: uses the variableCUDA_VISIBLE_DEVICES;global optimizationfrom parameter spaces (e.g. for hyperparameter tunning) usingscikit-optimize.Installationpip install exppipenv install expwithpipenvAvailable CLI toolsEXP provides two CLI modules:exp.run:python -m exp.run -p basic.conf -m runnable.py --workers 10exp.gopt:python -m exp.gopt -p basic.conf -m runnable.py --workers 4 -n 100 --plotfor more information check each commands help:python -m exp.run -hGetting Started: Optimization1. Runnable ModuleThe first step is to create a module to use in our experiments. A basic configurable modulerunnable.pylooks like this:defrun(x=1,**kwargs):returnx**2The module simply computes the square of a parameterx. Note thatkwargsis included in case the experiment runner dispatcher sends other parameters (that might not be used by your module). Since run receives a dictionary, you could also define it as follows.defrun(**kwargs):x=kwargs.get('x',1)returnx**22. Parameter Space DefinitionNext, we need a configuration filebasic.confwere the parameters are specified:[x]
type = "range"
bounds = [-10,10]This defines a parameter space with a single parameterxwith values in the range[-10,10]. For how to specify parameter spaces, see the Parameter Space Specification.3. Module OptimizationOur simple module returns thex**2, the optimizer tries to find the minimum value of this function based on the parameter space given by the configuration file. In this case, the optimizer will look at values ofxbetween[-10,10]and try to find the minimum value.python-mexp.gopt--paramsbasic.conf--modulerunnable.py--n20--workers4finds a solution very close to0. By default, the optimizer assumes a range defines the boundaries of a real-valued variable. If you wish to optimize discrete integers use the following specification:[x]
type = "range"
bounds = [-10,10]
dtype = "int"The optimizer will explore discrete values between -10 and 10 inclusively. Also, using the--plotflag displays a real-timeconvergence plotfor the optimization process.which in this case converges immediately because the function to be optimized is quite simple, but the goal is to optimize complex models and choosing from a large set of parameters without having to run an exhaustive search through all the possible parameter combinations.Parameter Space SpecificationParameter space files useTOMLformat, I recommend taking a look at the specification and getting familiar with how to define values, arrays, etc. If a parameter space the file cannot be parsed an error will be thrown. ParamSpaces in EXP has4 types of parametes, namely:value: single value parameter;range: a range of numbers between bounds;random: a randomreal/intvalue between bounds;list: a list of values (used for example to specify categorical parameters);Bellow, I supply an example for each type of parameter:ValueSingle value parameter.# this a single valued parameter with a boolean value[some_param]type="value"value=trueRangeA parameter with a set of values within a range.# TOML files can handle comments which is useful to document experiment configurations[some_range_param]type="range"bounds=[-10,10]step=1# this is optional and assumed to be 1dtype="float"# also optional and assumed to be floatThe commandsrunandgoptwill treat this parameter definition differently. The optimizer will explore values within the bounds including the end-points. The runner will take values betweenbounds[0]andbounds[1]excluding the last end-point (much like a python range or numpy arange).Thedtypealso influences how the optimizer looks for values in the range, if set to"int", it explores discrete integer values within the bounds; if set to"float", it assumes the parameter takes a continuous value between the specified bounds.RandomA parameter withnrandom values sampled from "uniform" or "log-uniform" between the given bounds. If used withrun, a parameter space will be populated with a list of random values according to the specification. If used withgopt,nis ignored and bounds are used instead, along with the prior.For optimization purposes, this works like range, except that you can specify the prior which can be "uniform" or "log-uniform", range assumes that the values are generated from "uniform" prior, when the parameter is used for optimization.The other difference between parameter grids and optimization is that the bounds do not include the end-points when generating parameter values for grid search. The optimizer will explore random values within the bounds specified, including the high end-point.[random_param]type="random"bounds=[0,3]# optional, default range is [0,1]prior="uniform"# optional, default value is "uniform"dtype="float"# optional, default value is "float"n=1# optional, default value is 1 (number of random parameters to be sampled)ListA list is just an homogeneous series of values a parameter can take.[another_param]type="list"value=[1,2,3]The array in"value"must be homogenous, something likevalue=[1,2,"A"]would throw aNot a homogeneous arrayerror. List parameters are treated bygoptcommand as acategoricalparameter. This is encoded using aone-hot-encodingfor optimization.Also, for optimization purposes, a list is treated like a set, if you provide duplicate values it will only explore the unique values. For example if you want to specify a boolean parameter, use a list:[some_boolean_decision]type="list"value=[true,false]Library ModulesEXP also provides different tools to specify param spaces programmaticallyParamSpaceTheexp.params.ParamSpaceclass provides a way to create parameter spaces and iterate over all the possible
combinations of parameters as follows:>>>fromexp.paramsimportParamSpace>>>ps=ParamSpace()>>>ps.add_value("p1",1)>>>ps.add_list("p2",[True,False])>>>ps.add_range("p3",low=0,high=10,dtype=int)>>>ps.size20grid=ps.param_grid(runs=2)gridhas2*ps.sizeconfigurations because we repeat each configuration2times (number of runs). Each configuration dictionary includes 2 additional parameters"id"and"run"which are the unique configuration id and run id respectively.forconfigingrid:# config is a dictionary with the params of a unique configuration in the parameter spacedo_something(config)ParamDict & NamespaceParamDictfromexp.argsmodule is a very simple dictionary where you can specify default values for different parameters.exp.args.Paramis a named tuple:(typefn,default,options)wheretypefnis a type function likeintorfloatthat transforms strings into values of the given type if necessary,defaultis a default value,optionsis a list of possible values for the parameter.This is just a very simple alternative to using argparse with a lot of of parameters. Example of usage:fromexp.argsimportParamDict,Namespace# these are interpreted by a ParamDict as a exp.args.Param named tupleparam_spec={'x':(float,None),'id':(int,0),'run':(int,1),'cat':(str,"A",["A","B"])}defrun(**kargs):args=ParamDict(param_spec)# creates a param dict from default values and optionsargs.from_dict(kargs)# updates the dictionary with new values where the parameter name overlapsns=args.to_namespace()# creates a namespace object so you can access ns.x, ns.run etc...Another nice thing is that there is basic type conversions from string to boolean, int, float, etc. Depending
on the arguments received inkwargs,ParamDictconverts the values automatically according to the parameter
specifications.Created byDavide NunesLicenceApache License 2.0