Package: optimyzer
OptimyzerA hyperparameter optimization framework that fits into every workflowThis project is still inbetastage. This means that interfaces might change
more often than expected. If you have any kind of issue or suggestion, please
drop us a line:info@gauss-ml.com.IntroductionWhile creating machine learning models is fun, optimizing their hyperparameters usually isn't. Many
people do this manually, which is often ironically referred to asexpert descent.Coming more from the world of numerical optimization, many available hyperparameter optimization
frameworks see the problem as a function optimization problem. However, this means that the code of
the project has to be structured in a way that the entire machine-learning pipeline (loading data,
preprocessing, training, evaluating) is one function call. Furthermore, all parameters have to be
available as function arguments. For many projects, wrapping the entire machine learning pipeline
into a function call is cumbersome and a waste of time.Optimyzerprovides a framework to make hyperparameter optimization easy. It features a very simple
interface, such that the optimization can be done with just a few lines of code, and in any existing
workflow. It logs all relevant data into a file-based structure, where each experiment gets its own
folder. At the momentOptimyzeronly uses random search because it is easy to use, highly
parallelizable and already way better thanexpert descentorgrid search.If you find a bug, or have an improvement suggestion or feature request, you can directly add it
into ourissue tracker. If you have
general remarks, feedback or other comments, please don't hesitate to contact us atinfo@gauss-ml.com.Example Use-caseLet's say you want to do handwritten digit recognition on the MNIST dataset with a simple multilayer
perceptron (MLP). Already with such a setup there are a couple of parameters to tune: The depth
(number of layers) and width (units per layer) of the MLP, as well as the learning rate of the
optimization algorithm, batch size and so on. Usually there are many more. Getting those
hyperparameters right can make the difference between mediocre and world-class performance.Sure, you canquicklytry out what happens when you change one of the parameters. And then tweak
another parameter. A couple of hours later,just one more tweak and the performance will be good
enough.We have been there. We didn't like it. WithOptimyzer, we would like to help people to
avoid wasting their time with parameter tuning.Getting StartedStarting to useOptimyzeris really simple, as you will see from theInstallationandUsagesections. Additionally, we will explain how theOptimyzer'sinformation flowdiffers from most other optimizers and howOptimyzerstores the necessary data.InstallationInstallingOptimyzeris very easy, just runpip install optimyzerin your favorite Python
environment and you're done. If you like it, you might want to addOptimyzerto your project's
dependencies.By the way,Optimyzerdoes not have any dependency, it is powered by the Python Standard Library
only. Also, there is no database, no inter-process communication, nothing to install, nothing to
configure, nothing to take care of.Optimyzer UsageOptimyzerhas been built to be included in any kind of workflow as easily as possible. This is
the minimal version:First we have to import it (of course):importoptimyzerThen we instantiate theOptimyzerobject and select a directory where it runs:oy=optimyzer.Optimyzer(".")# initializes an Optimyzer in the current workdirFor each parameter we want to tune, we have to tellOptimyzerthe type and
range. This is done by adding a parameter (IntParameter,FloatParameterorCategoricalParameter) toOptimyzer:oy.add_parameter(optimyzer.IntParameter("int_name",(minimum,maximum)))oy.add_parameter(optimyzer.FloatParameter("float_name",(minimum,maximum)))oy.add_parameter(optimyzer.CategoricalParameter("cat_name",["a","b","c"]))Note that the parameters can be freely named, except for the metadata namesworkdir,idandvalue.Once we have added the parameters, we can sample from the parameter space:config=oy.create_config()# create a configuration by random samplingThis returns a configuration that holds all your parameters as properties, e.g.,config.int_nameorconfig.float_name. For convenience, this configuration also contains theidandworkdirof
the instance. Note that you cannot add or change your parameters after creating a configuration.Now you can run the rest of your pipeline, using the sampled parameters from theconfig. After
everything is done and you have evaluated theperformanceof your model, you report it back toOptimyzer:oy.report_loss(performance)That's it already! With just a few extra lines of code, you can run your training pipeline using
hyper-parameters sampled from your search space.You can now execute the optimization by running this file a couple of times, either sequentially in
a simple loop (if you only have one GPU) or in parallel. It doesn't matter whether you're running
this on your laptop or your GPU cluster. The only thing that you need is a shared file system where
the base directory is located.After running this as many times as you wish, you can get the optimal configuration directly from a
static top-level function of the package:optimal_config=optimyzer.get_optimal_config(basedir)Alternatively, you can find the optimal parameters in a human-readable JSON file located at
(best_instance/.optimyzer/config.json), in thebasediryou gave toOptimyzer.Note that this framework is entirely transparent; there are no secret hooks for particular
libraries, nothing hidden happening. You're in control and know what is going on in your code.Parameter TypesThere are tree main types of parameters available:IntParameterfor integer-valued options, like the number of layers in a
neural network or the number of features for a spectral method.FloatParameterfor real-valued options, like the learning rate in neural
networks or length scales in kernel methods.CategoricalParameterfor choices, like the type of optimization algorithm
for a neural network or the covanriance function in Gaussian processes.Since quite often numerical parameters roughly follow Zipf's law (in the sense
that larger parameters are less likely than smaller parameters), and due to the
fact that often the relative change in parameters during optimization is much
more meaningful than the absolute change, it makes sense to define certain
parameter on a log scale. This can be done by adding the keyword argumentlogarithmic=Truewhen initilizing the parameter:optimyzer.IntParameter("int_name",(minimum,maximum),logarithmic=True)optimyzer.FloatParameter("float_name",(minimum,maximum),logarithmic=True)Using a logarithmic distribution is helpful, for instance, when optimizing
length scales, learning rates, noise levels, signal variances, lengths, masses,
etc.Illustrative Integration in ML CodeA cartoon example of a machine learning pipeline based on our imaginary neural network frameworkneuralnetworksis shown below. A full Keras tutorial is located in thenotebooksdirectory, both ascell-based scriptand asJupyter
notebook.# import stuffimportneuralnetworksasnnimportoptimyzertrain_inputs,train_targets,test_inputs,test_targets=nn.load_preprocessed_data('MNIST')# Optimyzer: initialize and configure parameterspaceoy=optimyzer.Optimyzer(".")# initializes an Optimyzer in the current workdir# int between 1 and 10 for the depth (number of layers)oy.add_parameter(optimyzer.IntParameter("depth",(1,10)))# int between 32 and 1024 for the width (nodes per layer)oy.add_parameter(optimyzer.IntParameter("width",(32,1024)))# float between 1e-3 and 1e0 for learning_rateoy.add_parameter(optimyzer.FloatParameter("learning_rate",(1e-3,1e0)))# two different neural network optimizersoy.add_parameter(optimyzer.CategoricalParameter("opt",["SGD","ADAM"]))# Optimyzer: freeze configuration and sampleconfig=oy.create_config()# we create a config by random sampling# create a modelmodel=nn.MLP(depth=config.depth,width=config.width)# use sampled values# select training algorithm based on sampled categoryifconfig.opt=="SGD":opt=nn.opt.SGD(model,learning_rate=config.learning_rate)ifconfig.opt=="ADAM":opt=nn.opt.ADAM(model,learning_rate=config.learning_rate)# train the modelmodel.train(opt,train_inputs,train_targets)# after the training: check how many predictions were correctpred=model.predict(test_inputs)correct=sum(pred==test_targets)# Optimyzer: report the lossoy.report_loss(performance)That's just four additional lines of code for usingOptimyzer, plus one line for each parameter.Directory HandlingOftentimes, machine learning pipelines write data into a working directory. For example, training
progress often is logged into atensorboarddirectory and model checkpoints are stored as well.Optimyzertherefore provides a working directory for each configuration instance, so that logging
and checkpoint saving can be done without risking to overwrite the data from other runs.Optimyzeritself stores the instance configuration and resulting performance within that directory in its.optimyzerfolder.Absolute PathsIf you already use absolute paths, great! If the path where you do your experiments is, for example,/var/tmp/ml-experiments/, you can give that path directly tooptimyzerlike this:experiment_dir="/var/tmp/ml-experiments"oy=optimyzer.Optimyzer(experiment_dir)When you create a configurationconfig=oy.create_config()thisconfigwill contain a working directoryconfig.workdirthat you can use subsequently, for
instance to save a modelmodel.save(os.path.join(config.workdir,"model.nn"))Relative Paths and Current Working DirectoryIf you just have a script running that uses the current working directory (where the script is
executed), no problem either! You can initializeOptimyzerin the current working directory
as well:oy=optimyzer.Optimyzer(".")When you create a configuration, you can pass thechdir=Trueswitch:config=oy.create_config(chdir=True)This will change the current working directory from the location of your script to the working
directory of this instance. This means that all relative file system commands are now executed
relative to the working directory. This means that you don't have to take care of the paths and can
just continue using commands that operate in the current working directory, for instancemodel.save("model.nn")Further Reading: How the Metadata is StoredWhileOptimyzerworks equally fine for algorithms that never store any data, it was designed to
work well with pipelines that need some kind of working directory to store results or configuration
or logging information. That's why there is one directory created for each instance. The instance
name is based on the configuration, which is hashed to generate a unique name.This means that after executing a couple of runs, yourexperimentsdirectory may look like thisexperiments
|- 5c49826a83751767729e
|- 71ae3a23d782a0465750
|- 7633242a2c695641532b
|- d0b85d53f19420d1a7a1
|- dee9fecd1f0b91346bc4
|- best_instance -> 71ae3a23d782a0465750
|- ...where each directory represents one experiment instance, one configuration that has been evaluated.
The directorybest_instanceis a symbolic link that points to the directory of the best
configuration seen so far.Each of the experiment instances contains a metadata folder.optimyzer, which holds the
configuration and the value of the experiment in a JSON file eachexperiments
|- 5c49826a83751767729e
   |- .optimyzer
       |- config.json
       |- value.json
|- 71ae3a23d782a0465750
   |- .optimyzer
       |- config.json
       |- value.json
|- ...