Package: hpo4dl
Hyperparameter Optimization for Deep Learning (HPO4DL)HPO4DL is a framework for multi-fidelity (gray-box) hyperparameter optimization.
The core optimizer in HPO4DL is DyHPO, a novel Bayesian Optimization approach to
hyperparameter optimization tailored for deep learning. DyHPO dynamically determines
the best hyperparameter configurations to train further by using a deep kernel for
Gaussian Processes that captures the details of the learning curve and an acquisition function
that incorporates multi-budget information.InstallationTo install the package:pipinstallhpo4dlGetting startedFollowing is a simple example to get you started:fromtypingimportList,Dict,Unionfromhpo4dl.tunerimportTunerfromConfigSpaceimportConfigurationSpacedefobjective_function(configuration:Dict,epoch:int,previous_epoch:int,checkpoint_path:str)->Union[Dict,List[Dict]]:x=configuration["x"]evaluated_info=[{'epoch':i,'metric':(x-2)**2}foriinrange(previous_epoch+1,epoch+1)]returnevaluated_infoconfigspace=ConfigurationSpace({"x":(-5.0,10.0)})tuner=Tuner(objective_function=objective_function,configuration_space=configspace,minimize=True,max_budget=1000,optimizer='dyhpo',seed=0,max_epochs=27,num_configurations=1000,output_path='hpo4dl_results',)incumbent=tuner.run()Key Parameters Explained:objective_function: The function you aim to optimize.configuration_space: The hyperparameter configuration space over which the optimization is performed.minimize: Boolean flag indicates whether the objective function should be minimized (True) or maximized (False).max_budget: The cumulative number of epochs the tuner will evaluate. This budget gets distributed across various
hyperparameter configurations.optimizer: Specifies the optimization technique employed.seed: Random seed for reproducibility.max_epochs: Maximum number of epochs a single configuration is evaluated.num_configurations: Determines the number of configurations DyHPO reviews before selecting the next one for
evaluation. Essentially, it guides the balance between exploration and exploitation in the optimization process.output_path: Designates the location to save the results and the checkpoint for the best hyperparameter
optimization.Objective functiondefobjective_function(configuration:Dict,epoch:int,previous_epoch:int,checkpoint_path:str)->Union[Dict,List[Dict]]The objective function is tailored to support interrupted and resumed training processes.
Specifically, it should continue training from aprevious_epochto the designatedepoch.The function should return a dictionary or a list of dictionaries upon completion.
Every dictionary must include theepochandmetrickeys. Here's a sample return value:{
    “epoch”: 5,
    “metric”: 0.76
}For optimal performance with DyHPO, ensure the metric is normalized.Lastly, thecheckpoint_pathis allocated for saving any intermediate files produced
during training pertinent to the current configuration. It facilitates storing models, logs,
and other relevant data, ensuring that training can resume seamlessly.Detailed ExamplesFor a detailed exploration of the HPO4DL framework, we've provided an in-depth example
under:examples/timm_main.pyTo execute the provided example, use the following command:pythonexamples/timm_main.py--datasettorch/cifar100--train-splittrain--val-splitvalidation--optimizerdyhpo--output-dir./hpo4dl_resultsCitation@inproceedings{
wistuba2022supervising,
title={Supervising the Multi-Fidelity Race of Hyperparameter Configurations},
author={Martin Wistuba and Arlind Kadra and Josif Grabocka},
booktitle={Thirty-Sixth Conference on Neural Information Processing Systems},
year={2022},
url={https://openreview.net/forum?id=0Fe7bAWmJr}
}