Package: mle-hyperopt
Lightweight Hyperparameter Optimization 🚂Themle-hyperoptpackage provides a simple and intuitive API for hyperparameter optimization of your Machine Learning Experiment (MLE) pipeline. It supports real, integer & categorical search variables and single- or multi-objective optimization.Core features include the following:API Simplicity:strategy.ask(),strategy.tell()interface & space definition.Strategy Diversity: Grid, random, coordinate search, SMBO & wrapping FAIR'snevergrad, Successive Halving, Hyperband, Population-Based Training.Search Space Refinementbased on the top performing configs viastrategy.refine(top_k=10).Export of configurationsto execute via e.g.python train.py --config_fname config.yaml.Storage & reload search logsviastrategy.save(<log_fname>),strategy.load(<log_fname>).For a quickstart check out thenotebook blog📖.The API 🎮frommle_hyperoptimportRandomSearch# Instantiate random search classstrategy=RandomSearch(real={"lrate":{"begin":0.1,"end":0.5,"prior":"log-uniform"}},integer={"batch_size":{"begin":32,"end":128,"prior":"uniform"}},categorical={"arch":["mlp","cnn"]})# Simple ask - eval - tell APIconfigs=strategy.ask(5)values=[train_network(**c)forcinconfigs]strategy.tell(configs,values)Implemented Search Types 	🔭Search TypeDescriptionsearch_configGridSearchSearch over list of discrete values-RandomSearchRandom search over variable rangesrefine_after,refine_top_kCoordinateSearchCoordinate-wise optimization with fixed defaultsorder,defaultsSMBOSearchSequential model-based optimization(Hutter et al., 2011)base_estimator,acq_function,n_initial_pointsNevergradSearchMulti-objectivenevergradwrapperoptimizer,budget_size,num_workersHalvingSearchSuccessive Halving(Karmin et al., 2013)min_budget,num_arms,halving_coeffHyperbandSearchHyperband(Li et al., 2018)max_resource,etaPBTSearchPopulation-Based Training(Jaderberg et al., 2017)explore,exploitVariable Types & Hyperparameter Spaces 🌍VariableTypeSpace SpecificationrealReal-valuedDict:begin,end,prior/bins(grid)integerInteger-valuedDict:begin,end,prior/bins(grid)categoricalCategoricalList: Values to search overInstallation ⏳A PyPI installation is available via:pip install mle-hyperoptIf you want to get the most recent commit, please install directly from the repository:pip install git+https://github.com/mle-infrastructure/mle-hyperopt.git@mainSearch Method Highlights 🔎Grid Search 🟥strategy=GridSearch(real={"lrate":{"begin":0.1,"end":0.5,"bins":5}},integer={"batch_size":{"begin":1,"end":5,"bins":1}},categorical={"arch":["mlp","cnn"]},fixed_params={"momentum":0.9})# Add fixed param setting to each configconfigs=strategy.ask()Hyperband 🎸strategy=HyperbandSearch(real={"lrate":{"begin":0.1,"end":0.5,"prior":"uniform"}},integer={"batch_size":{"begin":1,"end":5,"prior":"log-uniform"}},categorical={"arch":["mlp","cnn"]},search_config={"max_resource":81,"eta":3},seed_id=42,# Fix randomness for reproducibilityverbose=True)configs=strategy.ask()Population-Based Training 🦎strategy=PBTSearch(real={"lrate":{"begin":0.1,"end":0.5,"prior":"uniform"}}search_config={"exploit":{"strategy":"truncation","selection_percent":0.2},"explore":{"strategy":"perturbation","perturb_coeffs":[0.8,1.2]},"steps_until_ready":4,"num_workers":10,},maximize_objective=True# Max score instead of min)configs=strategy.ask()Further Options 🚴Saving & Reloading Logs 🏪# Storing & reloading of results from .json/.yaml/.pklstrategy.save("search_log.json")strategy=RandomSearch(...,reload_path="search_log.json")# Or manually add info after class instantiationstrategy=RandomSearch(...)strategy.load("search_log.json")Search Decorator 🧶frommle_hyperoptimporthyperopt@hyperopt(strategy_type="Grid",num_search_iters=25,real={"x":{"begin":0.,"end":0.5,"bins":5},"y":{"begin":0,"end":0.5,"bins":5}})defcircle(config):distance=abs((config["x"]**2+config["y"]**2))returndistancestrategy=circle()Storing Configuration Files 📑# Store 2 proposed configurations - eval_0.yaml, eval_1.yamlstrategy.ask(2,store=True)# Store with explicit configuration filenames - conf_0.yaml, conf_1.yamlstrategy.ask(2,store=True,config_fnames=["conf_0.yaml","conf_1.yaml"])Storing Checkpoint Paths 🛥️# Ask for 5 configurations to evaluate and get their scoresconfigs=strategy.ask(5)values=...# Get list of checkpoint paths corresponding to config runsckpts=[f"ckpt_{i}.pt"foriinrange(len(configs))]# `tell` parameter configs, eval scores & ckpt paths# Required for Halving, Hyperband and PBTstrategy.tell(configs,scores,ckpts)Retrieving Top Performers & Visualizing Results 📉# Get the top k best performing configurationsid,configs,values=strategy.get_best(top_k=4)# Plot timeseries of best performing score over search iterationsstrategy.plot_best()# Print out ranking of best performersstrategy.print_ranking(top_k=3)Refining the Search Space of Your Strategy 🪓# Refine the search space after 5 & 10 iterations based on top 2 configurationsstrategy=RandomSearch(real={"lrate":{"begin":0.1,"end":0.5,"prior":"log-uniform"}},integer={"batch_size":{"begin":1,"end":5,"prior":"uniform"}},categorical={"arch":["mlp","cnn"]},search_config={"refine_after":[5,10],"refine_top_k":2})# Or do so manually using `refine` methodstrategy.tell(...)strategy.refine(top_k=2)Note that the search space refinement is only implemented for random, SMBO andnevergrad-based search strategies.Simple Command Line interface ⌨️You can also directly launch a search for your applications. This requires a couple of things: A python script<script>.pycontaining a functionmain(config), which runs your simulation for a given configuration dictionary. It should return a single scalar performance score, which will be logged.defmain(config):...returnscoreFurthermore, you will need a search configuration<search>.yamlfile and can add default fixed parameter settings in<base>.yaml.mle-search <script>.py -base <base>.yaml -search <search>.yaml -iters <search_iters>Have a look at theexample, which can be executed viamle-search run_mle_search.py -search search.yaml -base base.yaml. You can reload a previous search log by adding the option-reload.Citing the MLE-Infrastructure ✏️If you usemle-hyperoptin your research, please cite it as follows:@software{mle_infrastructure2021github,
  author = {Robert Tjarko Lange},
  title = {{MLE-Infrastructure}: A Set of Lightweight Tools for Distributed Machine Learning Experimentation},
  url = {http://github.com/mle-infrastructure},
  year = {2021},
}Development 👷You can run the test suite viapython -m pytest -vv tests/. If you find a bug or are missing your favourite feature, feel free to create an issue and/or startcontributing🤗.