Package: ultraopt
UltraOpt:Distributed Asynchronous Hyperparameter Optimization better than HyperOpt.UltraOptis a simple and efficient library to minimize expensive and noisy black-box functions, it can be used in many fields, such as HyperParameter Optimization(HPO) and
Automatic Machine Learning(AutoML).After absorbing the advantages of existing optimization libraries such asHyperOpt[5],SMAC3[3],scikit-optimize[4]andHpBandSter[2], we developUltraOpt, which implement a new bayesian optimization algorithm : Embedding-Tree-Parzen-Estimator(ETPE), which is better than HyperOpt' TPE algorithm in our experiments.
Besides, The optimizer ofUltraOptis redesigned to adaptHyperBand & SuccessiveHalving Evaluation Strategies[6][7]andMapReduce & Async Communication Conditions.
Finally, you can visualizeConfig Spaceandoptimization process & resultsbyUltraOpt's tool function. Enjoy it !Other Language:中文READMEDocumentationEnglish Documentation is not available now.中文文档TutorialsEnglish Tutorials is not available now.中文教程Table of ContentsInstallationQuick StartUsing UltraOpt in HPOUsing UltraOpt in AutoMLOur AdvantagesAdvantage One: ETPE optimizer is more competitiveAdvantage Two: UltraOpt is more adaptable to distributed computingAdvantage Three: UltraOpt is more function comlete and user friendlyCitationReferanceInstallationUltraOpt requires Python 3.6 or higher.You can install the latest release bypip:pipinstallultraoptYou can download the repository and manual installation:gitclonehttps://github.com/auto-flow/ultraopt.git&&cdultraopt
pythonsetup.pyinstallQuick StartUsing UltraOpt in HPOLet's learn whatUltraOptdoing with several examples (you can try it on yourJupyter Notebook).You can learn Basic-Tutorial inhere, andHDL's Definition inhere.Before starting a black box optimization task, you need to provide two things:parameter domain, or theConfig Spaceobjective function, acceptconfig(configis sampled fromConfig Space), returnlossLet's define a Random Forest's HPOConfig SpacebyUltraOpt'sHDL(Hyperparameter Description Language):HDL={"n_estimators":{"_type":"int_quniform","_value":[10,200,10],"_default":100},"criterion":{"_type":"choice","_value":["gini","entropy"],"_default":"gini"},"max_features":{"_type":"choice","_value":["sqrt","log2"],"_default":"sqrt"},"min_samples_split":{"_type":"int_uniform","_value":[2,20],"_default":2},"min_samples_leaf":{"_type":"int_uniform","_value":[1,20],"_default":1},"bootstrap":{"_type":"choice","_value":[True,False],"_default":True},"random_state":42}And then define an objective function:fromsklearn.ensembleimportRandomForestClassifierfromsklearn.datasetsimportload_digitsfromsklearn.model_selectionimportcross_val_score,StratifiedKFoldfromultraopt.hdlimportlayering_configX,y=load_digits(return_X_y=True)cv=StratifiedKFold(5,True,0)defevaluate(config:dict)->float:model=RandomForestClassifier(**layering_config(config))return1-float(cross_val_score(model,X,y,cv=cv).mean())Now, we can start an optimization process:fromultraoptimportfminresult=fmin(eval_func=evaluate,config_space=HDL,optimizer="ETPE",n_iterations=30)result100%|██████████| 30/30 [00:36<00:00,  1.23s/trial, best loss: 0.023]

+-----------------------------------+
| HyperParameters   | Optimal Value |
+-------------------+---------------+
| bootstrap         | True:bool     |
| criterion         | gini          |
| max_features      | log2          |
| min_samples_leaf  | 1             |
| min_samples_split | 2             |
| n_estimators      | 200           |
+-------------------+---------------+
| Optimal Loss      | 0.0228        |
+-------------------+---------------+
| Num Configs       | 30            |
+-------------------+---------------+Finally, make a simple visualizaiton:result.plot_convergence()You can visualize high dimensional interaction by facebook's hiplot:!pipinstallhiplotresult.plot_hi(target_name="accuracy",loss2target_func=lambdax:1-x)Using UltraOpt in AutoMLLet's try a more complex example: solve AutoML'sCASH Problem[1](Combination problem of Algorithm Selection and Hyperparameter optimization)
by BOHB algorithm[2](CombineHyperBand[6]Evaluation Strategies withUltraOpt'sETPEoptimizer) .You can learn Conditional Parameter and complexHDL's Definition inhere,  AutoML implementation tutorial inhereand Multi-Fidelity Optimization inhere.First of all, let's define aCASHHDL:HDL={'classifier(choice)':{"RandomForestClassifier":{"n_estimators":{"_type":"int_quniform","_value":[10,200,10],"_default":100},"criterion":{"_type":"choice","_value":["gini","entropy"],"_default":"gini"},"max_features":{"_type":"choice","_value":["sqrt","log2"],"_default":"sqrt"},"min_samples_split":{"_type":"int_uniform","_value":[2,20],"_default":2},"min_samples_leaf":{"_type":"int_uniform","_value":[1,20],"_default":1},"bootstrap":{"_type":"choice","_value":[True,False],"_default":True},"random_state":42},"KNeighborsClassifier":{"n_neighbors":{"_type":"int_loguniform","_value":[1,100],"_default":3},"weights":{"_type":"choice","_value":["uniform","distance"],"_default":"uniform"},"p":{"_type":"choice","_value":[1,2],"_default":2},},}}And then, define a objective function with an additional parameterbudgetto adapt toHyperBand[6]evaluation strategy:fromsklearn.neighborsimportKNeighborsClassifierimportnumpyasnpdefevaluate(config:dict,budget:float)->float:layered_dict=layering_config(config)AS_HP=layered_dict['classifier'].copy()AS,HP=AS_HP.popitem()ML_model=eval(AS)(**HP)scores=[]fori,(train_ix,valid_ix)inenumerate(cv.split(X,y)):rng=np.random.RandomState(i)size=int(train_ix.size*budget)train_ix=rng.choice(train_ix,size,replace=False)X_train,y_train=X[train_ix,:],y[train_ix]X_valid,y_valid=X[valid_ix,:],y[valid_ix]ML_model.fit(X_train,y_train)scores.append(ML_model.score(X_valid,y_valid))score=np.mean(scores)return1-scoreYou should instance amulti_fidelity_iter_generatorobject for the purpose of usingHyperBand[6]Evaluation Strategy :fromultraopt.multi_fidelityimportHyperBandIterGeneratorhb=HyperBandIterGenerator(min_budget=1/4,max_budget=1,eta=2)hb.get_table()iter 0iter 1iter 2stage 0stage 1stage 2stage 0stage 1stage 0num_config421213budget1/41/211/211let's combineHyperBandEvaluation Strategies withUltraOpt'sETPEoptimizer , and then start an optimization process:result=fmin(eval_func=evaluate,config_space=HDL,optimizer="ETPE",# using bayesian optimizer: ETPEmulti_fidelity_iter_generator=hb,# using HyperBandn_jobs=3,# 3 threadsn_iterations=20)result100%|██████████| 88/88 [00:11<00:00,  7.48trial/s, max budget: 1.0, best loss: 0.012]

+--------------------------------------------------------------------------------------------------------------------------+
| HyperParameters                                     | Optimal Value                                                      |
+-----------------------------------------------------+----------------------+----------------------+----------------------+
| classifier:__choice__                               | KNeighborsClassifier | KNeighborsClassifier | KNeighborsClassifier |
| classifier:KNeighborsClassifier:n_neighbors         | 4                    | 1                    | 3                    |
| classifier:KNeighborsClassifier:p                   | 2:int                | 2:int                | 2:int                |
| classifier:KNeighborsClassifier:weights             | distance             | uniform              | uniform              |
| classifier:RandomForestClassifier:bootstrap         | -                    | -                    | -                    |
| classifier:RandomForestClassifier:criterion         | -                    | -                    | -                    |
| classifier:RandomForestClassifier:max_features      | -                    | -                    | -                    |
| classifier:RandomForestClassifier:min_samples_leaf  | -                    | -                    | -                    |
| classifier:RandomForestClassifier:min_samples_split | -                    | -                    | -                    |
| classifier:RandomForestClassifier:n_estimators      | -                    | -                    | -                    |
| classifier:RandomForestClassifier:random_state      | -                    | -                    | -                    |
+-----------------------------------------------------+----------------------+----------------------+----------------------+
| Budgets                                             | 1/4                  | 1/2                  | 1 (max)              |
+-----------------------------------------------------+----------------------+----------------------+----------------------+
| Optimal Loss                                        | 0.0328               | 0.0178               | 0.0122               |
+-----------------------------------------------------+----------------------+----------------------+----------------------+
| Num Configs                                         | 28                   | 28                   | 32                   |
+-----------------------------------------------------+----------------------+----------------------+----------------------+You can visualize optimization process inmulti-fidelityscenarios:importpylabaspltplt.rcParams['figure.figsize']=(16,12)plt.subplot(2,2,1)result.plot_convergence_over_time();plt.subplot(2,2,2)result.plot_concurrent_over_time(num_points=200);plt.subplot(2,2,3)result.plot_finished_over_time();plt.subplot(2,2,4)result.plot_correlation_across_budgets();Our AdvantagesAdvantage One: ETPE optimizer is more competitiveWe implement 4 kinds of optimizers(listed in the table below), andETPEoptimizer is our original creation, which is proved to be better than otherTPE based optimizerssuch asHyperOpt's TPEandHpBandSter's BOHBin our experiments.Our experimental code is public available inhere, experimental documentation can be found inhere.OptimizerDescriptionETPEEmbedding-Tree-Parzen-Estimator, is our original creation,  converting high-cardinality categorical variables to low-dimension continuous variables based on TPE algorithm, and some other aspects have also been improved, is proved to be better thanHyperOpt's TPEin our experiments.ForestBayesian Optimization based on Random Forest. Surrogate model importscikit-optimize'sskopt.learning.forestmodel, and integrate Local Search methods inSMAC3GBRTBayesian Optimization based on Gradient Boosting Resgression Tree. Surrogate model importscikit-optimize'sskopt.learning.gbrtmodel.RandomRandom Search for baseline or dummy model.Key result figure in experiment (you can see details inexperimental documentation) :Advantage Two: UltraOpt is more adaptable to distributed computingYou can see this section in the documentation:Asynchronous Communication Parallel StrategyMapReduce Parallel StrategyAdvantage Three: UltraOpt is more function comlete and user friendlyUltraOpt is more function comlete and  user friendly than other optimize library:UltraOptHyperOptScikit-OptimizeSMAC3HpBandSterSimple Usage likefminfunction✓✓✓✓×SimpleConfig SpaceDefinition✓✓✓××Support ConditionalConfig Space✓✓×✓✓Support SerializableConfig Space✓××××Support VisualizingConfig Space✓✓×××Can Analyse Optimization Process & Result✓×✓×✓Distributed in Cluster✓✓××✓Support HyperBand[6]& SuccessiveHalving[7]✓××✓✓Citation@misc{Tang_UltraOpt,
    author       = {Qichun Tang},
    title        = {UltraOpt : Distributed Asynchronous Hyperparameter Optimization better than HyperOpt}
    month        = January,
    year         = 2021,
    doi          = {10.5281/zenodo.4430148},
    version      = {v0.1.0},
    publisher    = {Zenodo},
    url          = {https://doi.org/10.5281/zenodo.4430148}
}Reference[1]Thornton, Chris et al. “Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms.” Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (2013): n. pag.[2]Falkner, Stefan et al. “BOHB: Robust and Efficient Hyperparameter Optimization at Scale.” ICML (2018).[3]Hutter F., Hoos H.H., Leyton-Brown K. (2011) Sequential Model-Based Optimization for General Algorithm Configuration. In: Coello C.A.C. (eds) Learning and Intelligent Optimization. LION 2011. Lecture Notes in Computer Science, vol 6683. Springer, Berlin, Heidelberg.[4]https://github.com/scikit-optimize/scikit-optimize[5]James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS'11). Curran Associates Inc., Red Hook, NY, USA, 2546–2554.[6]Li, L. et al. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.” J. Mach. Learn. Res. 18 (2017): 185:1-185:52.[7]Jamieson, K. and Ameet Talwalkar. “Non-stochastic Best Arm Identification and Hyperparameter Optimization.” AISTATS (2016).