Package: hyanova
# HyANOVAHyANOVA is a pure python implementation of fuctional ANOVA algorithm,
which can be used to analyze the importance of hyperparameters in
machine learning algorithm.Quick StartTo install the package, please use thepipinstallation as follows:pipinstallhyanovaHere is a short example of usage. You can download thedatafrom the example
folder.importhyanovapath='./iris[GridSearchCV]Model1.csv'# gridsearch results generated by sklearnmetric='mean_test_score'# metric for model performancedf,params=hyanova.read_csv(path,metric)# df,params = hyanova.read_df(df,metric)              You can also load data from pd.DataFrameimportance=hyanova.analyze(df)Themetricis the feature you choose to evaluate the model
performance, it must appears in the.csvfile or thepandas.DataFrameobject’s column. And the result you got will be
similar to this below, see the next section(ANOVA) for more details.print(importance)>>>uv_uF_u(v_u/v_all)0(alpha,)0.0568850.8920571(l1_ratio,)0.0024890.0390302(alpha,l1_ratio)0.0043940.068912APIsLoad DataHyANOVA is designed to analyze the grid search results generated by
sklearn. It provides two ways to load the data.read_df(df,metric)You can useread_df(df,metric)to load data from a<class 'pandas.core.frame.DataFrame'>object.Parameters:df:<class ‘pandas.core.frame.DataFrame’>, theDataFrameyou want to analyze.metric:string, the metric you choose.Returns:result_df:<class ‘pandas.core.frame.DataFrame’>,aDataFramewith all hyperparameters’ value and the value of
metric you chooseparams_list:list, alistof all hyperparameters’ name.read_csv(path,metric)Usehyanova.read_csv(path,metric)to load data from.csvfile.
It is equivalent tohyanova.read_df(pandas.read_csv(path),metric).Parameters:path:string, path of theDataFrameyou want to analyze.metric:string, the metric you choose.Returns:result_df:<class ‘pandas.core.frame.DataFrame’>,aDataFramewith all hyperparameters’ value and the value of
metric you chooseparams_list:list, alistof all hyperparameters’ name.ExampleThetemplate
filecan be find at the example folder. Here is an example.print(df.head)>>>mean_fit_timestd_fit_timemean_score_timestd_score_timeparam_alpha\00.0038990.0001940.0485130.0076210.00097710.0034010.0005840.0424540.0112950.00097720.0027060.0005020.0485440.0090590.00097730.0033040.0005310.0407090.0030310.00097740.0018010.0001160.0002890.0000140.000977param_l1_ratioparams\00.00{'alpha':0.0009765625,'l1_ratio':0.0}10.25{'alpha':0.0009765625,'l1_ratio':0.25}20.50{'alpha':0.0009765625,'l1_ratio':0.5}30.75{'alpha':0.0009765625,'l1_ratio':0.75}41.00{'alpha':0.0009765625,'l1_ratio':1.0}split0_test_scoresplit1_test_scoresplit2_test_scoremean_test_score\00.8285710.9714290.9714290.92381010.8857140.9714290.9428570.93333320.8857141.0000000.9428570.94285730.8857140.9142860.9142860.90476240.8857141.0000000.9428570.942857std_test_scorerank_test_score00.067344410.035635320.046657130.013469540.0466571df,params=hyanova.read_df(df,'mean_test_score')print(df.head)>>>alphal1_ratiomean_test_score00.0009770.000.92381010.0009770.250.93333320.0009770.500.94285730.0009770.750.90476240.0009771.000.942857print(params)>>>['alpha','l1_ratio']ANOVAanalyze(df,max_iter=-1)Usehyanova.analyze(df,max_iter=-1)to do the functional ANOVA
decomposition.Parameters:df:<class ‘pandas.core.frame.DataFrame’>, theDataFrameyou want to analyze.max_iter:int, default to -1.Returns:result_df:<class ‘pandas.core.frame.DataFrame’>Thedfparameter needs apnadas.DataFrameobject which has a
format similar to the following table. You can use the methods HyANOVA
provides to load data easily.alphal1_ratiomeantestscore00.009770.000.92381010.009770.250.93333320.009770.500.94285730.009770.750.904762Note:The metric(meantestscore) should always be in the last
column.ExampleThehyanova.analyze(df)will return aDataFramewith
hyperparameters’ name, variance(vu) and the importance(Fu).importance=hyanova.analyze(df)>>>100%|██████████████████████████████████|3/3[00:00<00:00,11.32it/s]print(importance)>>>uv_uF_u(v_u/v_all)0(alpha,)0.0568850.8920571(l1_ratio,)0.0024890.0390302(alpha,l1_ratio)0.0043940.068912Note:The Fu is the ratio of the variance caused by the
hyperparameter itself(vu) to the variance of all trials(vall), so
all Fu sums always equal to 1.See references for more details.Due to the performance limitations of Python, the functional ANOVA will
be very slow when the number of hyperparameters is high (more than 5).
You can end the analysis early by setting themax_iterparameter. In
fact, we usually only need the univariate importance, so set themax_iterparameter to equal the number of features for shorter
runtime.importance=hyanova.analyze(df,max_iter=2)>>>100%|██████████████████████████████████|2/2[00:00<00:00,8.12it/s]print(importance)>>>uv_uF_u(v_u/v_all)0(alpha,)0.0568850.8920571(l1_ratio,)0.0024890.039030Example usageYou can use sklearn to do hyperparameters search and then use hyanova to
analyze the importance of hyperparameters.importsklearn.datasetsfromsklearn.model_selectionimportGridSearchCVfromsklearn.svmimportSVCimportpandasaspdimporthyanovairis=sklearn.datasets.load_iris()X=iris.datay=iris.targetmodel=SVC()grid={'C':np.linspace(1e-9,128,10000)'kernel':('rbf','linear','poly','sigmoid')}grid_search=GridSearchCV(model,grid)result=grid_search.fit(X,y)df=pd.DataFrame(result.cv_results_)metric='mean_test_score'df,params=hyanova.read_df(df,metric)importance=hyanova.analyze(df)DependenciesnumpypandastqdmWhy created HyANOVA?I am completing my undergraduate thesis. In order to better understand
the models used in my article, I looked for a lot of algorithms that can
measure the importance of hyperparameters. Among them, functional ANOVA
seems to be the most effective. But the original author’s implementation
is based on java and uses python to call java files, which confuses me.
I hope there is a module that is easier to understand and implemented
completely based on python, which can help me with ANOVA decomposition,
so I created HyANOVA. Hope that will help you too!ReferencesHutter, F., Hoos, H. & Leyton-Brown, K.. (2014). An Efficient
Approach for Assessing Hyperparameter Importance. Proceedings of the
31st International Conference on Machine Learning, in PMLR
32(1):754-762https://github.com/frank-hutter/fanova