Package: scikit-optimize-adapter
Scikit-Optimize-Adapter (Adapter: “A DAsk Parallel TunER”) is an efficient light weight library built on top of Scikit-Optimize and Dask that lets the user do Bayesian optimization hyperparameter tuning with different schemes of parallelized cross-validations.Installpip install --index-url https://test.pypi.org/simple/ --no-deps scikit-optimize-adapter --upgradeGetting startedLet’s start with the below dummy training data:importpandasaspdimportnumpyasnpdata=np.arange(30*4).reshape(30,4)df=pd.DataFrame(data=data,columns=['target','f1','f2','f3'])features=['f1','f2','f3']target='target'K=5orderby=Nonenum_partition=Nonewindow_size=Nonefromskopt.spaceimportSpace,Categorical,Integer,Real,Dimensionspace=[Real(0.5,10),# learning rate       (learn_rate)Real(0,1),# gamma               (min_split_improvement)Integer(3,4),# max_depth           (max_depth)Integer(11,13),# n_estimators        (ntrees)Integer(2,4),# min_child_weight    (min_rows)Real(0,1),# colsample_bytree    (col_sample_rate_per_tree)Real(0,1)]# subsample           (sample_rate)Adapter is shipped with XGBoost regressor and classifier, but you can pass in a callable estimator of your design if you wish to customize it.fromadapterimportAdapteradapt=Adapter(df,features,target,K,groupby=None,cross_validation_scheme='random_shuffle',search_method="bayesian_optimization",estimator="xgboost_regression")# "xgboost_regression" or "xgboost_classification" or callable estimator (more on this later)Try copying the link to the web browser to check out the dask dashboard:http://127.0.0.1:8789/status.You can visualize the Dask delayed computation graph:delayed_graph=adapt.construct_delayed_graph(num_iter=3,search_space=space)# we will set n_iter to 3 to make visualizing manageable.delayed_graph.visualize()Let’s run the code.num_initialis the number of random initial searches andnum_iteris the total number of search steps taken, including thenum_initialstep counts. (Example:num_initial=5, num_iter=15means 5 random search and 10 Bayesian search)res=adapt.run(num_initial=5,num_iter=15,search_space=space)While it runs, checkout the dashboard again, and click on theGraphtab. You will see the above computation graph being worked on in real time!Now you can retrieve the results:adapt.plot_improvements()# to show the improvementsoptimal_params=adapt.get_optimal_params()# which you can use to train your final modelIf you are running this in a local machine, you must take responsibility of removing the temporary directory:adapt.cleanup()Cross-validation schemesThere are 5 different cross-validation schemes supported by the adapter:random_shuffle: create K cross-validation folds from randomly shuffled rowsDefault mode for most regression tasks .ordered: create K cross-validation folds after sorting the train data by a certain columnUsed for regression tasks where data has time series nature with high temporal auto-correlation.Must supplyorderbyargument.binary_classification: create K cross-validation folds where positive/negative label proportion is preservedUsed for classification task.This mode will preserve the positive and negative label proportions in each fold.stratified_sampling: create K cross-validation folds such that the skew distribution of response is preservedUsed for regression task where the continuous response variable is highly skewed.This mode will preserve the skew distribution of the response values by sampling from stratification.Must supplynum_partitionargument.expanding_window: mainly for time series modelingRefer to:Tuning for multiple models in parallelAgain, let’s take a look at a specific example data:importpandasaspdimportnumpyasnpgroup_col=np.asarray([1]*10+[2]*10+[3]*10+[4]*10+[5]*10+[6]*10).reshape(-1,1)# this time we have a column specifying groupdata=np.arange(60*4).reshape(60,4)data=np.hstack((data,group_col))df=pd.DataFrame(data=data,columns=['target','f1','f2','f3','groups'])features=['f1','f2','f3']target='target'K=5orderby=Nonenum_partition=Nonewindow_size=Nonefromskopt.spaceimportSpace,Categorical,Integer,Real,Dimensionspace=[Real(0.5,10),# learning rate       (learn_rate)Real(0,1),# gamma               (min_split_improvement)Integer(3,4),# max_depth           (max_depth)Integer(11,13),# n_estimators        (ntrees)Integer(2,4),# min_child_weight    (min_rows)Real(0,1),# colsample_bytree    (col_sample_rate_per_tree)Real(0,1)]# subsample           (sample_rate)We can tune the models for each group by passing bygroupbyargument.fromadapterimportAdapteradapt=Adapter(df,features,target,K,groupby='groups',cross_validation_scheme='random_shuffle',search_method="bayesian_optimization",estimator="xgboost_regression")Run the adapter the same way:res=adapt.run(num_initial=5,num_iter=15,search_space=space)You can visualize the Dask delayed computation graph:Passing in an arbitrary callable estimatorYou can pass in an arbitrary callable estimator as long as it implements the standard scikit-learn estimator API:fromabcimportABCMeta,abstractmethodclassBaseEstimator(object,metaclass=ABCMeta):"""
    Base class for all Algorithm classes.
    """def__init__(self,**kwargs):pass@abstractmethoddeffit(self,X,y,params):pass@abstractmethoddefscore(self,X,y):pass@abstractmethoddefpredict(self,X):passFor example, we can even do something like:fromadapterimportBaseEstimator# import BaseEstimator!classDummyEstimator(BaseEstimator):def__init__(self):passdeffit(self,train_X,train_y,params):a=len(train_X)/10.foriinrange(int(a*5000000)):i+1print(len(train_X),len(train_y))defscore(self,validation_X,validation_y):print(len(validation_X),len(validation_y))return1.5defpredict(self,test_X):returnlen(test_X)my_estimator=DummyEstimator()Then you can use it with the Adapter:fromadapterimportAdapteradapt=Adapter(df,features,target,K,groupby='groups',cross_validation_scheme='random_shuffle',search_method="bayesian_optimization",estimator=my_estimator)# your own estimatorTuning multiple models with highly skewed training data sizesWhen the data size for each group is highly skewed, a suboptimal resource allocation can occur. In this case, it is more advantageous to throttle the feeding of delayed graphs to the Dask client by using multiple thread instances. Let’s again look at an example case:importpandasaspdimportnumpyasnpimporttimegroup_col=np.asarray([1]*100+[2]*2+[3]*2+[4]*2+[5]*2+[6]*2+[7]*2+[8]*2+[9]*2+[16]*2+[26]*2+[17]*2+[18]*2+[19]*2+[116]*2+[126]*2).reshape(-1,1)data=np.arange(130*4).reshape(130,4)data=np.hstack((data,group_col))df=pd.DataFrame(data=data,columns=['target','f1','f2','f3','groups'])features=['f1','f2','f3']groupby='groups'target='target'K=5fromadapterimportBaseEstimator# import BaseEstimator!classDummyEstimator(BaseEstimator):def__init__(self):passdeffit(self,train_X,train_y,params):a=len(train_X)/10.foriinrange(int(a*5000000)):i+1print(len(train_X),len(train_y))defscore(self,validation_X,validation_y):print(len(validation_X),len(validation_y))return1.5defpredict(self,test_X):returnlen(test_X)my_estimator=DummyEstimator()orderby=Nonenum_partition=Nonewindow_size=Nonefromskopt.spaceimportSpace,Categorical,Integer,Real,Dimensionspace=[Real(0.5,10),# learning rate       (learn_rate)Real(0,1),# gamma               (min_split_improvement)Integer(3,4),# max_depth           (max_depth)Integer(11,13),# n_estimators        (ntrees)Integer(2,4),# min_child_weight    (min_rows)Real(0,1),# colsample_bytree    (col_sample_rate_per_tree)Real(0,1)]# subsample           (sample_rate)In such a case, we userun_with_threadsmethod call, where we pass an additional argument ofnum_threads:fromadapterimportAdapteradapt=Adapter(df,features,target,K,groupby='groups',cross_validation_scheme='random_shuffle',search_method="bayesian_optimization",estimator=my_estimator)# your own estimatorres=adapt.run(num_initial=5,num_iter=15,search_space=space,num_threads=2)# num_threadsYou can check fromt the Dask dashboard that only two delayed computation graphs are worked on at the same time, achieving a dynamic resource allocation in effect:Todo:rest of the cross validation schemestesting hard thresholded submit process (and testing speed without it)supervised encodingsadd unit testscontinuous integration set uprandom search methodmulti GPU environmentdocumentations~~getting the results of the optimization~~~~visualization of optimizations~~early stop criterion using callbacks~~beta readme.rst for install and tutorial~~full readme.rst for install and tutorialperiodic trainingbayesian warm start trainingdependency managementsactive per worker threadpool managements