Package: tune-sklearn
tune-sklearnTune-sklearn is a drop-in replacement for Scikit-Learn’s model selection module (GridSearchCV, RandomizedSearchCV) with cutting edge hyperparameter tuning techniques.⚠️tune-sklearnis no longer being maintainedThe latest release0.5.0is the last version of the library that will be released by the Ray team, and it is compatible
withray>=2.7.x, ray<=2.9.x. The library will not be guaranteed to work with future Ray versions.The recommended alternative to keep up with the latest version of Ray is to migratetune-sklearnusage
to theRay Tune APIsto accomplish the same thing.Feel free to post an issue on theRay Githubif you run into any issues in migrating.FeaturesHere’s what tune-sklearn has to offer:Consistency with Scikit-Learn API: Change less than 5 lines in a standard Scikit-Learn script to use the API [example].Modern tuning techniques: tune-sklearn allows you to easily leverage Bayesian Optimization, HyperBand, BOHB, and other optimization techniques by simply toggling a few parameters.Framework support: tune-sklearn is used primarily for tuning Scikit-Learn models, but it also supports and provides examples for many other frameworks with Scikit-Learn wrappers such as Skorch (Pytorch) [example], KerasClassifier (Keras) [example], and XGBoostClassifier (XGBoost) [example].Scale up: Tune-sklearn leveragesRay Tune, a library for distributed hyperparameter tuning, to parallelize cross validation on multiple cores and even multiple machines without changing your code.Check out ourAPI DocumentationandWalkthrough(formasterbranch).InstallationDependenciesnumpy (>=1.16)ray(>=2.7.0)scikit-learn (>=0.23)User Installationpip install tune-sklearn ray[tune]orpip install -U git+https://github.com/ray-project/tune-sklearn.git && pip install 'ray[tune]'Tune-sklearn Early StoppingFor certain estimators, tune-sklearn can also immediately enableincremental training and early stopping. Such estimators include:Estimators that implement 'warm_start' (except for ensemble classifiers and decision trees)Estimators that implement partial fitXGBoost, LightGBM andCatBoostmodels (via incremental learning)To read more about compatible scikit-learn models, seescikit-learn's documentation at section 8.1.1.3.Early stopping algorithms that can be enabled include HyperBand and Median Stopping (see below for examples).If the estimator does not supportpartial_fit, a warning will be shown saying early stopping cannot be done and it will simply run the cross-validation on Ray's parallel back-end.Apart from early stopping scheduling algorithms, tune-sklearn also supports passing custom stoppers to Ray Tune. These
can be passed via thestopperargument when instantiatingTuneSearchCVorTuneGridSearchCV.
Seethe Ray documentation for an overview of available stoppers.ExamplesTuneGridSearchCVTo start out, it’s as easy as changing our import statement to get Tune’s grid search cross validation interface, and the rest is almost identical!TuneGridSearchCVaccepts dictionaries in the format{ param_name: str : distribution: list }or a list of such dictionaries, just like scikit-learn'sGridSearchCV. The distribution can also be the output of Ray Tune'stune.grid_search.# from sklearn.model_selection import GridSearchCVfromtune_sklearnimportTuneGridSearchCV# Other importsimportnumpyasnpfromsklearn.datasetsimportmake_classificationfromsklearn.model_selectionimporttrain_test_splitfromsklearn.linear_modelimportSGDClassifier# Set training and validation setsX,y=make_classification(n_samples=11000,n_features=1000,n_informative=50,n_redundant=0,n_classes=10,class_sep=2.5)X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=1000)# Example parameters to tune from SGDClassifierparameters={'alpha':[1e-4,1e-1,1],'epsilon':[0.01,0.1]}tune_search=TuneGridSearchCV(SGDClassifier(),parameters,early_stopping="MedianStoppingRule",max_iters=10)importtime# Just to compare fit timesstart=time.time()tune_search.fit(X_train,y_train)end=time.time()print("Tune Fit Time:",end-start)pred=tune_search.predict(X_test)accuracy=np.count_nonzero(np.array(pred)==np.array(y_test))/len(pred)print("Tune Accuracy:",accuracy)If you'd like to compare fit times with sklearn'sGridSearchCV, run the following block of code:fromsklearn.model_selectionimportGridSearchCV# n_jobs=-1 enables use of all cores like Tune doessklearn_search=GridSearchCV(SGDClassifier(),parameters,n_jobs=-1)start=time.time()sklearn_search.fit(X_train,y_train)end=time.time()print("Sklearn Fit Time:",end-start)pred=sklearn_search.predict(X_test)accuracy=np.count_nonzero(np.array(pred)==np.array(y_test))/len(pred)print("Sklearn Accuracy:",accuracy)TuneSearchCVTuneSearchCVis an upgraded version of scikit-learn'sRandomizedSearchCV.It also provides a wrapper for several search optimization algorithms from Ray Tune'ssearchers, which in turn are wrappers for other libraries. The selection of the search algorithm is controlled by thesearch_optimizationparameter. In order to use other algorithms, you need to install the libraries they depend on (pip installcolumn). The search algorithms are as follows:Algorithmsearch_optimizationvalueSummaryWebsitepip install(Random Search)"random"Randomized Searchbuilt-inSkoptSearch"bayesian"Bayesian Optimization[Scikit-Optimize]scikit-optimizeHyperOptSearch"hyperopt"Tree-Parzen Estimators[HyperOpt]hyperoptTuneBOHB"bohb"Bayesian Opt/HyperBand[BOHB]hpbandster ConfigSpaceOptuna"optuna"Tree-Parzen Estimators[Optuna]optunaAll algorithms other than RandomListSearcher accept parameter distributions in the form of dictionaries in the format{ param_name: str : distribution: tuple or list }.Tuples represent real distributions and should be two-element or three-element, in the format(lower_bound: float, upper_bound: float, Optional: "uniform" (default) or "log-uniform"). Lists represent categorical distributions.Ray Tune Search Spacesare also supported and provide a rich set of potential distributions. Search spaces allow for users to specify complex, potentially nested search spaces and parameter distributions. Furthermore, each algorithm also accepts parameters in their own specific format. More information inTune documentation.Random Search (default) accepts dictionaries in the format{ param_name: str : distribution: list }or a list of such dictionaries, just like scikit-learn'sRandomizedSearchCV.fromtune_sklearnimportTuneSearchCV# Other importsimportscipyfromrayimporttunefromsklearn.datasetsimportmake_classificationfromsklearn.model_selectionimporttrain_test_splitfromsklearn.linear_modelimportSGDClassifier# Set training and validation setsX,y=make_classification(n_samples=11000,n_features=1000,n_informative=50,n_redundant=0,n_classes=10,class_sep=2.5)X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=1000)# Example parameter distributions to tune from SGDClassifier# Note the use of tuples instead if non-random optimization is desiredparam_dists={'loss':['squared_hinge','hinge'],'alpha':(1e-4,1e-1,'log-uniform'),'epsilon':(1e-2,1e-1)}bohb_tune_search=TuneSearchCV(SGDClassifier(),param_distributions=param_dists,n_trials=2,max_iters=10,search_optimization="bohb")bohb_tune_search.fit(X_train,y_train)# Define the `param_dists using the SearchSpace API# This allows the specification of sampling from discrete and# categorical distributions (below for the `learning_rate` scheduler parameter)param_dists={'loss':tune.choice(['squared_hinge','hinge']),'alpha':tune.loguniform(1e-4,1e-1),'epsilon':tune.uniform(1e-2,1e-1),}hyperopt_tune_search=TuneSearchCV(SGDClassifier(),param_distributions=param_dists,n_trials=2,early_stopping=True,# uses Async HyperBand if set to Truemax_iters=10,search_optimization="hyperopt")hyperopt_tune_search.fit(X_train,y_train)Other Machine Learning Libraries and ExamplesTune-sklearn also supports the use of other machine learning libraries such as Pytorch (using Skorch) and Keras. You can find these examples here:KerasLightGBMSklearn Random ForestSklearn PipelinePytorch (Skorch)XGBoostDocumentationSee the auto-generated docshere.These are generated bylazydocsand should be updated on every release:pipinstalllazydocs
lazydocs/path/to/tune-sklearn/tune-sklearn--src-base-url="https://github.com/ray-project/tune-sklearn/blob/master"--overview-file="README.md"More informationRay Tune