Package: hypopt
A Python machine learning package for grid searchhyper-parameter optimization using a validation set(defaults to cross validation when no validation set is available). This package works for Python 2.7+ and Python 3+, for any model (classification and regression), andruns in parallel on all threads on your CPU automatically.scikit-learnprovides a package forgrid-search hyper-parameter optimization **using cross-validation** on the training dataset. Unfortunately, cross-validation is impractically slow for large datasets and fails for small datasets due to the lack of data in each class needed to properly train each fold. Instead, we use a constant validation set to optimize hyper-parameters – thehypoptpackage makes this fast (distributed on all CPU threads) and easy (one line of code).hypopt.model_selection.fit_model_with_grid_searchsupports grid search hyper-parameter optimizationwhen you already have a validation set, eliminating the extra hours of training time required when using cross-validation. However, when no validation set is given, it defaults to using cross-validation on the training set. This allows you to alows usehypoptanytime you need to do hyper-parameter optimization with grid-search, regardless of whether you use a validation set or cross-validation.InstallationPython 2.7, 3.4, 3.5, and 3.6 are supported.Stable release:$ pip install hypoptDeveloper (unstable) release:$ pip install git+https://github.com/cgnorthcutt/hypopt.gitTo install the codebase (enabling you to make modifications):$ conda update pip # if you use conda
$ git clone https://github.com/cgnorthcutt/hypopt.git
$ cd hypopt
$ pip install -e .ExamplesBasic usage# Assuming you already have train, test, val sets and a model.fromhypoptimportGridSearchparam_grid=[{'C':[1,10,100],'kernel':['linear']},{'C':[1,10,100],'gamma':[0.001,0.0001],'kernel':['rbf']},]# Grid-search all parameter combinations using a validation set.gs=GridSearch(model=SVR(),param_grid=param_grid)gs.fit(X_train,y_train,X_val,y_val)print('Test Score for Optimized Parameters:',gs.score(X_test,y_test))Choosing the scoring metric to optimizeThe default metric is the themodel.score()function, so in the previous exampleSVR().score()is optimized, which defaults to accuracy.It’s easy to use a different scoring metric using thescoringparameter inhypopt.GridSearch.fit():# This will use f1 score as the scoring metric that you optimize.gs.fit(X_train,y_train,X_val,y_val,scoring='f1')For classification,hypoptsupports these string-named metrics: ‘accuracy’, ‘brier_score_loss’, ‘average_precision’, ‘f1’, ‘f1_micro’, ‘f1_macro’, ‘f1_weighted’, ‘neg_log_loss’, ‘precision’, ‘recall’, or ‘roc_auc’.For regression,hypoptsupports: “explained_variance”, “neg_mean_absolute_error”, “neg_mean_squared_error”, “neg_mean_squared_log_error”, “neg_median_absolute_error”, “r2”.You can also create your own metricyour_custom_score_func(y_true, y_pred)by wrapping it into an object usingsklearn.metrics.make_scorerlike:fromsklearn.metricsimportmake_scorerscorer=make_scorer(your_custom_scoring_func)opt.fit(X_train,y_train,X_val,y_val,scoring=scorer)Minimal working examplesClassification minimal working exampleRegression minimal working exampleOther Examples including a working example with MNISTA simple tutorial of the hypopt package.A working example with MNIST.Usehypoptwith any model (PyTorch, Tensorflow, caffe2, scikit-learn, etc.)All of the features of thehypoptpackage work withany model. Yes, any model. Feel free to use PyTorch, Tensorflow, caffe2, scikit-learn, mxnet, etc. If you use a scikit-learn model, allhypoptmethods will work out-of-the-box. It’s also easy to use your favorite model from a non-scikit-learn package, just wrap your model into a Python class that inherets thesklearn.base.BaseEstimator. Here’s an example for a generic classifier:fromsklearn.baseimportBaseEstimatorclassYourModel(BaseEstimator):# Inherits sklearn base classifierdef__init__(self,):passdeffit(self,X,y,sample_weight=None):passdefpredict(self,X):passdefscore(self,X,y,sample_weight=None):pass# Inherting BaseEstimator gives you these for free!# So if you inherit, there's no need to implement these.defget_params(self,deep=True):passdefset_params(self,**params):passPyTorch MNIST CNN ExampleCheck out a PyTorch MNIST CNN wrapped in the above classhere. You use any object instantion of this class withhypoptjust as you would any scikit-learn model. Another example of a fully compliant class is theLearningWithNoisyLabels() model.If you don’t wish to write this code yourself, there are existing packages to do this for you. For PyTorch, check out
theskorchPython package <https://skorch.readthedocs.io/en/stable/>which will wrap yourpytorchmodel
into ascikit-learncompliant model.