Hyperparameter Tuning in Python | Towards Data Science

Kaggle’s Don’t Overfit II competition presents an interesting problem. We have 20,000 rows of continuous variables, with only 250 of them belonging to the training set.
The challenge is not to overfit.
With such a small dataset — and even smaller training set, this can be a difficult task!
In this article, we’ll explore hyperparameter optimization as a means of preventing overfitting.
The full notebook can be found here.
Wikipedia states that “hyperparameter tuning is choosing a set of optimal hyperparameters for a learning algorithm”. So what is a hyperparameter?
A hyperparameter is a parameter whose value is set before the learning process begins.
Some examples of hyperparameters include penalty in logistic regression and loss in stochastic gradient descent.
In sklearn, hyperparameters are passed in as arguments to the constructor of the model classes.
We will explore two different methods for optimizing hyperparameters:
Grid Search
Random Search
We’ll begin by preparing the data and trying several different models with their default hyperparameters. From these we’ll select the top two performing methods for hyperparameter tuning.
We then find the mean cross validation score and standard deviation:
RidgeCV Mean:  0.6759762475523124STD:  0.1170461756924883LassoCV Mean:  0.5STD:  0.0ElasticNetCV Mean:  0.5STD:  0.0LassoLarsCV Mean: 0.5STD:  0.0BayesianRidgeCV Mean:  0.688224616492365STD:  0.13183095412112777LogisticRegressionCV Mean:  0.7447916666666667STD:  0.053735373404660246SGDClassifierCV Mean:  0.7333333333333333STD:  0.03404902964480909
Our top performing models here are logistic regression and stochastic gradient descent. Let’s see if we can improve their performance through hyperparameter optimization.
Grid search is a traditional way to perform hyperparameter optimization. It works by searching exhaustively through a specified subset of hyperparameters.
Using sklearn’sGridSearchCV, we first define our grid of parameters to search over and then run the grid search.
Fitting 3 folds for each of 128 candidates, totalling 384 fitsBest Score:  0.7899186582809224Best Params:  {'C': 1, 'class_weight': {1: 0.6, 0: 0.4}, 'penalty': 'l1', 'solver': 'liblinear'}
We improved our cross validation score from 0.744 to 0.789!
The benefit of grid search is that it is guaranteed to find the optimal combination of parameters supplied. The drawback is that it can be very time consuming and computationally expensive.
We can combat this with random search.
Random search differs from grid search mainly in that it searches the specified subset of hyperparameters randomly instead of exhaustively. The major benefit being decreased processing time.
There is a tradeoff to decreased processing time, however. We aren’t guaranteed to find the optimal combination of hyperparameters.
Let’s give random search a try with sklearn’s RandomizedSearchCV. Very similar to grid search above, we define the hyperparameters to search over before running the search.
An important additional parameter to specify here is n_iter. This specifies the number of combinations to randomly try.
Selecting too low of a number will decrease our chance of finding the best combination. Selecting too large of a number will increase our processing time.
Fitting 3 folds for each of 1000 candidates, totalling 3000 fitsBest Score:  0.7972911250873514Best Params:  {'penalty': 'elasticnet', 'loss': 'log', 'learning_rate': 'optimal', 'eta0': 100, 'class_weight': {1: 0.7, 0: 0.3}, 'alpha': 0.1}
Here we improved the cross validation score from 0.733 to 0.780!
Here we explored two methods for hyperparameter turning and saw improvement in model performance.
While this is an important step in modeling, it is by no means the only way to improve performance.
In future articles we will explore other means to prevent overfitting including feature selection and ensembling.