How to Optimize a Deep Learning Model | by Zachary Warnes | Towards Data Science

Hyperparameter optimization is a critical part of any machine learning pipeline. Just selecting a model is not enough to achieve exceptional performance. You also need to tune your model to better perform on the problem.
This post will discuss hyperparameter tuning for deep learning architectures. There are code chunks to recreate the example shown.
Feel free to bookmark this post to copy code to optimize your deep learning models quickly.
If you are developing different deep learning models, replace code in the function ‘create_model’ with your specific model and update the relevant hyperparameters for each function. The rest of the optimization code is model-independent.
When developing models using scikit-learn, hyperparameter tuning is a relatively straightforward procedure. Both random and grid searches have their advantages and disadvantages. However, a third option known as Bayesian optimization provides a practical, balanced approach to hyperparameter optimization that can lead to more robust models.
Optimization in machine learning generally follows the same format. First, define a function that represents a loss. Then, by minimizing this loss, the model is forced to produce increasingly improved performance. Loss functions are chosen for two main reasons. The first is that they represent the problem well.
The second aspect of the standard loss functions is that the gradient is well defined. By having a well-defined gradient, many different optimization techniques become available. The most common of which is gradient descent which relies on the gradient of the function.
But what happens when the function does not have a gradient? This scenario is precisely the case when you have a black-box model.
A black-box model takes in an input and has an output. But what happens within the box is unknown. This property makes the model a black-box model.
The problem of optimization is more difficult when the model is a black-box.
Unfortunately, black boxes are the standard when building deep learning models, so standard optimization methods don’t perform well.
Briefly consider why optimization methods like gradient descent would not work to optimize hyperparameters. If you wanted to optimize for a hyperparameter, you would need to determine the derivative with respect to that hyperparameter. For example, imagine taking the derivative of the number of estimators for a random forest. It’s impossible.
However, black-box optimization does not care about determining the derivative of a function. Instead, the goal is to select a function based on multiple samples from the hyperparameter space. Then once this function closely represents hyperparameter space, you can optimize this function. The results of this optimization produce an optimal hyperparameter configuration that should perform well on the original black-box function.
In a previous post, I discussed different hyperparameter tuning methods and introduced Bayesian optimization. For the details regarding the other tuning methods available, their pitfalls, and benefits, refer to my previous post here:
towardsdatascience.com
Bayesian optimization is the process of sampling from the possible hyperparameter spaces, modeling a function based on these samples, and then optimizing that model
Bayesian optimization is the process of repeatedly sampling from the possible hyperparameter spaces, modeling a function based on these samples, and then optimizing that model
In contrast to gradient descent which requires optimization using the derivative, Bayesian optimization creates a representative function generated by sampling from the hyperparameter space multiple times.
This process of multiple sampling and generating an approximation of the space is why the method is Bayesian. Finally, the representative function is adjusted using prior information from the repeated sampling.
The process requires two things, a method to sample from the hyperparameter space and a method to optimize the function produced.
The method for sampling from a black-box will require sampling the hyperparameter space, creating a representative function based on multiple samples, then optimizing that function. Thus, the input becomes the hyperparameters, and the output is the performance of the model.
A good approximation of this function is generated with a sufficient number of samples from the hyperparameter space. From this, the model is then optimized for an optimal configuration of hyperparameters.
Deep learning models require a lot of tuning. When you manually tune your deep learning models, it is incredibly time-consuming.
The number of hyperparameters used to define deep learning models is numerous.
By repeatedly changing the parameters in a network manually, you are effectively performing a hands-on grid search.
Instead of manually checking different network configurations, you can define the distributions of each hyperparameter that you want to analyze. Then with many samples, you can find a more optimal model, and you can clearly understand how each hyperparameter affects your model overall.
This method of architecture tuning requires at least two functions. First, a process to evaluate the function and store the best result and the function to generate the deep learning model based on a set of hyperparameters.
For this post, I will focus on optimizing the architecture of a neural network with dropout layers. I’ll be using Keras within TensorFlow as it is straightforward to generate different models quickly.
A quick pause here to say thank you to those responsible for Keras and TensorFlow. If you’ve ever created a neural network, CNN, or LSTM from scratch, you know it is incredibly tedious and complicated. So I’m am extremely grateful that these libraries exist and that they are open to the public.
The model is generated using the hyperparameters sampled from each distribution defined. These values control the number of layers, the number of nodes across each of these layers, the dropout layer probability, the learning rate, optimizer, the learning decay for the optimization method, and the activation function. This model is set up for classification.
It is possible to apply these functions to different networks, such as a convolutional neural network. However, due to the nature of convolutional layers, each subsequent layer depends on the previous layer. This aspect increases the complexity of the model creation function. CNN optimization will likely be the focus of a future post. However, a simple workaround for CNNs is fixing the number of layers and optimizing the remaining parameters.
The first step is to import some packages and initialize some parameters. Here I also initialize the distributions of different hyperparameters that I was to test. You’ll notice that I don’t set up all of my hyperparameters as distributions, so I won’t be testing every combination possible.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.datasets import load_breast_cancerfrom sklearn.model_selection import train_test_splitimport tensorflow as tffrom tensorflow.keras import backend as Kfrom tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Dense, Dropoutfrom tensorflow.keras.callbacks import TensorBoardfrom tensorflow.keras.optimizers import Adam, SGDfrom tensorflow.keras.models import load_modelfrom tensorflow.keras.callbacks import EarlyStoppingfrom tensorflow.keras.optimizers.schedules import ExponentialDecayimport skoptfrom skopt.space import Real, Categorical, Integerfrom skopt import gp_minimize, forest_minimizefrom skopt.plots import plot_convergence, plot_objective, plot_evaluationsfrom skopt.utils import use_named_argsprint( "Tensorflow version: ", tf.__version__)data = load_breast_cancer()y = data.target.astype(float)X = data.dataX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=2)X_validation, X_test, y_validation, y_test = train_test_split(X_test, y_test, test_size=0.4, random_state=2)
Next, distributions for the hyperparameters are defined. These distributions can be real continuous values, integer values, or categorical values. Moreover, the type of distribution is added for continuous distributions.
For example, a log-uniform distribution may work better for a learning rate decay when most of the values that you want to test fall in a subset of the distribution. The hyperparameters also require default values. These defaults are used in the first model built in the optimization process.
dim_optimization = Categorical(categories=['adam', 'SGD'],name='optimization')dim_learning_rate = Real(low=1e-3, high=1e-1, prior='log-uniform',name='learning_rate')dim_learning_decay = Real(low=0.9, high=0.999, prior='uniform',name='learning_decay')dim_num_layers = Integer(low=1, high=5, name='num_layers')dim_num_dense_nodes = Integer(low=5, high=50, name='num_dense_nodes')dim_dropout_prob = Real(low=0.5, high=0.99, prior='uniform',name='dropout_prob')dim_activation = Categorical(categories=['sigmoid', 'softmax', 'relu'],name='activation')dimensions = [    dim_optimization, dim_learning_rate, dim_learning_decay,    dim_num_layers, dim_num_dense_nodes, dim_dropout_prob, dim_activation]NUM_LAYERS = 1BATCH_SIZE = 128LEARNING_RATE = 0.001DECAY_STEPS = 1000DENSE_UNITS = 20DROPOUT_PROB = 0.8ACTIVATION_FUNC = 'relu'LOSS_FUNC = 'binary_crossentropy'METRIC = 'accuracy'LEARNING_DECAY = 0.9OTIMIZATION_FUNC = 'adam'ACQ_FUNC = 'EI' # Expected Improvementbest_score = 0EPOCHS = 50BATCH_SIZE = 256N_CALLS = 1000default_parameters = [OTIMIZATION_FUNC, LEARNING_RATE, LEARNING_DECAY, NUM_LAYERS, DENSE_UNITS, DROPOUT_PROB, ACTIVATION_FUNC]
If you want to change the underlying model, this function will need to be changed. Depending on the hyperparameter input, some of the model parameters may be conditioned on other parameters. For example, in a CNN the filter dimensions affect the output shape into the next layer. Currently skopt does not support conditional features.
For this example, I control the number of units in each of the dense layers and then add a dropout layer. This structure is a fairly simple network, but this code is easily modified to include different models of your choice.
The next code chunk generates a deep neural net based on the hyperparameters from the sample.
def create_model(optimization, learning_rate, learning_decay, num_layers, num_dense_nodes, dropout_prob, activation):    model = Sequential()    name = 'layer_{0}_dense_units'.format(0)    model.add( Dense( num_dense_nodes, input_dim=X_train.shape[1],        activation=activation, name=name))    for i in range(num_layers-1):        name = 'layer_{0}_dense_units'.format(i+1)         model.add( Dense( num_dense_nodes, activation=activation, name=name))        model.add( Dropout(dropout_prob))    model.add( Dense(1, activation=activation))    lr_schedule = ExponentialDecay( initial_learning_rate=learning_rate,        decay_steps=DECAY_STEPS, decay_rate=learning_decay)        if optimization == 'adam':        optimizer = Adam(learning_rate=lr_schedule)    if optimization == 'SGD':        optimizer = SGD(learning_rate=lr_schedule)    model.compile(        optimizer=optimizer, loss=LOSS_FUNC, metrics=[METRIC])    return model
Fitness is the function that skopt will optimize with ‘gp_minimize’, which minimizes the Gaussian process generated to model the hyperparameter space. This function is minimization by construction. Hence the score that is being maximized becomes a minimization of the negative scores.
@use_named_args(dimensions=dimensions)def fitness(optimization, learning_rate, learning_decay, num_layers, num_dense_nodes, dropout_prob, activation):    # Create the neural network    model = create_model( optimization=optimization,        learning_rate=learning_rate, learning_decay=learning_decay,        num_layers=num_layers, num_dense_nodes=num_dense_nodes,        dropout_prob=dropout_prob, activation=activation)    # Save log for tensorboard    callback_log = TensorBoard(        log_dir ="./21_logs/opt_{0}_lr_{1:.0e}_lr_decay_{2:.0e}_layers_{3}_nodes_{4}_dropout_{5}_activation_{5}/".format(        optimization, learning_rate, learning_decay, num_layers,         num_dense_nodes, dropout_prob, activation),        histogram_freq=0, write_graph=True, write_grads=False, write_images=False)    # Train the model.    history = model.fit(        x= X_train,y= y_train,        epochs=EPOCHS,batch_size=BATCH_SIZE,        validation_data=(X_validation, y_validation),        callbacks=[callback_log],verbose=0)    # Get the final model performance.    col = [x for x in list(history.history.keys()) if 'val_'+METRIC in x]    score = history.history[col[0]][-1]    print("--> Validation {0}: {1:.2%}".format(METRIC, score))    global best_score    # Track scores and save best model    if score > best_score:        model.save('Optimal-NN')    best_score = score    # Clear model to save space    del model    K.clear_session()        # Skopt minimizes black-box functions, return the negative    return -score
The skopt function ‘gp_minimize’ runs Bayesian optimization on the black-box model to determine the optimal hyperparameter configuration. The ‘acq_func’, which is used to optimized the Gaussian process, is one of several options for optimization.
search_result = gp_minimize(func=fitness,    dimensions=dimensions,    acq_func=ACQ_FUNC,    n_calls=N_CALLS,    x0=default_parameters)
Once the search is completed the best model found is saved locally. This model can be loaded in and used from this point and trained further if necessary.
The search results contain the hyperparameters at each evaluation as well as the performance of each model tested.
To see how the search improved the model performance over time, run the following code chunk:
plot_convergence(search_result)
However, just seeing the improvement over the different models tested is not the largest benefit of Bayesian optimization. The behavior of each of the hyperparameters is analyzed independently, and how they interact with other hyperparameters.
_ = plot_objective(result=search_result)
You can see from the image above that there still is some uncertainty around some of the parameters. The results from changing the number of dense nodes vary a fair amount. However, the learning rate appears to be modeled effectively. You can see the optimization has selected a learning rate around 10e-2. Similarly, sigmoid activation and adam optimization appear to perform the best across all models generated.
Bayesian optimization is useful when trying to fine-tune and optimize a computationally expensive model. In addition, Bayesian optimization requires few samples and rapidly closes in on a more optimal hyperparameter configuration, so it is ideal for deep learning models.
For deep learning models such as neural networks and network variants, fine-tuning architecture is crucial. However, don’t waste your time manually tuning your model. Instead, use Bayesian optimization to determine what the role of each hyperparameter is for your model.
If you’re interested in reading articles about novel data science tools and understanding machine learning algorithms, consider following me on Medium.
If you’re interested in my writing and want to support me directly, please subscribe through the following link. This link ensures that I will receive a portion of your membership fees.