TensorBoard: Hyperparameter Optimization | by Renu Khandelwal | Towards Data Science

Prerequisites:
TensorBoard- A Visualization suite for Tensorflow models
In this article, you will learn hyperparameter optimization and then display the results of the hyperparameter optimization using TensorBoard.
What is Hyperparameter in the context of a deep neural network?
Your objective in a deep learning neural network is to find the weights of the nodes that will help us understand data patterns in an image, any text or speech.
You can do this by designing your neural network parameters with the values that provide the best accuracy and precision for the model.
So, what are these parameters that are referred to as hyperparameters?
Different parameters used for training the neural network model is called Hyperparameters. These hyperparameters are tuned like knobs to improve the performance of a neural network resulting in an optimized model.
Some of the Hyperparameter in a neural network are
No. of hidden layers
No. of units or nodes in a hidden layer
Learning rate
Dropout rate
Epochs or iterations
Optimizers like SGD, Adam, AdaGrad, Rmsprop, etc.
Activation functions like ReLU, sigmoid, leaky ReLU etc.
Batch size
How to implement hyperparameter optimization?
Hyperparameter optimization is the process to find the value for hyperparameter like optimizers, learning rate, dropout rates, etc. of a deep learning algorithm that will give the best model performance.
You can perform a hyperparameter optimization using the following techniques.
Manual search
Grid search: An exhaustive search of all possible combinations of the specified hyperparameters resulting in a cartesian product.
Random search: Hyperparameters are randomly selected, not every combination of Hyperparameter is tried. As the number of hyperparameters increases, the random search is a better option as it arrives at a good combination of hyperparameters faster.
Bayesian optimization: Incorporates prior data about hyperparameters, including accuracy or loss of the model. Prior information help determine the better approximation of hyperparameter selection for the model.
For visualizing the hyperparameter tuning for the model on the TensorBoard, we will use a Grid Search technique where we will use a few hyperparameters like no. of nodes, different optimizers, or learning rate, and different dropout rates and look at the accuracy and loss of the model.
Why use TensorBoard for hyperparameter optimization?
A picture is a thousand words, and this is also applied to sophisticated deep learning models. Deep learning models were considered as a black box where you send some input data, the model does some complex computation, and voila, you now have your results!!!
TensorBoard is a visualization toolkit from Tensorflow to display different metrics, parameters, and other visualizations that help debug, track, fine-tune, optimize, and share your deep learning experiment results
With TensorBoard, you can track the accuracy and loss of the model at every epoch; and also with different hyperparameters values. Tracking accuracy for different values of Hyperparameter will help you to fine-tune the model faster.
Finally, here is the code implementation in Python...
We will visualize the scalars, graphs, and distribution using TensorBoard using cats and dogs dataset.
Import TensorFlow and the TensorBoard HParams plugin along with Keras libraries for preprocessing the image and creating the model.
import tensorflow as tffrom tensorboard.plugins.hparams import api as hpimport datetimefrom tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2Dfrom tensorflow.keras.preprocessing.image import ImageDataGenerator, img_to_array, load_imgimport numpy as np
I have used TensorFlow 2.0.0 version.
# Load the TensorBoard notebook extension%load_ext tensorboard
Setting critical parameters for the training
BASE_PATH = 'Data\\dogs-vs-cats\\train\\'TRAIN_PATH='Data\\dogs-vs-cats\\train_data\\'VAL_PATH='Data\\dogs-vs-cats\\validation_data\\'batch_size = 32 epochs = 5IMG_HEIGHT = 150IMG_WIDTH = 150
Rescale and Apply different Augmentation to the training image
train_image_generator = ImageDataGenerator(                                                rescale=1./255,                                              rotation_range=45,                                                width_shift_range=.15,                                                height_shift_range=.15,                                                horizontal_flip=True,                                                zoom_range=0.3)
Rescale Validation data
validation_image_generator = ImageDataGenerator(rescale=1./255)
Generate batches of normalized data for train and validation data set
train_data_gen = train_image_generator.flow_from_directory(batch_size = batch_size,                                                     directory=TRAIN_PATH,                                                     shuffle=True,                                                     target_size=(IMG_HEIGHT, IMG_WIDTH),                                                     class_mode='categorical')val_data_gen = validation_image_generator.flow_from_directory(batch_size = batch_size,                                                              directory=VAL_PATH,                                                              target_size=(IMG_HEIGHT, IMG_WIDTH),                                                              class_mode='categorical')
We are using four hyperparameters for running our experiment by listing the different values or ranges of value for the Hyperparameter.
For discrete hyperparameters, all possible combinations of parameters will be tried, and for real-valued parameters, only the lower and upper bound will be used.
Num of units in the first Dense layer: 256 and 512Drop out rate: the range is between 0.1 and 0.2. So a dropout rate of 0.1 and 0.2 will be used.Optimizers: adam, SGD, and rmspropLearning rate for the optimizers:0.001, 0.0001 and 0.0005,
Num of units in the first Dense layer: 256 and 512
Drop out rate: the range is between 0.1 and 0.2. So a dropout rate of 0.1 and 0.2 will be used.
Optimizers: adam, SGD, and rmsprop
Learning rate for the optimizers:0.001, 0.0001 and 0.0005,
We also set the metrics as accuracy to be displayed on the TensorBoard
## Create hyperparametersHP_NUM_UNITS=hp.HParam('num_units', hp.Discrete([ 256, 512]))HP_DROPOUT=hp.HParam('dropout', hp.RealInterval(0.1, 0.2))HP_LEARNING_RATE= hp.HParam('learning_rate', hp.Discrete([0.001, 0.0005, 0.0001]))HP_OPTIMIZER=hp.HParam('optimizer', hp.Discrete(['adam', 'sgd', 'rmsprop']))METRIC_ACCURACY='accuracy'
Creating and Configuring the log files
log_dir ='\\logs\\fit\\' + datetime.datetime.now().strftime('%Y%m%d-%H%M%S')with tf.summary.create_file_writer(log_dir).as_default():    hp.hparams_config(    hparams=    [HP_NUM_UNITS, HP_DROPOUT,  HP_OPTIMIZER, HP_LEARNING_RATE],    metrics=[hp.Metric(METRIC_ACCURACY, display_name='Accuracy')],    )
Hyperparameters are not hardcoded but are taken from hparams dictionary for different parameters: HP_DROPOUTfor dropout, HP_NUM_UNITS for the number of units in the first Dense layer, HP_OPTIMIZER sets the different optimizers. We take the optimizer that gets used and set the learning rate based on HP_LEARNING_RATE.
The function returns the validation accuracy of the last epoch.
def create_model(hparams):    model = Sequential([    Conv2D(64, 3, padding='same', activation='relu',            input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),    MaxPooling2D(),    #setting the Drop out value based on HParam    Dropout(hparams[HP_DROPOUT]),    Conv2D(128, 3, padding='same', activation='relu'),    MaxPooling2D(),    Dropout(hparams[HP_DROPOUT]),    Flatten(),    Dense(hparams[HP_NUM_UNITS], activation='relu'),    Dense(2, activation='softmax')])        #setting the optimizer and learning rate    optimizer = hparams[HP_OPTIMIZER]    learning_rate = hparams[HP_LEARNING_RATE]    if optimizer == "adam":        optimizer = tf.optimizers.Adam(learning_rate=learning_rate)    elif optimizer == "sgd":        optimizer = tf.optimizers.SGD(learning_rate=learning_rate)    elif optimizer=='rmsprop':        optimizer = tf.optimizers.RMSprop(learning_rate=learning_rate)    else:        raise ValueError("unexpected optimizer name: %r" % (optimizer_name,))        # Comiple the mode with the optimizer and learninf rate specified in hparams    model.compile(optimizer=optimizer,              loss='categorical_crossentropy',              metrics=['accuracy'])        #Fit the model     history=model.fit_generator(    train_data_gen,    steps_per_epoch=1000,    epochs=epochs,    validation_data=val_data_gen,    validation_steps=1000,    callbacks=[        tf.keras.callbacks.TensorBoard(log_dir),  # log metrics        hp.KerasCallback(log_dir, hparams),# log hparams            ])    return history.history['val_accuracy'][-1]
For each run of the model, log the hparams summary with Hyperparameter and the final epochs accuracy. We need to convert the validation accuracy of the last epoch to a scalar value.
def run(run_dir, hparams):  with tf.summary.create_file_writer(run_dir).as_default():    hp.hparams(hparams)  # record the values used in this trial    accuracy = create_model(hparams)    #converting to tf scalar    accuracy= tf.reshape(tf.convert_to_tensor(accuracy), []).numpy()    tf.summary.scalar(METRIC_ACCURACY, accuracy, step=1)
Experimentation here uses Grid Search and tests all possible combinations of hyperparameters for the number of units for the first layer, dropout rate, optimizers, and their learning rates, and accuracy is used for accuracy.
session_num = 0for num_units in HP_NUM_UNITS.domain.values:  for dropout_rate in (HP_DROPOUT.domain.min_value, HP_DROPOUT.domain.max_value):    for optimizer in HP_OPTIMIZER.domain.values:        for learning_rate in HP_LEARNING_RATE.domain.values:          hparams = {              HP_NUM_UNITS: num_units,              HP_DROPOUT: dropout_rate,              HP_OPTIMIZER: optimizer,              HP_LEARNING_RATE: learning_rate,          }          run_name = "run-%d" % session_num          print('--- Starting trial: %s' % run_name)          print({h.name: hparams[h] for h in hparams})          run('logs/hparam_tuning/' + run_name, hparams)          session_num += 1
You can view the HParams TensorBoard dashboard using different commands: either in Jupyter notebook or using cmd
Using cmd
you will display the Hparam dashboard by providing the directory path where the different run logs were stored using the following command
python -m tensorboard.main --logdir="logs/hparam_tuning"
When sorting the accuracy in descending order, you can see that the most optimized model is with 256 units with a dropout rate of 0.2 and rmsprop optimizer with a learning rate of 0.0005.
Using Jupyter notebook
%tensorboard --logdir='\logs\hparam_tuning'
you can also view the Parallel Coordinates View, displaying the individual runs for each hyperparameter and also displaying accuracy
Tensorboard Hparams dashboard help find the most optimized hyperparameters for the best model accuracy
TensorBoard hyperparameters tuning provides a visual way to understand which hyperparameters can be used for fine-tuning the deep learning model for best accuracy