Deep Dive into TensorBoard: Tutorial With Examples
By: Patrycja Jenkner
Tags: Machine Learning, Experiment Tracking, Tensorboard, Tutorial, Data Science

Here are some things we’ll cover in this text:

Visualizing images in TensorBoard

in TensorBoard Checking model weights and biases on TensorBoard

on TensorBoard visualizing the model’s architecture

sending a visual of the confusion matrix to TensorBoard

to TensorBoard profiling your application so as to see its performance , and

your application so as to see its , and using TensorBoard with Keras, PyTorch, and XGBoost

Let’s get to it.

How to use TensorBoard

This section will focus on helping you understand how to use TensorBoard in your machine learning workflow.

How to install TensorBoard

Before you can start using TensorBoard you have to install it either via pip or via conda

pip install tensorboard

conda install -c conda-forge tensorboard

Using TensorBoard with Jupyter notebooks and Google Colab

With TensorBoard installed, you can now load it into your Notebook. Note that you can use it in a Jupyter Notebook or Google’s Colab.

%load_ext tensorboard

Once that is done you have to set a log directory. This is where TensorBoard will store all the logs. It will read from these logs in order to display the various visualizations.

log_folder = 'logs'

In the event that you want to reload the TensorBoard extension, the command below will do the magic — no pun intended.

%reload_ext tensorboard

You might want to clear the current logs so that you can write fresh ones to the folder. You can achieve that by running this command on Google Colab

!rm -rf /logs/

on Jupyter Notebooks

rm -rf logs

If you are running multiple experiments, you might want to store all logs so that you can compare their results. This can be achieved by creating logs that are timestamped. To do that, use the command below:

import datetime

log_folder = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")

How to run TensorBoard

Running Tensorboard involves just one line of code. In this section you’ll see how to do this.

Let’s now walk through an example where you will use TensorBoard to visualize model metrics. For that purpose, you need to build a simple image classification model.

import tensorflow as tf mnist = tf.keras.datasets.mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train, X_test = X_train / 255.0, X_test / 255.0 model = tf.keras.models.Sequential([

tf.keras.layers.Flatten(input_shape=(28, 28)),

tf.keras.layers.Dense(512, activation='relu'),

tf.keras.layers.Dropout(0.2),

tf.keras.layers.Dense(10, activation='softmax')]) model.compile(optimizer='sgd',

loss='sparse_categorical_crossentropy',

metrics=['accuracy'])

Next, load in the TensorBoard notebook extension and create a variable pointing to the log folder.

%load_ext tensorboard

log_folder = 'logs'

How to use TensorBoard callback

The next step is to specify the TensorBoard callback during the model’s fit method. In order to do that you first have to import the TensorBoard callback.

This callback is responsible for logging events such as Activation Histograms, Metrics Summary Plots, Profiling and Training Graph Visualizations.

from tensorflow.keras.callbacks import TensorBoard

With that in place, you can now create the TensorBoard callback and specify the log directory using log_dir. The TensorBoard callback also takes other parameters:

histogram_freq is the frequency at which to compute activation and weight histograms for layers of the model. Setting this to 0 means that histograms will not be computed. In order for this to work you have to set the validation data or the validation split .

is the frequency at which to compute activation and weight histograms for layers of the model. Setting this to 0 means that histograms will not be computed. In order for this to work you have to set the or the . write_graph dictates if the graph will be visualized in TensorBoard

dictates if the graph will be visualized in TensorBoard write_images when set to true, model weights are visualized as an image in TensorBoard

when set to true, model weights are visualized as an image in TensorBoard update_freq determines how losses and metrics are written to TensorBoard. When set to an integer, say 100, losses and metrics are logged every 100 batches. When set to batch the losses and metrics are set after every batch. When set to epoch they are written after every epoch

determines how and are written to TensorBoard. When set to an integer, say 100, losses and metrics are logged every 100 batches. When set to batch the losses and metrics are set after every batch. When set to epoch they are written after every epoch profile_batch determines which batches will be profiled. By default, the second batch is profiled. You can also set, for example from 5 and to 10, to profile batches 5 to 10, i.e profile_batch=’5,10′ . Setting profile_batch to 0 disables profiling.

determines which batches will be profiled. By default, the second batch is profiled. You can also set, for example from 5 and to 10, to profile batches 5 to 10, i.e profile_batch=’5,10′ . Setting profile_batch to 0 disables profiling. embeddings_freq the frequency at which the embedding layers will be visualized. Setting this to zero means that the embeddings will not be visualized

callbacks = [TensorBoard(log_dir=log_folder,

histogram_freq=1,

write_graph=True,

write_images=True,

update_freq='epoch',

profile_batch=2,

embeddings_freq=1)]

The next item is to fit the model and pass in the callback.

model.fit(X_train, y_train,

epochs=10,

validation_split=0.2,

callbacks=callbacks)

How to launch TensorBoard

If you installed TensorBoard via pip, you can launch it via the command line

tensorboard -- logdir=log

On a Notebook, you can launch it using:

%tensorboard -- logdir={log_folder}

The TensorBoard is also available via the browser using the following URL

Running TensorBoard remotely

When working on a remote server, you can use SSH tunneling to forward the port of the remote server to your local machine at port (port 6006 in this example). This is how this would look like:

ssh -L 6006:127.0.0.1:6006 your_user_name@my_server_ip

With that inplace, you can run the TensorBoard in the normal way.

Just remember that the port you specify in tensorboard command (by default it is 6006) should be the same as the one in the ssh tunneling.

tensorboard --logdir=/tmp --port=6006

Note: If you are using the default port 6006 you can drop –port=6006. You will be able to see the TensorBoard on the local machine but TensorBoard will actually be running on the remote server.

TensorBoard dashboard

Let us now look at the various tabs on the TensorBoard.

TensorBoard scalars

The Scalars tab shows changes in the loss and metrics over the epochs. It can be used to track other scalar values such as learning rate and training speed.

TensorBoard images

This dashboard has images that show the weights. Adjusting the slider displays the weights at various epochs.

TensorBoard graphs

This tab shows your model’s layers. You can use this to check if the architecture of the model looks as intended.

TensorBoard distributions

The distribution tab shows the distribution of tensors. For example in the dense layer below, you can see the distribution of the weights and biases over each epoch.

TensorBoard histograms

The Histograms show the distribution of tensors over time. For example, looking at dense_1 below, you can see the distribution of the biases over each epoch.

Using the TensorBoard projector

You can use TensorBoard’s Projector to visualize any vector representation e.g. word embeddings and images.

Words embeddings are numerical representations of words that capture their semantic relationship. The projector helps you see those representations. You can find it under the Inactive dropdown.

Plot training examples with TensorBoard

You can use TensorFlow Image Summary API to visualize training images. This is especially useful when working with image data like in this case.

Now, create a new log directory for the images as shown below.

logdir = "logs/train_data/"

The next step is to create a file writer and point it to this directory.

file_writer = tf.summary.create_file_writer(logdir)

At the beginning of this article (in the “How to run TensorBoard” section), you specified that the image shape was 28 by 28. It is important information when reshaping the images before writing them to TensorBoard. You also need to specify the channel to be 1 because the images are grayscale. Afterward, you use the file_write to write the images to TensorBoard.

In this example, the images at index 10 to 30 will be written to TensorBoard.

import numpy as np with file_writer.as_default():

images = np.reshape(X_train[10:30], (-1, 28, 28, 1))

tf.summary.image("20 Digits", images, max_outputs=25, step=0)

Visualize images in TensorBoard

Apart from visualizing image tensors, you can also visualize actual images in TensorBoard. In order to illustrate that, you need to convert the MNIST tensors to images using Matplotlib. After that, you need to use `tf.summary.image` to plot the images in Tensorboard.

Start by clearing the logs, alternatively you can use timestamped log folders. After that specify the log directory and create a `tf.summary.create_file_writer` that will be used to write the images to TensorBoard

!rm -rf logs # if Juypter Notebook remove ! import io

import matplotlib.pyplot as plt class_names = ['Zero','One','Two','Three','Four','Five','Six','Seven','Eight','Nine']

logdir = "logs/plots/"

file_writer = tf.summary.create_file_writer(logdir)

Next, create a grid that will hold the images. In this case, the grid will hold 36 digits.

def image_grid():

figure = plt.figure(figsize=(12,8)) for i in range(36):

plt.subplot(6, 6, i + 1)

plt.xlabel(class_names[y_train[i]])

plt.xticks([])

plt.yticks([])

plt.grid(False)

plt.imshow(X_train[i], cmap=plt.cm.coolwarm) return figure figure = image_grid()

Now convert the digits into a single image to visualize it in the TensorBoard.

def plot_to_image(figure):

buf = io.BytesIO()

plt.savefig(buf, format='png')

plt.close(figure)

buf.seek(0) digit = tf.image.decode_png(buf.getvalue(), channels=4)

digit = tf.expand_dims(digit, 0) return digit

The next step is to use the writer and `plot_to_image` to display the images on TensorBoard.

with file_writer.as_default():

tf.summary.image("MNIST Digits", plot_to_image(figure), step=0) %tensorboard -- logdir logs/plots

Log confusion matrix to TensorBoard

Using the same example, you can log the confusion matrix for all epochs. First, define a function that will return a Matplotlib figure holding the confusion matrix.

import itertools def plot_confusion_matrix(cm, class_names):

figure = plt.figure(figsize=(8, 8))

plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Accent)

plt.title("Confusion matrix")

plt.colorbar()

tick_marks = np.arange(len(class_names))

plt.xticks(tick_marks, class_names, rotation=45)

plt.yticks(tick_marks, class_names) cm = np.around(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis], decimals=2)

threshold = cm.max() / 2. for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):

color = "white" if cm[i, j] > threshold else "black"

plt.text(j, i, cm[i, j], horizontalalignment="center", color=color)



plt.tight_layout()

plt.ylabel('True label')

plt.xlabel('Predicted label') return figure

Next, clear the previous logs, define the log directory for the confusion matrix, and create a writer variable for writing into the log folder.

!rm -rf logs logdir = "logs"

file_writer_cm = tf.summary.create_file_writer(logdir)

The step that follows this is to create a function that will make predictions from the model and log the confusion matrix as an image.

After that use the `file_writer_cm to write` the confusion matrix to the log directory.

from tensorflow import keras

from sklearn import metrics def log_confusion_matrix(epoch, logs):

predictions = model.predict(X_test)

predictions = np.argmax(predictions, axis=1) cm = metrics.confusion_matrix(y_test, predictions)

figure = plot_confusion_matrix(cm, class_names=class_names)

cm_image = plot_to_image(figure)



with file_writer_cm.as_default():

tf.summary.image("Confusion Matrix", cm_image, step=epoch)

This will be followed by the definition of the TensorBoard callback and the LambdaCallback .

The LambdaCallback will log the confusion matrix on every epoch. Finally fit the model using these two callbacks.

Since you’ve already fitted the model before, it would be advisable to restart your runtime and ensure that you are fitting the model just once.

callbacks = [

TensorBoard(log_dir=log_folder,

histogram_freq=1,

write_graph=True,

write_images=True,

update_freq='epoch',

profile_batch=2,

embeddings_freq=1),

keras.callbacks.LambdaCallback(on_epoch_end=log_confusion_matrix)

] model.fit(X_train, y_train,

epochs=10,

validation_split=0.2,

callbacks=callbacks)

Now run TensorBoard and check the confusion matrix on the Images tab.

%tensorboard -- logdir logs

Hyperparameter tuning with TensorBoard

Another cool thing you can do with TensorBoard is use it to visualize parameter optimization. Sticking to the same MNIST example, you can attempt to tune the hyperparameters of the model (manually or using automated hyperparameter optimization) and visualize them in TensorBoard.

Here’s the final result that you expect to obtain. The dashboard is available under the HPARAMS tab.

To achieve this you have to clear the previous logs and import the hparams plugin.