A Quickstart Guide to TensorBoard | by Rohan Jagtap | Towards Data Science

Everyone agrees that “visuals are better than text”. Visualizations offer feasibility and interactivity in any kind of interpretation. Same goes for ML modeling. One may wish to monitor the training losses or weights and biases to improve the model performance. This can be visualized using TensorBoard.
TensorBoard is TensorFlow’s visualization toolkit. It provides various functionalities to plot/display various aspects of a machine learning pipeline.
In this article, we will cover the basics of TensorBoard, and see how to visualize some of the essentials in various machine learning tasks.
First things first, we need to see how to import and launch TensorBoard using command line/notebook. We load the TensorBoard notebook extension using this magic command:
%load_ext tensorboard
Launch TensorBoard through the command line or within a notebook. In notebooks, use the %tensorboard line magic. On the command line, run the same command without "%".
%tensorboard --logdir <log_directory>
We will see what a log directory is and what significance it holds in the coming sections.
A model should be generic and must make accurate predictions on unseen data as well. For this, we monitor the losses and other metrics on validation (or dev) data and ensure that the losses don’t differ much. This can be done in a more insightful manner by visualizing these metrics over the training time.
Aforementioned is the TensorBoard’s interactive visualization of Train (orange) and Val (blue) losses and accuracies respectively (top to bottom). It can be clearly observed that the train and val losses have been decreasing over the course of training (horizontal axis is epochs), which gives an intuition that the model is achieving generalization as well as low bias. Read bias-variance tradeoff.
So, how do we do this?
When training a model using the keras API, we create a tensorboard callback which ensures that the metrics are logged in the specified directories.
When defining training loops manually, we need to log the losses manually. The tf.summary API facilitates this. First, we define a file writer for the specified log directories (separate for train and test), then with this file writer instance, we log the quantities as scalars after every epoch. Hence, we can observe trends in these metrics over the course of training, step by step.
In the previous section, while creating a tensorboard callback, we set the histogram_freq parameter to 1.
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
The Histograms and Distributions dashboards allow us to visualize the distribution of a tensor over time. Using these, we can monitor the weights, biases, activations, and more.
histogram_freq tells tensorflow to compute the distributions after every epoch. If it is set to 0, then it won’t plot any distributions.
Above is a Histogram plot of the distribution of weights of a dense (fully connected) layer from a model. To the left, is the offset view, while to the right, is the overlay view of the same distribution.
Observe that the distribution is similar to a normal distribution.
In the offset view, we have a 3-dimensional figure:
The horizontal axis corresponds to the values of the weights; basically the range of values taken by the weights of this layer. In this case, [-0.6 to 0.6].The vertical axis corresponds to the distribution. Intuitively, these values show the number of occurrences of the corresponding weight value in the layer, relative to other values. So for example, in the image above, we can see a spike of 685 at the value 0.0626, it superficially means that 685 inputs assume the value 0.0626.The depth axis is essentially the steps (or epochs). We can observe the changes in weight distribution over these steps.
The horizontal axis corresponds to the values of the weights; basically the range of values taken by the weights of this layer. In this case, [-0.6 to 0.6].
The vertical axis corresponds to the distribution. Intuitively, these values show the number of occurrences of the corresponding weight value in the layer, relative to other values. So for example, in the image above, we can see a spike of 685 at the value 0.0626, it superficially means that 685 inputs assume the value 0.0626.
The depth axis is essentially the steps (or epochs). We can observe the changes in weight distribution over these steps.
The overlay view is essentially a 2-dimensional representation of the same histogram. Here, the depth (epoch) axis is squeezed to get a direct comparison between the distributions. We can hover over the plots to have a particular step’s distribution highlighted.
There is another way of visualizing the distribution of weights over the epochs.
This is the Distributions tab of the tensorboard’s interactive toolkit.
The adjacent graph is essentially the plot between the depth and horizontal axis from the previous histogram.
The plot essentially depicts the changes in the range of the weight values of a layer over the epochs.
Using TensorBoard, along with the previously discussed default metrics, we can also log and visualize any custom metrics that we have used in our modeling. In this example, we will visualize the learning rate which would be adjusted by a custom learning rate scheduler.
We first define a custom learning rate schedule. Then we create a callback for it which ensures that the scheduler gets called while defining the learning rate for each epoch. We have logged the learning rate using the tf.summary API which we discussed in the previous sections.
It can be observed that there is a drop in the learning rate wherever we have defined so in our scheduler (See epochs 10, 20, and 50).
We can easily view a conceptual graph of our model’s structure and ensure it matches our intended design using the Graphs dashboard. This also is quite an insight into how TensorFlow construes the model.
Each expanded view shows an extra level of insight into the model’s computation workflow.
In the case of Keras models, no special effort is required except for creating a tensorboard callback to view the graphs dashboard.
However, in a situation where one may have to define a custom computation, the tf.function annotation transforms the python computation function into a TensorFlow graph. This graph can be traced and logged using the tf.summary trace API.
As mentioned in the docs, to use the Summary Trace API:
Use tf.summary.trace_on() immediately before your function call site.
Add profile information (memory, CPU time) to graph by passing profiler=True
With a Summary file writer, call tf.summary.trace_export() to save the log data
Plotting images is like plotting any other scalar value. However, to visualize matplotlib’s plots with TensorBoard, they need to be converted to images first. For this, we have a boilerplate code for converting a pyplot figure to an image:
Here, we take an example of plotting a visually vibing version of confusion matrix using tensorboard.
First, we add a LambdaCallback that would be called after every epoch, which computes the confusion matrix and logs it.
Then, we create a visual of the confusion matrix using pyplot.
This returns a pyplot figure which we convert to an image using the previously discussed boilerplate code.
Embeddings are essentially the representation of a very complex quantity in terms of vectors, wherein the values in the vectors have a notion of belonging with respect to other related quantities. We can use TensorBoard’s Embedding Projector to visualize high-dimensional embeddings. This can be useful for monitoring the keras Embedding layer.
We take an example of sequence classification task to classify imdb movie reviews as positive or negative. We build a simple model with an Embedding layer and a couple of fully connected layers.
For logging and visualizing, we first write the words from the vocabulary to a file.
Then, we checkpoint the trained embedding weights into a variable and then to a file.
Finally, we set up the projector config and project the embeddings:
We saw what TensorBoard is and explored some basic stuff in it. However, there are some special mentions I’d like to make, that I haven’t covered here.
Hyper-parameter TuningThe What-If ToolProfilingFairness Indicators (Beta)
Hyper-parameter Tuning
The What-If Tool
Profiling
Fairness Indicators (Beta)
You can check these out here if interested.
Here is the link to the complete code for the topics covered in this article, feel free to fork it.
The code used in this guide is referred from the following official TensorFlow Documentation: