Exploring confusion matrix evolution on tensorboard | by Sushrut Ashtikar | Towards Data Science

Tensorboard is the best tool for visualizing many metrics while training and validating a neural network. In most of the case, we need to look for more details like how a model is performing on validation data. Sometimes training and validation loss and accuracy are not enough, we need to figure out the performance of validation data. One of the ways is to visualize using a confusion matrix.
In the field of machine learning and specifically the problem of statistical classification, a confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one (in unsupervised learning it is usually called a matching matrix). Each row of the matrix represents the instances in a predicted class while each column represents the instances in an actual class (or vice versa). The name stems from the fact that it makes it easy to see if the system is confusing two classes (i.e. commonly mislabeling one as another).
I won’t be digging deep into coding, I will highlight only important code portions which shows how to set up and implement a custom callback in tensorboard in python. In case if you want to view complete code you can check my repository, which I have added a link at the bottom of this story.
I assume you have already built and compiled a Keras sequential model.
Defining a function to plot cm
def plot_confusion_matrix(cm, class_names):    """    Returns a matplotlib figure containing the plotted confusion matrix.        Args:       cm (array, shape = [n, n]): a confusion matrix of integer classes       class_names (array, shape = [n]): String names of the integer classes    """        figure = plt.figure(figsize=(8, 8))    plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)    plt.title("Confusion matrix")    plt.colorbar()    tick_marks = np.arange(len(class_names))    plt.xticks(tick_marks, class_names, rotation=45)    plt.yticks(tick_marks, class_names)        # Normalize the confusion matrix.    cm = np.around(cm.astype('float') / cm.sum(axis=1)[:, np.newaxis], decimals=2)        # Use white text if squares are dark; otherwise black.    threshold = cm.max() / 2.        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):        color = "white" if cm[i, j] > threshold else "black"        plt.text(j, i, cm[i, j], horizontalalignment="center", color=color)            plt.tight_layout()    plt.ylabel('True label')    plt.xlabel('Predicted label')    return figure
We are now ready to train the CNN and regularly log the confusion matrix during the process. Using the below code, you will create a Keras TensorBoard callback to log basic metrics.
logdir = "logs/image/" + datetime.now().strftime("%Y%m%d-%H%M%S")tensorboard_callback = keras.callbacks.TensorBoard(log_dir = logdir, histogram_freq = 1)file_writer_cm = tf.summary.create_file_writer(logdir + '/cm')
Unfortunately, the Matplotlib file format cannot be logged as an image, but the PNG file format can be logged. So, we will create a helper function that takes a Matplotlib figure and converts it to PNG format so it can be written.
def plot_to_image(figure):    """    Converts the matplotlib plot specified by 'figure' to a PNG image and    returns it. The supplied figure is closed and inaccessible after this call.    """        buf = io.BytesIO()        # Use plt.savefig to save the plot to a PNG in memory.    plt.savefig(buf, format='png')        # Closing the figure prevents it from being displayed directly inside    # the notebook.    plt.close(figure)    buf.seek(0)        # Use tf.image.decode_png to convert the PNG buffer    # to a TF image. Make sure you use 4 channels.    image = tf.image.decode_png(buf.getvalue(), channels=4)        # Use tf.expand_dims to add the batch dimension    image = tf.expand_dims(image, 0)        return image
We will define a function that calculates the confusion matrix.
def log_confusion_matrix(epoch, logs):        # Use the model to predict the values from the test_images.    test_pred_raw = model.predict(test_images)        test_pred = np.argmax(test_pred_raw, axis=1)        # Calculate the confusion matrix using sklearn.metrics    cm = sklearn.metrics.confusion_matrix(test_labels, test_pred)        figure = plot_confusion_matrix(cm, class_names=class_names)    cm_image = plot_to_image(figure)        # Log the confusion matrix as an image summary.    with file_writer_cm.as_default():        tf.summary.image("Confusion Matrix", cm_image, step=epoch)
We will set up tensorboard callback to log confusion matrix on epoch end
cm_callback = keras.callbacks.LambdaCallback(on_epoch_end=log_confusion_matrix)
We need to specify values as a list to callbacks parameter in model.fit to specify Keras to use our custom callback functions while training.
# Start TensorBoard.%tensorboard --logdir logs/image# Train the classifier.model.fit(train_images,          train_labels,          epochs=5,          verbose=0, # Suppress chatty output          callbacks=[tensorboard_callback, cm_callback],          validation_data=(test_images, test_labels))
The tensorboard server runs on port 6006 by default, in case you want to specify any other port you need to specify as an arg to tensorboard command.Jupyter users can simply type%load_ext tensorboard in the first cell and run before importing libraries, this will load tensorboard inside Jupyter notebook.
Wanna try out yourself
github.com
Refer to my Github repo link. I have trained a CNN classifier on the Fashion Mnist dataset and setup a confusion matrix. The above output is from my tensorboard server.
Deep Dive into TensorBoard: Tutorial With Exampleshttps://neptune.ai/blog/tensorboard-tutorial
Deep Dive into TensorBoard: Tutorial With Exampleshttps://neptune.ai/blog/tensorboard-tutorial