Metrics to Evaluate your Semantic Segmentation Model
By: Ekin Tiu
Tags: Machine Learning, Evaluation, Metrics, Deep Learning, Segmentation

Metrics to Evaluate your Semantic Segmentation Model

How do you know your segmentation model performs well? Find out here.

Illustration of IoU and Dice Coefficient.

Semantic segmentation. My absolute favorite task. I would make a deep learning model, have it all nice and trained… but wait. How do I know my model is performing well? In other words, what are the most common metrics for semantic segmentation? Here’s a clear cut guide to the essential metrics that you need to know to ensure your model performs well. I have also included Keras implementations below.

If you want to learn more about Semantic Segmentation with Deep Learning, check out this Medium article by George Seif.

Contents:

Pixel Accuracy Intersection-Over-Union (Jaccard Index) Dice Coefficient (F1 Score) Conclusion, Notes, Summary

1. Pixel Accuracy

Pixel accuracy is perhaps the easiest to understand conceptually. It is the percent of pixels in your image that are classified correctly.

While it is easy to understand, it is in no way the best metric.

At first glance, it might be difficult to see the issue with this metric. To expose this metric for what it really is, here’s a scenario: Let’s say you ran the following image(Left)through your segmentation model. The image on the right is the ground truth, or annotation (what the model is supposed to segment). In this case, our model is trying to segment ships in a satellite image.

Image from Vlad Shmyhlo in article: Image Segmentation: Kaggle experience (Part 1 of 2) in TDS

You see that your segmentation accuracy is 95%. That’s awesome! Let’s see how your segmentation looks like!

Not exactly what you were hoping for, huh. Is there something wrong with our calculation? Nope. It’s exactly right. It’s just that one class was 95% of the original image. So if the model classifies all pixels as that class, 95% of pixels are classified accurately while the other 5% are not. As a result, although your accuracy is a whopping 95%, your model is returning a completely useless prediction. This is meant to illustrate that high pixel accuracy doesn’t always imply superior segmentation ability.

This issue is called class imbalance. When our classes are extremely imbalanced, it means that a class or some classes dominate the image, while some other classes make up only a small portion of the image. Unfortunately, class imbalance is prevalent in many real world data sets, so it can’t be ignored. Therefore, I present to you two alternative metrics that are better at dealing with this issue:

2. Intersection-Over-Union (IoU, Jaccard Index)

The Intersection-Over-Union (IoU), also known as the Jaccard Index, is one of the most commonly used metrics in semantic segmentation… and for good reason. The IoU is a very straightforward metric that’s extremely effective.

IoU calculation visualized. Source: Wikipedia

Before reading the following statement, take a look at the image to the left. Simply put, the IoU is the area of overlap between the predicted segmentation and the ground truth divided by the area of union between the predicted segmentation and the ground truth, as shown on the image to the left. This metric ranges from 0–1 (0–100%) with 0 signifying no overlap and 1 signifying perfectly overlapping segmentation.

For binary (two classes) or multi-class segmentation, the mean IoU of the image is calculated by taking the IoU of each class and averaging them. (It’s implemented slightly differently in code).

Now let’s try to understand why this metric is better than pixel accuracy by using the same scenario as section 1. For the sake of simplicity, let’s assume all the ships (colored boxes) are part of the same class.

But wait, what exactly is overlap and union in our context? The image above shows a pretty clear picture, but I found it a bit difficult to understand in the context of predicted vs. ground truth because they aren’t necessarily overlapping like the image depicts above. Let’s take a look at the predicted segmentation and the ground truth side-by-side.