All about Structural Similarity Index (SSIM): Theory + Code in PyTorch
By: Pranjal Datta
Tags: Image Comparison, Pytorch, Computer Vision, Image Processing, Image Similarity

The Theory

SSIM was first introduced in the 2004 IEEE paper, Image Quality Assessment: From Error Visibility to Structural Similarity. The abstract provides a good intuition into the idea behind the system proposed,

Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information.

Summary: The authors make 2 essential points,

Most Image quality assessment techniques rely on quantifying errors between a reference and a sample image. A common metric is to quantify the difference in the values of each of the corresponding pixels between the sample and the reference images (By using, for example, Mean Squared Error).

The Human visual perception system is highly capable of identifying structural information from a scene and hence identifying the differences between the information extracted from a reference and a sample scene. Hence, a metric that replicates this behavior will perform better on tasks that involve differentiating between a sample and a reference image.

The Structural Similarity Index (SSIM) metric extracts 3 key features from an image:

Luminance

Contrast

Structure

The comparison between the two images is performed on the basis of these 3 features.

Fig 1 given below shows the arrangement and flow of the Structural Similarity Measurement system. Signal X and Signal Y refer to the Reference and Sample Images.

Fig 1: The Structural Similarity Measurement System. Source: https://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf

But what does this metric calculate?

This system calculates the Structural Similarity Index between 2 given images which is a value between -1 and +1. A value of +1 indicates that the 2 given images are very similar or the same while a value of -1 indicates the 2 given images are very different. Often these values are adjusted to be in the range [0, 1], where the extremes hold the same meaning.

Now, let’s explore briefly, how these features are represented mathematically, and how they contribute to the final SSIM score.

Luminance: Luminance is measured by averaging over all the pixel values. Its denoted by μ (Mu) and the formula is given below,

where xi is the ith pixel value of the image x. N is the total number of pixel values. Source: https://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf

Contrast: It is measured by taking the standard deviation (square root of variance) of all the pixel values. It is denoted by σ (sigma) and represented by the formula below,

Where x and y are the two images and mu is the mean of the pixel values of the image. Source: https://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf

Structure: The structural comparison is done by using a consolidated formula (more on that later) but in essence, we divide the input signal with its standard deviation so that the result has unit standard deviation which allows for a more robust comparison.

where x is the Input Image

So now we have established the mathematical intuition behind the three parameters. But hold on! We are not yet done with the math, a little bit more. What we lack now, are comparison functions that can compare the two given images on these parameters, and finally, a combination function that combines them all. Here, we define the comparison functions and finally the combination function that yields the similarity index value

Luminance comparison function: It is defined by a function, l(x, y) which is shown below. μ (mu) represents the mean of a given image. x and y are the two images being compared.

where C1 is a constant to ensure stability when the denominator becomes 0. C1 is given by,

Contrast comparison function: It is defined by a function c(x, y) which is shown below. σ denotes the standard deviation of a given image. x and y are the two images being compared.

where C2 is given by,

Structure comparison function: It is defined by the function s(x, y) which is shown below. σ denotes the standard deviation of a given image. x and y are the two images being compared.

where σ(xy) is defined as,

And finally, the SSIM score is given by,

where α > 0, β > 0, γ > 0 denote the relative importance of each of the metrics. To simplify the expression, if we assume, α = β = γ = 1 and C3 = C2/2, we can get,

But there’s a plot twist!

While you would be able to implement SSIM using the above formulas, chances are it won’t be as good as the ready-to-use implementations available, as the authors explain that,

For image quality assessment, it is useful to apply the SSIM index locally rather than globally. First, image statistical features are usually highly spatially nonstationary. Second, image distortions, which may or may not depend on the local image statistics, may also be space-variant. Third, at typical viewing distances, only a local area in the image can be perceived with high resolution by the human observer at one time instance (because of the foveation feature of the HVS [49], [50]). And finally, localized quality measurement can provide a spatially varying quality map of the image, which delivers more information about the quality degradation of the image and may be useful in some applications.

Summary: Instead of applying the above metrics globally (i.e. all over the image at once) it’s better to apply the metrics regionally (i.e. in small sections of the image and taking the mean overall).

This method is often referred to as the Mean Structural Similarity Index.

Due to this change in approach, our formulas also deserve modifications to reflect the same (it should be noted that this approach is more common and will be used to explain the code).

(Note: If the content below seems a bit overwhelming, no worries! If you get the gist of it, then going through the code will give you a much clearer idea.)

The authors use an 11x11 circular-symmetric Gaussian Weighing function (basically, an 11x11 matrix whose values are derived from a gaussian distribution) which moves pixel-by-pixel over the entire image. At each step, the local statistics and SSIM index are calculated within the local window. Since we are now calculating the metrics locally, our formulas are revised as,

Where wi is the gaussian weighting function.

If you found this a bit unintuitive, no worries! It suffices to imagine wi as a multiplicand that is used to calculate the required values with the help of some mathematical tricks.

Once computations are performed all over the image, we simply take the mean of all the local SSIM values and arrive at the global SSIM value.

Finally done with the theory! Now onto the code!