Markov Random Fields And Image Processing | by Arun Jagota | Towards Data Science

This post covers Markov Random Fields and their application to specific problems in image processing. We don’t go deep or formal, just intuitive.
A Markov Random Field is a graph whose nodes model random variables, and whose edges model desired local influences among pairs of them. Local influences propagate globally, leveraging the connectivity of the graph.
Here is an example. Consider an image on a rectangular grid. It’s composed of pixels. Each pixel has a value, denoting its color. Internally a pixel may be represented as a binary variable for black-and-white images, a continuous variable for grey-scale images, and use mixtures of multiple variables (e.g., RGB) for color images. These differences will not matter to us here. As far as we are concerned, a pixel has a value that represents its color.
A natural MRF that models this has a node for each of the pixels. An edge connects two nodes that are adjacent (on the grid). Below is an example MRF on a 3x3 grid. The vertical and horizontal lines are its edges.
o — o — o|   |   |o — o — o|   |   |o — o — o
Okay, the structure of this MRF matches the grid. That sounds like a good thing. Is there a deeper reason for choosing this structure? Yes.
Real-world images tend to be smooth. That is, adjacent pixels tend to have similar colors. Sure, they might also be abrupt transitions such as edges or object boundaries. However, there is generally a lot of smoothness between these transitions.
It is the MRF’s edges that somehow favor that adjacent pixels have similar values. How? First, it will help to introduce some new concepts. Then we will come back to answer this question.
Cliques And Energy Functions
This material is going to be a bit obtuse. You might wonder, why do I need to know all this? The dots will start connecting soon, so bear with me a while.
A clique in a graph is a set of nodes in which every pair is connected by an edge. A clique is maximal if it is not part of a larger one.
Maximal cliques play a crucial role in how an MRF operates. Intuitively each such clique specifies a set of nodes, all of whose values directly influence each other.
In our 2D image example, the maximal cliques are the graph’s edges. No triangles are possible since an edge can connect only adjacent pixels.
Okay, so we’ve characterized which sets of nodes directly influence each other. How do we control the nature of this influence? In our 2D image example, we want adjacent pixels to have similar values.
We control the specifics of the interaction among the nodes’ values in a clique by a so-called energy function of our choice. This function (one per maximal clique) inputs values for all the clique nodes and outputs its energy, a scalar. The energy function we choose should assign low energy to a good combination of values of these nodes. That is, low energy is good.
The global energy for a particular assignment of values to its nodes is the sum of all the maximal cliques’ energies in the graph.
Smoothness-favoring Energy Function. In our image MRF, what energy function would deliver smoothness? Consider |xi-xj|. Here i and j denote adjacent pixels, with xi and xj denoting their values. This energy function prefers that xi and xj have similar values. As a result, global energy will be minimized by an image that is maximally smooth.
Note that, under-the-hood, xi, and xj, might have multiple dimensions, e.g., RGB for color. We don’t care so long as we can meaningfully calculate a distance between the two colors, which is what |xi-xj| is.
Well, that’s nice, but what good is a maximally smooth image? All pixels would have the same color.
Below we see some specific useful image processing problems in which smoothness, hence this MRF, plays an important role.
Image Denoising
Consider this example.
Look at the white dots (stars). Let’s say someone would like to think of them as noise (even though they are not). With a click of a button, the person can ask this image to be denoised. The denoiser’s job is to identify these white dots as the most likely culprits and blacken them.
Why would we expect the MRF to be able to detect these white dots as noise? For the sake of discussion, assume that each white dot is a single pixel. A dot’s pixel is white, and all its neighbors are black. Moreover, black and white are very different-looking colors. The MRF doesn’t like this disharmony. That is, global energy would decrease significantly if we made the spot’s color black. This reasoning applies to all the white dots.
As to any pixel in the bright red sphere, it has at least some neighbors with a similar color. So drastically changing the color of such a pixel will not reduce the energy.
A Richer MRF for Image Denoising
This MRF has two sheets of nodes. The nodes in both sheets represent the pixels, as laid out on the grid. The edges of the first sheet are determined by the grid, as before. There are no edges among nodes in the second sheet. An edge connects the two nodes in the two sheets that represent the same pixel.
Below is an example MRF on a 1X3 grid.
o — o — o      sheet one|   |   |o   o   o      sheet two
We’ll denote the nodes in sheet one as Y and those in sheet two as X. We’ll set X’s values from the image to be denoised. These values stay frozen. The values in Y represent the denoised image. These values change as the denoising process evolves.
We’ll use the same energy function as before. What is the effect of this on the global energy? It tries to find a denoised image that is both close to the input image and smooth(er) than it.
Image Restoration
Here we are given an image that is both noisy (i.e. some of its pixels have incorrect values) and incomplete (i.e., some of its pixels don’t have a value). The two-sheet MRF is a good fit for solving this problem as well. As before, we’ll set X’s values from the input image. Except that pixels in the input image that are missing values are left free.
Image Segmentation
Here we’d like to segment the image into maximally homogenous regions. Homogeneity means that some property remains similar in the entire area. The property might be color. Or texture. Or a combination. Maximally means that different regions, especially adjacent ones, are dissimilar. Sound like clustering? Yes, in a 2D setting on image properties: color, etc.
A particular critical case of segmentation is to split the image into foreground and background. In our sphere example, the sphere would be the foreground, and the black sky (including the stars) the background.
In general, even such binary segmentation is not easy to do. Unlike the simple background in our example (black sky), others may have varying textures or colors. Plus, the image may contain multiple objects. In fact, when this is the case, how is the foreground even defined?
An effective way to both simplify and disambiguate the problem is called interactive segmentation. The user draws a bounding box around the desired object to be segmented. The algorithm takes this box as a rough initial segmentation and tries to improve upon it.
Take the sky example modified so that there are two distinct spheres in it. Think of them like the sun and the moon. The user might put the bounding box around the moon. So both the sun and the sky would be in the background.
Architecture and Energy Function
We’ll use the two-sheet MRF structure here as well. As before, sheet X will contain the image. Sheet Y will capture its segmentation. Sheet Y’s nodes are binary-valued, with zero denoting background and one foreground. The edges in Y will form the same grid structure as before. The edges connecting X to Y will also be the same.
We’ll use a different energy function tailored to this problem. Why? It seems reasonable to impose substantial homogeneity on the foreground and weaker (if any) on the background. How can we make this happen? By choosing an energy function that propagates a 1-value with more ‘force’ than a 0-value to neighboring nodes.
We’ll illustrate this algorithm and energy function on this, especially simple example. (Albeit, not good-looking.)
______ __|___ ____| ____ ______
This example is a 1D binary image. The horizontal line segments are runs of black pixels of varying lengths. These are punctuated by runs of white pixels. The vertical bars demarcate the bounding box.
Now to the illustration. The initial assignment of Y given the bounding box is
Y           ________X ______ __|___ ____| ____ ______
Note that the white pixel in the interior of the box in X is already black in Y. It is as if we were using the prior belief that the interior of the bounding box specified by the user should all be deemed as being in the foreground unless there is substantive evidence to the contrary.
From this initial Y, the algorithm will extend the bounding box two pixels to the left. Why? Because each of the components of the energy function will like this extension. Why? Because the smoothness component likes to propagate a 1-value to its neighbors. And it so happens that the actual pixels in the image at those points are also 1-valued (black). So the algorithm will stop at the Y depicted below.
Y        ___________X ______ __|___ ____| ____ ______
Higher-Order Models
Can we enrich our model without sacrificing its simplicity and elegance? Here we touch on how? For more extensive treatment, see [1].
One approach is to extend the neighborhood of a pixel, for example, by adding diagonal connections, or connecting two pixels if they are within a specific distance d. For the latter, we might use Euclidean distance or Manhattan distance.
If we are going to extend the neighborhood of a pixel, we might introduce weights on the edges decaying with the distance between the pixels they connect.
A second approach is to have nodes represent not individual pixels but square patches of pixels. The value of a patch would be whatever we want it to be. For instance, the average intensity over the intensities of the pixels in the patch. An edge will connect the nodes representing two patches if the patches are sufficiently close. A weight could also be attached to an edge based on the distance between the patches.
Further Reading
Models for gridsGibbs Fields & Markov Random Fields 1 Gibbs FieldsMarkov Random Fields and Gibbs Sampling for Image Denoising
Models for grids
Gibbs Fields & Markov Random Fields 1 Gibbs Fields
Markov Random Fields and Gibbs Sampling for Image Denoising