CNN-Based Single-Image Super-Resolution: A Comparative Study
By: Mohammad Farahmand
Tags: Deep Learning, Convolutional Network, Super Resolution

The following article is a brief report on my Bachelor’s project, a comparative study regarding CNN-Based Single-Image Super-Resolution (SISR) techniques. We’ll begin with a brief look into the first applications of Deep Learning in SISR, followed by a discussion regarding the state-of-the-art CNN-based models for this task. We’ll be comparing the performance of the models on an open-source dataset of satellite images concerning their output quality, speed, and resource demands. You can find the scripts for our experiments as well as the trained super-resolution models here.

Learning in Single-Image Super-Resolution

SISR aims to recover a high-resolution (HR) image from a corresponding low-resolution (LR) version. Given the lower sampling rate of the LR image, smaller details would fall victim to the Nyquist limit, making the problem fundamentally ill-posed, since any combination of amplitudes and phase shifts for the aliased components can yield a valid result for the problem. When you apply a conventional image upsampling technique, such as the bicubic interpolation (BI), to the LR image, the algorithm attempts to fit a specific curve to the 2-d plane of the LR image. It then estimates the HR image pixel values from the interpolated curves.

When the scale factor between the HR image and its LR counterpart surpasses 2, this curve-fitting process results in very smooth images, devoid of sharp edges and sometimes, with “artifacts.” This is because an interpolation technique is not, in fact, adding any new information to the signal. This can be changed with the aid of learning-based techniques. The idea is that if a neural network is provided with enough LR-HR pairs, it will be able to recognize the obscured entities in the LR images and reconstruct them based on the samples it’s seen during training. In other terms, the neural network will learn to “hallucinate” the details lost in the LR image. Deep convolutional neural networks are an obvious candidate for the job, given their outstanding success in image processing problems.

Early Attempts

In their 2014 publication, Chao Dong and his colleagues introduced the first learning-based method for SISR, a deep convolutional neural network that came to be known as SRCNN [1]. Not long after, the same lab published another article, introducing an accelerated version of the same model, unironically called “Fast SRCNN,” or FSRCNN [2], for short, which also improved the performance of the network. FSRCNN makes several modifications to the original SRCNN architecture, enabling it to run in real-time, processing up to 43.5 frames per second with a custom implementation in C++. Real-time processing, unfortunately, has not been a primary goal of the state-of-the-art models. Nevertheless, the structures of these two models have greatly influenced the models that came after. Figure 1 compares the structure of the two. In this figure, Conv(f, n, c) denotes a convolution layer where f, n, and c stand for the filter size, the number of filters, and the number of channels, respectively.