Compression artifact removal using multi-scale reshuffling convolutional network

In this work, we aim to build a high efficient deep network to remove artifact of compressed image. The degeneracy and small receptive field problem might be caused by reducing the computational cost of convolutional network via general approaches, such as pooling features to low resolution space, reducing the width of network and using small size convolutional kernel. For the reasons, we propose a multi-scale reshuffling network to efficiently reduce the compression artifact of compressed images without degeneracy. We firstly present a reshuffling network which includes a downscaling and a upscaling reshuffling operation. The downscaling reshuffling periodically rearranges high resolution to low resolution space without any information loss. The upscaling reshuffling is the reverse transformation of downscaling reshuffling, which allows us reconstructing high resolution image from the low resolution features. A densely connected structure is applied to efficiently extract features without degeneracy. The low resolution representations is gradually recovered to the higher resolution spaces which leads to a multi-scale structure. Results show the effectiveness of the proposed method. 1. Degeneracy problem and computational cost of convolutional network In image restoration scenario, including compression artifact removal, we need to predict the latent high quality image from a degraded input image. We find that general methods for computational cost reduction of convolutional neural network (CNN) might lead network to suffer from the degeneracy. Degeneracy of network might force it learning a lot of redundancy, which results in an inefficient network, to obtain an accurate prediction. 1.1. Degeneracy problem Orhan [8] explained that the difficulty of training deep network is mainly due to the degeneracy problem. We ascribe degeneracy to the singularity of linear transformation, rectified linear unit (ReLU) and pooling layer, as these might cause too much input information to be lost and therefore decrease the performance of deep network. Zhang et al. [11] shown that most convolutional or pooling layers would cause information loss through an information theoretic view. Singularity would lead consistent deactivation of nodes and decrease the effective representation dimension of deep features. ReLU ignores the negative activations. Downscaling pooling layers directly reduce the spatial size of feature maps. The shallow layers should be wide enough to keep sufficient effective information, which leads to a redundant structure. He et al. [5] successfully trained very deep network using skip connections. Orhan et al. [9] explained that skip connections can eliminate singularities. However, skip connection can not prevent pooling layer from information loss and is not efficient enough in image restoration tasks. 1.2. Computational cost of convolutional layer We assume that convolutional layer operates with stride 1 and pads the input to produce output feature maps with the same spatial size of input. The convolutional layer is parameterized by a convolution kernel F of shape K×L×M×N , where K and L is the width and height of convolution kernel, M is the number of input channels and N is the number of output channels. A Convolutional layer takes as input X with shape of I × J × M and outputs A with shape of I×J ×N , where I and J is the width and height of feature maps. The output feature maps of convolutional layer are computed as: