Residual Non-local Attention Networks for Image Restoration

1 INTRODUCTION . Image restoration aims to recover high-quality ( HQ ) images from their corrupted low-quality ( LQ ) observations and plays a fundamental role in various high-level vision tasks . It is a typical ill-posed problem due to the irreversible nature of the image degradation process . Some most widely studied image restoration tasks include image denoising , demosaicing , and compression artifacts reduction . By distinctively modelling the restoration process from LQ observations to HQ objectives , i.e. , without assumption for a specific restoration task when modelling , these tasks can be uniformly addressed in the same framework . Recently , deep convolutional neural network ( CNN ) has shown extraordinary capability of modelling various vision problems , ranging from low-level ( e.g. , image denoising ( Zhang et al. , 2017a ) , compression artifacts reduction ( Dong et al. , 2015 ) , and image super-resolution ( Kim et al. , 2016 ; Lai et al. , 2017 ; Tai et al. , 2017 ; Lim et al. , 2017 ; Zhang et al. , 2018a ; Haris et al. , 2018 ; Wang et al. , 2018c ; Zhang et al. , 2018b ; Wang et al. , 2018b ) ) to high-level ( e.g. , image recognition ( He et al. , 2016 ) ) vision applications . Stacked denoising auto-encoder ( Vincent et al. , 2008 ) is one of the best well-known CNN based model for image restoration . Dong et al . proposed SRCNN ( Dong et al. , 2014 ) for image superresolution and ARCNN ( Dong et al. , 2015 ) for image compression artifacts reduction . Both SRCNN and ARCNN achieved superior performance against previous works . By introducing residual learning to ease the training difficulty for deeper network , Zhang et al . proposed DnCNN ( Zhang et al. , 2017a ) for image denoising and compression artifacts reduction . The denoiser prior was lately introduced in IRCNN ( Zhang et al. , 2017b ) for fast image restoration . Mao et al . proposed a very deep fully convolutional encoding-decoding framework with symmetric skip connections for image restoration ( Mao et al. , 2016 ) . Tai et al . later proposed a very deep end-to-end persistent memory network ( MemNet ) for image restoration ( Tai et al. , 2017 ) and achieved promising results . These CNN based methods have demonstrated the great ability of CNN for image restoration tasks . However , there are mainly three issues in the existing CNN based methods above . First , the receptive field size of these networks is relatively small . Most of them extract features in a local way with convolutional operation , which fails to capture the long-range dependencies between pixels in the whole image . A larger receptive field size allows to make better use of training inputs and more context information . This would be very helpful to capture the latent degradation model of LQ images , especially when the images suffer from heavy corruptions . Second , distinctive ability of these networks is also limited . Let ’ s take image denoising as an example . For a noisy image , the noise may appear in both the plain and textural regions . Noise removal would be easier in the plain area than that in the textural one . It is desired to make the denoising model focus on textual area more . However , most previous denoising methods neglect to consider different contents in the noisy input and treat them equally . This would result in over-smoothed outputs and some textural details would also fail to be recovered . Third , all channel-wise features are treated equally in those networks . This naive treatment lacks flexibility in dealing with different types of information ( e.g. , low- and high-frequency information ) . For a set of features , some contain more information related to HQ image and the others may contain more information related to corruptions . The interdependencies among channels should be considered for more accurate image restoration . To address the above issues , we propose the very deep residual non-local attention networks ( RNAN ) for high-quality image restoration . We design residual local and non-local attention blocks as the basic building modules for the very deep network . Each attention block consists of trunk and mask branches . We introduce residual block ( He et al. , 2016 ; Lim et al. , 2017 ) for trunk branch and extract hierarchical features . For mask branch , we conduct feature downscaling and upscaling with largestride convolution and deconvolution to enlarge receptive field size . Furthermore , we incorporate non-local block in the mask branch to obtain residual non-local mixed attention . We apply RNAN for various restoration tasks , including image denoising , demosaicing , and compression artifacts reduction . Extensive experiments show that our proposed RNAN achieves state-of-the-art results compared with other recent leading methods in all tasks . To the best of our knowledge , this is the first time to consider residual non-local attention for image restoration problems . The main contributions of this work are three-fold : • We propose the very deep residual non-local networks for high-quality image restoration . The powerful networks are based on our proposed residual local and non-local attention blocks , which consist of trunk and mask branches . The network obtains non-local mixed attention with non-local block in the mask branch . Such attention mechanis helps to learn local and non-local information from the hierarchical features . • We propose residual non-local attention learning to train very deep networks by preserving more low-level features , being more suitable for image restoration . Using non-local lowlevel and high-level attention from the very deep network , we can pursue better network representational ability and finally obtain high-quality image restoration results . • We demonstrate with extensive experiments that our RNAN is powerful for various image restoration tasks . RNAN achieves superior results over leading methods for image denoising , demosaicing , compression artifacts reduction , and super-resolution . In addition , RNAN achieves superior performance with moderate model size and performs very fast . 2 RELATED WORK . Non-local prior . As a classical filtering algorithm , non-local means ( Buades et al. , 2005 ) is computed as weighted mean of all pixels of an image . Such operation allows distant pixels to contribute to the response of a position at a time . It was lately introduced in BM3D ( Dabov et al. , 2007b ) for image denoising . Recently , Wang et al . ( 2018a ) proposed non-local neural network by incorporating non-local operations in deep neural network for video classification . We can see that those methods mainly introduce non-local information in the trunk pipeline . Liu et al . ( 2018 ) proposed non-local recurrent network for image restoration . However , in this paper , we mainly focus on learning nonlocal attention to better guide feature extraction in trunk branch . Attention mechanisms . Generally , attention can be viewed as a guidance to bias the allocation of available processing resources towards the most informative components of an input ( Hu et al. , 2017 ) . Recently , tentative works have been proposed to apply attention into deep neural networks ( Wang et al. , 2017 ; Hu et al. , 2017 ) . It ’ s usually combined with a gating function ( e.g. , sigmoid ) to rescale the feature maps . Wang et al . ( 2017 ) proposed residual attention network for image classification with a trunk-and-mask attention mechanism . Hu et al . ( 2017 ) proposed squeezeand-excitation ( SE ) block to model channel-wise relationships to obtain significant performance improvement for image classification . In all , these works mainly aim to guide the network pay more attention to the regions of interested . However , few works have been proposed to investigate the effect of attention for image restoration tasks . Here , we want to enhances the network with distinguished power for noise and image content . Image restoration architectures . Stacked denoising auto-encoder ( Vincent et al. , 2008 ) is one of the most well-known CNN-based image restoration method . ( Dong et al. , 2015 ) proposed ARCNN for image compression artifact reduction with several stacked convolutional layers . With the help of residual learning and batch normalization ( Ioffe & Szegedy , 2015 ) , Zhang et al . proposed DnCNN ( Zhang et al. , 2017a ) for accurate image restoration and denoiser priors for image restoration in IRCNN ( Zhang et al. , 2017b ) . Recently , great progresses have been made in image restoration community , where Timofte et al . ( Timofte et al. , 2017 ) , Ancuti et al . ( Ancuti et al. , 2018 ) , and Blau et al . ( Blau et al. , 2018 ) lead the main competitions recently and achieved new research status and records . For example , Wang et al . ( Wang et al. , 2018c ) proposed a fully progressive image SR approach . However , most methods are plain networks and neglect to use non-local information . 3 RESIDUAL NON-LOCAL ATTENTION NETWORK FOR IMAGE RESTORATION . 3.1 FRAMEWORK . The framework of our proposed residual non-local attention network ( RNAN ) is shown in Figure 1 . Let ’ s denote IL and IH as the low-quality ( e.g. , noisy , blurred , or compressed images ) and highquality images . The reconstructed image IR can be obtained by IR = HRNAN ( IL ) , ( 1 ) where HRNAN denotes the function of our proposed RNAN . With the usage of global residual learning in pixel space , the main part of our network can concentrate on learning the degradation components ( e.g. , noisy , blurring , or compressed artifacts ) . The first and last convolutional layers are shallow feature extractor and reconstruction layer respectively . We propose residual local and non-local attention blocks to extract hierarchical attentionaware features . In addition to making the main network learn degradation components , we further concentrate on more challenging areas by using local and non-local attention . We only incorporate residual non-local attention block in low-level and high-level feature space . This is mainly because a few non-local modules can well offer non-local ability to the network for image restoration . Then RNAN is optimized with loss function . Several loss functions have been investigated , such as L2 ( Mao et al. , 2016 ; Zhang et al. , 2017a ; Tai et al. , 2017 ; Zhang et al. , 2017b ) , L1 ( Lim et al. , 2017 ; Zhang et al. , 2018c ) , perceptual and adversarial losses ( Ledig et al. , 2017 ) . To show the effectiveness of our RNAN , we choose to optimize the same loss function ( e.g. , L2 loss function ) as previous works . Given a training set { IiL , I i H } N i=1 , which contains N low-quality inputs and their high-quality counterparts . The goal of training RNAN is to minimize the L2 loss function L ( Θ ) = 1 N N∑ i=1 ∥∥HRNAN ( IiL ) − IiH∥∥2 , ( 2 ) where ‖·‖2 denotes l2 norm . As detailed in Section 4 , we use the same loss function as that in other compared methods . Such choice makes it clearer and more fair to see the effectiveness of our proposed RNAN . Then we give more details to residual local and non-local attention blocks . 3.2 RESIDUAL NON-LOCAL ATTENTION BLOCK . Our residual non-local attention network is constructed by stacking several residual local and nonlocal attention blocks shown in Figure 2 . Each attention block is divided into two parts : q residual blocks ( RBs ) in the beginning and end of attention block . Two branches in the middle part : trunk branch and mask branch . For non-local attention block , we incorporate non-local block ( NLB ) in the mask branch , resulting non-local attention . Then we give more details to those components .

Summary: This paper proposes a residual non-local attention network for image restoration. Specifically, the proposed method has local and non-local attention blocks to extract features which capture long-range dependencies. The local and non-local blocks consist of trunk branch and (non-) local mask branch. The proposed method is evaluated on image denoising, demosaicing, compression artifacts reduction, and super-resolution.