Title: RESIDUAL NON-LOCAL ATTENTION NETWORKS FOR IMAGE RESTORATION

Abstract: In this paper, we propose a residual non-local attention network for high-quality image restoration. Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial-and channel-wise features. To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts. Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block. The trunk branch is used to extract hierarchical features. Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions. The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map. Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network. Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution. Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

1. INTRODUCTION:

Image restoration aims to recover high-quality (HQ) images from their corrupted low-quality (LQ) observations and plays a fundamental role in various high-level vision tasks. It is a typical ill-posed problem due to the irreversible nature of the image degradation process. Some most widely studied image restoration tasks include image denoising, demosaicing, and compression artifacts reduction. By distinctively modelling the restoration process from LQ observations to HQ objectives, i.e., without assumption for a specific restoration task when modelling, these tasks can be uniformly addressed in the same framework. Recently, deep convolutional neural network (CNN) has shown extraordinary capability of modelling various vision problems, ranging from low-level (e.g., image denoising (Zhang et al., 2017a) , compression artifacts reduction (Dong et al., 2015) , and image super-resolution (Kim et al., 2016; Lai et al., 2017; Tai et al., 2017; Lim et al., 2017; Zhang et al., 2018a; Haris et al., 2018; Wang et al., 2018c; Zhang et al., 2018b; Wang et al., 2018b) ) to high-level (e.g., image recognition (He et al., 2016) ) vision applications.

Stacked denoising auto-encoder (Vincent et al., 2008) is one of the best well-known CNN based model for image restoration. Dong et al. proposed SRCNN (Dong et al., 2014) for image superresolution and ARCNN (Dong et al., 2015) for image compression artifacts reduction. Both SRCNN and ARCNN achieved superior performance against previous works. By introducing residual learning to ease the training difficulty for deeper network, Zhang et al. proposed DnCNN (Zhang et al., 2017a) for image denoising and compression artifacts reduction. The denoiser prior was lately introduced in IRCNN (Zhang et al., 2017b) for fast image restoration. Mao et al. proposed a very deep fully convolutional encoding-decoding framework with symmetric skip connections for image restoration (Mao et al., 2016) . Tai et al. later proposed a very deep end-to-end persistent memory network (MemNet) for image restoration (Tai et al., 2017) and achieved promising results. These CNN based methods have demonstrated the great ability of CNN for image restoration tasks.

However, there are mainly three issues in the existing CNN based methods above. First, the receptive field size of these networks is relatively small. Most of them extract features in a local way with convolutional operation, which fails to capture the long-range dependencies between pixels in the whole image. A larger receptive field size allows to make better use of training inputs and more context information. This would be very helpful to capture the latent degradation model of LQ images, especially when the images suffer from heavy corruptions. Second, distinctive ability of these networks is also limited. Let's take image denoising as an example. For a noisy image, the noise may appear in both the plain and textural regions. Noise removal would be easier in the plain area than that in the textural one. It is desired to make the denoising model focus on textual area more. However, most previous denoising methods neglect to consider different contents in the noisy input and treat them equally. This would result in over-smoothed outputs and some textural details would also fail to be recovered. Third, all channel-wise features are treated equally in those networks. This naive treatment lacks flexibility in dealing with different types of information (e.g., low-and high-frequency information). For a set of features, some contain more information related to HQ image and the others may contain more information related to corruptions. The interdependencies among channels should be considered for more accurate image restoration.

To address the above issues, we propose the very deep residual non-local attention networks (RNAN) for high-quality image restoration. We design residual local and non-local attention blocks as the basic building modules for the very deep network. Each attention block consists of trunk and mask branches. We introduce residual block (He et al., 2016; Lim et al., 2017) for trunk branch and extract hierarchical features. For mask branch, we conduct feature downscaling and upscaling with largestride convolution and deconvolution to enlarge receptive field size. Furthermore, we incorporate non-local block in the mask branch to obtain residual non-local mixed attention. We apply RNAN for various restoration tasks, including image denoising, demosaicing, and compression artifacts reduction. Extensive experiments show that our proposed RNAN achieves state-of-the-art results compared with other recent leading methods in all tasks. To the best of our knowledge, this is the first time to consider residual non-local attention for image restoration problems.

The main contributions of this work are three-fold: 

2. RELATED WORK:

Non-local prior. As a classical filtering algorithm, non-local means (Buades et al., 2005) is computed as weighted mean of all pixels of an image. Such operation allows distant pixels to contribute to the response of a position at a time. It was lately introduced in BM3D (Dabov et al., 2007b) Attention mechanisms. Generally, attention can be viewed as a guidance to bias the allocation of available processing resources towards the most informative components of an input (Hu et al., 2017) . Recently, tentative works have been proposed to apply attention into deep neural networks (Wang et al., 2017; Hu et al., 2017 ). It's usually combined with a gating function (e.g., sigmoid) to rescale the feature maps. Wang et al. (2017) proposed residual attention network for image classification with a trunk-and-mask attention mechanism. Hu et al. (2017) proposed squeezeand-excitation (SE) block to model channel-wise relationships to obtain significant performance improvement for image classification. In all, these works mainly aim to guide the network pay more attention to the regions of interested. However, few works have been proposed to investigate the effect of attention for image restoration tasks. Here, we want to enhances the network with distinguished power for noise and image content.

Image restoration architectures. Stacked denoising auto-encoder (Vincent et al., 2008) The first and last convolutional layers are shallow feature extractor and reconstruction layer respectively. We propose residual local and non-local attention blocks to extract hierarchical attentionaware features. In addition to making the main network learn degradation components, we further concentrate on more challenging areas by using local and non-local attention. We only incorporate residual non-local attention block in low-level and high-level feature space. This is mainly because a few non-local modules can well offer non-local ability to the network for image restoration.

Then RNAN is optimized with loss function. Several loss functions have been investigated, such as L 2 (Mao et al., 2016; Zhang et al., 2017a; Tai et al., 2017; Zhang et al., 2017b) , L 1 (Lim et al., 2017; Zhang et al., 2018c) , perceptual and adversarial losses (Ledig et al., 2017) . To show the effectiveness of our RNAN, we choose to optimize the same loss function (e.g., L 2 loss function) as previous works. Given a training set

I i L , I i H N i=1

, which contains N low-quality inputs and their high-quality counterparts. The goal of training RNAN is to minimize the L 2 loss function

EQUATION

where • 2 denotes l 2 norm. As detailed in Section 4, we use the same loss function as that in other compared methods. Such choice makes it clearer and more fair to see the effectiveness of our proposed RNAN. Then we give more details to residual local and non-local attention blocks. 

RB RB RB q q RB RB t Conv ReLU Conv RB RB m RB RB 2m RB RB m Subscale Upscale Conv Sigmoid NLB Trunk RB (NL) Mask

3.2. RESIDUAL NON-LOCAL ATTENTION BLOCK:

Our residual non-local attention network is constructed by stacking several residual local and nonlocal attention blocks shown in Figure 2 . Each attention block is divided into two parts: q residual blocks (RBs) in the beginning and end of attention block. Two branches in the middle part: trunk branch and mask branch. For non-local attention block, we incorporate non-local block (NLB) in the mask branch, resulting non-local attention. Then we give more details to those components.

3.2.1. TRUNK BRANCH:

As shown in Figure 2 , the trunk branch includes t residual blocks (RBs). Different from the original residual block in ResNet (He et al., 2016) , we adopt the simplified RB from (Lim et al., 2017) .

The simplified RB (labelled with blue dashed) only consists of two convolutional layers and one ReLU (Nair & Hinton, 2010) , omitting unnecessary components, such as maxpooling and batch normalization (Ioffe & Szegedy, 2015) layers. We find that such simplified RB not only contributes to image super-resolution (Lim et al., 2017) , but also helps to construct very deep network for other image restoration tasks.

Feature maps from trunk branch of different depths serve as hierarchical features. If attention mechanism is not considered, the proposed network would become a simplified ResNet. With mask branch, we can take channel and spatial attention to adaptively rescale hierarchical features. Then we give more details about local and non-local attention.

3.2.2. MASK BRANCH:

As labelled with red dashed in Figure 2 , the mask branches used in our network include local and non-local ones. Here, we mainly focus on local mask branch, which can become a non-local one by using non-local block (NLB, labelled with green dashed arrow).

The key point in mask branch is how to grasp information of larger scope, namely larger receptive field size, so that it's possible to obtain more sophisticated attention map. One possible solution is to perform maxpooling several times, as used in (Wang et al., 2017) for image classification. However, more pixel-level accurate results are desired in image restoration. Maxpooling would lose lots of details of the image, resulting in bad performance. To alleviate such drawbacks, we choose to use large-stride convolution and deconvolution to enlarge receptive field size. Another way is considering non-local information across the whole inputs, which will be discussed in the next subsection.

From the input, large-stride (stride ≥ 2) convolutional layer increases the receptive field size after m RBs. After additional 2m RBs, the downscaled feature maps are then expanded by a deconvolutional denotes element-wise addition.

layer (also known as transposed convolutional layer). The upscaled features are further forwarded through m RBs and one 1 × 1 convolutional layer. Then a sigmoid layer normalizes the output values, ranging in [0, 1]. Although the receptive field size of the mask branch is much larger than that of the trunk branch, it cannot cover the whole features at a time. This can be achieved by using non-local block (NLB), resulting in non-local mixed attention.

3.2.3. NON-LOCAL MIXED ATTENTION:

As discussed above, convolution operation processes one local neighbourhood at a time. In order to obtain better attention maps, here we seek to take all the positions into consideration at a time. Inspired by classical non-local means method (Buades et al., 2005) and non-local neural networks (Wang et al., 2018a) , we incorporate non-local block (NLB) into the mask branch to obtain non-local mixed attention (shown in Figure 3 ). The non-local operation can be defined as

EQUATION

where i is the output feature position index and j is the index that enumerates all possible positions.

x and y are the input and output of non-local operation. The pairwise function f (x i , x j ) computes relationship between x i and x j . The function g(x j ) computes a representation of the input at the position j.

As shown in Figure 3 , we use embedded Gaussian function to evaluate the pairwise relationship

EQUATION

where W u and W v are weight matrices. As investigated in (Wang et al., 2018a) , there are several versions of f , such as Gaussian function, dot product similarity, and feature concatenation. We also consider a linear embedding for g: g(x j ) = W g x j with weight matrix W g . Then the output z at position i of non-local block (NLB) is calculated as

EQUATION

where W z is a weight matrix. For a given i, ∀j f (x i , x j )/ ∀j f (x i , x j ) in Eq. 3 becomes the softmax computation along dimension j. The residual connection allows us to insert the NLB into pretrained networks (Wang et al., 2018a ) by initializing W z as zero.

With non-local and local attention computation, feature maps in the mask branch are finally mapped by sigmoid function

EQUATION

where i ranges over spatial positions and c ranges over feature channel positions. Such simple sigmoid operation is applied to each channel and spatial position, resulting mixed attention (Wang et al., 2017) . As a result, the mask branch with non-local block can produce non-local mixed attention. However, simple multiplication between features from trunk and mask branches is not powerful enough or proper to form very deep trainable network. We propose residual non-local attention learning to solve those problems.

3.3. RESIDUAL NON-LOCAL ATTENTION LEARNING:

How to train deep image restoration network with non-local mixed attention remains unclear. Here we only consider the trunk and mask branches, and residual connection with them (Figure 2 ). We focus on obtaining non-local attention information from the input feature x. It should be noted that one form of attention residual learning was proposed in Wang et al. (2017) , whose formulation is

H RN A (x) = H trunk (x)(H mask (x) + 1).

(7) We find that this form of attention learning is not suitable for image restoration tasks. This is mainly because Eq. 7 is more suitable for high-level vision tasks (e.g., image classification), where low-level features are not preserved too much. However, low-level features are more important for image restoration. As a result, we propose a simple yet more suitable residual attention learning method by introducing input feature x directly. We compute its output H RN A (x) as

EQUATION

where H trunk (x) and H mask (x) denote the functions of trunk and mask branches respectively. Such residual learning tends to preserve more low-level features and allows us to form very deep networks for high-quality image restoration tasks with stronger representation ability.

3.4 IMPLEMENTATION DETAILS Now, we specify the implementation details of our proposed RNAN. We use 10 residual local and non-local attention blocks (2 non-local one). In each residual (non-)local block, we set q, t, m = 2, 2, 1. We set 3×3 as the size of all convolutional layers except for those in non-local block and convolutional layer before sigmoid function, where the kernel size is 1×1. Features in RBs have 64 filters, except for that in the non-local block (see Figure 3 ), where C = 32. In each training batch, 16 low-quality (LQ) patches with the size of 48 × 48 are extracted as inputs. Our model is trained by ADAM optimizer (Kingma & Ba, 2014) with β 1 = 0.9, β 2 = 0.999, and = 10 -8 . The initial learning rate is set to 10 -4 and then decreases to half every 2 × 10 5 iterations of back-propagation. We use PyTorch (Paszke et al., 2017) to implement our models with a Titan Xp GPU.

4. EXPERIMENTS:

We apply our proposed RNAN to three classical image restoration tasks: image denoising, demosaicing, and compression artifacts reduction. For image denoising and demosaicing, we follow the same setting as IRCNN (Zhang et al., 2017b) . For image compression artifacts reduction, we follow the same setting as ARCNN (Dong et al., 2015) . We use 800 training images in DIV2K (Timofte et al., 2017; Agustsson & Timofte, 2017) to train all of our models. For each task, we use commonly used dataset for testing and report PSNR and/or SSIM (Wang et al., 2004) to evaluate the results of each method. More results are shown in Appendix A. Non-local Mixed Attention. In cases 1, all the mask branches and non-local blocks are removed.

In case 4, we enable non-local mixed attention with same block number as in case 1. The positive effect of non-local mixed attention is demonstrated by its obvious performance improvement.

Mask branch. We also learn that mask branch contributes to performance improvement, no matter non-local blocks are used (cases 3 and 4) or not (cases 1 and 2). This's mainly because mask branch provides informative attention to the network, gaining better representational ability.

Non-local block. Non-local block also contributes to the network ability obviously, no matter mask branches are used (cases 2 and 4) or not (cases 1 and 3). With non-local information from low-level and high-level features, RNAN performs better image restoration.

Block Number. When comparing cases 2, 4, and 7, we learn that more non-local blocks achieve better results. However, the introduction of non-local block consumes much time. So we use 2 non-local blocks by considering low-and high-level features. When RNAB number is fixed in cases 5 and 7 or cases 6 and 8, performance also benefits from more RABs. 

4.2. COLOR AND GRAY IMAGE DENOISING:

We compare our RNAN with state-of-the-art denoising methods: BM3D (Dabov et al., 2007b) , CBM3D (Dabov et al., 2007a) , TNRD (Chen & Pock, 2017) , RED (Mao et al., 2016) , DnCNN (Zhang et al., 2017a) , MemNet (Tai et al., 2017) , IRCNN (Zhang et al., 2017b) , and FFD-Net (Zhang et al., 2017c) . Kodak24 (http://r0k.us/graphics/kodak/), BSD68 (Martin et al., 2001) , and Urban100 (Huang et al., 2015) are used for color and gray-scale image denoising. AWGN noises of different levels (e.g., 10, 30, 50, and 70) are added to clean images.

Quantitative results are shown in Tables 2 and 3 . As we can see that our proposed RNAN achieves the best results on all the datasets with all noise levels. Our proposed non-local attention covers the information from the whole image, which should be effective for heavy image denoising. To demonstrate this analysis, we take noise level σ = 70 as an example. We can see that our proposed RNAN achieves 0.48, 0.30, and 1.06 dB PSNR gains over the second best method FFDNet. This comparison strongly shows the effectiveness of our proposed non-local mixed attention.

We also show visual results in Figures 4 and 5 . With the learned non-local mixed attention, RNAN treats different image parts distinctively, alleviating over-smoothing artifacts obviously. Visual results are shown in Figure 6 . Although IRCNN can remove mosaicing effect greatly, there're still some artifacts in its results (e.g., blocking artifacts in 'img 026'). However, RNAN recovers more faithful color and alliciates blocking artifacts.

4.4. IMAGE COMPRESSION ARTIFACTS REDUCTION:

We further apply our RNAN to reduce image compression artifacts. We compare our RNAN with SA-DCT (Foi et al., 2007) , ARCNN (Dong et al., 2015) , TNRD (Chen & Pock, 2017) , and DnCNN (Zhang et al., 2017a) . We apply the standard JPEG compression scheme to obtain the compressed images by following (Dong et al., 2015) . Four JPEG quality settings q = 10, 20, 30, 40 are used in Matlab JPEG encoder. Here, we only focus on the restoration of Y channel (in YCbCr space) to keep fair comparison with other methods. We use the same datasets LIVE1 (Sheikh et al., 2005) and Classic5 (Foi et al., 2007) in ARCNN and report PSNR/SSIM values in Table 5 . As we can see, our RNAN achieves the best PSNR and SSIM values on LIVE1 and Classic5 with all JPEG qualities.

We further shown visual comparisons in Figure 7 . We provide comparisons under very low image quality (q=10). The blocking artifacts can be removed to some degree, but ARCNN, TNRD, and DnCNN would also over-smooth some structures. RNAN obtains more details with consistent structures by considering non-local mixed attention. 

4.5. IMAGE SUPER-RESOLUTION:

We further compare our RNAN with state-of-the-art SR methods: EDSR (Lim et al., 2017) , SR-MDNF (Zhang et al., 2018a) , D-DBPN (Haris et al., 2018) , and RCAN (Zhang et al., 2018b) . Similar to (Lim et al., 2017; Zhang et al., 2018c) , we also introduce self-ensemble strategy to further improve our RNAN and denote the self-ensembled one as RNAN+.

As shown in Table 6 , our RNAN+ achieves the second best performance among benchmark datasets: Set5 (Bevilacqua et al., 2012) , Set14 (Zeyde et al., 2010) , B100 (Martin et al., 2001) , Urban100 (Huang et al., 2015) , and Manga109 (Matsui et al., 2017) . Even without self-ensemble, our RNAN achieves third best results in most cases. Such improvements are notable, because the parameter number of RNAN is 7.5 M, far smaller than 43 M in EDSR and 16 M in RCAN. The network depth of our RNAN (about 120 convolutional layers) is also far shallower than that of RCAN, which has about 400 convolutional layers. It indicates that non-local attention make better use of main network, saving much network parameter.

In Figure 8 , we conduct image SR (×4) with several state-of-the-art methods. We can see that our RNAN obtains better visually pleasing results with finer structures. These comparisons further demonstrate the effectiveness of our proposed RNAN with the usage of non-local mixed attention. 

4.6. PARAMETERS AND RUNNING TIME ANALYSES:

We also compare parameters, running time, and performance based on color image denoising in Table 7 . PSNR values are tested on Urban100 (σ=50). RNAN with 10 blocks achieves the best performance with the highest parameter number, which can be reduced to only 2 blocks and obtains second best performance. Here, we report running time for reference, because the time is related to implementation platform and code. Here, we retrain our RNAN on small training sets for 5 tasks. In Table 8 , for each task, we only refer the second best method from our main paper to compare. For training data, we use BSD400 (Martin et al., 2001) for color/gray-scale image denoising and demosaicing. We use 91 images in Yang et al. (2008) 

A.2 VISUAL RESULTS:

Color and Gray Image Denoising. We show color and gray-scale image denoising comparisons in Figures 9 and 10 respectively. We can see that our RNAN recovers shaper edges. Unlike most of other methods, which over-smooth some details (e.g., tiny lines), RNAN can reduce noise and maintain more details. With the learned non-local mixed attention, RNAN treats different image parts distinctively, alleviating over-smoothing artifacts obviously.

Image Compression Artifacts Reduction (CAR).:

In Figure 11 , we provide comparisons under very low image quality (q=10). The blocking artifacts can be removed to some degree, but ARCNN, TNRD, and DnCNN would also over-smooth some structures. In contrast, RNAN obatins more details with consistent structures by considering non-local mixed attention.

Image Super-Resolution (SR). In Figure 12 , we conduct image SR (×4) with several state-of-theart methods, such as SRCNN (Dong et al., 2016) , VDSR (Kim et al., 2016) , LapSRN (Lai et al., 2017) , MemNet (Tai et al., 2017) , EDSR (Lim et al., 2017) , SRMDNF (Zhang et al., 2018a) , and D-DBPN (Haris et al., 2018) . We can see that most of compared methods would suffer from distortion or output totally wrong structures. For tiny line, edge structures, or some textures, they fail to recover and introduce blurring artifacts. Instead, our RNAN obtains better visually pleasing results with finer structures. These comparisons further demonstrate the effectiveness of our proposed RNAN with the usage of non-local mixed attention.

:



Figures and Tables:

Figure 1: The framework of our proposed residual non-local attention network for image restoration. 'Conv', 'RNAB', and 'RAB' denote convolutional layer, residual non-local attention block, and residual local attention block respectively. Here, we take image denoising as a task of interest. 3 RESIDUAL NON-LOCAL ATTENTION NETWORK FOR IMAGE RESTORATION 3.1 FRAMEWORK The framework of our proposed residual non-local attention network (RNAN) is shown in Figure 1. Let's denote I L and I H as the low-quality (e.g., noisy, blurred, or compressed images) and highquality images. The reconstructed image I R can be obtained by I R = H RN AN (I L ) , (1) where H RN AN denotes the function of our proposed RNAN. With the usage of global residual learning in pixel space, the main part of our network can concentrate on learning the degradation components (e.g., noisy, blurring, or compressed artifacts).

Figure 2: Residual (non-)local attention block. It mainly consists of trunk branch (labelled with gray dashed) and mask branch (labelled with red dashed). The trunk branch consists of t RBs. The mask branch is used to learning mixed attention maps in channel-and spatial-wise simultaneously.

Figure 3: Non-local block. Red dashed line denotes matrix reshaping. H×W×C means C features with height H and width W. ⊗ denotes matrix multiplication.denotes element-wise addition.

Figure 4: Color image denoising results with noise level σ = 50.

We propose the very deep residual non-local networks for high-quality image restoration.

Table content: <table/>

is one of the most well-known CNN-based image restoration method.(Dong et al., 2015) proposed AR-CNN for image compression artifact reduction with several stacked convolutional layers. With the help of residual learning and batch normalization(Ioffe & Szegedy, 2015),Zhang et al. proposed  DnCNN (Zhang et al., 2017a)  for accurate image restoration and denoiser priors for image restoration in IRCNN(Zhang et al., 2017b). Recently, great progresses have been made in image restoration community, where Timofte et al.(Timofte et al., 2017),Ancuti et al. (Ancuti et al., 2018), and Blau et al.(Blau et al., 2018) lead the main competitions recently and achieved new research status and records. For example, Wang et al.(Wang et al., 2018c)  proposed a fully progressive image SR approach. However, most methods are plain networks and neglect to use non-local information.

Table content: <table><tr><td>Conv</td><td>RNAB</td><td>RAB</td><td>RAB</td><td>RAB</td><td>RNAB</td><td>Conv</td></tr></table>

Ablation study of different components. PSNR values are based on Urban100 (σ=30).

Table content: <table><tr><td>Case Index</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td></tr><tr><td>Mask Branch</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Non-local Block</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>RAB Number</td><td>7</td><td>7</td><td>5</td><td>5</td><td>1</td><td>2</td><td>5</td><td>8</td></tr><tr><td>RNAB Number</td><td>0</td><td>0</td><td>2</td><td>2</td><td>1</td><td>2</td><td>1</td><td>2</td></tr><tr><td>PSNR (dB)</td><td>30.96</td><td>31.17</td><td>31.20</td><td>31.32</td><td>30.78</td><td>30.99</td><td>31.27</td><td>31.50</td></tr><tr><td>4.1 ABLATION STUDY</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="9">We show ablation study in Table 1 to investigate the effects of different components in RNAN.</td></tr></table>

Quantitative results about color image denoising. Best results are highlighted.

Table content: <table><tr><td>Method</td><td>10</td><td colspan="2">Kodak24 30 50</td><td>70</td><td>10</td><td>30</td><td colspan="2">BSD68</td><td>50</td><td>70</td><td>10</td><td colspan="2">Urban100 30 50</td><td>70</td></tr><tr><td>CBM3D</td><td>36.57</td><td>30.89</td><td>28.63</td><td>27.27</td><td>35.91</td><td colspan="2">29.73</td><td colspan="2">27.38</td><td>26.00</td><td>36.00</td><td>30.36</td><td>27.94</td><td>26.31</td></tr><tr><td>TNRD</td><td>34.33</td><td>28.83</td><td>27.17</td><td>24.94</td><td>33.36</td><td colspan="2">27.64</td><td colspan="2">25.96</td><td>23.83</td><td>33.60</td><td>27.40</td><td>25.52</td><td>22.63</td></tr><tr><td>RED</td><td>34.91</td><td>29.71</td><td>27.62</td><td>26.36</td><td>33.89</td><td colspan="2">28.46</td><td colspan="2">26.35</td><td>25.09</td><td>34.59</td><td>29.02</td><td>26.40</td><td>24.74</td></tr><tr><td>DnCNN</td><td>36.98</td><td>31.39</td><td>29.16</td><td>27.64</td><td>36.31</td><td colspan="2">30.40</td><td colspan="2">28.01</td><td>26.56</td><td>36.21</td><td>30.28</td><td>28.16</td><td>26.17</td></tr><tr><td>MemNet</td><td>N/A</td><td>29.67</td><td>27.65</td><td>26.40</td><td>N/A</td><td colspan="2">28.39</td><td colspan="2">26.33</td><td>25.08</td><td>N/A</td><td>28.93</td><td>26.53</td><td>24.93</td></tr><tr><td>IRCNN</td><td>36.70</td><td>31.24</td><td>28.93</td><td>N/A</td><td>36.06</td><td colspan="2">30.22</td><td colspan="2">27.86</td><td>N/A</td><td>35.81</td><td>30.28</td><td>27.69</td><td>N/A</td></tr><tr><td>FFDNet</td><td>36.81</td><td>31.39</td><td>29.10</td><td>27.68</td><td>36.14</td><td colspan="2">30.31</td><td colspan="2">27.96</td><td>26.53</td><td>35.77</td><td>30.53</td><td>28.05</td><td>26.39</td></tr><tr><td>RNAN (ours)</td><td>37.24</td><td>31.86</td><td>29.58</td><td>28.16</td><td>36.43</td><td colspan="2">30.63</td><td colspan="2">28.27</td><td>26.83</td><td>36.59</td><td>31.50</td><td>29.08</td><td>27.45</td></tr></table>

Quantitative results about gray-scale image denoising. Best results are highlighted.

Table content: <table><tr><td>Method</td><td>10</td><td colspan="2">Kodak24 30 50</td><td>70</td><td>10</td><td>30</td><td colspan="2">BSD68</td><td>50</td><td>70</td><td>10</td><td colspan="2">Urban100 30 50</td><td>70</td></tr><tr><td>BM3D</td><td>34.39</td><td>29.13</td><td>26.99</td><td>25.73</td><td>33.31</td><td colspan="2">27.76</td><td colspan="2">25.62</td><td>24.44</td><td>34.47</td><td>28.75</td><td>25.94</td><td>24.27</td></tr><tr><td>TNRD</td><td>34.41</td><td>28.87</td><td>27.20</td><td>24.95</td><td>33.41</td><td colspan="2">27.66</td><td colspan="2">25.97</td><td>23.83</td><td>33.78</td><td>27.49</td><td>25.59</td><td>22.67</td></tr><tr><td>RED</td><td>35.02</td><td>29.77</td><td>27.66</td><td>26.39</td><td>33.99</td><td colspan="2">28.50</td><td colspan="2">26.37</td><td>25.10</td><td>34.91</td><td>29.18</td><td>26.51</td><td>24.82</td></tr><tr><td>DnCNN</td><td>34.90</td><td>29.62</td><td>27.51</td><td>26.08</td><td>33.88</td><td colspan="2">28.36</td><td colspan="2">26.23</td><td>24.90</td><td>34.73</td><td>28.88</td><td>26.28</td><td>24.36</td></tr><tr><td>MemNet</td><td>N/A</td><td>29.72</td><td>27.68</td><td>26.42</td><td>N/A</td><td colspan="2">28.43</td><td colspan="2">26.35</td><td>25.09</td><td>N/A</td><td>29.10</td><td>26.65</td><td>25.01</td></tr><tr><td>IRCNN</td><td>34.76</td><td>29.53</td><td>27.45</td><td>N/A</td><td>33.74</td><td colspan="2">28.26</td><td colspan="2">26.15</td><td>N/A</td><td>34.60</td><td>28.85</td><td>26.24</td><td>N/A</td></tr><tr><td>FFDNet</td><td>34.81</td><td>29.70</td><td>27.63</td><td>26.34</td><td>33.76</td><td colspan="2">28.39</td><td colspan="2">26.29</td><td>25.04</td><td>34.45</td><td>29.03</td><td>26.52</td><td>24.86</td></tr><tr><td>RNAN (ours)</td><td>35.20</td><td>30.04</td><td>27.93</td><td>26.60</td><td>34.04</td><td colspan="2">28.61</td><td colspan="2">26.48</td><td>25.18</td><td>35.52</td><td>30.20</td><td>27.65</td><td>25.89</td></tr></table>

Quantitative results about color image demosaicing. Best results are highlighted. one of the best methods for image demosaicing and limited space, we only compare with IRCNN in Table 4. As we can see, mosaiced images have very poor quality, resulting in very low PSNR and SSIM values. IRCNN can enhance the low-quality images and achieve relatively high values of PSNR and SSIM. Our RNAN can still make significant improvements over IRCNN. Using local and non-local attention, our RNAN can better handle the degradation situation.

Table content: <table><tr><td>Method</td><td colspan="2">McMaster18 PSNR SSIM</td><td colspan="2">Kodak24 PSNR SSIM</td><td colspan="2">BSD68 PSNR SSIM</td><td colspan="2">Urban100 PSNR SSIM</td></tr><tr><td>Mosaiced</td><td>9.17</td><td>0.1674</td><td>8.56</td><td>0.0682</td><td>8.43</td><td>0.0850</td><td>7.48</td><td>0.1195</td></tr><tr><td>IRCNN</td><td>37.47</td><td>0.9615</td><td>40.41</td><td>0.9807</td><td>39.96</td><td>0.9850</td><td>36.64</td><td>0.9743</td></tr><tr><td>RNAN (ours)</td><td>39.71</td><td>0.9725</td><td>43.09</td><td>0.9902</td><td>42.50</td><td>0.9929</td><td>39.75</td><td>0.9848</td></tr><tr><td colspan="2">4.3 IMAGE DEMOSAICING</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="9">Following the same setting in IRCNN (Zhang et al., 2017b), we compare image demosaicing results</td></tr><tr><td colspan="9">with those of IRCNN on McMaster (Zhang et al., 2017b), Kodak24, BSD68, and Urban100. Since</td></tr><tr><td>IRCNN has been</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Quantitative results about compression artifacts reduction. Best results are highlighted.

Table content: <table><tr><td>Dataset</td><td>q</td><td>JPEG PSNR SSIM</td><td>SA-DCT PSNR SSIM</td><td>ARCNN PSNR SSIM</td><td colspan="2">TNRD PSNR SSIM</td><td>DnCNN PSNR SSIM</td><td>RNAN (ours) PSNR SSIM</td></tr><tr><td/><td>10</td><td colspan="7">27.77 0.7905 28.65 0.8093 28.98 0.8217 29.15 0.8111 29.19 0.8123 29.63 0.8239</td></tr><tr><td>LIVE1</td><td>20 30</td><td colspan="7">30.07 0.8683 30.81 0.8781 31.29 0.8871 31.46 0.8769 31.59 0.8802 32.03 0.8877 31.41 0.9000 32.08 0.9078 32.69 0.9166 32.84 0.9059 32.98 0.9090 33.45 0.9149</td></tr><tr><td/><td>40</td><td colspan="3">32.35 0.9173 32.99 0.9240 33.63 0.9306</td><td>N/A</td><td>N/A</td><td colspan="2">33.96 0.9247 34.47 0.9299</td></tr><tr><td/><td>10</td><td colspan="7">27.82 0.7800 28.88 0.8071 29.04 0.8111 29.28 0.7992 29.40 0.8026 29.96 0.8178</td></tr><tr><td>Classic5</td><td>20 30</td><td colspan="7">30.12 0.8541 30.92 0.8663 31.16 0.8694 31.47 0.8576 31.63 0.8610 32.11 0.8693 31.48 0.8844 32.14 0.8914 32.52 0.8967 32.78 0.8837 32.91 0.8861 33.38 0.8924</td></tr><tr><td/><td>40</td><td colspan="3">32.43 0.9011 33.00 0.9055 33.34 0.9101</td><td>N/A</td><td>N/A</td><td colspan="2">33.77 0.9003 34.27 0.9061</td></tr></table>

Quantitative image SR results. Best and second best results are highlighted and underlined

Table content: <table><tr><td>Method</td><td>Scale</td><td>PSNR</td><td>Set5</td><td>SSIM</td><td colspan="2">Set14 PSNR SSIM</td><td colspan="2">B100 PSNR SSIM</td><td colspan="2">Urban100 PSNR SSIM</td><td colspan="2">Manga109 PSNR SSIM</td></tr><tr><td>Bicubic</td><td>×2</td><td>33.66</td><td colspan="2">0.9299</td><td>30.24</td><td>0.8688</td><td>29.56</td><td>0.8431</td><td>26.88</td><td>0.8403</td><td>30.80</td><td>0.9339</td></tr><tr><td>EDSR</td><td>×2</td><td>38.11</td><td colspan="2">0.9602</td><td>33.92</td><td>0.9195</td><td>32.32</td><td>0.9013</td><td>32.93</td><td>0.9351</td><td>39.10</td><td>0.9773</td></tr><tr><td>SRMDNF</td><td>×2</td><td>37.79</td><td colspan="2">0.9601</td><td>33.32</td><td>0.9159</td><td>32.05</td><td>0.8985</td><td>31.33</td><td>0.9204</td><td>38.07</td><td>0.9761</td></tr><tr><td>D-DBPN</td><td>×2</td><td>38.09</td><td colspan="2">0.9600</td><td>33.85</td><td>0.9190</td><td>32.27</td><td>0.9000</td><td>32.55</td><td>0.9324</td><td>38.89</td><td>0.9775</td></tr><tr><td>RCAN</td><td>×2</td><td>38.27</td><td colspan="2">0.9614</td><td>34.12</td><td>0.9216</td><td>32.41</td><td>0.9027</td><td>33.34</td><td>0.9384</td><td>39.44</td><td>0.9786</td></tr><tr><td>RNAN (ours)</td><td>×2</td><td>38.17</td><td colspan="2">0.9611</td><td>33.87</td><td>0.9207</td><td>32.32</td><td>0.9014</td><td>32.73</td><td>0.9340</td><td>39.23</td><td>0.9785</td></tr><tr><td>RNAN+ (ours)</td><td>×2</td><td>38.22</td><td colspan="2">0.9613</td><td>33.97</td><td>0.9216</td><td>32.36</td><td>0.9018</td><td>32.90</td><td>0.9351</td><td>39.41</td><td>0.9789</td></tr><tr><td>Bicubic</td><td>×4</td><td>28.42</td><td colspan="2">0.8104</td><td>26.00</td><td>0.7027</td><td>25.96</td><td>0.6675</td><td>23.14</td><td>0.6577</td><td>24.89</td><td>0.7866</td></tr><tr><td>EDSR</td><td>×4</td><td>32.46</td><td colspan="2">0.8968</td><td>28.80</td><td>0.7876</td><td>27.71</td><td>0.7420</td><td>26.64</td><td>0.8033</td><td>31.02</td><td>0.9148</td></tr><tr><td>SRMDNF</td><td>×4</td><td>31.96</td><td colspan="2">0.8925</td><td>28.35</td><td>0.7787</td><td>27.49</td><td>0.7337</td><td>25.68</td><td>0.7731</td><td>30.09</td><td>0.9024</td></tr><tr><td>D-DBPN</td><td>×4</td><td>32.47</td><td colspan="2">0.8980</td><td>28.82</td><td>0.7860</td><td>27.72</td><td>0.7400</td><td>26.38</td><td>0.7946</td><td>30.91</td><td>0.9137</td></tr><tr><td>RCAN</td><td>×4</td><td>32.63</td><td colspan="2">0.9002</td><td>28.87</td><td>0.7889</td><td>27.77</td><td>0.7436</td><td>26.82</td><td>0.8087</td><td>31.22</td><td>0.9173</td></tr><tr><td>RNAN (ours)</td><td>×4</td><td>32.49</td><td colspan="2">0.8982</td><td>28.83</td><td>0.7878</td><td>27.72</td><td>0.7421</td><td>26.61</td><td>0.8023</td><td>31.09</td><td>0.9149</td></tr><tr><td>RNAN+ (ours)</td><td>×4</td><td>32.56</td><td colspan="2">0.8992</td><td>28.90</td><td>0.7883</td><td>27.77</td><td>0.7424</td><td>26.75</td><td>0.8052</td><td>31.37</td><td>0.9175</td></tr></table>

Parameter and time comparisons. '*' means being applied channel-wise for color images. this paper, we propose residual non-local attention networks for high-quality image restoration. The networks are built by stacking local and non-local attention blocks, which extract local and non-local attention-aware features and consist of trunk and (non-)local mask branches. They're used to extract hierarchical features and adaptively rescale hierarchical features with soft weights.

Table content: <table><tr><td>Methods</td><td>RED  *</td><td>DnCNN</td><td>MemNet  *</td><td>RNAN (1LB+1NLB)</td><td>RNAN (8LBs+2NLBs)</td></tr><tr><td>Parameter Number</td><td>4,131 K</td><td>672K</td><td>677K</td><td>1,494k</td><td>7,409K</td></tr><tr><td>PSNR (dB)</td><td>26.40</td><td>28.16</td><td>26.53</td><td>28.36</td><td>29.15</td></tr><tr><td>Time (s) (Platform)</td><td>58.7 (Caffe)</td><td>16.5 (Matlab+GPU Array)</td><td>239.0 (Caffe)</td><td>2.6 (PyTorch)</td><td>6.5 (PyTorch)</td></tr><tr><td colspan="2">5 CONCLUSIONS</td><td/><td/><td/><td/></tr></table>

and 200 images inMartin et al. (2001) (denoted as SR291) for image compression artifacts reduction and super-resolution. FFDNet used BSD400(Martin et al., 2001), 400 images from ImageNetDeng et al. (2009), and 4,744 images in Waterloo Exploration DatabaseMa et al. (2017). Here, 'BSD400+' is used to denote 'BSD400+ImageNet400+WED4744'. According to Table8, we use the same or even smaller training set for our RNAN and obtain better results for 5 tasks. These experiments demonstrate the effectiveness of our RNAN for general image restoration tasks. Quantitative results about image restoration. Best results are highlighted. Color image denoising. σ denotes noise level. Gray image denoising. σ denotes noise level. Image compression artifacts reduction. q denotes JPEG quality.Task 5: Image super-resolution (SR). Low-resolution (LR) images are obtained by Bicubic downsampling.

Table content: <table><tr><td colspan="5">Task 1: Data Kodak24 Method σ=10 σ=30 σ=50</td><td>σ=70</td><td>σ=10</td><td colspan="3">BSD68 σ=30 σ=50</td><td>σ=70</td><td>σ=10</td><td colspan="2">Urban100 σ=30 σ=50</td><td>σ=70</td></tr><tr><td>BSD400+</td><td>FFDNet</td><td>36.81</td><td>31.39</td><td>29.10</td><td>27.68</td><td>36.14</td><td colspan="2">30.31</td><td>27.96</td><td>26.53</td><td>35.77</td><td>30.53</td><td>28.05</td><td>26.39</td></tr><tr><td>BSD400</td><td>RNAN</td><td>37.16</td><td>31.81</td><td>29.55</td><td>28.15</td><td>36.60</td><td colspan="2">30.73</td><td>28.35</td><td>26.88</td><td>36.39</td><td>30.90</td><td>28.65</td><td>27.11</td></tr><tr><td colspan="5">Task 2: Data Kodak24 Method σ=10 σ=30 σ=50</td><td>σ=70</td><td>σ=10</td><td colspan="3">BSD68 σ=30 σ=50</td><td>σ=70</td><td>σ=10</td><td colspan="2">Urban100 σ=30 σ=50</td><td>σ=70</td></tr><tr><td>BSD400+</td><td>FFDNet</td><td>34.81</td><td>29.70</td><td>27.63</td><td>26.34</td><td>33.76</td><td colspan="2">28.39</td><td>26.29</td><td>25.04</td><td>34.45</td><td>29.03</td><td>26.52</td><td>24.86</td></tr><tr><td>BSD400</td><td>RNAN</td><td>35.15</td><td>29.87</td><td>27.92</td><td>26.50</td><td>34.18</td><td colspan="2">28.72</td><td>26.61</td><td>25.34</td><td>35.08</td><td>29.82</td><td>27.30</td><td>25.57</td></tr><tr><td colspan="8">Task 3: Image demosaicing Task 4: Data Method Kodak24 BSD68</td><td>Data</td><td/><td>Method</td><td colspan="2">LIVE1 q=20 q=40</td><td colspan="2">Classic5 q=20 q=40</td></tr><tr><td>BSD400</td><td>IRCNN</td><td colspan="2">40.41</td><td colspan="2">39.96</td><td/><td colspan="2">SR291</td><td/><td>DnCNN</td><td>31.59</td><td>33.96</td><td>31.63</td><td>33.77</td></tr><tr><td>BSD400</td><td>RNAN</td><td colspan="2">42.86</td><td colspan="2">42.61</td><td/><td colspan="2">SR291</td><td/><td>RNAN</td><td>31.87</td><td>34.32</td><td>31.99</td><td>34.03</td></tr><tr><td>Data</td><td>Method</td><td>×2</td><td>Set5 ×3</td><td>×4</td><td>×2</td><td>Set14 ×3</td><td/><td>×4</td><td>×2</td><td>B100 ×3</td><td>×4</td><td>×2</td><td>Urban100 ×3</td><td>×4</td></tr><tr><td>SR291</td><td>MemNet</td><td>37.78</td><td>34.09</td><td>31.74</td><td>33.28</td><td>30.00</td><td colspan="2">28.26</td><td>32.08</td><td>28.96</td><td>27.40</td><td>31.31</td><td>27.56</td><td>25.50</td></tr><tr><td>SR291</td><td>RNAN</td><td>37.95</td><td>34.30</td><td>31.93</td><td>33.41</td><td>30.22</td><td colspan="2">28.43</td><td>32.23</td><td>29.09</td><td>27.51</td><td>31.64</td><td>27.84</td><td>25.71</td></tr></table>

Peer review (iclr2019):
 The paper proposes a convolutional neural network architecture that includes blocks for local and non-local attention mechanisms, which are claimed to be responsible for achieving excellent results in four image restoration applications. # Results The strongest point of the paper is that the quantitative and qualitative image restoration results appear to be very good, although they seem almost a bit too good. # Novelty I*m not sure about the novelty of the paper, but I suspect it to be rather incremental. The paper says "To the best of our knowledge, this is the first time to consider residual non-local attention for image restoration problems." Does that mean non-local attention (in a very similar way) has already been used, just not in a residual fashion? If so, that would not constitute much novelty. I have to admit that I*m not familiar with the related work on attention, but I did not understand *why* the results of the proposed method are supposed to be much better than that of previous work. # Clarity I think the paper is not self-contained enough, since it seems to implicitly assume substantial background knowledge on attention mechanisms in CNNs. Furthermore, the introduction of the paper identifies three problems with existing CNNs that I don*t necessarily fully agree with. None of these supposed problems are backed up by (experimental) evidence. I don*t think it is sufficient to just show superior results than previous methods. It is also important to disentangle why the results are better. However, the presented ablation experiments are not very illuminating to me. The attempts at explaining what the novel attention blocks do and why they lead to superior results are very vague to me. Maybe they are understandable in the context of related work, but I found many statements, such as the following, devoid of meaning: - "Without considering the uneven distribution of information in the corrupted images, [...]" - "However, in this paper, we mainly focus on learning non-local attention to better guide feature extraction in trunk branch." - "We only incorporate residual non-local attention block in low-level and high-level feature space. This is mainly because a few non-local modules can well offer non-local ability to the network for image restoration." - "The key point in mask branch is how to grasp information of larger scope, namely larger receptive field size, so that it’s possible to obtain more sophisticated attention map." # Experiments - The experimental results are the best part of the paper. However, it would*ve been nice to include some qualitative results in the main paper. - The proposed RNAN model is trained on a big dataset (800 images with ~2 million pixels each). Are the competing methods trained on datasets of similar size? If not, this could be a major reason for improved performance of RNAN over competing methods. At least in the appendix, RNAN and FFDNet are compared more fairly since they are trained with the same/similar data. - The qualitative examples in the appendix mostly show close-ups/details of very structured regions (mostly stripy patterns). Please also show some other regions without self-similar structures. # Misc - Residual non-local attention learning (section 3.3) was not clear to me. - The word "trunk" is used without definition or explanation. - Fig. 2 caption is too short, please expand. # Update (2018-11-29) Given the substantial author feedback, I*m willing to raise my score.