Title: -LOCAL ENHANCED NET-WORKS

Abstract: Existing neural networks proposed for low-level image processing tasks are usually implemented by stacking convolution layers with limited kernel size. Every convolution layer merely involves in context information from a small local neighborhood. More contextual features can be explored as more convolution layers are adopted. However it is difficult and costly to take full advantage of long-range dependencies. We employ non-local operation to build up connection between every pixel and all remain pixels. Moreover a novel Pyramid Non-local Block is devised to robustly estimate pairwise similarity coefficients between different scales of content patterns. Considering computation burden and memory consumption, we exploit embedding feature maps with coarser resolution to represent content patterns with larger spatial scale. Through elaborately combining the pyramid non-local blocks and dilated residual blocks, we set up a Pyramid Non-local Enhanced Network for edge-preserving image smoothing. It achieves state-of-the-art performance in imitating three classical image smoothing algorithms. Additionally, the pyramid non-local block can be directly incorporated into existing convolution neural networks for other image processing tasks. We integrate it into two state-ofthe-art methods for image denoising and single image super-resolution respectively, achieving consistently improved performance.

1. INTRODUCTION:

Recently impressive progress has been achieved in low-level computer vision tasks as the development of convolution neural networks (CNN), e.g. edge-preserving image smoothing (Li et al., 2016; Fan et al., 2017; Zhu et al., 2019) , image denoising (Mao et al., 2016; Zhang et al., 2017; 2018) and image super-resolution (Tai et al., 2017a; b; Zhang et al., 2018) . This paper focuses on devising a novel pyramid non-local block oriented for effectively and efficiently mining long-range dependencies in low-level image processing tasks.

Typical convolution layers operate on a small local neighborhood without considering non-local contextual information. One common practice for capturing long-range dependencies is to enlarge the receptive field by stacking large number of convolution layers (Kim et al., 2016a) or dilating convolution layers (Yu et al., 2017) . However, it is difficult to deliver information between distant positions in such a manner (Hochreiter & Schmidhuber, 1997) .

Discovering similar patterns (including small texture patches and large object parts) residing in natural images is valuable to low-level image processing (Buades et al., 2005; Dabov et al., 2007; Mairal et al., 2009) . Dependencies on similar patterns can be used for recognizing out real region boundary in image smoothing and recovering contaminated or missing details in image restoration tasks. Inspired from this concept, a few literatures (Lefkimmiatis, 2017; Wang et al., 2018; Liu et al., 2018; Li et al., 2018a) introduce non-local operation into deep networks in form of self-similarity strategy. Wang et al. (2018) presents a non-local component in video classification, which is placed after high-level, sub-sampled feature maps. Huge computational cost and memory consumption of their non-local operations hinder its adaption to low-level computer vision tasks, where high-resolution feature maps are demanded to produce appealing pixel-wise outputs. NLRN (Liu et al., 2018) and NLEDN (Li et al., 2018a) apply non-local operation for image restoration and rain removal, respectively. One fatal drawback of this kind of methods is that the estimation of self-similarity is confined in a neighborhood of tens of pixels. Besides it is a very common phenomenon that the same type of patterns appear to own various spatial scales, which has not been taken into consideration in existing non-local operations.

To settle the above issues, we propose a pyramid structure, named Pyramid Non-local Block (PNB), to perform non-local operation with a cost-effective amount of computation load. It utilizes a query feature map with full resolution and a pyramid of reference feature maps to robustly estimate similarities between different scales of patterns. Accordingly, a pyramid of embedding feature maps are extracted to enhance the input feature map with the help of estimated similarity matrices. To ensure acceptable computation burden and memory consumption, resolutions of the reference and embedding feature maps are downscaled. Through intervening pyramid non-local blocks and dilated residual blocks (Yu et al., 2017) , we set up Pyramid Non-Local Enhanced Networks for edge-preserving image smoothing. It achieves the state-of-the-art performance in imitating various classical image smoothing filters.

In addition, efficient computation allows our proposed pyramid non-local block to be incorporated into deep CNN-based methods for pixel-level image processing tasks. We demonstrate the effectiveness of PNB on two classical tasks, image denoising and single image super-resolution (SISR). Two state-of-the-art models, RDN (Zhang et al., 2018) and MemNet (Tai et al., 2017b) , are adopted as the baseline models. The PNB acts as a critical component to exploit long-range dependencies. Performance improvements over baseline have been consistently obtained thanks to the adoption of pyramid non-local blocks.

2. RELATED WORK:

Deep Learning Based Image Processing Edge-preserving image smoothing aims to preserve significant image structures while smoothing out trivial details. Because of the strong feature learning ability, deep neural networks attracts a lot of attention in this field. The pioneering work (Xu et al., 2015) employs a three-layer CNN to predict a gradient map which is subsequently used to guide the smoothing procedure. Recurrent network is adopted to efficiently propagate spatial contextual information across pixels (Liu et al., 2016) . Joint filtering (Li et al., 2016) and edge map (Fan et al., 2017) are dedicated to using extra guidances for image smoothing. Meanwhile, deep neural networks have been proposed for image denoising and super-resolution (Kim et al., 2016a; Zhang et al., 2017) . Residual connections (Lim et al., 2017) and dense connections (Tong et al., 2017; Zhang et al., 2018) are added to alleviate the vanishing-gradient problem. Features at different scales are fused to get efficacious information (Li et al., 2018b) . Other representative methods include Huang et al. (2015) ; Shi et al. (2016) ; Tai et al. (2017a; b); Ahn et al. (2018) ; Hui et al. (2018) . We do not elaborate due to limited space.

Non-local Context Information. Image non-local self-similarity has been widely exploited in many non-local methods for image restoration, such as BM3D (Dabov et al., 2007) and WNNM (Gu et al., 2014) . Recently a few studies attempt to incorporate non-local operations into deep neural networks for capturing long-range dependencies. Wang et al. (2018) presented trainable non-local neural networks in video classification. However, the computation complexity of their non-local operation grows dramatically as size of the input feature map increases. Liu et al. (2018) and Li et al. (2018a) incorporated non-local module into RNN architecture and encoder-decoder architecture for image restoration and image de-raining, respectively. Nevertheless, the measurement of self-similarity is restricted within a small neighborhood.

Our method differs from existing models proposed for low-level image processing in two aspects. Firstly, our method can robustly measure similarities between different scales of content patterns as we adopt a pyramid structure for the non-local operation. Secondly, computation burden and memory consumption is greatly relieved because the spatial resolutions of reference features in the non-local operation are downscaled.

3. METHOD:

Non-local correlations between patterns are paramount for edge-preserving image smoothing. Longrange contextual features are beneficial to suppress textures inside objects (object parts) while identifying out real region boundaries in image smoothing. We propose a deep pyramid non-local enhanced network (PNEN), which introduces pyramid non-local block (PNB) to explicitly mine long-range dependencies. The overall architecture is illustrated in Figure 1 . The pyramid non-local block (PNB) is carefully designed to involve in correlation with distant pixels when inferring every pixel, as well as guaranteeing computation efficiency in terms of complexity and tolerant memory consumption. Besides, dilated residual blocks (DRB) (Yu et al., 2017) are introduced to enlarge receptive field for taking advantage of full structural and textural information in the image. In the following sections, each component of the proposed architecture will be elaborated with more details.

Input Entry … … Exit ⊕ " # " $ " % Output PNB DRB PNB DRB PNB DRB & # & $ & % & '

3.1. ENTRY AND EXIT NETWORK:

Define the input color image as X with size of h × w × 3. h and w represents the height and width of the input image respectively. Our proposed PNEN employs one convolution layer as the entry net to transform X into a feature map F 0 which is a h × w × c tensor. Formally, we have

EQUATION

where the F entry (•, •) denotes the convolution operations in the entry net and W entry represents related convolution parameters. Subsequently, M blocks, each of which is consisting of a pyramid non-local block and a dilated residual block, are stacked to induce deep features. We define the feature produced by the m-th block as F m . We have

EQUATION

where F P N B (•, •) and F DRB (•, •) indicates the calculation procedure inside the pyramid non-local block and dilated residual block which will be elaborated in Section 3.2 and 3.3 respectively. W m DRB and W m P N B represent their parameters correspondingly. Inspired by MemNet (Tai et al., 2017b) , features generated by all blocks {F m |m = 1, • • • , M } are accumulated to generate residual images using the exit network. The residual image produced with F m is defined as

EQUATION

where F exit (•, •) denotes the convolution operations in the exit network and W m exit represents the related parameters. The exit network contains three convolutional layers. Suppose Y m = X + R m . The final reconstructed images is computed by

EQUATION

where 

{w m |m = 1, • • • , M }

L = 1 hwc ( G -Y 2 + M m=1 G -Y m 2 ).

(5)

3.2. PYRAMID NON-LOCAL BLOCK (PNB):

Let F ∈ R h×w×c denotes the input feature activation map. Here h, w and c represents the height, width and channel, respectively. A general formulation of non-local operation (Wang et al., 2018) can be defined as

EQUATION

⊗ ⊗ !"×!" !×"×$ softmax %: (1, $, 1) +: (1, ,, 1) -: (1, ,, 1) !"×, ⊗ ⊗ !"×!" !×"×$ softmax X Y ,×!"

!"×$ ":

(1, $, 1)

$:

(1, ,, 1) %:

(1, ,,

EQUATION

EQUATION

":

(1, ), 1)

$:

(1, ,,

⊕ & -: (1, ,, 1) where F is the enhanced feature representation. M(F) ∈ R hw×hw is the self-similarity matrix, where each element M(F) i,j indicates the similarity between pixel i and j. G(X) ∈ R hw×n gives rise to a n-dimensional pixel-wise embedding. D(X) is a diagonal matrix for normalization purpose. T (•) is the transformation function which converts the embedded features back into the original space of input feature. In this way, the feature representation is non-locally enhanced through considering all positions of the feature map. One instantiation (Wang et al., 2018) can be constructed by taking the linear embedded Gaussian kernel as the distance metric to compute correlation matrix M, and linear function to compute G:

+ / : (2, ,, 2) $ / : (2, ), 2) ⊗ ⊗ ⊗ !"×! ' " ' !"×! ( " ( ! ( " ( ×) ! ' " ' ×) C ⊕ ψ: (1, 2, 1) !×"×3) ) & + 4 : (4, ,, 4) $ 4 : (4, ), 4) softmax ⊗ ⊗ ⊗ ! * " * ×) !"×! * " * softmax softmax $ 6 : (2 3 , ), 2 3 ) + 6 : (2 3 , ,, 2 3 ) ) & & ψ: (1, 2, 1) + , 7 + , 8 + , 9 (b) Pyramid Non-local Block

EQUATION

The embedding function F emb (F, W) can be implemented by first applying convolutional operation of parameter W on F, and then transforming the result into a 2-dimensional tensor in which each column represents one embedding channel. When calculating M(F), a query and a reference feature with same size of hw × m are generated using convolution kernel W θ and W φ respectively. The output dimension of W g is denoted as n. The diagonal elements of D(F) are obtained through calculating column summation of M(F). T (•) is also implemented with a convolution operation of parameter W ψ to convert the embedding feature back into the original c-dimensional space. All convolutions use kernel size of 1 × 1. An example non-local block is illustrated in Figure 2 (a). The computation complexity and memory occupation of the correlation matrix increases exponentially as the number of pixels grows. For sake of reducing computation burden, previous works (Liu et al., 2018; Li et al., 2018a) utilize a small neighborhood to restrict the range of non-local operation.

In comparison, we propose a novel pyramid non-local block to effectively soften the computation demand while achieving appealing performance in low-level image processing tasks.

At first, we produce one query feature E θ = F emb (F, W θ ). The spatial kernel size and stride of W θ is 1 × 1 and 1 respectively. Then, multi-scale reference features and embedding features are generated with convolutional layers using different kernel sizes and strides. Suppose there are totally S scales. The computation procedure of the non-local operation in the s-th scale is as follows:

EQUATION

EQUATION

where {E s Φ |s = 1, • • • , S} are query features for calculating similarity matrices with respect to E θ and {E s g |s = 1, • • • , S} are embedding features in all scales. The stride of convolutional layers in the s-th scale is set to 2 s . This implies that the number of rows in E s Φ and E s g is reduced to hw/4 s , which greatly reduces the amount of computation for estimating the self-similarity matrix. We adopt larger convolutional kernels to extract reference feature representations of larger content patterns. The pyramidal design is motivated by two reasons: 1) The computation and memory requirement of non-local operations can be controlled within an acceptable load. 2) The robustness of self-similarity estimation is enhanced since the self-similarity is measured at various resolutions.

X !: (1, &, 1) ( ) : (2, &, 2) + ) : (2, ,, 2) ⊗ ⊗ ⊗ -.×-" . " -.×-#.# -#.#×, -" . " ×, C ⊕ ψ: (1, 2, 1) -×.×2 -×.×3, -.×& ⊗ ⊗ -.×-. -×.×, -×.×2 softmax -×.×2 Y ( 4 : (4, &, 4) + 4 : (4, ,, 4) ( 6 : (7, &, S) + 6 : (7, ,, 7) softmax ⊗ ⊗ ⊗ -$ . $ ×, -.×-$ . $ softmax softmax X Y &×-.

-.×, %:

(1, ,, 1)

':

(1, &, 1) (: 

(1, &,

F = F Ψ ({ Ê1 , • • • , ÊS }, W ψ ) + F.

(11) We summarize Eq. ( 9), ( 10) and ( 11) with the function F P N B (•, •), as mentioned in Section 3.1. One characteristic of pyramid non-local block is the flexibility of balancing accuracy and computation resources through adjusting kernel sizes and strides in different scales. An illustration of the pyramid non-local block is shown in Figure 2(b) . We set m = 64, n = 32 and S = 3 in practice.

3.3. DILATED RESIDUAL BLOCK (DRB):

In pixel-wise image processing tasks, high-resolution feature maps are favorable for reconstructing complicate textural details while large receptive field benefits the extraction of features which can grab high-level contextual information. Considering the above issues, we employ dilated convolution (Yu et al., 2017) to rapidly increase the receptive fields without sacrificing spatial resolutions of learned features in our proposed model. As shown in Figure 3 , we devise a dilated residual block with 5 cascaded residual modules which is set up with dilated convolutions. The calculation procedure of the dilated residual block is indicated with the function F DRB (•, •), as mentioned in Section 3.1.

As illustrated in Figure 1 , our model is built up through intervening PNB-s and DRB-s. Every group of consecutive PNB and DRB contributes a feature map for the final prediction.

3.4. DISCUSSION:

The benefits of our proposed pyramid non-local block are three folds: First, the pyramidal strategy adopts convolutions with various kernel sizes to generate embedding features for self-similarity estimation. This improves the robustness of estimating correlation across pattern scales.

Second, existing deep models equipped with non-local modules are implemented via connecting all pairs of pixels in the feature map Wang et al. (2018) or limiting the nonlocal dependencies within a constant neighborhood size Liu et al. (2018) ; Li et al. (2018a) . The former kind of methods merely plug non-local modules after high-level feature maps with small resolution, because the computational complexity and memory usage will grow exponentially as the number of pixels in the feature map increases. The later kind of methods inevitably neglect valuable correlations from pixels outside the fixed neighborhood. We solve the problem ingeniously through embedding the input feature into a query feature map with full resolution and multiple reference feature maps with downscaled resolutions. In such a manner, the computation burden can be greatly relieved when applying our method in low-level pixel-wise image processing tasks.

Third, the pyramid non-local block can be easily incorporated into existing CNN-based models devised for low-level image processing, such as RDN (Zhang et al., 2018) for denoising or MemNet (Tai et al., 2017b) for single image super-resolution. Experiments on various datasets indicate that the pyramid non-local block consistently improves the performance in image smoothing, denoising and super-resolution.

Table 1 : Quantitative comparison in terms of PSNR/SSIM on three smoothing filters. The best performance is marked in bold. Model depth and parameter number are also indicated.

DJF (Li et al., 2016) CEILNet (Fan et al., 2017) ResNet (Zhu et al., 2019 

4.1. DATASET:

The image datasets are from Zhu et al. (2019) . There are 500 images of clear structures and visible details to evaluate the learning apability of deep model in reproducing edge-preserving filters. Following Zhu et al. (2019) , we use 400 images as training set and the remaining 100 images as testing set. We train models to reproduce three representative filters in our experiments, including weighted median filter (r = 10, σ = 50) (Zhang et al., 2014) , L 0 smoothing (λ = 0.02, κ = 2) (Xu et al., 2011) and SD Filter (λ = 15) (Ham et al., 2017) .

Implementation Details Without specification, all convolutional layers have 64 filters with kernel size 3 × 3. We stack three PNB-s and DRB-s consecutively as the feature extractor, resulting in a 37-layer deep network. During the training stage, random horizontal flip and rotation are applied for data augmentation. Training images are split into 96 × 96 patches. The mini-batch size is set to 8. Adam optimizer (Kingma & Ba, 2015) with β 1 = 0.9, β 2 = 0.999, = 10 -8 is used for optimization.

The initial learning rate is set to 5 × 10 -4 and reduced by half when training loss stops decreasing, until it reduces to 10 -4 . It takes around 2 days to train a model on one TITAN Xp GPU. For inference, the model takes 1.2 seconds to process a testing image of 500 × 400 pixels.

4.2. COMPARISON WITH THE STATE-OF-THE-ART:

Recently, piles of CNN-based approaches (Xu et al., 2015; Liu et al., 2016; Li et al., 2016; Fan et al., 2017; Zhu et al., 2019) have been proposed to reproduce edge-preserving smoothing filters. We compare our proposed PNEN against three state-of-the-art methods, including DeepJointFilter (DJF) (Li et al., 2016) , Cascaded Edge and Image Learning Network (CEILNet) (Fan et al., 2017) and Residual Networks (ResNet) (Zhu et al., 2019) . For fair comparison, the networks are re-trained from scratch on the same training dataset as described above. We evaluate the quality of the generated images using two metrics, including Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index (SSIM) (Wang et al., 2004) .

Quantitative results are reported in Table 1 . When reproducing WMF, L 0 smoothing and SD Filter, our proposed method (PNEB) outperforms the second best method ResNet (Zhu et al., 2019 ) by 1.15dB, 1.14dB and 0.98dB in PSNR metric respectively. Higher SSIM performance also indicates that our method can give rise to smoothed images with better structural information. A visual comparison of learning L 0 smoothing filter is provided in Figure 4 . As can be seen in close-ups of the selected patch, the region smoothed by our proposed PNEN appears to be cleaner and flatter than other methods. Our result is closer to the groundtruth image.

As shown in Figure 5 , we visualize the similarity map derived from the last pyramid non-local block at two locations, marked by red and blue respectively. Figure 5 (b), (c) and (d) show the similarity map at 'Scale 2', 'Scale 4' and 'Scale 8', respectively. We can see that the pixels with similar features show high correlation in these maps. Thus, the features are non-locally enhanced by exploiting long-range dependencies. Particularly, for the pixel marked by red, there exist noises when estimating correlation coefficients with similar patterns (left-bottom area) in 'Scale 2' while 'Scale 4' performs well on these patterns. It indicates that the adoption of pyramid non-local operations can benefit the robustness of estimating correlations between different scales of patterns.

4.3. ABLATION STUDY:

To validate the effectiveness and necessity of pyramidal strategy, we exhaustively compare PNB with its variants on learning L0 smoothing filter. The performance changes in terms of PSNR and SSIM are shown in networks with a significant margin. It is noteworthy that the design of PNB is flexible according to the trade-off between computation efficiency and accuracy.

The required computational resources, in terms of floating point operations (FLOPs) and memory consumption, are summarized in Table 3 . The performance is obtained by testing methods on a 96 × 96 patch. We compare our proposed PNB with the non-local block proposed in Wang et al. (2018) . Memory consumption increases dramatically since non-local operation needs to store a large correlation matrix. Our memory requirement is 57% less than that of Wang et al. (2018) . Under the same GPU memory constraint, our method allows larger training patches and larger receptive field for low-level image processing tasks.

5. EXTENSION: EXPERIMENTS ON IMAGE RESTORATION:

Deep convolutional networks are widely used in image restoration tasks (Kim et al., 2016a; b; Zhang et al., 2017; Lim et al., 2017; Tai et al., 2017b; Hui et al., 2018; Zhang et al., 2018) . We adopt two state-of-the-art methods, RDN (Zhang et al., 2018) and MemNet (Tai et al., 2017b) , as the baseline models for image denoising and image super-resolution task, respectively. As discussed in Section 3.4, efficient computation allows our proposed pyramid non-local block be incorporated into these low-level baseline models. The PNB acts as a basic component to exploit non-local image self-similarity. Model architectures and visual comparisons are provided in the appendix.

Image Denoising: Three PNB-s are incorporated into RDN. To demonstrate the effectiveness of PNB, we compare with a variant of RDN by stacking more convolutional layers. The deeper RDN has more parameters than the PNB-enhanced RDN. We follow the training and testing protocols as in Zhang et al. (2018) for fair comparison. 800 images from DIV2K dataset (Timofte et al., 2017) are utilized as training data. We add white Gaussian noise with standard deviation σ = 25 to the original images to synthesize noisy images. Results on three widely used benchmarks, Set12 (Zhang et al., 2017) , BSD68 (Roth & Black, 2009) , Urban 100 (Huang et al., 2015) , are reported in Table 4 . The PNB-enhanced RDN yields better results than original RDN and its variant. The training data are constructed by bicubic downsampling with factor 3, and then upscaled to the original size. Results on four widely used benchmarks, Set5 (Bevilacqua et al., 2012) , Set14 (Zeyde et al., 2010) , BSD100 (Martin et al., 2001) and Urban100 (Huang et al., 2015) , are presented in Table 5 . The PNB-enhanced MemNet yields better results than original MemNet and its variant.

6. CONCLUSION:

In this paper, we present a novel and effective pyramid non-local enhanced network (PNEN) for edge-preserving image smoothing. The proposed pyramid non-local block (PNB) is computationfriendly, which allows it be a plug-and-play component in existing deep methods for low-level image processing tasks. Methods incorporating our proposed pyramid non-local block achieve significantly improved performance in image denoising and image super-resolution. 

C IMAGE SUPER-RESOLUTION: MODELS AND VISUAL COMPARISON:

The architecture of MemNet (Tai et al., 2017b) is shown in Figure 12 (a). They proposed memory block as their basic component. Please refer to Tai et al. (2017b) for more details. As shown in Figure 12 (b), we incorporate three our proposed PNB-s into their network without affecting the dense connections between memory blocks. 

:

EXPERIMENTS IN EDGE-PRESERVING SMOOTHINGEdge-preserving smoothing is a fundamental topic in image processing. It aims at detecting major image structures while neglecting insignificant details, which is critical in many computer vision tasks such as image segmentation and contour detection.

Figures and Tables:

Figure 1: The overall architecture of our proposed pyramid non-local enhanced network (PNEN).

are trainable weights. During the training stage, supervisions are imposed to intermediate predictions Y m -s and final output Y. The loss function is defined as the mean squared error towards groundtruth images G:

Figure2: Architecture of prior non-local block(Wang et al., 2018) and our pyramid non-local block (PNB). In PNB, φ k and g k are implemented with convolutional layers of different strides, while θ is shared. The kernel size k, number of filters f and stride s are indicated as (k, f, s) for each convolution layer.

Figure 4: Visual comparison of learning L0 smoothing filter. There exist unwanted details in previous methods if we examine closely. In contrast, the result generated by our PNEN is closer to the groundtruth.

Figure 5: Correlation map of pyramid non-local operation at different scales. The first row shows the correlation map computed at the red point. The second row shows correlation map computed at the blue point.

Super-Resolution: We add three PNB-s into MemNet. We also compare with a variant of MemNet by stacking more convolutional layers. The deeper MemNet and PNB-enhanced MemNet have similar parameter numbers. We follow the training and testing protocols as in Tai et al. (2017b) for fair comparison. The training sets consist of 291 images where 200 images are BSD train set and other 91 images are from Yang et al. (2010).

Figure 8: Correlation map of pyramid non-local operation at different scales. The first row shows the correlation map computed at the red point. The second row shows correlation map computed at the blue point.

Figure 12: Baseline MemNet architecture, and the enhanced MemNet architecture by incorporating our PNB.

1)

Table content: <table><tr><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td></tr><tr><td>Dilation: 1</td><td/><td/><td/><td>2</td><td/><td/><td>4</td><td/><td/><td>2</td><td/><td/><td>1</td><td/></tr><tr><td colspan="15">Figure 3: Architecture of dilated residual block (DRB). Our proposed DRB contains five Conv-BN-</td></tr><tr><td colspan="12">ReLU-Conv groups, which have dilation factor 1, 2, 4, 2, 1, respectively.</td><td/><td/><td/></tr><tr><td colspan="15">After performing non-local operation in all scales, enhanced embedding features { Ês |s = 1, • • • , S}</td></tr><tr><td colspan="15">are concatenated together, followed by one 1 × 1 convolution layer to generate residual values to F.</td></tr><tr><td colspan="12">Formally, the final output of pyramid non-local block can be achieved by</td><td/><td/><td/></tr></table>

Ablation study on different design choices of pyramid non-local block.

Table content: <table><tr><td/><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td></tr><tr><td>Scale 2</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Scale 4</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Scale 8</td><td/><td/><td/><td/><td/><td/></tr><tr><td>PSNR</td><td>31.95</td><td>32.57</td><td>33.17</td><td>32.87</td><td>33.21</td><td>33.44</td></tr><tr><td colspan="7">SSIM 0.9665 0.9706 0.9708 0.9707 0.9720 0.9741</td></tr></table>

Comparison of computation resource requirements in terms of floating point operations (FLOPs) and memory consumption.

Table content: <table><tr><td/><td>w/o</td><td>w/ Wang et al. (2018)</td><td>w/ our PNB</td></tr><tr><td>FLOPs</td><td>40.6G</td><td>46.2G</td><td>42.5G</td></tr><tr><td>Memory</td><td>1.1GB</td><td>9.6GB</td><td>4.3GB</td></tr><tr><td colspan="2">#Parameters 1334k</td><td>1519k</td><td>1875k</td></tr><tr><td>PSNR</td><td>31.95</td><td>32.40</td><td>33.44</td></tr></table>

The multi-scale pyramid non-local networks outperforms single-scale non-local

Table content: <table/>

Image denoising results.

Table content: <table><tr><td>Aver-</td></tr></table>

SISR results. Average PSNR/SSIM results are reported on Set5, Set14, BSD100 and Urban100 dataset.

Table content: <table><tr><td>Dataset</td><td>MemNet</td><td colspan="2">MemNet(deeper) MemNet(w/ PNB)</td></tr><tr><td>Set5</td><td>34.09/0.9248</td><td>34.12/0.9250</td><td>34.18/0.9267</td></tr><tr><td>Set14</td><td>30.00/0.8350</td><td>30.04/0.8361</td><td>30.12/0.8409</td></tr><tr><td>BSD100</td><td>28.96/0.8001</td><td>28.97/0.8008</td><td>29.02/0.8072</td></tr><tr><td colspan="2">Urban100 27.56/0.8376</td><td>27.65/0.8380</td><td>27.88/0.8474</td></tr><tr><td>#Parmeters</td><td>677k</td><td>1056k</td><td>1047k</td></tr></table>

Peer review (iclr2020):
In this work the authors propose the Pyramid Non-local Block (PNB) as a neural network module to estimate similarity coefficients between parts of an image. The PNB, unlike a convolution, can efficiently compute correlations between spatially distant parts of the input. In the pyramid non-local enhanced networks, the PNB is combined with standard dilated residual blocks. Applications of this new network are shown and evaluated on edge-preserving smoothing, image denoising and image restoration, where in all cases the evaluations show a small but consistent gain in PSRN compared to baseline models. The PNB is based on prior work proposing such non-local blocks, and the main innovation is the introduction of multi-scale processing in the form of a scale pyramid for the reference and embedding feature maps (obtained with strided convolutions with larger kernels). This makes it possible to compute correlations over larger spatial scales without paying the exponential computational cost. The paper cites prior work appropriately and lists the relevant hyperparameters used for training. The examples in the appendices show interesting visual improvements. As is, I believe this paper would be a better match for a more specialized venue (e.g. a computer vision conference). The main shortcomings of the paper are its limited novelty (multiscale pyramids are an old *trick* in computer vision), and the use of baselines that are behind SOTA for the discussed problems -- e.g. Dai el al, *Second-order Attention Network for Single Image Super-Resolution* could be a more current baseline for SISR (instead of MemNet) and Liu et al, *Non-Local Recurrent Network for Image Restoration* for denoising (which the authors cite and briefly discuss in the introduction). Also, in the RDN paper (which the authors used as a baseline for denoising) applications to superresolution are presented, with reported results better than the ones achieved with PNEN in Table 5. I found it surprising that a less well performing method (MemNet) was used as the baseline for the SISR task in the present work. For edge preserving smoothing, where the largest relative gains in evaluation scores are seen in the present work, the baselines and evaluation protocols seem to be less well established than for the remaining two tasks (the original paper using ResNets for this doesn*t use PSNR/SSIM for evaluation). It should also be noted that Zhu et al (2019) formed the ground truth set by asking for human preferences between results of existing algorithms, which makes the interpretation of the score gains less clear than in SISR and denoising where the actual target image is completely known. Suggestions for improvements: * The visual examples in the appendices do a good job of showing the improvements from using PNB. Are there any new failure modes that the baseline networks do not exhibit? * For superresolution evaluation, it would be more informative to list (at least in appendices) results for more downsampling factors than 3x (related work seems to examine 2x, 3x, 4x, and 8x). Similarly, for denoising, data for various noise levels would provide more insight into the behavior of the proposed model. * In Table 4, and 5, what are the averages computed over? * In Zhu et al (2019) which was used a ground truth for the edge-preserving smoothing experiments, each undistorted image is associated with multiple filtered results selected as preferred by human evaluators. How is this information incorporated into the PSNR/SSIM scores in Table 1?