Title: -LOCAL ENHANCED NET-WORKS

Abstract: Existing neural networks proposed for low-level image processing tasks are usually implemented by stacking convolution layers with limited kernel size. Every convolution layer merely involves in context information from a small local neighborhood. More contextual features can be explored as more convolution layers are adopted. However it is difficult and costly to take full advantage of long-range dependencies. We employ non-local operation to build up connection between every pixel and all remain pixels. Moreover a novel Pyramid Non-local Block is devised to robustly estimate pairwise similarity coefficients between different scales of content patterns. Considering computation burden and memory consumption, we exploit embedding feature maps with coarser resolution to represent content patterns with larger spatial scale. Through elaborately combining the pyramid non-local blocks and dilated residual blocks, we set up a Pyramid Non-local Enhanced Network for edge-preserving image smoothing. It achieves state-of-the-art performance in imitating three classical image smoothing algorithms. Additionally, the pyramid non-local block can be directly incorporated into existing convolution neural networks for other image processing tasks. We integrate it into two state-ofthe-art methods for image denoising and single image super-resolution respectively, achieving consistently improved performance.

1. INTRODUCTION:

Recently impressive progress has been achieved in low-level computer vision tasks as the development of convolution neural networks (CNN), e.g. edge-preserving image smoothing (Li et al., 2016; Fan et al., 2017; Zhu et al., 2019) , image denoising (Mao et al., 2016; Zhang et al., 2017; 2018) and image super-resolution (Tai et al., 2017a; b; Zhang et al., 2018) . This paper focuses on devising a novel pyramid non-local block oriented for effectively and efficiently mining long-range dependencies in low-level image processing tasks.

Typical convolution layers operate on a small local neighborhood without considering non-local contextual information. One common practice for capturing long-range dependencies is to enlarge the receptive field by stacking large number of convolution layers (Kim et al., 2016a) or dilating convolution layers (Yu et al., 2017) . However, it is difficult to deliver information between distant positions in such a manner (Hochreiter & Schmidhuber, 1997) .

Discovering similar patterns (including small texture patches and large object parts) residing in natural images is valuable to low-level image processing (Buades et al., 2005; Dabov et al., 2007; Mairal et al., 2009) . Dependencies on similar patterns can be used for recognizing out real region boundary in image smoothing and recovering contaminated or missing details in image restoration tasks. Inspired from this concept, a few literatures (Lefkimmiatis, 2017; Wang et al., 2018; Liu et al., 2018; Li et al., 2018a) introduce non-local operation into deep networks in form of self-similarity strategy. Wang et al. (2018) presents a non-local component in video classification, which is placed after high-level, sub-sampled feature maps. Huge computational cost and memory consumption of their non-local operations hinder its adaption to low-level computer vision tasks, where high-resolution feature maps are demanded to produce appealing pixel-wise outputs. NLRN (Liu et al., 2018) and NLEDN (Li et al., 2018a) apply non-local operation for image restoration and rain removal, respectively. One fatal drawback of this kind of methods is that the estimation of self-similarity is confined in a neighborhood of tens of pixels. Besides it is a very common phenomenon that the same type of patterns appear to own various spatial scales, which has not been taken into consideration in existing non-local operations.

To settle the above issues, we propose a pyramid structure, named Pyramid Non-local Block (PNB), to perform non-local operation with a cost-effective amount of computation load. It utilizes a query feature map with full resolution and a pyramid of reference feature maps to robustly estimate similarities between different scales of patterns. Accordingly, a pyramid of embedding feature maps are extracted to enhance the input feature map with the help of estimated similarity matrices. To ensure acceptable computation burden and memory consumption, resolutions of the reference and embedding feature maps are downscaled. Through intervening pyramid non-local blocks and dilated residual blocks (Yu et al., 2017) , we set up Pyramid Non-Local Enhanced Networks for edge-preserving image smoothing. It achieves the state-of-the-art performance in imitating various classical image smoothing filters.

In addition, efficient computation allows our proposed pyramid non-local block to be incorporated into deep CNN-based methods for pixel-level image processing tasks. We demonstrate the effectiveness of PNB on two classical tasks, image denoising and single image super-resolution (SISR). Two state-of-the-art models, RDN (Zhang et al., 2018) and MemNet (Tai et al., 2017b) , are adopted as the baseline models. The PNB acts as a critical component to exploit long-range dependencies. Performance improvements over baseline have been consistently obtained thanks to the adoption of pyramid non-local blocks.

2. RELATED WORK:

Deep Learning Based Image Processing Edge-preserving image smoothing aims to preserve significant image structures while smoothing out trivial details. Because of the strong feature learning ability, deep neural networks attracts a lot of attention in this field. The pioneering work (Xu et al., 2015) employs a three-layer CNN to predict a gradient map which is subsequently used to guide the smoothing procedure. Recurrent network is adopted to efficiently propagate spatial contextual information across pixels (Liu et al., 2016) . Joint filtering (Li et al., 2016) and edge map (Fan et al., 2017) are dedicated to using extra guidances for image smoothing. Meanwhile, deep neural networks have been proposed for image denoising and super-resolution (Kim et al., 2016a; Zhang et al., 2017) . Residual connections (Lim et al., 2017) and dense connections (Tong et al., 2017; Zhang et al., 2018) are added to alleviate the vanishing-gradient problem. Features at different scales are fused to get efficacious information (Li et al., 2018b) . Other representative methods include Huang et al. (2015) ; Shi et al. (2016) ; Tai et al. (2017a; b); Ahn et al. (2018) ; Hui et al. (2018) . We do not elaborate due to limited space.

Non-local Context Information. Image non-local self-similarity has been widely exploited in many non-local methods for image restoration, such as BM3D (Dabov et al., 2007) and WNNM (Gu et al., 2014) . Recently a few studies attempt to incorporate non-local operations into deep neural networks for capturing long-range dependencies. Wang et al. (2018) presented trainable non-local neural networks in video classification. However, the computation complexity of their non-local operation grows dramatically as size of the input feature map increases. Liu et al. (2018) and Li et al. (2018a) incorporated non-local module into RNN architecture and encoder-decoder architecture for image restoration and image de-raining, respectively. Nevertheless, the measurement of self-similarity is restricted within a small neighborhood.

Our method differs from existing models proposed for low-level image processing in two aspects. Firstly, our method can robustly measure similarities between different scales of content patterns as we adopt a pyramid structure for the non-local operation. Secondly, computation burden and memory consumption is greatly relieved because the spatial resolutions of reference features in the non-local operation are downscaled.

3. METHOD:

Non-local correlations between patterns are paramount for edge-preserving image smoothing. Longrange contextual features are beneficial to suppress textures inside objects (object parts) while identifying out real region boundaries in image smoothing. We propose a deep pyramid non-local enhanced network (PNEN), which introduces pyramid non-local block (PNB) to explicitly mine long-range dependencies. The overall architecture is illustrated in Figure 1 . The pyramid non-local block (PNB) is carefully designed to involve in correlation with distant pixels when inferring every pixel, as well as guaranteeing computation efficiency in terms of complexity and tolerant memory consumption. Besides, dilated residual blocks (DRB) (Yu et al., 2017) are introduced to enlarge receptive field for taking advantage of full structural and textural information in the image. In the following sections, each component of the proposed architecture will be elaborated with more details.

Input Entry … … Exit ⊕ " # " $ " % Output PNB DRB PNB DRB PNB DRB & # & $ & % & '

3.1. ENTRY AND EXIT NETWORK:

Define the input color image as X with size of h × w × 3. h and w represents the height and width of the input image respectively. Our proposed PNEN employs one convolution layer as the entry net to transform X into a feature map F 0 which is a h × w × c tensor. Formally, we have

EQUATION

where the F entry (•, •) denotes the convolution operations in the entry net and W entry represents related convolution parameters. Subsequently, M blocks, each of which is consisting of a pyramid non-local block and a dilated residual block, are stacked to induce deep features. We define the feature produced by the m-th block as F m . We have

EQUATION

where F P N B (•, •) and F DRB (•, •) indicates the calculation procedure inside the pyramid non-local block and dilated residual block which will be elaborated in Section 3.2 and 3.3 respectively. W m DRB and W m P N B represent their parameters correspondingly. Inspired by MemNet (Tai et al., 2017b) , features generated by all blocks {F m |m = 1, • • • , M } are accumulated to generate residual images using the exit network. The residual image produced with F m is defined as

EQUATION

where F exit (•, •) denotes the convolution operations in the exit network and W m exit represents the related parameters. The exit network contains three convolutional layers. Suppose Y m = X + R m . The final reconstructed images is computed by

EQUATION

where 

{w m |m = 1, • • • , M }

L = 1 hwc ( G -Y 2 + M m=1 G -Y m 2 ).

(5)

3.2. PYRAMID NON-LOCAL BLOCK (PNB):

Let F ∈ R h×w×c denotes the input feature activation map. Here h, w and c represents the height, width and channel, respectively. A general formulation of non-local operation (Wang et al., 2018) can be defined as

EQUATION

⊗ ⊗ !"×!" !×"×$ softmax %: (1, $, 1) +: (1, ,, 1) -: (1, ,, 1) !"×, ⊗ ⊗ !"×!" !×"×$ softmax X Y ,×!"

!"×$ ":

(1, $, 1)

$:

(1, ,, 1) %:

(1, ,,

EQUATION

EQUATION

":

(1, ), 1)

$:

(1, ,,

⊕ & -: (1, ,, 1) where F is the enhanced feature representation. M(F) ∈ R hw×hw is the self-similarity matrix, where each element M(F) i,j indicates the similarity between pixel i and j. G(X) ∈ R hw×n gives rise to a n-dimensional pixel-wise embedding. D(X) is a diagonal matrix for normalization purpose. T (•) is the transformation function which converts the embedded features back into the original space of input feature. In this way, the feature representation is non-locally enhanced through considering all positions of the feature map. One instantiation (Wang et al., 2018) can be constructed by taking the linear embedded Gaussian kernel as the distance metric to compute correlation matrix M, and linear function to compute G:

+ / : (2, ,, 2) $ / : (2, ), 2) ⊗ ⊗ ⊗ !"×! ' " ' !"×! ( " ( ! ( " ( ×) ! ' " ' ×) C ⊕ ψ: (1, 2, 1) !×"×3) ) & + 4 : (4, ,, 4) $ 4 : (4, ), 4) softmax ⊗ ⊗ ⊗ ! * " * ×) !"×! * " * softmax softmax $ 6 : (2 3 , ), 2 3 ) + 6 : (2 3 , ,, 2 3 ) ) & & ψ: (1, 2, 1) + , 7 + , 8 + , 9 (b) Pyramid Non-local Block

EQUATION

The embedding function F emb (F, W) can be implemented by first applying convolutional operation of parameter W on F, and then transforming the result into a 2-dimensional tensor in which each column represents one embedding channel. When calculating M(F), a query and a reference feature with same size of hw × m are generated using convolution kernel W θ and W φ respectively. The output dimension of W g is denoted as n. The diagonal elements of D(F) are obtained through calculating column summation of M(F). T (•) is also implemented with a convolution operation of parameter W ψ to convert the embedding feature back into the original c-dimensional space. All convolutions use kernel size of 1 × 1. An example non-local block is illustrated in Figure 2 (a). The computation complexity and memory occupation of the correlation matrix increases exponentially as the number of pixels grows. For sake of reducing computation burden, previous works (Liu et al., 2018; Li et al., 2018a) utilize a small neighborhood to restrict the range of non-local operation.

In comparison, we propose a novel pyramid non-local block to effectively soften the computation demand while achieving appealing performance in low-level image processing tasks.

At first, we produce one query feature E θ = F emb (F, W θ ). The spatial kernel size and stride of W θ is 1 × 1 and 1 respectively. Then, multi-scale reference features and embedding features are generated with convolutional layers using different kernel sizes and strides. Suppose there are totally S scales. The computation procedure of the non-local operation in the s-th scale is as follows:

EQUATION

EQUATION

where {E s Φ |s = 1, • • • , S} are query features for calculating similarity matrices with respect to E θ and {E s g |s = 1, • • • , S} are embedding features in all scales. The stride of convolutional layers in the s-th scale is set to 2 s . This implies that the number of rows in E s Φ and E s g is reduced to hw/4 s , which greatly reduces the amount of computation for estimating the self-similarity matrix. We adopt larger convolutional kernels to extract reference feature representations of larger content patterns. The pyramidal design is motivated by two reasons: 1) The computation and memory requirement of non-local operations can be controlled within an acceptable load. 2) The robustness of self-similarity estimation is enhanced since the self-similarity is measured at various resolutions.

X !: (1, &, 1) ( ) : (2, &, 2) + ) : (2, ,, 2) ⊗ ⊗ ⊗ -.×-" . " -.×-#.# -#.#×, -" . " ×, C ⊕ ψ: (1, 2, 1) -×.×2 -×.×3, -.×& ⊗ ⊗ -.×-. -×.×, -×.×2 softmax -×.×2 Y ( 4 : (4, &, 4) + 4 : (4, ,, 4) ( 6 : (7, &, S) + 6 : (7, ,, 7) softmax ⊗ ⊗ ⊗ -$ . $ ×, -.×-$ . $ softmax softmax X Y &×-.

-.×, %:

(1, ,, 1)

':

(1, &, 1) (: 

(1, &,

F = F Ψ ({ Ê1 , • • • , ÊS }, W ψ ) + F.

(11) We summarize Eq. ( 9), ( 10) and ( 11) with the function F P N B (•, •), as mentioned in Section 3.1. One characteristic of pyramid non-local block is the flexibility of balancing accuracy and computation resources through adjusting kernel sizes and strides in different scales. An illustration of the pyramid non-local block is shown in Figure 2(b) . We set m = 64, n = 32 and S = 3 in practice.

3.3. DILATED RESIDUAL BLOCK (DRB):

In pixel-wise image processing tasks, high-resolution feature maps are favorable for reconstructing complicate textural details while large receptive field benefits the extraction of features which can grab high-level contextual information. Considering the above issues, we employ dilated convolution (Yu et al., 2017) to rapidly increase the receptive fields without sacrificing spatial resolutions of learned features in our proposed model. As shown in Figure 3 , we devise a dilated residual block with 5 cascaded residual modules which is set up with dilated convolutions. The calculation procedure of the dilated residual block is indicated with the function F DRB (•, •), as mentioned in Section 3.1.

As illustrated in Figure 1 , our model is built up through intervening PNB-s and DRB-s. Every group of consecutive PNB and DRB contributes a feature map for the final prediction.

3.4. DISCUSSION:

The benefits of our proposed pyramid non-local block are three folds: First, the pyramidal strategy adopts convolutions with various kernel sizes to generate embedding features for self-similarity estimation. This improves the robustness of estimating correlation across pattern scales.

Second, existing deep models equipped with non-local modules are implemented via connecting all pairs of pixels in the feature map Wang et al. (2018) or limiting the nonlocal dependencies within a constant neighborhood size Liu et al. (2018) ; Li et al. (2018a) . The former kind of methods merely plug non-local modules after high-level feature maps with small resolution, because the computational complexity and memory usage will grow exponentially as the number of pixels in the feature map increases. The later kind of methods inevitably neglect valuable correlations from pixels outside the fixed neighborhood. We solve the problem ingeniously through embedding the input feature into a query feature map with full resolution and multiple reference feature maps with downscaled resolutions. In such a manner, the computation burden can be greatly relieved when applying our method in low-level pixel-wise image processing tasks.

Third, the pyramid non-local block can be easily incorporated into existing CNN-based models devised for low-level image processing, such as RDN (Zhang et al., 2018) for denoising or MemNet (Tai et al., 2017b) for single image super-resolution. Experiments on various datasets indicate that the pyramid non-local block consistently improves the performance in image smoothing, denoising and super-resolution.

Table 1 : Quantitative comparison in terms of PSNR/SSIM on three smoothing filters. The best performance is marked in bold. Model depth and parameter number are also indicated.

DJF (Li et al., 2016) CEILNet (Fan et al., 2017) ResNet (Zhu et al., 2019 

4.1. DATASET:

The image datasets are from Zhu et al. (2019) . There are 500 images of clear structures and visible details to evaluate the learning apability of deep model in reproducing edge-preserving filters. Following Zhu et al. (2019) , we use 400 images as training set and the remaining 100 images as testing set. We train models to reproduce three representative filters in our experiments, including weighted median filter (r = 10, σ = 50) (Zhang et al., 2014) , L 0 smoothing (λ = 0.02, κ = 2) (Xu et al., 2011) and SD Filter (λ = 15) (Ham et al., 2017) .

Implementation Details Without specification, all convolutional layers have 64 filters with kernel size 3 × 3. We stack three PNB-s and DRB-s consecutively as the feature extractor, resulting in a 37-layer deep network. During the training stage, random horizontal flip and rotation are applied for data augmentation. Training images are split into 96 × 96 patches. The mini-batch size is set to 8. Adam optimizer (Kingma & Ba, 2015) with β 1 = 0.9, β 2 = 0.999, = 10 -8 is used for optimization.

The initial learning rate is set to 5 × 10 -4 and reduced by half when training loss stops decreasing, until it reduces to 10 -4 . It takes around 2 days to train a model on one TITAN Xp GPU. For inference, the model takes 1.2 seconds to process a testing image of 500 × 400 pixels.

4.2. COMPARISON WITH THE STATE-OF-THE-ART:

Recently, piles of CNN-based approaches (Xu et al., 2015; Liu et al., 2016; Li et al., 2016; Fan et al., 2017; Zhu et al., 2019) have been proposed to reproduce edge-preserving smoothing filters. We compare our proposed PNEN against three state-of-the-art methods, including DeepJointFilter (DJF) (Li et al., 2016) , Cascaded Edge and Image Learning Network (CEILNet) (Fan et al., 2017) and Residual Networks (ResNet) (Zhu et al., 2019) . For fair comparison, the networks are re-trained from scratch on the same training dataset as described above. We evaluate the quality of the generated images using two metrics, including Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index (SSIM) (Wang et al., 2004) .

Quantitative results are reported in Table 1 . When reproducing WMF, L 0 smoothing and SD Filter, our proposed method (PNEB) outperforms the second best method ResNet (Zhu et al., 2019 ) by 1.15dB, 1.14dB and 0.98dB in PSNR metric respectively. Higher SSIM performance also indicates that our method can give rise to smoothed images with better structural information. A visual comparison of learning L 0 smoothing filter is provided in Figure 4 . As can be seen in close-ups of the selected patch, the region smoothed by our proposed PNEN appears to be cleaner and flatter than other methods. Our result is closer to the groundtruth image.

As shown in Figure 5 , we visualize the similarity map derived from the last pyramid non-local block at two locations, marked by red and blue respectively. Figure 5 (b), (c) and (d) show the similarity map at 'Scale 2', 'Scale 4' and 'Scale 8', respectively. We can see that the pixels with similar features show high correlation in these maps. Thus, the features are non-locally enhanced by exploiting long-range dependencies. Particularly, for the pixel marked by red, there exist noises when estimating correlation coefficients with similar patterns (left-bottom area) in 'Scale 2' while 'Scale 4' performs well on these patterns. It indicates that the adoption of pyramid non-local operations can benefit the robustness of estimating correlations between different scales of patterns.

4.3. ABLATION STUDY:

To validate the effectiveness and necessity of pyramidal strategy, we exhaustively compare PNB with its variants on learning L0 smoothing filter. The performance changes in terms of PSNR and SSIM are shown in networks with a significant margin. It is noteworthy that the design of PNB is flexible according to the trade-off between computation efficiency and accuracy.

The required computational resources, in terms of floating point operations (FLOPs) and memory consumption, are summarized in Table 3 . The performance is obtained by testing methods on a 96 × 96 patch. We compare our proposed PNB with the non-local block proposed in Wang et al. (2018) . Memory consumption increases dramatically since non-local operation needs to store a large correlation matrix. Our memory requirement is 57% less than that of Wang et al. (2018) . Under the same GPU memory constraint, our method allows larger training patches and larger receptive field for low-level image processing tasks.

5. EXTENSION: EXPERIMENTS ON IMAGE RESTORATION:

Deep convolutional networks are widely used in image restoration tasks (Kim et al., 2016a; b; Zhang et al., 2017; Lim et al., 2017; Tai et al., 2017b; Hui et al., 2018; Zhang et al., 2018) . We adopt two state-of-the-art methods, RDN (Zhang et al., 2018) and MemNet (Tai et al., 2017b) , as the baseline models for image denoising and image super-resolution task, respectively. As discussed in Section 3.4, efficient computation allows our proposed pyramid non-local block be incorporated into these low-level baseline models. The PNB acts as a basic component to exploit non-local image self-similarity. Model architectures and visual comparisons are provided in the appendix.

Image Denoising: Three PNB-s are incorporated into RDN. To demonstrate the effectiveness of PNB, we compare with a variant of RDN by stacking more convolutional layers. The deeper RDN has more parameters than the PNB-enhanced RDN. We follow the training and testing protocols as in Zhang et al. (2018) for fair comparison. 800 images from DIV2K dataset (Timofte et al., 2017) are utilized as training data. We add white Gaussian noise with standard deviation σ = 25 to the original images to synthesize noisy images. Results on three widely used benchmarks, Set12 (Zhang et al., 2017) , BSD68 (Roth & Black, 2009) , Urban 100 (Huang et al., 2015) , are reported in Table 4 . The PNB-enhanced RDN yields better results than original RDN and its variant. The training data are constructed by bicubic downsampling with factor 3, and then upscaled to the original size. Results on four widely used benchmarks, Set5 (Bevilacqua et al., 2012) , Set14 (Zeyde et al., 2010) , BSD100 (Martin et al., 2001) and Urban100 (Huang et al., 2015) , are presented in Table 5 . The PNB-enhanced MemNet yields better results than original MemNet and its variant.

6. CONCLUSION:

In this paper, we present a novel and effective pyramid non-local enhanced network (PNEN) for edge-preserving image smoothing. The proposed pyramid non-local block (PNB) is computationfriendly, which allows it be a plug-and-play component in existing deep methods for low-level image processing tasks. Methods incorporating our proposed pyramid non-local block achieve significantly improved performance in image denoising and image super-resolution. 

C IMAGE SUPER-RESOLUTION: MODELS AND VISUAL COMPARISON:

The architecture of MemNet (Tai et al., 2017b) is shown in Figure 12 (a). They proposed memory block as their basic component. Please refer to Tai et al. (2017b) for more details. As shown in Figure 12 (b), we incorporate three our proposed PNB-s into their network without affecting the dense connections between memory blocks. 

:

EXPERIMENTS IN EDGE-PRESERVING SMOOTHINGEdge-preserving smoothing is a fundamental topic in image processing. It aims at detecting major image structures while neglecting insignificant details, which is critical in many computer vision tasks such as image segmentation and contour detection.

Figures and Tables:

Figure 1: The overall architecture of our proposed pyramid non-local enhanced network (PNEN).

are trainable weights. During the training stage, supervisions are imposed to intermediate predictions Y m -s and final output Y. The loss function is defined as the mean squared error towards groundtruth images G:

Figure2: Architecture of prior non-local block(Wang et al., 2018) and our pyramid non-local block (PNB). In PNB, φ k and g k are implemented with convolutional layers of different strides, while θ is shared. The kernel size k, number of filters f and stride s are indicated as (k, f, s) for each convolution layer.

Figure 4: Visual comparison of learning L0 smoothing filter. There exist unwanted details in previous methods if we examine closely. In contrast, the result generated by our PNEN is closer to the groundtruth.

Figure 5: Correlation map of pyramid non-local operation at different scales. The first row shows the correlation map computed at the red point. The second row shows correlation map computed at the blue point.

Super-Resolution: We add three PNB-s into MemNet. We also compare with a variant of MemNet by stacking more convolutional layers. The deeper MemNet and PNB-enhanced MemNet have similar parameter numbers. We follow the training and testing protocols as in Tai et al. (2017b) for fair comparison. The training sets consist of 291 images where 200 images are BSD train set and other 91 images are from Yang et al. (2010).

Figure 8: Correlation map of pyramid non-local operation at different scales. The first row shows the correlation map computed at the red point. The second row shows correlation map computed at the blue point.

Figure 12: Baseline MemNet architecture, and the enhanced MemNet architecture by incorporating our PNB.

1)

Table content: <table><tr><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td><td>Conv</td><td>Relu</td><td>Conv</td></tr><tr><td>Dilation: 1</td><td/><td/><td/><td>2</td><td/><td/><td>4</td><td/><td/><td>2</td><td/><td/><td>1</td><td/></tr><tr><td colspan="15">Figure 3: Architecture of dilated residual block (DRB). Our proposed DRB contains five Conv-BN-</td></tr><tr><td colspan="12">ReLU-Conv groups, which have dilation factor 1, 2, 4, 2, 1, respectively.</td><td/><td/><td/></tr><tr><td colspan="15">After performing non-local operation in all scales, enhanced embedding features { Ês |s = 1, • • • , S}</td></tr><tr><td colspan="15">are concatenated together, followed by one 1 × 1 convolution layer to generate residual values to F.</td></tr><tr><td colspan="12">Formally, the final output of pyramid non-local block can be achieved by</td><td/><td/><td/></tr></table>

Ablation study on different design choices of pyramid non-local block.

Table content: <table><tr><td/><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td></tr><tr><td>Scale 2</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Scale 4</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Scale 8</td><td/><td/><td/><td/><td/><td/></tr><tr><td>PSNR</td><td>31.95</td><td>32.57</td><td>33.17</td><td>32.87</td><td>33.21</td><td>33.44</td></tr><tr><td colspan="7">SSIM 0.9665 0.9706 0.9708 0.9707 0.9720 0.9741</td></tr></table>

Comparison of computation resource requirements in terms of floating point operations (FLOPs) and memory consumption.

Table content: <table><tr><td/><td>w/o</td><td>w/ Wang et al. (2018)</td><td>w/ our PNB</td></tr><tr><td>FLOPs</td><td>40.6G</td><td>46.2G</td><td>42.5G</td></tr><tr><td>Memory</td><td>1.1GB</td><td>9.6GB</td><td>4.3GB</td></tr><tr><td colspan="2">#Parameters 1334k</td><td>1519k</td><td>1875k</td></tr><tr><td>PSNR</td><td>31.95</td><td>32.40</td><td>33.44</td></tr></table>

The multi-scale pyramid non-local networks outperforms single-scale non-local

Table content: <table/>

Image denoising results.

Table content: <table><tr><td>Aver-</td></tr></table>

SISR results. Average PSNR/SSIM results are reported on Set5, Set14, BSD100 and Urban100 dataset.

Table content: <table><tr><td>Dataset</td><td>MemNet</td><td colspan="2">MemNet(deeper) MemNet(w/ PNB)</td></tr><tr><td>Set5</td><td>34.09/0.9248</td><td>34.12/0.9250</td><td>34.18/0.9267</td></tr><tr><td>Set14</td><td>30.00/0.8350</td><td>30.04/0.8361</td><td>30.12/0.8409</td></tr><tr><td>BSD100</td><td>28.96/0.8001</td><td>28.97/0.8008</td><td>29.02/0.8072</td></tr><tr><td colspan="2">Urban100 27.56/0.8376</td><td>27.65/0.8380</td><td>27.88/0.8474</td></tr><tr><td>#Parmeters</td><td>677k</td><td>1056k</td><td>1047k</td></tr></table>

Peer review (iclr2020):
Specific Problem Tackled and Approach The paper presents a novel method for low level image feature extraction called pyramid non-local block (PNB) which is the family of pyramid non-local enhanced networks (PNENs). The authors showed its primary utility in the application of edge preserving image smoothing. However, one additional and important impact of the paper is that the PNB approach can be used as a plug and play component in other image processing and computer vision applications such image denoising and image super resolution and this adds to the generality of the approach which is commendable. The paper sets forth quantitative analysis of all the applications against state-of-the-art approaches and demonstrate the high efficiency without loss of performance in case of PNB. Impact and motivation In terms of impact and motivation, PNB’s impact is threefold. Firstly, the pyramid approach uses multiple convolutions with different kernel sizes for generating embedding features for self-similarity (correlation) estimation. This improves the robustness of estimating correlation across pattern scales. Also, besides this, the PNB approach preserves long range dependencis in non-local features in a convolution network while remaining computationally efficient. This is one of the primary impacts of the paper. Finally, the pyramid non-local block can be incorporated into existing convolution based models used for low-level image processing in applications like denoising or for single image super-resolution. Experiments have been presented on several datasets to indicate that the pyramid non-local block improves the performance in image smoothing, denoising and super-resolution. Strengths and Rooms for Improvement The abstract, while clear, can be improved with some linguistic clarity. Some of the sentences are fragment but this does not hinder understanding. Following the abstract, the introduction properly presents the motivation and related literature in an impressive manner. It might have been stronger if a brief introduction of pyramid based approaches and its applications were discussed. Still, the motivation of the fact that capturing long-range dependencies is a limitation in most modern convolution approaches has been conveyed very well. As for Section 2 (Related Works), instead of mentioning so many works in literature, I would recommend you to cite review papers which can hold more information without taking much too much space. This way you can elaborate a bit more than just listing them including the sentence, “We do not elaborate due to limited space,” would then be unneeded. The sections following Section 3 (Method) are the strongest. The mathematical presentation is detailed and very well presented. Figure 2 serves as a very useful architecture and motivation as to why the method presented in the paper is novel. Equation 9 and 10 does a splendid job in explaining why this is computationally efficient. The two equation mathematically shows that it is motivated by two ideas. Firstly, the computation and memory requirement of non-local operations can be controlled within an acceptable load; and secondly, the robustness of self-similarity estimation is enhanced since the self-similarity is measured at various resolutions. The experimental findings are well presented for image smoothing. I appreciate showing the spatial/image output using multiple approaches in figure 4. As a support, the quantitative findings showing PNB’s superiority over several state-of-the-art methods in table 1 is welcome. However, if this pattern were to continue for also image denoising (as from salt-and -pepper noise etc.) and image super-resolution, then the paper would be even stronger. Currently, the quantitative metrics are measured in tables 3-5 which is great but if the reader is presented with demonstrations like figure 4, then it can be excellent. As a whole, the novelty and impact of the approach was conveyed clearly and in a scholarly manner. The work is commendable. Some clarity issues can be improved. For example, the sentence “Besides it is a very common phenomenon that the same type of patterns appear to own various spatial scales, which has not been taken into consideration in existing non-local operations” – is long and unclear. There are several such examples and I encourage the authors to break the long sentences and replace them with smaller and simpler sentences.