Multi-Scale Adaptive Network for Single Image Denoising
Authors: Yuanbiao Gou
Peng Hu
Jiancheng Lv
Joey Tianyi Zhou
Xi Peng
Venue: NeurIPS 2022 Accept (2022)
Keywords: image denoising
multi-scale architecture
image restoration

Abstract: Multi-scale architectures have shown effectiveness in a variety of tasks thanks to appealing cross-scale complementarity. However, existing architectures treat different scale features equally without considering the scale-specific characteristics, \textit{i.e.}, the within-scale characteristics are ignored in the architecture design. In this paper, we reveal this missing piece for multi-scale architecture design and accordingly propose a novel Multi-Scale Adaptive Network (MSANet) for single image denoising. Specifically, MSANet simultaneously embraces the within-scale characteristics and the cross-scale complementarity thanks to three novel neural blocks, \textit{i.e.}, adaptive feature block (AFeB), adaptive multi-scale block (AMB), and adaptive fusion block (AFuB). In brief, AFeB is designed to adaptively preserve image details and filter noises, which is highly expected for the features with mixed details and noises. AMB could enlarge the receptive field and aggregate the multi-scale information, which meets the need of contextually informative features. AFuB devotes to adaptively sampling and transferring the features from one scale to another scale, which fuses the multi-scale features with varying characteristics from coarse to fine. Extensive experiments on both three real and six synthetic noisy image datasets show the superiority of MSANet compared with 12 methods. The code could be accessed from https://github.com/XLearning-SCU/2022-NeurIPS-MSANet.

Multi-Scale Adaptive Network for Single Image Denoising
Yuanbiao Gou', Peng Hu', Jiancheng Lv Joey Tianyi Zhou', Xi Peng
'College of Computer Science, Sichuan University, China.
Institute of High Performance Computing, S Singapore.
fgouyuanbiao, genghu.ml, joey. boey.tianyi tianyi. zhou, pengx.gm (oggxx.gmmgmmai...m com 1vjianchengdscu. eew.ggguscu..uuu.ccmm.m edu.cn
Abstract
Multi-scale architectures have shown effectiveness in variety of tasks thanks appealing cross-scale complementarity. However, existing architectures treat differ- ent scale features equally without considering the scale-specifer characteristics, i.e., the within-scale characteristics are ignored in the architecture design. In this paper, reveal this missing piece for multi-scale architecture design and accordingly pro- pose a novel Multi-Scale Adaptive Network (MSANet) for single image denoising. Specifically, MSANet simultaneously embraces the within-scale characteristics and the cross-scale complementarity thanks to three novel neural blocks, i.e., adaptive feature block (AFeB), adaptive multi-scale block (AMB), and adaptive fusion block (AFuB). In brief, AFeB is designed to adaptively preserve image details and filter noises, which is highly expected for the features with mixed details and noises. AMB could enlarge the receptive field and aggregate the multi-scale infor- mation, which meets the need of contextually informative features. AFuB devotes to adaptively sampling and transferring the features from one scale to another scale, which fuses the multi-scale features with varying characteristics from coarse to fine. Extensive experiments on both three real and six synthetic noisy image datasets show the superiority of MSANet compared with 12 methods. The code could be accessed from http://iiiioccooo//eeinnnngnnuu0
1 Introduction
In the real world, images are often contaminated by various signal-dependent or -independent noises during the image acquisition process. result, the imaging quality will deteriorate, thus hindering people and computers from receiving image information. To solve this problem, image denoising, as an essential step for image perception, has been extensively studied in the past decades [10 19 27 50)
In the early, filtering-based methods remove the image noise by manually designing low-pass filters, egg., median filtering [8] bilateral filtering [36], and wiener filtering 77 Afterward, model-based
methods remove the image noise by optimizing problem of maximum posteriori [13 40 43
For instance, ITS 440] proposed an intrinsic tensor sparsity regularization on the non-local similar image patches by assuming they could be sparsely represented. Looking from the other side, both the filtering- and model-based methods are based on image priors from the statistics of natural images, and thus could be referred to as prior-based methods. Although remarkable performance has been achieved, an unpleasant denoising result will obtained once the priors are inconsistent with the real data distribution. To avoid prior engineering, learning-based methods adopt data-driven fashion to remove noise by learning the mapping from the noisy image to the corresponding clean image.
Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

F.Rss. E. Cla. R. Fie. N.Ams.
N. Rob, G. Det. C.
Mulli-scale Features
(Kihhuu characlenviics
Croseccll amplenantarty
Higher
Less
Smaller
Moce
Less
More
Less
Hoenoloort Archiediue
Add Fraiun
Concatorate Fucon
Hontologon Archiectuee
Withit scale chamoteristis Cross scale complementariy
Sctle-pecific Archnoctine
Scale specitic Trandlr Fusim
Satlespocific Archnocting
Lower
More
Larger
Less
More
Figure The motivations of MSANet. Left: the features different resolutions show varying characteristics. In the figure, "F. Res." and "F. Chs." " denote feature resolutions and channels, respectively; "R. Fie." denotes receptive field; "N. Amt." and "N. Rob." denote noise amount and robustmess, respectively; "G. Det." and "C. Inf." denote geometric details and contextual information, respectively. Right: the major difference of MSANet with existing multi-scale architectures, i.e., different scale features show varying characteristics and should be processed by scale-specific structures rather than homologous architectures.
With the popularity of deep neural networks, various network architectures have been designed and achieved state-of-the-art performance 66 26 32 5T]. For example, DnCNN [50] introduced residual learning and batch normalization to implement denoising convolution network. SwinIR [19 introduced the shifted windowing scheme to implement a restoration Transformer network.
Among the network design paradigms, multi-scale architectures play significant role in perfor- mance improvement thanks to multi-scale features. However, existing studies design their network architectures by only considering the cross-scale complementane while ignoring the within-scale characteristics (see Fig. D.. To exact, the shallow and high-resolution features, which lack aware- ness of contextual information, are sensitive to inputs and contain a lot of noises. Nevertheless, they contain abundant image geometric information, such as edges and textures. In consequence, they are critical for the recovery of fine-grained image details, and their network structures should take the advantages and make up for the disadvantages. The deep and low-resolution features are competitive in
and contextual information, which image context, but the too-low resolution will destroy the image structure. Hence, their network structures should also make full consideration of their characteristies during the architecture design. To summarize, as different scale features show varying characteristics, it is deserved to deal with them via scale-specific structures to fully adapt to their characteristics.
critical for the recovery of coarse- marse-grained
Based on the motivation, we propose a novel Multi-Scale Adaptive Network (MSANet) for single image denoising, which simultaneously incorporates the within-scale characteristies and the cross- scale complementarity into architecture design by overcoming the following three difficulties, i.e., i) how to adaptively sample image details and filter noises; ii) how adaptively extract contextually informative features without changing feature resolutions; iii) how to adaptively fuse the multi-scale features with varying characteristics. Accordingly, three neural blocks, i.e., adaptive feature block (AFeB), adaptive multi-scale block (AMB) and adaptive fusion block (AFuB) are proposed. In brief, AFeB handles the features with mixed details and noises through adaptively sampling and weighting, and thus the image details are preserved while filtering noises. AMB extracts the contextual information through dilated convolutions and adaptive aggregation, and thus contextually informative features are obtained while keeping the resolution unchanged. AFuB adaptively samples and transfers the features from one scale to another scale, and the multi-scale features with varying characteristics are fused from coarse to fine.
To summarize, the contributions
follows:
We propose a novel neural network for single image denoising, termed as MSANet. The major difference with existing methods that MSANet simultaneously considers and incorporates the within-scale characteristics and the cross-scale complementarity into multi- scale architecture design, which is a missing piece before and the first revelation so far.
To exploit the within-scale characteristics and achieve the cross-scale complementarity. we design three neural blocks, i..., AFeB, AMB, and AFuB, which are used implement our

---

idea, i.e. building scale-specific subnetworks corresponding to different scale features by considering their characteristics.
Extensive experiments are conducted on three real and six synthetic noisy image datasets to show the effectiveness of MSANet, and the significance of the within-scale characteristics.
2
Related Work
In this section, we briefly introduce existing single image denoising methods and multi- scale archi- tectures, and discuss the major differences between MSANet and them.
2.1 Single Image Denoising
In general, most existing single image denoising approaches could categorized as prior- and learning-based methods. Prior-based methods are based on some priors from natural images, such
as local smoothing [1]] self-similarity (36 TO [3], and sparsity [43] [4]. For instance, BM3D [TO]
eliminates noisy pixels by transforming the 3D stack of non-local similar patches and employing a shrinkage function to obtain sparse coefficients. WNNM [33 introduced low-rank weight coefficient based on the nuclear norm minimization and exploited the non-local similar image patches to remove noise. Different from prior-based methods that heavily rely on handcrafted priors, soamingg-aseed methods learn the mapping from the noisy image the clean image in end-to-end manner. In recent, a large number of methods have been proposed and achieved state-of-the-an performance [6] 44 32 [8 47). For example, MemNet [35] proposed a persistent memory network to fuse both short- and long-term memories for capturing different levels of information. FFDNet [5]] enhanced the denoising network for non-uniform noise by using the noise level map. Non-local attention (37] was designed to exploit the image self-similarity. and NLRN [20] incorporated it into recurrent neural network. SADNet 6 proposed residual apatial-adaptiin
block and multi-scale context block to constitute denoising network. DeamNet [32) introduced an adaptive consistency prior and designed an interpretable deep denoising network. With the rise of vision Transformers, Uformer (38) proposed a U-shape Transformer based on the -eeaane-n window Transformer block and multi-scall restoration modulator. Restormer [47] introduced the channel-base self- attention and gating mechanism to implement an efficient Transformer.
Different from the aforementioned methods, MSANet proposed three neural blocks, i.e., AFeB, AMB, and AFuB, by simultaneously considering the within-scale characteristics and the cross-scale complementarity, constitute the scale-specific subnetworks corresponding to different scale features for adapting their varying characteristics.
2.2 Multi-Scale Architecture
Multi-scale architectures have played a significant role in many fields of computer vision 55] 21 48), thanks to multi-scale features and their cross-scale complementarity. The straightforward way for multi-scale architecture is to separately feed malti-/singlee-rsolutee
images/features into For example,
single/multiple subnetworks, and then fuse the outputs as a result [18] 25 39
45
HRNet (34 proposed multi-scale network by gradually adding high-to-low resolution subnetworks
and repeating multi-scale fusions for human pose estimation. CLEARER [12] proposed a multi-scale neural architecture search to automatically determine where to fuse multi-scale features for image
restoration. GDN [2 exploited the multi-scale information using a grid-like network and employed
an attention mechanism to improve the performance. DID-MDN [49] proposed a multi multi-stream densely connected network to efficiently leverage features of different scales for image deraining.
MSCNN 333 consists of a coarse-scale network and a fine-scale network to learn a transmission map for image dehazing. PANet 25 proposed pyramid attention network for image restoration by capturing long-range feature correspondences from a multi-scale feature pyramid. In addition, 6
38 47 employed encoder-decoder architecture combine the high-to- with low-to-high resolution features through kkip-connections.
Although the aforementioned studies and our work share similarities in multi-scale architecture, they are remarkably different. The existing methods only consider the cross-scale complementarity and use homologous architectures for different scale features. In contrast, work additionally considers
3

---

Enceder
Saloeswork
Decoder
AFeB
Roidkel
Blok
JMM
Coos2d
ARVR
Deformabee
High
Cresd
Resolation
Traepesol
Creed
Figure 2: The framework of MSANet. It employs asymmetric incoder-decoder architecture with multiple subnetworks to capture and fuse the specifc features. In addition, three neural blocks designed exploit the within-scale characteristics and achieve the cross-scale complementarity of multi-scale features. Note that we take four scales of features as a showcase and for experimental evaluations, more scales allowed in practice.
the within-scale characteristics and designs scale-specific structures to embrace both, which is the missing piece and the first revelation for multi-scale architecture design.
3 Proposed Method
In this section, we propose novel single image denoising network, i.e., MSANet, which simultane- ously embraces the within-scale characteristies and the cross-scale complementarity of multi-scale features through the three neural blocks. In the following, we will first introduce the architecture principles of MSANet and then elaborate on the three neural blocks.
3.1 Architecture Principles
As illustrated in Fig. 2 MSANet employs an asymmetric encoder-subeetworss-eeoddd architecture. Specifically, the encoder adopts four residual blocks [15] to extract features of four scales. The first residual block aims extract the initial features without changing the resolution. The other residual blocks respectively decrease the resolutions to half while increasing the channels to double. With multi-scale features extracted by the encoder, the subnetworks aim to exploit their within-scale characteristics via AFeB and AMB blocks. As the characteristics of multi-scale features gradually change from high- to low-resolution, we take the two bottom and two top subnetworks in Fig. the high- and low-rcsolution branches, respectively.
The bottom subnetworks need handle the high-resolution features which usually are charac- terized by a mixture of details and noises. It is highly expected to remove the noises without losing the fine-gu gre-ggraiedd image details, and thus AFeB designed for adaptively preserving the indispens- able details and filtering unpleasant noises. Meanwhile, another characteristic of high-resolution features is the limited contextual information. The corresponding subnetworks should also enrich the contextual information while keeping the feature resolution unchanged in order not to lose details. As result, AMB is designed for adaptively extracting contextually informative features. As AFeB and AMB could mutually boost each other, i.e., low noise features from AFeB allow AMB to better capture the image contexts, while contextually informative features from AMB help AFeB to better distinguish the noises and details, alternately stack AFeB and AMB to build the subnetworks. In addition, as the higher-resolution features usually have fewer channels (i.e., fewer parameters), a deeper subnetwork is allowed to further alleviate their disadvantages, such as massive noises and limited contexts.
Different from the bottom subnetworks, the two top subnetworks aim to exploit the within-scall char- acteristics the low-resolution features, which are characterized by less noise and more contextual information. As the too-low resolution will destroy the image structure, the structure consistency cannot guaranteed during the image recovery. Hence, we employ AMB to capture more image contexts for cnriching the contextual information without reducing the feature resolution. In addition,

---

as the low-resolution features are more robust to noise and have more channels, the top subnetworks mainly consist of AMB and are with shallower depth for efficiency. Meanwhile, except for the lowest resolution, the first and the last blocks in the top subnetworks are AFeB for adaptively controlling the input and output features.
To achieve the cross-scale complementarity, the decoder consists of four AFuB blocks followed by convolution layer to output the recovered image. More specifically, the first three AFuB blocks increase the resolutions to double and decrease the channels to half, while adaptively sampling and transferring the fine-gres inn-grained detail features to the coarse- -rerseed contextual features. The last AFuB block keeps the resolution and channel unchanged, and directly samples and transfers the image details from the noisy input, which an effective alternative to the global coipeconn,ction, for which could avoid introducing noises from the noisy input.
For given noisy input æt MSANet employs the encoder to extract the features of different scales and then feeds the features into different subnetworks to learn the scale-specific features. After that, the decoder fuses the multi-scale features with varying characteristics from coarse to fine to obtain the recovered image D = f(r), where f() denotes MSANet. To optimize MSANet, we employ the following objective function,
L == 11y - 013
(1)
where y is the ground truth of x, and p = {1,2].
3.2 Adaptive Neural Blocks
In this section, we elaborate on the proposed three neural blocks which are designed to learn better multi-scale features for single image denoising.
Adaptive Feature Block (AFeB). To preserve the indispensable details and filter unpleasant noises, the block is expected to adaptively sample and weight the input features f. based on themselves, i.e.,
{Ax, Ay, Aw)
F(f:),
(2)
where F-)) is used to compute the offset (Ar, Ay) w.r.t. the positions (z, y). as well as the corre- sponding weight Aw. Then, the output features fi+1 could be aggregated by
fi+1(1,)) y) Σ wj * f(x Axy,y AW₂) Awl,
(3)
where k is the number of samples, and w denotes the learnable weights. In this way, AFeB could learn the sampling locations to indicate where are important for restoration, while assigning different weights to show how important the locations are. As a result, AFeB preserves the indispensable details and filters unpleasant noises for better restoration. However, it is prohibitive to traverse all positions for sampling and weighting w.r.t. each position in the input features. Hence, for the convenience of implementation, AFeB employs the modulated deformable convolution [55) to implement the
aggregation operation in Eq. 3 In detail, AFeB sets the sample number to the kernel size and the learnable weights as the convolutional weights. Although this setting would decrease the number of samples and the flexibility of weights, it is efficient in the calculation of high-resolution features, and the limitations could be alleviated by stacking AFeB blocks. In summary, AFeB consists of a convolution layer F(.). a modulated deformable convolution, LeakyRel.U layer, a convolution layer and a skip-connection, i.e., Jout = Ss + Feono(Frete (f++1))-
(4)
Adaptive Multi-scale Block (AMB). Contextually informative features are highly expected for both the high- and low-resolution branches. For the high-resolution features, reducing the resolution leads to the loss of image details. For low-resolution features, a too-low resolution destroys the structure consistency of recovered images. Therefore, to capture more contextual information without changing the resolution, we propose AMB by using several convolutions with different dilation rates. Convolution with a large dilation rate could provide large receptive field, and multiple convolutions with different dilation rates could smoothly capture multi-scale information. To reduce the cost caused by the multiple convolutions, AMB compresses the channels of each convolution that the concatenated channels of all convolutions are equal to the output channels, i.e.,
f++1 Concat (d. (fF))
N+}), }),
(5)
5

---

where f; and fi+1 are imput and output features, and FL is the k-th convolution with the dilation rate d. As the concatenation assigns different scale features into different channels, the distinctive importance of multi-scale features is not considered. To address this issue, AMB adaptively scales different channels and features, i.e.
ch = 2 Fssig poolaaag_poolfftt
fi+2 ch fi+1,
(6)
sp 2 Fsig (P (enno (mean(
f+43 = sp fi+2.
where org rig_pool an adaptive average pooling on space domain, mean denotes mean operation along the channels, Frc is a linear layer, Feons is a convolution layer, Fsig is sigmoid layer, and 2+ used to control the amplification (> 1) or suppression (< 1). With the fi+3, AMB passes through a LeakyReLUU layer, a convolution layer and a skipconnectio,, i.e.,
ou Sa + FeonolFrette (J++3)).
Adaptive Fusion Block (AFuB). As the high-resolution features contain lot of disordered fine- grained image details, and the low-resolution features contain abundant coarse-grained image con- textual information, it is desirable transfer the fine- see-graaidd image details into the coarse-grait coarse-grained image context. To this end, AFuB first upsamples the coarse -anss-e features to the resolution of the fine-grained features via
fcoorse FrCone (f seoarse):
(8)
where FrCone is transpose convolution. Then, address the issue of the disordered details, AFuB adaptively samples and weights the fine-grained features by using coarse gaaseegrained features to provide contextual information and fine-grained features provide details information, i.e.,
(Ar, Ay, (a,)) F(Scoarses
where -(::-) is used to compute the offset (Ax, Ay) w.r.t the positions (Ir, v). as well as the corresponding weight Aw. After that, AFuB transfers the fine-graing one-igained details to the coarse-grained context via
f fearse fcoorse + Ew,+
W fiiee + A Ayj) * Awj,
(10)
j=1
where l is the number of fine- tne-grained detail features, w is the learnable weights. Similar to AFeB, AFuB employs a convolution layer as the function F' and a modulated deformable convolution to achieve the aggregation in TO). Finally, AFuB uses a convolution layer, LeakyReLU layer, convolution layer, and skip-connection to further refine features, i.e.,
f fout = f fine + Feono (Feeuu Fretu(Fon (Foono (ffine fine (onsse)))
(11)
Experiments
In this section, we first introduce the experimental settings, and then show the quantitative and qualitative results on nine datasets. Finally, we perform analysis experiments including ablation study and feature-based visualization. Due space limitations, we present more experimental details and results in the supplementary material.
4.1 Experimental settings
We evaluate the MSANet on both real and synthetic noisy datasets. For the evaluations on real noise, we employ the SIDD [], RENOIR B1. Poly (42) datasets for training, and use SIDD Validation, Nam (28 and DnD 30 datasets for testing. For synthetic noise, we train MSANet on DIV2K 2 dataset, which contains 800 images of 2K resolution, by adding Additive White Gaussian Noise (AWGN) with the noise levels of 30, 50, and 70. We use color McMaster [52] (CMcMaster), color Kodak24 (CKodak24), color BSD68 (24] (CBSD68) for testing color image denoising, and grayscale McMaster (GMeMaster), grayscale Kodak24 (GKodak24), grayscale BSD68 (GBSD68) for testing grayscale image denoising.

---

We implement MSANet in Pytorch 29 and carry out all experiments on Ubuntu 20.04 with GeForce RTX 3090 GPUs In our implementations, we use four scales of features with channels of 32, 64, 128 and 256. Moreover, we train MSA 100 epochs via I11 loss for real noise and 300 epochs via L.2 loss for synthetic noise. Both real and synthetic noise training are with the batch size of 16 and the patch size of 128. To optimize MSANet, the Adam [16] optimizer is used, and the learning rate is initially set to le-4 and decays to zero via the cosine annealing strategy [23]. During the training, we randomly crop, flip and rotate the patches for data augmentation. In the testing, we employ PSNR and SSIM to evaluate the performance.
4.2 Comparisons on Real Noise Images
Table 1: Quantitative results on SIDD sRGB validation dataset.
Method
CDaCNN-B
CBM3D
CBDNet
PD
RIDNet
SADNet
DeamNet
MSANet
PSNR
26.21
30.88
33.07
33,96
38.71
39.46 0.9103
39.47 0.9105
39,56
SSIM
0.8324
0,8195
0,9052
0.9118
Table 2: Quantitative results on Nam dataset with JPEG compression.
Method
CDCCNN-B
CBM3D
CBINNet
PD
RIDNee
SADNet
DeamNet
Net
PSNR SSIM
37.49
39.84
41.31
41.09
41.04
42,92 0.9839
42,03 0.9790
43.52
0.9272
0,9657
0.9784
0,9780
0,9814
0,9863
Table 3: Quantitative results on DnD sRGB dataset.
Method
CDeCNN-B
CBM3D
FFDNete
CBDNet
N3Net
PR
RIDNet
SADNet
DeamNet
MSANet
PSNR
32.43
34,51
37,61
38.06
38.32
39.00
39.26
39.59
39.63
39.65
SSIM
0.7900
0,8507
0,9415
0.9421
0.9384
0.9542
0.9528
0,9523
0,9531
0,9553
Real noise image denoising is challenging due to the real noise being usually signal-dependent and -patial-variant hinges the in-camera pipeline. Therefore, we carry out denoising experiments on three real noise image datasets, i.e., SIDD Validation, Nam, and DnD. In brief, the validation dataset of SIDD contains 1,280 256 256 noisy-clean image pairs captured by the smartphone. Nam includes 15 large image pairs with JPEG compression, and we evaluate MSANet on the selected 25 512 x 512 patches by following CBDNet [14]. DnD contains 50 pairs of real noisy-clean images captured by cameras, and 1,000 512 x 512 patches are extracted for testing. Due to the ground truths of the patches are not publicly available, we obtain the PSNR and SSIM results via the online submission system (30). Besides, since JPEG compression makes the noise more stubborn on the Nam, we first train our model on the combination of SIDD) and RENOIR for the evaluations SIDI) Validation and DND, and then fine-tune the trained model on the Poly for the evaluations on Nam.
We compare MSANet with 10 denoising methods that are comparable in model complexity, i.e., CDnCNN-B, BB, CBM3D 9. FFDNet+, CBDNet, N3Net [3I, PD [54), PR [46), RIDNet (4. SADNet and DeamNet. In experiments, we the corresponding pretrained models provided by their authors, and refer to their results reported in the online submission system and papers.
Table T shows the quantitative results on SIDD validation dataset. In brief, MSANet achieves the highest PSNR and SSIM values compared to other methods, e.g., 0.85dB, I.B, 0.09dB gains in PSNR, and 0.0066, 0.0015, 0.0013 gains in SSIM over the RIDNet, SADNet, and DeamNet, respectively. For visual comparisons in Fig. 3 CBDNet and PD result in residual noises and pscudo artifacts, RIDNet, SADNet and DeamNet severely destroy the textures and obtain over-smoothed results. In contrast, our method MSANet recovers textures and structures more subtly and obtains clearer restoration. Some areas are highlighted by color rectangles and zooming-in is recommended for better visualization.
The quantitative results on Nam dataset are shown in Table 2 which demonstrates that our method achieves significant improvements over the other methods. Specifically, MSANet outperforms RIDNet with 2.48dB (0.0049). SADNet with 0.6dB (0.0024), DeamNet with 1 1.49dB (0.0073) in PSNR (SSIM) values. For the visual comparisons shown Fig. 4 our method obtains the best result for details recovery and noises removal, which is closer to the ground truth than other results.
7

---

Figure 3: Qualitative results real noise image from SIDD validation dataset. From left right, show the real noise image, the results of CBDNet, PD, RIDNet, SADNet, DeamNet, MSANet, and the ground truth.
Figure 4: Qualitative results on real noise image from Nam dataset. From left to right, we show the real noise image, the results of CBDNet, PD, RIDNet, SADNet, DeamNet, MSANet, and the ground truth.
Figure 5: Qualitative results on real noise image from DnD dataset. From left to right, we show the real noise image, the results CBDNet, RIDNet, PD, SADNet, DeamNet, and MSANet.
Table 3 reports the quantitative results on DnD dataset, which are obtained from the DnD benchmark website. From the table, one could observe that MSANet outperforms all methods both in PSNR and SSIM values. Moreover, we further perform qualitative comparison on the DnD dataset. As shown in Fig. [5] the other methods achieve blurred results wherein many image details are corroded by noises, while our method can effectively remove the noises and obtain clearer details.
Comparisons on Synthetic Noise Images
Table Quantitative results on synthetic color noise image datasets.
Datasct
CBM3D
DoCNN
FFDNet
CLEARER
SADNet
RNAN
DeanNet
MSANet (Ours)
29.64/0
30,0500
32.01/0
32.000 8862
32.070.8876
CMcMaster
25.92/0.7153
25.9910.7147
26.23/0.7244
28.92/0.8143
29.7210.8344
29.69/0.8333
29.78/0.8393
29.82/0.8403
70
23.1220.6398
23.03/0.6297
23.19/06-406
26.9600.504
28.25/0.7988
1400.9077918
28. 28.27/0.5000
28.35/0.0028
30
30.330 8417
30.770088588
3062200854
85917008590
31.72808730
31.720.8716
31.76/0.8366
31.780.8744
CKodak24
27.2800.5722
27.6330.7718
27.5480
78.94079977
29.49/88499
8143
29 29.5308115
29.57/0.8169
70
24
1448400.660 6890
24.90166912
24.88016890
27. 79.59997703
28. 10.108
27.990.7688
28. 140.77211
28.17/0.7731
30
29.220 00/22/08336 8378
29.7210.8556
29.51/0
30.3510.8665
30.63/0.8740
30.61/0.8733
30.65/0.8749
30.67/0.5758
CBSD68
26.06/0.7378
26-18/0.7600
28.01/0.7996
28.34V0.800
28.361)8107
23.7010.6548
23. 23.8660.6626
23.80/0.6584
26.58/0.7433
26.91/0.7577
26.81/0.5511
26.92/0.7574
26.96/0.7391
We carry out experiments on three color and three grayscale noisy image datasets. Specifically, the datasets are obtained by adding AWGN with the levels of 30, 50, and 70 to the color and grayscale version of BSD68, Kodak24 and McMaster, respectively. In brief, BSD68 contains 68 images which are frequently used for measuring image denoising performance, Kodak24 contains 24 images captured by film cameras, and McMaster contains 18 images with statistics closer natural images.
For comparisons, we choose seven representative denoising methods, iee,, BM3D 100, DnCNN (50),
FFDNet (51), CLEARER [2]. RNAN [53), SADNet [6] and DeamNet [32]. We call the python

---

Table 5: Quantitative results synthetic grayscale noise image datasets.
Dataset
BM3D
DoCNN
FFDNet
CLEARER
SADNat
RNAN
DeamNet
MSANet (Ours)
30
29. 39.4500.1151
19.80088119
29 29,88008299
30.2900.8491
30,92/0 8644
30.92/0.8629
30,9400 8656
30.96/088661
GM.Master
26.22007218
26.24/0.281
26.46/0.7319
28.31/0.7945
28.61/0.8052
28.57/080014
18.65/0.8070
28.68/0.0022
70
23.78/0.6517
23.63/0.6682
23.6100.6166
26.8300.427
27.18/0.6666
27.06/0.7526
27.200.7616
27.22/0.7620
30
28.71/0.7854
29.2180.7946
29.15818027
29.49/088132
29,870088388
29. 29.89918208
29 29.90/08241
29.91/0.8348
GKodak24
50
066222006990
26.52/0.7190
26.52/0.7177
27.27/0.411
27.77/0.7559
27.73/0.7494
27.79/0.7567
27.81/0, 7554
70
34.3770.6393
24.31806647
24,280064199
26.12N16931
26,51A1.0090
26,42/16999
26.53007107
26.54/0
30
27.43/0.7721
27.96/0.7762
27.8910.7982
28.27/0.8112
28.58/0.8165
28.59/0.81:00
28. 15.59/0.8116
28.61/0.8174
GBSD68
50
24.5000.7715
25.19/0.6826
25,19/0.6909
26.09/0.7295
26.5010.7382
26/46/0.7333
26 06.5000.739
26.51/0.3393
23.07/0.5985
23.04/0.6107
22.98/0.5942
25.03/0.6734
25.23/0.6828
25.16/066736
25.23/0.6831
25.25/0.6526
library for the evaluation of BM3D, and the pretrained models, provided by authors or retrained by us, for the evaluations of the other methods.
reports the quantitative results on color image denoising, which shows that MSANet achieves the highest PSNR and SSIM values. Taking the noise level of 70 as an example, our method can achieve PSNR gains about 0.03 ~ 0.27dB, and SSIM gains about 0.0010 ~ 0.0111 over the state- of-the-art methods, i..,, SADNet, RNAN, and DeamNet. T the quantitative results on grayscale image denoising. From the table, one could observe that MSANet achieves the highest PSNR values, and outperforms the other methods about 0.01 3.59dB w.r.t. PSNR.
4.4 Analysis Experiments
The ablation study is conducted to demonstrate the significance of utilizing the within-scale charac- teristics (WSC) and the cross-scale complementarity (CSC). As shown in Table "ED" and "ResB" are with homologous architectures, which use uniform Identity Mapping and Residual Block to process different scale features, respectively. As using AFeB and AMB alone cannot exploit WSC well, "AFeB"/"AMB" and over "ResB" and "AFuB", respectively, When using AFeB and AMB together, "AFeB+AMB" and MSANet (i.e., "AFcB+AMB+AFul "AFuB", verifying our claim on the role of AFBB+AMB w.r.t. WSC. Furthermore, thanks to CSC through AFuB, "AFuB" and MSANet (i.e, "AFeB+AMB+AFUB") "ResB" and "AFeB+AMB", respectively. In summary, the ablation study not only demonstrates the significance of utilizing WSC and CSC, but also shows the effectiveness of our proposed solution.
only slightly improve the performance
significantly improve the performance over "ResB" and
significantly better than
Table 6: Ablation study CMcMaster with the noise level of 30. "ED" denotes the encoder-decoder architecture with skip connections. "ResB" denotes to substitute the blocks in MSANet with the residual blocks. "AFeB", "AMB", "AFuB", "AFcB+AMB", "AFcB+AFuB" and "AMB+AFuB" denote to use the corresponding blocks on the basis of "ResB".
Ablations
ED
ResB
AFeB
AMB
AFuB AFeB+AMB
AFeB+AFuB
AMB+AFuB
MSANet
PSNR
31.70
31.93
31.94
31.94
32.01
31.98
32.04
32.03
32.07
SSIM
0.8801
0.8851
0.8854
0.8851
0.8864
0.8860
0.8869
0.8866
0.8876
To further demonstrate the significance of the within-scale characteristics, and the effectiveness of our solution in exploiting it, we show the qualitative comparisons on the intermediate multi-scale features before (i.e., without) and after (i.e., with) our subnetworks in MSANet. From the Fig. one could observe that our subnetworks could well exploit the within-scale characteristics, i.e., preserving the indispensable details while filtering the unpleasant noises for high-resolution features (i.e., 2nd and 3rd columns), and capturing rich contextual information for low-resolution features (i.e., 4th and 5th columns). Besides, the different scale features show varying characteristies and significant cross-scale complementarity. which would be further exploited by our AFuB.
5 Conclusions
In this paper, we proposed MSANet with three neural blocks, i.e., AFeB, AMB, and AFuB, for single image denoising. Different from existing multi-scale architectures, MSANet considers not only the cross-scale complementarity but also the within-scale characteristics, thus boosting the recovery performance verified in experiments. As this work could be regarded as finding the missing piece

---

Figure 6: Qualitative comparisons on the intermediate multi-scale features in MSANet. The top left is the noise image, and the bottom left is the corresponding clean image. From left to right, the feature resolution varies from high to low. The top row is the features before (i.e., without) subnetworks, and the bottom row is the features after (i.e., with) subnetworks.
of multi-scale architecture design, we will explore other solutions to simultaneously exploit the within-scale characteristics and the cross-scale complementarity, and investigate their effectiveness in broader tasks such as deblur, segmentation.
Broader Impact Statement. MSANet is specifically designed architecture for supervised single image denoising, which requires intensive labor to collect a lot of noisy-clean image pairs, and thus has the potential make more opportunities for employment. However, MSANet is general neural network and might be trained with uncertain data and used for uncertain purposes, such as watermark removal, which might prejudice the rights of others. Besides, MSANet involves novel idea of multi-scale architecture design and might be used to design new networks for uncertain purposes. Moreover, the training and running of the model consume lot of electricity causing carbon emission.
Acknowledgements
This work was supported in part by NSFC under Grant U21B2040, U19A2078, and 61836006; and part by Sichuan Science and Technology Planning Project under Grant 2022YFQ0014.
References
[1] AbdeIrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 1692-1700, 2018.
[2] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: dataset and study. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh. July 2017.
[3] Josue Anaya and Adrian Barbu. Renoir-a dataset for real low-light image noise reduction. Journal of Visual Communication and Image Representation, 51:144-154, 2018
[4] Saeed Anwar and Nick Barnes. Real image denoising with feature attention. In Int. Conf. Comput. Vis., pages 3155-3164, 2019.
[5] Xiaochun Cao, Wenqi Ren, Wangmeng Zuo, Xiaojie Guo, and Hassan Foroosh. Scene text deblurring using text- text-specific multiscale dictionaries. IEEE Trans. Image Process., 24(4):1302- 1314, 2015.
[6] Meng Chang, Li, Huajun Feng, and Zhihai Xu. Spatial-adaptive network for single image denoising. In Eur. Conf. Comput. Vis., pages 171-187. Springer, 2020.
10

---

[7] Jingdong Chen, Jacob Benesty, Yiteng Huang, and Simon Doclo. New insights into the noise reduction wiener filter. IEEE Trans. on Audio, Speech, and Language Process.. 14(4):1218- 1234, 2006.
[8] Tao Chen, Kai-Kuang Ma, and Li-Hui Chen. Tri-state median filter for image denoising. IEEE Trans. Image Process. 8(12):1834-1888 1999.
[9] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Color image de- noising via sparse 3d collaborative filtering with grouping constraint in suminance-hromminncee space. In JEEE Ins. Conf. Image Process., volume 1, pages I-313. IEEE, 2007.
[10] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8):2080- 2095, 2007.
[11] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming- Hang- Yang. Multi-scale boosted dehazing network with dense feature fusion. In IEEE Conf. Comput. Vis. Pattern Recog., June 2020.
[12] Yuanbiao Gou, Boyun Li, Zitao Liu, Songfan Yang, and Xi Peng. Clearer: Multi-scale neural architecture search for image restoration. Adv. Neural Inform. Process. Syst. 33, 2020.
[13] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm min- imization with application to image denoising. In JEEE Conf. Comput. Vis. Pattern Recog., pages 2862 2869, 2014.
[14] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In IEEE Conf. Comput. Vis. Pattern Recog. June 2019.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 770-778, 2016.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[17] Boyun Li, Yuanbiao Gou, Shuhang Gu, Jerry Zitao Liu, Joey Tianyi Zhou, and Xi Peng. You only look yourself: Unsupervised and untrained single image dehazing neural network, Int. Comput. Vis., 12965):154-11777, 2021.
[18] Boyun Li. Yuanbiao Gou, Jerry Zitao Liu, Hongyuan Zhu, Jocy Tianyi Zhou, and Xi Peng. Zero-shot image dehazing. IEEE Trans. Image Process., 29:8457-8466, 2020.
[19] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Int. Conf. Comput. Vis. Worksh. October 2021.
[20] Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. In Ady. Neural Inform. Process. Syst., pages 1680- 1689, 2018
[21] Jinyuan Liu, Xin Fan, Ji Jiang, Risheng Liu, and Zhongxuan Luo. Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion. IEEE Trans. Circuit Syst. Video Technol., 2021.
[22] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-based multi-scale network for image dehazing. IEEE Conf. Comput. Vis. Pattern Recog., pages 73 7314-7323, 2019.
[23] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint
2016.
[24] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human seg- mented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Int. Conf. Comput. Vis., volume 2, pages 416-423. IEEE, 2001.

---

[25] Yiqun Mei, Yuchen Fan, Yulun Zhang, Jiahui Yu, Yuqian Zhou, Ding Liu, Yun Fu, Thomas S Huang, and Humphrey Shi. Pyramid attention networks for image restoration. ar preprint arXiv:2004. 13824, 2020.
[26] Chong Mou, Qian Wang, and Jian Zhang. Deep generalized unfolding networks for image restoration. IEEE Conf. Comput. Vis. Pattern Recog.. 2022.
[27] Chong Mou and Jian Zhang. Graph attention neural network for image restoration. In Int. Conf. Multimedia and Expo, pages 1-6, 2021.
[28] Seonghyeon Nam, Youngbae Hwang, Yasuyuki Matsushita, and Seon Joo Kim. A holistic approach to cross-channel image noise modeling and its application to image denoising. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1683-1691, 2016.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Adv. Neural Inform. Process. Syst., 32:8026-80333 2019.
[30] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 1586-1593 2017.
[31] Tobias Plötz and Stefan Roth. Neural nearest neighbors networks. Adv. Neural Inform. Process. Syst. 31:1087-11088, 2018
[32] Chao Ren, Xiaohai He, Chuncheng Wang. and Zhibo Zhao. Adaptive consistency prior based deep network for image denoising. In IEEE Conf. Comput. Vis. Puttern Recog., pages 8596- 8606, 2021.
[33] Wenqi Ren, Jinshan Pan, Hua Zhang, Xiaochun Cao, and Ming-Hsuan Yang. Single image dehazing via multi-scale convolutional neural networks with holistic edges. Int. J. Comput. Vis., 128(1):240-2555 259, 2020.
[34] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In IEEE Conf. Comput, Vis. Pattern Recog., June 2019.
[35] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In Int. Conf. Comput. Vis., pages 4539-4547, 2017.
[36] Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In Int, Conf. Comput. Vis., pages 239-846. IEFF, 1998.
[37] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 7794-7803, 2018
[38] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In IEEE Conf. Comput. Vis. Puttern Recog., pages 17683-17693, June 2022.
[39] Wenhao Wu, Yuxiang Zhao, Yanwu Xu, Xiao Tan, Dongliang He, Zhikang Zou, Jin Ye, Yingying Li, Mingde Yao, Zichao Dong, et al. Dsanet: Dynamic segment aggregation network for video-level representation learning. In ACM Int. Conf. Multimedia, pages 1903-1911, 2021.
[40] Qi Xie, Qian Zhao, Deyu Meng, Zongben Xu, Shuhang Gu, Wangmeng Zuo, and Lei Zhang. Multispectral images denoising by intrinsic tensor sparsity regularization. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1692-1700, 2016.
[41] Jinjun Xu and Stanley Osher. Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising. IEEE Trans. Image Process. 16(2):534-544, 2007.
[42] Jun Xu, Hui Li, Zhetong Liang, David Zhang, and Lei Zhang. Real-world noisy image denoising: new benchmark. arXiv preprint arXiv: SEiv:1804.02666 2018.
[43] Jun Xu, Lei Zhang, and David Zhang. A trilateral weighted sparse coding scheme for real- world image denoising. In Eur. Conf. Compus. Vis., pages 20-36, 2018.
12

---

[44] Ruikang Xu, Zeyu Xiao, Jie Huang, Yueyi Zhang, and Zhiwei Xiong. Edpn: Enhanced deep pyramid network for blurty image restoration. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 413-423, 2021.
[45] Hongyu Yang, Di Huang, Yunhong Wang, and Anil K. Jain. Learning face age progression: A pyramid architecture of gans. In IEEE Conf. Comput. Vis. Pattern Recog,, June 2018.
[46] Ke Yu, Xintao Wang. Chao Dong, Xiaoou Tang, and Chen Change Loy. Path-restore: Learning network path selection for image restoration. IEEE Trans. Pattern Anal. Mach. Intell., 2021.
[47] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5728-5739, June 2022.
[48] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In IEEE Conf. Comput. Vis. Pattern Recog., pages 14821-14831, 2021.
[49] He Zhang and Vishal M Patel. Density-aware single image de-raining using a multi-stream dense network. In IEEE Conf. Comput. Vis. Pattern Recog., pages 695-704, 2018
[50] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Trans. Image Process., 26(7):3142-3155 2017.
[51] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for cnn-based image denoising. IEEE Trans. Image Process. 27(9):4608 4622, 2018.
[52] Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic Imaging, 20(2):02301,, 2011.
[53] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. In Int. Conf. Learn. Represent. 2019.
[54] Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, and Thomas Huang. When awgn-based denoiser meets real noises. AAAI, volume 34, pages 13074-13081, 2020.
[55] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In JEEE Conf. Comput. Vis. Pattern Recog., pages 9308-9316, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Broader Impact Statement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL))? [Yes] See the URL of Github in Abstract.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] is detailed in the source code.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No] We use the public datasets in our experiments.
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [N/A] We conduct experiments the public datasets, which are widely used for research. The discussion of personally identifiable information offensive content could be referred to the data publisher.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "uoEYQWc2jOY", "reviewer": "Reviewer_D7k7", "summary": "Overall, this paper aims to tackle the task of image denoising based on the drawback (not exploring the internal information of each scale in the network) of multi-scale networks. Specifically, the authors propose three modules to extract feature maps, multi-scale construction and feature aggregation to run a noisy image. Extensive experimental results are shown in the paper.", "questions": "[-] My problem is mainly how can the authors prove that the existing approach does not make use of the information within the scale?\n\n[-] The authors' proposed approach does not seem to be innovative.", "limitations": "Limitations have been illustrated.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the architecture of the paper is clear and the writing is good, but the theoretical and methodological parts are not enough to meet me.\n\nStrengths\n\n[+]  The authors seem to propose an effective image denoising solution, described in the motivation, related work, methodology and experiments.\n\nWeaknesses\n\n[-] First of all, the title does not show the author's intention, which is a bad case (No expressed motivation).\n\n[-] The abstract description starts by depicting a big problem, namely the general weakness of multi-scale networks, but focuses on a simple application.\n\n[-] The introduction and related works ignore Transformer-based models.\n\n[-] Figure 1 does not convey the author's intention, which is confusing. Why not use the feature map presentation to illustrate that there is a problem with information extraction within the scale?\n\n[-] Do not insert ‘so on’ to follow ‘such as’.\n\n[-] The biggest problem is that the motivation is not valid and I don't see cases to illustrate the problems with multi-scale networks nowadays.\n\n[-] In addition, the solution is essentially adding convolutional layers (adaptive) between scales, does this solve the problem?\n\n[-] What worries me is that the methods of CVPR'22 have been released, so why are there no relevant experiments to compare these methods?\n\n[-] relu---> ReLU, pytorch--->PyTorch.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rPrT6V62Qq4", "reviewer": "Reviewer_H79u", "summary": "Different from existing works treat different scale features equally, in this paper, the authors reveal the different scale features show varying characteristics and should be processed by scale-speciﬁc rather than homologous structures. To simultaneously embrace the within-scale characteristics and cross-scale complementarity of multi-scale features, the authors propose a novel Multi-Scale Adaptive Network (MSANet) for single image denoising. Speciﬁcally, three neural blocks are designed, i.e., adaptive feature block (AFeB) for adaptively sampling and ﬁltering features; adaptive multi-scale block (AMB) for expanding the receptive ﬁeld and adaptively aggregate multi-scale features; adaptive fusion block (AFuB) for adaptively fusing the multi-scale features with varying characteristics. Ablation studies indicate the blocks are eﬀective, and comparison experiments show MSANet achieves better performance than alternatives.\n", "questions": "1. In the paper, “adaptive” is frequently used to describe the proposed network as well as the three neural blocks, but itself is not explicitly discussed. Some discussions are recommended for a better understanding to this work.\n2. AFeB is designed to adaptively sampling and weighting the input features, which is highly expected for ﬁne-grained features. However, why adaptively sampling and weighting the ﬁne-grained features is good for the denoising task to preserve the image details and filter unpleasant noise?\n3. As Table 6 suggests, the performance gains of using either AFeB or AMS alone are slight. However, why using them together could bring significant performance improvements? Please give some interpretations.\n", "limitations": "Yes, the authors have discussed the related problems.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors reveal the different scale features show varying characteristics and should be processed by scale-speciﬁc structures for this first time. Based on the observation, MSANet is proposed to utilize the within-scale characteristics and cross-scale complementarity of multi-scale features. Extensive experiments are conducted on both three real and six synthetic noisy image datasets to show its superiority.\n2. Three neural blocks are designed by considering the within-scale characteristics and cross-scale complementarity of multi-scale features. Ablation studies indicate these blocks are eﬀective.\n3. In general, the paper is well written. The related works are clearly illustrated, the shortcomings of existing multi-scale networks are analyzed, and accordingly MSANet is proposed by solving the shortcomings. The framework is clear and easy to understand, and the experimental results are explained in detail.\n\nWeaknesses:\n1. The idea of taking advantage of within-scale characteristics and cross-scale complementarity is not limited to denoising task, but is a general idea of multi-scale architecture design. Therefore, exploring and verifying this idea in more tasks and areas would make this work be more significant.\n2. MSANet contains more parameters than most of the baselines due to multiple subnetworks for multi-scale features.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_PbaoqR0LGz", "reviewer": "Reviewer_9KYY", "summary": "To exploit the potential of the within-scale characteristics and cross-scale complementarity of multi-scale features, three elaborate neural blocks and a novel Multi-Scale Adaptive Network (MSANet) are proposed for single image denoising. Specifically, AFeB and AMB are designed by taking the within-scale characteristics of multi-scale features into consideration, AFuB is designed to exploit the cross-scale complementarity of multi-scale features. Three neural blocks are combined to be the scale-wise subnetwork in MSANet by adapting to the feature characteristics of the corresponding scale. Ablation studies demonstrate the effectiveness of the proposed neural blocks and network. Sufficient comparison experiments on three real and six synthetic noisy image datasets compared with 12 baselines show the advantages of the proposed method.", "questions": "1. Although different-scale features are processed by different subnetworks, and the subnetworks in the finest and coarsest scale are clearly scale-specific. However, it’s difficult to understand the scale-specific designs in the two middle subnetworks. More clear interpretations are needed for a better understanding.\n2. As shown in Table 6, using AFeB and AMS together could signiﬁcantly improve the performance. However, using either AFeB or AMB alone slightly gains the performance over ResB. Why is that? Some clear explanations are needed for a better understanding.\n3. As shown in Table 7, although MSANet contains more parameters, the running time and the FLOPs are not much. Why does this happen? Please give some explanations.\n4. Some typos, e.g., “disorderly” -> “disordered” in the paragraph of Adaptive Fusion Block (AFuB).\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A. Strengths\n1. The idea of simultaneously exploiting the within-scale characteristics and cross-scale complementarity of multi-scale features is interesting and reasonable. The accordingly proposed MSANet incorporates them into multi-scale architecture design, and shows impressive performance on single image denoising task.\n2. The design of AFuB is novel and ingenious. Taking the disordered ﬁne-grained image details in high-resolution features and the coarse-grained image context in low-resolution features as input to adaptively sample, weight and transfer the ﬁne-grained image details into the coarse-grained image context.\n3. Sufficient experiments are conducted to demonstrate the effectiveness of the proposed designs and the advantages over the previous works. Moreover, the supplementary material also shows more details and results together with the main paper.\n\nB. Weaknesses\n1. Since the cross-scale complementarity and the within-scale characteristics are not only restricted to single image denoising task. Therefore, the designs will be more persuasive if they could also achieve improvements in other image restoration tasks.\n2. Compared with the three real and the three synthetic color noisy image datasets, the performance improvements over the baselines on the three synthetic grayscale noisy image datasets are less significant.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AmJAp4s1Use", "reviewer": "Reviewer_rLGu", "summary": "This paper proposes Multi-Scale Adaptive Network (MSANet) for single image denoising with three blocks: Adaptive Feature Block (AFeB), Adaptive Multi-scale Block (AMB) and Adaptive Fusion Block (AFuB). The network architecture design explores the within-scale and cross-scale characteristics of multi-scale networks. Extensive experiments demonstrate that these presented modules gain better performance than the baseline network and some other denoising methods on several real or synthetic datasets.\n", "questions": "1) In Fig.1, the authors claim that the robustness of high-level and low-level scales is different. What is the definition of robustness there? Is there any experiment or reference to support this statement?\n\n2) How does the motivation of exploring within-scale characteristics cause the design of the network architecture? The proposed three modules seem to be only simple combinations of residual blocks and deformable/dilated convolution. The performance improvements may come from the additional parameters rather than the designed architecture.\n\n3) How do we combine the features weighted by channel attention and the features weighted by spatial attention in AMB？ The detailed process seems to be omitted.\n", "limitations": "None\n", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+This paper is well-written, and the proposed architecture is straightforward to follow.\n\n+The summary of the difference between low-resolution and high-resolution scales in Fig.1 is valuable, and the motivation to explore the within-scale characteristics in architecture design is reasonable.\n\nWeaknesses:\n\n-The relationship between the motivation of exploring the within-scale characteristics and the design of the three proposed modules is farfetched. There is still a lack of related experiments or visualization results that prove that the presented modules can utilize the within-scale characteristics well.\n\n-Comparison with the latest SOTAs such as MPRNet [1], Uformer [2], and DGUNet [3] is missing. The performances reported in this paper seem to be much worse than these previous denoising methods.\n\n-The ablation study in Section 4.4 is limited to demonstrating the different characteristics of each module. Adding the proposed modules seems only to gain some improvements over the baseline network, but the in-depth analysis is missing. I can not get any intuitive conclusions from this part.\n\n[1] Zamir, Syed Waqas, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)\n\n[2] Wang, Zhendong, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)\n\n[3] Mou, Chong, Qian Wang, and Jian Zhang. Deep Generalized Unfolding Networks for Image Restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "64WafatBSlw", "reviewer": "Reviewer_9gQL", "summary": "The paper investigates existing multi-scale methods, and discovers the within-scale characteristics of multi-scale features are ignored. Therefore, the paper reveals this missing piece for multi-scale architecture design, and accordingly proposes a novel Multi-Scale Adaptive Network (MSANet) by simultaneously exploiting the within-scale characteristics and the cross-scale complementarity. MSANet uses AFeB, AMB to build different subnetworks corresponding to diﬀerent scales for exploiting the within-scale characteristics, and AFuB to fuse the multi-scale features with varying characteristics from coarse to ﬁne for exploiting the cross-scale complementarity. Extensive experiments on both three real and six synthetic noisy image datasets show the effectiveness of the proposed designs and the advantages over the previous methods. Moreover, supplementary material shows more results together with the main paper.", "questions": "(1) Both AFeB and AFuB are based on deformable convolution to adaptively sample and weight the features, but one is used to adaptively select details and ﬁlter noises, the other is used to fuse the multi-scale features with varying characteristics. What caused this crucial difference? Why adaptively sampling and weighting the feature is important for exploiting the within-scale characteristics and the cross-scale complementarity?\n\n(2) Why using AFeB and AMS together could significantly improve the performance while only using one of them slightly improve the performance?\n\n(3) In Table 7, only three baselines are investigated w.r.t. the model complexity, which is inadequate. More baselines should be investigated and compared.", "limitations": "yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n(1) Simultaneously taking the within-scale characteristics and the cross-scale complementarity into multi-scale architecture design increases the performance of single image denoising task. The reviewer agrees that the within-scale characteristics is significative for multi-scale architecture and single image denoising.\n\n(2) The motivations of the proposed designs, i.e., MSANet and the three neural blocks, are clear and convincing. The experimental results show the effectiveness and robustness of the proposed designs.\n\n(3) The authors conduct adequate experiments on both synthetic and real noise image datasets, and investigate the effectiveness of the proposed three blocks as well as the model complexity. Overall, the experiments are complete.\n\nWeaknesses\n\n(1) The proposed MSANet is not competitive in parameter numbers as there are different subnetworks corresponding to different scale features. This limits the use of the model on memory-constrained devices. In contrast, the running time and FLOPs are competitive due to multi-resolution features.\n\n(2) Using AFeB and AMS together could significantly improve the performance, while using either AFeB or AMB alone slightly improve the performance, i.e., they are bound together for using. This limits the separate use of the two blocks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept