Enhanced Latent Space Blind Model for Real Image Denoising via Alternative Optimization
Authors: Chao Ren
Yizhong Pan
Jie Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Guidance Constraint
Latent Space
Self-Correction
Blind Model
Real Image Denoising
Alternative Optimization

Abstract: Motivated by the achievements in model-based methods and the advances in deep networks, we propose a novel enhanced latent space blind model based deep unfolding network, namely ScaoedNet, for complex real image denoising. It is derived by introducing latent space, noise information, and guidance constraint into the denoising cost function. A self-correction alternative optimization algorithm is proposed to split the novel cost function into three alternative subproblems, i.e., guidance representation (GR), degradation estimation (DE) and reconstruction (RE) subproblems. Finally, we implement the optimization process by a deep unfolding network consisting of GR, DE and RE networks. For higher performance of the DE network, a novel parameter-free noise feature adaptive enhancement (NFAE) layer is proposed. To synchronously and dynamically realize internal-external feature information mining in the RE network, a novel feature multi-modulation attention (FM2A) module is proposed. Our approach thereby leverages the advantages of deep learning, while also benefiting from the principled denoising provided by the classical model-based formulation. To the best of our knowledge, our enhanced latent space blind model, optimization scheme, NFAE and FM2A have not been reported in the previous literature. Experimental results show the promising performance of ScaoedNet on real image denoising. Code is available at https://github.com/chaoren88/ScaoedNet.

Enhanced Latent Space Blind Model for Real Image Denoising via Alternative Optimization
Jie Huang College of Electronics and Information Engineering, Sichuan University, Chengdu, China chaoren@scu, www.ssccueeedeu......eeeee....ee en, (panyizhong, huangjiechn, @estu. scu edu. cn
Chao Ren"
Yizhong Pan
Abstract
Motivated by the achievements in model-based methods and the advances in deep networks, we propose novel enhanced latent space blind model based deep unfolding network, namely ScaoedNet, for complex real image denoising. It is derived by introducing latent space, noise information, and guidance constraint into the denoising cost function. A self-correction alternative optimization algorithm is proposed to split the novel cost function into three alternative subproblems, i.e., guidance representation (GR), degradation estimation (DE) and reconstruction (RE) subproblems. Finally, we implement the optimization process by a deep unfolding network consisting of GR, DE and RE networks. For higher performance of the DE network, novel parameter-free noise feature adaptive enhancement (NFAR) layer is proposed. To synchronously and dynamically realize internal- external feature information mining in the RE network, a novel feature multi- modulation attention (FM2A) module proposed. Our approach thereby leverages the advantages of deep learning, while also benefiting from the principled denoising provided by the classical model-based formulation. To the best of knowledge, our enhanced latent space blind model, optimization scheme, NFAE and FM A have not been reported in the previous literature. Experimental results show the promising performance of ScaoedNet on real image denoising. Code is available at www.cha.c/eeeeoco ScaoeedNet
Introduction
Image denoising is important for various computer vision tasks, and numerous outstanding methods
32, 20]. The flttriig-based methods are representative, such as the classic local low-pass filtering methods: mean filtering and median filtering. In recent decades, many non-local filtering methods 13 16 25 have achieved great success. However, their results may suffer from blurring artifacts caused by their block-wise operations.
have been developed [13 17 46
18
36
26
49 54
27
The model-based methods largely depend on image priors, e.g., graph-based (34), sparsity 3
30 (7). local smoothing (33 45]. and low-rank [17 46
43 55 priors. Pang et al. [34]
interpreted neighborhood graphs of pixel patches discrete counterparts of Riemannian manifolds, providing insights into several fundamental aspects of graph Laplacian regularization for denoising. By introducing three weight matrices into the data and regularization terms of the sparse coding framework, Xu et al. [7]] developed a denoising method to characterise the statistics of real noise and image priors. Gu et al. [17 proposed an image restoration method by studying the weighted nuclear norm minimization problem. However, most of these methods focus on removing the additive white Gaussian noise (AWGN), without taking the real noise and external guidance into consideration.
The Barning-based methods are promising ways for addressing the image denoising problem. Gener- ally, it can be divided into traditional and deep network-based methods. The deep network-based
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Noisy
CBDNet
InvDN
DANet
DeamNet
ScaoedNet
Figure I: Denoising results on real noisy image from SIDD dataset [2]. Compared with CBDNet
InvDN 277 DANet 49 and DeamNet [36]. our ScaoedNet achieves better results.
methods have superiority in denoising due their excellent modeling ability. For example, the noise map is introduced as the input for removing noise at different levels in [53]. The studies
[52] 39 31 show that it is feasible to learn a single model for blind Gaussian denoising, but these
blind models may be over-fitted to AWGN and fail to handle real noise [19]. Different from the previous empirically designed networks, the deep unfolding-based methods
[5 24 2T were
proposed by implementing traditional methods via deep networks. For instance, an advanced image denoising network was developed by solving a fractional optimal control problem in (1
The above methods generally aim at removing AWGN with regular distributions. However, real image denoising a more challenging task since the distribution of real noise is more complex. For real image denoising, Guo et al. [19] adopted denoising network. A apatiil-adaptive denoising network was proposed in [10] for efficient blind noise removal. Byun et al. 7 proposed a fast noise estimation network and an efficient blind-spot network for
Gaasssann noise model and presented a blind
noise removal. Anwar al. (5] incorporated feature attention into denoising and developed a one-stage real image denoising network. Cheng el al. [12 improved the
image denoising performance by plugging the subspace attention module. Liu al. (28 presented an invertible denoising network for real image denoising. Based on the adaptive consistency prior, Ren et al. (36) proposed a novel model-based denoising method inform the design of the network for both synthetic and real denoising. Yue et al. [4 proposed unified framework, namely dual adversarial network, to simultaneously deal with the real noise removal and noise generation tasks.
In this paper, motivated by the advances in deep networks and relying on the rich body of the model-based methods, we propose novel enhanced latent space (LS) blind model based deep unfolding network for real image denoising. As shown in Fig. 1, the whole architecture of ScaoedNet achieves the best promising visual performance when compared to other denoising methods. The main contributions of our method are as follows:
For the theoretical novelty, novel enhanced model-based denoising cost function is proposed based on LS, noise information, and guidance constraint (GC). Then, a self-correction (SC) based alternative optimization algorithm is proposed to split the cost function into three alternative subproblems (i.e., guidance representation (GR), degradation estimation (DE) and reconstruction (RE) subproblems).
To merge the power of the model-based framework with the recent advances in deep learning for real image denoising, a novel interpretable network (ScaoedNet) is designed via implementing the optimization process by a deep network, which consists of the GR, DE, and RE networks.
An effective DE network with novel noise feature adaptive enhancement (NFAE) layer is proposed. NFAE does not add extra parameters to the DE network, and leads to better noise estimation results.
To enable internal-external feature mining, novel feature multi-modulation attention residual block (FM2ARB) is proposed in the RE network, which is mainly based on feature self-modulation (FSM) and degradation iiternalmoddllaton (DEM). FM2 ARB can dynamically fuse the encoded degradation representation and the image feature representation, leading to higher performance.
2 Proposed Model-based Denoising Method
2.1 Enhanced Latent Space Blind Model for Denoising
Let y E " be a noisy image with e channels and n pixels, x E R be its clean version, and (p()) be the regularizer weighted by p. The traditional madel-based denoising method can be given by
x = arg min H(x. y) + (x)(x).
x
where H(x, y) is the data fidelity which usually set to yy x x|3 (p()) can any image regularizers,
e.g. sparsity prior [ 30 47). local smoothing prior [33 45 and low-rank prior [1] 44 46 43 etc.

---

Stage
Stage
Stage
esHl>WW
BIX
HI
Uer-NNet
u-Net
uf
Ie-et
(k-1)
MIX 2
cxH<W
Latrunt
Module
Module
Module
gg
9
(--1
Z-Net
Z-Net
N-Net
Noisy Input
Clean Output
E( 1r u 10) (y . ) g" G(u Oah n 2(6") y. u
00 U( y.u O Die (.,).
3
Ory)
Enhanced
Input
Festure Map
3)
Came
ReLL
Input
Output
Cont
Output
ini Net
u-Net
ourpor clumnels
PS-1/2
Cooy
Z-Net
Ely)
G-Module
DEM Module
Rel.U
Coan
Fusie
Can
Re
Con
ReLU
Conn
ReLu
Com
EMFA Module
PSPARH
ESFARR
FXPARB
Cour
+
Cony
FMPARB
DEM
Cony
Weight
ReLu
Tensur
Input
Guidance Information
Output
Figure 2: Architecture of Set alternatively optimizes the guidance representation (GR), degradation estimation (DE) and reconstruction (RE) subproblems in latent space (LS), consisting of LS encoding module (E), LS decoding module (D), G-modules, initial DE network (Net,,Net,, DE
networks (u-Nets), and RE networks (z-Nets).
Analysis and Enhancement. We perform an in-depth analysis of Eq. ① in the following:
1) H(x,y) y) in Eq. I only utilizes the squared error distance ly x between y and x, which mainly derived from AWGN. Inspired by the latest unfolding works [36 6]. we parameterize
the metric ly x|2 by a pre-specified high-dimensional I.S encoding function E = Rm
(m > c), to obtain high-dimensional image embeddings z = F(x) for better performance, instead of limiting the objective the squared error in nov-dimensional image space. It allows us directly model the image formation process in LS 36 6), and to integrate LS priors into the prediction.
2) In real image denoising, noise complex and unknown, and thus blind model is needed. Since the noise information is largely ignored in Eq. (I). it is difficult to well address the real noise removal problem. By introducing the real noise information and LS, we can learn more general distance measure H(z y) (u is the unknown real noise map), and further replace the regularizer by T2(2) + (u) (regularizers for z and u weighted by r and v) respectively, where T, 7) > 0).
3) Compared with the regularizer based general image statistical characteristics, better results can be obtained if certain specific information of the given image is exploited to guide the denoising process. Let g be certain sophisticated guidance information. Then, we improve ő(z) by specific constraint (zz, g) for further performance enhancement, and we call it the GC scheme.
Proposed Model-based Denoising Method. Before the introduction of our denoising framework, we first introduce the real noise model. Different from AWGN, real image noise generally complex and signal-dependent. Let
p(x) E n be the irradiance image of RAW pixels, and be the mapping function from RGB to RAW. According to [19, the following heteroscedastic Gaussian reasonable approximation for real noise distribution: nxx,) n. (x,) + n. (0, u u²())) (x.)). Specifically, n, is a stationary noise component with variance ue. and n, is a signal-dependent noise component with space- variant variance x,. Then, y q(x, + n(x,)), where q is the mapping function from RAW to RGB. Inspired by [19], the real image noise variance can be formulated by
u (x,x)) x 2 2 where U, and ue uniformly sampled from the ranges of [0, 0.16) and [0, 0.06). For more details, please refer to [19]. Then, the noise map is defined
(2)
/ ( e R" which is signal-dependent

---

and space-variant with the same size of x, instead of scalar. Next. according to the 'Analysis and Enhancement" subsection, the proposed enhanced denoising method can be given as
{2, i arg min H(z y) + TT g) + + s.t., x D z.u
(3)
where D :R R is the LS decoding function that maps the latent embeddings & to st.
2.2 Self-Correction (SC) Alternative Optimization
According to the widely used traditional alternative optimization algorithm g
38 29 in image
restoration, Eq. (3) can be solved by splitting into two alternative subproblems with respect to (reconstruction (RE) subproblem) and u (degradation estimation (DE) subproblem):
z (++1) arg min M(z n(),, ,) T u arg min H( (++1), u, - (o(,), (4)
where: is the iteration number. However, the last estimates of z and u are not used in Eq. (4) For higher performance, it is better to use the last estimates for SC when estimating and u1 In addition, since the estimated noise map is available during the iterations, it can be reasonable information to guide the reconstruction (i.e., g can be obtained according to ) Thus, instead of uusinn Eq. 4
we use the SC scheme (introducing u and (1)) and specify the form g, and Eq. becomes:
g = G))
(++1)) arg min, H(z, u0),, y) + T (Z, (0) ((z y.g. (())) ((+1)) arg min, H(z y, mp u, u()) = (z(+1)) .y (0),
(5)
where G(:) is the guidance information generator, (()) becomes the joint constraint GC and SC for z, and (-) becomes the joint constraint of noise information and SC for u. The two functions Z() and (-) are the optimization solving processes for the and subproblems in Eq. 5
Finally, can be obtained by applying the LS decoding function D to z. Note that ( can be directly estimated by any reasonable DE estimator Unii (i.e., u = Uni (y)), and z (0) can be simply set to E(y). The overall algorithm is called SC alternative optimization based enhanced denoising (SCAOED), and is described in Algorithm of the 'Supplementary Material'.
3 SCAOED Driven Denoising Network
3.1 Implicitly Implement SCAOED Algorithm via Deep Network
It is very challenging to manually design the optimal operators in SCAOED, e.g. E, D, U.Z. Therefore, the deep unfolding scheme is applied to implicitly implement these operators via deep network, leading ScaoedNet. Specifically, we parameterize these operators by introducing learnable parameters θ O E + D. Og, U.n: Ou Oz) ), which are learned in discriminative manner. As shown in Fig. ScaoedNet is an iterative real denoising framework in LS. Overall, LS encoding module, LS decoding module, G-module, initial DE network (ue),, DE network (u-Net), and RE network (z-Net) are used to implement E, D, G, Unni U, and Z, respectively.
Specifically, in order to reduce the computational complexity while increasing the receptive field, the LS encoding module (the learned version of E) is implemented by pixel-shufile layer with factor 1/2 (PSx x 1/2) followed by 3 x 3 Conv layer, which can project the original low-dimensional image space input y E R W (H W n) to the high-dimensional embedding space by F.(,0g)) E R HmxxH/2xW22 (m is set to 64) as initial latent image embeddings 20)), i.e. z00 = ( g) In the meanwhile, (0)) E R HxW is estimated by - (the learned version of mm) to implicitly implement Umn (y). Then, GR is obtained by inputting ()) into the G-module (the learned version of (0) z (0)),, and (o) are fed into z-Net (the learned version of Z) to get 211), After that,
G). Next, y. g y, ( and z are fed into the next u-Net (the learned version of U) obtain ()), Then, g. and z are alternatively updated by the following G-modules, u-Nets, and z-Nets, wheih implicitly implement Eq. ( Finally, (K)) is reconstructed by the LS decoding module (the learned version of D) to obtain the final image space estimate for x, i.e. x (z (K) (O)) where the LS decoding module is implemented by 'Conv-PSXX) X2' layers. Note that the parameters of all u-Nets, G-modules, and Nets are respectively shared. For the reconstruction loss, we adopt the L, loss between the output and the ground-truth image. In addition, we adopt structural similarity (SSIM) loss [5]] Ls to pay attention to image structures. For the noise map estimation loss, we adopt the T,, loss between the estimated noise map and the ground-truth version. Note that the initial DE network is pre-trained

---

CIG-SA Module
M
Poool
APool
Cony
Siennod
Sigmsni
Module
DEM
Output Map
CIG Spetial Feature
APool
Feature Map
Inpat
M.Pool
Concil
Stennii
DEM
Squanekd
Conv
Degradation Representation
MFool
Conv
Sbannel
CIG-CA Module
FSM Module
Figure 3: Architecture the proposed FM2 A module consisting of FSM and DEM modules.
only with the noise map estimation loss. Let (x,- Yolg=1 be N clean-noisy training pairs, @ be the element-wise product, and 77, ? be two positive constants. The total loss function can be given by
N
K-1 N
Σ (K) - 9=1
x,))+
Me @ ( Ill
N
=11==1
where Ug is the estimated noise map the ith DE network for and u, is the ground-truth version. Kigg - 1/- Ogg. Vi is the weight for the ith DE network (uy-s compose the geometric sequence with common ratio and vector a for the noise constraint. For a synthetic training pair with known noise map, Og = 1. For real training pair without known noise map, 0g 0.
ie,, V/ (1- 1)<--/(((K-1 1)). ag is the gth element of the indicator
3.2 Initial Degradation Estimation (DE) Network (Uit--Net and DE Network (u-Net)
Basic Architecture of Uni--Ntt and u-Net. In -met, the input y is first processed by noise feature enhancement module Q consisting of a PSx1/2 layer. four 'Conv-ReLU" layers, and an NFAE layer, to get the enhanced noise feature map. Next, the feature map is projected to the noise map by Conv layer and a PSx2 layer. The procedure of u -Net can be written as:
Umi ( O
where m (-1O0mm Um) is the initial DE estimator parameterized by O
The u-Net implicitly implements DE subproblem via a deep network similar to Unni-Net. ( is first
downsampled by PSx1/2 followed by a Conv layer to obtain the feature map up
(6) In addition, y
is input into the feature extraction module Q with shared parameters of Q in Um--Net to implicitly
enhance the non-linear fitting ability of u-Net. Then, Q(y) processed by Conv layer, and further (4) fused with up and z (f+1) by Concat layer and 1 x 1 Conv layer to obtain the fused feature map. Next, four Conv-ReLLU' layers followed by an NEAE layer are used, After that, the enhanced feature map is projected to the noise residual map by Conv layer and a PS x2 layer. Finally, skip connection from (A to the noise residual map is added. The procedure of u-Net can be written as:
(i+1) Ulz (i+-1,, ,,,, ( (Q:),
where u))@gt) ) is the optimization solving process the u subproblem parameterized by Oy-
Noise Feature Adaptive Enhancement (NFAF) Layer. The feature of real image noise should have the following properties: a salient voxel with strong degradation should be much different from the voxels around it. The difference can be easily measured by the variance, and salient voxel with
" (E) (E( is the input of NFAE) should have larger in weight than non-salient voxel. Let of be the variance for the hwrth voxel within the /th channel of
larger variance in and between channels of E. (i++1
E(++) Znw be the mean of uh the location hw, and Zr be the mean of of within the channel. We calculate the weight without adding parameters Unni Nei-Net and u-Net by
vt,, h, w) S(10v / + 8), 10 (Z, + 6)), where S(..)) the product of the Sigmoid results of its two inputs, i.e., S(a, b) Sigmoid(a)
Sigmoid 1/[(1 +e (1 + e e)), where the Euler's number. (10-16) is small positive
factor for numeral stability. Thus, the output of NFAE can be obtained by v 6 (++1)
in
3.3 Guidance Representation Module (G-Module) and Reconstruction Network (z-Net)
Since the estimated noise map is available during the iterations, it can be reasonable information to guide the reconstruction. Therefore, g( in Eq. (5) is simply implemented by the encoded degradation
5

---

representation obtained by the G-Module in Fig. 2 The procedure of G-Module can be written as:
G( Oc),
(10)
where G(.10g) is the guidance information generator parameterized by Og.
z-Net is designed to restore the latent z, which implicitly implement the z subproblem via a deep network. As shown in Fig. 2 with the degradation representation g(), we further propose the FM module shown in Fig. 3 to adjust the feature map according to the internal and external information. In detail, we adopt the FSM module to capture spatial and channel information of image features themselves, and the DEM module to dynamically recalibrate the image features according to the guidance information. By applying FM A to a residual unit, FMARR is proposed to comprehensively modulate the residual features. After that, an skip connection from the Conv layer after z to the recalibrated feature by DEM is added. Finally, the feature map is adjusted by Conv layer, and then added with Ely). The procedure of u--et can be written as:
1++1) [(z (0)
y, y,g(
u ((z),
(11)
where Z(. is the optimization solving process of the subproblem parameterized by θz.
3.4
Feature Multiple-Modulation Attention
As shown in Fig. novel FM2 A module consisting of FSM and DEM modules is proposed.
Feature Self-Mooulation (FSM). Attention mechanism very important in image restoration tasks. Although self-attention mechanism 42 has been widely used recently, because of its high parameter number and complexity, the spatial/channel attention mechanism is finally adopted in our work for a balance between performance and complexity. Different from the traditional spatial attention (SA) and channel attention (CA) mechanisms [50], we introduce complementary feature information into the proposed FSM module to enhance the recalibration ability in and between channels of the feature map. It consists of complementary information guided (CIG) SA (i.e., CIG-SA) and CIG-CA. For CIG-SA, CIG spatial feature is first achieved by channel-wis wise modulation, and then used to caleulate the spatial weights. For CIG-CA, CIG channel feature is first achieved by space-wise wise modulation, and then used to calculate the channel weights. After both the spatial and channel weights are obtained, they are respectively used to recalibrate the input feature map, and then their results are further fused. Note that by removing the CIG scheme in FSM, it becomes spatial-channel attention (SCA) based on traditional SA and CA. For more details about CIG-SA and CIG-CA, please refer to the 'Supplementary Material'.
Degradation Aaternal-Mooulatiee (DEM). Because of the complexity of real degradation, it is great challenge to estimate the clean image directly by the usual deep network. As analyzed in section 2 it is promising to exploit the guidance information (degradation representation) g to guide the reconstruction. Therefore, to exploit g in the implementation of GC, the feature map processed by the FSM module is further recalibrated by the tensor generated by the DEM weight tensor generator.
For the input feature map a. by applying both FSM and DEM, the FM A module can synchronously realize external GC and the internal important information mining to get the output feature map b:
b
Cison (a) X
dem (g),
(12)
where Catt is the DEM weight tensor generator, and Cisn is the FSM operator.
4
Experiments
4.1
Training and Testing
To train ScaoedNet, Div2K 41 is adopted to generate the synthetic clean-noisy image pairs according to the noise model in subsection 2. and the real clean-noisy image pairs from SIDD Medium Dataset 2 and RENOIR Dataset
are also used for training. Random rotations of 90" 180", 270", and horizontal flipping are performed for each training sample to achieve data augmentation. We empirically set K 5. 17 = 0.5. y = 0.25, = 4, and number, kernel size) are provided in Figs.2-3, and the default number m is set to 64. Adam (23) is used as the optimizer for the network model with default settings. During the training, we initially set the patch size to 64 x 64, the mini-batch size to 64, and the learning rate to 10-2. For fine-tuning, we set the patch size to 256 x 256, the mini-batch size to 4, and the learning rate to 10-5, We use Py PoTchh to implement Sea and train it with an Nvidia GeForce RTX 3090 GPU. Furthermore,
16. Network details (e.g., channel

---

Table I: The average PSNR(dB/SSSMM results
Table 2: The average PSNR(dBYSSIM results on
DnD benchmark dataset.
SIDD benchmark and validation datasets.
Method
Blind/Non-Blin Blind
PSNRT
SSIMT
Method
Blind/
SIDD benchmark
SIDD validation
14
Non-Blind
34.51
0.8507
Non-Blind
PSNRT
SSIM1
PSNR
SSIM1
TNRD
Non Blind
33.65
0.8306
CBM3D
Non-Blind
25.65
0.685
31.25
0.7061
DaCNN
Blind
32.43
0.7900
TNRD M
Non-Blind
24.73
0.643
26.99
0.7440
FFDNet
Mon-Blinnd
37.61
0,9415
DrCNN
Blind
23.66
0.583
26.20
0.4414
DCDiel 57
Non Blind
35.90)
0.9150
FFDNet 3
Non-Blind
29.30
0.694
26.21
0.6052
CBDNet
Blind
38.06
0.9421
DCDieL
Non-Blind
63.68
0.860
33.76
0.8171
48
Blind
39.38
0,9518
CBDNel 19
Blind 33.28
0.868
30.83
0.7341
RIDNet 5
Blind
39.25
0,9528
VDN ES
Blind
39.26
0.955
39.29
0.9109
AINDNet
Blind
39.37
0.9505
RIDNel
0.943
38.76
InvDN 27
Blind
39.57
0,9522
lavDN
el 22
Blind
Blind 38.95
39.28
0.952
38.30
38.96
0.9064
0.9123
DANet 4M
Blind
39.47
0,9548
DANe 39
Blind
39.25
0.955
39.30
0.9164
DeamNet 30
39.63
0.9531
DeamNei 36
Blind
39.35
0.955
39.40
0.9169
Nca
Blind
40.12
0.9603
ScanedNet
Blind
39,44
0.956
39,48
0.9186
ScaoedNet
Blind
40.17
0.9597
ScaoedNet
Blind
39.48
0.957
39.52
0.9187
Noisy Image
DOCNN
CBDNet
VDN
RIDNet
InvDN
DANet
DeamNet
ScaoedNett
Noisy DRCNN
CBDNet
VDN
RIDNet
InvDY
DANet
Deam
Scaoed
Figure 4: Real denoising results on DnD Benchmark dataset.
PSNR and SSIM metrics are utilized for performance evaluation the following real noisy datasets: DnD benchmark [35] SIDD benchmark [2], and SIDD validation dataset (2). The near noise-free images of DnD and SIDD benchmarks are not publicly available. However, we can submit the denoised image to the DnD and SIDD online servers to obtain the PSNR/SSIM results. Due to the limited space, please refer to the 'Supplementarry Material' for more experimental results.
4.2 Real-Worle Denoising on DnD
DnD consists
pairs of real clean-noisy scenes, where 20 smaller images of size 512 x 512 extracted from each scene. However, the corresponding ground-truth images are not available users. this subsection, we evaluate the performance ScaoedNet on the DnD benchmark.
Specifically, 12 competing methods are tested, including CBM3D [14], TNRD MT. DnCNN (52)
FFDNet [53], DCDicL [57]. CBDNet [19]. VDN (88), RIDNet (5), AINDNet (22). InvDN (27].
DANet 44). and DeamNet (36). Note that (INDNe((TF) is tested in the experiment since it can
get the best overall performance on both DnD and SIDD among all AINDNet models according 12 36.. DCDieL. (57) gets lower results because it is designed for AWGN. To better show the denoising performance of ScaoedNet, the results the self-ensemble 40 version denoted by the super script are also reported. Table I and Fig. 4 show the objective and visual results. Due to the limited space, please enlarge the figures on the screen for better comparison. According to the results, ScaoedNet significantly outperforms these competing methods in PSNR/SSIM results. In addition, these methods prone to generate either over-smooth results or distorted results with remaining noise. On the contrary, ScaoedNet can provide a clean output image with robust noise removal, effective artifacts suppression, and excellent image edges preservation.
4.3 Real-World World Denoising SIDD
We further evaluate the generalization and performance of ScaoedNet on SIDD captured by five representative smartphones. Only 320 clean-noisy image pairs (SIDD Medium Dataset) are provided for training. For testing, SIDD benchmark only with the noisy images is provided. In addition, SIDD validation consists of 1280 clean-noisy image pairs can also used. In this subsection, extensive experiments are carried out on SIDD validation and SIDD benchmark to evaluate the superiority of

---

Forward
Noisy Image
DOCNN
CBDNet
VDN
RIDNet
InvDN
DANet
DeamNet
ScaoedNet
Noisy Image
DRCNN
CBDNet
VDN
RIDNet
InvDN
DANet
DeamNet
ScaoedNet
Figure 5: Real denoising results on SIDD validation dataset.
Table 3: Results on SIDD validation for K
Table 4: Results on SIDD validation for T'-s.
2
3
5
8
16
20
PSNR 39.24
39.32
39.40
39.45
39.48
39.48
PSNR
39.10
39.26
39.40
39.48
39.49
SSIM
0.9170
0.9177
0.9182
0.9184
0.9186
0.9187
SSIM 0.9158
0.9169
0.9179
0.9186
0.9187
ScaoedNet. Table and Fig. Sjillustrate the quantitative and visual comparisons between the previous methods and SeaoedNet. We can notice that ScaoedNet achieves the best results compared to other competing methods in terms of both the objective evaluation and the visual performance.
4.4 Influence of Parameters K and
Since the iteration stage number K and the FM2 number T are crucial for denoising performance, the PSNR/SSIM evolutions versus K and 7 are analyzed. Specifically, we train six ScaoedNet
variants by setting K = 1, 2, 3, 4, 5, 6 and five ScaoedNet variants by setting The PSNR/SSIM results SIDD validation dataset are provided in Tables 3 and 4 According to the results, when = 1, ScaoedNet can already achieve promising denoising performance with 39. 29.24dB/0.9170. When K reaches 5, PSNR/SSIM further converge to 99.49dB/0.911 186, which indicates that G-Module, ueNet, and z-Net have learned to cooperate with each other this case. For that T 16 is significantly better than T 4, 8, 12, and there is little PSNR/SSIM improvements when larger than 16. These suggest that K = and T = 16 are good choices.
4. 8, 12, 16. 20.
T,
4.5 Ablation Study
Ablation on LS. By removing the LS encoding and decoding modules from ScaoedNet, we can obtain the image space denoising network. Because the Shuffle operator with factor 1/2 is used, the output of each RE network in the image space version have only 12 channels for color images and 4 channels for gray images. We can see from Fig. that the PSNR/SSIM results of the version without LS on SIDD validation dataset are only 39. 12.199B/09911 and the performance reaches to 39, 39.48dB/0.99 991B/ 186 by adding LS. Therefore, the scheme is essential for the denoising performance.
These reasons make the LS scheme essential for the performance. In Sec. 4.5 'Ablation Study', "w/o LS' corresponds to the network implementation without LS, and the performance decreases significantly in this case, which verifies our analyses. In addition, by visualizing the features after E, we find it can well separate the image signal related features and noise related components, which also makes it easier to denoise complex real noise.
Ablation SC. In method, the SC scheme is proposed for enhancing the optimization results. The ablation study on SC is conducted and the performance comparisons are shown Fig. 6 Note that the version without SC means (*) and (+) are respectively removed when estimating u (++ and
(++1), We can see that the PSNR/SSIM results decrease from 999.88B/00.911 to 39, 99.224B/0.9170
without SC. This observation indicates that the SC scheme is an effective scheme in ScaoedNet.
Ablation on GC. For further performance enhancement, the GC scheme is proposed. Note that the version without GC means the branches related to g (f) removed. In other words, G-Module and DEMs are removed. We can conclude from Fig. 6 that the PSNR/SSIM results decrease 0. 1.133/000015 without GC. This observation indicates the superiority of GC in ScaoedNet.
Ablation on Attention. The proposed attention module can synchronously realize external-internal important information mining. Fig. shows that, by removing FM A. the PSNR/SSIM will decrease

---

Method
L distancel
PSNR
WWO LS
with NEAE
5.80
33.16
WWO
without NEAB
5.97
32.53
FMF
W/O
AINDNet 22 6.11
CBDNet 1D
6.11
32.30
32.92
25NR 08
20.55
0503
wwo S5IM Loss
WWO
PRIDNet
56
6.19
33.01
ScacedNNt
069666
Table The effectiveness of the NFAE layer.
Figure 6: PSNR/SSIM scores of different ScaoedNet variants.
from 99.488B/0.91866 to 39. 29B././4/.91.4 Consequently, the effectiveness of FM is verified. More details of FSM and DEM are provided in 'Supplementary Material'.
Ablation on SSIM Loss. To verify the influence of the SSIM loss in our method, we remove from the loss function in Eq. 6 We can see from Fig. that PSNR/SSIM of the one without SSIM loss will decrease slightly to 39. 39.466//0 can also achieve higher performance than other comparisons baselines.
We should note that, even without the SSIM loss, our method
4.6 Study on NFAE layer
The proposed parameter- free NFAE layer is an important component in ScaoedNet. To show the effectiveness of NFAE, we use BSD68 337 and Kodak24 [ to synthesize the noisy images according to the noise model in subsection 2.1 Then, we build 'nossy image and noise map' pairs as the testing dataset. -et with NEAE, Uini -et without NFAE, and the noise estimation networks from CBDNet [19], AINDNet (2), and PRIDNet (56) are used for comparisons. The PSNR result and the L, distance Σy= 16g uy uylli /lll://H// (H,WW (V is the number the test images, and Hg. W are the image height and width of the gth test image) between the estimated ú, and the ground-truth noise map Ug reported in Table 5) Note that during the caleulations of the Lı distance and PSNR, u is scaled to lie in the range of [0 255J. The results show that the NFAE layer is essential for the estimation performance, and the network with NFAE obtain more accurate estimates than the noise estimation networks from CBDNet [9] AINDNet [22]. and PRIDNet [56].
4.7 Parameter Number
In this subsection, report the network parameter numbers the average PSNRs/SSIMs of the competing methods in Fig. 7 We can see that ScaoedNet has a moderate parameter number, which significantly lower than CBDNet (14) VDN (88), AINDNet [22], and DANet 49. Furthermore, although other methods have smaller parameter numbers than ScavedNet, their objective performances much lower. In summary, ScaoedNet is effective with moderate network parameter number.
ScaoedNNtt DeamNet
9.86
ScaoedNNet DeamNet
39
RIDNet
VDN
AINDNet
DANet
FFDNet
RIDNet
CBDNet
VDN
AINDNet
DANet
PSNR (dB)
38
FFDNet
CBDNet
0.52
SSIM
8.88
TNRD
8.62
33
DrCNN
DnCNN
32
0.78
8
10
12
62
64
2
8
10
12
14
02
64
Number of Parameter (M)
Number of Parameter (M)
Figure 7: The numbers of parameter vs. average PSNR/SSIM values of different models on DnD.
5
Conclusion
In this paper, we propose a new denoising network for real noise images. Although many denoising networks have been proposed, most of them are limited AWGN and ignore the valuable achieve- ments of the classic denoising methods. In contrast, we first propose novel model-based denoising cost function, and then solve it by the proposed SC alternative optimization algorithm. After that, deep unfolding denoising network is proposed according to the optimization process. Our denois- ing network combines some valuable achievements of the classical denoising methods. Extensive experiments verify that our method achieves excellent performance on real image denoising.

---

Broader Impact
As an important computer vision task, image denoising has largely benefited the society in various areas and has no negative impact yet. The proposed method could further improve the performance especially in the challenging real-worke cases where the noise information is unknown. This work has no negative impact on the ethical and societal aspects.
Acknowledgements
This work was supported by the National Natural Science Foundation China under Grant 62171304, and in part by the Key Research and Development Program of Sichuan Province under Grant 2022YFS0098.
References
[1] "kodak lossless true color image suite," http://rOk..u/graphiikkkkaal
[2] A. Abdelhamed, S. Lin, and M. S. Brown. high-quality denoising dataset for smartphone cameras. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1692-1700, Jun. 2018.
[3] M. Aharon, M. Elad, and A. Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11):4311- 4322, 2006.
[4] J. Anaya and A. Barbu. Renoir-a dataset for real low-light image noise reduction. Journal of Visual Communication and Image Representation, 51:144-154, 2018.
[5] S. Anwar and N. Barnes. Real image denoising with feature attention. In IEEE International Conference on Computer Vision (ICCV), pages 3155-3164, Oct. 2019.
[6] G. Bhat, M. Danelljan, F. Yu, L. Van Gool, and R. Timofte. Deep reparametrization of multi- frame super-resolution and denoising. In IEEE International Conference on Computer Vision (ICCV), pages 2460-2470, Oct. 2021.
[7] Byun, Cha, and T. Moon. Fbi-denoiser: Fast blind image denoiser for poisson-gaussinn noise. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5768-5777, Jun. 2021.
[8] J. Cai, H. Ji, C. Liu, and Z. Shen. Blind motion deblurring from single image using sparse approximation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 104-114 111, Jun. 2009.
[9] J. Cai, H. Ji, C. Liu, and Z. Shen. Framelet-based blind motion deblurring from a single image. IEEE Transactions Image Processing, (1(())562-552 2012.
[10] M. Chang, Q. Li, H. Feng, and Z. Xu. Spatial- network for single image denoising. In European Conference on Computer Vision (ECCV), pages 17 171-187, Sep. 2020.
[11] Y. Chen and T. Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1256-122)) 1272, 2017.
[12] S. Cheng, Y. Wang, H. Huang, D. Liu, H. Fan, and Liu. Nbnet: Noise basis learning for image denoising with subspace projection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4896 4906, Jun. 2021.
[13] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d transform- domain collaborative filtering. IEEE Transactions on Image Processing, 1608):2080-2055, 2007.
10

---

[14] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Color image denoising via sparse 3d collaborative filtering with grouping constraint luminancechrominance space. In IEEE International Conference on Image Processing (ICIP), pages 1-313-1-316, Oct. 2007.
[15] W. Dong, P. Wang, W. Yin, and G. Shi. Denoising prior driven deep neural network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(10): 2305-23112 2019.
[16] V. Fedorov and C. Ballester. Affine non-local means image denoising. IEEE Transactions on Image Processing. 26(5):2137-2148, 2017.
[17] S. Gu, L.. Zhang, W. Zuo, and X. Feng. Weighted nuclear norm minimization with application to image denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2862-2869, Jun. 2014.
[18] S. Gu, Y. Li, L. V. Gool, and R. Timofte. Self Self-guided network for fast image denoising. In IEEE International Conference on Computer Vision (ICCV), pages 251 1-2520, Oct. 2019.
[19] S. Guo, Z. Yan, K. Zhang, W. Zuo, and L. Zhang. Toward convolutional blind denoising real photographs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1722, Jun 2019.
[20] J. Huang, Z. Zhao, C. Ren, Q. Teng, and X. He. A prior-guided deep network for real image denoising and its applications. Knowledge-Bacd Based Systems, 255:109776, 2022.
[21] X. Jia, S. Liu, X. Feng, and Z. Lei. Focnet: A fractional optimal control network for image denoising. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6047-6056, Jun. 2019.
[22] Y. Kim, J. W. Soh, G. Y. Park, and N. I. Cho. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3482-3492, Jun. 2020.
[23] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), pages 1-15, May 2015.
[24] Lefkimmiatis. Universal denoising networks : novel enn architecture for image denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3204-3213, Jun. 2018.
[25] X. Li, Y. Zhou, Zhang, and L. Wang. Multipatch unbiased distance non-local adaptive means with wavelet shrinkage. IEEE Transactions on Image Processing, 29:157-169, 2020.
[26] Y. Li, X. Fu, and Z.-J. Zha. Cross-patch graph convolutional network for image denoising. In IEEE International Conference on Computer Vision (ICCV), pages 4651 Oct. 2021.
[27] Y. Liu, Z. Qin, S. Anwar, P. Ji, D. Kim, S. Caldwell, and T. Gedeon. Invertible denoising network: A light solution for real noise removal. In JEFF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13365-13374, Jun. 2021.
[28] Y. Liu, Z. Qin, S. Anwar, P. Ji, D. Kim, S. Caldwell, and T. Gedeon. Invertible denoising network: A light solution for real noise removal. In IEEE Conference on Computer Vision and Pattem Recognition (CVPR), pages 13365 13365-13374, Jun. 2021.
[29] Z. Luo, Y. Huang, S. Li, L. Wang, and T. Tan. Unfolding the alternating optimization for blind super resolution. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 5632-5643, 2020.
[30] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image restoration. In IEEE international conference on computer vision (ICCV), pages 2272-2279, Oct. 2009.
[31] X. Mao, C. Shen, and Y.B. Yang. Image restoration using very deep convolutional encoder- decoder networks with symmetric skip connections. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, pages 2802-2810 10, 2016.

---

[32] S. Mohan, J. L. Vincent, R. Manzorro, P. Crozier, C. Fernandez- Granda, and E. Simoncelli. Adaptive denoising via gaintuning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 23727-23740, 2021.
[33] S. Osher, M. Burger, D. Goldfarb, Xu, and W. Yin. An iterative regularization method for total variation-based image restoration. Multiscale Modeling & Simulation, 4(2):460-489, 2005.
[34] J. Pang and G. Cheung. Graph laplacian regularization for image denoising: Analysis in the continuous domain. IEEE Transactions on Image Processing, 26(4):1770-1185, 2017.
2017.
[35] T. Plotz and S. Roth. Benchmarking denoising algorithms with real photographs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1586-1595, Jun. 2017.
[36] C. Ren, X. He, C. Wang, and Z. Zhao. Adaptive consistency prior based deep network for image denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1996-6606, Jun. 2021.
[37] S. Roth and M. J. Black. Fields of experts. International Journal of Computer Vision, 82(2): 205-229, 2009.
[38] W. Shao and M. Elad. Simple, accurate, and robust nonparametric blind super-resolution. In International Conference on Image and Graphics, pages 333-348, 2015.
[39] Y. Tai, J. Yang, X. Liu, and C. Xu. Memnet: A persistent memory network for image restoration. In IEEE international conference on computer vision (ICCV), pages 4539-4547, Oct. 2017.
[40] R. Timofte, R. Rothe, and I.. V. Gool. Seven ways to improve example-based single image super resolution. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1865-1873, Jun. 2016.
[41] R. Timofte, K. M. Lee, X. Wang. Y. Tian, Y. Ke, Y. Zhang, S. Wu, D. Chao, L. Liang, and Q. Yu. Ntire 2017 challenge on single image super-resolution: methods and results. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1110-112,, Jun. 2017.
[42] Z. Wang, X. J. Bao, W. J. and H. Li. Uformer: A general u-shaped transformer for image restoration. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 17683-17693, Jun. 2022.
[43] Xie, S. Li, and Sun. Hyperspectral images denoising via nonconvex regularized low-rank and sparse matrix decomposition. IEEE Transactions on Image Processing, 29:44-56, 2020.
[44] Y. Xie, S. Gu, Y. Liu, W. Zuo, W. Zhang, I.. Zhang, and Y. Xie. Weighted schatten p-norm minimization for image denoising and background subtraction, IEEE Transactions on Image Processing, 25(10)4882 4857, Oct. 2016.
[45] Xu and S. Osher. Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising. IEEE Transactions Image Processing, 16(2):334-544 2007.
[46] J. Xu, L. Zhang, D. Zhang, and X. Feng. Multi-channel weighted nuclear norm minimization for real color image denoising. In IEEE International Conference on Computer Vision (ICCV), pages 1096-1104, Oct. 2017.
[47] J. Xu, I.. Zhang, and D. Zhang. A trilateral weighted sparse coding scheme for real-world image denoising. In European Conference on Computer Vision (ECCV), pages 21-38, Sep. 2018
[48] Z. Yue, H. Yong, Q. Zhao, D. Meng, and L. Zhang. Variational denoising network: Toward blind noise modeling and removal. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pages 1690-1701, 2019.
[49] Z. Yue, Q Zhao, L.. Zhang, and D. Meng. Dual adversarial network: Toward real-world noise removal and noise generation. In European Conference on Computer Vision (ECCV), pages 41-58, Sep. 2020.
12

---

[50] S. YWoo. J. Park, J.-Y. Lee, and I. S. Kweon. Cbam: Convolutional block attention module. In European Conference on Computer Vision (ECCV), pages 3-19. Sep. 2018.
[51] H. Zhang, Y. Li, H. Chen, and C. Shen. Memory-efficient hierarchical neural architecture search for image denoising. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3657-3666, Jun. 2020.
[52] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: residual learning of deep for image denoising. IEEE Transactions Image Processing, 26(7): 3142-3155, 2017.
[53] K. Zhang, W. Zuo, and L. Zhang. Fidnet: Toward fast and flexible solution for enn based image denoising. IEEE Transactions on Image Processing, 27(9):4088-4622, 2018.
[54] Zhang, Y. Fu, and Li. Hyperspectral image denoising with realistic data. In IEEE International Conference on Computer Vision (ICCV), pages 2248 2257, Oct. 2021.
[55] X. Zhang, J. Zheng, D. Wang, and L. Zhao. Exemplar-based denoising: A unified low-rank recovery framework. IEEE Transactions on Circuits and Systems for Video Technology, 30(8): 2538-2544 2020.
[56] Y. Zhao, Z. Jiang, A. Men, and G. Ju. Pyramid real image denoising network. In IEEE Visual Communications and Image Processing (VCIP), Dec. 2019.
[57] H. Zheng, H. Yong, and L. Zhang. Deep convolutional dictionary learning for image denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 630 641, Jun. 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] All claims supported by extensive empirical experiments
(b) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see section 'Broader Impact'
(d) Did you describe the limitations of your work? [Yes] see section 'S15. Limitations and Future Works' in Supplementary Material
you including theoretical results...
(a) Did you state the full of assumptions of all thcoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you experiments...
(a) Did you include the code, data, and instructions needed reproduce the main experimental results (either in the supplemental material or a URL)? [Yes]
(b) Did you specify all training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] subsection '4.1 Training and Testing' in main paper
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See subsection '4.1 Training and Testing' in main paper
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See the beginning of Supplementary Material
(d) Did you discuss whether and how consent obtained from people whose data you 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
If you used crovdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "cs5Qu9yXAtE", "reviewer": "Reviewer_92hr", "summary": "This paper proposed an enhanced latent space blind model based deep unfolding network, namely ScaoedNet, for complex real image denoising.  Experiments demonstrate the superiority of ScaoedNet over many SOTA methods.", "questions": "Please clarify the difference between you and [1] on how to use the distinguishable space (named latent space or subspace), and the superiority of your usage. ", "limitations": "See in weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n \n1. An enhanced model-based denoising cost function is proposed to implicitly optimize the ScaoedNet, which addresses the challenging task of manually designing the optimal operators and making the optimal process more interpretable. \n\n2. The effective NFAE layer leads to better results without increasing extra parameters.\n\nWeaknesses:\n\n1. The description of the total loss is not clear enough. \n\n2. Map noise image to space that better distinguish noise has been proposed in [1] and therefore cannot be made as a prominent contribution.\n\n[1] 1. Cheng, Y. Wang, H. Huang, D. Liu, H. Fan, and S. Liu. Nbnet: Noise basis learning for 356 images denoising with subspace projection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4896–4906, Jun. 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aVi0roC7IQG", "reviewer": "Reviewer_YKat", "summary": "This paper proposes a real-world image denoising model with alternative optimization. The formulation is carefully discussed in order to ensure the validity of the proposed pipeline. This is an interesting approaching to solve the problem of image denoising. Experiments and ablation studies are sufficient to prove the superiority of the proposed method over other state-of-the-art methods and the efficiency of the designed modules.", "questions": "1.     The embeddings in the proposed Latent Space (LS) are seemingly just convolutional features of input images. Whether these latent embeddings have special semantics? If not, why the authors emphasize it in the paper (even in the title)?\n\n2.\tI notice that the proposed method uses improved channel attention (CIG-CA) and spatial attention (CIG-SA). Have the authors tried self-attention mechanism? If not, why?\n\n3.\tThe formulation of the proposed method is seemingly weak while the performances are excellent. I totally agree that this is a great work, but I’d like to ask the authors that have you think about to submit your work to ECCV or CVPR? I think it would be more suitable for those conferences instead of NeurIPS.\n", "limitations": "The limitations are clearly addressed in supplementary materials. This work does not have any potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe method of alternative optimization is novel to image denoising. In order to adopt this method, the formulation of denoising is strictly discussed and the pipeline is carefully designed.\n\n2.\tThe definition of the problem and the corresponding solution are clear. Instead of regarding NNs as a ‘black box’, the authors try to analysis the building reasons of every block.\n\n3.\tThe experiments and ablation studies are sufficient. The performance of the proposed method is better than SotA methods clearly.\n\nWeakness:\n\n1.\tThe proposed method is seemingly an incremental method to DAN [1]. Although the pipeline is specifically designed for the task of image denoising, I think that it somehow lacks of novelty.\n\n2.\tI miss a general description of the total pipeline. Such a complex framework is a little bit hard to read. I suggest the authors to modify the texts of Sec. 2 and 3.\n\n3.\tThe subjective results shown in the paper is not that impressive. I suggest the author choose clearer results to replace Fig. 4 and the upper row of Fig. 5.\n\n[1] Z. Luo, Y. Huang, S. Li, L. Wang, and T. Tan. Unfolding the alternating optimization for blind super resolution. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 5632–5643, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WBCHVg9lwU", "reviewer": "Reviewer_XEky", "summary": "This paper proposes an image denoising network based on latent space self-correction alternative optimization. A deep unfolding network implements alternative optimization, and state of the art denoising performance is achieved on multiple benchmark datasets.", "questions": "1. Would raise score if inference complexity/time is on par with other state of the art methods.\n2. Is the proposed model based framework applicable to other image restoration problems, e.g. deblurring, deraining, etc. ? This paper only discusses denoising and the value of the proposed methodology is therefore limited.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- the proposed method is backed with classic model based image restoration theory that provides strong regularization on the network design, which allows the proposed method to achieve state of the art performance with a modest model size.\n- implementation detail of the proposed network is well documented for reproduction\n- extensive ablation study is conducted to analysis each individual component of the proposed network architecture\n\nWeakness:\n- The experimental evaluation section lacks complexity and time cost analysis and comparison\n- section 3 is filled with an overwhelming amount of implementation detail that are better suited for supplementals. It would be better to put emphasis on theoretic/algorithmic reasoning & insights, and put implementation details in section 4 or a dedicated subsection in section 3. Right now section 3 provides very limited insight into *how* the network implements and enforces those ideas.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MEAoqydzf9m", "reviewer": "Reviewer_JAMB", "summary": "In this paper, the authors proposed a method for real image denoising. The authors designed the enhanced model-based denoising cost function by introducing latent space, noise information and guidance constraint. And then implement the proposed self-correction alternative optimization algorithm to minimize cost function via deep network. Besides, for higher performance, some optimized modules of subnetwork are proposed, including noise feature adaptive enhancement (NFAE) layer and feature multi-modulation attention residual block (FM2ARB).", "questions": "1. In ‘Analysis and Enhancement’ part, the author represents that LS encoding function can obtain high-dimensional image embeddings instead of limited squared error in low-dimensional image space. However, no clear theoretical reason and significance are indicated.\n2. As mentioned in the article, LS scheme is the essential for the performance of this work. However, there is still no enough explicit explanation for its ablation study.\n3. The author has already chosen the SSIM as one evaluation, why repeatedly adopt SSIM loss in the total loss function? \n4. Can author provide more comparison with other noise estimation network to further verify the effectiveness of NFAE layer?\n5. Authors also can compare the proposed method with some more advanced traditional methods, e.g.: Exemplar-Based Denoising: A Unified Low-Rank Recovery Framework, IEEE Transactions on Circuits and Systems for Video Technology, 2020; Deep Convolutional Dictionary Learning for Image Denoising，CVPR, 2021.", "limitations": "In this paper, the method that authors proposed makes full use of the advantages of classical denoising methods and deep network, which no longer sticks to AWGN and contributes to real image denoising to a certain extent.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Motivated by the advances in deep networks and relying on the rich body of the model based methods, authors propose a novel enhanced latent space (LS) blind model based deep unfolding network for real image denoising. Extensive experiments verify that this method achieves excellent performance on real image denoising. \nHowever, authors should pay more attention to clarity of the paper. For example, the abstract of this paper is too mess to emphasis the most important novelty. Author needs to write it in brief and appropriately. \nBesides, the layout and the visualization of some figures also need to be attached. \ne.g., the Figure 1 is not mentioned in the text but shown. In Fig.2, the name of subnetworks cannot be corresponded to the figure note. Readers still need to find each subnetwork’s name through article.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept