PUCA: Patch-Unshuffle and Channel Attention for Enhanced Self-Supervised Image Denoising
Authors: Hyemi Jang
Junsung Park
Dahuin Jung
Jaihyun Lew
Ho Bae
Sungroh Yoon
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised image denoising
low-level vision
TL;DR: We propose J-invariant U-Net structure to incorporate global contexts and coarse-and-fine features in self-supervised image denoising.

Abstract: Although supervised image denoising networks have shown remarkable performance on synthesized noisy images, they often fail in practice due to the difference between real and synthesized noise. Since clean-noisy image pairs from the real world are extremely costly to gather, self-supervised learning, which utilizes noisy input itself as a target, has been studied. To prevent a self-supervised denoising model from learning identical mapping, each output pixel should not be influenced by its corresponding input pixel; This requirement is known as J-invariance. Blind-spot networks (BSNs) have been a prevalent choice to ensure J-invariance in self-supervised image denoising. However, constructing variations of BSNs by injecting additional operations such as downsampling can expose blinded information, thereby violating J-invariance. Consequently, convolutions designed specifically for BSNs have been allowed only, limiting architectural flexibility. To overcome this limitation, we propose PUCA, a novel J-invariant U-Net architecture, for self-supervised denoising. PUCA leverages patch-unshuffle/shuffle to dramatically expand receptive fields while maintaining J-invariance and dilated attention blocks (DABs) for global context incorporation. Experimental results demonstrate that PUCA achieves state-of-the-art performance, outperforming existing methods in self-supervised image denoising.

PUCA: Patch-Unshuffle and Channel Attention for Enhanced Self-Supervised Image Denoising
Hyemi Jang', Junsung Park', Dahuin Jung', Jaihyun Lew?, Ho Bae² Sungroh Yoon 1.2,* 1.2
'Department of Electrical and Computer Engineering, Scoul National University
Inteendisciplinary Program in Artificial Intelligence, Seoul National University
DDpartment of Cyber Security, Ewha Womans University jerryraydsnu.c ac kr annajuug062000mu ac.kr
ikda195120smu ac kr fudojhl
madouhl@snu..o @snu. ac.ke kr
hobae@ewha.c ac.kr
sryoon acc.
Abstract
Although supervised image denoising networks have shown remarkable perfor- mance synthesized noisy images, they often fail in practice due to the difference between real and synthesized noise. Since clean-noisy image pairs from the real world are extremely costly to gather, self-supervised learning, which utilizes noisy input itself as target, has been studied. To prevent self-supervised denoising model from learning identical mapping, each output pixel should not be influ- enced by corresponding input pixel; This requirement known as J -invarriance. Blind-spot networks (BSNs) have been prevalent choice to ensure J-invariance in self-supervised image denoising. However, constructing variations of BSNs by in- jecting additional operations such as downsampling
expose blinded information, thereby violating J -iivaariance. Consequently, convolutions designed specifically for BSNs have been allowed only, limiting architectural flexibility. To overcome this limitation, we propose PUCA, a novel J-invariant U-Net architecture, for self-supervised denoising. PUCA leverages paich-unshuffl//huuffe to dramatically expand receptive fields while maintaining -iinvariance and dilated attention blocks (DABs) for global context incorporation. Experimental results demonstrate that PUCA achieves state-of-the-art performance, outperforming existing methods self- sul--supervieed image denoising.
Introduction
CLASSIC
CLASSIC
CLASSIC
CLASSIC
CLASSIC
Noisy Image 16.15dB/0.118
Ground Truth dB/1
DrCNN 45
AP-BSN 21
PUCA (Ours)
27.65dB/0.624
29.46dB/0.77
31.11dB/0.795
Figure 1: Visual comparison with other denoising methods on the SIDD validation dataset [T] DnCNN was trained with real clean-noisy pairs of SIDD. AP-BSN 21] and PUCA were trained in self-supervised manner solely on noisy images.
"Correspondigg Authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Image denoising is a traditional task in computer vision, aiming to recover clean image from a noisy one. With the advent of convolutional neural networks (CNNs) T7.. deep learning- based methods have dominated the field. 23 24
These methods train a denoiser in a supervised manner using synthesized deannnoisy image pairs. However, denoising models trained on synthesized image pairs have limitations when dealing with real noise because of the domain gap between synthetic and real noise [47]. To overcome the limitations, several researchers [] have collected real-world clean- noisy image pairs in a controlled environment. However, this process requires a lot of manual effort and post-processing, and it is challenging to collect enough data, resulting in limited generalization ability. Therefore, self-supervised denoising methods have been proposed to train denoising model solely relying on noisy images.
46
30
47
In self-supervised learning, where the model is trained using an identical image as input and target, is essential to satisfy J-invariance 4 to prevent the model from learning an identity mapping. J-invariance refers to eliminating the influence of each pixel on the corresponding output pixel. Blind-spot networks (BSN), which basically meet the J-invariance requirement, have been introduced to effectively denoise noisy inputs in self-supervised manner [18 36 211. BSNs [36] 21 achieve the requirement by incorporating centrally masked convolution, where the central pixel of the kernel has zero value, and dilated convolutions. BSNs can be constructed with two choices of layers only, centrally masked convolutions and dilated convolutions, and cannot utilize varying operations such downsampling. This constraint arises from the necessity maintain J Jiinvariance. Consequently, the previously proposed architectures have been restricted due to the harsh requirement. In contrast, supervised denoising models enjoy more flexibility in terms of network structures since real clean-noisy image pairs are provided, alleviating the need for J J-invariance. This flexibility enabled the exploration of diverse architectures, and recently, the encoder-decoder based U-Net which preserve hierarchical multi-scale representations, have
architectures 2 8 19 33
become prevalent.
As highlighted by the success of supervised denoising models, aggregating spatial information plays a critical role in effective denoising. In particular, U-Net architectures have shown superior improvements in modeling long-range dependencies and coarse-to- tone representation 2 g 19
33 41 43 44). To benefit from the U-Net structure (27) in self-supervised learning, we propose
technique that preserves J-invariance. By incorporating lownsamplinn/uussappigg as design option for BSN, we increase flexibility in network structures. Furthermore, patch-unshuffle expands receptive fields and utilizes multi- scale representation in BSN. We also introduce the dilated attention block (DAB), a J -invariant channel attention mechanism incorporating global information. DABs effectively remove noise by attentive connections between instances sub-sampled via patch-unshuffle. Hence, we integrate Mhich-Unshuffle and Channel Attention (PUCA), a novel U-Net architecture, while meeting the J-invariant requirement. PUCA significantly outperforms existing state-of-the-ar denoising methods and unsupervised/ unpaired approaches. As shown in Figure [ PUCA shows comparable denoising quality with the ground truth. Our contributions can be summarized as follows:
patch-unshufflesshuffitt a novel dlowssamlinn/upsampplie
supervisd
We propose patch-unshufff//shffftic a I invariiat downsampling(ussampiin method that effectively utilizes multi-scale representation and expands receptive fields. Moreover, halch-unshufflc/shufii unleashes the constrained architecture design for blind-spot net- works (BSNs).
We introduce dilated attention blocks (DABs), which effectively incorporate global context through channel attention. The combination of DAB with natch-unshuffleshhfftt provides an advantage for noise removal by leveraging information from an ensemble of subsamples.
By taking advantage of patch-unshufffe/shuffie and dilated attention block (DAB), PUCA, our proposed J T-invariant U-Net, outperforms other self self-supervised and unpaired solutions by a substantial margin. Code is available at
com wtw.coom/////eeeeee Hyem/ Esme/PUCa
2 Preliminaries
2.1
J-invariant Network
In self-supervised learning, the denoising model needs to maintain J-invariance to prevent learning identity mapping. J-invariance is to remove the influence of the input pixel on the corresponding

---

Figure 2: Dependency between the input and output pixels with a centrally masked convolution and d-dilated convolutions. The green pixels indicate the pixels that are dependent on the center pixel in input, while the blue pixels represent the area independent of the central pixel. The yellow pixels represent the convolution weights.
output pixel. Since spatially correlated noise can expose evidence for the masked pixels, previous works [18 4 [6)] assume zero-mean pixel-wise independent noise. We also adopt the same noise assumption, and the definition of J --invariance introduced in Noise2Self [4] is as follows.
Definition 1 (Batson and Royer, 2019) Consider partition I of the dimensions (1,....m) m), an observed noisy signal æ, and sub-sample ty of x limited to J E J. Let g R - Rm be a function. A function g is J-invariant if the value of g(x), does not depend on the particular value of IJ. is J --invariantt if it is J-invariant for every J E J.
The J-invariant network is trained to predict the value of each pixel using that of its surrounding pixels. To demonstrate the correspondence of self-supervised loss for supervised loss, we adopt the proposition from J -invariant definition of Noise2Self [4].
Proposition (Batson and Royer, 2019) Let noisy image) be an unbiased estimator of y (a clean image), denoted as [x[]y]] y. Considering the J J-invariant function g. then the self- loss,
Ellg(x) x²| Bl)g(=)
y| + El| y11
Under the assumption of zero-mean pixel-wise independent noise, the self-supervised loss is equal the general supervised loss in addition to the variance of the noise. Therefore, by training the I J-invariant function y using the self-supervised loss, g can learn a way of eliminating the encoded general noise.
incorporate blind spots to the denoising framework to preserve J-invariance in self-superw superviee learning. D-BSN 33) is popular BSN architecture that utilizes centrally masked convolution in the first layer to exclude input pixel information. To maintain J-invariance, D-BSN employs dilated convolution with dilation 2 when using 3 x 3 centrally masked convolution and dilation 3 when using a x centrally masked convolution. The dependence of output pixels on the central pixel of input after stacking centrally masked convolution and dilated convolution is illustrated in Figure 2 The combination of the centrally masked convolution and the dilated convolution satisfies the J Jinvariiance.
Blind-spot network (BSN) BSNs 18 20 36
2.2 Pixel-Shuffle Downsampling
Contrary to the assumption of zero-mean pixel-wise independent noise in BSN, real noise demon- strates correlations that enable noise prediction from neighboring pixels. To address this, Zhou et al. [47] introduced pixel-shuffle downsampling (PD), allowing denoisers trained on synthetic noise handle real noise effectively by breaking spatial noise correlation. However, the analysis of AP-BSN (2D on real noise in SIDD reveals that images downsampled by the PD with small stride factor still exhibit spatial noise correlation. To mitigate this, PD with large stride factor is used during training. Nonetheless, using a PD with a large stride factor leads to a loss of input details. To preserve the details, PD with small stride factor is employed during testing. We also adopt asynchronous PD, utilizing different PD sizes for train and test phases.
Method
We propose PUCA, a novel -iinvariant U-Net shown in Figure 3 Initially, we provide an overview of PUCA. Subsequently, we delve into the two essential elements of PUCA: (1) Patch Unshufflc/Shffle and (2) Dilated Attention Block (DAB).
3

---

Centrally
Image
Masked
olttolatoon
120
128
123
128
61
64
Ouiput
DAB
Mix W xC
Fo
light fevered
Shuflle
Ix1 Camoolaion
Patch Unsshalle
PD
Pissi-Shafle Downsimpling
DA B
DAB
Dilittd A tteition Black
LaverNom
FIN
DDC
Dilated Combuiato-omnooaation
Simplet-ae
2
SCA
Simplified Channel Atention
SimpleGate
Patch-Shuufle
PU
Pixck Shuffle Upompling
All
Attration Block
DAB
EON
Feed forward Network
Element-
Figure 3: Overview of PUCA. Our method utilizes an encoder--dccoder-basee U-Net architecture. During encoding, we extract global context through suuch-nnhuffting. and during decoding, combine local and global context through skip connections.
Unehuffle
Shuffl:
Shulll:
(a) Pixel-Unshuffle
(b) Patch-Unshufff
Figure 4: Difference between pixel-unshufflesshufle and patch-unshuffleshufffe The pixels except for blue are dependent the central pixel of input, and only blue pixels are independent. We assume 2-dilated convolution is applied.
PUCA Overview First, convolution and pixel-shufile downsampling are applied to break noise correlation in the noisy image / E #WW x3 Subsequently, centrally masked convolution and multiple 1 convolutions derive low-level feature embeddings Fo IR Hx И W represents the spatial dimensions, and C' denotes the number of channels. The initial features Fo are then processed through a 3-level encoder-decoder structure with multiple DABs at each level, resulting in deep features Fa E IR Mx The encoder begins with the high-resolution input and incrementally shrinks its spatial size while augmenting channel capacity. Conversely, the decoder starts with low-rcsolution latent features Fi E R " X4-C and progressively restores high-resolution representations. Patch-unshuffle and patch-shuffle operations (Figure 4b) used for feature downsampling and upsampling, respectively. To retain detailed structures and textures the restored images, skip connections merge features from the encoder and decoder. Finally, sequential application of convolution layers generates the output image Hxxx which represents the enhanced and denoised version the original noisy image.
3.1 Patch Unshuffle/Shuffte
We propose patch unshuffie/shuffle that dramatically increases receptive fields, shown in Fig- ure 5 Downsampling helps to capture long-range dependencies and multi-scale context. By in- corporating multi-scale context through downsampling, denoisers
better understand the rela- tionships and dependencies between different image parts. Among the downsampling methods, pixel-unshufflcshuiiie (28) that preserves original pixels known be effective in image restoration. However, mixel-unshune with BSN [36] disrupts J -invariiance.
Let's consider BSN denoted as composed of fully convolutional layers. comprises d-dilated convolutions (() for all E (1, L), with kernel size of 3 x 3. We can express the function as sequential application of f()), resulting in f(x) f(L)
Here,

---

C. -p,, illustrated AP-BSN 21
PUCA (Level-1)
PUCA (Level-2)
PUCA (I.evel-3)
PUCA (Level-4)
Figure 5: Visualization of the receptive field for AP-BSN [21] and PUCA based on depth variations. We calculate the influence of input pixels on the output's central pixel through gradients. Brighter colors indicate stronger influence PUCA exhibits significantly wider receptive fields than AP-BSN. We observe that the receptive fields become wider as the depth increases.
f(0) represents a convolutional layer using a (2d-1) 1) (2d 1) centrally masked filter, and denotes the input noisy image. The output features for each convolutional layer / are represented as /d) f (..) (r))()))))
Let Fijj pixel of a noisy image where i.j are the coordinates satisfying mod 0 and j mod y = 0. q denotes the scale factor pixel-unshuffle. The application of dilated convolution exposes æi, is features to its neighboring pixels. Pixel-unshuffle (Figure 4a) aligns pixels ym.., where (f)
i 5 m < i + and j < n < j + q to the channel axis of the same spatial position as Vigg and convolution operations following this would expose Tisj's features to v breaking the J J-invariance.
We propose patch-unshufile to solve the problem of pixel-unshuflle mentioned above. Patch-unshufile transforms tensor of size H W into a reshaped tensor of size
in Figure 4b This process can be mathematically defined as follows:
1 + mod p. C(pl mod p² mod p² Much-UnshuuMe(r(..2222 (p) (p- + mod p. P
J) k)
p
(2)
Incorporating patch-unshuffle function f(x) preserves I -invariance under specific conditions.
Proposition 2.
Patch-unshuffle maintains J-invariance if p. patch size, is multiple of the dilation used in dilated convolution
Proof. Assuming simplified one-dimensional case for clarity without loss of generality, the satisfaction of J -invariiance requires patch-une sachhuunshuffee to meet the condition where pixels with the same position as v, in each channel remain unaffected by æ. E (1...., m.).
After central masked convolution in the first layer, neighboring pixels are affected by æy, with the receptive field RF(((0) x))) = {J - (d 1). J - 1, J + 1, J + (d - 1)). (F x)) indicates the receptive field æ, in ( Sequentially, dilated convolution spreads the information æj.
". x.))
U jEl d.0.4)
(i Ji+ii (-1) xx)))
There exist pixels that do not depend on with period of d. To downsample y we use patch-unshuffle. facilitate comprehension, our emphasis lies on spatial location while disregarding the channel axis.
Batch-Unshuffleff/ 9 .p)) pl mod
(4)
The pixels that have identical spatial position with J satisfy
pl トラ + mod p = pl - oo + mod
(5)
J
p([-
2 1 1)
(J mod p
(6)
5

---

In Equation 6 the left-hand side is multiple of p. and the right-hand side is the remainder when divided by p. making it smaller than p. Thus, for the equation to be valid. both sides must be that the positional difference between and J is multiple of from J. By considering the periodicity d of pixels in yd) that do not depend on æJ, resulting from the combination of centrally masked convolution and dilated convolution, we can infer that when the patch size p is a multiple of the dilation factor d, Sunhhfunssufflee guarantees the independence on F. along the channel axis. O
By employing patch- unnsufle with patch size p that is multiple of the dilation factor d, J --invariant networks ensure the preservation of J //invariaance This successful integration of patch-unshuffle in BSN allows for effective downsampling operations. Moreover, patch-shuffle in Figure 4b) which acts as the reverse operation of patch-unshuffle, facilitates the upsampling of downsampled feature maps to restore them to their original image sizes.
3.2 Dilated Attention Block
We also adopt channel attention 7 in addition to U-Net to fully exploit the global information the image. The feature interaction that can be obtained from each is different. While channel attention based on global pooling extracts the most prominent features among the global information, U-Net is more suited for leveraging feature relations by expanding the receptive fields. However, directly applying channel attention to BSN is not straightforward due to the blind spot requirement. We aim to design channel attention block that satisfies the blind spot requirement when combined with patch-unshuffle and patch-shuffle operations. To fulfill this requirement, we incorporate a d-dilated 3 x 3 depth- wise convolution (DDC) before gating and attention, taking inspiration from D-BSN [6). The resulting dilated attention block (DAB), consists of LayerNorm [3 1 X 1 convolution, skip connection [44, SimpleGate and Simplified Channel Attention (SCA) 77. along with DDC (as shown in Figure 3 The utilization of SimpleGate and SCA enhances the integration of local and global information by suppressing less informative features 42]
For the input feature map X
SimpleGate (xaalGatte(1,,xX X2) X
X2.
(7)
where @ is element-wise multiplication, X, and X2 are two parts of X along the channel axis of the same size.
SCA(X) = X * (X),(X)),
(8)
where Pt. FF and * denotes global average pooling, a fully-connected layer, and channel-wise product operation, respectively.
4 Experiments
4.1 Datasets and Implementation Details
Smartphone Image Denoising Dataset (SIDD) [1 is collection of real-world images for denoising captured by five different smartphone cameras. Specifically, the SIDD-Medium dataset consists of 320 pairs of noisy and clean images for training purposes. In addition, the SIDD validation set and benchmark set are used for validation and evaluation, respectively. Both sets consist of 1,280 noisy patches with size of 256 256, and corresponding clean images are provided only for the validation set. We train our model for SIDD validation and evaluation on the
dataset.
Darmstadt Noise Dataset (DND) 226 is dataset used for benchmarking image denoising algorithms. contains 50 real-world noisy images without any ground truth provided, and the only way to obtain results is through an online submission system. A fully self-supervised learning approach can be used to train the denoising model directly on the test set without using any external data. We train our model for DND evaluation on the DND benchmark.
Implementation Details We trained the model using an NVIDIA TESLA P100 GPU and imple- mented it with Pytorch 2.0.0. The model was trained with L1 loss between the input noisy image and the output, using Adam optimizer with an initial learning rate of le-4. We trained the model for 20 epochs until it fully converged. More detailed information can be found in our supplementary material. We use peak signal-to-nn for denoising. The SIDD and DND benchmark results are obtained from online submission scores, and for the SIDD validation data, we evaluate the performance using the corresponding functions in the skimage metrics library.
ratio (PSNR) and structural similarity (SSIM) 34 as evaluation metrics

---

Table 1: Quantitative comparison on SIDD [ and DND [26] benchmarks While certain super- vised methods achieve better evaluation results by utilizing noisy-clean image pairs from SIDD, our approach solely relies on noisy sRGB images. We report the official results from the SIDD and DND benchmark websites. Results marked with R and A are reported from R2R 25 and AP-BSN 21
respectively, while * indicates results evaluated by us.
SIDD
DND
Method
PS
SSIM
PSNR(dB)
SSIM
BM3D
25.65
0.685
34.51
0.851
Non-lcarning based
WNNM
25.78
0.809
34.67
0.865
DrCNN
45
23.66
0.583
32.43
0.790
Supervised (Synthetic pairs)
CBNNet
33.28
0.868
38.05
0,942
Zhou et al.
44
34.00
0.898
38.40
0,945
DrCNN
45
35.50*
0,827*
35.43*
0,906
Supervised (Real pairs)
AINDNet (R)
16
38.84
0,951
39.34
0,952
VDN 40
39.26
0,955
39.38
0,952
DANet 41
39.43
0,956
39.58
0.955
NAFNet
40.30
0.961
GCBD 6
35.58
0.922
Unsupervised
C2N [5
DIDN 39
35.35
0.937
37.28
0,924
(Unpaired)
D-BSN 36 + MWCNN 23
37.93
0.937
2oid
18
27.68
0.668
2Slf
29.56
0.808
NAC 38
36.20
0.925
Self-supervised
R2R 25
34.78
0.898
AP-BSN
21
35.97
0.925
38.09
0.937
PUCA (Ours)
37.54
0,936
38.83
0.942
4.2 Real World Denoising
We trained with real-world images in self-supervised way and validate our method using the popular SIDD and DND benchmark datasets. Table a quantitative comparison of various methods the SIDD and DND benchmarks with our method performing the best for self-supervised and unsupervised denoising. Unsupervised methods model noise using unpaired clean-noisy pairs, resulting in domain gap between training and testing. Self-supervised methods such as NAC (38) and R2R (25) are built upon less pragmatic assumptions, such as weak noise levels and known ISP function, resulting poor performance in real-world scenarios. Our method is applicable without any restrictions as it uses sRGB noisy images for training without additional information. Figures 6 and7ssoww the qualitative results for the SIDD and DND benchmarks. As shown Figure PUCA takes advantage of the global context to distinguish between meaningful features and noise showing improved denoising quality compared to the baseline methods. On the other hand, baseline methods such AP-BSN (11 fail in exploiting global context due to the limited receptive field with notable noise remaining. PUCA, which utilizes the U-Net structure, leverages multi-scale representation capturing features different scales. This enables the extraction of rich contextual information and facilitates the comprehension of overall object structures and their surrounding environments. As demonstrated in Figure 7 PUCA exhibits remarkable enhancements both globally and locally, specifically for complex images containing multiple objects, beyond mere flat images.
4.3 Ablation Study
Table 2: Ablation study PUCA
level with SIDD validation I.
As the number of levels in the U-Net increases, the receptive field becomes wider. Table shows quantitative results according the number of levels of the U-Net. We built several variations of our U-Net with different numbers of levels. Table shows that PSNR and SSIM increase as the model gets deeper. However, in the 4-level architecture, PSNR and SSIM drop compared level-3. This phenomenon is presumed to occur because pixel-
Level
PSNR
SSTM
36.768
0.875
2
37.231
0.878
37.492
0.880
36.910
0.878
7

---

A
DrCNN C5
C2N+DIDN 15
R2R P
AP-BSN 21
PUCA (Ours)
Figure 6: Qualitative comparison on SIDD benchmark
Access to the ground truth image and
evaluation values for denoised image is unavailable.
30.85dB/0.825 29.76d8/0.79
29.23d8/0.807
33.04dB/0.889
33.62dB/0.901
DrCNN 45
Zhou al. 47
C2N+DIDN [5]
AP-BSN 21
PUCA (Ours)
Figure Qualitative comparison on DND benchmark [26].
shuffle downsampling and patch-ume patch-unshuffle drastically reduce the latent features an extremely small
resolution leading to the degradation of global semantics.
Table confirms the effect of DAB and patch-unshuffle. As explained in Section 3.1, the generated output of pixel unshuffle as downsampling in a 3-layer U-Net structure is equal to the original noisy input (Table 3 (a)) because the model learns identity mapping. We run experiments with each component to test the role of DAB and patch-
Table 3: Ablation study on PUCA
components with SIDD validation [I]
DAB Unshuffle PSNR
SSIM
(a)
Pixel
23.662
0.328
unshuffle. For Table (b), we use 1-level network without
(b)
V
36.768
0.875
downsampling. For (c), we utilize the D-BSN block instead of DAB. In accordance with our intuition, Table (d), which utilizes both DAB and patch-unshuflle, exhibits the best performance compared to the ablated results.
(c)
Patch
37.386
0.880
(d)
V Patch
37.492
0.880
Related Work
5.1 Image Denoising
Supervised Image Denoising with Synthetic Pairs Since the success of CNNs 17
29
14
deep learning- baaning-based methods with CNNs have dominated the field of computer vision, including image denoising. DnCNN 455 first adopted the CNN architecture to image denoising, training on synthesized image pairs. Following this pioneering work, many studies have aimed build better architecture for image denoising 23
24 46 30 These learning-based methods rely clean-noisy
image pairs for training, where noisy images are synthesized with additive white Gaussian noise (AWGN). Recently, Guo et al. [13 has shown that this assumption of AWGN does not hold in

---

practice, and these methods show poor generalization performance due to the domain gap between real and synthetic noise.
Unsupervised Image Denoising Various approaches have been introduced to deal with the men- tioned domain gap between training and testing in image denoising. One straightforward approach uses the clean-noisy pairs captured from real-world [1 [S]. However, this approach is impractical since data collection extremely labor-intensive and requires a sophisticated process.
Another line of work is synthesizing realistic clean-noisy image pairs. Guo et al. 13 considers the
in-camera Image Signal Processing (ISP) pipeline for realistic noise synthesis. GCBD (6] adopt
adversarial training [ to model the noise distribution and build a paired dataset for training. C2N [15] takes the properties of real-world noise into account noise simulation.
Due to the notorious difficulty of acquiring clean-noisy pairs, self-supervised methods without clean images have been proposed. Noise2Noise (22 showed that denoising models could be trained without clean images. Noise2Void [8] and 2elf [ use a masking strategy for self seff-supervision. The
concept of blind-Spot Networks (BSNs) [8] was further improved by Laine et al. [20] and Wu et
al. [36]. On the other hand, there have been some works on loss functions (37 32) instead of explicitly masking prevent learning identity mapping. These methods all based on strong assumption that noise is pixel-wise independent. Yet, they eventually learn identity mapping when applied to real-world scenarios where noise is spatially correlated. Zhou al. [47) suggests pixel-shuflle downsampling (PD) to break the spatial correlation of real noise so that models trained with AWGN could adapt well. AP-BSN (2]] enhances this with asymmetric strides during training and testing.
LG-BPN 35 contemporary work to ours, proposes denser sampling kernel to recover local texture better and global branch with larger receptive fields.
5.2 Long Range Dependency
Large receptive fields are important to extract meaningful features in consideration of context. Stacking convolutional layers can linearly expand the receptive fields of a network. Yet, downsampling operation enables exponential growth of receptive fields and thus is a more favored option in increasing receptive fields. U-Net (27] is a representative structure used in low-level vision, which employs downsampling and upsampling and it enables long-range interaction of pixels. Also, the effectiveness of U-Net's coarse-and-fing representation has been demonstrated by previous works 19 |3| 43 44. Self-attention [31,, another form of handling long-range dependencies, was first introduced in natural language processing (NLP) and it has been known be also effective in computer vision 10). Although self-attention has shown promising results recently, it has a critical shortcoming of complexity quadratic the input resolution. Restormer 42 proposes to operate self-attention on the channel dimension rather than the spatial dimension in order to capture global information while saving computational cost. NAFNet 7 shows that simple gating and channel attention is sufficient to incorporate global context with better efficiency and performance. Following these lines of work, we aimed to apply these findings to self-supervised image denoising incorporating global context and coarse-an soarse-and-ao fine features of large receptive fields.
6 Conclusion
We present PUCA, a novel J -invariian U-Net tailored for self-supervised image denoising. By incorporating patch-unsshuffle/hhfffee and dilated attention block (DAB), we successfully alleviate the constrained architecture design imposed by J-invariance. First, we propose a novel downsampling technique called patch- unshuffl,, which plays a vital role in leveraging multi-scale representation and significantly expanding the receptive fields. Second, we propose DAB, which integrates global context and suppresses less informative features. The experimental results demonstrate the superiority of PUCA over existing self-supervised image denoising methods. Despite achieving state-of -hhe-----heettt---tte art performance in self-supervised image denoising. PUCA still has certain limitations. One such limitation is the degradation of global semantics when using an excessive number of levels. One potential approach to address this issue is to explore methods that can break input noise correlation, thus preserving the resolution of the input. In terms of societal impacts, like other denoising methods, PUCA could be misused to invade privacy or make incorrect diagnoses.

---

7 Acknowledgements
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea (MSTTT) (2022R AIBIABB117777 1077720 and 1922R1A540003000111 Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea gMVI) [2021-0-01345 Artificial Intelligence Graduate School Program (Seoul National University) and 2021 100-020668 Artificial Intelligence Innovation Hub] and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University in 2023.
References
[1] AbdeIrahman Abdelhamed, Stephen Lin, and Michael Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1692-1700, 2018.
[2] Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In Computer Vision-ECC 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part X 16, pages 111-126. Springer, 2020.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 2016.
[4] Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning, pages 524-533, PMLR, 2019.
[5] Benoit Brummer and Christophe De Vleeschouwer. Natural image noise dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[6] Jingwen Chen, Jiawei Chen, Hongyang Chao, and Ming Yang. Image blind denoising with generative adversarial network based noise modeling. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3155-3164 2018.
[7] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII. pages 17-33. Springer, 2022.
[8] Sung-Jin Cho, Seo-War Seo-Won Ji, Jun-Pyo Hong, Seung- Won Jung, and Sung-Jea Ko. Rethink- ing coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF international conference computer vision, pages 4641-4650, 2021.
[9] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domale collaborative filtering. IEEE Transactions on image processing. 16(8):2080-2099 2090000209,,.00 2007.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[11] Ian Goodfellow, Jean Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 11:1391144, 2020.
[12] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm min- imization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2862-2869, 2014.
[13] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1712-1722, 2019.
10

---

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Geonwoon Jang, Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. C2n: Practical generative noise modeling for real-world denoising. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2350-2359, 2021.
[16] Yoonsik Kim, Jac Woong Soh, Gu Yong Park, and Nam Ik Cho. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3482-3492, 2020.
[17] Alex Krizhevsky, lla Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6))84-90, 2017.
[18] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-leariing denoising from single noisy images. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2129-21377 2019.
[19] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblur- ring (orders-of-mape) of-mannitde)) faster and better. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[20] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self- supervisee deep image denoising. Advances in Neural Information Processing Systems, 32, 2019.
[21] Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap-hsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17725-17734, 2022.
[22] Jaakko Lehtinen, Jacob Munkherg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data, arXiv preprint 1803.04 2018.
[23] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet- cnn for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 773-782, 2018
[24] Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. Image restoration using very deep con- volutional encoder-decods networks with symmetric skip connections. Advances in neural information processing systems, 29, 2016.
[25] Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji. Recorrupted to-rupted-to-rrcorrupeedddd unsupervised deep learning for image denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2043-2052, 2021.
[26] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 1586-1595, 2017.
[27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAN 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[28) Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.
[29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.

---

[30] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE international conference on computer vision, pages 4539-4547, 2017.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[32] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. Blind2unblind: Self-supervised image denoising with visible blind spots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2027-2036, 2022.
[33] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang
Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17683-17693, 2022.
[34] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (3)) 612, 2004.
[35] Zichun Wang, Ying Fu, Ji Liu, and Yulun Zhang. Lg-bpn: Local and global blind- network for self-supervised real-world denoising. arXiv preprint arXiv:2304.005: 00534, 2023.
[36] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In Computer ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, pages 352-368. Springer, 2020.
[37] Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2same: Optimizing a self-supervised bound for image denoising. Advances in Neural Information Processing Systems, 33:20320- 20330, 2020.
[38] Jun Xu, Yuan Huang, Ming-Ming Cheng, Li Liu, Fan Zhu, Zhou Xu, and Ling Shao. Noisy-as- clean:
Learning self-supervised denoising from corrupted image. IEEE Transactions on Image Processing, 29:9316-9329, 2020.
[39] Songhyun Yu, Bumjun Park, and Jechang Jeong. Deep iterative down-up enn for image denoising. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[40] Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. Variational denoising network: Toward blind noise modeling and removal. Advances in neural information processing systems, 32, 2019.
[41] Zongsheng Yue, Qian Zhao, Lei Zhang, and Deyu Meng. Dual adversarial network: Toward real-world noise removal and noise generation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part X 16, pages 41-58. Springer, 2020.
[42] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- -suan Yang. Restormer: Efficient transformer for nigh-reeolltion image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728 5738,577,,200 2022.
[43] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14821-14831, 2021.
[44] Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and- play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence,
2021.
[45] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7):3142-31) 2017.
12

---

[46] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for enn-based image denoising. IEEE Transactions on Image Processing. 27(9):6608-4555 4622,, 2018.
[47] Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, and Thomas Huang. When awgn-based denoiser meets real noises. In Proceedings of the AAAI Conference on Artificial Intellience, volume 34, pages 13081, 2020.
13


Reviews:
[{"review_id": "wFTUe9Wt2L", "reviewer": "Reviewer_nv4m", "summary": "This paper introduces PUCA, a J-invariant U-Net for self-supervised image denoising. Specifically, the authors propose a patch-unshuffle and dilated attention block to allow the use of the multi-scale structure for enlarging the receptive field. Extensive experiments demonstrate that the proposed PUCA outperforms the existing method by a notable margin. There is also adequate analysis to illustrate the correctness and properties of the proposed method. ", "questions": "In L28, the authors emphasize the generalization issue of the supervised method and note that the unsupervised method can generalize better. However, this claim is not justified. Can the author show the cross-dataset validation of both the supervised method and the unsupervised method? ", "limitations": "The paper contains an adequate discussion of social impacts and limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Good Quality. The submission is technically sound in my opinion and the advantages and limitations of this work are discussed carefully and honestly.\n2. Good Clarity. The submission is written with sufficient clear definitions and formulas. And the organizations are well-designed.\n3. Good motivation. The authors observe that the commonly used simple structure, such as multi-scale structure,  in image denoising violates the J-invariance and thus cannot be used in BSNs. \n4. Solid solution. The authors propose patch unsuffle to achieve downsampling without violating j-invariance. In addition, proof is also provided to demonstrate this property of patch unshuffle theoretically. \n5. Sufficient experiments. The authors conduct sufficient experiments and ablation studies to demonstrate the effectiveness of the proposed method. The results show that the proposed method outperforms all existing methods by a notable margin. ", "weaknesses": "No obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**\n In L28, the authors emphasize the generalization issue of the supervised method and note that the unsupervised method can generalize better. However, this claim is not justified. Can the author show the cross-dataset validation of both the supervised method and the unsupervised method?\n\n**QA1**\nSorry for the confusion. The meaning of L28 in the main text was that supervised denoising approaches require extreme cost in data collection, making it challenging to gather large-scale data, which would inevitably lead to limited generalization ability. We will revise the main text to resolve this confusion. In line with your request, we conducted cross-dataset experiments (testing the model trained on SIDD with the DND benchmark) and obtained the following reasonable results:\n\n|                              | SIDD (PSNR/SSIM) | DND (PSNR/SSIM) |\n|------------------------------|------------------|-----------------|\n| Restormer (SIDD trained)[43] | 40.02/0.960      | 40.03/0.956     |\n| PUCA (SIDD trained)          | 37.54/0.936      | 38.60/0.940     |\n\nThe results of Restormer, a type of supervised image denoiser, are from the results reported in the original paper [43]. We would like to emphasize the significance of our self-supervised image denoiser demonstrating comparable performance to the supervised image denoiser.\n \n"}, {"review_id": "T0YhIR8LIW", "reviewer": "Reviewer_daq5", "summary": "This paper presents a method for self-supervised image denoising. Specifically, a patch-unshuffle (and shuffle) operation together with a dilated attention block was proposed to achieve the goal. Experimental results on two real noise datasets (SIDD and DND) show the effectiveness of the proposed method over other self-supervised methods.", "questions": "It would be helpful if the authors could address the concerns raised in the above Weaknesses section. For example, the main motivation of the proposed method and its validity; the technical novelty of the newly proposed components (i.e. the patch-unshuffle and the DAB blocks); those concerns about the experiments.\n\nThe reviewer would be happy to change the rating if the concerns could be well addressed.", "limitations": "The authors clearly mention the limitations of their method (in the Conclusion) and potential societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Real noisy image denoising is an important problem in the denoising community and the authors also alleviate the limitations from the $\\mathcal{J}$-invariance to some extent.\n\n+ The proposed method was shown to perform generally better than other self-supervised methods on the SIDD and DND real noise datasets.\n\n+ The paper is generally well-written and easy to follow.", "weaknesses": "- From the description, it seems the authors were motivated by the flexibility in network structures. But it is unclear why the flexibility in network structure design is necessary, or how would it benefit the denoising community.\n\n- The novelty of the proposed patch-unshuffle/shuffle is a bit limited, especially considering the previously proposed pixel-unshuffle/shuffle approach [47]. The key idea is very similar with the only difference as pixel vs. patch. Whereas the significance of such a change and novelty were not well clarified with convincing justifications. For example, what if replace the patch-unshuffle/shuffle blocks in the proposed method with the corresponding pixel-wise version? It is also unclear what is the contribution of the PD and PU processing at the input/output ends. What if remove these processing operations and only use the proposed patch-wise blocks?\nOn the other hand, from what was shown in Fig.5 in [47], it seems that they already used a similar idea of patch-shuffle.\n\n- The proposed patch-unshuffle was motivated by the claimed issue of breaking the \\mathcal{J}-invariance in pixel-unshuffle. But as also acknowledged by the authors, the proposed patch-unshuffle can still break it in some cases.\n\n- The novelty of the proposed dilated attention block is also a bit limited. As mentioned by the authors, it was based on the channel attention [7] being applied to U-Net, with inspirations from D-BSN [36] and SCA [7]. If there is a new design involved, the contribution and novelty are unclear, and it lacks convincing (experimental) validation, e.g. what if without the new design? how the performance relates to the proposed design?\n\n- It is unclear what dilation rate was used in the proposed DAB block. And unclear how this hyper-parameter d affects the model's performance.\n\n- The proposed method performs worse than the C2N+DIDN for the SSIM (Table 1). We know that SSIM usually represents the visual or structural quality of the reconstructed image. But there is no clarification for this.\n\n- In the ablation study (Table 2), the authors attribute the decreased performance when the level increased from 3 to 4, to the resolution of the latent features. To validate this, an experiment with a larger input image (thus a higher resolution for the latent features when level=4) should have been included.\n\n- Although LG-BPN [35] may not be published (and not published at CVPR) when the proposed method was submitted, the idea of local-global branches and the larger receptive fields through dilated convolutions is very similar to the proposed method. The novelty is as a result weakened.\n\n- The proposed method claimed the issues within existing methods about the \\mathcal{J}-invariance, and motivated by this, claimed that the proposed method \"successfully alleviates the constrained architecture design imposed\" by this. But it is unclear how the constraints for architecture design were alleviated. This was not validated, unless other designs were shown to be effective with the proposed method.\n\n- Missing definition to p (L136-144). If it's the same as the p in L145, please specify (the first time it appears).\n\n- It is unclear what \"PUCA\" indicates, \"Patch-Unshuffle Channel Attention\", or \"xxxUnetxxx\"? Please clearly define it the first time it was introduced.\n\n- In the caption of Fig. 3, \"During encoding...through patch shuffling...\" should be \"patch unshuffling\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Table 3: Ablation study on PUCA components with SIDD validation \n|     | DAB | Unshuffle | PSNR   | SSIM  |\n|-----|-----|-----------|--------|-------|\n| (a) | -   | Pixel     | 23.662 | 0.328 |\n| (b) | V   | -         | 36.768 | 0.875 |\n| (c) | -   | Patch     | 37.386 | 0.880 |\n| (d) | V   | Patch     | 37.492 | 0.880 |\n\n**W1** BSN's evolution has shifted from input-masking to a centrally-masked kernel approach for efficiency and reduced artifacts. Due to BSN's constraints, only stacked dilated convolutions were viable. In contrast, supervised denoising, like U-Net [2,8,19,33,41,42,46], enjoys more design flexibility. U-Net extends receptive fields, extracts multi-scale global context, and combines details through skip connections, making it effective. By infusing scalability into self-supervised denoising design, we foresee new network structures emerging for enhanced performance.\n\n**W2** In self-supervised denoising, the target image matches the input noisy image. To prevent identity mapping, BSNs have been used, requiring j-invariance. However, centrally masked/dilated convolutions with pixel-shuffle/unshuffle can disrupt j-invariance and hamper denoising. [47] introduced pixel-shuffle down-sampling adaptation as shown in Figure 5 of [47]. To maintain j-invariance and successful denoising, we propose patch-unshuffle/shuffle for downsampling/upsampling. Table 3 (a) in the main text highlights that pixel-shuffle/unshuffle generates identical outputs to the input image.\n\nReal noise displays correlations distinct from synthetic noise, challenging the assumption of independent zero-mean noise in synthetic methods. Thus, synthetic-based approaches struggle with real noise generalization. PD disrupts noise correlations by arranging subsamples along the channel axis onto a plane as seen in AP-BSN’s Figures 4 and 5. PU recombines subsampled images. While j-invariance remains intact without PD and PU, real noise's characteristics allow predictions influenced by neighboring pixels, resembling identity mapping. This applies to AP-BSN and LG-BPN as well.\n\n**W3** D-BSN constructs its network through a combination of centrally masked/dilated convolution. As demonstrated in Noise2Kernel*, maintaining J-invariance necessitates a dilation of dilated convolution, $d \\geq \\mathrm{ceil}(K/2)$ (where $\\mathrm{ceil}(a)$ represents the smallest integer greater than or equal to a, and $K$ signifies the kernel size of centrally masked convolution). Similar to adjusting the dilation of dilated convolution to satisfy BSN's requirement, adjusting the patch size of patch-unshuffle serves as a method to meet the demands of BSN.\n\n**W4** Using the channel attention [7] directly would break j-invariance. Hence, we made the modification of changing the convolution in the channel attention to dilated convolution. While D-BSN could be utilized as is, we incorporated channel attention to gain ensemble effects from the subsamples generated by patch-shuffle/unshuffle. In Table 3 (c) and (d), it can be observed that the presence of DAB results in a PSNR increase of 0.106.\n\n**W5** In the main text, we utilized a dilation of 2, and in accordance with your request, the effects of experimentation with different dilations are as follows:\n\n| Dilation | PSNR   | SSIM  |\n|----------|--------|-------|\n| 2        | 37.492 | 0.880 |\n| 3        | 37.284 | 0.884 |\n| 4        | 36.824 | 0.879 |\n\n As the dilation size increases, the PSNR decreases. We infer that this phenomenon occurs due to the simultaneous increase in dilation and patch-unshuffle size, which presents challenges in reconstructing local information.\n\n**W6** \nOur intuition is that C2N+DIDN's stabilizing loss term helps maintain the luminance and contrast of the dataset, and the addition of C2N-generated noise to the clean image enhances structural aspects, leading to high SSIM scores. Furthermore, we infer that the SIDD dataset's larger object sizes relative to the noise positively contribute to C2N+DIDN achieving high SSIM values.\n\n**W7** As per your request, we measured the performance based on level variations in the DND dataset with a size of 512x512. The results are as follows:\n\n| level   | PSNR   | SSIM  |\n|---------|--------|-------|\n| level-3 | 38.884 | 0.942 |\n| level-4 | 39.030 | 0.944 |\n\nAs anticipated in the main text, it is evident that as the resolution increases, performance improves with deeper levels (level 3 to level 4).\n\n**W8** LG-BPN augmented receptive fields through expanded convolutions and a local-global branch. PUCA effectively expands the receptive field by downsampling/upsampling feature maps using patch-unshuffle/shuffle. Furthermore, by extracting global context from a multi-scale representation and integrating fine details through skip connections, PUCA benefits denoising without the need for local-global separation.\n\n**W9** Sorry for the confusion. We intended that we introduced scalability to network design. To assess the potential for extension in U-Net variants, we modified [8], considering the requirements of BSN. As a result, we obtained a PSNR of 37.209 dB and an SSIM of 0.875 on the SIDD validation. While further optimization seems necessary, we anticipate that the components of PUCA can be applied to various U-Net variants to enhance performance.\n\n**W10** With the same $p$ as in L145, we will define it according to your opinion when it first appears.\n\n**W11** \"PUCA\" stands for \"Patch-Unshuffle and Channel Attention,\" and, we will make sure to properly include it in the introduction as you suggested. Thank you for the advice.\n\n**W12** We will make the correction to \"Patch Unshuffling\" in Figure 3 as you mentioned.\n\n[*] Noise2kernel: Adaptive self-supervised blind denoising using a dilated convolutional kernel architecture. Sensors, 2022\n\n\n\n\n\n"}, {"review_id": "RMYFI5ZEq4", "reviewer": "Reviewer_DQiP", "summary": "This work extends the field of blind spot networks (BSN) for self-supervised image denoising. They propose patch-unshuffle/shuffle a technique for downsampling/upsampling that preserve J-invariance, which allows to build U-Net like architectures expanding the receptive field and utilizing multi-scale representations in BSN.\nSecond, the dilated attention block is introduced, a J-invariant channel attention mechanism incorporating global information. The method outperforms existing self-supervised methods by 1.6dB (SIDD dataset) and 0.7dB (DND dataset) in PSNR resulting also in improved perceived image quality for the depicted examples.", "questions": "1. Choosing the by far worst performing supervised baseline as the only visual reference for supervised training in Figure 1 seems a bit misleading with respect to the comparison of supervised and self-supervised methods. \n2. The caption in Table 3 (3b with DAB) does not fit the description in the text (3b no DAB).\n3. Line 256 refers to other work without citing it.\n4. The paper uses a numeric citation style in the text but alphabetic in the references making it impossible to match citations to entries in the references.", "limitations": "The authors briefly touch on one limitation of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is interesting and alleviates the constrained architecture design imposed by J-invariance opening up possibilities for new designs of BSNs.\n2. The experiments on the two most common benchmarks for self-supervised image denoising are extensive and the method is compared to a range of baseline methods.\n3. The authors theoretically derive the J-invariance property of their proposed patch-unshuffle method.", "weaknesses": "1. Section 3.2 introduces the dilated attention block (DAB) that preserve J-invariance. However, from the sentence \"To fulfill this requirement we incorporate a d-dilated 3x3 depth-wise convolution (DDC) before gating and attention, taking inspiration from D-BSN\" it is not clear to me how exactly J-invariance is preserved nor if or how the approach differs from the previous work.\n2. From Section 2.2 it is not clear to me how pixel-shuffle downsampling (PD) works. From Figure 4 it seems as PD changes the dimension of the input image, however this is not discussed in Section 2.2 and the algorithm overview in Figure 3 shows the signal before and after PD to exhibit the same dimension.\n3. I find that dropping the channel dimension in the proof in Section 3.1 is a bit confusing as now several input indices are mapped to the same output index. Maybe an illustration with an explicit example of how a set of indices is transformed might be helpful.\n4. Summarizing the above points the clarity of the work could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1** Section 3.2 introduces the dilated attention block (DAB) that preserve J-invariance. However, from the sentence \"To fulfill this requirement we incorporate a d-dilated 3x3 depth-wise convolution (DDC) before gating and attention, taking inspiration from D-BSN\" it is not clear to me how exactly J-invariance is preserved nor if or how the approach differs from the previous work.\n\n**WA1**\nIn Figure 3 of the main text, layers of the DAB block are depicted. The DAB block integrates components including LayerNorm and 1x1 convolution, DDC, SimpleGate, and SCA. LayerNorm and 1x1 convolution do not affect j-invariance. SimpleGate performs element-wise multiplication on two chunks divided along the channel axis, thereby maintaining j-invariance similarly. SCA goes through global average pooling and a fully connected layer to create channel-attention. Then, it multiplies the input and the channel-attention along the channel axis Consequently, even in this scenario, it has no impact on j-invariance.\nIn the context of the conventional channel-attention block, using simple depth-wise convolution instead of DDC disrupts the j-invariance upheld by centrally masked/dilated convolutions, as illustrated in Figure 2. To address this, DDC (a dilated depth-wise convolution) is proposed to maintain j-invariance. While D-BSN can be used as is, channel attention was introduced to harness ensemble effects from the subsamples generated by patch-shuffle/unshuffle. As indicated in Table 3 (c) and (d), the presence of DAB leads to a PSNR increase of 0.106.\n\nTable 3: Ablation study on PUCA components with SIDD validation \n|     | DAB | Unshuffle | PSNR   | SSIM  |\n|-----|-----|-----------|--------|-------|\n| (a) | -   | Pixel     | 23.662 | 0.328 |\n| (b) | V   | -         | 36.768 | 0.875 |\n| (c) | -   | Patch     | 37.386 | 0.880 |\n| (d) | V   | Patch     | 37.492 | 0.880 |\n\n---\n\n**W2** From Section 2.2 it is not clear to me how pixel-shuffle downsampling (PD) works. From Figure 4 it seems as PD changes the dimension of the input image, however, this is not discussed in Section 2.2 and the algorithm overview in Figure 3 shows the signal before and after PD to exhibit the same dimension.\n\n**WA2**\nPUCA adopts the pixel-downsampling approach from AP-BSN [21]. Similar to the observation in Figures 4 and 5 of AP-BSN, pixel-downsampling (PD) serves to arrange the subsamples that are listed along the channel axis of pixel-unshuffle onto a single plane. Consequently, Figure 3 of the main text exhibits the same channel dimensions before and after PD.\n\n---\n\n**W3** I find that dropping the channel dimension in the proof in Section 3.1 is a bit confusing as now several input indices are mapped to the same output index. Maybe an illustration with an explicit example of how a set of indices is transformed might be helpful.\n\n**WA3**\nWe agree with your suggestion, multiple input indices mapping to the same output index could potentially lead to confusion. We visualized the operation principle of patch-unshuffle in Figure 4 (b). Patches with the same color are assembled along the same channel axis, and patches of different colors are arranged in the same order as indicated by the red patches.\n\n---\n\n**Q1** Choosing the by far worst performing supervised baseline as the only visual reference for supervised training in Figure 1 seems a bit misleading with respect to the comparison of supervised and self-supervised methods.\n\n**QA1**\nWe agree with your suggestion and we will include additional supervised models into the final version.\n\n---\n\n**Q2** The caption in Table 3 (3b with DAB) does not fit the description in the text (3b no DAB).\n\n**QA2** As you mentioned, the table caption in the main text has indeed been changed. We willl make the necessary corrections. Thank you for bringing this to our attention.\n\n---\n\n**Q3** Line 256 refers to other work without citing it.\n\n**QA3** We will add the citation to the work at line 256, as you suggested.\n\n---\n\n**Q4** The paper uses a numeric citation style in the text but is alphabetic in the references making it impossible to match citations to entries in the references.\n\n**QA4** We will match the bibliography style to the number citation style. This should help avoid any confusion.\n"}, {"review_id": "xVKTX2eVMo", "reviewer": "Reviewer_jJKN", "summary": "This work addresses the problem of developing a self-supervised learning-based denoiser. To this end, this work proposes the concepts of patch shuffle/unshuffle operations to effectively downsample and upsample the features while ensuring that the J-invariance property holds with the resulting network. Using dilated attention blocks and patch shuffle/unshuffle, this work comes up with a variant of Unet architecture that can significantly improve J-invariance networks' spatial information aggregation ability. The proposed method improves the denoising performance on public benchmark datasets.", "questions": "- Please address my comments under weaknesses. \n\n- Visual comparisons with other self supervised methods (example, Figure 6) reveals as if the proposed method lacks details while removing the noise completely whereas other methods retains more details. Is there an explanation on why so? Is it possible to control the denoising level of the proposed method to earn back the lost details?", "limitations": "Limitations has been addressed adequately.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written well.\n2. The new notions of patch shuffle/unshuffle and dilated attention blocks to ensure the holding of J-invariance properly is interesting\n3. The proposed method outperforms prior arts significantly", "weaknesses": "1. A study is missing to elaborate more on what other network architectures the proposed ideas can find a perfect fit. There have been numerous variants of Unet proposed in the literature. Can those extensions boost the performance further remain unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1** A study is missing to elaborate more on what other network architectures the proposed ideas can find a perfect fit. There have been numerous variants of U-Net proposed in the literature. Can those extensions boost the performance further remain unclear.\n\n**WA1**\nStructures employing self-attention, such as Uformer [42], are left for future work due to the potential of self-attention to compromise J-invariance. Adhering to BSN's requirements, we adapted the MIMO-UNet [8]. Our training yielded results of PSNR: 37.209 dB and SSIM: 0.875 on the SIDD validation set. Due to time constraints, extensive hyperparameter searching was not conducted. However, our modified MIMO-UNet surpassed the denoising performance of AP-BSN [21]. Thus, we anticipate that PUCA could enhance performance across various U-Net variants.\n\n---\n\n**Q1** Visual comparisons with other self supervised methods (example, Figure 6) reveals as if the proposed method lacks details while removing the noise completely whereas other methods retains more details. Is there an explanation on why so? Is it possible to control the denoising level of the proposed method to earn back the lost details?\n\n**QA1**\nSorry for the confusion. It might seem that we have lost some details by not visualizing the ground truth together, but it is apparent from Figure 6 that PUCA indeed displays sharper edges compared to other models. Given this observation, we believe that our method does not miss the details. It seems that the baselines might preserve detail that appears more akin to less-removed noise, rather than enhancing real image details, as per our perception. Upon reviewing Figure 1 and the supplementary materials, we can confirm that PUCA represents finer details like text with greater clarity. Adjusting the level or dilation of the model does offer the possibility to control details, but along with that, denoising also becomes weaker, making it less practical.\n"}, {"review_id": "l0eM9z5l88", "reviewer": "Reviewer_3t6u", "summary": "This paper propose a novel network architecture for self-supervised image denoising. The network is build with foundation blocks of NAFNet, which combines mask and dilated convolutions to implement the blind-spot network. Patch-unshuffle is introduced as downsample/upsample operations with enables multi-scale design of the network while maintaining the blind-spot mechanism. Experimental results show the effectiveness of proposed method on real-world denoising datasets.", "questions": "My major concern is weakness 2 and 3, please complete the necessary experiments.", "limitations": "The authors hav adequately addressed the limitations and potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of searching for flexible network architecture is reasonable for improving the self-supervised denoising performance.\n2. Patch-unshuffle is well designed to enable downsample/upsample operations for implementing multi-scale network architecture.\n3. Experimental results show the effectiveness of proposed method in self-supervised image denoising on real-world images.\n4. The paper is well written and easy to follow.", "weaknesses": "1. The overall novelty is limited. The training process is the same as AP-BSN, and the network architechture is assembled with existing DBSN and NAFNet. The only contribution is the idea of improving the network flexibility and to implement a multi-scale network with patch-unshuffle, which is not technically sound.\n2. The ablation study is not complete. The central idea of this paper is to improve DBSN used in AP-BSN with multi-scale architecture, however, there are existing multi-scale BSN architechture such as [1]. What's the performance of porposed network compare with [1]?\n3. The competing methods is not complete. This paper shows good performance on real-world image denoising, but the latest self-supervised denoising methods [2][3] are ignored.\n\n[1] High-quality self-supervised deep image denoising. NIPS, 2019.\n\n[2] LG-BPN: Local and Global Blind-Patch Network for Self-Supervised Real-World Denoising. CVPR, 2023.\n\n[3] Spatially Adaptive Self-Supervised Learning for Real-World Image Denoising. CVPR, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**\nThe overall novelty is limited. The training process is the same as AP-BSN, and the network architechture is assembled with existing DBSN and NAFNet. The only contribution is the idea of improving the network flexibility and to implement a multi-scale network with patch-unshuffle, which is not technically sound.\n\n**WA1** \nAs you have mentioned in the strengths section, we devised multi-scale network architectures through patch-unshuffle/shuffle to enhance self-supervised denoising performance. This outcome emerges from the expansion of network design, which was previously constrained due to the requirements of BSN. Additionally, by incorporating the J-invariant block with channel attention, known as DAB, we achieved an ensemble effect from the subsamples generated by patch-unshuffle. This successful integration allowed us to effectively elevate denoising performance.\n\n---\n\n**W2**\nThe ablation study is not complete. The central idea of this paper is to improve DBSN used in AP-BSN with multi-scale architecture, however, there are existing multi-scale BSN architechture such as [1]. What's the performance of porposed network compare with [1]?\n\n**WA2** \nBased on our understanding of [1] you referred to, it appears that all convolutional layers employ a shifted upwards kernel with zero values below the center row and incorporate offsets. The same approach is used for downsampling/upsampling, utilizing offset methods along with either average pooling or nearest-neighbor upsampling. It seems plausible that for ablation studies: 1) DAB could be replaced with the proposed convolution and 2) patch-unshuffle/shuffle could be substituted with offset-incorporated avg. pooling/nearest-neighbor upsampling.\nIn [1], the maintenance of j-invariant is achieved through a combination of downward rows with zero-value convolutions, offsets, and avg. pooling/nearest-neighbor upsampling. Our method is optimized for centrally masked convolutions and dilated convolutions, which is why using the alternative approach that we suggested could potentially break the j-invariant. Consequently, both methods might yield results similar to pixel-unshuffle/shuffle. \n\n---\n\n**W3**\n The competing methods is not complete. This paper shows good performance on real-world image denoising, but the latest self-supervised denoising methods [2][3] are ignored.\n\n**WA3**\nAt the time of our proposed method's submission, neither [2] nor [3] had been published. However, in response to your request, we conducted a comparison.\nLG-BPN [2] augmented receptive fields through expanded convolutions and a local-global branch. SASL [3] employed a Blind-Neightborgood Network and a Locally Aware Network to separate denoising into textured and flat regions.\nPUCA effectively expands the receptive field by downsampling/upsampling feature maps using patch-unshuffle/shuffle. Furthermore, by extracting global context from a multi-scale representation and integrating fine details through skip connections, PUCA benefits denoising without the need for distinct regions or local-global separation, unlike other methods.\nThe performance on the SIDD and DND benchmarks is as follows, with results from [2] and [3] extracted from the original texts:\n\n|             | SIDD benchmark (PSNR/SSIM) | DND benchmark (PSNR/SSIM) |\n|-------------|----------------------------|---------------------------|\n| PUCA (Ours) | 37.54/0.936                | 38.83/0.942               |\n| LG-BPN [2]  | 37.28/0.936                | 38.43/0.942               |\n| SASL [3]    | 37.41/0.934                | 38.18/0.938               |\n\nThank you for your suggestion. we will incorporate it into the final version.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"}]

Author rebuttal:
We thank reviewers for the positive comments and encouraging remarks: 

“The authors observe that the commonly used simple structure, such as multi-scale structure, in image denoising violates the J-invariance and thus cannot be used in BSNs.” (**nv4m**)

“The proposed method is interesting and alleviates the constrained architecture design imposed by J-invariance opening up possibilities for new designs of BSNs.”(**nv4m**, **daq5**, **DQiP**,  **jJkN**, **3t6u**)

“The proposed method outperforms prior arts significantly” (**nv4m**, **daq5**, **jJkN**, **3t6u**)

“The submission is technically sound in my opinion and the advantages and limitations of this work are discussed carefully and honestly.” (**nv4m**, **daq5**, **jJkN**, **3t6u**)

“The submission is written with sufficiently clear definitions and formulas. And the organizations are well-designed.” (**nv4m**, **daq5**, **jJkN**, **3t6u**)


We sincerely appreciate your thorough understanding of the method and careful review of the paper. We are truly grateful for your valuable insights and advice.


Meta-review: This paper presents a method for self-supervised image denoising. Specifically, a patch-unshuffle (and shuffle) operation together with a dilated attention block was proposed to achieve the goal. Experimental results on two real noise datasets (SIDD and DND) show the effectiveness of the proposed method over other self-supervised methods. During the review stage, issues like motivation, novelty/technical details, clarification/presentation, and experiments were raised. The authors provided detailed rebuttal responses which address the above concerns. Since all reviewers are positive on this work, it is clear that this paper is above the threshold for publication.
Decision: Accept (poster)