SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We trained with real-world images in a self-supervised way and validate our method using the popular SIDD and DND benchmark datasets. Table 1 shows a quantitative comparison of various methods on the SIDD and DND benchmarks with our method performing the best for self-supervised and unsupervised denoising. Unsupervised methods model noise using unpaired clean-noisy pairs, resulting in domain gap between training and testing. Self-supervised methods such as NAC [38] and R2R [25] are built upon less pragmatic assumptions, such as weak noise levels and known ISP function, resulting in poor performance in real-world scenarios. Our method is applicable without any restrictions as it uses sRGB noisy images for training without additional information. Figures 6 and 7 show the qualitative results for the SIDD and DND benchmarks. As shown in Figure 6, PUCA takes advantage of the global context to distinguish between meaningful features and noise showing improved denoising quality compared to the baseline methods. On the other hand, baseline methods such as AP-BSN [21] fail in exploiting global context due to the limited receptive field with notable noise remaining. PUCA, which utilizes the U-Net structure, leverages multi-scale representation capturing features at different scales. This enables the extraction of rich contextual information and facilitates the comprehension of overall object structures and their surrounding environments. As demonstrated in Figure 7, PUCA exhibits remarkable enhancements both globally and locally, specifically for complex images containing multiple objects, beyond mere flat images.
As the number of levels in the U-Net increases, the receptive field becomes wider. Table 2 shows quantitative results according to the number of levels of the U-Net. We built several variations of our U-Net with different numbers of levels. Table 2 shows that PSNR and SSIM increase as the model gets deeper. However, in the 4-level architecture, PSNR and SSIM drop compared to level-3. This phenomenon is presumed to occur because pixel
Table 2: Ablation study on PUCA level with SIDD validation [1].
Level	PSNR	SSIM
1	36.768	0.875
2	37.231	0.878
3	37.492	0.880
4	36.910	0.878
Figure 6: Qualitative comparison on SIDD benchmark [1]. Access to the ground truth image and evaluation values for a denoised image is unavailable.
Figure 7: Qualitative comparison on DND benchmark [26].
shuffle downsampling and patch-unshuffle drastically reduce the latent features to an extremely small resolution leading to the degradation of global semantics.
Table 3 confirms the effect of DAB and patch-unshuffle. As explained in Section 3.1, the generated output of pixel unshuffle as downsampling in a 3-layer U-Net structure is equal to the original noisy input (Table 3 (a)) because the model learns identity mapping. We run experiments with each component to test the role of DAB and patch-unshuffle. For Table 3 (b), we use a 1-level network without downsampling. For Table 3 (c), we utilize the D-BSN block instead of DAB. In accordance with our intuition, Table 3 (d), which utilizes both DAB and patch-unshuffle, exhibits the best performance compared to the ablated results.
Table 3: Ablation study on PUCA components with SIDD validation [1]
DAB	Unshuffle	PSNR	SSIM
(a)	-	Pixel	23.662	0.328
(b)	✓	-	36.768	0.875
(c)	-	Patch	37.386	0.880
(d)	✓	Patch	37.492	0.880
5.1 Image Denoising
Supervised Image Denoising with Synthetic Pairs Since the success of CNNs [17, 29, 14], deep learning-based methods with CNNs have dominated the field of computer vision, including image denoising. DnCNN [45] first adopted the CNN architecture to image denoising, training on synthesized image pairs. Following this pioneering work, many studies have aimed to build a better architecture for image denoising [23, 24, 46, 30]. These learning-based methods rely on clean-noisy image pairs for training, where noisy images are synthesized with additive white Gaussian noise (AWGN). Recently, Guo et al. [13] has shown that this assumption of AWGN does not hold in
practice, and these methods show poor generalization performance due to the domain gap between real and synthetic noise.
Unsupervised Image Denoising Various approaches have been introduced to deal with the mentioned domain gap between training and testing in image denoising. One straightforward approach uses the clean-noisy pairs captured from real-world [1, 5]. However, this approach is impractical since data collection is extremely labor-intensive and requires a sophisticated process.
Another line of work is synthesizing realistic clean-noisy image pairs. Guo et al. [13] considers the in-camera Image Signal Processing (ISP) pipeline for realistic noise synthesis. GCBD [6] adopt adversarial training [11] to model the noise distribution and build a paired dataset for training. C2N [15] takes the properties of real-world noise into account in noise simulation.
Due to the notorious difficulty of acquiring clean-noisy pairs, self-supervised methods without clean images have been proposed. Noise2Noise [22] showed that denoising models could be trained without clean images. Noise2Void [18] and Noise2Self [4] use a masking strategy for self-supervision. The concept of blind-Spot Networks (BSNs) [18] was further improved by Laine et al. [20] and Wu et al. [36]. On the other hand, there have been some works on loss functions [37, 32] instead of explicitly masking to prevent learning identity mapping. These methods are all based on a strong assumption that noise is pixel-wise independent. Yet, they eventually learn identity mapping when applied to real-world scenarios where noise is spatially correlated. Zhou et al. [47] suggests pixel-shuffle downsampling (PD) to break the spatial correlation of real noise so that models trained with AWGN could adapt well. AP-BSN [21] enhances this with asymmetric strides during training and testing. LG-BPN [35], a contemporary work to ours, proposes a denser sampling kernel to recover local texture better and a global branch with larger receptive fields.
5.2 Long Range Dependency
Large receptive fields are important to extract meaningful features in consideration of context. Stacking convolutional layers can linearly expand the receptive fields of a network. Yet, downsampling operation enables exponential growth of receptive fields and thus is a more favored option in increasing receptive fields. U-Net [27] is a representative structure used in low-level vision, which employs downsampling and upsampling and it enables long-range interaction of pixels. Also, the effectiveness of U-Net's coarse-and-fine representation has been demonstrated by previous works [2, 8, 19, 33, 41, 43, 44]. Self-attention [31], another form of handling long-range dependencies, was first introduced in natural language processing (NLP) and it has been known to be also effective in computer vision [10]. Although self-attention has shown promising results recently, it has a critical shortcoming of complexity quadratic to the input resolution. Restormer [42] proposes to operate self-attention on the channel dimension rather than the spatial dimension in order to capture global information while saving computational cost. NAFNet [7] shows that simple gating and channel attention is sufficient to incorporate global context with better efficiency and performance. Following these lines of work, we aimed to apply these findings to self-supervised image denoising incorporating global context and coarse-and-fine features of large receptive fields.
We present PUCA, a novel  $\mathcal{I}$ -invariant U-Net tailored for self-supervised image denoising. By incorporating patch-unshuffle/shuffle and dilated attention block (DAB), we successfully alleviate the constrained architecture design imposed by  $\mathcal{I}$ -invariance. First, we propose a novel downsampling technique called patch-unshuffle, which plays a vital role in leveraging multi-scale representation and significantly expanding the receptive fields. Second, we propose DAB, which integrates global context and suppresses less informative features. The experimental results demonstrate the superiority of PUCA over existing self-supervised image denoising methods. Despite achieving state-of-the-art performance in self-supervised image denoising, PUCA still has certain limitations. One such limitation is the degradation of global semantics when using an excessive number of levels. One potential approach to address this issue is to explore methods that can break input noise correlation, thus preserving the resolution of the input. In terms of societal impacts, like other denoising methods, PUCA could be misused to invade privacy or make incorrect diagnoses.
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (2022R1A3B1077720 and 2022R1A5A708390811), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [2021-0-01343: Artificial Intelligence Graduate School Program (Seoul National University) and 2021-0-02068: Artificial Intelligence Innovation Hub] and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2023.
[1] Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1692-1700, 2018.
[2] Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pages 111-126. Springer, 2020.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[4] Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning, pages 524-533. PMLR, 2019.
[5] Benoit Brummer and Christophe De Vleeschouwer. Natural image noise dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
[6] Jingwen Chen, Jiawei Chen, Hongyang Chao, and Ming Yang. Image blind denoising with generative adversarial network based noise modeling. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3155-3164, 2018.
[7] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pages 17-33. Springer, 2022.
[8] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4641-4650, 2021.
[9] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080-2095, 2007.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
[12] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2862-2869, 2014.
[13] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1712-1722, 2019.
[46] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Transactions on Image Processing, 27(9):4608-4622, 2018.
[47] Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, and Thomas Huang. When awgn-based denoiser meets real noises. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13074–13081, 2020.