Self-Supervised Image Restoration with Blurry and Noisy Pairs
Authors: Zhilu Zhang
RongJian Xu
Ming Liu
Zifei Yan
Wangmeng Zuo
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-supervised learning
image restoration
blurry and noisy pair

Abstract: When taking photos under an environment with insufficient light, the exposure time and the sensor gain usually require to be carefully chosen to obtain images with satisfying visual quality. For example, the images with high ISO usually have inescapable noise, while the long-exposure ones may be blurry due to camera shake or object motion. Existing solutions generally suggest to seek a balance between noise and blur, and learn denoising or deblurring models under either full- or self-supervision. However, the real-world training pairs are difficult to collect, and the self-supervised methods merely rely on blurry or noisy images are limited in performance. In this work, we tackle this problem by jointly leveraging the short-exposure noisy image and the long-exposure blurry image for better image restoration. Such setting is practically feasible due to that short-exposure and long-exposure images can be either acquired by two individual cameras or synthesized by a long burst of images. Moreover, the short-exposure images are hardly blurry, and the long-exposure ones have negligible noise. Their complementarity makes it feasible to learn restoration model in a self-supervised manner. Specifically, the noisy images can be used as the supervision information for deblurring, while the sharp areas in the blurry images can be utilized as the auxiliary supervision information for self-supervised denoising. By learning in a collaborative manner, the deblurring and denoising tasks in our method can benefit each other. Experiments on synthetic and real-world images show the effectiveness and practicality of the proposed method. Codes are available at https://github.com/cszhilu1998/SelfIR.

Self-Supervised Image Restoration with Blurry and Noisy Pairs
Zhilu Zhang's
Rongjian Xu', Ming Liu', Zifei Yan Wangmeng Zuo 1.2
Harbin Institute of Technology, Harbin, China;
2 Peng Cheng Laboratory, China
cszlzhang@outlook com, ronjon www.ooooonaall...... com, csmliu@outlook. com, manzifeihhic edu. en, wmzuoohit. edu.
Abstract
When taking photos under environment with insufficient light, the exposure time and the sensor gain usually require to be carefully chosen to obtain images with satisfying visual quality. For example, the images with high ISO usually have inescapable noise, while the long-exposure ones may be blurry due to camera shake or object motion. Existing solutions generally suggest seek a balance between noise and blur, and learn denoising or deblurring models under either full- or self-supervision. However, the real-world training pairs are difficult collect, and the self- supervieed methods merely rely on blurry or noisy images are limited in performance. In this work, we tackle this problem by jointly leveraging the short-exposure noisy image and the long-exposure blurry image for better image restoration. Such setting is practically feasible due to that short-exposure and long- exposure images can be either acquired by two individual cameras synthesized by a long burst of images. Moreover, the short-cxposure images are hardly blurry, and ones have negligible noise. Their complementarity makes it fea- sible to learn restoration model self-supervised manner. Specifically, the noisy images can be used as the supervision information for deblurring, while the sharp areas in the blurry images can be utilized as the auxiliary supervision information for supervueee denoising. By learning in collaborative manner, the deblurring and denoising tasks in our method can benefit each other. Experiments on synthetic and wallw images show the effectiveness and practicality of the proposed method. Codes are available at https: thub. com/ cszhi 1u1998 (a1199///ellfT SR.
the
Introduction
is a common yet challenging task to acquire visually appealing photos with appropriate brightness under low light environment. Traditional ways increase the image brightness include enlarging the aperture, adopting a higher ISO, and reducing the shutter speed (i.e., lengthening exposure time). As for smartphone cameras with fixed aperture, brightness can only be adjusted by setting the sensor gain (i.e., ISO) and exposure time. Nonetheless, they are negatively correlated to maintain the appropriate brightness level of the image, i.e., the shorter-exposure image generally adopts higher ISO, while the longer-exposune image usually has lower ISO. Moreover, high ISO configuration introduces inevitable and complex noise due to the limited photon amount and the process of camera image signal processing (ISP) pipeline, while expoourr is prone to produce blurry images due to the camera shake and scene variations. Consequently, photographers have to make compromise between noise and blur.
Correspondence author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Recent advances in image restoration make it possible to further improve the visual quality of the acquired low light images by leveraging deep image denoising or deblurring networks. Taking supervised denoising as an example, synthetic or real-world noisy-clean image pairs are required to train the deep networks [9, 18,47-49,52, 533. However, the models trained with synthetic training pairs are hard to generalize to real noisy images, and the real-world clean reference images are usually obtained by averaging hundreds of noisy ones [I] or with complicated capturing and processing procedure [29], making the collection of large scale real-world training datasets laborious, expensive, and time-consuming. Such problems greatly limit the deployment of models on more devices with different noise distributions. Alternatively, surge of self-supervised image denoising methods [11, have been developed to avoid the collection of ground- (GT) training images, yet are limited in handling complex real-world image noise. Another possible solution is to perform motion deblurring on long-exposure images. Early explorations [8, 21, 1,,,,,4,1,, 3,3,,35,45 35, mainly given spatially uniform deblurring caused by camera motion. Recently, the proposal of non-uniform datasets (e.g.. GoPro [25], REDS [24], HIDE [36], RealBlur [32], etc.) has greatly boosted the research of deblurring in more practical scenes involving both camera shake and object
13, 15,11,7,,,,7,311
motion [6, 24,25,36 38, 51].
In this paper, we suggest to improve low light imaging by jointly leveraging the short-exposure noisy and long-exposure blurry images. First, such setting is practically feasible. For example, multiple cameras have been equipedd in modern smartphones, which can be designed to acquire short-exposure and of blurry and noisy images from a long burst of images captured by camera. Second, the noisy and blurry images convey complementary information, which is beneficial to improve restoration performance and makes self-supprvised image restoration (SelfIR) possible. We note that several methods [5, 14,23, 24,33,46) have been suggested to combine the blurry image with their noisy counterpart for better image restoration, yet it remains uninvestigated under the self-supervised regime.
images, respectively. Moreover, one can also synthesize a pair
We further present a SelfIR model with blurry and noisy pairs. Even though the blurry and noisy images are both disturbed, the short-exposure images taken with high ISO are hardly blurry, while the long-exposure images taken with low ISO are generally near noise-free. Thus, the long-exposure and short-exposure images can be used to provide some supervision information for each other. On the one hand, the noisy image can serve as an alternative of sharp image to supervise deblurring with negligible performance degradation. On the other hand, the static regions in long-exposure images forse-free and sharp, which in turn can provide auxiliary supervision information for image denoising. Taking these two aspects into account, we present a collaborative learning (co-learning) method termed SelfIR for deblurring and denoising, which is effective in leveraging the complementary information of long- and short-exposure images and can be learned in a self-supervised manner.
Extensive experiments on synthetic data are conducted to evaluate our SelfIR. Both quantitative and qualitative results show that SelfIR outperforms the state-of-the-an yelf-supervised denoising methods, as well as the supervised denoising and deblurring counterparts. To further verify the practicality of our SelfIR model, we have also collected set of 61 real-world blurry and noisy pairs using smartphones. Since there are no corresponding ground-truth images for calculating full-reference image quality assessment (IQA) metrics, we evaluate the restoration results using no-reference IQA metrics. The results show that our method also performs favorably against the competing methods.
To sum up, the main contributions of this work include:
We take step forward in leveraging blurry and noisy image pairs for image restoration. Going beyond leveraging their complementarity in improving restoration performance, we show that it can also be utilized for self- suse-sipeeeieed learning of the restoration model.
A self-supervised image restoration model (SelfIR) is proposed, where short-exposure images serve as supervision for the corresponding deblurring task, while the sharp regions in long-exposure images provide auxiliary supervision for self-supervised denoising.
Extensive experiments on both synthetic and real-world image pairs show that our SelfIR performs favorably against the state-of-the-art self-supervised denoising methods, as well as the baseline supervised deblurring and denoising methods.
2

---

2
Related Work
In this section, briefly review burst image denoising and deblurring, as well as self-supervised image denoising and deblurring methods. In addition, we recommend [7] for a comprehensive introduction the relevant mobile computational photography.
Burst Image Denoising and Deblurring, In comparison with single image, burst images can provide more information that is beneficial for image restoration. Hasinoff el al. [10] utilize an FFT-based alignment algorithm and a hybrid 2D/3D Wiener filter to denoise and merge a burst of underexposed frames for low-light photography. KPN [21] predicts spatially variant kernels for every burst noisy image to merge them. BPN [44] extends the KPN method with a basis prediction network and achieves larger denoising kernels under certain computing resource constraints. Aittala al. [2] take both noise and blur into account, and restore sharp and noise-free images from burst images in an inder-independent manner.
Self-Supervised Image Denoising and Deblurring. Recently, self-supervised learning has drawn upsurging attention in low-level vision. DIP [39] utilizes the image prior implicitly captured by the network structure to repair corrupted images. SelfDeblur [30] respectively models the deep priors of clear image and blur kernel for self-supervised deblurring. For these methods, the networks are required to re-train from scratch for each test image, which is less efficient, especially for mobile edge devices. Noise2Noise [17] demonstrates that noisy pairs with mutually independent noise can be used to train denoising network, opening the door to self-supervised denoising. Neighbor2Neighbor [11] utilizes a random neighbor sub-sampler to generate the training pairs from noisy images themselves. In addition, some works [13, 15, 16, 41, 43] elaborately design blind- spot networks to avoid learning the identity mapping for self-supervised denoising. However, the self-supervised denoising methods are limited in handling complex image noise. In this work, we utilize the complementarity of long-exposure blurry and short-exposure noisy images for better self-supervised image restoration.
3 Proposed Method
In this section, we first show the feasibility of taking noisy images as the supervision of deblurring. Then, introduce the sharp area detection method in long-exp mingexxpsure images and auxiliary loss for self-supervised denoising. Finally, we present the proposed co-learning framework SelfIR.
3.1 Deblurring with Noisy Image
When taking long-exposure photos, the shake of the camera and the motion of objects usually lead to a blurry image Is, which can formulated by,
Is K(I) NB.
where is the latent clear image, IC denotes the blur process with non-unifomm kernels, Ng represents the low-intensity noise. Existing supervised deblurring methods generally utilize a network (denoted by D&) to estimate I from Ig. For training the parameters of Ds, which is denoted by Os, the optimization objective can be defined by,
deep neural
O arg min E [C ())
0g
where L denotes the loss functions for supervised learning. However, collecting clear images is troublesome in real-world scenes. Inspired by Noise2Noise [17], we show that the noisy short- exposure image can be substitution of the latent clear image to supervise the task of deblurring.
When taking short-exposure photos under low light environment, the limited photon amount and inherent defects of camera ISP make the images noisy (denoted by Iy), which can be formulated by,
IN I+ -INNs
(3)
where Ny represents the noise (with much higher intensity than N₂). When using the noisy image In as the supervision of deblurring, the optimization of Og can be expressed as,
O arg min Big Ay [C (Da(I;; θs), Lw) arg min Big [Etyitty w [C (Da(Ie; Os), Lv)]] θg
(4)

---

Suppose that the loss function L in Eqn. (4) is the & loss, we have,
Elwlle (Dx (B); [w]] = ElvlIe [1 [I: 08) IN
EI,Nw [D 0g) (I Nx
Emjsa
[|| 0g 1
(5)
2EI,NwiT [
(I:: 08) -I) Nv]
EN
[|
where EN Ie assume that Ny is zero-mean, NN and are independent, we can get,
[ can be regarded as constant and be safely discarded from Eqn. (5). Further,
Nx [(D (B) 1) NN) =
In this case, the optimal solution Øg in Eqn. (4) and that in Eqn. (2) are the same. Thus, it is feasible to utilize noisy short-exposure images instead of clear ones the supervision of deblurring.
3.2 Denoising with Long-Expostt Image
Self-supervised denoising makes possible to re- move the noise without clean image supervision, but may give rise obvious performance degrada- tion, especially when handling complex real-world noises. Here we propose alleviate this prob- lem by introducing the long-exposure counterpart. Obviously, taking the whole long-exposure image supervision will bring adverse effects, making the results to blurry. Nonetheless, it is worth noting that, the blur process K. Eqn. (1) is generally non-uniform, and sometimes not all areas are blurry, As shown in Fig. (d)), there exist some approximately sharp regions in the long-exposure image, which can pro- vide partial supervision information that benefits self-supe denoising.
extra supervision from
(+) Blurrylmage
( Blurryamao
Blunyy areas with ringing artiacts
(4) Approdimatdl sharp areas
Figure 1: A blurry image example where the blur is non-uniform in (a). It includes some common blurry areas (b), severe blurry areas with ringing artifacts (c), and some approxi- mately sharp areas (d).
Therefore, it is crucial to pinpoint the sharp areas in the long-exposure image. Otherwise, fer to go without sharp areas than accept a shoddy option, misjudgments of sharp areas will lead to worse denoising results. However, without any discriminative clues, it is very likely to misjudge the sharp areas. Considering that Iv nearly non-blurry due the short exposure time, it may be a reference for sharp area detection. To avoid noise interference, we pre-process Iy with self- supervised denoising model, and take the result IN to help detect sharp regions in the corresponding blurry image Ig.
would pre-
Specifically, first divide Ig and In into N non-overlapping patches. For each patch pair I and IN < S N). our goal is to obtain mask m " e {0, 10 that indicates whether I is sharp patch. Since Iv is nearly non-blurry, when some severe motion blurs exist Ig the difference between Ig and 1 in textures and edges should be evident. Taking the above into account, we adopt a similarity metric to detect the areas with severe motion blurs, i.e.,
m agnnmaa((0) s (Lg. -x)))
(7)
where structural similarity (SSIM) [40] is utilized for the similarity metric 8, e, denotes the threshold, while max (o,bb b) and sgn(-) denote the maximum and sign function, respectively.
However, the initial denoising result iN may be over-smooth, in other words, Eqn. (7) may fail when facing some mildly blurred regions in 1g. Therefore. further measure the difference in variance between Ig and ig. When the variance Ig greater than that of in. we consider that Ig is potential to be sharp patch. It should be noted that the difference in variance is not suitable for

---

Cuax Eqn. (9)
Blurry image
Noisy image
91
(1p)
Restoration
network
Blurry image
21(-)
Output
91 (Ix)
Live & Ligg
Equs. (10) (1))
Restoration network D
Noisy image (1,x)
92 (L,))
Training
Sub-sampling operation (+)
66
Neighboring Sub-Images
Sub-sampling operation (ta(-)
Output
(b) Example of neighbor sub-sampler
Testing
Figure 2: Overview of our proposed SelfIR framework. (a) Training phase of SelfIR. Sub-samples blurry image 91 (IB) and noisy image 91 (Iw) are taken as the inputs. 92 is used for calculating the reconstruction loss Lree (see Eqn. (10)) and regularization loss Lreg (see Eqn. (I1)), while 91 (Ig) taken for calculating auxiliary loss (see Eqn. (9)). (b) Example of neighbor sub-sampler. In each 2 x 2 cell, two pixels randomly selected for respectively composing the neighboring sub-images. (c) Testing phase SelfIR. The blurry and noisy images can be directly taken for restoration.
detecting some severely blurry areas with ringing artifacts (see Fig. I(c)), In such blurry regions, the variance of IB may also be greater than that of Ig. When synthesizing blurry images, some works 52,542,56] remove the artifacts by interpolating the frames before averaging the sharp images. However, the artifacts also exist in real-world blurry images, especially in areas with flickering lights. Therefore, we still take the ringing artifacts into consideration in this work.
As result, we jointly use the SSIM and variance measure for judging sharp regions, m" can be formulated as,
m sgn(max((0.)mmm s (Ig. [x) - cs)) sgn(max(0,, var(Ig) var var(ī cx)), (8)
where var(.) and en denote the variance function and the threshold, respectively. and ey are set to 0.99 and le-5 when color intensity values of I and In are normalized to [0, 1]. Defining the final output image I, the auxiliary loss function for denoising can be denoted as,
Lowa (î, (i.
m
I
(9)
3.3 Co-Learning of Deblurring and Denoising
As illustrated in Secs. 3.1 and 3.2, the noisy images can provide effective supervision information for deblurring, while the blurry images can also provide auxiliary supervision for self-supervised denoising. In other words, we learn deblur denoise without additional clear and clean ground-truths. However, Iearning them independently will lead to limited performance. As some works [5, 14,23,444 46] have suggested, better restoration performance can be obtained by incorporating blurry with noisy images. Thus, it should be further considered to achieve self-superrised image restoration that takes two disturbed images input. In this section, delicately design the self- supervised model SelfIR that learns from both deblurring and denoising tasks collaborative learning manner.
For training SelfIR, required to avoid the trivial solution when taking both blurry and noisy images the input. For example, when taking noisy images for supervising deblurring, the model
5

---

may simply output the noisy images instead of learning the deburring results. Fortunately, Neigh- bor2Neighbor [11] shows the noises in two sub-sampled images (i.e.. 91 (Iv) and g2(IN), see Fig. 2(b)) from noisy image IN are almost independent. Therefore, they can be respectively taken as the input and target for training the denoising subtask in a self-supervised manner. Simultaneously, combining the derivation in Eqn. (5), 91 (Ig) and ga (Tv) can be respectively taken as the input and target to train the deblurring subtask. Therefore, our SelfRR can take a step forward and deliver both (Ig) and gi (Iv) (i.e., {yı (IB, 91 (Iv))) }} into the restoration network D, as shown in Fig. 2(a). The sub-sampled image 92 (2(Ix) is taken for calculating reconstruction loss Crees which can be written as,
re = | (TB), 91 (Iv)) - 92(Tw)]
(10)
We also calculate regularization loss Creg similar to [1]].
Creg (g (Is), (Iw)) 22 (Iw) (or (D(Is, Iw)) 92 (D((. 1))
(11)
where D has same parameters with D but has no gradient for back-propagattnn Moreover, the sharp areas in In (Ig) can provide auxiliary supervision for the model learning and the performance can be further improved. Specifically, we calculate auxiliary loss Lau (D(g) (Is). gi (Iw)), 91 (Ig)) in Eqn. (9). For obtaining the mask m" we replace Ig and In in Eqn. (8) with Φi (IB) and D(gi (Is), 9n (Iw)), respectively. Finally, the learning objective of parameters Op with the co- learning manner can be formulated as,
0 = arg min Erg Ix (Cree + >reg + Maus))
(12)
O
where Arag and Naur are
for balancing the loss terms.
Experiments
In this section, first describe the dataset configuration and the training/ evaluation protocols in detail. Then, both the quantitative and qualitative results on synthetic and real-world images are given for comprehensive evaluation.
4.1 Implementation Details
For synthesizing blurry images, early methods [23,46 convolve sharp images with simulated blur kernels, which is quite different from the real-world blur model. Recently, the GoPro dataset² [25] offers a more realistic way to synthesize blurry images, which has been widely adopted for motion deblurring tasks 38, exists in the sharp video frames that are captured by a high-speed camera (i.e., GoPro Hero4 Black), it has little effect model training and evaluation. We directly regard the sharp images clean images. The blurry image is generated by averaging consecutive sharp frames, which can better simulate both camera shake and object motion. Thus, can get the blurry-clear pairs as Ig and respectively. For synthesizing the noisy image Iws we consider three noise distributions: 1) Gaussian noise with a [5/255, 50/ /255), 2) Poisson noise with E [5, 50), and 3) sensor noise [4]. Please refer to the supplementary material for the noise formulations. Experiments are conducted in both sRGB and raw-RGB spaces. For the sRGB space, we add Gaussian or Poisson noise the sRGB sharp images. While for the raw-RGB space, the more complex sensor noise is added to the sharp raw-RGB images, and the blurry images are also converted back the raw-RGB space through the unprocessing [4] pipeline. Finally, there are 2,103 image pairs for training, and we use the remaining 1,111 pairs for testing.
Synthetic Datasets.
In the dataset, although slight inherent noise
Real-world Datasets. Furthermore, we have also collected 61 real-world blurry and noisy raw-RGB pairs with Huawei P40 smartphones in the professional camera mode. In particular, the blurry images are captured with a low ISO and long exposure time, while the corresponding noisy images are captured with a high ISO and short exposure time in the same scene. Please refer the supplementary material for detailed collection process. Due to the limited amount of real-world images, we use 30 pairs fine-tune the model pre-trained on raw-RGB images with synthetic sensor noise, and take the remaining 31 pairs for evaluation.
Phttps: http///suugnnunaaa github. wo//at//aeeee www.aaaatttaaaeeeaaa gopro htmll

---

Table Quantitative results on synthetic sRGB images.
Gaussian [5/255, 50/255) PSNR+/ SSIM LPIPS1
Poisson E 50] PSNR SSIM LPIPS.
Method
Supervised Deblurring
Baselineg
28.24 / 0.8561 00119
DeepDeblur [2:5]
30.04 / 0.9015 0.133
Supervised
Baselinex
34.91 / 0,9360 0.098
33.15/ 0.126 32.45 / 0.9084 / 0.128
Denoising
DnCNN [52]
34.63 / 0.9308 /0.
Supervised IR
BaselineR
36.15 0.9534 0.070
34.74 / 0..4454 0.084
N2N [17]
34.88 0,9354 0.100
33.09 / 0.9216 0,129
N2V [13]
33.09 / 0,9180 0.111
31.81 / 00.999 0.137
19-mu [15]
33.61 / 0..927 104
32.29 / 00.9091 0.131
Sele-Supervised
Laine19-pmss 19- [15]
34.76 / 0,9322 0.086
32.77 / 0.9147 O. 16
Denoising
DBSN [43]
33.72 0.9224 0.111
31.46 0.8883 0.14
R2R [27]
33.74/ / 0,9223 000
30.05 / 0.7649 0,230
Neighbor2Neighhor [11]
34.29 0,9271 0.005
32.68 / 0,99100////011 0.111
Blind2Unblind [41]
34.69 / 0.9353 0.007
33.09 / 0.9216 / 0.132
Ours
SelfIR
35.74 / 0,9499 /.076
34.27 / 0.9404 0.092
Training Details. Following [11, 15, 17], we adopt a U-Net [33] architecture our restoration network, where two encoders are respectively deployed to the blurry and noisy input images for better domain-specific feature extraction, and then blurry and noisy features are fused in the decoder part. Detailed architecture is given in the supplementary material. During training, the batch size to 16 and the patch size 128 x 128. Adam optimizer [12] with B1 0.9 and 32 0.999 used to train the network for 200 epochs. The learning rate is initially set to 3 x 10 for synthetic experiments and x 10-4 for real-world experiments. And it reduces by half every 50 epochs. For the srper-parameters in Eqn. (12), Aaur is set to 2, Xreg is set to 2 and 4 for experiments in sRGB space and raw-RGB space, respectively. All experiments are conducted with Py [28] on an Nvidia GeForce RTX 2080Ti GPU.
Evaluation Configurations. We convert all raw-RGB results to sRGB space through post-processing pipeline, and the quantitative metrics are computed in the sRGB space. For the results of synthetic experiments, we take peak signal noise ratio (PSNR), structural similarity (SSIM) [40] and learned perceptual image patch similarity (LPIPS) [54] as evaluation metrics. For the results of real-world experiments, due the lack of ground-truth images, we utilize no-reference IQA metrics (i.e.. NIQE [22], NRQM [19], and PI [3]) to evaluate the generated images.
4.2 Experimental Results in sRGB Space
To assess our proposed SelfIR, we build several baseline methods for comparison, including 1) supervised deblurring baseline (denoted by Baselineg), supervised denoising baseline (denoted by Baseline ). 3) a supervised image restoration baseline which takes both blurry and noisy images as input (denoted by Baselineg),, and 4) Neighbor2Neighhor [11] since the denoising part of our co-learning framework is based on it. Note that the network structure is not the focus of this paper, so we deploy a simple U-Net [33] architecture for our restoration network and the first three baseline methods. Besides, we choose two classical supervised methods (i.e., DeepDeblur [25] and DrCNN [52]) for comparison. Self-supervised denoising methods (e.g.. N2N [17], N2V [13], L9ine [15]. DBSN [43], R2R [27]. and Blind2Unblind [41]) are also compared. For DeepDe- blur [25], we use the officially released model for testing. For other methods, the models are retrained with our synthetic images for fair comparison.
The quantitative results of synthetic experiments on Gaussian and Poisson noise are given in Tab. I. One can see that our SelfIR outperforms all other methods except the supervised IR model (Baselinex), which can seen as the upper limit of the restoration performance with the utilized U-Net architecture and the blurry-noisy pair input. In comparison to the self-supervised denoising baseline Neighbor2Neighhor [1]], 1.45 dB and 1.59 dB PSNR gains are obtained by our SelfIR on Gaussian and Poisson noise, respectively. Even compared to the supervised denoising baseline
7

---

ISTANBLA BATUM SANAT
ISTANBLIL BATUM SANAT
ISTANDUL DATUM SANAT
ISTANBUL DATUM
ISTANBUL DATUM
iSTANDUL DATUM
SANAT
SANAT
SANAT
Noisy Image
Baselinegr
N2V [13]
Laine19-pme [15]
DBSN [43]
R2R [27]
iSTANBUL BATUM SANAT
iSTANBUL
ISTANBUL
ISTANBUL
|STANBUL
iSTANBUL BATUM
BATUM SANAT
BATUM SANAT
DATUM
BATUM SANAT
SANAT
SANAT
Bluryy Image
Baseliness
NeighborrNeiibbbr [11]
Blind2Unblind
SelfIR (Ours)
Baselinex
Figure 3: Visual comparison on Gaussian noise. The texts in our result are sharper and clearer. The proposed SelfIR is almost comparable to supervised BaselineR, which can seen the upper limit of the restoration performance with the utilized U-Net architecture and the blurry-noisy pair input.
Noisy Image
Baseinee
N2V [13]
Laine Sainel9-pmoo |151
DBSN [43]
R2R [27]
Blurry Image
Baselineg
NeighborNeiighbor [11] BlindUUnbland [41]
SelfIR (Ouns)
Baselines:
Figure 4:
Visual comparison on Poisson noise. terms of the visual result, SelfTR combines the advantages of supervised denoising Bascliney and deblurring Baselineg.
model (Baseliney),, our method also improves the PSNR index by 0.83 dB and 1.12 dB. Due the severely ill- ill-posed nature of deblurring, SelfIR brings a PSNR gain by more than 6 dB comparing to the supervised deblurring baseline model (Baselineg)). The results clearly show that the blurry images can indeed provide beneficial features for denoising, and so do the noisy images for deblurring.
The qualitative results are shown in Figs. 3 and 4. One can see that our results are sharper and clearer than other denoising or deblurring methods. terms of visual quality, the proposed SeliR is almost comparable to the supervised image restoration model (Baseliner), and combines the ad advantages of supervised denoising Baseline and deblurring Baselineg. More results can be found in the supplemental material.
4.3 Experimental Results in Raw-RGB Space
Tab. shows the quantitative results of synthetic and real-world experiments on raw-RGB images. For synthetic experiments with more complex and realistic sensor noise [4], all models are retrained with our synthetic images. Our SelfIR achieves 1.69 dB PSNR gain in comparison with Neigh- bor2Neighbor [11]. For evaluation on real-world images, we directly use the models of N2N [17], Laine 1aine19-pme 19-pme [15], R2R [27] and all supervised methods, which are are-trained for sensor noise. For SelfIR and other self-supervised methods, we fine-tune the pre-trained model on our real-world training set. The PI [3] metric of SelfIR improved by 0.55 on real-world testing images through fine-tuning. And the results on no-reference IQA show that our method is very competitive in comparison with the competing methods on real-world images. The visual results are given in the supplemental material.

---

Table 2: Quantitative results on synthetic and raw-RGB images.
Sensor Noise [4]
Real-World Images
Method
PSNR1 SSIM; LPIPS)
NIQE,/ NRQM, Pl;
Supervised Deblurring
Baselineg
28.14 0,8547 /0.162
6.26/ 5,04 5.62 6.76 4.78 6.00
DeepDeblur [25]
29.75 / 0.8881 /0.115
Supervised Denoising
Baselinew
34.52 / 0,9461 /.005
5.69 4.85 5.43
DnCNN [52]
33.81 / 0.9325 0.076
6.05 5.10 5.48
Supervised IR
Baseliner
36.10/ 0.9574 0.035
5.14/5.1455.1
N2N [17]
34.67 / 0.9472/ 0.053
6.10/4.93 / 5.59
N2V [13]
31.39 / 0.9227 0.076
5.82 5.52 5.17
19ine19-mu [15]
32.74 / 0,9304 0.073
5.87 5.67 5.10
Sell-Supervised
Laine19-pme [15]
33.28 0.9119 0.095
7.26 6.03 5.62
Denoising
DBSN [43]
33.59 / 0,9389 0.066
/ 5,48 5.54
R2R [27]
32.21 / 0.8807 0.117
/ 4.99
Neighbor2Neighbor [1]]
32.82 0.9275 0.087
6,47 5.86 5.33
Blind2Unblind [41]
33.30/ / 0.9380 0.061
5.28 5.22 /
SelfTR
34.51 /.9440 0.055
5.48 5.83
Table Results of deblurring with clear images and noisy images as the supervision.
Supervision
Gaussian a E [5/255, 50/255)
Poisson À E [5, 50)
Sensor Noise [4]
Information
PSNRt / SSIM, LLPIPS,
PSNR1 SSIM / LPIPS;
PSNR+/ SSIM LPIPS)
Clear Images
28.24 / 0.8561 0.19
28.24 / 0.8561 / 091
28. 14 0.8547 /0.162 28.16 0.8545 0.164
Noisy Images
28.29 / 0.8578/ 0.190
28.23 / 0.8563 0.1
Table 4: Ablation study of auxiliary loss on Gaussian noise.
Neighbor2Neighbor []]] PSNRt/ SSIM// LPIPS.
SelfIR
PSNR / SSIM LPIPS1
w/o Lauz
34.29 0.9271 0.085
35.65 / 0.9492 0.080
Laux
34.45/0.955 0.9307 0.093
$5.74/0.9499 0.076
5
Ablation Study
5.1 Feasibility of Deblurring with Noisy Images
In order to verify the feasibility of taking noisy images the supervision of deblurring in Sec. 3.1, we replace the clear supervision the results shown in Tab. 3. It can be that taking noisy clear images the supervision leads to similar performance on the deblurring task, which confirms that the noisy images can be an alternative of clear images supervise the deblurring task.
Baselineg with noisy images of different noise distributions, and
5.2 Effect of Auxiliary Loss
In order evaluate the effect of auxiliary loss Laux in Eqn. (9), we add Caux to the loss terms of self- supervised demoising method Neighbor2Neighbor [11]. As shown in Tab. 4, we obtain 0.16 dB PSNR gain against the baseline Neighbor2Neighbor [11]. The result indicates that blurry images can provide some auxiliary supervision information and improve performance for self-supervised denoising. When removing Lauz from our SelfIR, the PSNR dropped by 0.09 dB. Since the long-exposure image is taken directly input, the loss term may be less effective, but is still beneficial.

---

Table 5: Ablation study on different weights
Table 6: Ablation study on different weights (Naur values) of auxiliary loss.
(Dreg values) of regularization loss.
Areg
PSNR SSIM+ LPIPS)
Agur
PSNR / SSIM LPIPS)
35.20 00.9433 00.997
35.65 0.9492 0.080
1
35.64 00,9492 /.0082
35.73 0,9498 0.0078
2
35.74 / 0.9499 0.076
35.74 / 0.9499 0.007
35.72 / 0.9496 0.075
35.73 0.9499 0.076
35.73 0.9497 0.072
35.67 0.9496 0.077
5.3 Effect of Different Loss Weights
We conduct ablation studies on different weighting ppper-paraeeters (i.e., Xreg and Areg) for bal- ancing regularization and auxiliary loss terms. The experiments are conducted on Gaussian noise in sRGB space. When varying one hyper-parameter, the other one is set to 2 by default. As shown in Tabs. 5 and 6, it can be seen that the sensitivity Xreg and Areg of SelfIR is acceptable.
6 Conclusion
The complementarity between long-expen oongexxpouure blurty and short-exposure noisy images not only improves the performance image restoration, but also makes it possible to learn a restoration model in self-supervised manner. Jointly leveraging the long- and short-exposure images, we present self-supervised image restoration method named SelfIR. On the one hand, we take the short-exposure images as the supervision information for deblurring. On the other hand, we utilize the sharp areas in the Long-exposure images as auxiliary supervision information in aid of self- seff-supervised denoising. SelfIR combines these two aspects by a collaborative learning scheme, and makes the deblurring and denoising tasks benefit from each other. Experiments on synthetic and real-world images show the effectiveness and practicality of our proposed method.
7 Limitation, Impact, Etc
This work still limited in assessing the results on real-world image pairs. Although no-reference IQA metrics adopted, these may be too unstable report the performance consistent with humans accurately. We will establish a real-world blurry-noisy pair dataset including high-quality GTs to solve this problem in the future work. As for societal influence, this work is promising to be applied to terminal devices (e.g., smartphones) for obtaining better images under low-light environments. It has no foresecable negative impact. Besides, the images used in this work are from natural human scenes. The GoPro dataset used for synthetic experiments is public under CC BY 4.0 license. There no personally identifiable information or offensive content the experimental data.
Acknowledgement
This work supported by the Major Key Project of Peng Cheng Laboratory (PCL2021A12) and the National Natural Science Foundation of China (NSFC) under Grants No.s U19A2073 and 61871381.
10

---

References
[1] AbdeIrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In CVPR, 2018.
[2] Miika Aittala and Frédo Durand, Burst image deblurring using permutation invariant convolutional neural networks. In ECCV, 2018.
[3] Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. The 2018 pirm challenge on perceptual image super-resolution. In ECCV Workshops, 2018.
[4] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron, Unpro- cessing images for learned raw denoising. CVPR, 2019.
[5] Meng Chang, Huajun Feng, Zhihai Xu, and Qi Li. Low-light image restoration with short-and long- exposure raw pairs. IEEE TMM, 2021.
[6] Sung-Jin Cho, Seo-Won Ji. Jun-Pyo Hong, Seung- Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fine approach in single image deblurring. In /CCV, 2021.
[7] Mauricio Delbracio, Damien Kelly, Michael S Brown, and Peyman Milanfar. Mobile computational photography: A tour. Annual Review of Vision Science, 2021.
[8] Rob Fergus, Barun Singh, Aaron Hertzmann, Sam Roweis, and William T Freeman. Removing camera shake from a single photograph. In ACM SIGGRAPH. 2006.
[9] Shi Guo, Zifei Yan, Kai Zhang. Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In CVPR, 2019.
[10] Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Mare Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM ToG, 2016.
[11] Tao Huang. Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neighhor: Self-supervised denoising from single noisy images. In CVPR, 2021.
[12] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[13] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-learning denoising from single noisy images. In CVPR, 2019.
[14] Green Rosh KS, Nikhil Krishnan, BH Pawan Prasad, and Sachin Lomte. Content preserving scale space network for fast image restoration from noisy-blurry pairs. In ICASSP, 2022.
[15] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. In NeurlPS, 2019.
[16] Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and Spon network. In CVPR, 2022.
[17] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In ICML, 2018.
[18] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet-cnn for image restoration. In CVPR, 2018
[19] Chao Ma, Chih- Yuan Yang, Xiaokang Yang. and Ming-Hsur Hsuan Yang. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding, 2017. [20] Tomer Michaeli and Michal Irani. Blind deblurring using internal patch recurrence. In ECCV, 2014.
[21] Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlct, Ren Ng, and Robert Carroll. Burst denoising with kemel prediction networks. In CVPR, 2018.
[22] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 2012.
[23] Janne Mustaniemi, Juho Kannala, Jiri Matas, Simo Särkkä, and Janne Heikkilä. Lsd2-joint denoising and deblurring of short and long exposure images with cnns. In BMVC, 2020.
[24] Seungjun Nah, Sungyong Baik. Seokil Hong. Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In CVPR Workshops, 2019.
[25] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
[26] Reyhaneh Neshatavar, Mohsen Yavartanoo, Sanghyun Son, and Kyoung Mu Lee. Cvf-sid: Cyclic multi- variate function for self-supervised image denoising by disentangling noise from image. In CVPR, 2022. Unsupervised deep
[27] Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji. Recorrpped--to-ecoorpppp learning for image denoising. In CVPR, 2021.
[28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NS 2019.
[29] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In CVPR, 2017.
11

---

[30]
Dongwei Ren, Kai Zhang. Qilong Wang, Qinghua Hu, and Wangmeng Zuo. Neural blind deconvolution using deep priors. In CVPR, 2020.
[31] Wenqi Ren, Jiawei Zhang, Lin Ma, Jinshan Pan, Xiaochun Cao, Wangmeng Zuo, Wei Liu, and Ming- Yang. Deep non-blind deconvolution via generalized low-rank approximation. In NewrIPS, 2018
[32] Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. weal-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV, 2020.
[33] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015,
[34] Christian Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf. Learning to deblur. IEEE
TPAMI, 2015.
[35] Qi Shan, Jiaya Jia, and Aseem Agarwala. High-quality motion dehlurring from a single image. ACM TOG,
2008.
[36] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haihin Ling, Tingfa Xu, and Ling Shao. Human- aware motion deblurring. In ICCV, 2019.
[37] Shakarim Soltanayev and Se Young Chun. Training deep learning based denoisers without ground truth data, In NeurlPS, 2018.
[38] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In CVPR, 2018.
[39] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In CVPR, 2018.
[40] Zhou Wang. Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
[41] Zejin Wang. Jiazheng Liu, Guoqing Li, and Hua Han. Blind2unblind: Self-supervised image denoising with visible blind spots. In CVPR, 2022.
[42] Patrick Wieschollek, Michael Hirsch, Bernhard Scholkopf, and Hendrik Lensch. Learning blind motion deblurring. In /CCV, 2017.
[43] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In ECCV, 2020.
[44] Zhihao Xia, Federico Perazzi, Michaël Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti. Basis prediction networks for effective burst denoising with large kernels. In CVPR, 2020.
[45] Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural 10 sparse representation for natural image deblurring. In CVPR, 2013.
[46] Lu Yuan, Jian Sun, Long Quan, and Heung- Yeung Shum. Image deblurring with blurred/noisy image pairs.
In SIGGRAPH, 2007.
[47] Zongsheng Yue, Qian Zhao, Lci Zhang, and Deyu Meng. Dual adversarial network: Toward real-world
noise removal and noise generation. In ECCV, 2020.
[48] Syed Waqas Zamir, Aditya Arora. Salman Khan. Munawar Hayat, Fahad Shahbaz Khan, and Ming-ISsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
[49] Syed Wagas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Ilsun Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. ECCV, 2020.
[50] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- H-san Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
[5I] Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming-Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In CVPR, 2018.
[52] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE TIP, 2017.
[53] Kai Zhang. Wangmeng Zuo, and Lei Zhang. Fidnet: Toward a fast and flexible solution for enn-based image denoising. JEEE TIP, 2018.
[54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018
[55] Yi Zhang, Dasong Li, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Idr: Self- supervised image denoising via iterative data refinement. In CVPR, 2022.
[56] Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo Haozhe Xie, Jinshan Pan, and Jimmy S Ren. Davanet: Stereo deblurring with view aggregation. In CVPR, 2019.
[57] Magauiya Zhussip, Shakarim Soltanayev, and Se Young Chun. Extending stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy images. In NewrlPS, 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Sec. 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see Sec. 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [No] The code will be available when the paper is published. The real-world data we collect proprietary,
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Sec. 4.1.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] It is too computationally expensive, and our computing resources are limited.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please see Sec. 4.1.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Please see Sec. 7.
(c) Did you include any assets either in the supplemental material URL? [No] The data we collect is proprietary.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] Please see Sec. 7.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Please see Sec. 7.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "oTf5v00B7F2", "reviewer": "Reviewer_njZ6", "summary": "This paper considers the problem of taking high quality clean images given the difficulty of the tradeoff between obtaining blurry images due to long exposure times, and noisy images from small exposure time. A typical way to handle the tradeoff is to find a compromise between the two, then use a debluring or denoising model to fix up the image (the models trained using supervision or self-supervision). Instead this paper proposes to make use of both a blurry and a noisy image, taking advantage of the fact that the blurry image will have very little noise, and the noisy model very little blur in order to co-train *both* a debluring and a denoising model. Using these two models the authors Provide a method for producing a single clean image given a blurry and a noisy image. \n", "questions": "See previous section.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe main contribution of this work seems to be the idea that noisy and blurry images contain complimentary information, that, if both images are collected, can be used to train both a debarring and a denoising model. Using both of theses models to restore images then leads to good clean images. This message seems clear and intuitive, and the method a useful contribution. Too I am happy to lean towards acceptance. But there is an important major caveat. It. Is non-trivial, even infeasible to take both a long- and short-expoosure image of the same scene. The authors solution is to use two cameras side by side taking photos simultaneously. So I am quite worried about the general availability of the type of data required to use this method. \n\n---\n\nTo summarize some points:\n- The paper is clearly written and highly accessible.\n- The idea of using the complimentary information in noisy vs blurry images is natural and intuitive\n- Strong quantitative and qualitative results: the proposed method seems to improve over existing  supervised & self-supervised denoising methods (& more). \n\n---\n\nSome questions:\n- The real-world dataset only contains 61 images, only 30 of which are used for training. This is clearly a really small dataset. I would really like to hear from the authors on their thoughts about this data issue, and how it affects the viability of the joint-training method. \n- Could other intermediate exposure time images also be incorporated? So instead of just having very short and relatively long exposures, also include medium length exposures.\n- The overall loss includes two weighting hypermarameters (Eqn 12). How sensitive is the method to the choice of these hyper parameters?\n- Recent works in diffusion modeling have found that when denying a sample it is better to parameterize the denoising model to *rpedict the noise* instead of directly predicting the clean sample. Did you at all look into this? It could either 1) improve the current method, or 2) provide interesting counter-evidence to the generality of the observation from diffusion modeling.\n\n---\n\nA disclaimer: I do not keep up to date with the recent literature in image restoration, so potentially will not be aware of related work.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jtGKw-IbBRq", "reviewer": "Reviewer_X2qR", "summary": "This paper proposed a self-supervised image restoration method named ``SelfIR''. The basic idea is to work with a pair of short-exposure noisy and long-exposure blurry images. The complementary nature of blurry and noisy pairs make it possible to jointly utilize the auxiliary supervision information for self-supervised denoising. The reported experimental results are promising and convincing. SelfIR can achieve better performance than competing self-supervised denoising methods on synthetic datasets and comparable performance to N2N (ref. [15]) on real-world datasets.", "questions": "1. Is the proposed SelfIR \"collaborative learning\" or \"self-supervised learning\" or both (Line 170-171)? It seems to me Eq. (9) is collaborative learning since it involves the blurry-noisy image pair; while Eqs. (10)-(11) can be interpreted as self-supervised learning since it uses the pseudo-GT (output of restoration network D). Please correct me if I am wrong.  \n2. How is the sharp patch detected in Sec. 3.2 used by co-learning in Sec. 3.3? \n3. Is there any risk of inconsistency between the patch-dividing strategy in Sec. 3.2 and sub-sampling operation in Sec. 3.3?\n4. Do you assume the blur process with non-uniform kernels K is known in Eq. (1)? If Yes, this seems a strong assumption because nonuniform kernel estimation from real-world blurry images is a long-standing open problem.\n5. Do you have visual quality comparison for denoising real-world raw sRGB images? Figs. 3-4 are for synthetic images only where supervised learning (baseline) is easy to obtain.  ", "limitations": "When does the proposed SelfIR fail? For example, Table 2 shows that the proposed method falls behind N2N [15] on PSNR and Blind2Unblind [37] on NIQE. Any discussion or explanation about the limitation of SelfIR? ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The idea of exploiting complementary nature of blurry and noisy image pairs is sensible. There is some novelty with the developed co-learning of deblurring and denoising in Sec. 3.3\n+ The reported experimental results are mostly convincing and promising. Improved denoising results over Ref. [9] (CVPR'2021) and Ref. [37] (CVPR'2022) have been reported on synthetic sRGB images.\n\nWeaknesses:\n- Originality. I have found more clarification is needed for Sec. 3.1 and Sec. 3.3. How is Sec. 3.1 different from Noise2Nosie [15] and how is Sec. 3.3 different from Neighbor2Neighbor [9]? Many technical details such as Eqs. (4)-(6) and Fig. 2(b) have strong similarity with previous works. Even though the authors are inspired by those existing works, they need to highlight what are the new insights brought by this paper. I am afraid that the novelty of these two subsections is limited in the current version.\n- Quality. I have two major concerns over Sec. 3.2 (denoising with long-exposure image). First, I have found that the strategy of dividing an image into non-overlapping patches ad-hoc. Even though \"it is crucial to pinpoint the sharp areas in the long-exposure images\", I don't understand why the authors take a patch-based (instead of a pixel-based) approach. Second, the use of SSIM in Eqs. (7)-(8) lacks substantial justification. Maybe it is based on heuristics but the quality of presentation for this paper is lacking, in my biased opinion.  \n- Clarity. I wish authors can clarify more about the self-supervised learning procedure in Sec. 3.3. The discussion about \"complementarity\" (line 169) is vague because authors did not explicitly discuss how the complementary nature was exploited by the design of co-learning method.\n- Significance. The importance of this work seems to be to show the proposed SelfIR works better on real-world situations but the majority of the experimental results (especially visual comparisons) are for synthetic images only.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "idCelgilBbs", "reviewer": "Reviewer_hPxE", "summary": "This paper proposes a self-supervised technique for image restoration from a blurry and a noisy image pair. By observing that each task can be trained from the other low-quality image, the authors propose a method that brings sharpness from the noisy image and the cleanliness from the blurry image. \nHowever, I’m not sure if this method is a true self-supervised method. The training objective seems to be recovering the real noisy image from the blurry and (synthetic) augmented noisy image pair.\n", "questions": "L158 The ghost artifact has been pointed as imperfect blurry image in the following paper:\n\nS. Nah et al., “NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study,” CVPRW 2019\n\nThey interpolate the frames before averaging the sharp (& noisy) images to remove the ghost artifact.\nFollowing similar approaches may benefit finding sharp regions from equation (8).\n", "limitations": "Limitations described in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nWhile there have been attempts to use the blurry and the noisy images to achieve better-quality images, this method is novel in terms of self-supervised design. The self-supervised learning process builds on top of the observations in the nature of deblurring and denoising. The experimental results look promising given my concerns are resolved. \n\n[Weaknesses]\n\n- Related Works\n\nI think the authors should also refer to previous burst image denoising and burst image deblurring works.\nThe authors mainly describe self-supervised image denoising papers only. (+ 1 self-supervised deblurring paper)\n\n- L196 \n\nFrom the definition in the introduction (L23-33) the blurry-sharp image pair in GoPro dataset is I_B and I_N pair (not I) as the sharp image is taken with a short exposure. The only difference from the assumption is that the GoPro dataset was captured in a bright environment where 240 fps video capturing is possible, however, noise exists and image quality is low.\nThe authors may augment the sharp & noisy image with more aggressive noise for training purposes (L196-199), however, the sharp image should not be considered a ground-truth.\n\nOtherwise, the proposed method is not a true self-supervised method as I_N is generated from I in the training process. Without I (which is considered GT here), training is impossible.\n\nThe authors should at least show the proposed method could achieve higher-quality image D(I_B, I) which is cleaner and sharper than I.\n\nIn Figures I and J, is the noise in the noisy image real or synthetic? Also, how could the blurry and the real image be well-aligned?\n\n- Basic setup \n\nAre the blurry and the noisy image pair assumed to be taken 1) at the same time from different cameras or 2) at different times from a single camera?\nFor 1), the pair should be stereo images with disparity and 2) should have spatial offset between the pair due to motion.\nHowever, training from a synthetic GoPro dataset has well-aligned image pairs. Does training with the aligned data generalize to realistic-scenario images?\n\n\n- typos\n\nL141 without than accept a shoddy option -> without (sharp areas?) than to accept a shoddy option\n\nL147 patches.For -> patches. For\n\nL172, 178 sub-sampling images -> sub-sampled images\n\nL194 Hero 4 Black -> Hero4 Black \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept