Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers the problem of taking high quality clean images given the difficulty of the tradeoff between obtaining blurry images due to long exposure times, and noisy images from small exposure time. A typical way to handle the tradeoff is to find a compromise between the two, then use a debluring or denoising model to fix up the image (the models trained using supervision or self-supervision). Instead this paper proposes to make use of both a blurry and a noisy image, taking advantage of the fact that the blurry image will have very little noise, and the noisy model very little blur in order to co-train *both* a debluring and a denoising model. Using these two models the authors Provide a method for producing a single clean image given a blurry and a noisy image. 

strengths_and_weaknesses: 
The main contribution of this work seems to be the idea that noisy and blurry images contain complimentary information, that, if both images are collected, can be used to train both a debarring and a denoising model. Using both of theses models to restore images then leads to good clean images. This message seems clear and intuitive, and the method a useful contribution. Too I am happy to lean towards acceptance. But there is an important major caveat. It. Is non-trivial, even infeasible to take both a long- and short-expoosure image of the same scene. The authors solution is to use two cameras side by side taking photos simultaneously. So I am quite worried about the general availability of the type of data required to use this method. 

---

To summarize some points:
- The paper is clearly written and highly accessible.
- The idea of using the complimentary information in noisy vs blurry images is natural and intuitive
- Strong quantitative and qualitative results: the proposed method seems to improve over existing  supervised & self-supervised denoising methods (& more). 

---

Some questions:
- The real-world dataset only contains 61 images, only 30 of which are used for training. This is clearly a really small dataset. I would really like to hear from the authors on their thoughts about this data issue, and how it affects the viability of the joint-training method. 
- Could other intermediate exposure time images also be incorporated? So instead of just having very short and relatively long exposures, also include medium length exposures.
- The overall loss includes two weighting hypermarameters (Eqn 12). How sensitive is the method to the choice of these hyper parameters?
- Recent works in diffusion modeling have found that when denying a sample it is better to parameterize the denoising model to *rpedict the noise* instead of directly predicting the clean sample. Did you at all look into this? It could either 1) improve the current method, or 2) provide interesting counter-evidence to the generality of the observation from diffusion modeling.

---

A disclaimer: I do not keep up to date with the recent literature in image restoration, so potentially will not be aware of related work.


questions: See previous section.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a self-supervised technique for image restoration from a blurry and a noisy image pair. By observing that each task can be trained from the other low-quality image, the authors propose a method that brings sharpness from the noisy image and the cleanliness from the blurry image. 
However, I’m not sure if this method is a true self-supervised method. The training objective seems to be recovering the real noisy image from the blurry and (synthetic) augmented noisy image pair.

strengths_and_weaknesses: [Strengths]

While there have been attempts to use the blurry and the noisy images to achieve better-quality images, this method is novel in terms of self-supervised design. The self-supervised learning process builds on top of the observations in the nature of deblurring and denoising. The experimental results look promising given my concerns are resolved. 

[Weaknesses]

- Related Works

I think the authors should also refer to previous burst image denoising and burst image deblurring works.
The authors mainly describe self-supervised image denoising papers only. (+ 1 self-supervised deblurring paper)

- L196 

From the definition in the introduction (L23-33) the blurry-sharp image pair in GoPro dataset is I_B and I_N pair (not I) as the sharp image is taken with a short exposure. The only difference from the assumption is that the GoPro dataset was captured in a bright environment where 240 fps video capturing is possible, however, noise exists and image quality is low.
The authors may augment the sharp & noisy image with more aggressive noise for training purposes (L196-199), however, the sharp image should not be considered a ground-truth.

Otherwise, the proposed method is not a true self-supervised method as I_N is generated from I in the training process. Without I (which is considered GT here), training is impossible.

The authors should at least show the proposed method could achieve higher-quality image D(I_B, I) which is cleaner and sharper than I.

In Figures I and J, is the noise in the noisy image real or synthetic? Also, how could the blurry and the real image be well-aligned?

- Basic setup 

Are the blurry and the noisy image pair assumed to be taken 1) at the same time from different cameras or 2) at different times from a single camera?
For 1), the pair should be stereo images with disparity and 2) should have spatial offset between the pair due to motion.
However, training from a synthetic GoPro dataset has well-aligned image pairs. Does training with the aligned data generalize to realistic-scenario images?


- typos

L141 without than accept a shoddy option -> without (sharp areas?) than to accept a shoddy option

L147 patches.For -> patches. For

L172, 178 sub-sampling images -> sub-sampled images

L194 Hero 4 Black -> Hero4 Black 

questions: L158 The ghost artifact has been pointed as imperfect blurry image in the following paper:

S. Nah et al., “NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study,” CVPRW 2019

They interpolate the frames before averaging the sharp (& noisy) images to remove the ghost artifact.
Following similar approaches may benefit finding sharp regions from equation (8).

limitations: Limitations described in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a self-supervised image restoration method named ``SelfIR''. The basic idea is to work with a pair of short-exposure noisy and long-exposure blurry images. The complementary nature of blurry and noisy pairs make it possible to jointly utilize the auxiliary supervision information for self-supervised denoising. The reported experimental results are promising and convincing. SelfIR can achieve better performance than competing self-supervised denoising methods on synthetic datasets and comparable performance to N2N (ref. [15]) on real-world datasets.
strengths_and_weaknesses: Strengths:
+ The idea of exploiting complementary nature of blurry and noisy image pairs is sensible. There is some novelty with the developed co-learning of deblurring and denoising in Sec. 3.3
+ The reported experimental results are mostly convincing and promising. Improved denoising results over Ref. [9] (CVPR'2021) and Ref. [37] (CVPR'2022) have been reported on synthetic sRGB images.

Weaknesses:
- Originality. I have found more clarification is needed for Sec. 3.1 and Sec. 3.3. How is Sec. 3.1 different from Noise2Nosie [15] and how is Sec. 3.3 different from Neighbor2Neighbor [9]? Many technical details such as Eqs. (4)-(6) and Fig. 2(b) have strong similarity with previous works. Even though the authors are inspired by those existing works, they need to highlight what are the new insights brought by this paper. I am afraid that the novelty of these two subsections is limited in the current version.
- Quality. I have two major concerns over Sec. 3.2 (denoising with long-exposure image). First, I have found that the strategy of dividing an image into non-overlapping patches ad-hoc. Even though "it is crucial to pinpoint the sharp areas in the long-exposure images", I don't understand why the authors take a patch-based (instead of a pixel-based) approach. Second, the use of SSIM in Eqs. (7)-(8) lacks substantial justification. Maybe it is based on heuristics but the quality of presentation for this paper is lacking, in my biased opinion.  
- Clarity. I wish authors can clarify more about the self-supervised learning procedure in Sec. 3.3. The discussion about "complementarity" (line 169) is vague because authors did not explicitly discuss how the complementary nature was exploited by the design of co-learning method.
- Significance. The importance of this work seems to be to show the proposed SelfIR works better on real-world situations but the majority of the experimental results (especially visual comparisons) are for synthetic images only.
questions: 1. Is the proposed SelfIR "collaborative learning" or "self-supervised learning" or both (Line 170-171)? It seems to me Eq. (9) is collaborative learning since it involves the blurry-noisy image pair; while Eqs. (10)-(11) can be interpreted as self-supervised learning since it uses the pseudo-GT (output of restoration network D). Please correct me if I am wrong.  
2. How is the sharp patch detected in Sec. 3.2 used by co-learning in Sec. 3.3? 
3. Is there any risk of inconsistency between the patch-dividing strategy in Sec. 3.2 and sub-sampling operation in Sec. 3.3?
4. Do you assume the blur process with non-uniform kernels K is known in Eq. (1)? If Yes, this seems a strong assumption because nonuniform kernel estimation from real-world blurry images is a long-standing open problem.
5. Do you have visual quality comparison for denoising real-world raw sRGB images? Figs. 3-4 are for synthetic images only where supervised learning (baseline) is easy to obtain.  
limitations: When does the proposed SelfIR fail? For example, Table 2 shows that the proposed method falls behind N2N [15] on PSNR and Blind2Unblind [37] on NIQE. Any discussion or explanation about the limitation of SelfIR?
review_title: null

Meta-review: metareview: All three reviewers voted to accept the paper, and the detailed rebuttals from the authors helped to clarify reviewers' original concerns. One remaining concern from one of the reviewers is whether this method should be referred to as "self-supervised".  However, authors clarified that it is reasonable to consider this method self-supervised for real-world data. I am fine with therefore leaving "self-supervised" in the title.

confidence: Certain
Decision: Accept