Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use a score-based model for the task of removing structured noise. Specifically, the method uses a pretrained (unconditional) NCSNv2 model as the signal prior and uses modified update rule during sampling that incorporates the score of a separate noise model.  Experimental results show that score-based models achieve superior performance to existing methods based on GAN and normalizing flow.
strength_and_weaknesses: **Strengths**
* Clear exposition and motivation.
* Problem is relevant to the field as inverse problems encompass a large family of problems and is an active area of research.

**Weaknesses**
* Experimental comparison is done on a single dataset -- especially when compared to the main baseline (Whang et al.), which considered various types of noises.
* Unclear whether the benefits are due to the difference in the model class, or other factors (e.g. hyperparameters, network size, etc).  While I don't doubt that score-based prior could outperform GAN/flow priors, it would have been nice to see some effort on trying to decouple the effects of model class vs. other architectural/implementation details.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
The paper is clearly written, and I had no problem understanding the motivation as well as the proposed method.

**Quality**
* Pages 2-6 are spent on background and the description of the method.  This seems a bit excessive and could be made more concise to make room for additional experiments.

**Novelty**
Given the recent success of score-based models, its application on the existing task of structured noise removal is a relatively straightforward idea.  That said, I do think a comparison between score-based prior vs. other generative priors have a merit.  This is why I hoped to see a more thorough experimental section.
summary_of_the_review: This paper proposes to combine two score-based models -- signal prior and noise prior -- for the task of removing structured noise.  The proposed method outperforms existing deep generative priors on the task of MNIST digit removal, and while the approach is simple, this discovery has some value.  That said, the experiments and analysis are lacking.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes an approach for solving the inverse problem via diffusion models. Particularly, the work considers structured noise 
that contaminates the observations. To solve the problem, another line of the diffusion process that describes noise is used except 
for the image generative process. The score models in both image generation and noise generation are pretrained, and then applied
to two reverse stochastic differential equations to sample images and noises. During inference, a data-consistency step is added.
Experiments about removing MNIST digits and compressive sensing are presented to verify the effectiveness of the proposed
methods.

strength_and_weaknesses: Strength: 
1. The work additionally takes the noise structure into consideration and proposes a noise diffusion process together with an image 
diffusion process to solve the inverse problem. 
2. The method that samples image and noise from joint conditional posterior is rational and interpretable, and also seems easy
to implement.

Weakness
1. The right side of Eq. (14) does not contain $n_t$.
2. In algorithm1, step 7 and step 8 seem to be covered by step 10 and step 15, respectively.
3. The necessity of using the diffusion model to describe the structural noise is not convincing. Why is it better than the other generative model, such as GAN, VAE and Flow?
4. How the structural noise is embedded into the diffusion model? This point is not clear.
5. It is not clear what the training data is when training the second score function $s_{\phi}(n,t)$. 
6. I wonder how the proposed method behaves compared with non-generative methods for removing MNIST digits and compressive sensing, and what is the superiority of the proposed method.

clarity,_quality,_novelty_and_reproducibility: Clarity: some clarity is not clear.
Quality: ordinary
Novelty: limited.
Reproducibility: not sure.

summary_of_the_review: The work is relatively easy to follow and the idea is clear. However, the clarity to embed the structural noise is not clear. Additionally, the necessity of using the diffusion model is not convincing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new method to solve a large class of inverse problems using diffusion models. The approach is to jointly learn both a source data diffusion model and a noise diffusion model, from only a training set of noisy observations. It is assumed that the noise is added to the source signal in a known manner, but a highlight here is that the noise is allowed to be structured and not assumed to be i.i.d.

A method to jointly learn the two diffusion models is proposed. The method is then tested by using CelebA as the source signal and then adding MNIST digits as a noise model. In experiments, the diffusion model is shown to outperform alternative solutions that use GANs and and flow models. 
strength_and_weaknesses: Strengths:
- The paper is clearly written and well-organized.
- The paper addresses a broad class of problems. Scientists in other fields (e.g. astronomy, physics) could find this very useful. 
- The method proposed is not obvious, and a significant contribution. 
- The experiments are well done. 

Weaknesses:
- None
clarity,_quality,_novelty_and_reproducibility: The paper is clear and original. 
summary_of_the_review: This is exciting work, and an excellent contribution to ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to remove structured noise from noised images using a diffusion model that describes the joint distribution of the clean data samples and the structured noise. Using an additional score model for the noise, an approximate score for the joint distribution (for all time $t$) is derived. Experimental results show that the proposed method outperforms the previous benchmarks based on normalizing flows and GANs.
strength_and_weaknesses: Strength:

- Removing structured noise is an interesting task and the idea of the paper is fashionable.
- The experimental results look promising.

Weaknesses:

- Technical soundness & Clarify: I found the technical part of the paper hard to follow and a little bit misleading.
    * At the beginning of Section 3.1, it seems that this work tries to build a conditional distribution $p(\mathbf{x},\mathbf{n}|\mathbf{y})$ so that the inversion problem can be solved by just sampling from this distribution or taking its MAP estimation. And then to build this sampling process (or to model this distribution), a diffusion process is imposed on the "joint space" of $(\mathbf{x},\mathbf{n})$, while everything is conditioned on the noise observation $\mathbf{y}$. However, in Eq.(14), the score is conditioned on $\mathbf{y}_t$, which is further explained below in Eq.(16) as a corrupted version of $\mathbf{y}$. What is $\mathbf{y}_t$ here? Why does the conditioning depend on time $t$? What is the exact definition of $p(\mathbf{y}_t|\mathbf{y})$ and why is it related to the diffusion process of $\mathbf{x}$ and $\mathbf{n}$?
    * In Eq.(15&16), why does the second term of RHS use a sample $\hat{\mathbf{y}}_t$? Is it a good approximation?
    * In page 3 it mentioned $s_{\phi}(\mathbf{n},t)\approx\nabla_{\mathbf{n}}\log p_N(\mathbf{n})$. Is the score model $s_{\phi}(\mathbf{n},t)$ actually independent with time $t$?
    * It is not very clear how are the two score networks trained. Is $s_{\phi}(\mathbf{n},t)$ trained jointly with $s_{\theta}(\mathbf{x},t)$?
- Experiments: It is mentioned in Section 4.1 that the baseline results are improved to be more competitive. However, in Whang et al. (2021) the results seem to be better than what is reported in this paper (see Figure1&2 in Whang et al. (2021)). The inconsistency seems to make the results here questionable. Can you please provide more details on this?


Minors:
- In Eq.(13) the “proportional to” is not correct as the RHS uses log probabilities.
- In Eq.(14) the drift term should be $f(t)(\mathbf{x}_t, \mathbf{n}_t)$.
clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: I have raised some concerns. Please see my comments above.
Novelty: The idea looks novel to me.
Reproducibility: The paper has described the reproducibility statement, but some details, such as how the score net for structured noise is trained, remain unclear.
summary_of_the_review: In conclusion, I have raised several concerns regarding the technical soundness and empirical results of the proposed method, which prevents me from giving this paper a higher rating at this time. I will consider increasing my rating if the authors address my concerns and clarify my misunderstandings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The problem considered is relevant, and the proposed method of removing structured noise with diffusion models is intuitive and shown to perform well for removing handwritten digits from celeb-A face images. Conceptually, the proposed methods is similar to Wang et al. 2021''s approach in that it uses generative priors to remove structured noise. A major difference is that the paper under review uses score based models and proposes a corresponding conditional sampling approach, while the prior works rely on other kinds of generative models. Using score based models in this context is very reasonable since score based models perform very well amongst methods imposing generative priors.

However, the paper's experiments are very simplistic and the main experiments pertain to removing handwritten digits from celeb-A pictures. It would be more interesting to study practically relevant cases of removing structured noise from scientific data for example, where structured noise actually occurs in practice, and to compare to a wider array of methods. For example, for such scientific data, structured noise removal with networks trained end-to-end also work very well. 

justification_for_why_not_higher_score: As written in the meta-review, the experiments are very simplistic and do not cover practically relevant cases of structured noise removal.
Decision: Reject