Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new method for the denoising of diffusion weighted images with diffusion denoising models. It brings about a series of modifications on existing literature that actually determines higher performance compared to state-of-the-art.

The article presents extensive validation which is highly appreciated, and shows results from comparing state-of-the-art techniques. My main concern with this is the lack of synthetic data validation, which is something that (unfortunately) many articles on the subject lack. I think that when talking about denoising it would be somehow mandatory to display synthetic results, even more when denoising is achieved through machine learning, where the risk of introducing "artefactual allucinations" in the images is very high.


strength_and_weaknesses: The article is interesting, inspiring and introduces a series of improvements that are of interest for the community, as also is the shown performance. I like the formulation of the solution to the problem in three stages and the advantage of each of them is clearly presented with quantitative information.

The main weakness remains, in my opinion, the lack of synthetic validation. Despite many "real data" examples it is not possible to assess the amount of allucinations generated by the methodology and the amount of intensity bias introduced with each pixel.

In my experience with Patch2Self on synthetically generated data I could prove that the estimation of quantitative measurements (e.g. the estimation of diffusivities) is subject to a bias, which is not present, for instance, with methdos based on MPPCA. In this sense, without synthetic results it is not possible to assess this kind of performance.

I note, especially in figure 11 and 14 , that indeed the DDM2 method may introduce relevant allucinations. For instance, in fig. 11 please look at the first raw, and observe the cerebrospinal fluid areas in the middle of the brain, which have very different shapes compared to the noisy image. This is an example of "invented" tissue contrast. Another example is found by looking at the first column image of each row of figure 14 (left hand side). In the bottom left part of the brain (just beneath the red square) there is an "island" of tissue contrast that is better preserved in the stage 1 compared to the final result.

I think that the authors should point out with arrows to this kind of differences. While reading I had the impression that authors were stressing a bit too much the fact that there were very few allucinations, and in my opinion, that is not the case. In fact, I strengthen the comment about the lack of synthetic experiments!


I have other points that I would like the author to address:
- Does the presented method generalize to the case when the noise variance varies spatially? In my opinion it does not, so please mention that a limitation.
- Noise in MRI is spatially correlated (use of Partial Fourier, for example): what are the consequences of this for the approach?
- Noise in MRI is non-Gaussian (e.g. Rician): how does this affect the proposed method which clearly only works for Gaussian data? In section 5 authors mention SENSE, which does not perturb the Gaussian nature of the noise on complex images, but still yields Rician magnitude images!

clarity,_quality,_novelty_and_reproducibility: The article presents novel content and is clear and the authors have made an effort to summarize the state-of-the-art and comparing techniques very well. I appreciated that. Although I have not found the link to the code, I believe that it will be sent out at publication. Link to the data is present, and tips on how to make everything to work are also present. Therefore, I believe that the article is reproducible.

Minor clarifications:
0) in the introduction of diffusion MRI, the gradients are not magnetic fields, but rather generate a spatially-varying magnetic field.
a) Page 2 → and uses them TO supervise each other
b) figure 1 → and black blocks indicate.. you mean blue blocks?
c) Page 4 → however 3D volume results are shown in the supplementary MATERIAL
d) I don’t understand the right-hand-side of eq.4 Where does it come from? Please give indications on its derivation.
e) The sentence at page 5 “We can determine the closest matching state S t of x by comparing G and posterior p(S t )—σ and β t .” is not intelligible. Please rephrase.
f) Page 7 → please fix the sentence “We do not that N2S did have improved worst-case results than DDM 2 but this was likely”
g) Page 7 → please fix sentence “are necessary for DDM 2 Ṫhis by its nature”
summary_of_the_review: The article tackles a challenging problem. It comes up with new ideas/modifications with respect to the state of the art. It achieves remarkable results although the full extent of these is difficult to evaluate. This is not just a problem of this article but of all the comparing state-of-the-art works, since they do not provide results on synthetic data. This is a problem with machine learning/generative techniques for denoising of medical data, because allucinations can cause dramatic effects. I would not use this denoising method on data from patients. Nevertheless, the methodology described in the article is certainly of interest for the community and can inspire other researchers. As such, the article has a great value and represent a step forward in the good direction. Despite synthetic validation is missing, favorable comparison with many competing methods has been clearly shown.

To summarize, the article discloses some key ideas to solve the denoising problem using diffusion methods. These ideas are valuable and of interest, and lead to improved performance. The problem statement is clearly formulated, and the literature well acknowledged. I recommend  the acceptance of the article as is. If a second revision stage is foreseen, I would recommend the authors to take into account my comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides an approach for self-supervised denoising of MRI data using diffusion-based generative models. The approach is applied to denoise diffusion-weighted MRI, hence the name DDM2. A three-stage approach is used to train/denoise: 1. learning a denoising function with existing self-supervised approaches; 2. fitting the noise statistics to a Gaussian; and 3. training a conditional diffusion model using the noisy data and noise statistics. The method is evaluated on four different datasets of diffusion-weighted MRI. Comparisons to other self-supervised methods are shown, as well as ablation experiments.

strength_and_weaknesses: Strengths:
- The landscape of self-supervised image denoising is quite crowded and can be difficult to navigate. The authors do a pretty good job contextualizing their work within this space.
- The approach to training a conditional diffusion model through spatially shuffling the noise residual is interesting.
- The ablation results lend credibility to the need for the three-stage approach, differentiating it from other self-supervised denoising works
- Extensive experiments are conducted on different datasets
- The paper is very detailed and code to reproduce results are provided

Weaknesses:
- It is unclear how this work fits in with other denoising methods such as [1] and [2]. For example, as the noise is assumed Gaussian, could [1] simply be used to denoise the volumes. Furthermore, could the denoiser in [1] be used according to the strategy in [2] to further refine the solution? To me this seems more straightforward than the proposed three-stage approach.
- How would the approach compare to running annealed Langevin dynamics using a pre-trained MRI diffusion model (for example, from [3]) with an identity A operator? In other words, is the scan-specific nature of the method necessary?
- As the approach is not actually specific to diffusion MRI, and the authors report that even n=1 is sufficient for learning the denoiser, could a different MRI dataset (where GT is known) be used to validate the methods (with noise added on top of the GT)? For example, from FastMRI or SKM-TEA?
- It is unclear from reading the paper if multiple diffusion directions is necessary
- It seems like all the datasets have multiple scans at the same b-value (for example 10 have b=0 and 60 have b=1000). Are these b-value images collected with different diffusion directions? What would happen if the images with the same b-value and direction are simply averaged together? Could this be used as a GT, either for comparison or for some other supervised approach?
- It is unclear how the dimensions of the data correspond to the directions/b-values. Please explain the dimension sizes as they relate to the # of directions and b-values.
- is the method only applicable to denoising with multiple b-values? do the b-values need to be the same for denoising to work?
- It is said that two neuroradiologists reviewed the images. Can you please comment more on this evaluation? How many images were reviewed per dataset? why wasn't a larger evaluation conducted using the neuroradiologists, for example to rate image quality, SNR, CNR, fine details, etc?
- Could the authors test (or at least comment) on the impact of Rician vs. Gaussianity, for instance with GT data and additive noise? It was unclear in the text whether the data used are complex-valued or magnitude only, and what the impact is of the Gaussianity assumption.

[1] Christopher A Metzler, Ali Mousavi, Reinhard Heckel, Richard G Baraniuk, Unsupervised learning with Stein's unbiased risk estimator, arXiv preprint arXiv:1805.10531
[2] Zahra Kadkhodaie, Eero P. Simoncelli, Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser, https://arxiv.org/abs/2007.13640 
[3] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, Jon Tamir, Robust compressed sensing mri with deep generative priors.  Advances in Neural Information Processing Systems

clarity,_quality,_novelty_and_reproducibility: The approach is clear and appears to be novel. The quality is high and the work appears to be reproducible. I commend the authors for their effort in building a clear and transparent work.

summary_of_the_review: The authors develop a three-stage approach to denoising diffusion-weighted MRI using self-supervised denoising diffusion models. The method appears to improve over other self-supervised baselines, though some details remain unclear. The contribution is clear and the work is well-organized and novel.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised denoising method for diffusion MRI with diffusion model. The model consists of three steps which 1) learn noise pattern, 2) match an input to different diffusion stages and 3) diffusion model that generate denoised image. The model is validated on 4 different dataset to demonstrate generalization capability of the method.  
strength_and_weaknesses: Strength:
- Clear text and step-by-step description of the method. 
- Differences from prior works well explained. 

Question (rather than weakness):
- It would be nice to have an explanation how prior denoising of MIR are performed with supervision, as typical denoising methods for images are unsupervised. 
- In Stage I, patches are replaced by image slices and slice-to-slice mappings are learned. There must be drawbacks from such a coarse mapping (e.g., losing local structures in the image space) and I am wondering if the authors have any thoughts on this. 
- Is there any backbone model to approximate necessary functions within the framework?
- What are $\lambda$ in equation (4)?
- In the method, it is not clear where the self-supervised training is occuring. 
- Is the noise in typical MRI or diffusion MRI iid or systematic? The noisy images in the experiments are synthesized with different $\beta$ levels (correct me if I misunderstood) and I am not sure if they truly mimic the noise pattern in reality. 
clarity,_quality,_novelty_and_reproducibility: - The manuscript is written quite clearly. 
- Most of the components should be able to be reproduced. 
- Differences from priors are well addressed. 
summary_of_the_review: I am not a super expert in diffusion model so I cannot judge very well on the novelty, but I think the differences of this work from the prior ones are well explained. 
Text is written clearly to demonstrate the authors' ideas and they are well validated through the experiments.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposes a new method for denoising diffusion MRI data using recent advances in reverse diffusion. The method proposed a three stage approach and builds on top of Patch2Self which is the state-of-the-art method in the field of Diffusion MRI. The method is developed using generative models which indeed show potential for denoising applications.
strength_and_weaknesses: Strengths

The proposed method makes use of diffusion models to perform the denoising. While the approach seems useful, there are some questions which need to be answered:

Weakness

The official implementation of Patch2Self does not have MLP as a regression model. It is stated clearly in the GitHub repository of the Patch2Self implementation that DIPY implementation is the official one. Was this the one that was used? If yes, please can you provide the model parameters set? 
In the supplement of the Patch2Self paper, the 1st figure shows that MLP performs worse than OLS and Ridge. Why was this chosen as the regression model for comparison?
The authors of the paper compare the performance of DDM2 with DIP, Noise2Noise and Noise2Score. DIP and Noise2Noise are not valid for dMRI data on may accounts. Was care taken to ensure that the training was performed on dMRI data? I would assume that these models would not do a good job out-of-the box. 

The authors say that the Patch2Self was compared in all cases with patch radius 0 and MLP model. But as per the Patch2Self paper (Fig. 6C), if the number of volumes is lower, the patch radius needs to be increased. This makes the comparison in Fig. 5 incorrect.  The official implementation of Patch2Self needs at least 10 volumes for optimal performance. 

It is also unclear why 2D slice-to-slice would not have structure in the residuals as opposed to 3D volume-to-volume mapping. This needs to be discussed in further detail. 

In Figure 15, for the Sherbrooke dataset, the residuals are flipped -- please rectify this mistake. 

The authors do not do a good job on validating the performance of the denoiser. While FA is an important metric the way SNR and CNR is computed is invalid. Furthermore the comparison needs to be done on different microstructure measures. 

Generative models typically run the risk of hallucinating structures -- which may not be visible in the residuals. An analysis showing this needs to be added. This can be shown via DEC FA maps and Tractography analyses. 

clarity,_quality,_novelty_and_reproducibility: 
The work is original but the paper lacks clarity and quality of analyses. A key shortcoming is the validation is done only on the raw signal. The analyses needs to happen on the derivative microstructure measures or tractography tasks. 

summary_of_the_review: While the method is new, it has quite a few shortcomings in its current form. Answering the above questions and improving the validation analyses should fix the issues.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper suggests a new method for denoising diffusion MRI images based on diffusion models. As can be seen from the reviews, one of the main objections to the paper, by Reviewer RsQ4, is that the comparison with existing baselines was not carried out in a fair way. In their rebuttal, the authors have carried out further experiments, and after a discussion with reviewer M8JP, they have suggested a more sober description of their results. It is a requirement for the acceptance of the paper that this description is also added to the paper.


Strengths:
- The paper is interesting, well written and novel
- The experimental section is strong, given the updates in the rebuttal


Weaknesses:
- As explained above, in the original version of the paper, the comparison with existing work was not fairly carried out; however, the authors rectified this in their rebuttal
- Most of the weaknesses listed by the reviewers are actually questions, which I take to be more of a sign of interest than a weakness.
justification_for_why_not_higher_score: The reviewer scores do not support it.
justification_for_why_not_lower_score: This should actually be a relatively clear accept; the one rejection recommendation clearly has to do with the unfair comparison with state of the art. The authors have updated this in the rebuttal phase, and as long as they update the paper accordingly, the paper can be accepted.
Decision: Accept: poster