Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript proposed an interesting method to infer the distribution of denoised images for a given input instead of a single denoised image. Then, the final denoise image was generated using MMSE (averaging multiple denoised images) or using MAP (finding the mode of the posterior distribution).

1) Unfortunately, this manuscript missed one very important relevant recent work:
[Quan2020] Quan et al., Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image, CVPR 2020.
This work also infers the distribution of denoised images instead of a single denoised image and yielded state-of-the-art results for both synthetic noise, real noise, salt-and-pepper noise and inpainting. 
The ways of implementing distribution generations seem similar ("a principled approach to incorporate explicit models of the imaging noise distribution in the decoder of a VAE" in this manuscript and also similar structure was implemented in the decoder part of the network in [Quan2020]) except that the proposed method here utilized random sampling in the latent space while [Quan2020] used dropout to implement it. In my view, [Quan2020] and the proposed method share some common aspects that can undermine an important contribution of this manuscript in its current form, and thus this manuscript must explain the advantages and novelty of the proposed method over [Quan2020] properly. 

Even though the proposed method with MAP is still new, for now [Quan2020] seems to have advantages over the proposed method such as (a) Self2Self requires a single noisy image while the proposed method requires noisy dataset, thus Self2Self can cover the case in this manuscript while the proposed method may not be able to cover the case of [Quan2020] with a single noisy image, (b) Self2Self yielded state-of-the-art performance on well known datasets such as Set9, BSD68, thus the readers can know that Self2Self is one of the state-of-the-art denoising methods. Even though the proposed method was evaluated on 13 datasets, none of them are used to evaluate denoisers in general. Therefore, it is hard to justify that the current proposed method also achieved state-of-the-art performance. (c) Self2Self was evaluated on both color, gray-scale images with synthetic noises as well as real noise (PolyU) while the proposed method was evaluated only on gray-scale images with real noise.

In addition, the way of implementing the generation network for samples was already proposed in [Kohl2018] for segmentation task. Thus, the contribution of the proposed method could be weakened substantially according to [Kohl2018], too, even though this manuscript nicely demonstrated that this sampling method in [Kohl2018] worked well for denoising problems.

2) Table 1 shows 13 datasets, but it is hard to use them to compare with other denoiser results in other literature. Many denoising works are using common datasets such as Set9, BSD68 (for synthetic noise), and often PolyU (for real noise) and it is helpful to have these results (at least in the appendix) for the comparison purpose with other previous works. Moreover, most images that were used in this manuscript seem to have specific structures (cells, characters), so I am not convinced that the proposed method will be generalized to regular images and color images. The proposed method has noise estimation components (co-learned, bootstrapped) that may be sensitive to underlying ground truth image structures and properties. Showing some results for common datasets (with complicated textures and patterns) as well as some synthetic noise with different noise levels (ideal cases) will convince readers that the proposed method can be generalized to other cases. The proposed method involves noise estimation and it seems to work well with texts, faces and microscopy imaging. However, these results do not seem to be easily extended to other imaging cases. 

In addition, it seems that generating 1000 images for averaging will be computationally expensive. Thus, in Table 1, it will be fair to indicate computation time for training / testing per image.




review_title: Estimating a distribution of denoised images instead of a single denoised image is not new (ref missing), but an interesting alternative was proposed. I am concerned on evaluations.

final_score: rating: 7: Good paper, accept
review_content: This paper devises a novel unsupervised denoising paradigm, DIVNOISING, that allows us, for the first time, to generate diverse and plausible denoising solutions, sampled from a learned posterior. This approach only requires noisy images and a suitable description of the imaging noise distribution, providing a new perspective for the image denoising field. It has demonstrated that the quality of denoised images is highly competitive, typically outperforming the unsupervised state-of-the-art, and at times even improving on supervised results. This paper is well-written and good-organized. However, the reviewer has the following concerns.

Q1. In DIVNOISING, noisy images are been created from a clean signal $s$ via a known noise model, i.e., $x\sim p_{NM}(x|s)$. One main practical concern is that is there any assumptions about this noise model and which type of noises are mainly considered to be removed?

Q2. The authors have demonstrated the excellent performance of DIVNOISING on several datasets, especially on microscopy datasets. Whether this denoiser can be applied to other noisy scenarios, such as a real color noisy image or MRI.

Q3. Open the source codes.

review_title: Official Blind Review #1

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new method of noise removal using convolutional VAE.  An observed image with noise is input to VAE, and after the expression $z$ in the latent space, the noise removed image is finally output.  After that, it is possible to generate a pseudo noisy observation image according to the noise model.  The noise model part is flexibly designed using the Gaussian mixture model.  In the training, VAE and noise model can be learned at the same time. Since VAE is a generative model, and a clean denoising image can be obtained by averaging a large number of candidates of clean images, $s$, sampled from the periphery of the latent space representation $z$.

The advantages of the proposed method are that it is Fully unsupervised and that the noise model can be learned at the same time.

The paper itself is interesting, and the proposed method itself is good as one of the image processing methods, but there is a lack in the explanation part.  In this paper, the noise model $p (x|s)$ is well discussed, but I think that the discussion about the image prior, $p(s)$, for clean image is weak. As explained in p3 of the paper, $p(s|x) \propto p (x | s) p (s)$ is an important formula. In conventional image processing, $p(s)$ has been interpreted as smoothness or non-local similarity. In this method, it is treated as $p (s | z) p (z)$, but I think we should discuss a little more about what this is.

In the proposed method, there is no special assumption about the prior knowledge of the image, and it seems that it is normally distributed in the latent space. In this case, a large amount of data or some help of a network structure is needed to remove noise. The use of the convolutional structure in the proposed method is an indispensable element in the explanation of $p(s)$.

The noise reduction effect of the convolutional structure itself is well known after the study of Deep Image Prior, and recent research [a, b] also discusses the relationship between convolutional neural networks and non-local similarity. These may help the discussion about $p(s)$.

[a] Yokota, Tatsuya, et al. "Manifold Modeling in Embedded Space: A Perspective for Interpreting" Deep Image Prior "." ArXiv preprint arXiv: 1908.02995 (2019).
[b] Tachella, Julián, Junqi Tang, and Mike Davies. "CNN Denoisers As Non-Local Filters: The Neural Tangent Denoiser." ArXiv preprint arXiv: 2006.02379 (2020).

-------------
As a result of the feedback, the part I was concerned  became clear, so I would like to raise the score.

review_title: Discussion about p(s) is preferred 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a modified VAE for unsupervised image denoising. Unlike existing methods only predicting a single denoised image, the proposed one can generate diverse and plausible results. The experimental results show that this method can outperform existing unsupervised denoising methods. I have several concerns about this paper.

(1) The modifications relative to Kingma & Welling (2014) are marginal. The proposed method more likes an application of Kingma & Welling (2014).

(2) According to Eq. 1, the noises are pixel independent which does not hold for most of the applications e.g. images after demosaicing or ISP denoising. How can the proposed method deal with it?

(3) VAE framework seems only to work for small and constrained images. I am wondering about the denoising results of the proposed method for high-resolution nature images?

(4) At the beginning of Sec. 5, the authors claim that the noise model should be known. How does the proposed method deal with the noises without the noise model for example images after demosaicing or ISP denoising?
-------------------------------------------------------------------------------------------------------------------------------------

I appreciate that this paper has some merits. But I lower my rating because of the limited usage of the proposed method.
It seems it cannot handle high-resolution nature images. Most of the experiments use constrained images e.g. biomedical images or images with small resolution.

review_title: The idea sounds reasonable. But the novelty seems marginal and I have several concerns about the application range of the proposed method.

Meta-review: A simple but sensible idea to improve VAE with good experimental results.
Decision: Accept (Poster)