Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a hierarchical VAE model that produces good denoising results. In addition, the model can be adapted to perform removal of more structured artefacts. 
main_review: Strengths:

The proposed model is a novel unsupervised technique that is competitive or outperforms the state of the art using a multiscale variational autoencoder framework. I believe that this is a very interesting contribution. In addition, the paper provides extensive experiments and ablation analysis that provide interesting insights into the properties of the model. Also, the paper is written clearly.

Weaknesses:

In the abstract and introduction, the authors imply that their method produces meaningful uncertainty quantification because it enables one to sample possible restored images, but at the end of the day they use an average as the final denoised result in order to compare to other methods. The stochastic outputs are not evaluated. It would be very interesting (and would back up the tone of the abstract and introduction) if the authors could at least provide an example where the stochastic inputs are useful in some way. This is not a condition for publication, I think the method is interesting in any case. 
summary_of_the_review: The authors show that a multiscale VAE can produce good unsupervised denoising results, provide meaningful analysis, and show that their technique can be adapted to remove structured artefacts. I think this is a meaningful contribution, and therefore I recommend publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work extends the work of DivNoising (Prakash et al., 2021) to build an interpretable approach for unsupervised image restoration. The proposed method, namely Hierarchical DivNosing (HDN), is based on the well-studied concept of hierarchical Variational Autoencoder. This work is the first to use hierarchical Variational Autoencoder for the task of unsupervised image denoising. In doing so, HDN achieves state-of-the-art unsupervised image denoising results on twelve benchmark image datasets. The method is also shown to remove structured artifacts on three real microscopy datasets.
main_review: This paper has the following strong points

- First work to use hierarchical Variational Autoencoder for unsupervised image denoising task
- Yields state-of-the-art results on unsupervised image denoising
- Reconstruction results are more interpretable as compared to competing ones
- Could remove structured noise in some special cases of imaging

The only negative that I found concerning this work is that

Lack of discussion on training and inference efficiency comparisons of the proposed method with others
Recommendation: As I have stated under the strong points, this paper brings in multiple contributions in terms of the first attempt of its kind while yielding state-of-the-art results. Therefore I rate this work as novel enough to get accepted.

I have the following questions for the authors.

There has been no discussion on how much inferior the training and test time of the proposed method as opposed to the competing ones such as DivNoising or even other fully supervised methods. Also, how much time does it take to find the MMSE estimate with 100 samples? I would like authors to have a detailed comparison along these aspects.
DivNoising has reported the MAP estimate to be best performing. What is the plausibility of such a MAP estimate with the proposed HDN framework? I do not find any discussion regarding this possibility.
I would like the authors to address the below mentioned points in the final version of their paper.

Section 6 points out to “line artefacts in Fig. 4”. However for a reader, it is hard to understand what these artefacts are. It would be better to give a description or highlight such artefacts for improved revelation of the same. Also, providing additional visualization of the ground truth image can further improve clarity.

Section 7 says “the situation is more complicated for structured noises in other data domains and is slated for future work.” However, it would have been more informative if authors were to show examples of structured noises in these other domains, and how the proposed method works in these cases. This can help shed more light on the limitations of the proposed method and also for future research directions.
summary_of_the_review: This work advances the state-of-the-art in terms of use-case novelty and performance improvement. Henceforth, the contributions are significant enough for the acceptance
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Reviewed work presented a new and very promising approach to unsupervised denoising titled Hierarchical DivNoising. Hierarchical DivNoising is based on earlier published DivNoising architecture. The authors compared the performance of their HDN approach to the closest state-of-the-art unsupervised as well as supervised denoising methods on an impressive set of twelve datasets. In all datasets, the presented method has outperformed other unsupervised alternatives, while sometimes even improving on the performance of supervised counterparts. The authors also show that the layers of the proposed architecture encode interpretable levels of abstraction, which can be used to one's own advantage e.g. to get rid of the structured noise often present on the microscopy images. 
main_review: For the most part, the authors make claims that are well-supported by either the evidence presented in the paper or by the cited research. The presentation of the work, including the use of language, formulas, and graphics is self-consistent and favors smooth reading. The results discussed in the paper make it clear that the Hierarchical DivNoising is superior to all relevant baselines, sometimes even surpassing the performance of supervised methods.

Although overall the paper is very comprehensive and thorough, there are two main things that are worth considering:
1. Section 3, Non-hierarchical VAE Based Diversity Denoisers, discusses the most notable competitors for HDN method, namely classical VAEs algorithms and DivNoising. Later in Section 4, the authors introduce HDN, describing in detail its architecture, loss function, types of extracted representations. The main differences (from the architectural point of view) between HDN and DN need to be explicitly written out either in this or later sections to help the reader assess the novelty of the discussed approach. Also, the authors did not discuss the difference in the number of trainable parameters that can influence both the training time as well as the performance of the model. 
2. Hardware requirements. Multiple times across the paper, the authors highlight the low computational cost of the HDN method compared to other VAE-based approaches. Yet, to the best of my knowledge, no specific data on training times i.e. time for one epoch, inference time on the same size data, total training time have been presented in the text. 

A few less critical, yet still relevant comments:
1. Nor all variables used in equations are explained or defined. For example, variable s' from the equation (2), N, xi, and i in equation (6). It is important that every element is properly defined to help the reader follow the main idea. 
2. Bottom-up, top-down network definitions are missing. In the first paragraph of Section 4, the authors introduce two new terms that have not been defined anywhere above in the text, namely bottom-up and top-down networks. These terms should be defined and explained before being used, otherwise, they might cause confusion.
3. Some equations would benefit from short intuitive explanations. Equations such as (8) and (12) would be easier to comprehend if they would be supported by short and more intuitive explanations. 
4. Structured noise patterns are not clearly visible in Figure 4, which shows the results of removing structured noise. As input images also contain pixel-noise it is hard to observe the structured noise, except for N2V where it is possible to see it. Is it possible to make an example that very clearly illustrates the structured noise in the original images? 
summary_of_the_review: The reviewed paper makes claims that appear to be correct, supported by empirical evidence, theoretical insights, and prior literature. The proposed method is based on existing technology but extends in a number of ways, which makes the novelty of the paper significant. The potential impact of this research seems to be substantial, provided that denoising is an important problem in many applications related to computer vision. Overall, this is sound work, which seems to significantly improve the current state-of-the-art.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an image restoration method that can not only reduce pixelwise noises but also remove artifacts in resultant images. Introducing the idea of hierarchical representation of latent variables analogous to VAEs, the proposed method improves the denoising performance of DivNoising (DN). In addition, the authors propose a method for artifacts removal based on the analysis of the image components represented by the latent variables at each layer. The method requires no pairs of noisy images and the corresponding noise-free signals for training but requires the probability distribution of pixel values conditioned by the corresponding signal strength. The experimental results demonstrate the proposed method outperforms DNs.
main_review: Strengths:
The paper shows one can improve the performance of DN by introducing the hierarchical representation of the latent variables introduced in VAEs. In addition, the paper shows that the artifact removal can be realized by the analysis of the visualized image patterns corresponding to the latent variables in each layer: By deactivating the layers of latent variables at which the artifacts are encoded in the restoration, one can remove artifacts even though they are spatially structured perturbation of pixel values.

Weaknesses:
Artifacts targeted in the paper are not well characterized. It is not sure if the structured noises would be encoded in the first and/or the second layer(s) of the latent variables independently from the input images. The limitation of the performance of the proposed artifact reduction should be experimentally or theoretically explained more clearly. When the deactivation is not applied, would the HDN always generate similar artifacts independently from the input images or from the network architectures? Is it sure if the artifacts are always enced by the latent variables of layers near the bottom? If the artifacts change depending on the input images as described by Eq.(3) and are encoded by the layers near the top, can the proposed method still reconstruct high quality images even when the layers near the top are deactivated?　It is not clearly demonstrated by the current paper whether the proposed method will show high performance of artifact removal against variety of input images.

By the way, if the reviewer understands correctly, the artifacts handled by the proposed method are not generated in the process of imaging but essentially by the reconstruction from given images. The noises denoted by e in Eq.(1) does not include artifacts (spatially correlated noises) and the noises denoted by n in Eq.(3) include the artifacts. Denoting both e and n as "noise" makes it more difficult to understand the proposed method. Artifacts, structured noises, can be generated even in the process of imaging. It is recommended to denote e and n in different words. It is also recommended to clearly describe that only the artifacts generated in the reconstruction process are handled by the proposed method when describing the related works of structured noise removal.
summary_of_the_review: This paper improves  Prakash et al.(2021). The reported facts that the hierarchical representation of the latent variables improve the performance and that the artifacts are encoded at the layers near the bottom would have worth sharing. As for the artifact removal, on the other hand, it is not so clear how effective the proposed method is for variety of images and hence the reviewer downgrades the rating.
review_title: null

Meta-review: A multi-scale hierarchical variational autoencoder based technique is developed for unsupervised image denoising and artefact removal. The method is shown to achieve state of the art performance on several datasets. Further, the multi-scale latent representation leads to an interpretable visualization of the denoising process.

The reviewers unanimously recommend acceptance.
Decision: Accept (Spotlight)