Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper generalizes a Markov wavelet conditional probability model developed based on the renormalization group theory. The model is parameterized using conditional CNNs with local receptive fields, which enables stationary Markov properties. The method is extended to handle several face image reconstruction tasks, e.g. image denoising and synthesis. The results show that the model can capture image features with a small receptive field,
strength_and_weaknesses: Strength:

- This work is well-motivated.
- This paper addresses an important problem in learning prior probability models of images. 
- The paper seems to be solid.
- Two application scenarios are provided with technical details.

Cons:
- The image reconstruction results can be further improved.
- There is no comparison with the state-of-the-art methods on the two mentioned tasks.
clarity,_quality,_novelty_and_reproducibility: The paper has its novelty. The quality of the paper presentation is very good. The paper is reproducible.
summary_of_the_review: This is a well-written paper with solid theoretical content. The experiments are insufficient. Given the theoretical nature of the paper, this can be somehow tolerated.

The authors' responses have well addressed by concerns. I have raised the empirical novelty and significant score and also raised the final recommendation to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a multi-scale conditional probability model to reconstruct and denoise images. Specifically, the scales come from wavelet transform, and they generalize Markov conditional models by parametrizing the conditional gradients with a CNN with a local receptive field. They show how their method performs better than Markov methods conditioned and performed in the pixel domain. They study the effect of the receptive field. They use a previously proposed method to generate new images, i.e., to perform gradient ascent on the log probability of the learned CNNs to draw samples from the learned image distributions. They show applications in denoising, synthesis, and super-resolution.

strength_and_weaknesses: The paper shows the application of their method in several image tasks (generation, denoising, super-resolution).

The paper can improve on clarity. It is not clear what their contribution is from a general perspective on the performed tasks (e.g., denoising and super-resolution, generation). Please explain the advantages of your method compared to other general frameworks used for super-resolution, denoising, etc. It is not shown how their framework performed against multiple baselines. There seems to be only one method that they compared with which is the vanilla version of their Markov method. Here are detailed comments.

Lack of sufficient baselines:
- By using local convolutional filters and learning locally, their method alleviates the curse of dimensionality; However, how is their method compared to other popular methods in the literature in terms of performance, computational complexity, runtime, etc? (Perhaps, a quantitative comparison between their method and the cited methods on the last paragraph of page 8 is suggested)
- It is nice to see (in Figure 3) that their method outperforms a similar approach of Markov probability in the pixel domain. However, more experiments with non-Markov baselines should be included. How is the performance of this method for example compared to generic deep learning (e.g., DnCNN bias-free) for denoising, super-resolution, etc?

The literature review needs proper citations. Here are some examples in the introduction. Please provide citations to
- the second line in the intro talks about dimensionality.
- the fourth line in the intro refers to traditional methods.
- the last line of the first paragraph in the intro refers to global models.

It is hard to see the relationship between the conditional CNN and (3). Can the authors explain?

A thorough experimental analysis would be nice to add to the characterization of their method: how the performance changes as the number of wavelets, and the depth of scale change.

It is not clear if the models are trained on the same noise level compared to the test noise or a range. Please clarify and add such information.

How is the performance on natural images?

It is not clear how the right image in Figure 6 is generated. Please elaborate on "CNN denoiser with TF smaller than the image".

Minor comments

- Moving Figure 2 to the second page will help to understand the - model.

- Why the first row of Figure 4 is smaller? Please fix.

- Elaborate on the statement after Figure 4: "... less than three percent of the coefficients".
clarity,_quality,_novelty_and_reproducibility: The paper needs improvement in clarity and organization. For example, it would be nice if in terms of bullet points the authors can indicate the contributions of this paper and how it differs from prior works, and how it benefits the ICLR community. The method is novel in how it combines ideas to scale up wavelets. 
summary_of_the_review: The paper proposes a method to generalize multiscale wavelets and markov models to learn conditional distributions of wavelets to recover images. The method needs a thorough experimental study to show its performance against other learning frameworks and to clearly explain how its method differs from prior works and its advantages. Hence, the paper in its current form is not recommended for acceptance.


-----------

 Given the authors' responses, I have raised my score to 8.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors propose a local conditional probability model for images in multiscale representations.
In this model, the probability of the high-frequency (detail) components is conditioned to the low-frequency (low-resolution) component, and these conditional relations are spatially local. The probability of the final low-resolution residual is modeled globally. 

Both the conditional probabilities and the probability of the low-resolution residual are learnt by CNN denoisers using the relation between the denoiser errors and the gradient of the probability of the signal.
 
The local (Markov) structure of the conditional probabilities means that the parameters (receptive fields) of the CNNs are restricted to the local neighborhood and hence the conditional part of the model is relatively low dimensional. This assumes that the conditional relations at each scale are essentially stationary (or translation invariant). 

However, the receptive fields of the CNN to model the probability of the low-pass residual have to be the same size as the residual in order to capture its non-stationary nature. The requirement of dense connection for the low-pass residual is the main limitation of the proposed model in terms of dimensionality.
strength_and_weaknesses: STRENGTHS:

The paper should be accepted because it builds an explicit image probability model as opposed to implicit (non-accessible) models of autoencoders, GANs, normalizing flows, or diffusion methods. These results are an interesting connection between the efforts of (1) linking the gradient of the PDFs with optimal denoising [Miyasawa61, Raphan&Simoncelli11, Vincent11], and (2) establishing conditional relations between low-frequency and high-frequency aspects in spatial neighborhoods of natural images [Buccigrossi&Simoncelli99, Marchand et al.22].
 
WEAKNESSES:

* The model reminds me of other recent conditional models such as the PixelCNN [van den Oord et al.16, Salimans et al.17].
I guess the key difference is the formulation in a multiscale representation instead of in the input spatial domain.
The authors already compare with CNN models in the spatial domain. I think it would be interesting for the reader to see some example with more sophisticated conditional models in the spatial domain such as the PixelCNN. Of course, in this comparison the size of the images should be big enough for a couple of scales but small enough for proper training of pixelCNN.

* The authors acknowledge the main limitation of their proposal in the discussion "the non-stationary capabilities of the full model arise primarily from the terminal low-pass CNN, which uses global receptivie fields". I think this should be acknowledged from the very beginning (in the abstract and/or intro) because the interesting conditional relations (the core of the proposal) always need "a seed to grow" which is modeled in a conventional way.

* The dimensionality of the model is small because the Markov structure is spatially local (the interesting theorem 1).
Could we get extra reductions in the parameters using locality in the orientation and in the scale in redundant wavelets?
Note that nonorthonormal wavelets could also be applied since it would only imply the consideration of the determinant |W| in Eq. 1, right?. This would mean the extension to orientation and scale of the locality concept applied in space.

REFERENCES:

* van den Oord et al. 16a, Salimans et al.17 Conditional Image Generation with PixelCNN Decoders NeurIPS 2016 arxiv 1606.05328

* van den Oord et al. 16b Pixel Recurrent Neural Networks ICML 2016 arxiv 1601.06759

* Salimans et al.17 PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications ICLR arxiv 1701.05517

MINOR COMMENTS:

* You could cite Marchand et al. 22 the first time you mention that the conditional can be modeled with conditional Gibbs energies (Eq. 2)

* The first paragraph of section 5 mentiones the super-resolution problem and it does not mention the synthesis problem, but then, the next paragraphs alternate between one problem and the other. The two problems should be mentioned at the begining to avoid confusion.

* The proof of Eq. 5 (appendix B) already was done in Raphan&Simoncelli 11, right?. Why not just cite that? 
clarity,_quality,_novelty_and_reproducibility: Clarity: the writing/math is clear and interesting
Quality and novelty: high because it makes explicit models that are regularly hidden
Reproducibility: I missed links to code
summary_of_the_review: The paper should be accepted because it builds an explicit image probability model as opposed to implicit (non-accessible) models of autoencoders, GANs, normalizing flows, or diffusion methods. These results are an interesting connection between the efforts of (1) linking the gradient of the PDFs with optimal denoising [Miyasawa61, Raphan&Simoncelli11, Vincent11], and (2) establishing conditional relations between low-frequency and high-frequency aspects in spatial neighborhoods of natural images [Buccigrossi&Simoncelli99, Marchand et al.22]. Experimental results successfully show the good performance of the proposed model. 

However, some elaboration would be appreciated on: (1) advantages over alternative conditional models (such as PixelCNN) in experiments, (2) acknowledgement of the central role of the low-pass component (not a core of the proposal), (3) possible extensions of the locality concept in space to orientation and scale.

My current score is 7, but I will be happy to raise to 8 or 9 it if the authors address the suggested points.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission proposes an image model based on a wavelet framework.  This is quite similar to other works that attempt to understand the behavior of NNs in the frequency domain using multiple frequency transforms.  The main thing that sets it apart is the imposition of a Markov condition that severely theoretically restricts the model over previous families of solutions.  Quite interestingly, this results in quite performant models at a tiny fraction of the number of parameters.  This is an interesting family of models to study and refine in its own right, but it also gives insights into the performance of other coarse to fine strategies, which may be viewed as more heuristic or pragmatic.  The exposition is clear and technically competent, and the contributions of the paper were unanimously appreciated by the reviewers with Reviewer KhB3 recommending that it be highlighted in the conference.

The main weakness of the submission is the dismissal of the authors of multiple requests for additional empirical comparisons to SOTA models.  The baseline model is essentially an ablated version of the proposed model, and does not remotely reach the theoretical capacity of the current SOTA, which was requested by all reviewers.  The author response was
> The reviewer mentions the lack of comparison with state-of-the-art methods on denoising and super-resolution. This is intentional because the main goal was to demonstrate the existence of Markov conditional properties across scales but not in the pixel domain. For doing so, we used small size CNNs which have about 0.001 of parameters of SoTA architectures....

The point is somewhat taken, but that does not preclude including that information.  While the paper is interesting, there is still clearly a gap between the models evaluated in the paper and the existing literature.  A scatter plot of model size vs. accuracy could be a way to address this, and certainly a paragraph in the Discussion section prominently highlighting these limitations should be included.  As it stands, the authors are indicating that they intentionally left additional benchmarking and scaling of their family of models to future work, which is not a very compelling answer.
justification_for_why_not_higher_score: The main weakness of the submission, which was pointed out by *all* reviewers, and I agree, was disagreed with by the authors: that the method does not sufficiently compare to SOTA methods empirically.  The authors argue that other models have orders of magnitude more parameters and that it would be confusing to make the comparison.  I think the authors need to trust the readers more.  I understand the frustration of reviewers not getting the point and viewing everything as a benchmark competition where only accuracy matters, but clearly that did not happen in this review process.  The authors should trust that, with an appropriate presentation, e.g. some notion of accuracy vs. model size as a scatter plot, the ICLR readership is capable of discerning this fact.  The reader is now left with the additional task of either providing their own benchmarking, or in searching across different papers with the understanding that the evaluation setup may not be consistent.  Why???
justification_for_why_not_lower_score: Due to the above concern, part of me even wanted to reject the paper for lack of benchmarking - a common concern across *all* reviewers.  The authors response is not really satisfactory.  As such, I'm OK with this being bumped down.  On the other hand, the reviewer scores are quite high, and the main point of the paper is well conveyed and demonstrated - that a frequency domain Markov property is a surprisingly effective image model, with eventually lessons for design and understanding of SOTA models as well as being a very interesting model in its own right.
Decision: Accept: notable-top-25%