Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a multi-resolution framework for U-Net architecture, which provides some insights into the architecture: (1) it can relate UNets as diffusion processes (2) average pooling regularizes it by implicitly learning Haar wavelet basis representation of target data. The authors analyze Hierarchical Gaussian VAE (HVAE) by applying the proposed framework to the model, which can explain the instability of sampling in HVAE and helps reduce parameter redundancies.

strengths_and_weaknesses: ### Strong points
Although why UNet has been successful in many applications, its regularization effect is poorly understood ever. The framework can naturally explain the instability issue and parameter redundancies of HVAE. The proposed framework is not only elegant but also insightful. In addition, experimental results are overall convincing and support the theory part.
The presentation of the results is correct and easy to follow.

### Weak points
- My only issue is the connection between the insight into sampling instabilities and empirical observation in Sect. 3.3 is unclear (please see my question).
- It is not easy to refer to corresponding appendix parts (because appendix. A is long and the numbering for subsection is not used).
questions: - Sampling instabilities on multi-modal datasets such as CIFAR10 and ImageNet are really due to the instability from the hierarchical property? VAE w/o UNet structure also usually suffers from poor sampling performance on such vision datasets.
limitations: Limitations are addressed, no negative societal impact observed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work tries to make an abstraction of the UNet (through the multi-resolution framework). Based on this abstraction, this work analyzes properties of UNet, revealing a regularisation property of UNet, that is implicitly learning a Haar wavelet basis representation. This work further identifies HVAE (based on UNet) as an Euler discretisation of multi-resolution diffusion sum processes. This paper also proposes a weight sharing of HVAE, which reduces the number of parameters.
strengths_and_weaknesses: Overall, the analysis framework of multi-resolution is original from my perspective, but I preserve some questions on this framework. The paper builds upon heavy notations with less explanation. Therefore, I think the writting quality and clarity can be improved. Besides, it seems that the UNet used in practice has a gap between the UNet discussed in this work, and this gap is not discussed clearly.

Below are my concerns and questions in detail:

1. The multi-resolution framework use finite dimensional projection. So I expect a projection is defined on a discrete space, such as $Z^2$. However, the partition is w.r.t. functions with domain $R^2$ in Definition 1.

2. In definition 1, why a negative $j$ corresponds to zooming in? Suppose $f(\cdot) \in V_j$, where $j$ is negative, and the input space $[0, 2^j]^2$. Such an input space is scaled to $[0, 1]^2$ for $f(2^j \cdot)$ to keep the same graph. Therefore, a negative $j$ has a smaller graph (since $2^j < 1$) compared to $j=0$. From my perspective, a smaller graph corresponds to zooming out.

3. In definition 2, the author mentions the multi-resolution partition of $L^2(X)$, where $X$ is a compact set. But the partition on a compact set is not defined. It seems that directly applying definition 1 to compact set will have some problems, since A4 implies $X$ should be unbounded to allow a translation transformation. However, a compact is bounded. Therefore, I think it is necessary to define the multi-resolution partition for a compact set $X$ in some place.

4. This paper does not give an introduction to the UNet. This makes it difficult to link the UNet used in practice to the one discussed in this paper. The author could introduce the UNet used in practice somewhere and discuss how the practical UNet is abstracted in this paper.

5. The practical UNet has different number of channels in different layers. Does this work also support the different number of channels?

6. An important feature of UNet is the skip connection across different layers. It seems that the framework in this paper ignores this feature.

7. This paper relies on heavy notations. These notations should be explained more. For example, the author could explain what the practical counterpart is when introducing a notation.

8. When the author refers to the Appendix, the author could refer to a more detailed position.

9. This paper relies on UNet, HVAE and multi-resolution partition. It would be better to add a background section to introduce these important prior works. This would make this paper more readable.
questions:  
limitations:
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a multi-resolution framework of U-net. Theoretical results show the regularisation property of a U-Net with average pooling as implicitly learning a Haar wavelet representation of the data. This framework is also used to identify parameter redundancies and instabilities of HVAEs and verify them via experiments. 
strengths_and_weaknesses: Pros:

(1) This is a paper with theoritical insights that help to understand the regularization properties of U-net. 

(2) With the help of the proposed theories, the parameter efficiencies of HVAEs are improved as shown by extensive experiments on 4 benchmarks.

Cons:

(1) As admitted in the paper, the author only applies the framework to a single VAE model HVAEs so the theory is not guaranteed to generalize. 

(2) Though the theory is insightful and novel, the practical technique used to improve parameter efficiencies is just parameter sharing, which is well studied. 
questions: No.
limitations: (1) Only test on a single VAE model HVAEs.
review_title: null

Meta-review: metareview: The paper makes an observation that average pooling in U-Nets implicitly learn a Haar wavelet basis representation and build a theory for hierarchical VAEs (HVAEs) on top of it. The proposed interpretation of HVAEs lead to modification to HVAEs that reduce the number of parameters and improve stability. I think the Haar wavelet basis representation is somewhat obvious but the analysis of HVAEs look nontrivial. I would recommend accepting this paper with the following suggestions:

I suggest the authors to focus on the HVAE part and improve the presentation. More specifically, the definition of U-Net architectures used in HVAEs should be in the main text. Also make more connection between HVAEs and diffusion models. For example, time information is explicitly handled in diffusion models, whereas this paper suggest that it is handled implicitly in HVAEs. The parameter sharing is common for diffusion models. At the end of the day, the proposed improvement to HVAEs seem to make HVAEs closer to diffusion models.

confidence: Less certain
Decision: Accept