Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors extend an existing method and develop an approach to control each layer’s contribution to the rate independently in the hierarchy VAE. They identify the most general class of inference models to which their proposed method is applicable. In the experiments, they demonstrate that the proposed method better tunes hierarchical VAEs. 
strength_and_weaknesses: Strength:

The paper was well written with good structures. 

Weakness

The paper is a simple extension of existing works, and the significance and novelty of the paper are not strong enough for publication at the moment. The paper could be stronger with additional formal theoretical study. 


clarity,_quality,_novelty_and_reproducibility: The writing of the paper is clear and easy to follow. 

summary_of_the_review: The author could add more theoretical and experimental studies to make the paper stronger. For example, how does the distribution of $\beta$s affect the performance? How many layers of H-VAEs do we need given specific datasets?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a relatively thorough study of the effect of $\beta$ for the case of hierarchical VAEs. Specifically, the authors study the rate-distortion performance of two-layer VAEs with various combinations of $\beta$s and show that different tasks require different settings of $\beta$s.  The paper also presents a theoretical discussion that predicts the empirical results. 
strength_and_weaknesses: **Strengths**
* Very well-written paper with clear motivation, with an interesting generalization of $\beta$ VAEs.
* Thorough analysis and discussion.

**Weaknesses**
* Lack of empirical results for $L > 2$.  While I understand that it may not be feasible to perform a thorough grid search (as was done for the two-layer VAEs in this paper) with deeper VAEs, basing all empirical conclusions on $L=2$ experiments seems a little brittle.  While the theoretical discussions are sensible, it is unclear to me whether the empirical results presented would readily generalize to much larger values of $L$ -- especially given finite capacity models with possibly different optimization dynamics  in practice.


clarity,_quality,_novelty_and_reproducibility: **Clarity & Quality**
* The plots are beautiful, but I had some trouble interpreting them. In particular, identifying the (many) overlapping colored circles was pretty challenging, and required me to zoom in heavily.  Also it may help to present a separate plot for each of the three main tasks considered.
* Writing quality is high.

**Novelty**
* The generalization of $\beta$-VAEs to multiple layers is very sensible and well-motivated, and the perspective to study the impact of $\beta$ values on different tasks was certainly interesting. However, I did not find much novelty in the paper -- the theoretical exposition consists mostly of relatively straightforward generalization of existing bounds, with the remainder of the paper being a thorough empirical study.
summary_of_the_review: Overall, I enjoyed reading the paper as it is very well-written and the theoretical discussions conclusions are coherent with the empirical findings.  I particularly value the fact that the paper explicitly tries to provide useful guidelines for practitioners who may need to tune a hierarchical VAE for a specific task.  However as mentioned above, the paper does not present a novel idea, and is somewhat incomplete in experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper clarifies that the different applications in VAEs require different trade-offs in the IB formulation; that is, the single regularization coefficient of rates in IB cannot achieve the best performance in three application domains. Motivated by this analysis, the authors proposed to extend the existing HVAEs based on the IB formulation and introduce flexible regularization terms in HVAEs. The new model can show better trade-offs under different tasks thanks to this flexibility.
strength_and_weaknesses: # Strength
- This paper explicitly discusses how the three commonly used tasks of VAEs are related to the rates in the IB formulation. Then the authors showed theoretically and empirically that the different rates are preferable for different tasks, which seems missed in existing work.
- The authors extend the IB formulation of VAE to the different levels of regularization of rates in HVAE.
- The authors studied how the rates affect the performance of different tasks by controlling the regularization in detail.

# Weakness
- I am not satisfied with the explanation of how the rates of the representation $Z$ and accuracies in downstream tasks are related to each other. In particular, the authors introduced the bound Eq.11, but I do not think the assumptions of Eq. 11 in Meyen 2016 are satisfied in VAEs.
- Decomposing the rate term in IB and introducing different $\beta$s in HVAEs are natural. Although using different $\beta$s introduces additional flexibility and improves the performance, I do not think it is practical because we must choose $\beta$s, and it is not clear how to tune them.
- As the authors mentioned in Sec 6, it would be better to include the results about how we choose $\beta$s or the results that show the merit of using $\beta_1\neq \beta_2$ not $\beta_1=\beta_2$. In the current numerical experiments, I could not see that advantage.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, although the assumptions of some statements are unclear, such as Eq 11. I think the idea of using different $\beta$ for HVAE is straightforward, not novel. On the other hand, the numerical experiments that show the different tasks require different $\beta$ in Figure 1 are important in the field.
summary_of_the_review: I think the contributions of this work are two folds;
one is the analysis that the different tasks in VAEs require different levels of regularization about rates in IB problems. The other contribution is introducing the new HVAE that has flexible rate regularization.

I think the first contribution is insightful and should be shared in the community. Still, the second contribution seems insufficient since I could not be convinced why using different beta is important numerically or theoretically.
It would be better to include a discussion about how using different betas leads to resolving the problem of the necessity of different rates among different tasks by introducing additional flexibility.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to assign separate beta Lagrange factors to each level in hierarchical VAEs to control the rate for each level separately. They argue that different applications of VAEs have different trade-offs for the rate, e.g., classification, generation, and reconstruction. The experiments sweep over values of each beta in 2-level VAEs on MNIST, SVHN, and CIFAR10. They report various metrics for each setting, showing that different values of beta are useful in different settings.
strength_and_weaknesses: **Strengths**
- **Clear presentation.** The authors clearly lay out the setting (hierarchical VAEs), three application domains (classification, generation, and reconstruction), and the main contribution (evaluating different beta trade-offs). Supporting diagrams in Figure 2 show the setup considered by the authors. Mathematical equations provide clear and consistent definitions of the quantities under consideration without introducing superfluous notation. And the experiment section is laid out with separate sections and plots for each setting.
- **Great plots / empirical analysis.** Beyond the clear presentation, the empirical findings are presented extremely well. It’s apparent that a significant amount of time went into creating these plots. The plots present findings with a mix of color-coding, shapes, and 2D and 3D visualizations. The plots do a great job of presenting the distinction between VAE vs. beta-VAE vs. having multiple beta hyperparameters.
- **Somewhat practical/relevant.** The paper tackles a rather practical and straightforward question of whether adjusting the rate at each hierarchical level in VAEs significantly impacts performance on various applications. This is a well-defined question, which the authors directly address by sweeping over hyperparameters (betas) and evaluating performance using relevant metrics. While I’m not sure whether a reader would come away from the paper with clear guidance on how to adjust these hyperparameters for any given architecture, the main message—that these hyperparameters can and should be separately adjusted—is clear.

**Weaknesses**
- **Limited novelty.** The primary novel aspect of the paper is in proposing to separately adjust different beta hyperparameters for each level in hierarchical VAEs, showing that this is relevant for various applications. I did not find this to be particularly novel, as the paper boils down to performing a hyperparameter sweep over multiple (in this case, two) dimensions rather than one. Likewise, the authors do not propose any form of method for automatically adjusting these hyperparameters for a given application. Rather, the paper presents a set of hyperparameter sweeps, showing that different regions are relevant for different applications. It’s unclear how these findings can readily be extended to hierarchies of arbitrary depth beyond simply performing hyperparameter sweeps in these larger spaces. The primary way of fixing this weakness would be to demonstrate some new capability afforded by this technique and/or some new method for targeting a particular application. For instance, rather than separately sweeping over both beta hyperparameters, one might consider using the constrained optimization formulation presented by Rezende & Viola, 2018, in which a target rate (or distortion) is satisfied.
- **Simple-ish models and datasets, which may not be relevant for practical settings.** The authors explore their empirical question in the setting of somewhat small 2-level hierarchical VAEs using MNIST, SVHN, and CIFAR10, all of which are fairly small image datasets. While I entirely understand the need to manage the complexity of the empirical setting to make such analyses tractable, it does raise questions of whether these analyses offer practical guidance for a practitioner. I would have a hard time believing that any of the models and datasets in this paper would be a go-to choice for any of the applications (classification, generation, reconstruction) considered. I understand that the authors are not attempting to obtain SOTA performance in any of these domains. However, to address this weakness, it may be useful to target one particular application more directly, e.g., compression/reconstruction, and show that these findings are relevant with full-scale models and datasets.
- **Framing may not be fully general/applicable.** The authors consider a stripped-down version of hierarchical VAEs, in which the generative process is a strict Markov chain. However, such models are not typically considered within the literature. Starting with the IAF (Kingma et al., 2016) hierarchical VAE architecture, most hierarchical models have introduced deterministic connections between higher-level variables and lower-levels (including the conditional likelihood). This is true of modern large-scale hierarchical VAEs like NVAE (Vahdat & Kautz, 2020) and VDVAE (Child, 2021). These deterministic connections allow gradients to flow directly from the reconstruction to the higher-level variables, which undoubtedly alters the gradients and learning dynamics. I understand that the authors restricted their analysis to Markov hierarchies for practical purposes, but the findings may be moot if they are not relevant in the more general setting of non-Markov hierarchies. Fixing this weakness would involve performing analyses in this more general setting.
clarity,_quality,_novelty_and_reproducibility: Clarity - The paper is clear. Everything, from the setup, diagrams, and presentation of empirical findings, focuses around a clear message. Equations and other mathematical expressions are clearly defined. Plots are shown with clear color-coding, shapes, and labels. The authors separate various empirical findings into distinct sections, figures, and tables.

Quality - The paper appears to be high-quality. The authors take a known setting (hierarchical VAEs) and perform a set of analyses on benchmark datasets using accepted metrics for tracking performance on downstream tasks (PSNR, Inception Score, and classification accuracy). Special cases (i.e., VAEs and beta-VAEs) are shown within empirical findings. The analysis appears to be fairly thorough.

Novelty - The novelty of the paper is one of its weaker aspects. The primary novel contribution of the paper is in demonstrating that having separate beta hyperparameters for each level in hierarchical VAEs is beneficial when using these models for various applications. The main findings of the paper are in sweeping over two hyperparameter dimension rather than one. Admittedly, there is some novelty in this approach, as Lagrange factors have not been explored extensively in the setting of hierarchical models.

Reproducibility - The paper is likely reproducible. The results show clear trends across multiple datasets while varying hyperparameters. That is, many models were trained in order to present the results in the paper, and the trends appear to be fairly smooth and consistent. The authors also provided accompanying code with their submission, which will assist with reproducing the results.
summary_of_the_review: While the paper does not present any radically novel techniques or entirely surprising findings, the authors tackle a well-defined and practical question that, to the best of my knowledge, has not been addressed previously in the literature. The clarity of the presentation makes the paper approachable by readers with varying levels of background knowledge, and the empirical findings demonstrate a high degree of rigor (i.e., performing a large-scale sweep, multiple datasets, multiple metrics). For these reasons, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is about a bit-distortion rate analysis of hierarchical (beta)VAEs and using control of the bit-distortion rate for controlling the behavior and interpret each layer of the model.

The reviewers like the paper although they appreciate the experiments more than the novelty. Some also complain about that the hierarchy is maximum two stochastic layers. 
justification_for_why_not_higher_score: Lacking novelty in the theoretical part and hierarchies with more stochastic layers should be considered.  
justification_for_why_not_lower_score: One could argue for reject, but especially the empirical investigation is of a high quality that will be useful to see at ICLR.
Decision: Accept: poster