Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces Multi-Rate VAE, which is a hyper-network that outputs the parameters of a $\beta$-VAE network, when given a value of $\beta$. The rate-distortion curve created using the the networks computed from this hyper-network matches the one that is obtained by repeatedly training $\beta$-VAEs for different values of $\beta$. This hyper-network also has similar number of parameters as that of the $\beta$-VAE network it outputs, and has similar training time. The construction of the hyper-network is theoretically principled, because the paper proves that the hyper-network can output the optimal $\beta$-VAE when the VAE is linear.
strength_and_weaknesses: Strengths:
1. The proposed hyper-network construction seems to be theoretically principled. This is because for linear VAEs, the paper provides a theorem that proves that indeed the optimized hyper-network can output parameters for the optimal linear $\beta$-VAE.
2. The hyper-network is of practical importance since it bypasses the need to retrain $\beta$-VAEs for different values of $\beta$, while fine tuning for optimal $\beta$.
3. The hyper-network has similar size and training time as the $\beta$-VAE it outputs.
4. The results have been empirically verified on MNIST, OMNIGLOT, CIFAR10 and CelebA datasets, where it is shown that the rate-distortion curve created using the the networks computed from this hyper-network matches the one that is obtained by repeatedly training $\beta$-VAEs for different values of $\beta$.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the idea is novel.
summary_of_the_review: I think the idea of hyper-networks outputting $\beta$-VAE for different values of $\beta$ without the need for retraining from scratch for each $\beta$, is of practical importance. Furthermore, the proposed framework is both theoretically principled and empirically verified to work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes MR-VAE, an algorithm to train a VAE that corresponds to an interval of $\beta$ values in a single training run.  The proposed method trains a hypernetwork that modifies the weights of a regular VAE at inference time for a given value of $\beta$, thus eliminating the need to retrain a VAE for each value of $\beta$.  The choice of hyperparameters for the hypernetwork is based on the analysis on linear VAEs, and the empirical results demonstrate that the method generalizes well to modern VAE architectures.
strength_and_weaknesses: **Strengths**
* Very well-written paper with a clear motivation.
* Strong empirical results on diverse datasets.  In particular, the paper addresses the important questions that naturally arise: (1) does MR-VAE introduce performance degradation?  (2) does MR-VAE generalize to larger, complicated architectures?  (3) how sensitive is MR-VAE to the choice of its (only) additional hyperparameter $(a, b)$?  The experiments convincingly and positively answer all these questions.

**Weaknesses**
* I did not find any particular weakness.
clarity,_quality,_novelty_and_reproducibility: **Clarify & Quality**
The paper is very well-written, and the quality of the exposition is high.

**Novelty**
While the question studied in the paper is not very novel, the paper provides a clean algorithm with good justification that also works well in practice.  The analysis of optimal response functions on linear VAEs was insightful.
summary_of_the_review: This paper provides an answer to a somewhat obvious but practically important question of whether it's possible to train a VAE that can operate on a continuum of rate-distortion points.  The presentation is clear, with clear experimental results to back up the usability (i.e. R-D performance, computational/memory cost, stability to hyperparameters).  I believe this is a valuable contribution to the field.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers $\beta$-VAEs and treats the weight $\beta$ of the KL-divergence term as a hyper-parameter. The authors develop a hypernetwork model which learns the optimal VAE parameters (i.e. weights of the encoder and decoder) simultaneously for all $\beta$ values within a pre-specified range. The approach is experimentally tested on different datasets like MNIST, SVHN and others as well as for different VAE models like standard VAEs and hierarchical VAEs.
strength_and_weaknesses: Paper strengths:
The authors start from simple log-linear Gaussian VAEs and prove theoretically that a simple affine transform of the encoder/decoder weights parametrised by $\beta$ provides the dependence of the optimal wights on the hyper-parameter $\beta$. They conjecture that the hyper-network for complex, non-linear Gaussian VAEs is obtained by applying such affine transforms of the base weights in each layer of the encoder/decoder pair. The experiments compare achievable data likelihoods for VAEs (with  fixed architecture), when learned with fixed betas (aka response curve), with the one obtained from learning the hypernetwork. It is striking to see how good the corresponding rate curves are matched by the hypernetwork for different VAE variants.

Paper weaknesses:
There are a few conceptual choices and issues which remain unclear for me:
I was not able to fully follow the proofs given in the supplementary material for the log-linear Gaussian VAEs:
- Step from (18) to (19,20): Taking the expectation over the data and maximising it w.r.t. to $\mu$ will give a $\mu_{MLE}$ that depends on the other model parameters. It remains unclear how to derive the gradients of the objective w.r.t. the latter.
- Theorem 1: It remains unclear to me, why it is necessary to represent the parameter matrices of the model as products of 
matrices? Why are then the affine transforms applied directly to the VAE weights for the non-linear models in (8,9)?

It remains unclear to me, why applying the affine transform in all layers of the encoder/decoder of non-linear VAEs is sufficient for obtaining the required hypernetwork. Is this may be related to the known rescaling properties of ReLU networks? Would this work also for other non-linearities?. And more general, can this approach be used also for non-Gaussian VAEs with discrete latent variables?
clarity,_quality,_novelty_and_reproducibility: The paper is well organised and clearly written in most parts. However, the technical material given in the supplementary material is indispensable for understanding the approach. Even considering this, there are quite some open questions remaining (see above).
summary_of_the_review: The paper proposes an interesting hypernetwork approach for estimating the optimal weights of $\beta$-VAEs simultaneously for all $\beta$ values within a pre-specified range for given encoder/decoder architectures. The technical part (especially in  the supplementary material) is sometimes hard to follow and there are a few questions and issues remaining open. My current recommendation is a lower bound. The final recommendation will depend on the clarification of these questions in the discussion phase.

After rebuttal discussion: The clarifications provided by the authors answered almost all my questions and are convincing. I am raising my recommendation to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method to learn a hypernetwork for Beta VAEs - a network that can output the parameters of a VAE for all possible values of beta. The paper received very strong reviews which praised its very well-written, high-quality exposition, its theoretical justification and its strong experimental results. While there were a very small number of concerns by the reviewers, the authors have adequately addressed all of them, with the paper having no outstanding concerns from the reviewers. This paper is further of great interest to the community as Beta VAEs are widely used and the paper provides a principled approach to get rid of a fundamental difficulty of Beta VAEs, i.e. that one has to retrain the model for all different values of beta. I hence strongly recommend to accept this paper.
justification_for_why_not_lower_score: The paper had very strong reviews, with no concerns. At the same time, the paper is highly relevant to a large part of the community.
Decision: Accept: notable-top-5%