High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve

## Abstract

Abstract In the Variational Autoencoder (VAE), the variational posterior often aligns closely with the prior, which is known as posterior collapse and hinders the quality of representation learning.
To mitigate this problem, an adjustable hyperparameter beta has been introduced in the VAE.
This paper presents a closed-form expression to assess the relationship between the beta in VAE, the dataset size, the posterior collapse, and the rate-distortion curve by analyzing a minimal VAE in a high-dimensional limit.
These results clarify that a long plateau in the generalization error emerges with a relatively larger beta. As the beta increases, the length of the plateau extends and then becomes infinite beyond a certain beta threshold.
This implies that the choice of beta, unlike the usual regularization parameters, can induce posterior collapse regardless of the dataset size. Thus, beta is a risky parameter that requires careful tuning.
Furthermore, considering the dataset-size dependence on the rate-distortion curve, a relatively large dataset is required to obtain a rate-distortion curve with high rates. Extensive numerical experiments support our analysis.

## 1 INTRODUCTION

Deep latent variable models are generative models that convert latent variables generated from a prior distribution into samples that closely resemble the data through a neural network. Variational autoencoders (VAE) (Kingma and Welling,, ; Rezende et al.,, ), one of the deep latent variable models, have been applied in various fields such as image generation (Child,, ; Vahdat and Kautz,, ), text generation (Bowman et al.,, ), music generation (Roberts et al.,, ), clustering (Jiang et al.,, ), dimensionality reduction (Akkari et al.,, ), data augmentation (Norouzi et al.,, ), and anomaly detection (An and Cho,, ; Park et al.,, ).
However, direct likelihood maximization is intractable due to the marginalization of latent variables. Consequently, VAE often employs the evidence lower bounds (ELBO), which serve as computable lower bounds on the log-likelihood.

From an information-theoretical point of view, the ELBO has been interpreted in various studies (Alemi et al.,, ; Huang et al.,, ; Nakagawa et al.,, ), which emphasize that the ELBO can be decomposed into two terms representing a trade-off relationship. Drawing an analogy from the rate-distortion theory, these terms can be likened to ‘rate’ and ‘distortion’(Alemi et al.,, ). Furthermore, these studies suggest that during learning with ELBO, the variational posterior for latent variables tends to align with their prior, hindering effective learning of data representation. This phenomenon is commonly referred to as ‘posterior collapse’.

To address the posterior collapse, an additional parameter, denoted as $\beta_{\mathrm{VAE}}$, is introduced to control the trade-off between rate and distortion (Higgins et al.,, ); While models with a small $\beta_{\mathrm{VAE}}$ effectively reconstruct data points, i.e., have low distortion, they may generate inauthentic data due to significant mismatches between the variational posteriors and their priors (Alemi et al.,, ).
Conversely, models with a large $\beta_{\mathrm{VAE}}$ closely align their variational distributions with the prior, i.e., it has a low rate, but may ignore the encoding critical information. Thus, careful tuning of $\beta_{\mathrm{VAE}}$ in beta-VAEs is important for various applications (Kohl et al.,, ; Castrejon et al.,, ).
Moreover, $\beta_{\mathrm{VAE}}$
is intricately linked to disentanglement quality, generalization property, data compression, and posterior collapse.
However, theoretical understanding of the relationship between $\beta_{\mathrm{VAE}}$, the posterior collapse, and the rate-distortion curve has remained limited. Understanding these relationships is essential for practical application and further improvements.

In this study, we clarify the relationship using a minimal model, referred to as Linear VAE (Lucas et al.,, ), which captures the essence of beta-VAEs.
Our results show that $\beta_{\mathrm{VAE}}$ significantly affects the generalization performance and the rate-distortion curve.
In particular, depending on $\beta_{\mathrm{VAE}}$, the generalization performance becomes poor even when provided with abundant data, raising substantial concerns for real-world applications.

#### Contributions.

In our paper, we advance the theory concerning the relationship between the posterior collapse, the rate-distortion curve, and the dataset size. Throughout the manuscript, we consider a high-dimensional limit where the number of training data $P$ and the dimension $N$ are large ($N,P\to\infty$) while remaining comparable, i.e., $\alpha\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}N/P=\Theta(1)$.
Our main contributions are:

- •
A sharp closed-form formula is derived for the generalization properties and the measure of posterior collapse as a function of sample complexity $\alpha$ and $\beta_{\mathrm{VAE}}$.
- •
A phenomenon is observed where the generalization error has a peak at a certain sample complexity for a small $\beta_{\mathrm{VAE}}$.
As $\beta_{\mathrm{VAE}}$ increases, the peak continuously decreases, which is remarkably similar to the well-known interpolation peak in supervised regression problems for the regularization parameter (Opper and Kinzel,, ; Mignacco et al.,, ; Hastie et al.,, ).
- •
Our analysis shows that a long plateau in the generalization error exists with respect to sample complexity, $\alpha$ for a large $\beta_{\mathrm{VAE}}$. As $\beta_{\mathrm{VAE}}$ increases, the length of the plateau increases, and then eventually the length of the plateau is infinite regardless of the value of the sample complexity.
- •
Using macroscopic variables naturally introduced from replica analysis, we characterize three distinct phases, with the phase diagram pinpointing the boundary of the posterior collapse.
- •
In the infinite dataset-size limit, the rate-distortion curve of VAE, introduced from the analogy of the rate-distortion theory by Alemi et al., (), is confirmed to coincide exactly with that of the Gauss sources (Cover,, ).
In addition, the rate-distortion curve is evaluated as a function of sample complexity $\alpha$, revealing that a relatively large dataset is required to obtain the optimal rate-distortion curve in the high-rate and low-distortion regions.

Note that all the results are based on the replica method, which is quite standard in statistical physics and is generally believed to be exact, although not rigorous. Thus, the results are presented here as claims.

### 1.1 Preliminaries

Here we summarize the notations used in this paper. The expression $\|\cdot\|_{F}$ is Frobenius norm. The notation $\oplus$ denotes the horizontal concatenation of vectors; for vectors ${\bm{a}}\in{\mathbb{R}}^{N}$ and ${\bm{b}}\in{\mathbb{R}}^{M}$, ${\bm{a}}\oplus{\bm{b}}=(a_{1},\ldots,a_{N},b_{1},\ldots,b_{M})\in{\mathbb{R}}^{N+M}$. $I_{N}\in{\mathbb{R}}^{N\times N}$ denotes an $N\times N$ identity matrix, and ${\bm{1}}_{N}$ denotes the vector $(1,\ldots,1)^{\top}\in{\mathbb{R}}^{N}$ and ${\bm{0}}_{N}$ denotes the vector $(0,\ldots,0)^{\top}\in{\mathbb{R}}^{N}$. $Dx=e^{-x^{2}/2}/\sqrt{2\pi}dx$ is the standard Gaussian measure. $D_{\mathrm{KL}}[\cdot\|\cdot]$ is the Kullback–Leibler (KL) divergence. For a matrix $A=(A_{ij})\in{\mathbb{R}}^{N\times M}$ and a vector ${\bm{a}}=(a_{i})\in{\mathbb{R}}^{N}$, we use the shorthand expression $dA\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\prod_{i=1}^{N}\prod_{j=1}^{M}dA_{ij}$ and $d{\bm{a}}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\prod_{i=1}^{N}da_{i}$, respectively.

## 2 BACKGROUND

### 2.1 Variational Autoencoders

A VAE (Kingma and Welling,, ) is a latent generative model. Let $\mathcal{D}=\{x^{\mu}\}_{\mu=1}^{P}$ be training data with $x^{\mu}\in\mathbb{R}^{D}$ and $p_{\mathcal{D}}(x)$ be the empirical distribution of the training dataset.
In practical applications, VAEs are commonly trained using $\beta$-VAE objective (Higgins et al.,, ) defined by

$$ $\mathbb{E}_{p_{\mathcal{D}}}\left[\mathbb{E}_{q_{\phi}}[-\log p_{\theta}({\bm{x}}|{\bm{z}})]+\beta_{\mathrm{VAE}}D_{\mathrm{KL}}[q_{\phi}({\bm{z}}|{\bm{x}})\|p({\bm{z}})]\right]\\ \mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\mathbb{E}_{p_{\mathcal{D}}}[\mathcal{L}(\theta,\phi;{\bm{x}},\beta_{\mathrm{VAE}})],$ (1) $$

where $p({\bm{z}})$ is a prior for the latent variable, and a parameter $\beta_{\mathrm{VAE}}\geq 0$ is introduced to control the trade-off between the first and second terms in Eq. ([1](#S2.E1)).
The distributions $p_{\theta}({\bm{x}}|{\bm{z}})$ characterized by parameters ${\bm{\theta}}$ and $q_{\phi}({\bm{z}}|{\bm{x}})$ by ${\bm{\phi}}$ are often referred to as decoder and encoder, respectively.
Then, VAEs optimize both the encoder parameters ${\bm{\phi}}$ and decoder parameters ${\bm{\theta}}$ by minimizing the objective of Eq. ([1](#S2.E1)).
Note that when $\beta_{\mathrm{VAE}}=0$, the objective becomes a deterministic autoencoder, focusing more on minimizing the first term, which is referred to as reconstruction error.

### 2.2 Information-Theoretic Interpretation of VAEs

Alemi et al., (); Huang et al., (); Park et al., () show that
VAEs can be interpreted through the lens of rate-distortion theory (Davisson,, ; Cover,, ), which has been successfully applied to data compression.
The primary focus has been on the curve where the distortion achieves its minimum value for a given rate, or conversely; see Supplementary Materials [B](#S2a) for a detailed explanation.
From this perspective, the objective of $\beta$-VAEs can be derived from the optimal distortion condition at a given rate.

Drawing an analogy from the rate-distortion theory, Alemi et al., () decompose the $\beta$-VAE objective in Eq. ([1](#S2.E1)) into ‘rate’ $R$ and ‘distortion’ $D$ as follows:

$$ $\displaystyle R({\bm{\phi}})={\mathbb{E}}_{p_{{\mathcal{D}}}}[D_{\mathrm{KL}}[q_{\phi}({\bm{z}}|{\bm{x}})\|p({\bm{z}})]],$ (2) $\displaystyle D({\bm{\theta}},{\bm{\phi}})={\mathbb{E}}_{p_{{\mathcal{D}}}}[\mathbb{E}_{q_{\phi}}[-\log p_{\theta}({\bm{x}}|{\bm{z}})]].$ (3) $$

According to Alemi et al., (), there exists a trade-off between rate and distortion as in the rate-distortion theory, especially when encoder and decoder have infinite capacities. This relationship is derived from the identity:

$$ $H=-{\mathbb{E}}_{p_{{\mathcal{D}}}}[D_{\mathrm{KL}}[q_{\phi}({\bm{z}}|{\bm{x}})\|p_{\theta}({\bm{z}}|{\bm{x}})]]+R({\bm{\phi}})+D({\bm{\theta}},{\bm{\phi}}),$ $$

where $H$ is the data entropy, defined as $H={\mathbb{E}}_{p_{{\mathcal{D}}}}[-\log p_{{\mathcal{D}}}({\bm{x}})]$. From the non-negativity of the KL divergence, it follows that $H\leq R({\bm{\phi}})+D({\bm{\theta}},{\bm{\phi}})$, where the equality is achieved if and only if the variational posterior and the true posterior coincide, i.e., $\forall{\bm{x}},q_{\phi}({\bm{z}}|{\bm{x}})=p_{\theta}({\bm{z}}|{\bm{x}})$.

While this equality holds when the encoder and decoder with infinite capability meet the optimality conditions, the limitation of finite parameters makes this situation unfeasible.
Therefore, we aim to find an approximate optimal distortion at a given rate $R^{\ast}$ by solving the optimization problem given by

$$ $\bar{D}(R^{\ast})=\min_{{\bm{\theta}},{\bm{\phi}}}D({\bm{\phi}},{\bm{\theta}})~{}~{}\mathrm{s.t.}~{}R({\bm{\phi}})\leq R^{\ast}.$ (4) $$

To optimize without explicitly considering $R^{\ast}$, the Lagrangian function with a Lagrange multiplier $\beta_{\mathrm{VAE}}\geq 0$ can be utilized as

$$ $\min_{{\bm{\theta}},{\bm{\phi}}}D({\bm{\theta}},{\bm{\phi}})+\beta_{\mathrm{VAE}}R({\bm{\phi}}).$ $$

This formulation is identical to the $\beta$-VAE objective expressed in Eq. ([1](#S2.E1)). Thus, training various VAEs with different $\beta_{\mathrm{VAE}}$ essentially corresponds to distinct points on the rate-distortion curve.

## 3 SETTING

#### Generative model for the real data.

We consider that real dataset $\mathcal{D}=\{{\bm{x}}^{\mu}\}$ with $\mu=1,\ldots,P$ are drawn according to the generative model given by

$$ ${\bm{x}}^{\mu}=\sqrt{\frac{\rho}{N}}W^{\ast}\bm{c}^{\mu}+\sqrt{\eta}{\bm{n}}^{\mu},$ (5) $$

where $W^{\ast}\in\mathbb{R}^{N\times M^{\ast}}$ is a deterministic unknown $M^{\ast}$ feature matrix; ${\bm{c}}^{\mu}\in\mathbb{R}^{M^{\ast}}$ is a random vector drawn from a standard normal distribution $p({\bm{c}})=\mathcal{N}(0_{M},I_{M})$; ${\bm{n}}^{\mu}$ is a background noise vector whose components are i.i.d from the standard normal distribution $\mathcal{N}({\bm{0}}_{N},I_{N})$; and $\eta\in\mathbb{R}$ and $\rho\in{\mathbb{R}}$ are scalar parameters to control the strength of the noise and signal, respectively.
This generative model, known as the spiked covariance model (Johnstone and Lu,, ), is used in the theoretical studies of Principal Component Analysis (PCA). Even if $W^{\ast}$ is not orthogonal, $W^{\ast}{\bm{c}}^{\mu}$ can be rewritten as $(W^{\ast}R)(R^{-1}{\bm{c}})$, where $R$ is a matrix that orthogonalizes and normalizes the columns of $W^{\ast}$. This can be viewed as an equivalent system where the new feature vector is $R^{-1}{\bm{c}}$. Therefore, we assume,
without loss of generality, that $(W^{\ast})^{\top}W^{\ast}=I_{M}$.

#### Linear VAE model.

In this work, we analyze the following VAE model:

$$ $\displaystyle p_{W}({\bm{x}}|{\bm{z}})=\mathcal{N}\left({\bm{x}};\frac{1}{\sqrt{N}}W{\bm{z}},I_{N}\right),$ (6) $\displaystyle q_{V,D}({\bm{z}}|{\bm{x}})=\mathcal{N}\left({\bm{z}};\frac{1}{\sqrt{N}}V^{\top}{\bm{x}},D\right),$ (7) $\displaystyle p({\bm{z}})=\mathcal{N}({\bm{z}};0_{N},I_{N}),$ (8) $$

where the diagonal covariance matrix $D\in\mathbb{R}^{M\times M}$ is learning parameters, and $W\in\mathbb{R}^{N\times M}$ and $V\in\mathbb{R}^{N\times M}$ are also the learning parameters. This model is called linear VAE (Dai et al.,, ; Lucas et al.,, ; Sicks et al.,, ). We assume a fixed identity covariance matrix $I_{N}$ since this covariance matrix is often used in practice.

#### The training algorithm.

The VAE is trained to learn the generative model by following the optimization problem:

$$ $(\bar{W}(\mathcal{D}),\bar{V}(\mathcal{D}),\bar{D}(\mathcal{D}))\\ =\underset{W,V,D}{\mathrm{argmin}}~{}\mathcal{R}(W,V,D;\mathcal{D},\beta_{\mathrm{VAE}},\lambda),$ (9) $$

where

$$ $\mathcal{R}(W,V,D;\mathcal{D},\beta_{\mathrm{VAE}},\lambda)\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\sum_{\mu=1}^{P}{\mathcal{L}}(W,V,D;{\bm{x}}^{\mu},\beta_{\mathrm{VAE}})\\ +\frac{\lambda}{2}\|W\|_{F}^{2}+\frac{\lambda}{2}\|V\|^{2}_{F}.$ $$

Here ${\mathcal{L}}(W,V,D;{\bm{x}},\beta_{\mathrm{VAE}})$ is defined in Eq. ([1](#S2.E1)), and the last two terms regulate the magnitude of the parameters $W$ and $V$ with $\lambda>0$ being a regularization parameter.

#### The generalization metric.

The VAE can generate a sample ${\bm{x}}\sim p_{W}({\bm{x}})$ through the following procedure: First generate a latent variable ${\bm{z}}\sim p({\bm{z}})$ and then generate a sample ${\bm{x}}\sim p_{W}({\bm{x}}|{\bm{z}})$. Thus, the generalization error $\varepsilon_{g}$ is defined as

$$ $\varepsilon_{g}(W,W^{\ast})=\frac{1}{N}\mathbb{E}_{{\bm{c}}}\left[\left\|\sqrt{\rho}W^{\ast}{\bm{c}}-W{\bm{c}}\right\|^{2}\right],$ (10) $$

where $\mathbb{E}_{{\bm{c}}}[\cdot]$ denotes an average over $p({\bm{c}})=\mathcal{N}(0_{M},I_{M})$. The generalization error, $\varepsilon_{g}$, can measure the extent of signal recovery from the training data.

## 4 REPLICA FORMULA

### 4.1 Replica Formulation

Suppose we define the Boltzmann distribution as

$$ $p(W,V,D;\mathcal{D},\beta)\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\frac{1}{Z(\mathcal{D},\beta)}e^{-\beta\mathcal{R}(W,V,D;\mathcal{D},\beta_{\mathrm{VAE}},\lambda,\tilde{\lambda})}$ (11) $$

where $Z(\mathcal{D},\beta)$ is the normalization constant, known as the partition function in statistical mechanics.
Note that in the limit $\beta\to\infty$, Eq. ([11](#S4.E11)) converges to a distribution concentrated on the $(\bar{W}({\mathcal{D}}),\bar{V}({\mathcal{D}}),\bar{D}({\mathcal{D}}))$. Thus, the expectation of any function $\psi(\bar{W}({\mathcal{D}}),\bar{V}({\mathcal{D}}),\bar{D}({\mathcal{D}}))$ over the dataset can be written by an average over a limiting distribution as

$$ ${\mathbb{E}}_{{\mathcal{D}}}\psi(\bar{W}({\mathcal{D}}),\bar{V}({\mathcal{D}}),\bar{D}({\mathcal{D}}))=\\ \lim_{\beta\to\infty}{\mathbb{E}}_{{\mathcal{D}}}\int dWdVdD\psi(W,V,D)p(W,V,D;\mathcal{D},\beta).$ $$

To evaluate this expression, the idea of the replica method (Mézard et al.,, ; Mezard and Montanari,, ; Blundell,, ; Zdeborová and Krzakala,, ) is to compute the cumulant generating function (also known as the free energy density) as

$$ $f=-\lim_{\beta\to\infty}\frac{1}{\beta N}\mathbb{E}_{\mathcal{D}}\log Z(\mathcal{D},\beta).$ (12) $$

Although Eq. ([12](#S4.E12)) is difficult to calculate straightforwardly, this can be resolved by using the replica method (Mézard et al.,, ; Mezard and Montanari,, ; Blundell,, ), which is based on the following equality

$$ $\mathbb{E}_{\mathcal{D}}\log Z(\mathcal{D},\beta)=\lim_{n\to+0}\frac{\log\mathbb{E}_{\mathcal{D}}Z^{n}(\mathcal{D},\beta)}{n}.$ (13) $$

Instead of handling the cumbersome log expression in Eq. ([12](#S4.E12)) directly, one can calculate the average of the $n$-th power of $Z(\mathcal{D},\beta)$ for $n\in\mathbb{N}$, analytically continues this expression to $n\in\mathbb{R}$, and finally takes the limit $n\to+0$.
Based on this replica trick, it is sufficient to calculate

$$ $\mathbb{E}_{\mathcal{D}}Z^{n}(\mathcal{D},\beta)\\ =\mathbb{E}_{\mathcal{D}}\int\prod_{a=1}^{n}dW^{a}dV^{a}dD^{a}\prod_{a=1}^{n}e^{-\beta\mathcal{R}(W^{a},V^{a},D^{a};\mathcal{D},\beta_{\mathrm{VAE}},\lambda)}$ $$

up to the first order of $n$ to take the $n\to+0$ limit in the right-hand side of Eq. ([13](#S4.E13)).

### 4.2 Outline of The Derivation

This section provides a brief derivation; see the Supplementary Materials [C](#S3a) for comprehensive details. We notice that for every $\mu=1,\ldots,P$,  ${\bm{h}}^{\mu}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}{\bm{u}}^{\mu}\oplus\tilde{{\bm{u}}}^{\mu}\in\mathbb{R}^{2Mn}$ with

$$ $\displaystyle{\bm{u}}^{\mu}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\bigoplus_{a=1}^{n}\frac{1}{\sqrt{N}}(W^{a})^{\top}{\bm{n}}^{\mu},~{}\tilde{{\bm{u}}}^{\mu}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\bigoplus_{a=1}^{n}\frac{1}{\sqrt{N}}(V^{a})^{\top}{\bm{n}}^{\mu}$ $$

follows a Gaussian multivariate distribution with zero mean ${\bm{0}}_{2Mn}$ and covariance matrix

$$ $\displaystyle\mathbb{E}_{{\bm{n}}^{\mu},{\bm{n}}^{\nu}}{\bm{h}}^{\mu}({\bm{h}}^{\nu})^{\top}=\delta_{\mu\nu}\begin{pmatrix}Q&R\\ R&E\end{pmatrix}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\delta_{\mu,\nu}\Sigma,$ $\displaystyle Q=(Q^{ab}),~{}E=(E^{ab}),~{}R=(R^{ab}),$ $\displaystyle Q^{ab}=\frac{1}{N}(W^{a})^{\top}W^{b},~{}E^{ab}=\frac{1}{N}(V^{a})^{\top}V^{b},$ $\displaystyle R^{ab}=\frac{1}{N}(W^{a})^{\top}V^{b}.$ $$

Then, one can rewrite

$$ $\displaystyle\mathbb{E}_{\mathcal{D}}Z^{n}(\mathcal{D},\beta)=\int dQdEdRdmdd[\mathcal{S}\times\mathcal{E}],$ $\displaystyle\mathcal{S}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\int\prod_{a=1}^{n}dW^{a}dV^{a}\prod_{a,b}\prod_{s,l}\delta\left(Q^{ab}_{sl}-\frac{({\bm{w}}^{a}_{s})^{\top}{\bm{w}}^{b}_{l}}{N}\right)$ $\displaystyle\times\delta\left(E^{ab}_{sl}-\frac{({\bm{v}}^{a}_{s})^{\top}{\bm{v}}^{b}_{l}}{N}\right)\times\delta\left(R^{ab}_{sl}-\frac{({\bm{w}}^{a}_{s})^{\top}{\bm{v}}^{b}_{l}}{N}\right)$ $\displaystyle\prod_{a}\prod_{s,l^{\ast}}\delta\left(m^{a}_{sl^{\ast}}-\frac{({\bm{w}}_{s}^{a})^{\top}{\bm{w}}^{\ast}_{l^{\ast}}}{N}\right)\delta\left(d^{a}_{sl^{\ast}}-\frac{({\bm{v}}_{s}^{a})^{\top}{\bm{w}}^{\ast}_{l}}{N}\right)$ $\displaystyle\times e^{-\frac{\beta}{2}\sum_{a}(\lambda\|W\|_{F}^{2}+\lambda\|V\|_{F}^{2})},$ $\displaystyle\mathcal{E}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\int\prod_{a}dD^{a}\prod_{\mu=1}^{P}\mathbb{E}_{{\bm{c}}^{\mu}}\int d{\bm{h}}^{\mu}\mathcal{N}({\bm{h}}^{\mu},{\bm{0}}_{2Mn},\Sigma)$ $\displaystyle\times e^{-\beta\sum_{a}\mathcal{L}(Q,E,R,m,d;{\bm{h}}^{\mu},{\bm{c}}^{\mu},\beta_{\mathrm{VAE}},\lambda,\tilde{\lambda})},$ $$

where ${\bm{w}}^{a}_{l}$ and ${\bm{v}}^{a}_{l}$ are columns vector of $W^{a}$ and $V^{a}$, respectively. We denote $m=(m^{a})$ with $m^{a}=(W^{a})^{\top}W^{\ast}/N$ and $d=(d^{a})$ with $d^{a}=(V^{a})^{\top}W^{\ast}/N$.
Assuming the replica symmetric (RS) ansatz, one reads

$$ $\displaystyle Q^{aa}_{sl}=Q_{sl},~{}E^{aa}_{sl}=E_{sl},~{}R^{aa}_{sl}=R_{sl},$ (14) $\displaystyle m^{a}_{sl^{\ast}}=m_{sl^{\ast}},~{}~{}~{}d^{a}_{sl^{\ast}}=d_{sl^{\ast}},$ (15) $\displaystyle Q^{ab}_{sl}=Q_{sl}-\chi_{sl}/\beta,$ (16) $\displaystyle E_{sl}^{ab}=E_{sl}-\zeta_{sl}/\beta,$ (17) $\displaystyle R_{sl}^{ab}=R_{sl}-\omega_{sl}/\beta,$ (18) $$

where all parameters are denoted as $\Theta\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}(Q,E,R,m,d,\chi,\zeta,\omega)\in{\mathbb{R}}^{M\times(6M+2M^{\ast})}$.
The integration of the replicated weight parameters $\{W_{a},V_{a}\}$ over the entire $\mathbb{R}^{(2M\times N)\times n}$ domain is restricted to a subspace that obeys the constraints detailed in Eqs. ([21](#S3.E21))-([18](#S4.E18)). Utilizing the Fourier representation of the delta function, the integral ${\mathcal{S}}$ is further expressed as

$$ ${\mathcal{S}}=\int d\hat{Q}d\hat{E}d\hat{R}d\hat{m}d\hat{d}~{}e^{\frac{Nn\beta}{2}\left(\mathrm{tr}(Q\hat{Q}+(n-1)\chi\hat{\chi}-n\beta Q\hat{\chi})\right)}\\ \times e^{\frac{Nn\beta}{2}\left(\left(\mathrm{tr}(E\hat{E}+(n-1)\chi\hat{\zeta}-n\beta E\hat{\zeta})\right)+2\left(\mathrm{tr}(R\hat{R}+(n-1)\omega\hat{\omega}-n\beta R\hat{\omega})\right)\right)}\\ \times e^{N\log\int D{\bm{\xi}}_{2M}\left(\int d\tilde{{\bm{w}}}e^{-\beta\left(\frac{1}{2}\tilde{{\bm{w}}}^{\top}(\tilde{Q}+\lambda I_{2M})\tilde{{\bm{w}}}+({\bm{\xi}}_{2M}^{\top}\tilde{\chi}^{1/2}+{\bm{1}}_{2M}^{\top}\tilde{m})\tilde{{\bm{w}}}\right)}\right)^{n}},$ $$

where $\tilde{{\bm{w}}}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}(w_{1},\ldots,w_{M},{v}_{1},\ldots,v_{M})$ and

$$ $\displaystyle\tilde{Q}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\begin{pmatrix}\hat{Q}&\hat{R}\\ \hat{R}&\hat{E}\end{pmatrix},~{}\tilde{\chi}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\begin{pmatrix}\hat{\chi}&\hat{\omega}\\ \hat{\omega}&\hat{\zeta}\end{pmatrix},~{}\tilde{m}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\begin{pmatrix}\hat{m}&{\bm{0}}_{MM^{\ast}}\\ {\bm{0}}_{MM^{\ast}}&\hat{d}\end{pmatrix}.$ $$

Integrating analytically from this expression with respect to $\tilde{{\bm{w}}}$, and under the RS ansatz, the simplified form of ${\mathcal{E}}$ is obtained as

$$ ${\mathcal{E}}=\int D{\bm{\xi}}_{2M\times 2M}\\ \left(\int dVD{\bm{z}}_{2M\times 2M}e^{-\beta{\mathcal{L}}(\Theta;{\bm{z}}_{2M\times 2M},{\bm{\xi}}_{2M\times 2M},\beta_{\mathrm{VAE}})}\right)^{n}.$ $$

Furthermore, integrating with respect to $\Theta$ by the saddle-point approximation justified for large $N$ limit, and taking the limit $\beta\to\infty$ after $n\to+0$ yields eventually the following expression for $f$.

###### Claim 4.1 .

The free energy density $f$ is given by

$$ $f=\underset{\Theta}{\mathrm{extr}}\Bigg{\{}-\frac{\mathrm{tr}(Q\hat{Q}-\chi\hat{\chi})}{2}-\frac{\mathrm{tr}(E\hat{E}-\zeta\hat{\zeta})}{2}\\ -\mathrm{tr}(R\hat{R}-\omega\hat{\omega})+\mathrm{tr}(m\hat{m})+\mathrm{tr}(d\hat{d})\\ -\frac{1}{2}\left(\mathrm{tr}[(\tilde{\chi}^{\frac{1}{2}})^{\top}(\tilde{Q}+\lambda)^{-1}\tilde{\chi}^{\frac{1}{2}}]+{\bm{1}}_{2M}^{\top}\tilde{m}\tilde{Q}^{-1}\tilde{m}^{\top}{\bm{1}}_{2M}\right)\\ +\int D{\bm{\xi}}_{2M\times 2M}\min_{D,{\bm{z}}_{2M\times 2M}}{\mathcal{L}}(\Theta,D;{\bm{z}}_{2M\times 2M},{\bm{\xi}}_{2M\times 2M})\Bigg{\}},$ $$

where $\mathrm{extr}$ means taking an extremum with respect to $\Theta$.

In particular, when $M=M^{\ast}=1$, the free energy density $f$ can be expressed as follows.

###### Claim 4.2 .

When $M=M^{\ast}=1$, the free energy density is given by

$$ $f=\underset{\Theta}{\mathrm{extr}}\Bigg{\{}-\frac{1}{2}(\hat{Q}Q-\chi\hat{\chi})-\frac{1}{2}(\hat{E}E-\zeta\hat{\zeta})\\ -(\hat{R}R-\omega\hat{\omega})+\hat{m}m+\hat{d}d\\ +\frac{(\lambda+\hat{E})(\hat{m}^{2}+\hat{\chi})+(\lambda+\hat{Q})(\hat{d}^{2}+\hat{\zeta})-2\hat{R}(\hat{m}\hat{m}+\hat{\omega})}{2\hat{G}}\\ -\frac{\alpha}{2}\Bigg{(}\frac{(Q-\eta\chi+\beta_{\mathrm{VAE}})(\rho d^{2}+\eta E)}{G}-\frac{\eta\zeta(\rho m^{2}+\eta Q)}{G}\\ +\frac{2(\eta\omega-1)(\rho md+\eta r)}{G}+\beta_{\mathrm{VAE}}\log\frac{Q+\beta_{\mathrm{VAE}}}{\beta_{\mathrm{VAE}}}\Bigg{)}\Bigg{\}},$ $$

where

$$ $\displaystyle\hat{G}=(\lambda+\hat{Q})(\lambda+\hat{E})-\hat{R}^{2},$ $\displaystyle G=\eta\zeta(Q-\eta\chi+\beta_{\mathrm{VAE}})+(\eta\omega-1)^{2}.$ $$

From the gradient of the free energy, the extremum conditions are explicitly given by

$$ $\displaystyle Q=\frac{(\hat{E}+\lambda)\hat{H}}{\hat{G}^{2}}-\frac{\hat{d}^{2}+\hat{\zeta}}{\hat{G}},~{}E=\frac{(\hat{Q}+\lambda)\hat{H}}{\hat{G}^{2}}-\frac{\hat{m}^{2}+\hat{\chi}}{\hat{G}},$ $\displaystyle R=-\frac{\hat{R}\hat{H}}{\hat{G}^{2}}+\frac{\hat{m}\hat{d}+\hat{\omega}}{\hat{G}},$ $\displaystyle m=\frac{\hat{m}(\hat{E}+\lambda)-\hat{d}\hat{R}}{\hat{G}},~{}\tilde{m}=\frac{\hat{m}(\hat{E}+\lambda)-\hat{m}\hat{R}}{\hat{G}},$ $\displaystyle\chi=\frac{\hat{E}+\lambda}{\hat{G}},~{}\tilde{\chi}=\frac{\hat{Q}+\lambda}{\hat{G}},~{}\omega=-\frac{\hat{R}}{\hat{G}},$ $\displaystyle\hat{Q}=\alpha\left(\frac{\beta_{\mathrm{VAE}}}{Q+\beta_{\mathrm{VAE}}}+\frac{\eta Q+d^{2}\rho-\eta^{2}\chi}{G}-\frac{\eta\zeta H}{G^{2}}\right),$ $$

$$ $\displaystyle\hat{E}=\alpha\eta\left(\frac{Q-\eta\chi+\beta_{\mathrm{VAE}}}{G}\right),~{}\hat{R}=\alpha\eta\left(\frac{\eta\omega-1}{G}\right),$ $\displaystyle\hat{\chi}=\alpha\eta\left(\frac{G(\eta E+d^{2}\rho)-\eta\zeta H}{G^{2}}\right),$ $\displaystyle\hat{\zeta}=\alpha\eta\left(\frac{G(\eta Q+m^{2}\rho)-\eta\chi H}{G^{2}}\right),$ $\displaystyle\hat{\omega}=\alpha\eta\left(\frac{-G(\eta R+md\rho)+(\eta\omega-1)H}{G^{2}}\right),$ $\displaystyle\hat{m}=\alpha\rho\left(\frac{\eta m\chi-d(\eta\omega-1)}{G}\right),$ $\displaystyle\hat{d}=-\alpha\rho\left(\frac{d(Q-\eta\chi+\beta_{\mathrm{VAE}})+m(\eta\omega-1)}{G}\right),$ $$

where

$$ $\displaystyle\hat{H}=(\lambda+\hat{E})(\hat{m}^{2}+\lambda)+(\lambda+\hat{Q})(\hat{d}^{2}+\hat{\zeta})$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-2\hat{R}(\hat{m}\hat{d}+\hat{\omega}),$ $\displaystyle H=(d^{2}\rho+\eta E)(Q-\eta\chi+\beta_{\mathrm{VAE}})-\eta\zeta(m^{2}\rho+\eta Q)$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+2(\eta R+md\rho)(\rho\omega-1).$ $$

Thus, the generalization error and other summary statistics can be evaluated by numerically solving the self-consistent equations.

###### Claim 4.1 .

###### Claim 4.2 .

## 5 RESULTS

In this section, the generalization properties and the rate-distortion curve are derived from the free energy density obtained in the previous section.
In particular, the focus is on the representative case of the simplest VAE, i.e., $M=M^{\ast}=1$.
In addition, numerical experiments are conducted to verify the consistency of our theory and to compare the results obtained by training the VAE using a basic gradient descent algorithm.

### 5.1 Generalization Error and β VAE subscript 𝛽 VAE \beta_{\mathrm{VAE}}

Figure: Figure 1: (Left) Generalization error as a function of sample complexity $\alpha$ for fixed parameters $\beta_{\mathrm{VAE}}=1$ and varying $\lambda$ (Middle) Generalization error for different $\beta_{\mathrm{VAE}}$ with fixed parameters $\lambda=1$. (Right) KL divergence between the true posterior and the variational posterior with fixed parameters $\lambda=1$ for different $\beta_{\mathrm{VAE}}$. Each data point in all plots represents the average result of five different numerical simulations at $N=5000$ using gradient descent, and error bars represent standard deviations of the results.
Refer to caption: /html/2309.07663/assets/x1.png

We first clarify the relationship between the generalization error and $\beta_{\mathrm{VAE}}$. Our results can be summarized in three points as follows.

#### Interpolation peak as in supervised learning.

We show that the well-known interpolation peak in supervised regression (Mignacco et al.,, ; Hastie et al.,, ; Opper and Kinzel,, ) also occurs in VAEs, the unsupervised scenario. The interpolation peak in supervised regression has a characteristic peak in the generalization error at $\alpha=1$ with a small ridge regularization parameter, and the peak gradually decreases as the regularization parameter increases. Fig. [1](#S5.F1) shows the dependence of the generalization error $\varepsilon_{g}$ obtained by the replica method on $\beta_{\mathrm{VAE}}$ and $\lambda$, respectively, together with numerical experimental results with finite dataset size. The curves for small $\beta_{\mathrm{VAE}}$ and $\lambda$ show an peak at $\alpha=1$. This peak tends to disappear smoothly with increasing $\beta_{\mathrm{VAE}}$ and $\lambda$. This result implies that the peak is a universal phenomenon, commonly observed in supervised and unsupervised settings.

#### Long plateau in ε g subscript 𝜀 𝑔 \varepsilon_{g} with a large β VAE subscript 𝛽 VAE \beta_{\mathrm{VAE}} .

The left panel of Fig. ([1](#S5.F1)) shows $\alpha$ dependence of the generalization error $\varepsilon_{g}$ for various $\beta_{\mathrm{VAE}}$. For a smaller $\beta_{\mathrm{VAE}}$, the generalization error $\varepsilon_{g}$ begins to decrease from $\alpha=1$. Meanwhile, as $\beta_{\mathrm{VAE}}$ increases, a long plateau appears in a range of $\alpha$ before the curve begins to decrease.
Notably, the length of this plateau gets longer with the increasing values of the $\beta_{\mathrm{VAE}}$.
Moreover, when the value of $\beta_{\mathrm{VAE}}$ exceeds $2$, the decrease in the generalization error $\varepsilon_{g}$ seems to disappear completely.
The exact points at which the generalization error begins to decrease and remains $1$ will be explained in the following section with a corresponding phase diagram description.

#### The optimal β VAE subscript 𝛽 VAE \beta_{\mathrm{VAE}} depends on α 𝛼 \alpha and the state closest to the true posterior is not necessarily better.

We clarify that the optimal $\beta_{\mathrm{VAE}}$ that minimizes the generalization error $\varepsilon_{g}$ depends on $\alpha$.
Specifically, in a smaller $\alpha$ regime approximately from $\alpha=1$ to $\alpha\approx 2.6$, the generalization error $\varepsilon_{g}$ is minimized by $\beta_{\mathrm{VAE}}=1.5$.
On the other hand, in the larger $\alpha$ regime, the optimal value is $\beta_{\mathrm{VAE}}=1$.
In addition, the right panel of Fig. ([1](#S5.F1)) shows the KL divergence between the true posterior and the variational posterior, $D_{\mathrm{KL}}[p_{W}(z|{\bm{x}})\|q_{V,D}(z|{\bm{x}})]$, as a function of $\alpha$ for different values of $\beta_{\mathrm{VAE}}$. The figure shows that minimizing the generalization error $\varepsilon_{g}$ does not necessarily bring the true posterior $p_{W}(z|{\bm{x}})$ closer to the variational posterior $q_{V,D}(z|{\bm{x}})$.
In fact, even when the generalization error $\varepsilon_{g}$ is minimized at $\beta_{\mathrm{VAE}}=1.5$, the KL divergence for the $\beta_{\mathrm{VAE}}$ is not minimal in the range about $\alpha=1$ and $\alpha\approx 2.6$.

### 5.2 Phase Diagram

In this section, on the basis of the extreme values $m$ and $Q$ in Claim [4.2](#S4.ThmThm2), we discuss the behavior of $\beta_{\mathrm{VAE}}$ in terms of the phase diagram.
Before getting into the analysis of the phase diagram, let us recall the meaning of the order parameters $m$ and $q$.
The order parameter $m$ can be derived to satisfy the equation $m={\mathbb{E}}_{{\mathcal{D}}}[W^{\top}W^{\ast}/N]$, where $m$ corresponds to the overlap with the signal $W^{\ast}$ and the decoder parameter $W$. While $m\neq 0$ indicates that the VAE recovers the signal, when $m=0$, the VAE does not learn the signal, which implies the occurrence of posterior collapse.
On the other hand, the relation $q={\mathbb{E}}_{{\mathcal{D}}}[W^{\top}W/N]$ is also derived, meaning that $q$ is the norm of the decoder weights and is the quantity that observes the freedom of the parameter; a smaller $q$ indicates stronger regularization, yielding a smaller effective feasible region of the parameter, while a larger $q$ indicates weaker regularization, yielding the larger effective feasible region.
From these parameters, the following three distinct phases are identified, as shown in Fig [2](#S5.F2):

- •
Learning phase (green region, $m\neq 0,q\neq 0$): This phase represents a state where the VAE recovers the signal and avoids posterior collapse.
- •
Overlearning phase (red region, $m=0,q\neq 0$): This phase represents a state where the effects of the rate and ridge regularizations are small, resulting in the overlearning of the background noise in the data.
- •
Regularized phase (orange region, $m=0,q=0$): This phase represents a state where the degrees of freedom of the parameters are completely constrained by the rate and ridge regularizations.

The boundaries between the overlearning phase and the learning phase, as well as the regularized phase and the learning phase in the phase diagram, correspond precisely to the point where the generalization error begins to decrease, as noted in the previous section.
The phase diagram shows that as $\beta_{\mathrm{VAE}}$ becomes considerably larger, it becomes more difficult to reach the learning phase even with a sufficiently large $\alpha$, demonstrating the ‘long plateau’ described above.

Figure: Figure 2: Phase diagram for $\lambda=1$: Learning phase $(m\neq 0,q\neq 0)$, Overlearning phase $(m=0,q\neq 0)$, Regularized phase $(m=0,q=0)$. The overlearning and regularized phases fail to recover the true signal, which implies the appearance of posterior collapse.
Refer to caption: /html/2309.07663/assets/phase-diagram.png

### 5.3 Large α 𝛼 \alpha limit

The phase diagram in the previous section alone does not allow us to discuss whether it is possible to reach the learning phase by increasing $\alpha$ in any $\beta_{\mathrm{VAE}}$.
In this section, this feasibility is shown from the analysis at a large $\alpha$ limit. Furthermore, the optimal $\beta_{\mathrm{VAE}}$ that minimizes the generalization error in the large $\alpha$ limit is derived in closed form. We first present the following claim:

###### Claim 5.1 .

In the large $\alpha$ limit, when $\beta_{\mathrm{VAE}}<\rho+\eta$, the order parameter $m$ and the generalization error $\varepsilon_{g}$ are expressed as

$$ $\displaystyle m=\sqrt{\rho+\eta-\beta_{\mathrm{VAE}}},$ $\displaystyle\varepsilon_{g}=\rho-\sqrt{\eta+\rho-\beta_{\mathrm{VAE}}}(2\sqrt{\rho}-\sqrt{\eta+\rho-\beta_{\mathrm{VAE}}}),$ $$

respectively, and when $\beta_{\mathrm{VAE}}\geq\rho+\eta$, $m=0$ and $\varepsilon_{g}=1$.

From Claim [5.1](#S5.ThmThm1), once $\beta_{\mathrm{VAE}}$ exceeds the threshold $\beta_{\mathrm{VAE}}^{\ast}=\rho+\eta$, the learning phase is not reached even if $\alpha$ is increased, which means that the posterior collapse cannot be avoided. Thus, this result suggests that $\beta_{\mathrm{VAE}}$ can be a risky parameter and that learning may fail regardless of the dataset size. Furthermore, the extremum calculations of the generalization error in Claim [5.1](#S5.ThmThm1) show that the generalization error reaches a minimum value at $\beta_{\mathrm{VAE}}=\eta$, which means that the best result is achieved when $\beta_{\mathrm{VAE}}$ is equal to the strength of the background noise $\eta$.

###### Claim 5.1 .

### 5.4 Rate-Distortion Curve with Linear VAE

We present results on the rate-distortion curves of VAEs. First, we demonstrate that the following relationship holds in the large $\alpha$ limit.

###### Claim 5.2 .

In the large $\alpha$ limit, the rate-distortion curve $R_{\ast}$ of the linear VAE equals that of a Gaussian source (Cover,, ):

$$ $\displaystyle R_{\ast}$ $\displaystyle\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}{\mathbb{E}}_{{\mathcal{D}}}R(\bar{V}({\mathcal{D}}),\bar{D}({\mathcal{D}}))$ $\displaystyle=\begin{cases}\frac{1}{2}\log\frac{\eta+\rho}{2D_{\ast}}&0\leq D_{\ast}<\frac{\rho+\eta}{2},\\ 0&D_{\ast}\geq\frac{\rho+\eta}{2},\end{cases}$ (19) $$

where $D_{\ast}$ is the optimal distortion given by

$$ $\displaystyle D_{\ast}$ $\displaystyle\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}{\mathbb{E}}_{{\mathcal{D}}}D(\bar{W}({\mathcal{D}}),\bar{V}({\mathcal{D}}),\bar{D}({\mathcal{D}}))$ $\displaystyle=\begin{cases}\frac{\beta_{\mathrm{VAE}}}{2}&0\leq\beta_{\mathrm{VAE}}<\rho+\eta,\\ \frac{\rho+\eta}{2}&\beta_{\mathrm{VAE}}\geq\rho+\beta_{\mathrm{VAE}}.\end{cases}$ $$

See the Supplementary Material [B](#S2a) for a brief explanation of the rate-distortion function for Gaussian source.

Claim [5.2](#S5.ThmThm2) suggests that VAE achieves the optimal compression rate in the large $\alpha$ limit. Furthermore, the rate introduced by Alemi et al., () is found to coincide with the rate of discrete quantization of the rate-distortion theorem (Cover,, ) in the large $\alpha$ limit, indicating that the rate is truly a generalization of the rate of discrete quantization in the rate-distortion theory.
The results of the rate-distortion curve in the large $\alpha$ limit and in finite $\alpha$ are shown in Fig. [3](#S5.F3). The figure demonstrates that a relatively large dataset is required to obtain the optimal rate-distortion curve in the high-rate and low-distortion regions.
Moreover, when $dR(D_{\ast})/dD_{\ast}=-1$, VAE achieves the optimal generalization error with $\beta_{\mathrm{VAE}}=\eta$.

Figure: Figure 3: Rate-distortion curve for $\lambda=1$ with various values of $\alpha$. The dashed line represents the curve in the limit of infinite $\alpha$, which coincides with the rate-distortion curve for a Gaussian source (Cover,, ).
Refer to caption: /html/2309.07663/assets/x2.png

###### Claim 5.2 .

## 6 RELATED WORK

#### Replica Method.

The replica method is a non-rigorous but powerful heuristic of statistical physics (Mézard et al.,, ; Mezard and Montanari,, ; Blundell,, ). It has proven to be a valuable method for high-dimensional machine-learning problems.
Previous studies have investigated supervised learning: single-layer (Gardner and Derrida,, ; Opper and Haussler,, ; Barbier et al.,, ; Aubin et al.,, ) and multi-layer (Aubin et al.,, ) neural networks, as well as kernel methods(Dietrich et al.,, ; Bordelon et al.,, ; Gerace et al.,, ) and unsupervised learning: dimensionality reduction techniques such as principal component analysis (Biehl and Mietzner,, ; Hoyle and Rattray,, , ), and generative models such as energy-based models (Decelle et al.,, ; Ichikawa and Hukushima,, ) and denoising autoencoders (Cui and Zdeborová,, ).
However, there has been no analysis of VAEs, and in this study, we conducted an analysis of VAEs.
Efforts have been made to confirm the non-rigorous results of the replica method using other rigorous analysis techniques. For convex optimization problems, the Gaussian min-max theorem (Gordon,, ) can be used for rigorous analysis, which provides rigorous results consistent with those of the replica method (Thrampoulidis et al.,, ).

#### Linear VAEs.

The linear VAE is a simple model where both the encoder and decoder are constrained to be affine transformations (Lucas et al.,, ). While deriving analytical results for deep latent models is often intractable, the linear VAE can provide analytical results, facilitating a deeper understanding of VAEs.
Furthermore, despite this simplicity, the theoretical results can well explain the behavior of deeper and intricately structured VAEs (Lucas et al.,, ; Bae et al.,, ).
In fact, results proven effective for linear models have been applied to deeper models, leading to the new algorithms (Bae et al.,, ).
In addition, various theoretical results have been obtained; Dai et al., () show the connections between linear VAE, probabilistic PCA (Tipping and Bishop,, ), and robust PCA (Candès et al.,, ; Chandrasekaran et al.,, ). In parallel, studies by Lucas et al., () and Wang and Ziyin, () utilize linear VAEs to explore the origins of posterior collapse.
However, these analyses do not address the dataset-size dependence of generalization performance, the rate-distortion curve, and the robustness against background noise examined in our study.

## 7 CONCLUSION

In this paper, we clarify the relationship between dataset size, $\beta_{\mathrm{VAE}}$, the posterior collapse, and the rate-distortion curve by using a replica method to evaluate the sharp asymptotic properties of the linear VAE.
Building on these results, we observe a peak in the generalization error at a particular sample complexity for small values of $\beta_{\mathrm{VAE}}$, reminiscent of the well-known interpolation peak in supervised regressions.
This suggests that the peak is a universal phenomenon, independent of supervised and unsupervised learning.
We also find that a long plateau appears in the generalization error for a large value of $\beta_{\mathrm{VAE}}$. As $\beta_{\mathrm{VAE}}$ increases, the length of the plateau extends, and when $\beta_{\mathrm{VAE}}$ exceeds a critical value, it goes to infinity even for infinite dataset size.
This result suggests that, unlike the ridge regularization parameter, $\beta_{\mathrm{VAE}}$ is potentially risky.
Extending the data generative model and the linear VAE to more complex models and analyzing the learning dynamics of the linear VAE are interesting future directions.

Dataset Size Dependence of Rate-Distortion Curve and
Threshold of Posterior Collapse in Linear VAE: Supplementary Materials

## A OVERVIEW

This supplementary material provides extended explanations, implementation details, and additional results for the paper “Dataset Size Dependence of Rate-Distortion Curve and Threshold of Posterior Collapse in Linear VAE”.

## B REVIEW OF RATE DISTORTION THEORY

The rate-distortion theory was introduced by Shannon et al., () and then further developed by Berger et al., (); Berger and Gibson, ().
This theoretical framework describes the minimum bit rate (rate) required for encoding a source, subject to a given distortion measure.
In recent years, it has been used to understand machine learning (Gao et al.,, ; Alemi et al.,, ; Theis et al.,, ; Brekelmans et al.,, ; Isik et al.,, ).

Let $X^{P}=\{X_{1},\ldots,X_{P}\}\in{\mathcal{X}}^{P}$ be i.i.d random variables from the distribution $P(x)$.
An encoder $f_{P}:{\mathcal{X}}^{P}\to\{1,2,\ldots,2^{P\times R}\}$ maps the input $X^{P}$ into a quantized vector, and a decoder $g_{P}:\{1,2,\ldots,2^{P\times R}\}\to{\mathcal{X}}^{P}$ reconstructs the input by a decoded input $\hat{X}^{P}$ from the quantized vector.
To measure the discrepancy between the original and decoded inputs, a distortion function $d:{\mathcal{X}}\times{\mathcal{X}}\to{\mathbb{R}}_{+}$ is introduced. The distortion for the input $X^{P}$ and decoded input $\hat{X}^{P}$ is defined as the average distortion between each pair $X_{i}$ and $\hat{X}_{i}$.
Commonly used distortion functions are the Hamming distortion function defined as $d(x,\hat{x})=\mathbb{I}[x\neq\hat{x}]$ for $X=\{0,1\}$ where $\mathbb{I}$ is the indicator function, and the squared error distortion function defined as $d(x,\hat{x})=(x-\hat{x})^{2}$ for $X={\mathbb{R}}$.
We are ready to define the rate-distortion function.

###### Definition B.1 .

A rate-distortion pair $(R,D)$ is achievable if there exists a (probabilistic) encoder-decoder $(f_{P},g_{P})$ such that the quantized vector has size $2^{P\times R}$ and the expected distortion $\lim_{P\to\infty}[d(X^{P},g_{P}(f_{P}(X^{P}))]\leq D$.

###### Definition B.2 .

The rate-distortion function $R(D)$ is the infimum of rates $R$ such that the rate-distortion pair $(R,D)$ is achievable.

The main theorem of the rate-distortion theory (Cover,, ) states as follows,

###### Theorem B.3 .

Given an upper bound of distortion $D$, the following equation holds:

$$ $R(D)=\min_{P(\hat{X}|X):{\mathbb{E}}[d(X,\hat{X})]\leq D}I(X;\hat{X})$ (20) $$

The rate-distortion theorem provides the fundamental limit of data compression, i.e., how many minimum bits are needed to compress the input, given the quality of the reconstructed input.

###### Definition B.1 .

###### Definition B.2 .

###### Theorem B.3 .

### B.1 Rate-distortion of Gaussian source.

We give an example of the rate-distortion function for Gaussian input.

###### Proposition B.4 .

If $X\sim{\mathcal{N}}(0,\sigma^{2})$, the rate-distortion function is given by

$$ $R(D)=\begin{cases}\frac{1}{2}\log_{2}\frac{\sigma^{2}}{D}&D\leq\sigma^{2}\\ 0&D>\sigma^{2}\end{cases}.$ $$

If the required distortion is larger than the variance of the Gaussian variable $\sigma^{2}$, we simply transmit $\hat{X}=0$; otherwise, we transmit $\hat{X}$ such that $\hat{X}\sim{\mathcal{N}}(0,\sigma^{2}-D)$, $X-\hat{X}\sim{\mathcal{N}}(0,D)$ where $\hat{X}$ and $X-\hat{X}$ are independent.

###### Proposition B.4 .

## C DETAILED DERIVATION OF CLAIMS

Here, we present the derivation of Claims 4.1, 4.2, 5.1, and 5.2.

### C.1 Derivation of Claim 4.1

To calculate free-energy density, it is sufficient to calculate the replicated partition function, as mentioned in Section 4.1. The replicated partition function is expressed as

$$ $\displaystyle\mathbb{E}_{\mathcal{D}}Z^{n}(\mathcal{D},\beta)$ $\displaystyle=\mathbb{E}_{\mathcal{D}}\int\prod_{a=1}^{n}dW^{a}dV^{a}dD^{a}\prod_{a=1}^{n}e^{-\beta\mathcal{R}(W^{a},V^{a},D^{a};\mathcal{D},\beta_{\mathrm{VAE}},\lambda)}$ $\displaystyle=\mathbb{E}_{\mathcal{D}}\int\prod_{a=1}^{n}dW^{a}dV^{a}dD^{a}\prod_{a=1}^{n}e^{-\beta\sum_{\mu=1}^{P}{\mathcal{L}}(W^{a},V^{a},D^{a};{\bm{x}}^{\mu},\beta_{\mathrm{VAE}})}e^{-\beta\left(\frac{\lambda}{2}\|W^{a}\|_{F}^{2}+\frac{\lambda}{2}\|V^{a}\|^{2}_{F}\right)}$ $\displaystyle=\int\prod_{a=1}^{n}dW^{a}dV^{a}dD^{a}\prod_{\mu=1}^{P}{\mathbb{E}}_{{\bm{c}}^{\mu},{\bm{n}}^{\mu}}\left[\prod_{a=1}^{n}e^{-\beta\sum_{\mu=1}^{P}{\mathcal{L}}(W^{a},V^{a},D^{a};{\bm{c}}^{\mu},{\bm{n}}^{\mu}\beta_{\mathrm{VAE}})}\right]e^{-\beta\left(\frac{\lambda}{2}\|W^{a}\|_{F}^{2}+\frac{\lambda}{2}\|V^{a}\|^{2}_{F}\right)}$ $\displaystyle=\int\prod_{a=1}^{n}dW^{a}dV^{a}dD^{a}\left({\mathbb{E}}_{{\bm{c}},{\bm{n}}}\left[\prod_{a=1}^{n}e^{-\beta\sum_{\mu=1}^{P}{\mathcal{L}}(W^{a},V^{a},D^{a};{\bm{c}},{\bm{n}},\beta_{\mathrm{VAE}})}\right]\right)^{P}e^{-\beta\left(\frac{\lambda}{2}\|W^{a}\|_{F}^{2}+\frac{\lambda}{2}\|V^{a}\|^{2}_{F}\right)},$ $$

where ${\mathcal{L}}(W^{a},V^{a},D^{a};{\bm{c}},{\bm{n}},\beta_{\mathrm{VAE}})$ is given by

$$ ${\mathcal{L}}(W^{a},V^{a},D^{a};{\bm{c}},{\bm{n}},\beta_{\mathrm{VAE}})=\frac{1}{2}\Bigg{(}\left\|\sqrt{\frac{\rho}{N}}W^{\ast}{\bm{c}}+\sqrt{\eta}{\bm{n}}\right\|^{2}\\ -\frac{2}{\sqrt{N}}\left(\sqrt{\frac{\rho}{N}}(W^{a})^{\top}W^{\ast}{\bm{c}}+\sqrt{\eta}(W^{a})^{\top}{\bm{n}}\right)^{\top}\left(\frac{\sqrt{\rho}}{N}(V^{a})^{\top}W^{\ast}{\bm{c}}+\sqrt{\frac{\eta}{N}}(V^{a})^{\top}{\bm{n}}\right)\\ +\left(\frac{\sqrt{\rho}}{N}(V^{a})^{\top}W^{\ast}{\bm{c}}+\sqrt{\frac{\eta}{N}}(V^{a})^{\top}{\bm{n}}\right)^{\top}\frac{(W^{a})^{\top}W^{a}}{N}\left(\frac{\sqrt{\rho}}{N}(V^{a})^{\top}W^{\ast}{\bm{c}}+\sqrt{\frac{\eta}{N}}(V^{a})^{\top}{\bm{n}}\right)+\frac{1}{N}(W^{a})^{\top}W^{a}D^{a}\\ +\beta_{\mathrm{VAE}}\left(\left\|\frac{\sqrt{\rho}}{N}(V^{a})^{\top}W^{\ast}{\bm{c}}+\sqrt{\frac{\eta}{N}}(V^{a})^{\top}{\bm{n}}\right\|^{2}+\mathrm{tr}(D^{a})^{2}-\mathrm{tr}(\log D^{a})\right)\Bigg{)}.$ $$

To perform the average over ${\bm{n}}$, we notice that, since ${\bm{n}}$ follows a multivariate normal distribution ${\mathcal{N}}({\bm{0}}_{N},{\bm{I}}_{N})$, ${\bm{h}}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}{\bm{u}}\oplus\tilde{{\bm{u}}}\in\mathbb{R}^{2Mn}$ with

$$ $\displaystyle{\bm{u}}^{\mu}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\bigoplus_{a=1}^{n}\frac{1}{\sqrt{N}}(W^{a})^{\top}{\bm{n}}^{\mu},~{}\tilde{{\bm{u}}}^{\mu}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\bigoplus_{a=1}^{n}\frac{1}{\sqrt{N}}(V^{a})^{\top}{\bm{n}}^{\mu}$ $$

follows a following Gaussian multivariate distribution:

$$ $p({\bm{h}})={\mathcal{N}}({\bm{h}};{\bm{0}}_{2Mn},\Sigma),$ $$

where

$$ $\displaystyle\Sigma=\delta_{\mu\nu}\begin{pmatrix}Q&R\\ R&E\end{pmatrix},~{}Q=(Q^{ab})\in{\mathbb{R}}^{Mn\times Mn},~{}E=(E^{ab})\in{\mathbb{R}}^{Mn\times Mn},~{}R=(R^{ab})\in{\mathbb{R}}^{Mn\times Mn},$ $\displaystyle Q^{ab}=\frac{1}{N}(W^{a})^{\top}W^{b},~{}E^{ab}=\frac{1}{N}(V^{a})^{\top}V^{b},~{}R^{ab}=\frac{1}{N}(W^{a})^{\top}V^{b}.$ $$

By introducing the auxiliary variables through the trivial identities

$$ $\displaystyle 1=\prod_{(a,l);(b,s)}N\int\delta\left(NQ_{ls}^{ab}-({\bm{w}}_{l}^{a})^{\top}{\bm{w}}_{s}^{b}\right)dQ,$ $\displaystyle 1=\prod_{(a,l);(b,s)}N\int\delta\left(NE_{ls}^{ab}-({\bm{v}}_{l}^{a})^{\top}{\bm{v}}_{s}^{b}\right)dE,$ $\displaystyle 1=\prod_{(a,l);(b,s)}N\int\delta\left(NR_{ls}^{ab}-({\bm{w}}_{l}^{a})^{\top}{\bm{v}}_{s}^{b}\right)dR,$ $\displaystyle 1=\prod_{(a,s);(a,l^{\ast})}N\int\delta\left(Nm_{sl^{\ast}}^{a}-({\bm{w}}_{s}^{a})^{\top}{\bm{w}}^{\ast}_{l^{\ast}}\right)dm,$ $\displaystyle 1=\prod_{(a,s);(a,l^{\ast})}N\int\delta\left(Nd_{sl^{\ast}}^{a}-({\bm{v}}_{s}^{a})^{\top}{\bm{w}}^{\ast}_{l^{\ast}}\right)dd,$ $$

the replicated partition function is further expressed as

$$ $\displaystyle\mathbb{E}_{\mathcal{D}}Z^{n}(\mathcal{D},\beta)=\int dQdEdRdmdd[\mathcal{S}\times\mathcal{E}],$ $\displaystyle\mathcal{S}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\int\prod_{a=1}^{n}dW^{a}dV^{a}\prod_{a,b}\prod_{s,l}N\delta\left(NQ^{ab}_{sl}-({\bm{w}}^{a}_{s})^{\top}{\bm{w}}^{b}_{l}\right)N\delta\left(NE^{ab}_{sl}-({\bm{v}}^{a}_{s})^{\top}{\bm{v}}^{b}_{l}\right)N\delta\left(NR^{ab}_{sl}-({\bm{w}}^{a}_{s})^{\top}{\bm{v}}^{b}_{l}\right)$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\prod_{a}\prod_{s,l}N\delta\left(Nm^{a}_{sl^{\ast}}-({\bm{w}}_{s}^{a})^{\top}{\bm{w}}^{\ast}_{l^{\ast}}\right)N\delta\left(d^{a}_{sl^{\ast}}-({\bm{v}}_{s}^{a})^{\top}{\bm{w}}^{\ast}_{l}\right)\times e^{-\frac{\beta}{2}\sum_{a}(\lambda\|W\|_{F}^{2}+\lambda\|V\|_{F}^{2})},$ $\displaystyle\mathcal{E}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\int\prod_{a}dD^{a}\left(\int D{\bm{c}}\int d{\bm{h}}\mathcal{N}({\bm{h}},{\bm{0}}_{2Mn},\Sigma)\times e^{-\beta\sum_{a}\mathcal{L}(Q,E,R,m,d;{\bm{h}},{\bm{c}},\beta_{\mathrm{VAE}},\lambda,\tilde{\lambda})}\right)^{P},$ $$

where ${\bm{w}}^{\ast}_{l^{\prime}}$ ,${\bm{w}}^{a}_{l}$ and ${\bm{v}}^{a}_{l}$ are column vectors of $W^{\ast}$, $W^{a}$, and $V^{a}$, respectively.
Assuming the replica symmetric (RS) ansatz, one reads

$$ $\displaystyle Q^{aa}_{ls}=Q_{ls},~{}E^{aa}_{ls}=E_{ls},~{}R^{aa}_{ls}=R_{ls},~{}m^{a}_{sl^{\ast}}=m_{sl^{\ast}},~{}d^{a}_{sl^{\ast}}=d_{sl^{\ast}},$ (21) $\displaystyle Q^{ab}_{ls}=Q_{ls}-\frac{\chi_{ls}}{\beta},~{}~{}E_{ls}^{ab}=E_{ls}-\frac{\zeta_{ls}}{\beta},~{}~{}R_{ls}^{ab}=R_{ls}-\frac{\omega_{ls}}{\beta},$ (22) $$

where all parameters are denoted as $\Theta\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}(Q,E,R,m,d,\chi,\zeta,\omega)\in{\mathbb{R}}^{M\times(6M+2M^{\ast})}$.
This RS ansatz restricts the integration of the replicated weight parameters $\{W_{a},V_{a}\}$ across the entire $\mathbb{R}^{(2M\times N)\times n}$ to a subspace that satisfies the constraints in Eq. [21](#S3.E21) and [22](#S3.E22).
Using the Fourier transform of the delta functions, ${\mathcal{S}}$ is expanded as

$$ $\displaystyle{\mathcal{S}}$ $\displaystyle=\int d\hat{Q}d\hat{E}d\hat{R}d\hat{\chi}d\hat{\zeta}d\hat{m}d\hat{d}\prod_{a}dW^{a}dV^{a}~{}e^{\frac{1}{2}\sum_{ls}\sum_{a}(\beta\hat{Q}_{ls}-\beta^{2}\hat{\chi}_{ls})(NQ_{ls}-{\bm{w}}_{l}^{a}{\bm{w}}_{s}^{a})-\frac{1}{2}\sum_{ls}\sum_{a\neq b}\beta^{2}\hat{\chi}\left(Q_{ls}-\frac{\chi_{ls}}{\beta}-{\bm{w}}_{l}^{a}{\bm{w}}_{s}^{b}\right)}$ $\displaystyle\times e^{\frac{1}{2}\sum_{ls}\sum_{a}(\beta\hat{E}_{ls}-\beta^{2}\hat{\zeta}_{ls})(NE_{ls}-{\bm{v}}_{l}^{a}{\bm{v}}_{s}^{a})-\frac{1}{2}\sum_{ls}\sum_{a\neq b}\beta^{2}\hat{\zeta}\left(E_{ls}-\frac{\zeta_{ls}}{\beta}-{\bm{v}}_{l}^{a}{\bm{v}}_{s}^{b}\right)}$ $\displaystyle\times e^{\sum_{ls}\sum_{a}(\beta\hat{R}_{ls}-\beta^{2}\hat{\omega}_{ls})(NR_{ls}-{\bm{w}}_{l}^{a}{\bm{v}}_{s}^{a})-\sum_{ls}\sum_{a\neq b}\beta^{2}\hat{\omega}\left(R_{ls}-\frac{\omega_{ls}}{\beta}-{\bm{w}}_{l}^{a}{\bm{v}}_{s}^{b}\right)}$ $\displaystyle\times e^{-\sum_{ls}\sum_{a}\beta\hat{m}_{sl^{\ast}}(Nm_{sl^{\ast}}-{\bm{w}}^{a}_{s}{\bm{w}}^{\ast}_{l^{\ast}})-\sum_{ls}\sum_{a}\beta\hat{d}_{sl^{\ast}}(Nd_{sl^{\ast}}-{\bm{v}}_{s}^{a}{\bm{w}}_{l^{\ast}}^{\ast})}e^{-\frac{\beta\lambda}{2}\sum_{a}\left(\|W_{a}\|^{2}_{F}+\|V^{a}\|_{F}^{2}\right)}$ $\displaystyle=\int d\hat{\Theta}e^{\frac{n\beta N}{2}\left(\mathrm{tr}\left(\hat{Q}Q+(n-1)\hat{\chi}\chi-n\beta Q\hat{\chi}\right)+\mathrm{tr}\left(\hat{E}E+(n-1)\hat{\zeta}\zeta-n\beta E\hat{\zeta}\right)+2\mathrm{tr}\left(\hat{R}R+(n-1)\hat{\omega}\omega-n\beta R\hat{\omega}\right)-\mathrm{tr}(\hat{m}m)-\mathrm{tr}(\hat{d}d)\right)}$ $\displaystyle\times\int dW^{a}dV^{a}e^{-\frac{\beta}{2}\sum_{ls}\left((\hat{Q}_{ls}+\lambda)\sum_{a}{\bm{w}}_{l}^{a}{\bm{w}}_{s}^{a}+(\hat{E}_{ls}+\lambda)\sum_{a}{\bm{v}}_{l}^{a}{\bm{v}}_{s}^{a}+2\hat{R}_{ls}\sum_{a}{\bm{w}}^{a}_{l}{\bm{v}}^{a}_{s}\right)+\frac{\beta^{2}}{2}\sum_{ls}\left(\hat{\chi}_{ls}(\sum_{a}{\bm{w}}_{a})^{2}+\hat{\zeta}_{ls}(\sum_{a}{\bm{v}}_{a})^{2}+2\hat{\omega}_{ls}\sum_{a}{\bm{w}}_{a}\sum_{a}{\bm{v}}_{a}\right)}$ $\displaystyle\times e^{\beta\sum_{ls}\hat{m}_{sl^{\ast}}\sum_{a}{\bm{w}}_{s}^{a}{\bm{w}}_{l^{\ast}}^{\ast}+\beta\sum_{ls}\hat{d}_{sl^{\ast}}\sum_{a}{\bm{v}}_{s}^{a}{\bm{w}}_{l^{\ast}}^{\ast}},$ $$

where $d\hat{\Theta}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}d\hat{Q}d\hat{E}d\hat{R}d\hat{\chi}d\hat{\zeta}d\hat{m}d\hat{d}$.
This can be derived with the help of the identity for any symmetric positive matrix $M\in{\mathbb{R}}^{M\times M}$ and any vector ${\bm{x}}\in{\mathbb{R}}^{M}$, given by

$$ $e^{\frac{1}{2}{\bm{x}}^{\top}M{\bm{x}}}=\int D{\bm{\xi}}_{M}e^{{\bm{\xi}}^{\top}M^{\frac{1}{2}}{\bm{x}}},$ $$

where $D{\bm{\xi}}_{M}$ is the standard Gaussian measure on ${\mathbb{R}}^{M}$. Then, we obtain:

$$ $\displaystyle{\mathcal{S}}$ $\displaystyle=\int d\hat{\Theta}e^{\frac{n\beta N}{2}\left(\mathrm{tr}\left(\hat{Q}Q+(n-1)\hat{\chi}\chi-n\beta Q\hat{\chi}\right)+\mathrm{tr}\left(\hat{E}E+(n-1)\hat{\zeta}\zeta-n\beta E\hat{\zeta}\right)+2\mathrm{tr}\left(\hat{R}R+(n-1)\hat{\omega}\omega-n\beta R\hat{\omega}\right)-\mathrm{tr}(\hat{m}m)-\mathrm{tr}(\hat{d}d)\right)}$ $\displaystyle\times\left(\int D{\bm{\xi}}_{2M}\left(\int d\tilde{{\bm{w}}}e^{\frac{\beta}{2}\tilde{{\bm{w}}}^{\top}(\tilde{Q}+\lambda I_{2M})\tilde{{\bm{w}}}+\beta({\bm{\xi}}_{2M}^{\top}\tilde{\chi}^{\frac{1}{2}}+{\bm{1}}_{2M}^{\top}\tilde{m})\tilde{{\bm{w}}}}\right)^{n}\right)^{N}$ $\displaystyle=\int d\hat{\Theta}e^{\frac{n\beta N}{2}\left(\mathrm{tr}\left(\hat{Q}Q+(n-1)\hat{\chi}\chi-n\beta Q\hat{\chi}\right)+\mathrm{tr}\left(\hat{E}E+(n-1)\hat{\zeta}\zeta-n\beta E\hat{\zeta}\right)+2\mathrm{tr}\left(\hat{R}R+(n-1)\hat{\omega}\omega-n\beta R\hat{\omega}\right)-\mathrm{tr}(\hat{m}m)-\mathrm{tr}(\hat{d}d)\right)}$ $\displaystyle\times e^{N\log\int D{\bm{\xi}}_{2M}\left(\int d\tilde{{\bm{w}}}e^{\frac{\beta}{2}\tilde{{\bm{w}}}^{\top}(\tilde{Q}+\lambda I_{2M})\tilde{{\bm{w}}}+\beta({\bm{\xi}}_{2M}^{\top}\tilde{\chi}^{\frac{1}{2}}+{\bm{1}}_{2M}^{\top}\tilde{m})\tilde{{\bm{w}}}}\right)^{n}}$ $\displaystyle=\int d\hat{\Theta}e^{\frac{n\beta N}{2}\left(\mathrm{tr}\left(\hat{Q}Q-\hat{\chi}\chi\right)+\mathrm{tr}\left(\hat{E}E-\hat{\zeta}\zeta\right)+2\mathrm{tr}\left(\hat{R}R-\hat{\omega}\omega\right)-\mathrm{tr}(\hat{m}m)-\mathrm{tr}(\hat{d}d)+{\mathcal{O}}(n)\right)}$ $\displaystyle\times e^{Nn\left(\int D{\bm{\xi}}_{2M}\log\int d\tilde{{\bm{w}}}e^{\frac{\beta}{2}\tilde{{\bm{w}}}^{\top}(\tilde{Q}+\lambda I_{2M})\tilde{{\bm{w}}}+\beta({\bm{\xi}}_{2M}^{\top}\tilde{\chi}^{\frac{1}{2}}+{\bm{1}}_{2M}^{\top}\tilde{m})\tilde{{\bm{w}}}}+{\mathcal{O}}(n)\right)}$ $\displaystyle=\int d\hat{\Theta}e^{\frac{n\beta N}{2}\left(\mathrm{tr}\left(\hat{Q}Q-\hat{\chi}\chi\right)+\mathrm{tr}\left(\hat{E}E-\hat{\zeta}\zeta\right)+2\mathrm{tr}\left(\hat{R}R-\hat{\omega}\omega\right)-\mathrm{tr}(\hat{m}m)-\mathrm{tr}(\hat{d}d)+\mathrm{tr}[(\tilde{\chi}^{\frac{1}{2}})^{\top}(Q+\lambda)^{-1}\tilde{\chi}^{\frac{1}{2}}]+{\bm{1}}_{2M}^{\top}\tilde{m}\tilde{Q}^{-1}\tilde{m}^{\top}{\bm{1}}_{2M}\right)+o(n)+o(N)+o(\beta)}$ $$

where $\tilde{{\bm{w}}}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}(w_{1},\ldots,w_{M},{v}_{1},\ldots,v_{M})$ and

$$ $\displaystyle\tilde{Q}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\begin{pmatrix}\hat{Q}&\hat{R}\\ \hat{R}&\hat{E}\end{pmatrix},~{}\tilde{\chi}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\begin{pmatrix}\hat{\chi}&\hat{\omega}\\ \hat{\omega}&\hat{\zeta}\end{pmatrix},~{}\tilde{m}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\begin{pmatrix}\hat{m}&{\bm{0}}_{MM^{\ast}}\\ {\bm{0}}_{MM^{\ast}}&\hat{d}\end{pmatrix}.$ $$

The integral with respect to $\tilde{{\bm{w}}}$ can be calculated analytically using this expression. ${\mathcal{E}}$ is also expanded as

$$ $\displaystyle\log{\mathcal{E}}$ $\displaystyle=\log\int\prod_{a}dD^{a}\left(\int D{\bm{c}}\int d{\bm{h}}\mathcal{N}({\bm{h}},{\bm{0}}_{2Mn},\Sigma)e^{-\beta\sum_{a}\mathcal{L}(Q,E,R,m,d;{\bm{h}},{\bm{c}},\beta_{\mathrm{VAE}},\lambda,\tilde{\lambda})}\right)^{P}$ $\displaystyle=n\log\int dDe^{-\frac{\beta P}{2}\left(\mathrm{tr}[(Q+\beta_{\mathrm{VAE}})D]-\beta_{\mathrm{VAE}}\mathrm{tr}(\log D)\right)}$ $\displaystyle+P\log\int D{\bm{c}}\int d{\bm{h}}\mathcal{N}({\bm{h}},{\bm{0}}_{2Mn},\Sigma)e^{-\beta\sum_{a}\mathcal{L}(Q,E,R,m,d;{\bm{h}},{\bm{c}},\beta_{\mathrm{VAE}},\lambda,\tilde{\lambda})}$ $\displaystyle=n\log\int dDe^{-\frac{\beta P}{2}\left(\mathrm{tr}[(Q+\beta_{\mathrm{VAE}})D]-\beta_{\mathrm{VAE}}\mathrm{tr}(\log D)\right)}$ $\displaystyle+P\log\int D{\bm{c}}D{\bm{\xi}}_{2M\times 2M}\left(\int D{\bm{z}}_{2M\times 2M}e^{-\beta\mathcal{L}(Q,E,R,m,d;{\bm{\xi}}_{2M\times 2M},{\bm{z}}_{2M\times 2M},{\bm{c}},\beta_{\mathrm{VAE}},\lambda,\tilde{\lambda})}\right)^{n}$ $\displaystyle=n\log\int dDe^{-\frac{\beta P}{2}\left(\mathrm{tr}[(Q+\beta_{\mathrm{VAE}})D]-\beta_{\mathrm{VAE}}\mathrm{tr}(\log D)\right)}$ $\displaystyle+Pn\int D{\bm{c}}D{\bm{\xi}}_{2M\times 2M}\log\int D{\bm{z}}_{2M\times 2M}e^{-\beta\mathcal{L}(Q,E,R,m,d;{\bm{\xi}}_{2M\times 2M},{\bm{z}}_{2M\times 2M},{\bm{c}},\beta_{\mathrm{VAE}},\lambda,\tilde{\lambda})}+o(n).$ $$

Taking the limit $\beta\to\infty$, one can obtain

$$ $\log{\mathcal{L}}=-Pn\beta\int D{\bm{c}}D{\bm{\xi}}_{2M\times 2M}\log\min_{D,{\bm{z}}_{2M\times 2M}}{\mathcal{L}}(\Theta;{\bm{z}}_{2M\times 2M},{\bm{\xi}}_{2M\times 2M})$ $$

Substituting ${\mathcal{S}}$ and ${\mathcal{E}}$ into the expression of the replicated partition function yields

$$ ${\mathbb{E}}_{{\mathcal{D}}}Z_{\beta}^{n}({\mathcal{D}})=\int d\Theta d\hat{\Theta}e^{\frac{n\beta N}{2}(\mathrm{tr}(\hat{Q}Q-\hat{\chi}\chi)+\mathrm{tr}(\hat{E}E-\hat{\zeta}\zeta)+2\mathrm{tr}(\hat{R}R-\hat{\omega}\omega)-\mathrm{tr}(\hat{m}m)-\mathrm{tr}(\hat{d}d)+\mathrm{tr}[(\tilde{\chi}^{\frac{1}{2}})^{\top}(Q+\lambda)^{-1}\tilde{\chi}^{\frac{1}{2}}]+{\bm{1}}_{2M}^{\top}\tilde{m}\tilde{Q}^{-1}\tilde{m}^{\top}{\bm{1}}_{2M})}\\ \times e^{-\alpha Nn\beta\int D{\bm{c}}D{\bm{\xi}}_{2M\times 2M}\log\min_{D,{\bm{z}}_{2M\times 2M}}{\mathcal{L}}(\Theta;{\bm{z}}_{2M\times 2M},{\bm{\xi}}_{2M\times 2M})}$ $$

In the end, from the identity:

$$ $\lim_{n\to+0}\frac{\log{\mathbb{E}}_{{\mathcal{D}}}Z_{\beta}^{n}({\mathcal{D}})}{n},$ $$

one obtains

$$ $f=\underset{\Theta,\hat{\Theta}}{\mathrm{extr}}\Bigg{\{}-\frac{1}{2}\left(\mathrm{tr}(Q\hat{Q}-\chi\hat{\chi})+\mathrm{tr}(E\hat{E}-\zeta\hat{\zeta})+2\mathrm{tr}(R\hat{R}-\omega\hat{\omega})-\mathrm{tr}(m\hat{m})-\mathrm{tr}(d\hat{d})\right)\\ -\frac{1}{2}\left(\mathrm{tr}[(\tilde{\chi}^{\frac{1}{2}})^{\top}(\tilde{Q}+\lambda)^{-1}\tilde{\chi}^{\frac{1}{2}}]+{\bm{1}}_{2M}^{\top}\tilde{m}\tilde{Q}^{-1}\tilde{m}^{\top}{\bm{1}}_{2M}\right)+\alpha\int D{\bm{c}}D{\bm{\xi}}_{2M\times 2M}\log\min_{D,{\bm{z}}_{2M\times 2M}}{\mathcal{L}}(\Theta;{\bm{z}}_{2M\times 2M},{\bm{\xi}}_{2M\times 2M})\Bigg{\}}.$ (23) $$

### C.2 Derivation of Claim 4.2

When $M=M^{\ast}=1$, the following expressions holds:

$$ $\tilde{\chi}^{\frac{1}{2}}=\frac{1}{\sqrt{\hat{\chi}+\hat{\zeta}+2\sqrt{\hat{\chi}\hat{\zeta}-\hat{\omega}^{2}}}}\begin{pmatrix}\hat{\chi}+\sqrt{\hat{\chi}\hat{\zeta}-\hat{\omega}^{2}}&\hat{\omega}\\ \hat{\omega}&\hat{\zeta}+\sqrt{\hat{\chi}\hat{\zeta}-\hat{\omega}^{2}}\end{pmatrix},~{}~{}\tilde{Q}=\frac{1}{\hat{Q}\hat{E}-\hat{R}^{2}}\begin{pmatrix}\hat{E}&-\hat{R}\\ -\hat{R}&\hat{Q}\end{pmatrix}$ $$

Using these, a part of the exponential function of Eq. ([23](#S3.E23)) can be reduced as

$$ $-\frac{1}{2}\left(\mathrm{tr}[(\tilde{\chi}^{\frac{1}{2}})^{\top}(\tilde{Q}+\lambda)^{-1}\tilde{\chi}^{\frac{1}{2}}]+{\bm{1}}_{2}^{\top}\tilde{m}\tilde{Q}^{-1}\tilde{m}^{\top}{\bm{1}}_{2}\right)=-\frac{(\lambda+\hat{Q})(\hat{m}^{2}+\hat{\chi})+(\lambda+\hat{E})(\hat{d}^{2}+\hat{\zeta})-2\hat{R}(\hat{m}\hat{d}+\hat{\omega})}{2((\hat{Q}+\lambda)(\hat{E}+\lambda)-\hat{R}^{2})}.$ (24) $$

We notice that ${\bm{u}}$ and $\tilde{{\bm{u}}}$ can be expressed as

$$ $\displaystyle u_{a}$ $\displaystyle=\sqrt{\frac{\chi-\omega}{\beta}}z^{a}_{11}+\sqrt{\frac{\omega}{\beta}}z^{a}_{12}+\sqrt{Q-R}\xi_{11}+\sqrt{R}\xi_{12},$ $\displaystyle\tilde{u}_{a}$ $\displaystyle=\sqrt{\frac{\zeta-\omega}{\beta}}z_{22}^{a}+\sqrt{\frac{\omega}{\beta}}z_{12}^{a}+\sqrt{E-R}\xi_{22}+\sqrt{R}\xi_{12},$ $$

where $z_{11}^{a},z_{12}^{a},z_{22},\xi_{11},\xi_{12},\xi_{22}$ follows standard Gaussian distribution ${\mathcal{N}}(0,1)$.
By substituting ${\bm{u}}$ and $\tilde{{\bm{u}}}$, ${\mathcal{E}}$ is transformed to

$$ $\displaystyle{\mathcal{E}}$ $\displaystyle=\int D{\bm{c}}D{\bm{\xi}}_{2M\times 2M}\log\min_{D,{\bm{z}}_{2M\times 2M}}{\mathcal{L}}(\Theta;{\bm{z}}_{2M\times 2M},{\bm{\xi}}_{2M\times 2M})$ $\displaystyle=\alpha\int DcD\xi_{11}D\xi_{12}\xi_{22}\min_{D,z_{11},z_{12},z_{22}}{\mathcal{L}}(\Theta,D;z_{11},z_{12},z_{22},\xi_{11},\xi_{12},\xi_{22}).$ $$

The extremum conditions are represented as the following equations:

$$ $\displaystyle 0$ $\displaystyle=\frac{\partial{\mathcal{L}}}{\partial z_{11}}=\sqrt{\eta(\chi-\omega)}\left(c\sqrt{\rho}d+\sqrt{\eta}\left(\sqrt{\zeta-\omega}z_{22}+\sqrt{\omega}z_{12}+\sqrt{E-R}\xi_{22}+\sqrt{R}\xi_{12}\right)\right),$ $\displaystyle 0$ $\displaystyle=\frac{\partial{\mathcal{L}}}{\partial z_{12}}=\sqrt{\eta\omega}\Bigg{(}(1-(Q+\beta_{\mathrm{VAE}}))\left(c\sqrt{\rho}d+\sqrt{\eta}\left(\sqrt{\zeta-\omega}z_{22}+\sqrt{\omega}z_{12}+\sqrt{E-R}\xi_{22}+\sqrt{R}\xi_{12}\right)\right)$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\left(c\sqrt{\rho}m+\sqrt{\eta}(\sqrt{\chi-\omega}z_{11}+\sqrt{\omega}z_{12}+\sqrt{Q-R}\xi_{11}+\sqrt{R}\xi_{12})\right)\Bigg{)}$ $\displaystyle 0$ $\displaystyle=\frac{\partial{\mathcal{L}}}{\partial z_{22}}=\sqrt{\eta(\zeta-\omega)}\Bigg{(}c\sqrt{\rho}m+\sqrt{\eta}\left(\sqrt{\chi-\omega}z_{22}+\sqrt{\omega}x_{12}+\sqrt{Q-R}\xi_{11}+\sqrt{R}\xi_{12}\right)$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-(Q+\beta_{\mathrm{VAE}})\left(z\sqrt{\rho}mz_{11}+\sqrt{\eta}\left(\sqrt{\zeta-\omega}z_{22}+\sqrt{\omega}z_{12}+\sqrt{E-R}\xi_{22}+\sqrt{R}\xi_{12}\right)\right)\Bigg{)},$ $\displaystyle 0$ $\displaystyle=\frac{\partial{\mathcal{L}}}{\partial D}=(Q+\beta_{\mathrm{VAE}})-\frac{\beta_{\mathrm{VAE}}}{D}.$ $$

Solving these equations yields the following expression:

$$ ${\mathcal{E}}=n\beta\Bigg{(}\frac{(Q-\eta\chi+\beta_{\mathrm{VAE}})(\rho d^{2}+\eta E)-\eta\zeta(\rho m^{2}+\eta q)+2(\eta\omega-1)(\rho md+\eta r)}{\eta\zeta(Q-\eta\chi+\beta_{\mathrm{VAE}})+(\eta\omega-1)^{2}}\\ +\beta_{\mathrm{VAE}}\left(\log\frac{e(Q+\beta_{\mathrm{VAE}})}{\beta_{\mathrm{VAE}}}\right)\Bigg{)}$ (25) $$

In the end, substituting Eq.  ([24](#S3.E24)) and ([25](#S3.E25)) into Eq. ([23](#S3.E23)), one can easily obtain

$$ $f=\underset{\Theta}{\mathrm{extr}}\Bigg{\{}-\frac{1}{2}(\hat{Q}Q-\chi\hat{\chi})-\frac{1}{2}(\hat{E}E-\zeta\hat{\zeta})-(\hat{R}R-\omega\hat{\omega})+\hat{m}m+\hat{d}d\\ +\frac{(\lambda+\hat{E})(\hat{m}^{2}+\hat{\chi})+(\lambda+\hat{Q})(\hat{d}^{2}+\hat{\zeta})-2\hat{R}(\hat{m}\hat{m}+\hat{\omega})}{2\hat{G}}-\frac{\alpha}{2}\Bigg{(}\frac{(Q-\eta\chi+\beta_{\mathrm{VAE}})(\rho d^{2}+\eta E)-\eta\zeta(\rho m^{2}+\eta Q)}{G}\\ +\frac{2(\eta\omega-1)(\rho md+\eta r)}{G}+\beta_{\mathrm{VAE}}\log\frac{e(Q+\beta_{\mathrm{VAE}})}{\beta_{\mathrm{VAE}}}\Bigg{)}\Bigg{\}},$ $$

### C.3 Derivation of Claim 5.1

From the expansion in the first order term with respect to $\alpha$, one obtains the following solution:

$$ $\displaystyle Q=E=R=\chi=\zeta=\omega=m=d=0~{}~{}~{}(\rho+\eta\leq\beta_{\mathrm{VAE}}),$ (26) $\displaystyle Q=\eta+\rho-\beta_{\mathrm{VAE}},~{}E=\frac{\eta+\rho-\beta_{\mathrm{VAE}}}{(\eta+\rho)^{2}},~{}\chi=\zeta=\omega=0,~{}m=\sqrt{\eta+\rho-\beta_{\mathrm{VAE}}},~{}d=\frac{\eta+\rho-\beta_{\mathrm{VAE}}}{\eta+\rho}~{}~{}~{}(\rho+\eta>\beta_{\mathrm{VAE}}).$ (27) $$

Note that one can evaluate $\lim_{\beta\to\infty}{\mathbb{E}}_{{\mathcal{D}}}{\mathbb{E}}_{p(W,V,D;{\mathcal{D}},\beta)}\varepsilon_{g}(W,W^{\ast})$ as $\rho-2\sqrt{\rho}m+Q$. Thus, one obtains

$$ $\displaystyle\epsilon_{g}=\begin{cases}\rho-\sqrt{\eta+\rho-\beta_{\mathrm{VAE}}}(2\sqrt{\rho}-\sqrt{\eta+\rho-\beta_{\mathrm{VAE}}})&(\rho+\eta\leq\beta_{\mathrm{VAE}})\\ 1&(\rho+\eta>\beta_{\mathrm{VAE}}\end{cases}.$ (28) $$

The optimal condition for $\beta_{\mathrm{VAE}}$ yields optimal value $\beta^{\ast}_{\mathrm{VAE}}=\eta$.

### C.4 Derivation of Claim 5.2

We first notice that rate and distortion can be expressed as

$$ $\displaystyle R_{\ast}={\mathbb{E}}_{{\mathcal{D}}}R(\bar{W}({\mathcal{D}}),\bar{V}({\mathcal{D}}),\bar{D}({\mathcal{D}}))=\frac{1}{2}\left(\rho d^{2}+\eta E+\frac{\beta_{\mathrm{VAE}}}{Q+\beta_{\mathrm{VAE}}}-1-\log\frac{\beta_{\mathrm{VAE}}}{Q+\beta_{\mathrm{VAE}}}\right),$ (29) $\displaystyle D_{\ast}={\mathbb{E}}_{{\mathcal{D}}}D(\bar{W}({\mathcal{D}}),\bar{V}({\mathcal{D}}),\bar{D}({\mathcal{D}}))=\frac{1}{2}\left(\rho+\eta-2(\rho md+\eta R)+Q\left((\rho d^{2}+\eta E)+\frac{\beta_{\mathrm{VAE}}}{Q+\beta_{\mathrm{VAE}}}\right)\right),$ (30) $$

respectively.
Then, substituting Eq. ([26](#S3.E26)) and ([26](#S3.E26)) into Eq. ([29](#S3.E29)) and Eq. ([30](#S3.E30)), one can obtain

$$ $\displaystyle R_{\ast}$ $\displaystyle=\begin{cases}\frac{1}{2}\log\frac{\eta+\rho}{\beta_{\mathrm{VAE}}}&\rho+\eta\leq\beta_{\mathrm{VAE}}\\ 0&\rho+\eta>\beta_{\mathrm{VAE}}\end{cases},$ $\displaystyle D_{\ast}$ $\displaystyle=\begin{cases}\frac{\beta_{\mathrm{VAE}}}{2}&\rho+\eta\leq\beta_{\mathrm{VAE}}\\ \frac{\rho+\eta}{2}&\rho+\eta>\beta_{\mathrm{VAE}}\end{cases}.$ $$

From these equations, one obtains

$$ $R(D^{\ast})=\begin{cases}\frac{1}{2}\log\frac{\rho+\eta}{2D_{\ast}}&0\leq D^{\ast}<\frac{\eta+\rho}{2}\\ 0&D^{\ast}\geq\frac{\rho+\eta}{2}\end{cases}.$ $$

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: - Can you comment on how much the analysis is effected by the fact that $D$ is fixed?
- Are the RD curves computed for the training set or test set? These two curves can be widely different.  
- Do the authors mean overfitting by "overlearning"? If yes, I would say replace it with overfitting to avoid confusion :) 
- this is not clear to me but how did the authors at the values for Figure 3 Left? This is not seem to match other figures.
soundness: 2
strengths: - To the best of my knowledge, this is the first paper that studied RD curves in VAEs as a function of dataset size and data dimensions. This topic I think is a valuable topic of study and will indeed be of interest to the ICLR community. 

- The theory in the paper, to the best of my understanding, is sound. 

- The paper for the most reads well.
summary: This paper studies the RD curves in VAEs from a function of dataset size and dimensionality. The authors suggest that the RD curves as a function of data complexity $\alpha$ (# data points / dim of data) and $$\beta, can be divided to three categories of overfitting, learning, and underfitting; In high $\alpha$ regime, smaller $\beta$ is needed in order to avoid over-regularizing the model.
weaknesses: - There is no study of the network capacity in this work. While I understand that this is theoretical work, the authors do make a claim that the same results hold for more complex networks. However, there are prior works that suggest that RD curves for different network capacities behave differently [1,2]. Could the authors comment on this?

- It is also not clear to me what is the message of the paper. It ofcourse makes sense that when you don't have a lotta data in high dimensions, you want to incorporate prior knowledge (such as regularization). Similarly,  when you have a lotta data, you don't need a lotta regularization as it is evident from all the recent DGMs. Furthermore, the $\alpha < 1$ is hardly interesting as it is almost never the case. So practically, what does this mean for people employing VAEs? What is the core message here?

- I do not think the experiments are strong enough to back the claims made in this paper. First, $\alpha$ should have been studied as a function of both $n$ and $d$ separately. Here, $d$ was kept fixed. Furthermore, the data choice here is extremely specific. I understand the design choice, but some controlled experiments on real-world datasets are also necessary before showing Figure 3 left with those specific values. 


[1] Bozkurt, Alican, et al. "Rate-regularization and generalization in VAEs." arXiv preprint arXiv:1911.04594 (2019).

[2] Chérief-Abdellatif, Badr-Eddine, et al. "On PAC-Bayesian reconstruction guarantees for VAEs." International conference on artificial intelligence and statistics. PMLR, 2022.





**Minor comments**

- "Notations " should not be place in Related work I would say
- I would strongly advise to avoid using $D$ for the variances in $q$ and use $\sigma^2$ instead as it is the most common symbol in the literature  for this.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. The signal recovery error feels like a definition of reconstruction error, and a distortion metric. What’s the difference between the signal recovery error and the distortion (D) of the RD curve in the paper?
2. Typo: Page 8 line 397: “summary statics” -> “summary statistics”
soundness: 3
strengths: - The paper studied an important aspect of VAEs and how the different parameters and choices can affect the performance.
- The empirical findings of the relation between generalisation error and the sample complexity as well as the beta parameter is interesting.
summary: The paper investigates several aspects of beta-linear-VAEs, including posterior collapse, the effect of different values for the beta parameters, the effect of training dataset size.
It also introduced two summary statistics, and using these statistics, the authors derived a phase diagram for VAE learning in terms of the beta values and the relative scale of the training dataset size.
weaknesses: The paper discussed a list of different behaviours of VAEs, but it feels like they are rather loosely connected findings (i.e., the subsections in Section 6).

The findings themselves are interesting, but it is not surprising that changing one variable, such as beta or the number of training data, will lead to various changes in aspects like RD curves, posterior collapse. 

Therefore, I believe a more coherent story is important to connect the dots and make these findings more insightful.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - What is the main challenge that the replica method allows you to get around? Would be nice to provide some insight into this. Or perhaps a toy example of the usage of replica method demonstrating what are its benefits and why is it used in this particular context.
- What is the main reason to introduce the metric $\varepsilon_g$ in Eq (9)? How is it different than the distortion $D$?
- In Figure 3, what does "overlearning" mean? From the description in section 6.2, it seems it is the same as overfitting? If so, would be good to name it that way instead of introducing a new term.
soundness: 3
strengths: **Technical strengths**:
- The paper sharply characterizes high-dimensional asymptotics for learning the linear VAE (Eq (5)) under the spiked covariance model (Eq (4)) with the regularized $\beta$-VAE objective (Eq (6)).
- This is used to show interesting observations about the VAE learning process in Section 6.1 and 6.2. In particular, (1) Figure 2 shows a double-descent phenomenon w.r.to the sample complexity $\alpha$, with the reconstruction error (Eq (9)) peaking at $\alpha = 1$, and (2) Figure 2 also shows a long plateau in the reconstruction error for large values of $\beta$. This is backed by Claim 6.1 (in the large $\alpha$ limit) and lines 428-430 provide concrete guidance to practitioners about the risks of a large $\beta$ when training.
- Section 6.5 (and Figure 5) shows this on real-world datasets MNIST and FashionMNIST also, where the insight can be used to practically choose the "optimal" value of $\beta$ approximately equal to the noise ratio $\hat{\eta}$, which can be estimated using the training dataset.

**Presentation strengths**:
- The paper is largely well-written and easy to follow. The authors include relevant explanations in most places. For example, the choice of the spiked covariance model as the synthetic data generating process was backed by evidence in Figure 1 of MNIST following something similar.
summary: This paper aims to analyze the solution learnt by a $\beta$-VAE w.r.t. (1) the parameter $\beta$, and (2) the training dataset size. The authors work in the high-dimensional asymptotic setting, and aim to characterize certain phenomena about the quality of the learnt solution by the VAE.

To do this, the authors theoretically analyze a linear VAE model (Eq (5)), in the high-dimensional asymptotic regime where $n,d \rightarrow \infty$ and $\frac{n}{d} = \alpha$ (sample complexity) stays finite, using the replica method as a heuristic to get around intractable calculations. This is presented in Section 5.

The asymptotic formulae are empirically verified in Section 6.1 and 6.2, on a synthetic data model of the spiked covariance matrix (Eq (4)). This is then used to draw interesting observations about the learning process and the quality of the learnt VAE solution. Figure 2 in particular shows many of the findings.

The authors then empirically show some of the findings (from the linear VAE setting) hold true for non-linear VAEs also, trained on real-world datasets like MNIST and FashionMNIST. This is presented in Section 6.5.
weaknesses: **Technical Weaknesses**:
- The main weakness is the fact that the theoretical results are not exact, since they have been developed using the replica method, which is a heuristic to get around intractable calculations.
- The authors work in the simple setting of $k = k^\star = 1$. If I understand correctly, this means the true latent space is $1$-dimensional. It would have been nice to see the synthetic experiments with $k^\star$ varying, say in $[1, 2, 4]$. In particular, what would the trend of $\varepsilon_g$ w.r.to $k^\star$ look like?
- Some of the claims can be better substantiated. For example, in the context of Figure 2, it would have been nice to see a plot of $\varepsilon_g$ w.r.to $\alpha$ for the optimal $\beta$ choice. Is that perhaps monotonically decreasing? (This is similar to the double-descent observations in literature, where using the optimal parameter leads to a monotonically decreasing curve instead of double-descent).

**Minor notes on the typos I found**
- Line 146, $H$ is probably the "negative log-likelihood" instead of just "likelihood". I stress this because it is important whether we want to minimize or maximize $H$.
- Line 189, "Spectrum" of the covariance matrix, instead of "Spectral". This typo is present in many places throughout the paper (for eg, Fig 1(b)), would appreciate if it can be cleaned up.
- Line 214, "Note that" instead of "Noted that".
- Line 224, $\lambda \in \mathbb{R}_{+}$ maybe instead of  $\lambda \in \mathbb{R}$? I *assume* practitioners use a non-negative regularization parameter.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Could you explore the performance of this model in transitional or non-asymptotic regimes to reflect real-world data scenarios?
2. Is your analysis still valid for datasets with complex spectrums?
3. Could you clarify the derivation in Equation (9), especially the expectation notation?
soundness: 2
strengths: • Information-Theoretic Perspective: A thoughtful analysis of β-VAE’s Rate-Distortion properties.
• Insight on Hyperparameter Tuning: Suggests an optimal β based on sample complexity, a novel angle.
• Focus on an Underexplored Hyperparameter: Thorough analysis of β’s role in posterior collapse.
summary: This paper explores the influence of the β hyperparameter in a β-VAE, particularly regarding its impact on the posterior collapse phenomenon and the Rate-Distortion (RD) curve, from an information-theoretic perspective. Using a Spiked Covariance Model (SCM), the authors identify three phases (regularized, learning, overlearning) based on β and sample complexity α = n/d . The results suggest that high values of β inevitably lead to posterior collapse, regardless of data volume, and propose that the optimal β correlates with noise
strength ˆη in high-complexity settings. Experiments on MNIST and Fashion-MNIST datasets partially validate these claims.
weaknesses: • Over-Simplified Model and Limited Scope: Most analysis centers on a linear VAE with low latent dimensions (e.g., dimension 1), which reduces the generalizability to real-world settings. Minimal insight into non-asymptotic or intermediate regimes.
• Limited Novelty: The core claim that “high β is detrimental” is already known. While it validates an intuition, the work lacks a new method or paradigm. Moreover, it is unclear that the posterior collapse phenomenon occurs ”often” as stated in the abstract, especially when the latent dimension increases.
• Minimal Experiments: Only FID scores on two datasets (MNIST, Fashion-MNIST), without broader validation on diverse or complex data,
and lacks sensitivity analysis (for small changes on the value).
• Identifiability of Ground Truth Model: As is, the ground truth model does not seem to be identifiable (it is when dividing by √θ). This hinders the reliability of the analysis.

Meta-review: In the paper, the authors rigorously examine the factors contributing to posterior collapse in variational autoencoders (VAEs), focusing on the influence of the hyperparameter beta and data size in VAEs. 

While there is a consensus among the reviewers that the theories are sound and of interest to ICLR, there are major concerns about the impact and scope of the study, including (1) the model being studied is linear VAE with low latent dimensions. While it is understandable that the simple setting is useful for obtaining useful theoretical insights, it is unclear how the theoretical findings can be generalized to the general settings of VAEs. (2) Several claims in the paper, such as the bad effect of the high value of the hyperparameter beta, had been studied before, which limits the novelty of the current theories in the paper. (3) The experiments are quite poor and minimal, which limits the scope of the theories in the real-world settings.

Given the above major concerns, I recommend rejecting the paper at the current stage. I believe that the paper will be stronger after incorporating the feedback and suggestions of the reviewers.

Decision: Reject