Title: AdaVAE: Bayesian Structural Adaptation for Variational Autoencoders
Venue: NeurIPS 2023
Accepted: yes
Keywords: nonparametric Bayes

variational autoencoders
Abstract: The neural network structures of generative models and their corresponding inference models paired in variational autoencoders (VAEs) play a critical role in the models' generative performance. However, powerful VAE network structures are hand-crafted and fixed prior to training, resulting in a one-size-fits-all approach that requires heavy computation to tune for given data. Moreover, existing VAE regularization methods largely overlook the importance of network structures and fail to prevent overfitting in deep VAE models with cascades of hidden layers. To address these issues, we propose a Bayesian inference framework that automatically adapts VAE network structures to data and prevent overfitting as they grow deeper. We model the number of hidden layers with a beta process to infer the most plausible encoding/decoding network depths warranted by data and perform layer-wise dropout regularization with a conjugate Bernoulli process. We develop a scalable estimator that performs joint inference on both VAE network structures and latent variables. Our experiments show that the inference framework effectively prevents overfitting in both shallow and deep VAE models, yielding state-of-the-art performance. We demonstrate that our framework is compatible with different types of VAE backbone networks and can be applied to various VAE variants, further improving their performance.

AdaVAE: Bayesian Structural Adaptation for Variational Autoencoders

Abstract

The neural network structures of generative models and their corresponding infenrence models paired in variational autoencoders (VAEs) play a critical role in the models' generative performance. However, powerful VAE network structures are hand-crafted and fixed prior to training, resulting in a one-size-fits-all approach that requires heavy computation to tune for given data. Moreover, existing VAE regularization methods largely overlook the importance of network structures and fail to prevent overfitting in deep V AE models with cascades of hidden layers. To address these issues, we propose a Bayesian inference framework that automatically adapts VAE network structures to data and prevent overfitting as they grow deeper. We model the number of hidden layers with a beta process to infer the most plausible encoding/decoding network depths warranted by data and perform layer-wise dropout regularization with a conjugate Bernoulli process. We develop a scalable estimator that performs joint inference on both V AE network structures and latent variables. Our experiments show that the inference framework effectively prevents overfitting in both shallow and deep V AE models, yielding state-of-the-art performance. We demonstrate that our framework is compatible with different types of VAE backbone networks and can be applied to various VAE variants, further improving their performance.

1 Introduction

The inference models and the generative models paired in variational autoencoders (V AEs) are commonly constructed with neural networks, i.e., encoding networks and decoding networks, respectively [1, 2, 3]. Extensive research efforts show that well-designed encoding/decoding network structures for VAE models can constantly achieve state-of-the-art generative performance compared to other generative models [4, 5, 6]. However, powerful VAE network structures are hand-crafted and fixed prior to training. The issue with fixed network structures is that shallow ones limit VAE models' expressiveness, whereas overly deep networks are slow to use and prone to overfitting. Traditional model selection by training different V AE network structures for given data is difficult since finding optimal hyperparameters for each candidate structure is a daunting task, and training large VAE structures requires significant computation. On the other hand, network structure adaptation methods for discriminative model settings [7, 8, 9] cannot be straightforwardly applied to address the unique challenge posed by VAE estimation along with the latent variables.

Although the network structures play a critical role in the performance of V AE models, they are largely overlooked by current VAE regularization methods. This renders their failure to prevent overfitting when the network structures grow deeper. Amortized inference regularization (AIR) proposes two approaches: injecting random noise to the VAE objective for inference or directly restricting the inference models to a set of smooth functions [10, 11]. Another approach in VAE regularization

(a)

(b)

(c)

Betaprocess

Betaprocess

8

8

Bernoulliprocess

11

πT

0

Encodingnetwork

Decodingnetwork

incorporates additional constraints to the V AE objective, which enforces a similarity between original input and its semantic preserving transformation in the latent representation [12, 13]. Nonparametric Bayesian inference such as Indian buffet process deep generative models (IBP-DGM) [14] and betaBernoulli process VAE (BB-VAE) [15] address overfitting by inferring the dimensionality of VAE latent variables using an Indian buffet process, specifically, a marginalized beta-Bernoulli process prior. While these regularization methods are effective for shallow network structures, they fail to prevent overfitting in deep V AE models.

We propose a novel Bayesian inference framework that automatically adapt VAE network structures by inferring the most plausible encoding/decoding network depths based on the given data, as demonstrated in Figure 1. To achieve this, we employ beta processes [16, 17] to model the number of hidden layers in the encoding/decoding networks, allowing for infinite depths. A conjugate Bernoulli process is utilized to prune the neurons in each layer based on layer-wise activation probabilities generated by the beta process. In addition, to enable efficient joint inference on both the network structures and the latent variables, we extend multiply importance weighted autoencoder (MIWAE) [18] by introducing an additional sample size used for Monte Carlo estimation of the network structures to be tuned. Our theoretical and empirical analysis shows that this novel gradient estimation scheme leads to a tight lower bound with high signal-to-noise ratio of parameter gradients.

In summary, our contributions are: i) We propose AdaVAE, a novel VAE structural adaptation strategy based on Bayesian model selection to enhance model performance. ii) We introduce a scalable estimator that facilitates joint inference on both encoding/decoding network structures and latent variables. iii) We conduct a comprehensive analysis of AdaVAE's regularization capabilities and demonstrate its ability to effectively mitigate overfitting in both shallow and deep V AE models and achieve state-of-the-art performance. iv) We showcase the versatility of AdaV AE by demonstrating its compatibility with different types of V AE backbone networks. It can also be readily applied to various VAE variants, thereby enhancing their performance.

2 Related Works

Variational autoencoders (VAEs) have gained popularity as generative models across a wide range of applications [19, 20, 21, 22, 23, 24]. Extensive research efforts have focused on enhancing the performance of VAEs [25, 26, 27, 28, 29, 30, 31, 32]. Notably, well-designed encoding/decoding neural network structures constantly yield state-of-the-art generative performance [4, 5, 6, 33]. Ladder-VAE (LVAE) employs a shared top-down dependency structure in both the inference and generative models to facilitate information sharing between layers [4]. SkipVAE employs skip connections that connect latent variables to each layer in the generative model, mitigating posterior

collapse [34]. BIVA consists of a skip-connected generative model and an inference model formed by a bidirectional stochastic inference path [5]. NV AE designs expressive neural network structures tailored for V AEs based on deep residual networks [6]. Furthermore, advancements in V AE network structure design have also empowered recent works on hybrid models based on VAEs [35, 36, 37, 38].

Current VAE regularization methods often overlook the overfitting effect caused by deep network structures [12, 10, 15, 11, 13]. Amortized inference regularization (AIR) re-interprets the amortized inference model as a regularization for maximum likelihood training. AIR encourages the smoothness of the encoding network to restrict the model capacity, effectively mitigating overfitting and improving predictive performance [10]. Specifically, AIR proposes a denoising variational autoencoder (DV AE) that modifies the V AE objective using random perturbation training. An alternative AIR technique is weight-normalized inference VAE (WNI-VAE) that directly restricts the encoding networks to a set of smooth functions and achieves comparable performance to DVAE. Both approaches exhibit a lower inference gap than standard VAEs [39]. Consistency regularization for VAE (CR-VAE) extends the regularization techniques employed in semi-supervised learning [40, 41] to tackle the inconsistency problem of the inference models by enforcing the latent representations of an image and its semantic-preserving transformation to be similar, thereby yielding robust latent representations accounting for data variation [12].

IBP-DGM [14] and BB-VAE[15] as nonparametric Bayesian inference applied to VAE regularization focus on inferring the dimensionality of latent variable via an Indian Buffet Process (IBP) prior. The IBP prior is derived by marginalization over a beta process, resulting in a binary vector that masks the VAE latent variables. Without accommodating network structures, latent variable regularization alone is inadequate for mitigating overfitting.

3 Efficient VAE Estimators

Maximum likelihood estimation of a vector of parameters θ of a deep generative model p θ ( x , h ) with x denoting observed variables and h denoting latent variables is intractable in general due to the marginalization over h . Amortized variational inference optimizes an evidence lower bound (ELBO) L θ,ϕ ( x ) on the log marginal likelihood log p θ ( x ) = log ∫ p θ ( x , h ) d h by introducing a variational distribution q ϕ ( h | x ) :

where ϕ denotes variational parameters. For V AEs, q ϕ ( h | x ) denotes an inference model commonly constructed with a neural network. p θ ( x | h ) denotes a generative model that can also be parameterized by a neural network. The ELBO is optimized with gradient-based methods via reparameterization trick using stochastic Monte Carlo estimators of ∇L θ,ϕ [1].

IWAE obtains a tighter lower bound using K -sample importance weighting estimate of the log marginal likelihood [42]:

where h k ∼ q ϕ ( h | x ) . It shows that the bound gets tighter with increasing K . However, [18] presents theoretical and empirical evidence that increasing the importance weighted sample size K to tighten the bound degrades signal-to-noise ratio (SNR) of parameter gradients estimates for the encoding network, and hurts the learning process. A new estimator (MIWAE) is thus introduced to address the issue of diminishing SNR, and its gradient estimate is:

where h m,k ∼ q ϕ ( h m,k | x ) . For a fixed budget M × K of total number of hidden variable samples, the number of samples M reduces the variance in estimating the ELBO gradient. K is the importance sample size as in IWAE.

4 VAE Structure Inference Framework

Traditional model selection cannot effectively adapt pre-determined V AE network structures to data without incurring significant computation overhead. We thus propose AdaVAE that enables joint inference on the structures of the encoding/decoding networks using stochastic processes [43, 9] and latent variables, and optimizes V AE objective without requiring additional expensive computation.

4.1 Formulation of the Inference Model

Let the prior over the latent variables h be a zero-mean isotropic multivariate Gaussian p θ ( h ) = N ( h ; 0 , I ) . We formulate the inference model by letting the variational distribution over h be a multivariate Gaussian with a diagonal covariance structure:

where the mean µ and the standard deviation σ are outputs of the encoding neural network F ϕ ( x ) with the variational parameters ϕ . The binary matrix Z = [ z ol ∈ { 0 , 1 } ] denotes the network structural variable, as in Figure 1(a).

Let f l denote the l -th hidden layer of F ϕ ( x ) composed of neurons (i.e., non-linear activation functions) f ( · ) . The encoding network F ϕ ( x ) has the form:

where ϕ = { W l ∈ R O × O } , and W l is the layer l 's weight matrix. ⊙ denotes element-wise multiplication of two vectors, so that we drop out the l -th layer's outputs by multiplying them elementwisely by the column vector z · l of Z . Each random variable z ol takes the value 1 with π l ∈ [0 , 1] indicating activation probability of the l -th layer, as in Figures 1 (b),(c). O is the maximum number of neurons in a layer that is set to be the same for all hidden layers. We have skip connections to propagate the outputs of the hidden layers to the output layer. Note that when this network structure is used as the decoding network for the generative model p θ ( x | h ) , then h and x are swapped, and θ denotes the weights parameters. The output-layer can be readily replaced with a logistic function and the normal distribution with a Bernoulli distribution for binary data.

4.2 Beta Process Prior over Layer Number

A beta process B = ∑ l π l δ f l , where δ f l is a unit point mass at f l , is a completely random measure over countably infinite set of pairs ( f l , π l ) [16], where f l ∈ F denotes a hidden-layer function and π l is its activation probability π l ∈ [0 , 1] . Its conjugate Bernoulli process can be defined as Z o · ∼ BeP ( B ) , where Z o · = ∑ l z ol δ f l is at the same locations δ f l as B where z ol are independent Bernoulli variables with π l being the probability of z ol = 1 . As in Eqn. (5), z ol = 1 activates the o 'th neuron in layer l . Computationally, we employ the stick-breaking construction [17] of beta process and its conjugate Bernoulli process as

where ν l are sequentially drawn from a beta distribution. The hyperparameters α and β can be set to balance the network depth and width. Specifically, Figure 1 (b) demonstrates that if β > α > 1 , the network structure prior favors shallower but wider network structures with the first few layer-wise activation probabilities being high. If α > β > 1 , activation probabilities tend to be low over larger number of active layers, and the prior prefers a deeper but narrower network, as in Figure 1 (c).

We thus define the prior over the encoding network structural variable Z as

To enable asymmetric encoding/decoding network structures, we independently apply the prior to both networks. Further analysis on symmetric constraints can be found in the Appendix.

4.3 Joint Inference on VAE Network Structures and Latent Variables

We first expand the overall marginal likelihood over the V AE structures Z as

where the first RHS term denotes a Kullback-Leibler (KL) divergence of the approximate variational distribution from the true posterior of the V AE network structural variables. We specify the variational distribution as

where π t = ∏ t j =1 ν j , and { a t , b t } T t =1 are the variational parameters. T denotes a truncation level for the maximum number of hidden layers [17]. We also relax the constraint of the discrete variables by reparameterizing the Bernoulli distribution into a concrete Bernoulli distribution ConBer ( z ot | π t ) [44, 45]. This allows us to efficiently backpropagate the parameter gradients of the estimator while generating network structure samples.

The second RHS term in Eqn. (8) denotes the ELBO to the overall marginal likelihood:

The term log p θ ( x | Z ) in Eqn. (10) is the marginal likelihood over the latent variable h , which is an extension of Eqn. (1), in terms of conditioning on the structure variable Z . Thus, the ELBO to the marginal likelihood log p θ ( x | Z ) is:

Lemma 1 Let Q ( h | x ) = ∫ q ϕ ( h | x , Z ) q ( Z , ν ) d Z d ν be the variational distribution of the latent variable h marginalizing over Z , then

Lemma 1 indicates that the overall ELBO we derived on the left-hand side in Eqn. (12) bounds the lower bound to the marginal likelihood over h . The proof is in the appendix. In particular, q ( Z , ν ) is essentially a non-explicit mixing distribution [46]. It allows the variational distribution Q ( h | x ) to take complex form, and results in more informative latent representation.

We adopt Monte Carlo estimation of the expectations over both Z in Eqn. (10) and h in Eqn. (11) to estimate the overall ELBO. In particular, we extend the MIWAE estimator in Eqn. (3), and introduce three sample sizes to tune: the number of samples S used for Monte Carlo estimation of the expectation over the VAE network structure variable Z , the number of samples M s used for Monte Carlo estimation of the gradient of the latent variable ELBO conditioned on the structure samples in Eqn. (11), and the number of importance samples K s used for estimation of the expectation over the latent variables h . We thus express our gradient estimate in the general form as

where h m,k ∼ q ϕ ( h | x , Z ) . When S = 1 our estimator becomes equivalent to the MIWAE objective in Eqn. (3). Since our estimator generates latent variable samples conditioned on network structure samples, increasing S will not impact the SNR of the gradient estimator in Eqn. (13).

Theorem 1 Let L S be the lower bound with S structure samples of Z s ∼ q ( Z , ν ) , then:

Epoch:1

No.of activated layers

1

12

Encoding network

EncodingNetwork

DecodingNetwork

EncodingNetwork

DecodingNetwork

Decoding network

1

14

1

T1

T1

T1

h

0

0

0

4

10

6

0

Activated neuron(%)

0

20

40

6080100120140

2

Epochs

Epoch:100

160

S = 1

EncodingNetwork

DecodingNetwork

1

EncodingNetwork

DecodingNetwork

1

140

+

S =2

S = 4

→

S = 8

T1

T1

T1

1

77-

120

h

0

0

0

Q

100

80

20

40

6080100120140

Epochs

Proof of this theorem is in the appendix. The theorem shows the convergence of our estimator. Specifically, increasing S leads to a tighter lower bound for the overall marginal likelihood. Training an encoding/decoding network with depth L and width M , the time complexity is T c = O ( NBLM 2 ) with N training examples and B epochs. Our method is linearly scalable as ST c . With a proper thresholding, the number of active layers L is relatively small in each sample.

5 Experiments

We analyze the behavior of our inference framework across various tasks. We study how AdaVAE facilitates the evolution of encoding/decoding network structures for inferring the most plausible depth from the given data, while generating expressive latent representations. Next, we explore the impact of the structure sample size S on the convergence of the proposed estimator in Eqn.(13). Then we show that AdaVAE effectively mitigates overfitting in both shallow and deep network settings, leading to state-of-the-art performance on benchmark datasets. Finally, we demonstrate the framework's compatibility with different types of backbone networks and VAE variants. 2

5.1 Adaptive VAE Network Structures

AdaVAE enables us to perform joint inference on both encoding/decoding network structures and latent variables. To investigate how network structures evolve during training epochs, we set the truncation level T = 25 on MLP backbone nets with tanh non-linearities. We analyze adaVAE's behavior on 28 × 28 binarized MNIST images [47], employing structure sample sizes S = { 1 , 2 , 4 , 8 } . We run the experiments 3 times and averaging the outcomes. Figure 2 Left shows the evolution of the encoding/decoding network structures for one trial with S = 8 . AdaVAE initializes multiple hidden

MNIST

Omniglot

Caltech101

200

MIWAE

MIWAE+DO

180

MIWAE

MIWAE+DO

350

MIWAE

MIWAE+DO

180

IBP-DGM

IBP-DGM

IBP-DGM

160

CR-VAE

DVAE

→

160

DVAE

CR-VAE

→

300

CR-VAE

DVAE

140

Ours

7

L

140

Ours

250

Ours

120

200

120

100

150

80

100

100

A

0

5

10

15

20

25

0

5

10

15

20

25

0

5

10

15

20

25

Layers(T/L)

Layers(T/L)

Layers(T/L)

layers with sparsely activated neurons, gradually converging to fewer fully activated layers. Figure 2 right top presents the medians of the number of active hidden layers in the encoding/decoding networks, as well as the percentage of activated neurons in the truncation changing over epochs. The encoding network structure stabilizes at two active layers, while the decoding network settles at six active layers. The decoding network tends to have more active layers compared to the encoding network on Omniglot as well (see Appendix). Figure 2 right bottom shows the generative performance assessed by negative log-likelihood ( -LL ) converge faster with an increased structure sample size S , which is consistent with Theorem 1 .

5.2 Effect of Structure Sample Sizes

We assess our estimator using three benchmark datasets: MNIST [47], Omniglot [48], and Caltech101 Silhouettes [49]. In each minibatch, we set a budget of S × M s × K s = 64 total latent variable samples for each datapoint. We examine four settings of the V AE structure sample size S = { 1 , 2 , 4 , 8 } , along with latent variable sample sizes ( M,K ) = { (8 , 8) , (4 , 16) } as in [18]. The truncation level is T = 25 with a maximum width O = 200 . The distribution over the output from the decoding networks is factorized Bernoulli. Table 1 indicates larger values of S generally yield better performance. For MNIST, there is no statistically significant difference between S = 1 and S = 8 . Among the two structure sample sizes, the best importance sample configuration is ( M,K ) = (8 , 8) .

5.3 On Preventing Overfitting

To assess the performance of our method across varying truncation level T , we compare with existing VAE regularization methods: Denoising VAE (DVAE) [10], CR-VAE [12], and BB-VAE [15], MIWAE with dropout (MIWAE+DO), along with vanilla MIWAE [18]. as in Figure 3. All methods share the same maximum width of O = 200 and a latent variable dimensionality of 50 . For

BB-VAE

Ours

DVAE

Ours+DVAE

CR-VAE

Ours+CR-VAE

MNIST

Omniglot

Caltech101

15

Active Layers (Encoder)

25

Activated neuron(%)

20

Active Layers (Encoder)

Activated neuron(%)

6

Active Layers(Encoder)

Activated neuron(%)

Active Layers

12

Active Layers(Decoder)

Layers

Active Layers (Decoder)

25

Layers

Active Layers (Decoder)

25

20

15

9

20

6

20

15

Active

10

Active

6

15

15

3

3

10

5

10

10

Activated neuron(%)

Activated neuron(%)

Activated neuron(%)

0

50

100

150

200

250

0

50

100

150

200

250

0

50

100

150

200

250

IBP-DGM and BB-VAE, we set the maximum dimensionality of the latent variables to 100 , following the setup of [15]. In Figure 3, we set depth L for the regularization methods and truncation level T for our method over the range L = T = { 1 , 2 , 3 , 4 , 5 , 10 , 15 , 20 , 25 } , allowing us to compare their effectiveness in both shallow (i.e., L = T ≤ 5 ) and deep (i.e., L = T ≥ 5 ) VAE network structures.

Figure 3 shows that when L = T ≤ 5 , the VAE regularization methods can mitigate overfitting. However, even for the shallow cases as L = T = { 4 , 5 } , the performance of these methods is affected by overfitting, as evidenced by the 'U'-shaped performance curves, indicating the classical variance-bias trade-off. Despite our method exhibiting slight underfitting issues for L = T ≤ 2 , it still outperforms other methods. For deep VAE structures with T/L = { 10 , 15 , 20 , 25 } , Figure 3 shows that our method's performance is minimally impacted by large truncations. This robustness indicates our method's ability to mitigate overfitting for large V AE structure settings. In contrast, as the depth L increases, the steadily increasing negative log-likelihood ( -LL ) for other regularization methods suggests their inability to prevent overfitting in deep VAE models. Overall, AdaVAE consistently delivers superior performance for both shallow and deep V AE structures. An analysis of the computational time required by our framework and its comparison to the baselines is presented in the Appendix.

Table 2 highlights our superiority over other methods in terms of density estimation and mutual information (MI) [50]. Additionally, when incorporating our framework with DVAE through the addition of Gaussian noise to the input or with CR-V AE by integrating the consistency regularization term into our estimator, we observe enhanced performance on Omniglot and Caltech101 datasets. Furthermore, in Figure 4, we visualize the latent representations of the V AE methods. DV AE and CRVAE when combined with our framework result in well-separated clusters, indicating that application of our framework allows VAEs to learn a meaningful latent representation. A quantitative evaluation of the latent representations is presented in the Appendix by analyzing the performance of VAE methods on a downstream classification task.

5.4 Effects of the Maximum Width O

We further investigate the influence of the width O (i.e., the maximum number of neurons per layer). Figure 5 shows the evolution of the medians of the number of active hidden layers (i.e., the hidden layers with activated neurons) as O increases. When O ≤ 100 , we tend to have shallower

(a)

(b)

(c)

(d)

MNIST

Omniglot

Citeseer

300

CVAE

250

CVAE

250

Ours+cVAE

220

OurS+cVAE

0.9

0.9

7

→

200

β-CVAE

Ours+β-cVAE

→

190

160

β-CVAE

OurS+β-cVAE

C

↑

0.7

0.8

GAE

C

↑

0.7

0.8

GAE

150

130

0.6

VGAE

0.6

VGAE

100

Ours+VGAE

Ours+VGAE

100

0

5

10

15

20

25

0

5

10

15

20

25

0.5

2

4

6

8

10

0.5

2

4

6

8

10

Layers(T/L)

Layers(T/L)

Layers(T/L)

Layers(T/L)

encoding/decoding networks with a lower percentage of activated neurons in the truncation. However, when O is reasonably large as O ≥ 100 , it has no influence on the network depth. In particular, the percentage of activated neurons in the whole truncation also remains relatively stable. This suggests that our method can automatically balance network depth and width to maintain the best performance.

5.5 Application to VAE Backbone Networks

We demonstrate adaVAE's efficacy by applying it to VAEs with different types of encoding/decoding backbone networks. To infer the number of convolutional layers in a convolutional VAE (cVAE) using the beta process, we readily mask the convolution channels in layer l with z · l . Figure 6 (a) and (b) show that by adapting network structures to the data we improve the overall performance of cV AE and β -cVAE [51]. The adaptive backbone networks effectively prevent overfitting for deep structure settings (i.e., T/L ≥ 15 ).

Variational graph autoencoder (VGAE) [52] encodes graph nodes to latent embeddings with graph convolutional (GC) layers and re-creates the input graph from the embeddings by predicting the existence of an edge between the nodes. We combine our framework with VGAE by elementwisely multiplying the GC channels (i.e., the feature vectors) of layer l with z · l in both its encoding/decoding networks. We compare the hybrid method's performance with graph autoencoders (GAEs) [52] and vanilla VGAEs on two benchmark graph-structured datasets: Cora [53] and Citeseer [54]. The AUC scores of link prediction over varying numbers of GC layer settings are shown in Figure 6 (c) and (d). Our framework enables the VGAE to maintain its best performance for all the network depth settings by automatically adapting its structures to the data, whereas the performance of GAE and vanilla VGAE drops with the increase of the layer numbers (e.g., T/L = { 6 , 8 , 10 } ).

5.6 Application to VAE Variants

We assess the performance of our inference framework by leveraging it to adapt the network structures of VAE variants to data. Specifically, for β -VAE, we apply layer-wise binary masks z · l to the convolutional channels and infer the layer numbers using the beta process. In the case of LadderVAE (LVAE) [4], we adjust its depth by applying layer-wise binary masks to its deterministic layers in the bottom-up dependency structures and add

skip connections between the stochastic layers. For SkipV AE [34], we model its depth by employing layer-wise binary masks and skip connections in both its encoding/decoding networks. The expressive network structures of NVAE [6] consists of multiple blocks of convolutional layers. We apply our

framework to infer the number of blocks for a light NVAE version without top-down dependency between the blocks. The detailed settings of these V AE variants and additional results can be found in Appendix. The results in Table 3 on FashionMNIST [55] demonstrate that by inferring the encoding/decoding network structures we significantly improve the density estimation performance of the VAE variants. Our framework also boosts their ability to mitigate posterior collapse as indicated by MI and KL divergence.

6 Conclusion

We present a Bayesian inference framework and a scalable estimator that automatically adapts V AE network structures to data. The experiments demonstrate its effectiveness in preventing overfitting for both shallow and deep structure settings. Moreover, AdaVAE exhibits promising applications across various types of V AE backbone networks and VAE variants, including those with hierarchical structures such as LVAE. Notably, AdaVAE enhances the generative performance of these models without requiring pre-determined network structures prior to training. Our future work entails relaxing the constraint of truncation levels by incorporating the Russian roulette method [56] and scaling up the inference for large images.

References

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR) , 2014.

Danilo J. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML) , pages 1278-1286. PMLR, 2014.

Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Advances in Neural Information Processing Systems , volume 34, pages 480-493, 2021.

Casper Kaae S, Tapani Raiko, Lars Maalø e, Søren Kaae Sø nderby, and Ole Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems (NeurIPS) , volume 29, 2016.

Lars Maalœ, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 19667-19679, 2020.

Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. AdaNet: Adaptive structural learning of artificial neural networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 874-883. PMLR, 2017.

Konstantinos Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric Bayesian deep networks with local competition. In International Conference on Machine Learning , pages 4980-4988. PMLR, 2019.

Kishan KC, Rui Li, and Mohammad Mahdi Gilany. Joint inference for neural network depth and dropout regularization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, 2021.

Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer, and Stefano Ermon. Amortized inference regularization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning (ICML) , pages 1096-1103, 2008.

Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, 2021.

Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In International Conference on Machine Learning (ICML) , 2011.

Sotirios P Chatzis. Indian buffet process deep generative models for semi-supervised classification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2456-2460. IEEE, 2018.

Rachit Singh, Jeffrey Ling, and Finale Doshi-Velez. Structured variational autoencoders for the betabernoulli process. In NIPS 2017 Workshop on Advances in Approximate Bayesian Inference , 2017.

Romain Thibaux and Michael I Jordan. Hierarchical beta processes and the Indian buffet process. In Artificial Intelligence and Statistics (AISTATS) , pages 564-571. PMLR, 2007.

John W Paisley, Aimee K Zaas, Christopher W Woods, Geoffrey S Ginsburg, and Lawrence Carin. A stick-breaking construction of the beta process. In International Conference on Machine Learning (ICML) , 2010.

Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. In International Conference on Machine Learning (ICML) , pages 4277-4285. PMLR, 2018.

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230 , 2018.

Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Chris C Holmes. Multi-facet clustering variational autoencoders. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 8676-8690, 2021.

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Abhishek Kumar and Ben Poole. On implicit regularization in β -VAEs. In International Conference on Machine Learning (ICML) , pages 5480-5490. PMLR, 2020.

Dazhong Shen, Chuan Qin, Chao Wang, Hengshu Zhu, Enhong Chen, and Hui Xiong. Regularizing variational autoencoder with diversity and uncertainty awareness. arXiv preprint arXiv:2110.12381 , 2021.

Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 29246-29257, 2021.

Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research (PMLR) , pages 1086-1094. PMLR, 2018.

Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In Proceedings of the 35th International Conference on Machine Learning (ICML) , volume 80 of Proceedings of Machine Learning Research , pages 3403-3412. PMLR, 10-15 Jul 2018.

Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In Proceedings of The 33rd International Conference on Machine Learning (ICML) , volume 48 of Proceedings of Machine Learning Research (PMLR) , pages 324-333, 20-22 Jun 2016.

Yingzhen Li and Richard E Turner. Rényi divergence variational inference. In Advances in Neural Information Processing Systems (NeurIPS) , volume 29, 2016.

Arash Vahdat, Evgeny Andriyash, and William Macready. Dvae#: Discrete variational autoencoders with relaxed boltzmann priors. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. In 7th International Conference on Learning Representations, (ICLR) 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019.

James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the elbo! a linear vae perspective on posterior collapse. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchical priors in vaes. In Advances in Neural Information Processing Systems , volume 32, 2019.

Jakob D. Havtorn, Jes Frellsen, Søren Hauberg, and Lars Maaløe. Hierarchical VAEs know what they don't know. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning (ICML) , volume 139 of Proceedings of Machine Learning Research (PMLR) , pages 4117-4128, 18-24 Jul 2021.

Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. Avoiding latent variable collapse with generative skip models. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (AISTATS) , Proceedings of Machine Learning Research, pages 2397-2405, 2019.

Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS) , volume 29, 2016.

Vaden Masrani, Tuan Anh Le, and Frank Wood. The thermodynamic variational objective. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems , volume 32, 2019.

Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems , volume 34, pages 11287-11302, 2021.

Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. In International Conference on Machine Learning (ICML) , pages 1078-1086. PMLR, 2018.

Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems (NeurIPS) , volume 27, 2014.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS) , volume 29, 2016.

Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016.

Tamara Broderick, Michael I Jordan, and Jim Pitman. Beta processes, stick-breaking and power laws. Bayesian Analysis , 7(2):439-476, 2012.

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 , 2016.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference on Machine Learning (ICML) , pages 5660-5669. PMLR, 2018.

Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE , volume 86 (11), page 2278-2324, 2014.

Brenden M. Lake, Russ R. Salakhutdinov, and Josh Tenenbaum. One-shot learning by inverting a compositional causal process. In Advances in Neural Information Processing Systems (NeurIPS) , 2013.

Benjamin Marlin, Kevin Swersky, Bo Chen, and Nando Freitas. Inductive principles for restricted boltzmann machine learning. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 9, pages 509-516. PMLR, 13-15 May 2010.

Matthew D Hoffman and Matthew J Johnson. ELBO surgery: Yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NeurIPS , volume 1, 2016.

Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval , 3:127-163, 2000.

C Lee Giles, Kurt D Bollacker, and Steve Lawrence. CiteSeer: An automatic citation indexing system. In Proceedings of the third ACM Conference on Digital Libraries , pages 89-98, 1998.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Kai Xu, Akash Srivastava, and Charles Sutton. Variational russian roulette for deep bayesian nonparametrics. In International Conference on Machine Learning (ICML) , pages 6963-6972. PMLR, 2019.

Research Computing at Rochester Institute of Technology. https://doi.org/10.34788/0S3G-QD15 .

A Appendix

This section is in the supplemental material.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a structure adaptation algorithm specifically tailored to the well-known Variational Autoencoders (VAEs). The main motivation lies in the fact that the structure of a generative model plays a significant part in the overall performance, something that is very under-explored in related VAE literature. To this end, the authors turn to the solid Bayesian inference framework and utilize a structural adaptation approach based on the Beta Process to infer the optimal network depth and the Bernoulli process to prune neurons in each hidden layer. The experimental evaluation focuses on the adaptation abilities of the proposed framework, how the structure sample size affects convergence and its over-fitting prevention properties. Finally the authors discuss how to integrate the proposed approach on VAE backbones and other VAE variants in general.\n\nSTRENGTHS:\nThis work focuses on a very interesting aspect of modern architectures, that is their structure. The paper is overall easy to follow. The notation is clear and almost everything is well defined.\nBy now there has been a lot of work in the community that aim to address this challenge through different views such as pruning approaches, component omission mechanisms, e.t.c.  Compared to most of existing approaches the above formulation allows for simultaneously inferring both the network depth and width of each layer in a principled way via Bayesian arguments.\n\nWEAKNESSES:\n1) As the authors note, the idea of using a Beta-Bernoulli pair in order to adapt the model capacity is not something new. There exist several works in both discriminative and generative models that aim to tackle this issue [1,2,3,4,5]. The authors however fail to appropriately introduce and discuss the differences with and advantages/drawbacks compared to these different methods leading to a single two sentence mention of one of the most similar approaches, i.e., BB-VAE. Instead the authors seem to focus more to the regularization aspect of other dissimilar methods. \n\n2) In this context, the work presented in [4] should be the main focus on comparison because it is essentially the same exact method, with minor adaptations for the VAE structure. Apart from the difference in estimation, i.e., Gumbel Softmax vs MIWAE, the differences are minor and down to some notation changes. The authors do cite the paper at some point  without any kind of discussion. \n\n3) Taking into consideration that the method is very similar to [4], a core difference is the different estimator. Did the authors investigate the impact of other estimation methods? How does it compare to Gubmel Softmax in terms of both complexity and convergence?\n\n4) The authors should briefly expand on the chosen metrics in the respective tables. The negative log likelihood is a well known measure, while the MI and KL divergence can have substantially different interpretations. The authors cite the work of [6], which dissects the ELBO into three different terms that probably correspond to the Tables 2,3. Further clarifications for the reader are essential. For example, why is the largest KL the better? The purpose of optimization is the minimization of the KL divergence and in [6] it is also noted that \"whenever it is large it indicates a very strong and potentially unwanted regularization effect from the prior\"\n\n5) The stability of the training process might be an issue that isn't addressed in the main text, especially in the stick-breaking construction. Did issues arise during training due to very small or very large values? and how did the authors address them?\n\n6) There is no discussion about the computational and memory complexity of the approach. It is apparent that the introduction of the additional latent parameters and the KL terms will significantly contribute to the overall footprint of the method, especially with multiple samples due to the MIWAE estimation. Some wall time measurements are necessary for both training and inference. \n\n7) There is not a clear definition of the prediction process of the framework. Do you draw multiple samples from the learned posteriors and sample the active ones? do the authors use a threshold in \"each sample\"? (line 182)\n\n8) The performance of the method on different VAR backbones is not clear. The authors mention the masking of convolutional channels in order to infer the number of convolutional layers, without any further expansion on the specific formulation and results. Does adaptation take place in specific features maps similar to [2] or the options are to drop the layer or not?\n\n9) Apart from the t-SNE visualization, some reconstruction comparisons could be useful. \n\nConsidering these points and the similarity of this work to [4], I believe that the novelty and overall contribution of this work is very limited for publication. \n\n[1] Sotirios P Chatzis. Indian buffet process deep generative models for semi-supervised classification. In 2018\nIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)\n\n[2] Konstantinos Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric bayesian deep networks with\nlocal competition. In International Conference on Machine Learning, pages 4980–4988. PMLR, 2019.\n\n[3] Rachit Singh, Jeffrey Ling, and Finale Doshi-Velez. Structured variational autoencoders for the beta-\nbernoulli process. In NIPS 2017 Workshop on Advances in Approximate Bayesian Inference, 2017.\n\n[4] Kishan KC, Rui Li, and Mohammad Mahdi Gilany. Joint inference for neural network depth and dropout\nregularization. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.\n\n[5] Xu, W., Chen, R., Li, X., & Duvenaud, D. (2022). Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations. International Conference on Artificial Intelligence and Statistics.\n\n[6] Matthew D Hoffman and Matthew J Johnson. ELBO surgery: Yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NeurIPS, volume 1, 2016.\n\nQUESTIONS:\nPlease see the Weaknesses section.\n\nLIMITATIONS:\nPlease see Weaknesses, especially concerning complexity and stability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method called AdaVAE that adapt the sizes (depth and width) of the inference and the generative networks in VAEs.\nIt extends the idea of beta-Bernoulli process VAE to use a beta process over the expected number of activated neurons over depth and Bernoulli processes per layer for the actual activations, followed by a dense Gaussian latent layer.\nThe paper develops an estimator based on MIWAE to learn AdaVAE. \nResults show that AdaVAE can prevents overfitting and lead to performance improvement than previous regularization methods for VAEs.\n\nSTRENGTHS:\n### originality\n\nThe proposed AdaVAE for adapting VAE sizes is novel.\n\n### quality\n\nThe proposed method is technically sound.\n\n### clarity\n\nThe paper did a good presentation of the proposed methods with technical details, which is easy to follow.\n\n### significance\n\nThe proposed method is generally applicable to different VAE architecture and is potentially useful as a standard regularization technique.\n\nWEAKNESSES:\n### originality\n\nThe proposed estimator is a straightforward combination of previous works.\n\n### clarity\n\nThe paper is very related to [11,36,54] but only limited discussion is given.\nI encourage the author(s) to give some discussions between the AdaVAE and share intuitions of why those previous works (that only adapts the size of the latent layer) are not enough.\n\nSome of the figures with small legends are a bit hard to read.\n\nFigure 4 is purely qualitative. \nIt's hard to convince the readers that one method yields better representation than others.\nIt looks like the figure it trying to give an idea on how different clusters are separated.\nIf it is the case maybe some clustering metrics could be used to assess the quality quantitatively.\n\n### significance\n\nThe proposed method can be still computationally heavy to use, compared to a naive one without any structure adaptation.\n\nQUESTIONS:\nIn Figure 2, the last few layers of the encoder and the first few layers of the decoder seem to have no neuron activated.\nCan the author(s) clarify if it is the case and if so why the VAE still works in this case?\n\nDo we have any computation-wise comparison between the proposed methods and alternative/naive ones?\nIt is useful to address practical concerns of actually using the method.\n\nLIMITATIONS:\nThe paper mentions its limitations on the fixed truncation level and some future work on removing it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Bayesian structural adaptation framework that automatically adapts VAE network structures (both encoder and decoder) to the current data.  By modeling the number of hidden layers as a beta process and performing layer-wise dropout regularization with the conjugate Bernoulli process, the proposed model develops a joint ELBO that can optimize all parameters (network structures and latent variables) via the SGD algorithm. Empirical studies are conducted on three visual datasets and two graphical datasets. They also provide extensive ablation studies to show the robustness.\n\nSTRENGTHS:\n(1) The paper addresses one of the common issues in VAE communities. The developed AdaVAE attempts to prevent the overfitting issues by jointly learning the network structures and latent variables under the Bayesian framework. This motivation sounds good, and the novelty of the method is generally ok.\n\n(2) The paper is written clearly and the empirical results show the improvement of the proposed model.\n\nWEAKNESSES:\n(1) One of the main concerns comes from the experiment section. The proposed model is only tested on several simple network structures (the number of layer usually less than 25). Additional results on more expressive neural network structures (such as BIVA and NVAE) will improve the quality of this paper.\n\n(2) One of the goal of the proposed model is to prevent the overfitting by adapting the network structures under the Bayesian framework. The authors compared the proposed model with existing VAE regularization methods, and the results show the improvements. Unfortunately, other baselines (such as network structure search methods [1]) are not included in the experiments.\n\n[1] Corinna Cortes et.al. AdaNet: Adaptive Structural Learning of Artificial Neural Networks.\n\nQUESTIONS:\n(1) I suggest that the authors add a training algorithm, which can help the reader understand the algorithm more easily.\n\n(2) From Fig(6) (a) (b), we find that AdaVAE have similar results with cVAE. Can the authors provides deeply analysis?\n\n(3) Given that the proposed AdaVAE aims to find optimal structure for the dataset at hand, it is advisable to include a comparison of network parameters in Table 2.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel VAE structural adaptation strategy called AdaVAE based on Bayesian model selection to enhance model performance. It introduces a scalable estimator that facilitates joint inference on both encoding/decoding network structures and latent variables. The paper conducts a comprehensive analysis of AdaVAE's regularization capabilities and demonstrates its ability to effectively mitigate overfitting in both shallow and deep VAE models and achieve state-of-the-art performance. The versatility of AdaVAE is showcased by demonstrating its compatibility with different types of VAE backbone networks. It can also be readily applied to various VAE variants, thereby enhancing their performance.\n\nThe main contributions are:\n- Proposes AdaVAE, a novel VAE structural adaptation strategy based on Bayesian model selection to enhance model performance.\n- Introduces a scalable estimator that facilitates joint inference on both encoding/decoding network structures and latent variables.\n- Conducts a comprehensive analysis of AdaVAE's regularization capabilities and demonstrates its ability to effectively mitigate overfitting in both shallow and deep VAE models and achieve state-of-the-art performance.\n- Showcases the versatility of AdaVAE by demonstrating its compatibility with different types of VAE backbone networks.\n- Can be readily applied to various VAE variants, thereby enhancing their performance.\n\nSTRENGTHS:\n1. The authors introduce a novel Variational Autoencoder (VAE) structural adaptation strategy, dubbed AdaVAE, which employs Bayesian model selection as a mechanism to enhance model performance. This innovative approach pushes the boundaries of current practices in the field and sets a precedent for future explorations.\n\n2. The study further contributes by proposing a scalable estimator. This facilitates joint inference on not only the structures of the encoding and decoding networks but also the latent variables. This dual focus enhances the model's applicability and comprehensiveness, potentially opening new avenues in inferential methodologies.\n\n3.A thorough analysis is presented on AdaVAE's regularization capabilities, showcasing its efficacy in mitigating overfitting across both shallow and deep VAE models. This is a critical point, as it demonstrates the proposed method's capability to achieve state-of-the-art performance across varying levels of complexity.\n\n4. The versatility of AdaVAE is effectively demonstrated, as the authors show its compatibility with different types of VAE backbone networks. This level of adaptability reinforces the model's potential for broad application and usability within the field.\nThe authors execute a rigorous evaluation of the proposed method on three benchmark datasets: MNIST, Omniglot, and Caltech101 Silhouettes. This broad evaluation allows a comprehensive understanding of AdaVAE's performance and robustly positions it in relation to other state-of-the-art methods.\n\n5. The paper includes a qualitative evaluation of the latent representations, an element that strengthens the argument for the proposed method's compatibility with VAE regularization methods. This kind of analysis enhances the empirical rigor of the study and provides an additional perspective on the utility of AdaVAE.\n\n6. The manuscript offers a theoretical analysis of the proposed method and derives a tight lower bound with a high signal-to-noise ratio for parameter gradients. This theoretical grounding ensures the model is not only empirically valid but also theoretically sound.\n\n7.The manuscript is particularly well-executed in terms of its structure and style. The clear writing, logical organization, and the authors' ability to elucidate complex concepts makes it accessible and easy for readers to grasp the proposed method and its evaluation.\n\nIn essence, the paper is a substantial contribution to the field, demonstrating methodological innovation, theoretical robustness, and a thorough evaluation, all of which underscore the potential and value of AdaVAE within the realm of VAE structural adaptation.\n\nWEAKNESSES:\nThis manuscript is laudable for its comprehensive coverage, including a lucid motivation, rigorous theoretical proofs, and a thorough comparative analysis with benchmark models across various datasets. It demonstrates a high level of academic rigor and makes a compelling case for the proposed model.\n\nHowever, I do have a minor suggestion to enhance the paper further. While the evaluation conducted on Caltech101, among other datasets, certainly contributes to the robustness of the results, I believe there could be value in extending this evaluation to include more real-life, challenging datasets. Such datasets, replete with their inherent complexities, could serve to more thoroughly test and validate the model. This, in turn, would potentially bring the proposed model closer in comparison to other state-of-the-art generative models. It would also ensure that the model's performance is tested in conditions that mirror real-world applications, reinforcing the practical relevance and applicability of the model.\n\nQUESTIONS:\nPlease refer to the above sections for detailed discussion.\n\nLIMITATIONS:\nYes, there is a section at the end of the paper discussing limitations and future research opportunities.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}