SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Paribesh Regmi Rui Li* Rochester Institute of Technology {pr8537, rxlics}@rit.edu
The neural network structures of generative models and their corresponding inference models paired in variational autoencoders (VAEs) play a critical role in the models' generative performance. However, powerful VAE network structures are hand-crafted and fixed prior to training, resulting in a one-size-fits-all approach that requires heavy computation to tune for given data. Moreover, existing VAE regularization methods largely overlook the importance of network structures and fail to prevent overfitting in deep VAE models with cascades of hidden layers. To address these issues, we propose a Bayesian inference framework that automatically adapts VAE network structures to data and prevent overfitting as they grow deeper. We model the number of hidden layers with a beta process to infer the most plausible encoding/decoding network depths warranted by data and perform layer-wise dropout regularization with a conjugate Bernoulli process. We develop a scalable estimator that performs joint inference on both VAE network structures and latent variables. Our experiments show that the inference framework effectively prevents overfitting in both shallow and deep VAE models, yielding state-of-the-art performance. We demonstrate that our framework is compatible with different types of VAE backbone networks and can be applied to various VAE variants, further improving their performance.
The inference models and the generative models paired in variational autoencoders (VAEs) are commonly constructed with neural networks, i.e., encoding networks and decoding networks, respectively [1, 2, 3]. Extensive research efforts show that well-designed encoding/decoding network structures for VAE models can constantly achieve state-of-the-art generative performance compared to other generative models [4, 5, 6]. However, powerful VAE network structures are hand-crafted and fixed prior to training. The issue with fixed network structures is that shallow ones limit VAE models' expressiveness, whereas overly deep networks are slow to use and prone to overfitting. Traditional model selection by training different VAE network structures for given data is difficult since finding optimal hyperparameters for each candidate structure is a daunting task, and training large VAE structures requires significant computation. On the other hand, network structure adaptation methods for discriminative model settings [7, 8, 9] cannot be straightforwardly applied to address the unique challenge posed by VAE estimation along with the latent variables.
Although the network structures play a critical role in the performance of VAE models, they are largely overlooked by current VAE regularization methods. This renders their failure to prevent overfitting when the network structures grow deeper. Amortized inference regularization (AIR) proposes two approaches: injecting random noise to the VAE objective for inference or directly restricting the inference models to a set of smooth functions [10, 11]. Another approach in VAE regularization
*Corresponding author
(c)
Figure 1: (a): Demonstration of the VAE network structure inference framework. Beta processes induce infinite number of hidden layers for encoding/decodineg networks, and its conjugate Bernoulli process prunes the neurons in each layer with a layer-wise activation probability from the beta process. Filled circles indicate activated neurons, corresponding to a sample of 1 from the Bernoulli process, while empty circles correspond to deactivated neurons, corresponding to a sample of 0. (b) and (c): Two settings of the stick-breaking constructions of beta process. The sticks on top are random draws from the process, which act as the layer-wise activation probabilities. A stick location  $\delta_{\mathbf{f}_l}$  corresponds to a hidden-layer function  $\mathbf{f}_l \in \mathbf{F}$ , and the height denotes its activation probability  $\pi_l$ . The bottom shows the conjugate Bernoulli processes to activate or deactivate neurons in each layer (column-wisely).
incorporates additional constraints to the VAE objective, which enforces a similarity between original input and its semantic preserving transformation in the latent representation [12, 13]. Nonparametric Bayesian inference such as Indian buffet process deep generative models (IBP-DGM) [14] and beta-Bernoulli process VAE (BB-VAE) [15] address overfitting by inferring the dimensionality of VAE latent variables using an Indian buffet process, specifically, a marginalized beta-Bernoulli process prior. While these regularization methods are effective for shallow network structures, they fail to prevent overfitting in deep VAE models.
We propose a novel Bayesian inference framework that automatically adapt VAE network structures by inferring the most plausible encoding/decoding network depths based on the given data, as demonstrated in Figure 1. To achieve this, we employ beta processes [16, 17] to model the number of hidden layers in the encoding/decoding networks, allowing for infinite depths. A conjugate Bernoulli process is utilized to prune the neurons in each layer based on layer-wise activation probabilities generated by the beta process. In addition, to enable efficient joint inference on both the network structures and the latent variables, we extend multiply importance weighted autoencoder (MIWAE) [18] by introducing an additional sample size used for Monte Carlo estimation of the network structures to be tuned. Our theoretical and empirical analysis shows that this novel gradient estimation scheme leads to a tight lower bound with high signal-to-noise ratio of parameter gradients.
In summary, our contributions are: i) We propose AdaVAE, a novel VAE structural adaptation strategy based on Bayesian model selection to enhance model performance. ii) We introduce a scalable estimator that facilitates joint inference on both encoding/decoding network structures and latent variables. iii) We conduct a comprehensive analysis of AdaVAE's regularization capabilities and demonstrate its ability to effectively mitigate overfitting in both shallow and deep VAE models and achieve state-of-the-art performance. iv) We showcase the versatility of AdaVAE by demonstrating its compatibility with different types of VAE backbone networks. It can also be readily applied to various VAE variants, thereby enhancing their performance.
Variational autoencoders (VAEs) have gained popularity as generative models across a wide range of applications [19, 20, 21, 22, 23, 24]. Extensive research efforts have focused on enhancing the performance of VAEs [25, 26, 27, 28, 29, 30, 31, 32]. Notably, well-designed encoding/decoding neural network structures constantly yield state-of-the-art generative performance [4, 5, 6, 33]. Ladder-VAE (LVAE) employs a shared top-down dependency structure in both the inference and generative models to facilitate information sharing between layers [4]. SkipVAE employs skip connections that connect latent variables to each layer in the generative model, mitigating posterior
collapse [34]. BIVA consists of a skip-connected generative model and an inference model formed by a bidirectional stochastic inference path [5]. NVAE designs expressive neural network structures tailored for VAEs based on deep residual networks [6]. Furthermore, advancements in VAE network structure design have also empowered recent works on hybrid models based on VAEs [35, 36, 37, 38].
Current VAE regularization methods often overlook the overfitting effect caused by deep network structures [12, 10, 15, 11, 13]. Amortized inference regularization (AIR) re-interprets the amortized inference model as a regularization for maximum likelihood training. AIR encourages the smoothness of the encoding network to restrict the model capacity, effectively mitigating overfitting and improving predictive performance [10]. Specifically, AIR proposes a denoising variational autoencoder (DVAE) that modifies the VAE objective using random perturbation training. An alternative AIR technique is weight-normalized inference VAE (WNI-VAE) that directly restricts the encoding networks to a set of smooth functions and achieves comparable performance to DVAE. Both approaches exhibit a lower inference gap than standard VAEs [39]. Consistency regularization for VAE (CR-VAE) extends the regularization techniques employed in semi-supervised learning [40, 41] to tackle the inconsistency problem of the inference models by enforcing the latent representations of an image and its semantic-preserving transformation to be similar, thereby yielding robust latent representations accounting for data variation [12].
IBP-DGM [14] and BB-VAE[15] as nonparametric Bayesian inference applied to VAE regularization focus on inferring the dimensionality of latent variable via an Indian Buffet Process (IBP) prior. The IBP prior is derived by marginalization over a beta process, resulting in a binary vector that masks the VAE latent variables. Without accommodating network structures, latent variable regularization alone is inadequate for mitigating overfitting.
3 Efficient VAE Estimators
Maximum likelihood estimation of a vector of parameters  $\theta$  of a deep generative model  $p_{\theta}(\mathbf{x},\mathbf{h})$  with  $\mathbf{x}$  denoting observed variables and  $\mathbf{h}$  denoting latent variables is intractable in general due to the marginalization over  $\mathbf{h}$ . Amortized variational inference optimizes an evidence lower bound (ELBO)  $\mathcal{L}_{\theta ,\phi}(\mathbf{x})$  on the log marginal likelihood  $\log p_{\theta}(\mathbf{x}) = \log \int p_{\theta}(\mathbf{x},\mathbf{h})d\mathbf{h}$  by introducing a variational distribution  $q_{\phi}(\mathbf{h}|\mathbf{x})$ :
$$
\begin{array}{l} \log p _ {\theta} (\mathbf {x}) \geq \int q _ {\phi} (\mathbf {h} | \mathbf {x}) \log \frac {p _ {\theta} (\mathbf {x} , \mathbf {h})}{q _ {\phi} (\mathbf {h} | \mathbf {x})} d \mathbf {h} \tag {1} \\ = \mathbb {E} _ {q _ {\phi} (\mathbf {h} | \mathbf {x})} [ \log p _ {\theta} (\mathbf {x} | \mathbf {h}) ] - \mathrm {K L} [ q _ {\phi} (\mathbf {h} | \mathbf {x}) | | p _ {\theta} (\mathbf {h}) ] \\ = \mathcal {L} _ {\theta , \phi} (\mathbf {x}) \\ \end{array}
$$
where  $\phi$  denotes variational parameters. For VAEs,  $q_{\phi}(\mathbf{h}|\mathbf{x})$  denotes an inference model commonly constructed with a neural network.  $p_{\theta}(\mathbf{x}|\mathbf{h})$  denotes a generative model that can also be parameterized by a neural network. The ELBO is optimized with gradient-based methods via reparameterization trick using stochastic Monte Carlo estimators of  $\nabla \mathcal{L}_{\theta,\phi}$  [1].
IWAE obtains a tighter lower bound using  $K$ -sample importance weighting estimate of the log marginal likelihood [42]:
$$
\mathcal {L} _ {\theta , \phi} (\mathbf {x}) = \mathbb {E} _ {q _ {\phi} (\mathbf {h} | \mathbf {x})} \left[ \log \frac {1}{K} \sum_ {k = 1} ^ {K} \frac {p _ {\theta} (\mathbf {x} | \mathbf {h} _ {k}) p _ {\theta} (\mathbf {h} _ {k})}{q _ {\phi} (\mathbf {h} _ {k} | \mathbf {x})} \right] \tag {2}
$$
where  $\mathbf{h}_k\sim q_\phi (\mathbf{h}|\mathbf{x})$ . It shows that the bound gets tighter with increasing  $K$ . However, [18] presents theoretical and empirical evidence that increasing the importance weighted sample size  $K$  to tighten the bound degrades signal-to-noise ratio (SNR) of parameter gradients estimates for the encoding network, and hurts the learning process. A new estimator (MIWAE) is thus introduced to address the issue of diminishing SNR, and its gradient estimate is:
$$
\Delta_ {M, K} = \frac {1}{M} \sum_ {m = 1} ^ {M} \nabla_ {\theta , \phi} \log \frac {1}{K} \sum_ {k = 1} ^ {K} \frac {p _ {\theta} (\mathbf {x} | \mathbf {h} _ {m , k}) p _ {\theta} (\mathbf {h} _ {m , k})}{q _ {\phi} (\mathbf {h} _ {m , k} | \mathbf {x})} \tag {3}
$$
where  $\mathbf{h}_{m,k} \sim q_{\phi}(\mathbf{h}_{m,k}|\mathbf{x})$ . For a fixed budget  $M \times K$  of total number of hidden variable samples, the number of samples  $M$  reduces the variance in estimating the ELBO gradient.  $K$  is the importance sample size as in IWAE.
4 VAE Structure Inference Framework
Traditional model selection cannot effectively adapt pre-determined VAE network structures to data without incurring significant computation overhead. We thus propose AdaVAE that enables joint inference on the structures of the encoding/decoding networks using stochastic processes [43, 9] and latent variables, and optimizes VAE objective without requiring additional expensive computation.
4.1 Formulation of the Inference Model
Let the prior over the latent variables  $\mathbf{h}$  be a zero-mean isotropic multivariate Gaussian  $p_{\theta}(\mathbf{h}) = \mathcal{N}(\mathbf{h};0,I)$ . We formulate the inference model by letting the variational distribution over  $\mathbf{h}$  be a multivariate Gaussian with a diagonal covariance structure:
$$
q _ {\phi} (\mathbf {h} | \mathbf {x}, \mathbf {Z}) = \mathcal {N} (\mathbf {h}; \boldsymbol {\mu}, \sigma^ {2} I) \tag {4}
$$
where the mean  $\mu$  and the standard deviation  $\sigma$  are outputs of the encoding neural network  $\mathcal{F}_{\phi}(\mathbf{x})$  with the variational parameters  $\phi$ . The binary matrix  $\mathbf{Z} = [z_{ol}\in \{0,1\} ]$  denotes the network structural variable, as in Figure 1(a).
Let  $\mathbf{f}_l$  denote the  $l$ -th hidden layer of  $\mathcal{F}_{\phi}(\mathbf{x})$  composed of neurons (i.e., non-linear activation functions)  $f(\cdot)$ . The encoding network  $\mathcal{F}_{\phi}(\mathbf{x})$  has the form:
$$
\mathbf {f} _ {l} = f \left(\mathbf {W} _ {l} \mathbf {f} _ {l - 1}\right) \odot \mathbf {z}. l + \mathbf {f} _ {l - 1} \quad l \in \{1, 2, \dots , \infty \} \tag {5}
$$
where  $\phi = \{\mathbf{W}_l\in \mathrm{R}^{O\times O}\}$ , and  $\mathbf{W}_l$  is the layer  $l$ 's weight matrix.  $\odot$  denotes element-wise multiplication of two vectors, so that we drop out the  $l$ -th layer's outputs by multiplying them elementwisely by the column vector  $\mathbf{z}_{.l}$  of  $\mathbf{Z}$ . Each random variable  $z_{ol}$  takes the value 1 with  $\pi_l\in [0,1]$  indicating activation probability of the  $l$ -th layer, as in Figures 1 (b),(c).  $O$  is the maximum number of neurons in a layer that is set to be the same for all hidden layers. We have skip connections to propagate the outputs of the hidden layers to the output layer. Note that when this network structure is used as the decoding network for the generative model  $p_{\theta}(\mathbf{x}|\mathbf{h})$ , then  $\mathbf{h}$  and  $\mathbf{x}$  are swapped, and  $\theta$  denotes the weights parameters. The output-layer can be readily replaced with a logistic function and the normal distribution with a Bernoulli distribution for binary data.
4.2 Beta Process Prior over Layer Number
A beta process  $B = \sum_{l} \pi_{l} \delta_{\mathbf{f}_{l}}$ , where  $\delta_{\mathbf{f}_{l}}$  is a unit point mass at  $\mathbf{f}_{l}$ , is a completely random measure over countably infinite set of pairs  $(\mathbf{f}_{l}, \pi_{l})$  [16], where  $\mathbf{f}_{l} \in \mathbf{F}$  denotes a hidden-layer function and  $\pi_{l}$  is its activation probability  $\pi_{l} \in [0, 1]$ . Its conjugate Bernoulli process can be defined as  $\mathbf{Z}_{o} \sim \mathrm{BeP}(B)$ , where  $\mathbf{Z}_{o.} = \sum_{l} z_{ol} \delta_{\mathbf{f}_{l}}$  is at the same locations  $\delta_{\mathbf{f}_{l}}$  as  $B$  where  $z_{ol}$  are independent Bernoulli variables with  $\pi_{l}$  being the probability of  $z_{ol} = 1$ . As in Eqn. (5),  $z_{ol} = 1$  activates the  $o$ 'th neuron in layer  $l$ . Computationally, we employ the stick-breaking construction [17] of beta process and its conjugate Bernoulli process as
$$
z _ {o l} \sim \operatorname {B e r} \left(\pi_ {l}\right), \quad \pi_ {l} = \prod_ {j = 1} ^ {l} \nu_ {j}, \quad \nu_ {l} \sim \operatorname {B e t a} (\alpha , \beta) \tag {6}
$$
where  $\nu_{l}$  are sequentially drawn from a beta distribution. The hyperparameters  $\alpha$  and  $\beta$  can be set to balance the network depth and width. Specifically, Figure 1 (b) demonstrates that if  $\beta >\alpha >1$ , the network structure prior favors shallower but wider network structures with the first few layer-wise activation probabilities being high. If  $\alpha >\beta >1$ , activation probabilities tend to be low over larger number of active layers, and the prior prefers a deeper but narrower network, as in Figure 1 (c).
We thus define the prior over the encoding network structural variable  $\mathbf{Z}$  as
$$
p _ {\alpha , \beta} (\mathbf {Z}, \boldsymbol {\nu}) = p _ {\alpha , \beta} (\boldsymbol {\nu}) p (\mathbf {Z} | \boldsymbol {\nu}) = \prod_ {l = 1} ^ {\infty} \operatorname {B e t a} \left(\nu_ {l} \mid \alpha , \beta\right) \prod_ {o = 1} ^ {O} \operatorname {B e r} \left(z _ {o l} \mid \pi_ {l}\right) \tag {7}
$$
To enable asymmetric encoding/decoding network structures, we independently apply the prior to both networks. Further analysis on symmetric constraints can be found in the Appendix.
4.3 Joint Inference on VAE Network Structures and Latent Variables
We first expand the overall marginal likelihood over the VAE structures  $\mathbf{Z}$  as
$$
\log p _ {\alpha , \beta} (\mathbf {x}) = K L \left[ q \left(\mathbf {Z}, \boldsymbol {\nu} \mid \left\{a _ {t} \right\} _ {t = 1} ^ {T}, \left\{b _ {t} \right\} _ {t = 1} ^ {T}\right) \right] \left| \left| p _ {\alpha , \beta} \left(\mathbf {Z}, \boldsymbol {\nu} \mid \mathbf {x}\right) \right| + \mathcal {L} _ {\left\{a _ {t} \right\}, \left\{b _ {t} \right\}} (\mathbf {x}) \right. \tag {8}
$$
where the first RHS term denotes a Kullback-Leibler (KL) divergence of the approximate variational distribution from the true posterior of the VAE network structural variables. We specify the variational distribution as
$$
q \left(\mathbf {Z}, \boldsymbol {\nu} \mid \left\{a _ {t} \right\} _ {t = 1} ^ {T}, \left\{b _ {t} \right\} _ {t = 1} ^ {T}\right) = \prod_ {t = 1} ^ {T} \operatorname {B e t a} \left(\nu_ {t} \mid a _ {t}, b _ {t}\right) \prod_ {o = 1} ^ {O} \operatorname {C o n B e r} \left(z _ {o t} \mid \pi_ {t}\right) \tag {9}
$$
where  $\pi_t = \prod_{j=1}^t \nu_j$ , and  $\{a_t, b_t\}_{t=1}^T$  are the variational parameters.  $T$  denotes a truncation level for the maximum number of hidden layers [17]. We also relax the constraint of the discrete variables by reparameterizing the Bernoulli distribution into a concrete Bernoulli distribution  $\mathrm{ConBer}(z_{ot} | \pi_t)$  [44, 45]. This allows us to efficiently backpropagate the parameter gradients of the estimator while generating network structure samples.
The second RHS term in Eqn. (8) denotes the ELBO to the overall marginal likelihood:
$$
\begin{array}{l} \mathcal {L} _ {\{a _ {t} \}, \{b _ {t} \}} (\mathbf {x}) = \int q (\mathbf {Z}, \boldsymbol {\nu} | \{a _ {t} \} _ {t = 1} ^ {T} \{b _ {t} \} _ {t = 1} ^ {T}) (\log p _ {\theta} (\mathbf {x} | \mathbf {Z}) + \log p _ {\alpha , \beta} (\mathbf {Z}, \boldsymbol {\nu}) \\ - \log q (\mathbf {Z}, \boldsymbol {\nu} | \left\{a _ {t} \right\} _ {t = 1} ^ {T} \left\{b _ {t} \right\} _ {t = 1} ^ {T}) d \mathbf {Z} d \boldsymbol {\nu} \tag {10} \\ \end{array}
$$
The term  $\log p_{\theta}(\mathbf{x}|\mathbf{Z})$  in Eqn. (10) is the marginal likelihood over the latent variable  $\mathbf{h}$ , which is an extension of Eqn. (1), in terms of conditioning on the structure variable  $\mathbf{Z}$ . Thus, the ELBO to the marginal likelihood  $\log p_{\theta}(\mathbf{x}|\mathbf{Z})$  is:
$$
\begin{array}{l} \log p _ {\theta} (\mathbf {x} | \mathbf {Z}) \geq \int q _ {\phi} (\mathbf {h} | \mathbf {x}, \mathbf {Z}) \log \frac {p _ {\theta} (\mathbf {x} , \mathbf {h} | \mathbf {Z})}{q _ {\phi} (\mathbf {h} | \mathbf {x} , \mathbf {Z})} d \mathbf {h} \\ = \mathbb {E} _ {q _ {\phi} (\mathbf {h} | \mathbf {x}, \mathbf {Z})} [ \log p _ {\theta} (\mathbf {x} | \mathbf {h}, \mathbf {Z}) ] - \mathrm {K L} [ q _ {\phi} (\mathbf {h} | \mathbf {x}, \mathbf {Z}) | | p _ {\theta} (\mathbf {h} | \mathbf {Z}) ] \\ = \mathcal {L} _ {\theta , \phi} (\mathbf {x} | \mathbf {Z}) \tag {11} \\ \end{array}
$$
Lemma 1 Let  $Q(\mathbf{h}|\mathbf{x}) = \int q_{\phi}(\mathbf{h}|\mathbf{x},\mathbf{Z})q(\mathbf{Z},\nu)d\mathbf{Z}d\nu$  be the variational distribution of the latent variable  $\mathbf{h}$  marginalizing over  $\mathbf{Z}$ , then
$$
\mathbb {E} _ {q (\mathbf {Z}, \nu)} \mathbb {E} _ {q _ {\phi} (\mathbf {h} | \mathbf {x}, \mathbf {Z})} \log \frac {p _ {\theta} (\mathbf {x} , \mathbf {h} | \mathbf {Z})}{q _ {\phi} (\mathbf {h} | \mathbf {x} , \mathbf {Z})} \leq \mathbb {E} _ {Q (\mathbf {h} | \mathbf {x})} \log \frac {p _ {\theta} (\mathbf {x} , \mathbf {h} | \mathbf {Z})}{Q (\mathbf {h} | \mathbf {x})} \tag {12}
$$
Lemma 1 indicates that the overall ELBO we derived on the left-hand side in Eqn. (12) bounds the lower bound to the marginal likelihood over  $\mathbf{h}$ . The proof is in the appendix. In particular,  $q(\mathbf{Z},\pmb {\nu})$  is essentially a non-explicit mixing distribution [46]. It allows the variational distribution  $Q(\mathbf{h}|\mathbf{x})$  to take complex form, and results in more informative latent representation.
We adopt Monte Carlo estimation of the expectations over both  $\mathbf{Z}$  in Eqn. (10) and  $\mathbf{h}$  in Eqn. (11) to estimate the overall ELBO. In particular, we extend the MIWAE estimator in Eqn. (3), and introduce three sample sizes to tune: the number of samples  $S$  used for Monte Carlo estimation of the expectation over the VAE network structure variable  $\mathbf{Z}$ , the number of samples  $M_{s}$  used for Monte Carlo estimation of the gradient of the latent variable ELBO conditioned on the structure samples in Eqn. (11), and the number of importance samples  $K_{s}$  used for estimation of the expectation over the latent variables  $\mathbf{h}$ . We thus express our gradient estimate in the general form as
$$
\Delta_ {S, M, K} = \frac {1}{S} \sum_ {s = 1} ^ {S} \frac {1}{M _ {s}} \sum_ {m = 1} ^ {M _ {s}} \nabla_ {\theta , \phi} \log \frac {1}{K _ {s}} \sum_ {k = 1} ^ {K _ {s}} \frac {p _ {\theta} (\mathbf {x} | \mathbf {h} _ {m , k} , \mathbf {Z} _ {s}) p _ {\theta} (\mathbf {h} _ {m , k} | \mathbf {Z} _ {s})}{q _ {\phi} (\mathbf {h} _ {m , k} | \mathbf {x} , \mathbf {Z} _ {s})} \tag {13}
$$
where  $\mathbf{h}_{m,k} \sim q_{\phi}(\mathbf{h}|\mathbf{x},\mathbf{Z})$ . When  $S = 1$  our estimator becomes equivalent to the MIwAE objective in Eqn. (3). Since our estimator generates latent variable samples conditioned on network structure samples, increasing  $S$  will not impact the SNR of the gradient estimator in Eqn. (13).
Theorem 1 Let  $\mathcal{L}_S$  be the lower bound with  $S$  structure samples of  $\mathbf{Z}_s\sim q(\mathbf{Z},\pmb {\nu})$  , then:
$$
\mathcal {L} _ {S} \leq \mathcal {L} _ {S + 1} \leq \log p _ {\alpha , \beta} (\mathbf {x}), \mathcal {L} _ {S} = \mathbb {E} _ {q (\mathbf {Z}, \boldsymbol {\nu})} \mathbb {E} _ {q _ {\phi} (\mathbf {h} | \mathbf {x}, \mathbf {Z})} \log \left[ \frac {1}{S} \sum_ {s} \frac {p _ {\theta} (\mathbf {x} , \mathbf {h} | \mathbf {Z} _ {\mathbf {s}})}{q _ {\phi} (\mathbf {h} | \mathbf {x} , \mathbf {Z} _ {\mathbf {s}})} \right] \tag {14}
$$
Figure 2: Left: Evolution of the encoding/decoding network structures visualized through layer-wise activation probabilities  $\pi_{l}$  (top) and neuron activations  $\mathbf{Z}$  (bottom) with a reconstructed sample. Right: the top shows the median change of the number of active layers over training epochs and the percentage of activated neurons in the truncation. The bottom shows the convergence of the proposed estimator in terms of negative log-likelihood  $(-LL)$  for different structure sample size  $S$ .