A Note on Generalization in Variational Autoencoders: How Effective Is Synthetic Data and Overparameterization?
Authors: Tim Z. Xiao
Johannes Zenn
Robert Bamler
Venue: Accepted by TMLR ()

Abstract: Variational autoencoders (VAEs) are deep probabilistic models that are used in scientific applications. Many works try to mitigate this problem from the probabilistic methods perspective by new inference techniques or training procedures. In this paper, we approach the problem instead from the deep learning perspective by investigating the effectiveness of using synthetic data and overparameterization for improving the generalization performance. Our motivation comes from (1) the recent discussion on whether the increasing amount of publicly accessible synthetic data will improve or hurt currently trained generative models; and (2) the modern deep learning insights that overparameterization improves generalization. Our investigation shows how both training on samples from a pre-trained diffusion model, and using more parameters at certain layers are able to effectively mitigate overfitting in VAEs, therefore improving their generalization, amortized inference, and robustness performance. Our study provides timely insights in the current era of synthetic data and scaling laws.

Published in Transactions on Machine Learning Research (12/2024)
A Note on Generalization in Variational Autoencoders: How Effective Is Synthetic Data & Overparameterization
Tim Z. Xiao
AT Center Tübingen, University of Tübingen, IMPRS-TS
Johannes Zenn'
AI Center Tübingen, University of Tübingen, IMPRS-IS
Robert Bamler
robert. www.eeeeeereenn--teennnn...d..d
AT Center Tübingen, University of Tübingen
Reviewed on OpenReview: https: // openeeview. forum? id= buyllf5eery
Abstract
Variational autoencoders (VAEs) are deep probabilistic models that are used in scientifie applications (Cerri et al., 2019; Flam--hepherd et al., 2022; Zhou & Wei, 2020; Gondur et al., 2023) and are integral to compression (Ballé et al., 2018; Minnen et al., 2018; Cheng et al., 2020; Yang et al., 2023); yet they are susceptible ible to overfitting (Cremer et al., 2018). Many works try to mitigate this problem from the probabilistic methods perspective by new inference techniques or training procedures. In this paper, we approach the problem instead from the deep learning perspective by investigating the effectiveness of using synthetic data and overparameterization for improving the generalization performance. Our motivation comes from (1) the recent discussion on whether the increasing amount of publicly accessible synthetic data will improve or hurt currently trained generative models (Alemohammad et al., 2024; Bohacek & Farid, 2023; Bertrand et al., 2024; Shumailov et al., 2023); and (2) the modern deep learning insights that overparameterizatinn improves gencralization. Our investigation shows how both training on samples from a pre-trained diffusion model, and using more parameters at certain layers are able effectively mitigate overfitting in VAEs, therefore improving their generalization, amortized inference, and robustness performance. Our study provides timely insights in the current era of synthetic data and scaling laws.
1 Introduction
Variational autoencoders (VAEs, (Kingma & Welling, 2014; Rezende et al., 2014)) are deep probabilistic models that consist of an encoder and a decoder. VAFs were originally proposed in the framework of gen- erative models, where the decoder (together with a prior) models the underlying data distribution (aata())) while the encoder plays only an auxiliary role to speed up training. Nowadays, however, many applications use VAEs for their encoder: VAEs are widely used in science, e.g., for anomaly detection (Cerri et al., 2019) and for learning interpretable ropresentations in quantum optics "llan-Shepherd et al., 2022) and neuro- science (Zhou& & Wei, 2020; Gondur et al., 2023). In compression, VAEs are also the dominating models due to their connection to the rate-distortion theory (Ballé et al., 2018; Minnen et al., 2018; Cheng et al., 2020; Yang et al., 2023). These applications can be compromised if the VAE overfits. More specifically, it is empirically observed that the encoder fxx more susceptible overfitting than the decoder (Wu et al., 2017; Cremer et al., 2018; Shu et al., 2018). One possible explanation that a VAE normally trained with finite dataset, due to the ELBO objective and the training algorithm, the encoder is fed with the same data at cach epoch (by contrast, the decoder is fed with random samples from the approximate postcrior distribution, therefore less likely to see the same data twice during training).
* EEqual contribution.

---

Published in Transactions on Machine Learning Research (12/2024)
Overfitting in the encoder implies that the learned mapping fo( does not generalize well to unseen data, which can negatively impact the performance of generative modeling, amortized inference, and adversarial robustness. For generative modeling, as the number of training epochs increases, an overfitted VAE will have a higher evidence lower bound (ELBO) on the training sct but a lower ELBO on the test set. For amortized inference, an overfitted encoder is likely to map unseen data to a suboptimal set of variational paramelers. This results in a lower ELBO when compared to the ELBO obtained by directly optimizing these paramcters. For robustness, an overfitted encoder often learns a less smooth fa(#;) i.e,, a small change in æ can result in large difference in the latent space. This makes VAEs vulnerable to adversarial attacks, causing realistic and hardly distinguishable inputs to yield semantically different outputs (Kuzina et al., 2022).
In learning theory, overfitting is a result of the interaction between the size of a training set and the size of a model. According to the classical understanding, given a finite training set, as the model size (and presumably also the model complenity) increases, its performance first. increases and then decreases, resulting in a U-shape relation between test error and model size. Conversely, the modern deep learning literature shows that we keep increasing the model size, its performance gets better again, thus exhibiting a double descenl. behavior (Belkin et al., 2019; Nakkiran et al, 2021). In this work, investigate to what extent these findings apply to VAEs, and whether we can mitigate overfitting in VAEs and improve their gencralization performance by exploiting these findings. Starting from VAE in the overfitting regime, we investigate the effect of increasing either the size of the training set or the size of the model. Since the generalization property that we investigate is mostly affected by the relative size of the model compared to the dataset, we limit the discussion in this paper to datasets that allow us to test strongly overparameterized models (compared to the dataset size), as well as models that are underparameteriized.
There is subtlety in increasing the training data for VA Since the goal of VAEs is to model pdata (æ), naive data angmentations might alter the target distribution if not carefully crafted. Hence, we ask the question: "aan we have infinite training samples drawn from (a) pd The answer is likely to be "No", unless we have access to the true data gencrating process. However, there is a class of models, known as diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021), that can estimate Paata (x)) very well, and that can generate as many samples as we want. Contrary to diffusion models, VAEs learn an encoding procoss leading to the so-called variational information bottleneck (Alemi et al., 2016). This incentivizes VAEs to encode information into their latents efficiently by removing redundancies from the input differenn from diffusion maddels-whion makes VA the most important tool in compression (Ballé et al., 2018; Minnen et al., 2018; Cheng et al., 2020; Yang et al., 2023). Additionally, the learned representations are fast, controllable, and interpretable which enables semantically meaningful reprosentations useful for applications in e.g. science (Cerri et al., 2019; Flam-Shepherd et al., 2022; Zhou & Wei, 2020; Gondur et al., 2023). We therefore investigate whether training a VAE on samples from a pre-trained diffusion model alleviates overfitting in the encoder of the VAE, thus allowing us to learn better representations. This approach can be considered cross- moss-model-closs distillation, i.e., distilling from diffusion model to a VAE.
Contributions. We investigate generalization performance in VAEs from a deep learning perspective by considering synthetic data and werparameterization. Particularly, we show that in the overfitting regime (1) VAËs trained on samples from pre-trained diffusion models have better generalization, amortized in- ference and robustness performance; (2) we do not need infinite synthetic data to gain such generalization performance; and (3) increasing the number of parameters in the layers next to the latent variable (thus increasing the latent dimension) improves the generalization performance, but increasing the number of pa- rameters in other layers hurts performance. Our findings provide practitioners with practical guidance to avoid overfitting in VAEs. Moreover, we find indications of the double descent phenomenon in VAEs which might be explained as for non-deep models (Curth et al., 2023).
2 Performance Metrics for VAES
Variational autoencoders (VAEs). A VAE models the data distribution (xata( by assuming a gen- crative process that first draws a latent variable z from a prior (2(2) and then draws æ from conditional likelihood (z 1z), parameterized by the output of a neural network ("decoder") go(z) with weights 0. Given a training data distríbution p(a:) approximating parta(ar), naive maximum likelihood learning would
2

---

Published in Transactions on Machine Learning Research (12/2024)
maximize the negative cross entropy - -H(p(z), = (log (o))), where pe(x)) ( However, as the integration over is typically intractable, VA resort to variational inference (Blei et al., 2017) and instead maximize the evidence lower bound (ELBO)
(x)
[())
p(x)(())
(z)) - ()((()) pa po where
(1)
ELBO (x) = Ex (z a) [logpo po [x] =) + log Pz (z) - log go [=]]
(2)
Here, the variational distribution (z(2) is simple probability distribution (often fully factorized Ganssian) whose parameters are the outputs of the inference network (or "encoder") fa())), whose weights & are learned by maximizing [(O) jointly over all parameters θ = (0)
While we would like to use (x) for p(z:,, we only have access to a finite training set Dirains which can lead to overfitting We discuss three performance gaps quantifying the effects of overfitting.
Generalization gap. One signal for overfitting is that a model performs better on the training set Dirain than on the test set Duest and the test. set. performance decreases over training epochs. We refer to the difference between training and test set ELBO as the generulization gap,
gg = EEgDDuaai eLBOg(#)) - EewD ELBOg(z)).
(3)
Since Dirain and Drest both consist of samples from the same distribution Paata()),, and training maximizes the ELBO on Dirain the ELBO on Durain is typically greater than equal to the ELBO on Desst and 9g 2 0. A smaller 9g corresponds to betler generalization performance.
Amortization gap. VAEs use amortized inference, ie., they set the variational parameters of ga(z x)) for given to the output of the encoder fa()). At test time, can further maximize the ELBO over the individual variational parameters for each æ, which is more expensive but typically results in better variational distribution q²(z x). We then study the amortization gap (Cremer et al., 2018),
Ru=RaanDeo [LBO)(=)]
E.g.. [ELBOe(2)) 2
where we replaced 9 by 9" in ELBO The encoder of a VAE tends to be more susceptible to overfitting than the decoder (Wu et al., 2017: Cremer et al., 2018; Shu et al., 2018). When the encoder overfits, its inference ability might not generalize to test data, which results in a lower ELBO and larger amortization gap Ga. A smaller Gn corresponds to better generalization performance of the encoder.
Robustness gap. An overfitted encoder (x) polentially learns a less smooth function such that a small change in the input space can lead to a large difference in the output space. Hence, it is easier to construct an adversarial sample xx = xx' + E (I|e]] is within a given attack radius 5) from a real data point x' Drest- We construct adversarial samples by maximizing the symmetrized KI-divergence between gø(2 (=(=(=") and go(2 (*)) (Kuzina et al., 2022). For a successful attack, the reconstruction 5e go(2)), 2 qa(2 x*),, is very different from the real data reconstruction क' =g((²", x= ~ go(z x*), even though the inputs x and x" are similar. Using the image similarity metric MS-SSIM (Wang et al., 2003), where higher MS-SSIM means more similar images, we define the robusiness gap as
G Egerppa> ar) Drost MS-SSTM [at", xd)] MS-SSIM
[ [=]]]
A more robust, VAE makes attacks difficult, i.e., it has a higher MS-SSIM .e) and thus a smaller G, than a less robust VAE. For more details on the construction of the attack see Appendix A.
3
Related Work
We group related work into generalization and overparamoterization, using diffusion models, and closing the performance gaps. Work on augmentation and distillation is discussed in Section 4.
3

---

Published in Transactions on Machine Learning Research (12/2024)
Overparameterized models generalize. The classical theory of machine learning suggests not increase model parameters (relative to the size of the training set) to avoid fitting spurious patterns or noise in the training data, which is known as overfitting and can lead to poor generalization. However, empirical evidence in deep learning suggests that overparameterized models generalize well (Bartlett et al., 2020). Specifically, when we keep increasing the number of parameters, after the first descent, the test error will increase but a second descent will follow. This phenomenon known as double descent (Belkin et al., 2019). Understanding generalization in overparameteriiee models is still an active research direction (Brutzkus & Globerson, 2019; Allen-Zhu et al., 2019). Double descent has been demonstrated in many deep learning models (Nakkiran et al., 2021). Our work investigates both double descent and the effect of overparameterization in the setting of VAËs by analyzing two different sets of parameters.
Use samples from pre-trained diffusion models. There are many recent attempts to solve various tasks with data generated by diffusion models. Azizi et al. (2023) fine-tuned a tcxt-to-image diffusion model on ImageNet, generated state-of- thet--r-t samples with class labels, and trained a classifier on the samples. Their result shows that the classifier trained on generated data does not outperform the classifier trained on real data. In the adversarial training setting, using generated data by diffusion models shows significant improvements on classification robustness (Croce et al., 2021; Wang et al., 2023). Tian et al. (2023) found that the visual represent atioss learned from samples generated by text-to-image diffusion models outperform the representations learned by SimCLR (Chen et al., 2020) and CLIP (Radford et al., 2021). Alemohammad et al. (2024) trained new diffusion models with samples from previously trained dilfusion models, and they found that their sample quality and diversity progressively decrease. In this work, we find that using diffusion models as data sources improves the generalization performance of VAEs.
Improve generalization, amortized inference, and robustness in VAEs. Cremer et al. (2018) study the amortization gap in VAEs, and they notice that overfitting in the encoder is one of the contributing factors of the gap, and that it hurts generalization. Subsequent works try to close the amortization gap by introducing new inference techniques or procedures (Marino et al., 2018; Shu et al,, 2018; Zhao et al., 2019). To close the generalization gap and reduce encoder overfitting, Zhang et al. (2022) propose to freeze the decoder after a certain amount of training steps, but further train the encoder by using reconstruction samples as part of the training data. As for adversarial robustness, Kuzina et al. (2022) propose to defend a pre-trained VAE by running MCMC during inference to move z towards "safer" regions in the latent space. In our case, both using synthetic data and overparaeeterization do not require changing the original inference procedure. They can be used orthogonally and take into account all three gaps simultaneously.
4 Two Paths Towards Generalization
Given a VAE that overfits and generalizes poorly, we want to know whether using more training data (method A); or more model parameters (method B) can improve its generalization performance and performance gaps (see Section 2). Here, we compare the two approaches, and discuss how using samples generated from a diffusion model is fundamentally different from naive data augmentation.
4.1 Method A: More Data- Diffusion Model as Pdata (x)
An idcal training objective for VAEs to maximize
Table 1: Training distributions for VAFs (see Figure 1), and whether we (1) can sample arbitrarily many unique samples from (2) an accurate approximation of (1))
L
Exe- (mz ELBO0 (z)].
(6)
However, we only have Derain as a finite approximation of Pdata (z). Hence, we normally maximize
approx.
Dusin Panglx") PDM(("")
oc unique samples X
/
accurate
/
X
/
L Eg [L)O0<z)),
(7)
which can lead to overfitting. Rather than focusing on model architectures or training techniques as in prior works (Shu et al., 2018; Zhao et al., 2019; Zhang et al., 2022), we aim to mitigate overfitting by seeking a better approximation for (a)ta( than D An ideal training data distribution should enable us to
4

---

Published in Transactions on Machine Learning Research (12/2024)
(1) sample unlimited amount of unique samples to avoid overfitting; and it should be
(2) an accurate approximation of (a)ta(#), i.e., we are indeed modeling Pdaaa (x) rather than some different distribution (in practice, it needs to be an accurate model of D Perain).
Our hypothesis is that good diffusion model' that has been pre-trained on Derain satisfies these two criteria: (1) we generate unlimited samples from it, and (2) its training objective is designed to model Pdata (x). Therefore, we investigate training VAEs using a pre-trained diffusion model PDM((') instead of Dirain as an approximation of the underlying distribution Pdata (x), ie., by maximixing
C = E Ex''-ppst (x)) [ELBOe(z))
(8)
We denote this method DMaaPx, short for "Diffusion Model Paata(x))" Figure 1 illustrates the intuition behind DMaaPx. The blue dots represent the finite data set Dtrain They are i.i.d. samples from the unknown underlying data distribution Pdata (æ) which is shown by the dark-edged region. The green regions represent (a)) We use shaded areas (green), not dots, to highlight that PDM(2") models a continuous distribution which we can generate infinitely many samples from.
Diffusion models for data types other than images are less explored and might not accurately approximate Pdaza (2) and therefore might not satisfy criterion (2). Moreover, due to the data processing inequality, information Pa)ta( captured by diffusion model trained on Durain cannot exceed the information contained in Derain (but for injected information via inductive biases). In reality, state-of-the-art diffusion models cannot fit Dtrain perfectly. Many recent works observe in both image and text settings that training generative models on generated data degrades the overall performance (Alemohammad et al., 2024; Bohacek & Farid, 2023; Bertrand et al., 2024; Shumailov et al., 2023). Hence, the continuity we gain by replacing Durain with (a' comes the cost of potcntially losing some amount of information contained in Dirain.
4.1.1 Difference Between Data Augmentation and DMaaPx
Ptata
Data augmentation has a similar goal as DMaaPx as both ap- proaches aim to increase amount and diversity of training data. Training a VAE via data augmentation amounts to maximizing
Pany fa
== Emn-Drrae Dusai
B Bpoog (ee (x) [ELBOe(z']]]
(9)
where Paug (x' ax) is distribution over hand-crafted transfor- mations (e.g., scalings and rotations) of æ Dirain-
Prulee
Dirat
Like DMaaPx, data augmentation replaces Dtrain with a continuous distribution Pang (z') = Ea-Dran ((PPug("' x)]. But Paug(x') can be a less accurate approximation Panta(a:) than (x)(()) (Table 1). Firstly, typical data augmentation techniques generate new training points x' by conditioning on single original data point æ (i.e., the nested expectations in Eq. (9)). Thus, Paug cannot interpolate between points in Dirain (i.e., the gaps between purple regions in Figure 1). By contrast, in DMaaPx, each training data point æ' ~ (x')(x)) is drawn from diffusion model trained on the entire dataset Duriin Hence, each x' is effectively conditioned on the full Dusin.
Training distributions. Pang (x') = En-D.am (aug((' [x]] only extrapolates from individual data points æ ~ Duran and has density outside the support of (asta( (c.g.. when flipping the digit "2"). By contrast, pDM((* can interpolate be- tween data ~ Derain.
Figure 1:
Secondly, the transformations used for Paug(2') æ) are drawn from manually curated catalog. This catalog is heavily based on prior assumptions regarding invoriances in the data type under consideration, which can introduce bias. In
1 An anconditional diffusion model, oppused conditional like Stable Diffusion (Rombach al., 2022). UUsing samples from generative models for training is sometines considered kind of data augmentation in the context of supervised learning (Yang al., 2022). In the present work, deliberately scparate generative modcls from the broader sense of augmentation, and consider data augmentation in narrow sense such that augmented data output. of some transformation conditioned single Ptrain-
5

---

Published in Transactions on Machine Learning Research (12/2024)
practice, one has to make assumptions and decide whether the (unknown) true data distribution Pdata (x) is invariant under the considered transformations. For instance, with images, we assume invariance to minor translations, hue shifts, and zooms. For data types other than images, such as neural activations or molccules, manually crafting the appropriate set of augmentations might be less obvious. This manual design of augientattinss may result in problems of (i) not modeling the full exlend of Pasta (ar) or (ii) modeling density outside of Paata(2) Figure 1 depicts both: problem (i) corresponds to "empty" space between arcas of pang(a') (purple); problem (ii) corresponds to density of (ang(x') outside of (aata()). DMaaPx eliminates these assumptions, which makes the method more resilient against human bias (but less interpretable).
In summary, while traditional data augmentation introduces diversity based on invariances in the data, DMaaPx uses an expressive generative model to extrapolate from the empirical diversity of the data.
4.1.2 A Distillation Perspective
Using samples from pre-trained diffusion model to train a VAE can also be viewed from distillation perspective (Hinton et al., 2015). Distillation describes the process of transferring knowledge from a large model to a smaller one, In practice distillation is often used because a smaller model is less expensive to be deployed in production. On the contrary, here we consider transferring knowledge between models designed with different modeling assumptions or structures. We refer to this as cross-model- class distillation, and the conventional usage of distillation as wifhhin-model-class distillation. While both types soek to transfer knowledge from source to a larget model, cross-model-class distillation emphasizes more on enhancing functionalities that are unique to the target model rather than mirroring the capabilities shared with the source model. For instance, Frosst & Hinton (2017) distill neural network into a soft decision tree to improve its unique capability of providing explanations to decisions.
DMaaPx belongs to cross-model-class distillation, i.e., it distills diffusion models to VAEs. The goal of DMaaPx not to rival diffusion models in sample quality, but rather to improve the unique capabilities of VAEs such as its variational bottleneck (Alemi et al., 2016) for learning fast, controllable, and interpretable representations (see Section 1 for discussion). Hence, DMaaPx fundamentally differs from approaches that train VAËs samples produced by VAËs (Shumailov et al., 2023), diffusion models on outputs of diffusion models (Alemohammad et al., 2024), which belongs to within-modellclass distillation.
4.2 Method B: More Parameters
The success of deep learning and empirical evidence (Nakkiran et al., 2021) show that increasing the number of parameters in model can improve its generalization performance. However, increasing the number of parameters in one part of the model can have different effect than increasing the number of parameters in another part of the model. Different sets of parameters form distinct complerity azes, along which generalization performance can behave in different ways.
In practice, increasing the number of parameters in VAEs is usually done by changing two hyperparameters, which control two distinct sets of model parameters, θ O₂ O Here, O₂ contains the parameters that determine the latent dimension d₂ and interact directly with z, i.c., the weights in the last layer of the encoder [z) and the first layer of the decoder (=)... Changing the number |02| of parameters in Oz changes the dimension de of z. We denote the remaining parameters as O_g.
We investigate the effect of increasing |0z| and |0-z| separately as the increase has different implications for each parameter group. For both sets, we only consider changing the width of the VAF,, not the depth. For a VAE with mean field assumption, for example, increasing |Oz] implies that we update our belief about the underlying data generative process, which we now believe to involve more independent factors (hence, we increase dg to model these additional independent factors).
Explaining Double Descent with Multiple Complexity Axes. With the two complexity axes O0 and @-_1 mentioned above, we might potentially observe the phenomenon of double descent in VAEs. Recent work by Curth al. (2023) tries to explain double descent (see Section 1) in non-deep models such trees and linear regression. They suggest. that double descent is the result of plotting the test error along
6

---

Published in Transactions on Machine Learning Research (12/2024)
aggregated model complexity axis (i..,, the total number of parameters), even though the raw parameters are increased first along one complexity axis and then along another. In other words, the test error along each axis shows the classical U-shape, but plotting them into the same model complexity axis leads to the shape of double descent. We refer readers to (Curth et al., 2023, Figure 1) for an illustration. Whether this explanation applies to deep double descent is still unknown. Nevertheless, it highlights that increasing different sets of model parameters can impact the model differently.
5 Experiments and Discussions
In this section, we introduce the experimental setup and evaluate the generalization performance as well as the performance gaps (see Section 2) for both using more training data (method A) and more parameters (method B). We then evaluate and discuss the effects of these two methods.
5.1 Experimental Setup
Datasets. We evaluate both methods A & B, on CIFAR-10 (Krizhevsky et al., 2009). We further evalu- ate method A on BinaryMNIST (LeCun et al., 1998) and FashionMNIST (Xiao et al., 2017), which show consistent results with CIFAR-10 (see Appendix B.3). Due to compulation constraints, we did not evaluate method B on these two easier datasets. As a preparation for DMaaP'x in method A, we train a diffusion model PDM(2') on each training set Derain: which we then use to generate the training data for the VAEs. We use the implementation by Karras el. al. (2022). Further details can be found Appendix D.
VAE architectures. We assume fixed Gaussian priors p= = VO. For the conditional likelihood pa z); we use a discretized mixture of logistics (MoL; (Salimans et al., 2017)). For the inference model (0(2 m) we use fully factorized Gaussian distributions whose mcans and variances are the outputs of the encoder. Both the encoder and the decoder consist of convolutional layers with residual connections. The number of output channels for the encoder and the number of input channels for the decoder (i.e., the latent dimension of =) is denoted as d2 my x 64, where m, is an integer multiplier. The number of the internal channels, which governs the rest of the parameters, is denoted as ne. Hence, |0,1 d2 x e₁ and [0-z] ne C2, where e1 and 02 are the appropriate constants. For more details on the network architectures, hyperparancters, and training please consult Appendix B.1.
Basclines. The default models for method A and the bascline for method B use m, = 1 and n. = 256. For method A, we compare VA trained with DMaaPx against three other models trained on: (i) repetitions of Dusin ("Normal Training"); (ii) carefully tuned augmentation for Dusin (*Aug. Tuned") and (iii) plausible augmentation for images in general (4 ("AugNaave") The results are averaged over 3 random seeds. Note that "Au..Naive is not tuned towards a given training dataset Durain and can result in out-of-distribution data, e.g. a horizontally flipped digit "2" for MNIST. This mimics situations where the invariances of the data modality are less clear than for images. We report the specific augmentations used for Aug Tuned and Aug.N in Appendix E. For method B, we compare VAEs trained with various m, E (1,...,, 64) and ne E (1, 512). When documenting the training progress, the term "epoch" usually refers one complete pass over Deraiu. For DMaaPx, this term is not applicable since it can sample unlimited data from PDM(x') Therefore, we measure training progress of DMaaPx in "effective epochs", which count multiples of [Pirain] training samples. We train all models for 1000 (effective) epochs.
5.2 Training with synthetic data improves generalization and minimizes the gaps
Figure 2 shows the performance for generalization (left), amortized inference (mid), and robustness (right) across normal training, augmentations, and DMaaPx. We observe that DMaaPx has the highest. ELBO on Drest (left, solid green) and the smallest generalization, amortization, and robustness gaps (Eqs. 3-5). Augmentation provides competitive improvements, but not as good as DMaaPx. This implics that VAEs trained with DMaaPx approximate the underlying distribution (z) better than those trained on D solely, or on augmented data, as suspected in Section 4.1.1.
7

---

Published in Transactions on Machine Learning Research (12/2024)
Generalization Gap
Amortization Gap
Robustness Gap
10000
-10500
100
16900
0.0
MS SSIM x (rrput smiertt
ELBD (Tbst Sef)
55M [Reonstrcction demilert)
-13000
MS 15-55IM
0.6
-11200
-11000
-1440
0.6
-10000
Enilssead on Paua
Evalutte
-13500
-10500
400 Effective Epoch
420
400
Effective Epoch
Effective Epoch
Normal Training
DMaaPx
Aug. Tuned (for Dann)
Aug, Naive
Figure Generalization (left), amortized inference (mid), and robustness (right)) performance for VAFs trained with Eqs. (7)-19) Being slightly better than augmentations, DMaaPx consistently has the best performance on the test set and the smallest generalization, amortization, and robustness gap.
We also evaluate whether the improvements in generalization from DMaaPx and augmentation propagate to downstream tasks for which VAEs are better suited than diffusion models. We assess the represcntation learning performance by classifying the mean of (s(z) x) for each æ. We also test both the quality and robustness of the learned representations by using CIFAR-10-C (Hendrycks & Dietterich, 2019), a dataset containing 19 versions of the CIFAR-10 test set, each corrupted with a different corruption. For every test set, we encode each image to its μ and then split all representations into two separate subsets. We use one subset to train classifier and test on the other. Our experiments include four classífiers: logistic regression (LR), support vector machine (Boser et al., 1992) with lincar kernel (SVM-L), radial basis function kernel (SVM-RBF), and k-nearest neighbors (kNN) with k = 5. Figure 4 (left) shows that classifiers trained with representations from DMaaPx (green) outperform normal training on average (details in Appendix G). By contrast, augmentation (purple) hurts performance.
SVM with
Comparison to Prior Work on Modifying the Training Procedures. In Figure 5 we compare the generalization performance of DMaaPx to the previously proposed Reverse Half Asleep (RHA) method of (Zhang et al., 2022). Unlike DMaaPx, which focuses on the training data, RHA modifies the training procedure of a VAE such that first a VAE is trained 500 epochs, and then the decoder is fixed on only the encodor is trained on samples of the decoder for an additional 500 epochs. We find that DMaaPx, being conceptually much simpler, outperforms RHA significantly.
5.3 Smoothness of Latent Space
We hypothesize that an overfitted encoder often learns a less smooth fa( meaning that a small change in æ can result large difference in the latent space. Here, we report an exem- plary density for an interpolation in the latent space of a VAE trained on CIFAR-10 with normal training and DMaaPx. We map two data samples æp and æ from the test set of CIFAR- 10 to their latent means using the encoder 2n = 440 = f (2o) where f (x) denotes the encoder returning only the mean of (x()). We interpolate linearly in the latent space and evaluate log (za Ta) where za azo + (1 - a) a E [0, 1] and C.
100
togs aa))
(L
-5.0 0-
-7.5-
-10.0
Trating
0.0
0.2
0.4
0.6
0.8
1.0
is a reconstruction of 2,-
Density of log da evaluated on a line that linearly interpolates between two data samples from the test set of CIFAR-10. DMaaPx is seoother than normal training.
Figure 3:
We find that, firstly, the density is higher for DMaaPx compare to normal training and, secondly, the latent space appears to be smoother for DMaaPx compared to normal training.
8

---

Published in Transactions on Machine Learning Research (12/2024)
In Improvement in Accuracy
over Normal Training
mibu New
101
Evalated Piza
110
0.06
108
Evslatee on P.n.
11.0
-10
ELBO
0.00
-1..5
-006
-114
Normal Training (on Pusk A
12.0
Kareal Toaiing
Nareal Tea BBM
Dxslased
DM
DMafk
Dialatted
SVM-L SVM-RBF KNN
-11
200
4000
800
1000
Classifier
k (x Damm
Epoch
Figure Left: Improvements in classification accuracy over normal training for various classifiers trained on the latent rep- resentations of CIFAR-I0 10-C. Each column contains 19 3 points (i.c., 19 corruptions, cach VAE trained with 3 random sceds). Right: Generalization performance as a function of the amount k of training data sampled from a diffusion model. Horizontal blue lines show baseline performance (VAE trained directly on Dirain.. All VAEs were trained for 1000 effective epochs. k ~ 10 suffices.
Figure 5: Comparison between DMaaPx and Reverse Half Asleep (RHA) (Zhang et al., 2022). Dotted vertical line shows epoch when decoder is frozen for RHA. "Normal Training + RHA" improves upon baseline but it is still worse than DMaaPx. Also, adding RHA on top of DMaaPx does lead to improvements.
xin
Generalization Gap
Caps
X11
Amortization Gap
10
Robustness Gap
ELBO (Test Set)
Evaluated on Duss
MS. SSIM
0.9
MS-SSIM[e' 2x1
ELBO
-10
Evaluated on Dust
-10-
(1(x
go(=
0.8
(Input sim.)
MS-SSIM(a' .*1
-11
0.7
(Reconstruction sim.)
2001
400
603 Epoch
800
1000
200
diil
10000
200
600
600
800
1000
Epoch
Epoch
Normal Training (m,
Normal Training (m, 10)
Generalization (left), amortized inference (mid), and robustness (right) performance for models with different m, but the same n. 256, using normal training. Due to computation constraints, we only plot the amortization gap for m, 10 up to epoch 700.
Figure 6:
5.4
Do need "unlimited" synthetic data?
With the diffusion model DMaaPx, we can train a VAE on unlimited samples from (DM()=), which improves various performance metrics as demonstrated above. However, generating lots of samples from a diffusion model is computationally expensive (see also Appendix D.2). Therefore, further explore: "Do we really need infinite number of samples?" The answer, reassuringly, is "No".
Figure (right) shows the generalization performance of DMaaPx on CIFAR-10 where the amount of training data is restricted to & x [Ptrain| with & = 1,..., 1000. After i effective training epochs samples repeat. All models are trained for 1000 effective epochs. The horizontal blue lines represent the generalization gap of normal VAE training (on Drrain and & 1) epoch 1000 from Figure 2 (left). For 1, DMaaPx matches normal training CIFAR-10. The ELBO plateaus for k 2 10, indicating that approximately 10 [Ptrain] samples offer similar generalization 1000 x [Dtrainl samples.
5.5 Increasing (0zl improves generalization and minimizes the gaps
Figure shows the generalization (left), amortized inference (mid), and robustness (right) performance between the bascline parameter sctting (m, = 1, n, 256) and the sctting with an increased amount of parameters (m, 10, n. 256). Note that we only increase |Oz] here and keep |e-z/ unchanged. We observe that m, 10 has the higher ELBO Daea (left), and the smaller generalization, amortization, and

---

Published in Transactions on Machine Learning Research (12/2024)
Lass ELBO on Dast
Goneralization Gap
-10
Ceneralization
ELBO
-11
nakaned Peu
-00.3
nokuned se Pas
Norral Traiiig
-11.5
DitsaPs
ma 61
16 64
100)
230
400
500
1001
100
304)
000
300
R2
71.
(a)
(b)
(c)
(d)
Figure 7: (a) and (b) show the changes of ELBOs and generalization gaps while fixing ne 256 and
increasing mg- (c) and (d) show the effects of increasing ne while keeping my fixed. Besides normal training, we also evaluate DMaaPx in this setting.
robustness gaps (Eqs. 3-5). The resulting performance improvements are similar to DMaaPx in Figure 2, where we also see higher ELBO test and smaller gaps.
We also investigate whet her continuing to increase my will lead to better or worse generalization performance. Figure 7 (a, b) show trends of the ELBOs on Diest and the generalization gaps (Eq. (3)) as the result of increasing m, from to 64. We see that increasing m, leads to higher test ELBO for normal training until m, 16, after which the test ELBO stays around the same level. We also find that the benefit of using synthetic data (i.e., DMaaPx) for improving test ELBO diminishes as m, increases. In particular, after m, = 16, both normal training and DMaaPx have similar test ELBOs. However, we want to highlight that DMaaPx has always smaller generalization gap than normal training, even after m, = 16. Small generalization gaps are desirable as they imply that the ELBO evaluated on the training set can be used to prodict the performance of a VAE on a held-out test set.
5.6 Increasing 10-z| might hurt generalization
We further investigate the effect of increasing parameters along the other complexity axis (i.e., (0-xl). Figure 7 (c, d) show the test ELBOs and the generalization gaps (Eq. (3)) as functions of n. = 1 to 512. Subplot. (c) shows that when m, = and m, = 2, increasing n. results in the classical U-shape of overfitting for normal training, where the test performance first grows then drops. Increasing m, further to and 64 tilts the right-hand side of the U-shaped curve upward and eliminates the symptom of declining generalization w..t.. larger ne: which aligns with Figure 7 (a). In particular, when m, = 64, we see improved generalization as we keep increasing De: even though the improvement from enlarging m, is already saturated (as showed in Figure 7 (a)). This implies that: (I) the two sets of parameters O₂ and O- do indeed have different meanings; (2) increasing |0-z| when |Oz/ is small hurts generalization performance; (3) increasing both [0-z] and |0x| leads to better generalization than increasing only one of them.
The plot for generalization gaps (Figure 7 (d)) shows that, by increasing ne: the gaps for normal training become larger. However, increasing m. reduces the gaps (which aligns with Figure (b)). In addition, both plots (Figure (c) and (d)) show that using synthetic data (i.e., DMaaPx) on top of adding parameters does not hurt and often improves generalization. Espocially, using DMaaPx constantly results in near-zero generalization gaps (see the overlapping flat lines in Figure 7 (d)).
5.7 Double descent in VAEs?
Section 5.5 and Section 5.6 indicate that the modern wisdom of "larger models generalize better" does indeed apply to VAEs. Therefore, we are curious whether the double descent phenomenon also ap- plies here. Figure: 7 (a) and (c) show that if fix one complexity axis and only increase parame- ters along the other complexity axis, the test ELBO exhibits either a U-shaped or L-shaped curve, which does not look like double descent. However, one can artificially construct a double descent behavior if one sequentially increases parameters first along one axis and then along the other axis, and plots the test ELBO against the overall model complexity (i.c., the total number of parameters). This is illustrated in Figure& 8. The figure shows the negative lest. ELBO, which shows three phases:
10

---

Published in Transactions on Machine Learning Research (12/2024)
13 x100
(1) we start with fixed m; and increase n. from to 256 (the cyan U-curve); (2) we then fixed nz = 256 and start increasing m from 1 to 64 (the brown vertical line); (3) lastly, we fixed my 64 and increase ny from 256 to 512 (the cyan horizontal line). Phase (2) seems vertical due to the fact that increasing my by adds much loss raw parameters than increasing nz by
Negative ELBO
12
Fixed m0;
10-
Fixed 1%
10
9-
m0
Now we can indications of double descent behavior as the neg- ative test ELBO goes down, up, and down again we keep in- creasing the total number of parameters in the model. While the double-descent behavior was enforced artificially here by the choice of trajectory in model complexity space, it is conceivable that a trained models may allocate resources a similar way while follow- ing more natural trajectory. This aligns with our discussion in Section 4.2 and the work by Curth el. al. (2023), who point out. that double descent in non-deep models is due to increasing parameters along distinct complexity axis sequentially but plotting them on a
0.0
0.5
1.0
1.5
2.0
Number of Parameters ((0l) kt0
Figure 8: By increasing the total number of parameters first along the complexity axis [0-z| (i.e., larger ne: cyan), then along the complexity axis O (i.e., larger m,, brown line), then back to |6-=| (cyan), we see indica- tions of double descent.
combined complexity axis.
6 Conclusion
In this paper, we study overfitting and generalization in VAEs by investigating the effects of (a) training with synthetic data and (b) increasing the number of model parameters, both of which are timely questions to be investigated in generative models. We find that training samples from diffusion model that pre-trained on the training dataset consistently improves performance. Generally, increasing the amount of parameters also helps. However, when constrained by resources, one should prioritize the parameters related to the latent dimension over the parameters that do not directly impact the latent dimension. Applying (a) and (b) together can further improve the performance. Additionally, we find indications a double-descent phenomenon in VAFs which we defer to future work for an investigation in detail.
Acknowledgments
The authors would like to thank Anna Kuzina, Zhen Liu, and Weiyang Liu for helpful discussions. Funded by the Deutsche Porschungsgemeensccaal (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC number 2064/1 Project number 390727645. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S18039A. Robert Bamler acknowledges funding by the German Research Foundation (DFG) for project 448588364 of the Emmy Noether Programme. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Tim Z. Xiao and Johannes Zenn.
References
Alexander Alemi, Tan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottle- neck. In International Conference on Learning Representations, 2016. 2, 6
Sina. Alemohammad, Josue Casco-Rodriguer, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go MAD. In Interna- tional Conference on Learning Representations, 2024. 1, 4, 5,
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019. 4
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXin:2304.08466 2023. 4, 25
Johannes Ballé, Valero Laparra, and Ecro P Simoncelli. End-to-end optimized image compression.
International Conference on Learning Representations, 2017. 29
11

---

Published in Transactions on Machine Learning Research (12/2024)
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image
compression with a scale hyperprior. In International Conference on Learning Represenlations, 2018. 1,
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063 3000330070 2020. 4
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849- 15854, 2019. 2, 4
Quentin Bertrand, Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models their own data. In International Conference on Learning Representations, 2024. 1,
Scbastian Bischoff, Alana Darcher, Michael Deistler, Richard Gao, Franziska Gerken, Manuel Gloeckler, Lisa Haxel, Jaivardhan Kapoor, Janne K Lappalainen, Jakob H Macke, et al. A practical guide sample- based statistical distances for evaluating generative models in science. Transactions on Machine Learning Research, 2024. 26
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the Americon Statistical Association, 2017. 3
Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai models collapse. arviv preprint arXiv:2311. 12202, 2023. 1, 5
Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In Fifth Annual Workshop on Computational Learning Theory, 1992. 8, 29
Alon Brutzkus and Amir Globerson. Why do larger models generalize better? a theoretical perspective via the xor problem. In International Conference on Machine Learning, pp. 822-838 PMLR, 2019. 4
Olmo Cerri, Thong Q Nguyen, Maurizio Pierini, Maria Spiropulu, and Jean-Roch Vlimant. Variational autoencoders for new physics mining at the large hadron collider. Journal of High Energy Physics, 2019 (5):1 29, 2019. 2
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020. 4
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with dis- cretized gaussian mixture likelihoods and attention modules. In IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 7939 7948, 2020. 1,
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autocncoders. In International Conference on Machine Learning, 2018. 1, 3, 4
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateck Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. In Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 4, 25
Alicia Curth, Alan Jeffares, and Mihaela van der Schaar. A u-lurn on double descent: Rethinking parameter counting statistical learning. Xixiv preprint arXiv:2310. 18988, 2023. 2, 6, 7,
Daniel Flam-Shepherd, Tony C Wu, Xuemei Gu, Alba Cervera-Tierra, Mario Kremn, and Alan Aspuru- Guzik. Learning interpretable representations of entanglement in quantum optics experiments using deep generalive models. Nature Machine Intelligence, 4(6):544 554, 2022. 1,
Nicholas Frosst and Geoffrey Hinton. Distilling neural network into a soft decision tree. arXiv preprint NNi1:1711.0978 2017. 6
12

---

Published in Transactions on Machine Learning Research (12/2024)
Rabia Gondur, Usama Bin Sikandar, Evan Schaffer, Mikio Christian Aoi, and Stephen L Keeley. Multi-modal gaussian process variational autoencoders for neural and behavioral data. or preprint or arXim22310.03111 2023. 1, 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. 17, 26
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019. 8, 30
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Newral Information Processing Systems, 2017. 29
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprinl arXiv:1503. 02531, 2015. 6
Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020. 27
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Newral Information Processing Systems, 2022. 7, 25
Diederík P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014. 1
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7, 25 Solomon Kullback and Richard A Leibler, On information and sufficiency. The Annals of Mathematicul Statistics, 22, 1951. 17
Anna Kuzina, Max Welling, and Jakub Tomezak. Alleviating adversarial attacks on variational autoencoders with meme. In Advances in Neural Information Processing Systems, 2022. 2, 3, 4, 17
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to docu- ment recognition. In Proceedings of the IEEE, 1998. 7, 25
Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In International Conference on Machine Learning, 2018. 4
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems, 2018. 1,
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12):124003, 2021. 2, 4, 6
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Takshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2018. 19
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishknn, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 4
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. Tn International Conference Machine Learning, 2014. 1
13

---

Published in Transactions on Machine Learning Research (12/2024)
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Stalistics, 1951. 26
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022. 5
David E Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by back- propagating errors. Nature, 323, 1986. 26
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixcICNN++: Improving the pixcl- CNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017. 7, 17
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport.
In International Conference on Learning Representations, 2018. 29
Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer, and Stefano Ermon. Amortized inference regularization. In Advances in Neural Information Processing Systems, 2018. 1,
Mla Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. orXiv preprint armiv:2305 174933 2023. 1,
5,
Jascha Sohl-Dickstedd Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. Tn International Conference on Machine Learning, 2015. 2
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. 2, 25
Lucas Theis, van den Oord Aäron, and Matthias Bethge. note on the evaluation of generative models. In International Conference on Learning Representations, 2016. 29
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerop: Synthetic images from text-lo-image models make strong visual representation learners. arXiv preprint urXiw:2306. 00904. 2023. 4, 25
Ba-Hien Tran, Giulio Franzese, Pietro Michiardi, and Maurizio Filippone. One-line-of-code data mollif- cation improves optimization of likelihood- generative models. Advances in Neural Information Processing Systems, 2023. 21
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning, 2023. 4, 25
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality as- sessment. In The Seventh Asilomar Conference Signals, Systems & Computers, 2003, 2003.
3
Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse. On the quantitative analysis of decoder- based generative models. In International Conference on Learning Representations, 2017. 1,
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: a novel image datasct for benchmarking machine learning algorithms. arXiv preprint 1xXiv:100.0071. 2017. 7, 25
Tim Z. Xiao and Robert Bamler. Trading information between latents in hierarchical variational autoen- coders. In International Conference on Learning Representations, 2023. 29
Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. Image data augmentation for deep learning: A survey. ar preprint arXiv:2204.0861 2022.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Yibo Yang, Stephan Mandt, Lucas Theis, al. An introduction to neural data compression. Foundations and Trendsl in Computer Graphics and Vision, 2023. 1, 2
Mingtian Zhang, Peter Hayes, and David Barber. Generalization gap in amortized inference. In Advances in Neural Information Processing Systems, 2022. 4, 8, 9
Shengjia Zhao, Jiaming Song, and Stefano Frmon. Infovae: Balancing learning and inference in variational
autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019. 4
Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-vae. In Advances in Neural Information Processing Systems, 2020. 1, 2
15

---

Published in Transactions on Machine Learning Research (12/2024)
Appendix
Table of Contents
A Details The Adversarial Attack
17
B Further Details on VAEs and Ablations
17
B.1
Architectures and Training Cost
17
B.2 Reconstructions
18
B.3 Additional Datasets
................................
B.4
Different Conditional Likelihoods
................................
19
B.5
Training ELBO versus ELBO on Derain
................................
19
B.6
Adding Scheduled Noise to the Training Set
21
B.7 Mixing Synthetic Data with Real Data
21
B.8 Training the Diffusion Model on Subsets of the Training Set
21
B.9
Algorithms for Training VAEs with DMaaPx
22
C Quantitative Results on Performance Gaps
24
D Diffusion Model for DMaaPx
25
D.1 Samples From the Diffusion Model
25
Computational Cost DMaaPx ................................ ................................
25
D.3 Discriminating Synthetic and Real Samples by Classification
26
E Augmentation
27
Practical Evaluation VAEs on Three Tasks
G Practical Evaluation of VAEs on CIFAR-10-C
30
16

---

Published in Transactions on Machine Learning Research (12/2024)
A Details on The Adversarial Attack
We follow Kuzina et al. (Kizina et al., 2022) and construct an unsupervised encoder attack that optimizes the pertubation e to incur the largest possible change in 46(- x),
arg max SKL [gol: x" + 6) (s( x")]
(10)
|
where SKL denoles the symmetric Kullback-Lebble divergence (Kullback & Leibler, 1951). We optimize e for n iterations with projected gradient descent utilizing learning rate of n, The robustness gap (see
Scction 2) is computed over n' real images and n random soeds. The exact hyperparameters can be found in Table 2.
Table 2:
Hyperparameters for the unsupervisod encoder attack.
BinAryMNIST
FashionMNIST
CIFAR-10
n²
50
50
20
n
10
10
10
n
50
50
100
! 1.0
1.0
1.0
8
0.1
0.1
0.05
B Further Details on VAEs and Ablations
In this section we present further details on the VAE models and ablation studies.
B.1 Architectures and Training Cost
This section provides a detailed description of the VAE models utilized throughout the paper.
Table 3: Details on VAE architectures ordered by dataset. MoL refers to the discretized mixture of logistics likelihood model (Salimans et al., 2017).
dataset
likelihood
architecture
BinaryMNIST
Bernoulli
fully-connected
FashionMNIST
fixed-variance Gaussian
fully-connected
FNShionMNIST
MoL
fully-connected residual network
CIFAR-10
MoL
We consider a fully-connected architecture and a residual architecture (He et al., 2016). Table 3 gives more details on the likelihood model and architecture. For all VAFs, we choose the hyperparameteess the VAE models by consulting existing literature. The MoL likelihood for FashionMNIST is discussed in Appendix B.
The fully-connected architecture maps from input dimension of 32 to hidden dimension of 512. After hidden layer mapping from 512 to 512, the output is mapped to latent variable of dimension 16. The decoder mirrors the encoder and maps the latent variable of dimension 16, via three layers, to the original input size.
For the residual architecture we present specific tables for the encoder (Table 4), decoder (Table 5), and the residual block (Table 6).
We train each VAE single NVIDIA GeForce RTX 2080 Ti 11GB GPU.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Table 4: Residual encoder architecture.
layer
dimension
additional parameters
Convolution
Re
kernel size: 4, stride: padding: no bias
BatchNorm
ReLU
Convolution
n ne
kernel size: 4, stride: padding: 1, no bias
BatchNorm
Residual
Residual
ne
Convolution The + 2dz
kernel size: 1, stride: 1, padding: 0, bias
Table 5: Residual decoder architecture.
layer
dimension
additional paramcters
Convolution
d, +
ne
kernel size: 1, stride: 1, padding: 0, no bias
BatchNorm
fe
Residual
f2
Residual
The
Transposod Convolution ne > ne
kernel size: 4, stride: 2, padding: 1, no bias
BatchNorm
Tee
ReLU
Convolution
The - kernel size: 1, stride: 1, padding: 0, bias
Table Architocture of residual block.
layer
dimension additional parameters
ReLU
Convolution
Tee nee
kernel size: 3, stride: 1, padding: 1, no bias
BatchNorm
The
ReLU
Convolution
kernel size: 3, stride: 1, padding: 1, no bias
BatchNorm
The
B.2 Reconstructions
Figure 9 shows reconstructions for CIFAR-10. We reconstruct random subset of 9 samples from the test set. The VAEs model CIFAR-10 well. Therefore, qualitatively the differences between the methods are quite subtle. But still, we can see that the DMaaPx's reconstruction is slightly better for the rightmost image in the second row.
are quite small and simple, they are also non-hierachical, which makes it hard to
B.3 Additional Datasets
Figure 10 shows the three generalization gaps for the BinaryMNIST dataset. Figure shows the three generalization gaps for the Fashion-MNIST dataset. All claims that have been made the main text also hold for these datasets.
18

---

Published in Transactions on Machine Learning Research (12/2024)
Ground Truth
Normal Training
DMaaPx
Aug.Naive
Aug. Tuned
Figure Reconstructions for CIFAR-10 for a random subset of 9 samples from the test set.
Generalization Gap
Amortization Gap
Robustness Gap
10
-10
-70
0.8
MS 55 Mys x1 Orpu: simieely)
ELBO (les, Sett
MS-55 MS- (lecssamaction diiibrtt)
H5-S5IN
0.4
-10
-90
-15
Exalusted
Enalustad on D.e
2.0
100
800
1000
2.0
600
600
500
1000
200
600
1000
Effective Epoch
Effective Epoch
Effective Epoch
Normal Training
DMsaPx
Aug, Tuned (for Drsa)
Aug Naive
Figure 10: Generalization gap, amortization gap, and robustness gap on the BinaryMNIST dataset.
B.4 Different Conditional Likelihoods
VAEs' modeling assumptions for the conditional likelihood po 2) often differ based on data or use case. While a Gaussian likelihood is used for applications that foeus on low reconstruction compression), MoL likelihood is used if the density of the data matters (e.g., generative modeling lossless data compression). While the experiments in the main text considered a MoL likelihood, the model trained on FashionMNIST (Figure 11) uses a Gaussian likelihood and the model trained on MinaryMNIST (Figure 10) uses Bernoulli likelihood (also compare Appendix B.1). Figure 12 also evaluates MoL. for FNshionMNIST, and we observe similar behavior as its Gaussian counterpart in Figure 11 (left). summary, DMaaPx is less prone to overfitting than normal training and augmentation, for all investigated likelihoods.
(e-g., lossy data
B.5 Training ELBO versus ELBO on D traai
Remark (Test data entropy can also affect the ELBO value). Note that from Eqs. (2) and (1), we have
Emspp() [EL)Oe(=)) < Eus=y())) [log - -H(p(a)) s - p(x)),
(11)
where H denotes the (cross) entropy. Therefore, the ELBO on Deest can be higher than the ELBO on Diratu: Dirain and Drest are not drawn from the same distribution, and Drest has lower entropy than Drrain. Indeed, this phenomenon has been observed the out-of-distributtin setting when testing on a low-entropy data set (Nalismick et al., 2018).
Figure 13 shows the ELBOs analogous to the generalization gaps in Figure 2, Figure 10, and Figure 11, but the dotted lines plot the ELBO on the actual training distribution (e.g,, on samples from (D)( for DMaaPx). The point of this plot to warn that comparisons between ELBOs under such different distributions are not mcaningful, and should not be used calculate the generalization gap. For example, note that the plot would suggest a negative generalization gap for data augmentation (purple) on BinaryMNIST. This is consistent with the remark above: since the FLBOO is bounded by the negative entropy of the distributions
19

---

Published in Transactions on Machine Learning Research (12/2024)
Generalization Gap
Amortization Gap
Robustness Gap
400
Enelusted Diac
Exalustted Mas
-100
-100
200
-1240
-2000
-2000
0.6-
SSIMjs wi] (leoal sinlatoo
-100
MS SSiMjar ar) [Racosstrction innliing) W
-200
230
400
600
800
1003
-2000
200
630
2000
200
403
800
1000
Effective Epoch
Effective Epoch
Effective Epoch
Normal
DMaaPx
Aug, Tuned Drau)
Naive
Figure 11: Generalization gap, amortization gap, and robustness gap on the MNSH da Laset.
MNshionMNIST (MoL)
E1300
-199
-1907
-130
400
300
11001
Effective Epoch
Evaluated Puxe
DMaaPs (Proposed)
Evaluated on Dis
Aug. Tuned (for Dras)) Aug,
Normal Training
Figure 12: Generalization performance for MashionMINST with MoL likelihood. We observe similar behavior to left panel in Figure 11, which uses a Gaussian likelihood.
BinaryMNIST
FashionMNIST
CIFAR-10
-05
110000
-75
108000
-500
-55
-11500
2030
-1163
-100
-2000
11880
602
1002
4001
1000
403
1000
4013
1403
1000
Effective Epoch
Eflective
Epoch
Effective
Epech
Training ELSO
Evaluxted on Disi
Normal Training
DMaaPx
Aug. Tuned (for P....)
Aug. Naive
Figure 13: ELBO evaluted on the distribution that is actually used for training (dotted, see Eqs. (7)-(8)).
For augmentations, the test ELBO (solid) is higher than the training ELBO (dotted) in the left two panels, which is an artifact of different entropies of the distributions, see Remark.
which it is evaluated, evaluating it on two different distributions with different entropies exhibits differences unrelated to the generalization gap.
20

---

Published in Transactions on Machine Learning Research (12/2024)
B.6 Adding Scheduled Noise to the Training Set
10500
11000
11500
ELBO
12000
100000
-00000
Fushutted
DMsaPx
Evaluated on Pica
DMaaPt+AAu Mullily
24000
Training
Aug Mullify
200
400
6000
800
1000
Effective Epoch
Figure 14: Generalization gap as function of effective epochs. We compare normal training to DMaaPx, and the combination of DMaaPx and Aug.Mollify. Aug.Molliff reaches a similar performance as the baseline. Combining DMaaPx and Aug.Mollify does not improve upon DMaaPx.
Tran et al. (Tran al., 2023) show in their recent work that adding (scheduled) noise the training data during the training process helps the optimization of likelihood- generative models. More specifically, they anneal the noise schedule from large variance to small one for half of the training epochs and, finally, use the original training data for the other half of the training epochs. They argue that data typically resides on a low-dimensionall manifold that is embedded in a larger dimension. By adding noise to the data during training one circunvents the problem of density estimation in input space with few samples and, additionally, prevents the model from overfitting the data manifold. One can also view this idea from continuation method perspective as the method finds a good initialization point for the optimization.
Figure 14 shows the performance of the mollification method (Aug.Mollify) applied to VAEs the CIFAR- 10 dataset (dark). We compare the performance to normal training (blue), DMaaPx (green), and the combination of DMaaPx and the Aug.Mollity. We find that Aug.Mollify reaches a similar performance as the baseline but does not improve upon it. Also, combining DMaaPx and Aug.Mollify does not, bring improvements compared to DMaaPx. However, DMaaPx and Aul.Mollify have similarly small generalization gaps.
B.7 Mixing Synthetic Data with Real Data
We evaluate the performance of VAEs trained on a mix of x E {0, 20, 40, 60, 80, 1003% of real data and 100 - 2% of synthetic data. We train the models without augmentation and with Aug. Tuned. Note that x 0% (without augmentation) corresponds the DMaaPx method and x 100% (without augmentation) corresponds to the normal training baseline. 100% (with augmentation) corresponds the Aug. Tuned baseline.
Considering augmentation, when moving from mix of 100% of real data to 0% of real data, we find that the performance is continuously increasing (i.e., the ELBO evaluated on Drest increases) and the gen- eralization gap is shrinking. Adding augmentation improves both, the ELBO evaluated on D and the generalization gap. Note that DMaaPx (ær = 0%, no augmentation) shows the highest ELBO evaluated on Drest and the smallest generalization gap.
B.8 Training the Diffusion Model on Subsets of the Training Set
We evaluate the perfomance of DMaaPx trained on samples from a diffusion model which has been trained on æ E {10, 30, 50, 70, 90%% of CIFAR-10.
Figure 16 shows ELBO values evaluated on Dirain and Dustt The test performance DMaaPx using samples from diffusion model that has been trained on 30% of data, equals the test performance of normal training.
21

---

Published in Transactions on Machine Learning Research (12/2024)
x
10
-10.50
No Augmentation
Evalusted on Dusis
Augmentation
Evalucted on Post
10.75
ELBO
11.00
11.25
11.50
0%
20%
40%
60%
80%
100%
Percent Real Data
Figure 15: Last ELBO evaluated on Drran (dashed) and Deest (solid) with respect to 2% of real data for no augmentation (brown) and Tuned (pink). x = 0%, brown: DMaaPx. x = 100%, brown: Normal Training. x 100%, pink: Aug. Tuned. The vertical line shows the ELBO evaluated on the test set for DMaaPx. DMaaPx has highest ELBO and smallest generalization gap. More details are provided Appendix B.7.
However, DMaaPx shows significantly smaller generalization gap. Using more data for training the diffusion model improves upon normal training with diminishing returns for 2 70% of data.
x10
-1..0
ELBO
11.55
-2200
Training DMaaPx
Evaluated Evaluated on Dass
10%
30%
50%
70%
90%
Amount of CIFAR-10 Data Used for Training the Diffusion Model
Figure 16: ELBO evaluated on Dtrain (dashed) and Dtest (solid) for DMaaPx (green) trained on samples from a diffusion model which has been trained on æ (10,30, 30, 50, 70, 903% of CIFAR-10. We plot the performance of normal training (i.e., VAE trained with the original full CIFAR-10 10 only) as horizontal blue lines. Using 30% of data for DMaaPx shows similar performance than normal training (with significantly smaller gap). Using more data outperforms normal training. More details are provided in Appendix B.8.
B.9 Algorithms for Training VAEs with DMaaPx
Below, we provide pseudocode for training VAEs with normal training (Algorithm 1) and DMaaPx (Algo- rithm 2).
22

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm Training VAEs Normal
Given: Derains Og {&v. Φul: Number of epochs T, Batch size M
1; for t t= 1,... T do
for i i= 1,... p Pusinl do
Get M training examples: [I,,... [MM] = Dtrain - x M : M];
[( Im j-11) - Em [ELBO0,-1 (2m));
, = --11-1 VC:
j -=j+1; 1;
end
end
Algorithm 2 Traiiing VAËs DMaaPx
Given: (D) O, = {8v, Φo), Number of epochs T, Batch size M, N [Dusiul
// Step 1: Sampling the training data
DDM [TxN]] PDM(#) // Parallelizable; only need to do this once.
// Step 2: Training the VAE
;=1; for t= 1......,5 do
for i= 1,... do
Get M training examples: "., [.M] = DDMli x (i- ( 1) x M: : M]:
[(( [mM] 9-11) M Σm [ELBO (2mm));
0-=0-11-11 VC
j=j+1;
end
end
23

---

Published in Transactions on Machine Learning Research (12/2024)
C
Quantitative Results on Performance Gaps
Table 7 assigns quantitative values to the visual evidence in Figure (CIFAR-10), Figure 11 (FashionMNNIST), and Figure 10 (BinaryMNIST).
Table 7: Quantitative values of the performance gaps visualized in Figure 2, Figure 11, and Figure 10. Bold numbers indicate the smallest gap within a dataset.
generalization gap
amorization gap
robustness gap
(gg, Eq. (3))
(Gn Eq. (4))
(gr. Eq. (5)
Normal Training
25.76
20.32
0.49
Binary MNIST
DMaaPx (ours)
0.78
7.01
0.50
Aug. Tuned
8.16
9.34
0.79
Aug.Naive
6.38
8.16
0.74
Normal Training
1234.50
1135.89
0.39
Fashion MNIST
DMaaPx (ours)
136.57
593.39
0.31
Aug. Tuned
614.93
815.52
0.21
Aug.Naive
500.33
729.83
0.11
Normal Training
841.54
835.86
0.41
DMaaPx (ours)
5.44
288.82
0.30
CIFAR-10
Aug. Tuned
94.28
328.08
0.35
Aug.Naive
228.05
390.25
0.35
24

---

Published in Transactions on Machine Learning Research (12/2024)
D Diffusion Model for DMaaPx
We follow the setup of Karras et. al. (Karras et al., 2022) for the design and training of our diffusion model. However, we do not use the proposed augmentation pipeline during training.
We train the diffusion model on 50,000 training data points (i.e., the size of the original CIFAR-10 training set in the case of CIFAR-10) for 4000 epochs. Each model is trained 8 NVIDIA A100 80GB GPUs for approximately 1 days.
We utilized the deterministic second-order sampler as proposed by Karras et al. (Karras et al., 2022) with 18 integration steps. Fach sampled image utilizes a unique initial seed. We sample on a single NVIDIA A100 40GB GPU. Sampling 50, 000 images takes approximately 25 to 30 minutes.
D.1 Samples From the Diffusion Model
Figure 17 shows samples from the diffusion models trained. On CIFAR-10 we report a FID score of 3.9537. Scores on BinaryMNIST and FashionMNIST are ommited as those are not widely reported and heavily depend preprocessing (Song et al., 2021).
5
8
7
2
4
3
4
5
0
/
8
2
4
0
0 8
+
9
D
7
5
1
8
/
9
9
8
5 9
2
5
O
4
3
0
9
3
1
1
6
7.
6
7
0 2
3
9
5
7
2
5
1
1
0
6
9
4
&
0 $
BinaryMNIST
FahionMNIST
CIFAR-10
Figure 17: Samples of the diffusion models trained BinaryMNIST (LeCun et al., 1998), FASH (Xiao et al., 2017), and CIFAR-10 (Krizhevsky et al., 2009).
D.2 Computational Cost of DMaaPx
This section discusses the computational requirements of the DMaaPx method in general.
In this paper, we use the method DMaaPx as tool to study the difference between naive augmentations and synthetic data from diffusion models when training VAEs. Our evaluations show that there are scenarios where exploiting the modelling assumption (i.e., both VAEs and diffusion models are probabilistic models) helps, which often overlooked by the generative modelling community.
Ia pre-trained diffusion model available, DMaaPx can be applied with a negligible compulational overhead (see Figure 4, right). This setup follows recent trends where using synthetic data from pre-trained diffusion models is becoming common practice for, e.g. classification (Azizi et al., 2023), robustness (Croce et al., 2021; Wang et al., 2023), and representation learning (Tian et al., 2023).
If no pre-trained diffusion model is available, a diffusion model has to be trained a first step to apply DMaaPx. Whether this is feasible might depend on the specific application. However, we want emphasize that once the diffusion model trained, can be used to generate as many samples as needed for training many VA Es needed.
25

---

Published in Transactions on Machine Learning Research (12/2024)
D.3 Discriminating Synthetic and Real Samples by Classification
One way to evaluate the quality of the generated samples is by training a classifier to discriminate the synthetic and the real data. This method is also known as Classifier Two-Sample Tests (C2ST), which is a metric for evaluating generative models (Bischoff ct al., 2024). Herc, we train five classifiers to discriminato real data (images taken from CIFAR-10) and synthetic data (images sampled from the diffusion models) where we use a E (30, 50, 70, 90, 100)% % of the original CTFAR-10 to train the diffusion models, respectively.
We ResNet 18 (He al., 2016) for classifying real data (with label 1) and synthetic data (generated by the diffusion model). We train each classifier for 14 epochs with batch size of 256 with stochastic gradient descent. (Robbins & Monro, 1951) with a learning rate of 0.001, a momentum (Rumelhart et al., 1986) of 0.9 and weight decay of 5 x 10-4.
Figure 18 shows the classification accuracies for discriminating between real and synthetic data. While all samples can be classified with an accuracy > 50%, we find a negative trend between the amount of data that the diffusion model is trained on (ar {30, 50, 70, 90, 1003%) and the classification accuracy. Hence, it is easier to classify samples from a diffusion model that has been trained less data than samples from diffusion model that has been trained more data. Using the maximum amount of training data available, which DMaaPx docs, leads to samples that can be distinguished least from real data.
100
Classification Accuracy
90
80
70
60
50
30%
50%
70%
90%
100%
CIFAR-10
CIFAR-10 CIFAR-10 CIFAR-10
CIFAR-10 (DMaaPx)
Figure 18: Accuracy of classifier trained to distinguish real data (CIFAR-10) and data sampled from diffusion model. The diffusion model is trained on z {30, 50, 70, 90, 100)% of real data. The more data is used the closer synthetic samples are to real samples (i.e., smaller classification accuracy). Using 100% of CIFAR-10 leads to samples least distinguishable from real data. Accuracies are averaged over random seeds. Error bars show +1.9 standard deviations. For details see Appendix D.3.
26

---

translation Published in Transactions on Machine Learning Research (12/2024)
E
Augmentation
We use the augmentation pipelinc originally proposed for GAN training following Karras et al. (Karras et al., 2020). Each specific augmentation is applied with a probability of E 00.1, For each dataset we compare two sets of specific augmentations.
1.
The hyperparameters for each specific augmentation are tuned by hand with the goal of imitating the data generating distribution that produced the dataset.
We use naive of specific augmentations that is targeted image datasets (in general).
Table lists naive augmentation for BinaryMNIST, FashionMNIST, and CIFAR-10. Table lists augmen- tation tuned to the BinaryMNIST and the PashionMNIST dataset. Table 10 lists augmentation tuned to the CIFAR-10 datset.
They perform similarly overall in Figures 2, 10 and 11. However, Aug.Naive outperforms Tuned in generalization BinaryMNIST and FashionMNIST, and in robustness across all datasets. This is surprising naive augmentation might produce out-oo--istribuution data, like a horizontally flipped digit "2", potentially impairing performance. Thus, designing augmentation can be labor- inor-intensive.
Table 8: List of specific augmentations applied to BinaryMNISTT, MNSH and CIFAR-10. We refer to this set as "naive" augmentation as is targeted towards images in general (and not towards specific datasets). Each specific augmentation applied with probability
augmentation
description and hyperparameters
horizontal flip
flip an image horizontally
translation
translate image in æ and direction for / 1,,1)).3 pixels
scaling
scale image by 2" with &scale [0, 0.2]
rotation
rolate an image by d degrees with E [0, 10]
anisotropic scaling
do anisotropic scaling with scale 20aniso-scal (Aaniso E [0, 0.2]
anisotropic rotation
do anisotropic rotation with probability of 0.5
brightness
change the brightness of image by Obrightnesa E [0,0.2] change contrast of an image by 20oontratt where conntrast E [0, 0.25] change the hue by rotation of Thue with Thue E 0.25.25.m m]
contrast
hue saturation
change the saturation of an image by 20saturation where Cesturation E [0,0.5]
Table 9: List of specific augmentations applied BinaryMNIST and FashionMNIIT.. The set tuned lowards MNIST and PashionMNIST. Each specific augmentation is applied with probability b.
augmentation
description and hyperparameters
translate an image in x and direction for t E 1,,1, pixels
scaling
scale an image by 2 with Tsaale E [0, 0.15)
rotation
rotate image by d degrees with [0, 10]
anisotropic scaling
do anisotropic scaling with scale 2aantoo-scae (a scale E [0, 0.15) do anisotropic rotation with a probability of 0.4
anisotropic rotation
27

---

Published in Transactions on Machine Learning Research (12/2024)
Table 10: List of specific augmentations applied to CIPAR-10. The set is tuned lowards CIFAR-10. Each specific augmentation is applied with probability
augmentation
description and hyperparameters
horizontal flip vertical flip
flip image horizontally (applied with probability 1)
flip an image vertically
scaling
scale an image by 2 with Gsale E 0.2)
rotation
rotate image by d degrees with d E [0, 360) do anisotropic scaling with scale 2anteo--ooil aoansso E (0,0.2)
anisotropic scaling
anisotropic rotation
do anisotropic rotation with probability of 0.5
28

---

Published in Transactions on Machine Learning Research (12/2024)
F Practical Evaluation of VAEs on Three Tasks
The improvements of generalization performance, amortized inference and robustness in VAEs have direct impacts on the applications of them. In this section, we evaluate three popular tasks that VA are used for, based on whether task only involves the encoder, the decoder, or both as in (Xiao & Bamler, 2023): (a) reprosentation learning (i.e., using only the encoder); (b) data reconstruction (i.e., using both the encoder and the decoder); and (c) sample generation (i.e., using only the decoder).
Representation lcarning (with classification as the downstream task). We evaluate the represen- tation learning performance by classification accuracies on the mean la of qø(2 ()) for each æ. First, we find the learned representations f for all data points in the CIFAR-10 test set. Afterwards, we split them into two separate subsets. We use one subset to train the classifier, and test on the other subset. Our experiments include four different classifiers: logistic regression, & support vector machine (Boser et al., 1992) with radial basis function kernel (SVM-RBF), SVM with linear kernel (SVM-L), and k-nearest neighbors (kNN) with k- = 5. Table 11 (representation learning; RL) shows the resulting test accuracies across all models considered. We find that VAEs trained with DMaaPx (in bold) are slightly better than other models on average (with overlapping standard deviations), which implies the task of representation learning might. benefits from the smaller gaps evaluated in Section 5.2.
Data reconstruction. Tasks such as lossy data compression (Ballé et al., 2017) rely on the reconstruction performance of VAEs. We evaluate the reconstruction performance of VAEs trained on CIFAR- 10 using the peak signal-to-noise ratio (PSNR; higher is better). Table 11 (reconstruction; RC) shows that DMaaPx performs slightly better than the others average, but with overlapping standard deviations.
Sample generation. We evaluate the quality of samples generated by VAEs trained on CIFAR-10 with the methods explained in the main text (Normal Training, DMaaPx, Aagve Aug.Tunele Tuned). We report Fréchet Inception Distance (Heusel et al., 2017) (FID; lower is better) and Inception Score (Salimans et al., 2018) (IS; higher is better). Table 11 (sample quality; SQ) shows that DMaaPx performs slightly better than the others on average, with overlapping standard deviations, when sample quality is measured in FID, but Normal Training performs better when sample quality is measured in 1S.
Overall, VAEs trained with DMaaPx show improvements for representation learning and data reconstruction, and perform similarly to normal training on sample quality. At the same time, VA trained with both augmentations seem to have slightly worse performance for representation learning and sample generation, and perform similarly on the reconstruction task when compared to normal training. The results of DMaaPx in the table consistent with our claim that the proposed method mainly fixes the encoder, which affects representation learning and reconstruction but not sample quality. Additionally, Theis al. (Theis et al., 2016) show that a generative model with good log- likelhooldood (i.e., high test ELBO in the case of a VAE) does not necessarily produce great samples.
Table 11: Evaluation of downstream applications of VAEs on CIFAR-10: representation learning with classification as the downstream task (RL), reconstruction (RC), and sample quality (SQ). Results are averaged over 3 random seeds. Note that most differences are smaller than the standard deviations. See Appendix for a discussion of the results.
Normal Training
DMaaPx (ours)
Aug.Naive
Aug. Tuned
log. reg, (t)
0.370 + 0.018
0.383 + 0.018
0.359 0.004
0.361 0.014
RL SVM-RBF (1)
0.427 1 0.014
0.438 + 0.015
0.421 1: 0.004
0.420 0.016
SVM-L (1)
0.367 1 0.015
0.380 + 0.014
0.365 1 0.005
0.366 0.022
KNN (1)
0.325 + 0.006
0.327 + 0.035
0.300 0.004
0.299 0.028
RC PSNR (t)
16.087 + 0.042
16.370 + 0.195
16. + 0.017
15.924 + 0.205
FID (L)
219.256 12.124
219.081 + 14.894
237.238 43.218
240.898 + 11.072
SQ
IS (1)
1.818 + 0.155
1.614 + 0.076
1.656 1: 0.047
1.612 0.083
29

---

the strongest degree of Published in Transactions on Machine Learning Research (12/2024)
G Practical Evaluation of VAEs on CIFAR-10-C
On CIFAR-10-C (Hendrycks & Dictterich, 2019), we evaluate the representation learning capabilities of VAEs trained with DMaaPx, Aug-Naive, and Aug. Tuned by evaluating the classification performance of a classifier trained in latent space. The experiment are similar to the evaluation of representation learning above (see Appendix F). For each the classes of corruptions we train four classifiers on a subset of the CIFAR-1O-C test set and evaluate its performance on the subset that was not used for training (termed "overall" performance). Additionally, we sample random (one out of 19 possible) corruption for each of the images of CIFAR-10-C and evaluate its performance. During training corruption.
Figure 19 shows results across all 19 corruptions and the "overall" performance. DMaaPx achieves consis- tently the best mcan across all corruptions for the lincar classifiers and DMaaPx achieves the best mean for almost all of the corruptions when regarding the non-linear classifiers. See also Section 5.2 for a discussion of the experiments.
30

---

Published in Transactions on Machine Learning Research (12/2024)
IPER Fampessics
Fastie Traddor
03--
0.0
LR
SWM Unoe SVM RBF
MNN
LR
SKM Linw SVW RBF
JWN
LR
SYM Linar SVM REF
ANN
UR
SYM Linar S4M RSF
KNN
Satarnna
Gounton Baace
Shot Rarre
Goet Bar
0.5
0.1-
SUM UieM
1R
Linar RBF
JMNN
SVM Livar SWM
SVM 1.WWY SUM
Piedute
0.2
0.0
SVR Linar SYM
MMN
LH
SKIM HBT
JMN
SWM Liwar
SVM 183
6MN
UK SUM LENDr SWI
Accusny
Figure 19: Classification performance of four classifiers (LR: linear regression, SVM kernel: support vector machine with the specified kernel, kNN: d-nearest-neighbor classifier) trained in latent space across the 19 corruptions that are part of CIFAR-10-C. DMaaPx achieves the best mean performance for almost. all corruptions under consideration. For details consult Appendix
31


Reviews:
[{"review_id": "qPA0P0RLpX", "reviewer": "Reviewer_NGVK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well organized and clearly presented.\n- The paper offers practical guidelines for improving VAEs, such as prioritizing latent dimension parameterization and effectively using synthetic data.\n- The paper provides valuable insights into the double descent phenomenon for VAEs.\n\n\nWeaknesses:\n- Experiments are limited to image datasets, limiting generalization to other domains.\n- DMaaPx implies that diffusion models are better than other types of generative models (e.g., VAE, GANs) at capturing the real data distribution. Additional theoretical or experimental evidence would strengthen the paper.\n\nQuestions:\n- The paper mentions training diffusion models on CIFAR-10/BinaryMNIST/FashionMNIST for 2.5 days on 8 GPUs. Given the small sizes of these datasets, is this a mistake? If accurate, the computational cost is orders of magnitude higher than training the VAE itself. Is this significant overhead worthwhile? Is it computationally efficient in terms of learning representation compared to other methods?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores improving VAE generalization using synthetic data from diffusion models and overparameterization. It shows synthetic data reduces overfitting better than augmentation, and increasing latent parameters enhances performance, revealing practical insights and the double descent phenomenon.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Typo: \"double-decent\" should be \"double-descent\" in the last paragraph.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sQECwygLd7", "reviewer": "Reviewer_aqGx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n\nFirst of all, the article is well written and the main ideas and contributions are easy to follow. The new \"DMaaPx\" procedure seems very promising to improve the general performances of a VAE. Next, the practical interest of a VAE, in particular the dimensionality reduction providing a compressed representation of high-dimensional complex data, is really well explained and motivated. Moreover, I really appreciated the descriptions of both methods A and B in Section 4. Arguments in favor of using a diffusion model rather than data augmentation are convincing and illustrated numerically in Section 5.\n\n**Weaknesses**:\n\nSince it is the main point of this article, I would first appreciate in the introduction more explanations on why the encoder is more susceptible to overfit than the decoder. It would add even more strenght to the motivation of the article.\n\nMoreover, once the diffusion model is trained, I don't really understand when the new data are generated according to $p_{DM}$ and how they are used during the training process. It would then highligh the contribution to present the \"DMaaPx\" procedure as a pseudo-code for example.\n\nIn addition, my main methodological concern about the new procedure is the training of the diffusion model. Indeed, we can see in Appedix D that the diffusion model is trained on 200,000,000 data. It is a huge amount of data which might not be easily computationally tractable. The resulting diffusion model is then a very good approximation of $p_{data}$ and can explain the very good results of the \"DMaaPx\" procedure. Thus, it would be very beneficial for the impact of the contribution to evaluate the numerical performances of the \"DMaaPx\" procedure when training the diffusion model with a more reasonnable amount of data. If the corresponding results are satisfying, the \"DMaaPx\" procedure may could prevent overfitting even in a small data context.\n\n\n**Questions and perspectives**:\n\nHere are some additional questions and remarks I was wondering during the reading :\n- It is mentionned that a diffusion model is less susceptible to overfit than a VAE. With a more reasonnable amount of training data, can the diffusion model overfit ? And if not, why ?\n- Are the metrics in Section 2 proposed in the article or are they extracted from other articles ? If so, some additional references would be appreciated to clarify this point.\n- How can we compare the proposed contributions to classical methods to prevent overfitting, like early stopping or dropout for example ? A little discusion on it can be beneficial for the article.\n- At last, can the new \"DMaaPx\" procedure be a remedy to the posterior collapse phenomenon [1,2] which can badly affect the performances of a VAE ? It would be interesting to mention it at least as a perspective.\n\n**References**:\n\n[1] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015\n\n[2] Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv:1901.05534, 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors investigate overfitting when training a VAE from a \"deep learning perspective\". To that end, they propose first a new procedure called \"DMaaPx\" which aims to prevent as much as possible overfitting when training a VAE. The main idea is to use a pre-trained diffusion model, trained on the available dataset, which provides an accurate approximation of the real distribution $p_{data}$ and allows to draw as many samples as we want according to the resulting distribution $p_{DM}$. They show on some benchmark datasets that the new \"DMaaPx\" procedure improves generalization compared to a classical VAE. They second study the influence of the size of the model on overfitting, and show that increasing the size of the layers next to the latent space improves the generalization on test data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- See the \"Weaknesses\" and the \"Questions and perspectives\" sections.\n- In Section 2, some notations are quite confusing. In particular, for clarity, I would appreciate to change $p(\\mathbf{z})$ to $p_{\\mathbf{Z}}(\\mathbf{z})$ in order to clearly distinguish the prior distribution to the one of the data.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9kGPWzVY6D", "reviewer": "Reviewer_tiKc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is extremely well-written, clear to follow.\n- The explanations are great, I really like Figure 1. It's also great to have the focus both on the data and also on the model capacity.\n - I appreciate the details in the appendix.\n\nWeaknesses:\n- A shortcoming of this work is the motivation. The authors are training an unsupervised model to train an unsupervised model. There are some works around learning latent variable directly through diffusion models. So it's not clear why someone should use the method's presented by the author instead of training a single diffusion model. On a similar line, the paper would have been much stronger is the authors were able to run similar evaluations as they did on the VAE directly on their diffusion model (several papers show that diffusion models can be seen as classifier and are able to solve many downstream tasks).  \n- Lack of analysis over the quality of the latent learned by DMaaPx. I would have been curious to know wether changing the data distribution between real and synthetic could change some properties in the latent space that is learned. I would expect that since the distribution learned by the diffusion model should be smoother, it might also be easier to interpolate in the VAE latent space.\n- Another shortcoming is the over-reliance on the ELBO as metric. As noted by the authors, Nalisnick et al., 2018, show that ELBO on test can be higher than the one on the training data. And several works have highlighted the limitations over such metric. So, it would have been fine in the main paper to have maybe one figure with the ELBO but since all figures in the main paper rely on the ELBO, I have some concerned about the reliability of the plots. On a following point, the most interesting results are probably the ones in Table 11 in the appendix. For this figure, we have a bunch of downstream tasks such as classifications ones and generative ones. For the generative ones, the number between using real and generated data are very close (and FID, IS are not very reliable metric so, it doesn't seem that there is any statistically significant differences between those two methods to say that one is better than the other ones). For the classification tasks, it seems that the difference between DMaaPx and traditional training data is 0.01, which I do not think this is significant enough for claiming any \"outperforming\" statement. If we look at the std, there is a clear overlap between normal training and DMaaPx. So one could learn a model than could be better using normal data than DMaaPx.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the effectiveness on synthetic data and over-parametrization on variational autoencoder training. The authors introduce a method coined as DMaaPx which consists in training a generative diffusion model over the training data distribution and then use it to sample a potential infinite training dataset for training a variational autoencoder. The authors compare DMaaPx with traditional handcrafted data augmentation technique and show that DMaaPx gives better results than their methods on the following metric Generalization Metric, Amortization Gap and Robustness GAP that are based on the ELBO. The authors also state that DMaaPx is outperforming baselines on downstream and generation tasks. They also study the relationship between the training data and the model's capacity, the authors give some guidelines on how to increase the model's capacity without hurting generalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Can you clarify that the VAE when using DMaaPx is trained only on synthetic generate data and not on a mix of real/synthetic? From the paper, my understanding is that the authors are only training on synthetic data. In any case, I think it would add strength to the paper to add some ablation over mixing real with synthetic data (with and without additional data augmentation).\n- One point that often come back when using synthetic data from diffusion models is that despite having visually appealing sample, it's still easy for a model to learn to discriminate between real and generate data. Since one of the assumption made in the paper is that the diffusion model is able to provide an accurate approximation of D_train, I would have expected a more in depth analysis on wether this is the case and what are the limitations. In other work, what would happen if we train a simple model is discriminating between fake and real sample? If we can learn a model that could learn that, would that still mean that we are accurately approximating d_train?\n- It it possible to add samples in the appendix from the trained VAE? Would be great to have some reconstruction along table 11.\n- Can you downplay the claims around DMaaPx outperforming other methods on downstream tasks?\n\nOverall, I think this paper can have value for the community, but instead of focusing only on DMaaPx outperforming other tasks, it would be better to recenter the paper on achieving competitive performances using only synthetic data which is already a great contribution.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have addressed the comments of the reviewers in the revision. I do not request additional changes and suggest to accept the paper as is.
Decision: Accept (poster)