Complexity Matters: Rethinking the Latent Space for Generative Modeling
Authors: Tianyang Hu
Fei Chen
Haonan Wang
Jiawei Li
Wenjia Wang
Jiacheng Sun
Zhenguo Li
Venue: NeurIPS 2023 spotlight (2023)
Keywords: generative model
latent space
distance between distributions
generative adversarial network
vqgan
TL;DR: This work characterizes the optimal latent distribution for generative models from the perspective of minimizing model complexity and proposes a two-stage training scheme that achieves practical improvements on GAN, VQGAN and DiT.

Abstract: In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion models the latent space induced by an encoder and generates images through a paired decoder. Although the selection of the latent space is empirically pivotal, determining the optimal choice and the process of identifying it remain unclear. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity. Our investigation starts with the classic generative adversarial networks (GANs). Inspired by the GAN training objective, we propose a novel "distance" between the latent and data distributions, whose minimization coincides with that of the generator complexity. The minimizer of this distance is characterized as the optimal data-dependent latent that most effectively capitalizes on the generator's capacity. Then, we consider parameterizing such a latent distribution by an encoder network and propose a two-stage training strategy called Decoupled Autoencoder (DAE), where the encoder is only updated in the first stage with an auxiliary decoder and then frozen in the second stage while the actual decoder is being trained. DAE can improve the latent distribution and as a result, improve the generative performance. Our theoretical analyses are corroborated by comprehensive experiments on various models such as VQGAN and Diffusion Transformer, where our modifications yield significant improvements in sample quality with decreased model complexity.

Complexity Matters: Rethinking the Latent Space for Generative Modeling
Tianyang Hu
Fei Chen
Haonan Wangi.
Jiawei Li
Wenjia Wang
Jiacheng Sun Zhenguo Li
Huawei Noah's Ark Lab, Hong Kong University of Science and Technology (Guangzhou)
National University of Singapore
Abstract
generative modeling, numerous successful approaches leverage a low- dimensional latent space, e.g., Stable Diffusion [68] models the latent space induced by an encoder and generates images through paired decoder. Although the se- lection of the latent space empirically pivotal, determining the optimal choice and the process of identifying it remain unclear. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity. Our investigation starts with the classic generative adver- sarial networks (GANs). Inspired by the GAN training objective, we propose a novel "distance" between the latent and data distributions, whose minimization coincides with that of the generator complexity. The minimizer of this distance characterized as the optimal data-dependent latent that most effectively capitalizes on the generator's capacity. Then, we consider parameterizing such latent dis- tribution by encoder network and propose a two-stage training strategy called Decoupled Autoencoder (DAE), where the encoder only updated in the first stage with an auxiliary decoder and then frozen in the second stage while the actual decoder is being trained. DAE can improve the latent distribution and as a result, improve the generative performance. Our theoretical analyses are corroborated by comprehensive experiments on various models such as VQGAN [1 and Diffu- sion Transformer [60), where modifications yield significant improvements in sample quality with decreased model complexity.
Introduction
In the past decade, deep generative models have achieved great success across various domains such as images, audio, videos, and graphs [32, 59 33
34) The advances are epitomized by recent
breakthroughs in text-driven image generation 66 66
65 68 Behind the empirical success are fruitful developments of wide variety of deep generative models, among which, many can be
associated with latent space, e.g., generative adversarial networks (GANs) 23 6 40 Variational Autoencoders (VAEs) [44] and latent diffusion models 779 for structured data such as images, the intrinsic dimension much lower than the actual dimension
68 60 It is commonly believed that
4 62 By utilizing a proper cow-dimensional latent space, generative modeling can be more efficient Take text-to-image generation as an example, otate-of-the-art models such as DALL-EE Parti [8) Stable Diffusion [68] all utilized discrete Autoencoders to alleviate the computation
50
78
65
cost by downsampling images, e.g., from 256x256 32x32.
In practice, the choice of the latent plays a critical role. Stable Diffuusion [68 employs the Autoencoder from VQGAN [1I], which is an improvement over VQVAE 80 by incorporating adversarial training.
"Corresponding to: Jiacheng Sun (sunjiachengl @huawci.com)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The discrete Autoencoder was later substituted to continuous ones regularized with
(KL) divergence. ViT- VQGAN [87] further boosted VQGAN by using vision transformers [20] as the backhone and it was adopted by Parti [88] Despite the methodological advances, our theoretical understanding of the latent space is still limited. Important questions such as what constitutes good
latent space, what is the optimal choice, and how it depends on data remain elusive.
Ideally, the latent distribution Pe should preserve the important information about the data distribution Ps as much as possible so that the required effort from the model learning is minimal. Finding such a data-dependent latent can be viewed as a self-suuprvised learning (SSL) task. Can we borrow insights from the vast literature on SSL to understand and improve the latent? Unfortunately. existing methods
[4 25 28 [9] are designed for discriminative tasks and not suited for generative modeling. To
illustrate. consider the classic GANs [24] [6] where the latent distribution is usually pre-defined.
When modeling the CIFAR-10 dataset (45), if we substitute the DCGAN's s latent 63) from standard
Gaussian to the features learned from SimCLR [14] with the same dimension, the Inception Score
(IS) drops from 5.68 to 3 Therefore, new theoretical insights are in dire need to elucidate the ideal Ps and uncover its connection to Pz.
In this work, we aim to provide a new understanding of the latent space in generative modeling from the angles of SSL and minimizing model complexity, where we first formalize the problem for GAN models and then generalize it to Autoencoders.
Drawing inspiration from the training objective of GANs, a novel distance between distributions in different dimensions be defined measure the closeness between the latent and the data (Section 3.1). As is typically emphasized in learning theory 82 an important factor in this formulation, where latent is deemed closer to the data if generator can achieve the same (better) performance with lower (the same) complexity. The latent closest to the data in GAN-induced distance characterized the optimal choice for that GAN, which leads to the simplest map between the latent and the data.
83
35
61 the complexity of the generator
With our formulation of the optimal latent distribution, the immediate question is how estimate it, which gives rise to a new SSL task for generative modeling. Naturally, we consider utilizing an encoder network to parameterize the latent, which uncovers the popular Autoencoder architecture with the paired generator as a decoder (Section 3.2). In the learning process, we further emphasize the importance of relatively weak decoder and the trade-off between the informativeness of the latent and the capability of the decoder (Section 4. To this end, we propose a 2-stage training scheme called Decoupled Autoencoder (DAE), where the encoder is only updated in the first stage with an auxiliary decoder and then frozen in the second stage while the actual decoder is being trained (Section 4.2). Our theoretical analyses are corroborated by comprehensive experiments on both synthetic data and real image data with various models such as DCGAN (63). VAEGAN [47 90), VQGAN 21 and Diffusion Transformer (DiT) 600, where our modifications yield significant improvements in image quality with decreased model complexity (Section
2
Preliminary
For function f Ω -> R. let || (J2 denotes its norm. I.p and cp are used to distinguish function norms and vector norms. For two positive sequences {a, InEN and (bn)nen that a, < Cb,, for all sufficiently large n. We use Pa and pa to denote the probability distribution and density of the random variable x. go f(x)= == (x(())) denotes composition of functions.
Notations.
For vector æ. ||~ llp
we write an S b, if there exists a constant C > 0 such
Generative adversarial networks are a type of implicit generative models that aim to learn transformations g E G from random noises to the data æ. At the population level, its objective function can be expressed in general as
inf (M,,,, Pa, gEg
(2.1)
where Ps is usually pre-determined to be some simple distribution such as (mixture) Gaussian or uniform, and DA is realized by the adversarially-train discriminator and when optimized properly, can give rise to various well-established distances, e.g., Jlensnn-Shanno divergence 24].
CCalculated using reconstructed data. More details can be found in Section 6.2

---

maximum mean discrepancy 488 f --ivergeence [58] Wasserstein distance [5, etc. Exploiting the
flexible architectures of various neural networks as generators 41 42 GAN enjoys state-of-the-ar
with notable advantage in the sampling speed, especially compared with diffusion probabilistic models (DPMs) 72 [2]. However, GAN models often suffer from mode collapse, failing to keep the data diversity intact. Besides the usual suspect of the min-max adversarial training, the poor choice of latent noise distribution also to blame [22).
performance in generating various data types 41 85 74, 89
Sele-supervieed learning. The goal of SSL is to learn a (low-dimensions) feature representation f()) from unlabeled data that is suited for various discriminative downstream tasks, e.g., classifi- cation, detection, etc. There are mainly three categories, i.e., pretext task-based [57], contrastive methods such as SimCLR 14, MoCo [28], BYOL [25], SimSiam [15), and generative-based such as Masked Autoencoder (MAE) 229. The learned feature map fx) defines distribution over the feature space E IRd denoted as Pj. Ideally, Pj should preserve the important information about Pz as much as possible. In this sense, representation learning in general can be seen as preserving certain "distances between distributions in different dimensions 36 8).
Distance between distributions in different dimensions. To quantitatively measure the closeness, various similarity measures between Pj and Pa are defined. Since d, < d, typical distances between distributions will not work due to the different metric spaces and æ reside in. Fortunately, there are two existing tools we can turn to. The first is Gromov-Wasserster distance [56] [7],, which circumvents the metric spaces mismatch by comparing pairwise joint distributions. Typical manifold learning methods such as stochastic neighbor embedding can be thought of as a special case 31 81 36). Second, as in [9], popular distances such as Wasserstein-P distance and f f-divergence can all be naturally extended to such settings, either by embedding P3 to R or projecting PN to Rd. and taking the infimum over all orthogonal linear projections. Specifically, for any distance between distributions D(.,)) in the Wasserstein-p or f-divergence family, define its generalized version to be
inf D* (* Pa) = PsE#+(P..d)
D(Pa, Pz),
where (* = (Pg AT b A Ed.xd E Rd. AAT Ta, } is the family of distributions in IRd embedded from P. by linear orthogonal transformations from Rd. Rd Similarly, we can define D (Pz. P2) and [9] showed that D+ P.) = D D= P.).
Contrastive learning has deep connections with manifold learning [ in terms of preserving pairwise similarity, which can be viewed as a special case of preserving the Gromov- Wasserstein distance [36). In contrast, as we will demonstrate in this work, the optimal latent for generative modeling such as GANs more closely related D+ Pz).
3 Optimal latent distribution for generative models
Recall the typical training objective of GANs 2.1 and denote its empirical version as L,, Pz). where n the sample size. For any given Ps, with limited capacity on the generator, miny Ln P.) reflects the compatibility or closeness between P, and P.. If Ps, is a better latent than Pz,, intuitively min, In [gl P.,) S ming Ln (g) This can serve as an empirical and relative evaluation of the quality of GAN latents. Inspired by this, we can generalize the distance proposed in [9] to construct more meaningful version specifically for implicit generative models such as GANs.
3.1 GAN-induced distance
Let M(2)) denote the set of all Borel probability measures on Q. For any distance between distri- butions -(.--) in the data space M(R.) define the generalized version associated with a generator g E G mapping from Rdd to Rd as:
(P, Pz) inf D(P (())))): Pw).
(3.1)
gEg
We use the word "distance" quite generally, including only well-defined metrics, but also divergences.
3

---

Compared with D in 19, D substitutes the linear orthogonal mappings to general function space. Although D is not a valid metric as it is defined between different probability measure spaces, can serve as viable measurement of the closeness between P= and P..-
In order for D9 to be nontrivial, the function space cannot be too large, it is known that one- dimensional distributions can approximate higher dimensional distributions arbitrarily well by neural network transformations in Wasserstein-p distance [86]. On the other hand, cannot be too small, or it may not be able to adequately extract the information from the latent distribution. In this work, we consider G to be a family of neural networks with bounded complexity, i.e. ge = {g E G C(g) S c). where e > 0 and C-) is some complexity measure to be specified. C(.) can be as specific as the Lipschitz constant, or as general as the size (width, depth, etc.) of the network. Remark 3.1. It worth emphasizing that in (3.1), we do not consider the optimization problem but instead, focus on the existence or the global minimum of inf geg. A toy example is given in Appendix A. to illustrate the global minimizers of g E G.
Remark 3.2 (Complexity scaling law). In the toy case of Appendix A.I the relationship between D and e is embodiment of the popular scaling law phenomenon [9| [9]] with respect to the model complexity. It's worth noting that the requirement on the complexity can be drastically decreased we consider multiple steps of generation, rather than single-step fush-forward transformations. From this perspective, one of DPM's main advantages may just be the increased complexity due to the recursive sampling process.
Directly following the definition in 3.1 we have the following propositions.
Proposition 3.3. Let D be a p- Wasserstein metric. a Sensen-Shannon divergence, a total variation, or an f-divergence. Then, D (Pz, Pa) in 3.1 is zero if and only if there exists g E such that Pg (x) = Pz.
Proposition 3.4. For two distributional distances, generalizing them across different dimensions via (3.1) does not change their relative relationships. e.g., if D is Wasserstein distance, we still have W (Ps.Ps) S W (Pz: (e..P)..f f=).for for p
q
Equipped with D to compare different latent distributions, we can characterize the ideal Pa as the one that minimizes (P,,, P2). i..., the latent that requires the lowest generator capacity for L(g Px) to be below a certain threshold" Such a latent depends on both the data and the GAN training algorithm and can be seen an SSL task, targeting specifically minimizing the required capacity of the corresponding generator, thus easing the training of the generator.
Remark 3.5 (Dimensional collapse). In contrast to the common dimensional collapse phenomenon found in contrastive learning where the learned feature distribution is only supported on low-dimensional subspace, minimizing D (Pe, P2) with respect to Ps encourages it to maintain the intrinsic dimension of the data so that D(P, (z), Ps) is not ill-posed, The dimensional collapsed latent contributes to the poor performance of SimCLR+DCGAN mentioned in Section |
38
Remark 3.6 (Change-ofe sc-slle problem). Although De serves as a well-defined measurement of closeness between Ps and Pa, the latent P= that minimizes D& (p2. Pa) might be ill-posed. If the complexity is not invariant to scaling, e.g., In norm, Lipschitz constant, etc., (P (A=)) Pa) is non-increasing with X. In the next section, we impose extra constraints to circumvent this problem, and the optimal P is characterized therein.
3.2 Learning the latent by an encoder
Now that have characterized the ideal latent distribution given the generator family and the data, the next question is how to find such a good latent. Casting it as an estimation problem, the first decision to make is how to parameterize Pg. We could adopt Gaussian mixtures as in (84). Nonetheless, a more universal and powerful way is to employ an encoder network f E F Rd + Rdd similar to SSL methods. With Ps parameterized by an encoder, the overall structure is an Autoencoder. In the remaining part, we also refer to the generator as the decoder and use the two words interchangeably.
At the population level, we want to match the distributions of x and the reconstructed g f(x) i...
inf D(Pz, P (oo)(()) = inf inf D(Pz, P Pgos(=))) = inf D& P(()))
JEF.ygg
JEF (
SEF
Moore mathematical ground for this argument be found in Appendix A.3

---

Looking at the encoder and the decoder together, we can see that optimizing D(P Pa) amounts to learning an encoder f that minimizes DC, This strategy is employed by VQGAN [21], where the encoder and decoder are optimized together with the discretized latent code. Inspired by this
observation, we can characterize the optimal latent distribution P for given Pz from the perspective of minimizing the distance between distributions in different dimensions by defining Py. as
f" argmin D (Px: P(x))) JEF
(3.2)
Notice that Pj not only depends P. and D, but also F and G. The change-of-scale problem in Remark 3.6 is mostly taken care of by the inherent boundedness of F. More discussion on the choice of G can be found Section 1
Remark 3.7 (Classification). As the solution of a new SSL task, f in 3.2) will preserve well- separated clusters of Pz. though not as discriminative the features learned from existing SSL methods. See Proposition A.3 in the appendix for more details. We also conducted experiments on simulated data evaluating the classification accuracy in Section 6.1
With the encoder, (3.2 is not the only option to characterize the optimal latent distribution. Besides directly parameterizing Pz. F can also serve an auxiliary role in defining a more general P To this end, in parallel to DE
can also define D (P=, Pa) = inf D(P=, P)cz)) As D is to Combining DS and D== we get generalized measure of
D+ D generalizes the D in g distance between the latent and data for Autoencoders as (P,,, P2) = D9 P.) D (P.. P.), which resembles the condition number of matrices, but for latent distributions. The optimal latent distribution can also be characterized by
P argmin D (P.., P.
(3.3)
If d_ = d, then identity mapping, i.e., P Pa. obviously solves 3.2 and 3.3) If like DPMs or VAES where the latent distributions are chosen to be data-agnostic, e.g., standard Gaussian, D (P2, Pa) can be significantly larger for complicated data.
Remark 3.8. D and D can be zero, especially when Pa is supported low-dimensional space
91. As a result, the defined f or P. might not be unique. We provide a toy example in Appendix A.2 to illustrate how P minimizes the complexity.
4 Improving the latent with Decoupled AutoEncoder
In practice, the encoder and decoder family should be as powerful as possible. However, as stated before, G and F should be chosen carefully in order for the induced D and D to be meaningful.
4.1 Necessity of relatively weak decoders
Given total budge of G and J, intuitively. C(G)= tend to result in the best approximation of G o F to the identity mapping- This is reflected in practice where the encoder-decoder pair is often designed with symmetric architecture, with the same number of blocks and parameters, e-g.. up-sampling vs. down-sampling, convolution vs. deconvolution, etc. However, when it comes to the quality of the encoded latent, only targeting the reconstruction error may not be the ideal strategy. As will demonstrate in Section 4.2, better way is to first target good latent distribution, and then do reconstruction using the learned latent. Consider two symmetric cases: (larger encoder, smaller decoder) vs (smaller encoder, larger decoder). The reconstruction ability may not be significantly different, although both are sub-optimal, the former tends to produce better latent distribution. See Section 6. for empirical evidence in the toy case comparing the two cases.
For concrete demonstration, consider the simple linear case where the encoder and the decoder are two matrices, denoted as W. E Rdxd, and W2 E Rd, respectively. Take the matrix 2-norm as measurement of complexity, which equal to the largest singular value. similar setting is also considered in [37]. In the linear case, the quality of the latent is equivalent to the reconstruction error and the optimal encoder should recover the principal components. The next theorem states that the Autoencoder could fail if the encoder is not large enough.
'The budget and C(-) here refer to the general concept of size (width, depth, etc.) of the network.
5

---

Stage I: Update Encoder with Auxiliary Decoder
JEF
Real Sumples
Generated Samples
g
Stage 2: Freeze Encoder, Update Actual Decoder
Figure 1: Overview of the DAE 2-stage training. DAE addresses the trade-off between generation quality and the latent quality where the encoder is only updated in the first stage with an auxiliary weaker decoder and then frozen in the second stage while the actual decoder being trained.
Theorem 4.1. Let the singular values of Wa be \1 ... Ad. If < <//2a the linear Autoencoder cannot recover the principal components, and the reconstruction error is sub-optimal.
The linear case provides insights into neural networks as Autoencoders, indicating that the encoder should be relatively strong to better learn the latent distribution. In the more practical nonlinear cases, things become more complicated. The intuition that (larger encoder, smaller decoder) is preferred to (smaller encoder, larger decoder) for learning more informative latent is supported
by the well-known posterior collapse phenomenon in the VAE literature [T] 27 67 (54,, where the encoded distribution becomes completely uninformative if the decoder is too powerful. However, for implicit generative models such as GANs, the generator is obviously the key component and should be powerful possible. There exists trade-off between generation quality and the quality of the latent. Using decoder too powerful can hinder the learning process of a good latent while decoder too small will hurt the generation performance. In practice, whether decoder family is weak or not can be summarized in an oversimplified way by its size, i.e., a smaller decoder is weaker. Remark 4.2. We could consider pairing a sufficiently large G with an even larger F. Unfortunately, such combination may not be good solution. Besides the added computational cost, encoder- decoder pair too powerful can drive D + D too close to zero, so that it may get overwhelmed by random noise. The tignal-t-noisee ratio may be too low for effective optimization.
4.2 Decoupled AutoEncoder
To address the trade-off between generation quality and the quality of the latent, we can introduce an auxiliary decoder GA, potentially much smakler/weaker, substitute for training the encoder. Once the encoder is trained, we can then freeze it and pair it with G for the decoder training. The key 2
idea to decouple the learning process of the encoder and the decoder 2-stage training scheme we call Decoupled AutoEncoder (DAE). Figure illustrates the method overview where the two stages DAE can be summarized as:
DAE Stage 1: Train j E F with a small decoder G learn good latent.
DAE Stage Freeze the trained encoder f. Then train the regular decoder y E 9 to ensure good generation performance.
By the 2-stage training, one with gd and one with G, without much hyperparameter tuning, DAE can learn a better latent distribution and as a result, achieve better reconstruction and generative modeling. Remark 4.3 (Realizing g-A). The essence of the first stage to employ an auxiliary decoder that is relatively weak compared with the encoder. Straightforwardly. family. For an easier way of implementation, we can also use various forms of regularization to realize g-A, For instance, with a balanced encoder-decoder pair, we can apply strong Dropout 773 to the decoder in the first stage to make it weaker. Both methods are experimented in Section 6
introduce extra decoder
DAE can be applied in a plug-and-play style to any existing Autoencoder training pipelines with the decoupled 2-stage modification. We demonstrate in Section 6 that compared to various basclines,

---

e.g. VAE, VQGAN, etc., our DAE-modified versions are better in terms of both reconstruction error and quality of the latent. For instance, we refer to Figure 3 in the appendix where we experimented with VQGAN (21). In terms of the reconstruction error, Stage is worse than the baseline, which is to be expected since a weaker decoder used. However, paired with the Stage encoder, the decoder (the same size as the baseline) can achieve better reconstruction performance.
4.3 Sampling from the latent distribution
In most Autoencoder- Autoencoder (76), etc., the latent distributions all modeled by Gaussian distributions. DPMs 72 32 can also be seen as a type of Autoencoder-based method with the forward noising process as the encoder and the denoising process as the decoder, with the latent distribution being Gaussian. As discussed in Remark 3.2 the decoding process of DPM enjoys drastically increased capacity due to the multi-step generative process.
generative models, e.g., VAE, adversarial Autoencoder [55]. Wasserstein
In our work, with Pz parameterized by the encoder, how to generate new samples from it requires extra modeling of the latent. Otherwise, we can only do reconstruction. There are two mainstream methods to sample from the latent distribution. The first is inspired by natural language processing. In VQGAN 21 and Parti [88]. the discretized latent codes autoregressively modeled by transformers as a sequence, similar to GPT [64] while masked image modeling methods [2] resembles BERT 19]. The second is by diffusion process. Latent space diffusion models [68, 60)
can be seen as
using DPMs to model the latent space of a powerful Autoencoder. We mainly adopt the VQGAN formulation in our numerical experiments for training the Autoencoders. To validate the effectiveness of our DAE, we consider modeling the latent by both transformers and diffusion models.
5 Related works
proposed to model the latent distribution as Gaussian mixtures as a remedy for the mode collapse problem of GANs. Instance-condittione GAN 1T) proposed to partition the data manifold into mixture of overlapping neighborhoods described by a data point and its nearest neighbors to improve the quality of unconditional generation. There is also a line of work that considers post-sampling latent space mining of GANs by exploiting the trained discriminator [75] [3) 49 further proposed adding an implicit latent transform before the mapping function improve the latent from its initial distribution in bi-level optimization fashion. Though these works targeted improving the latent distribution of GANs, they did not characterize the optimal choice nor employ an encoder to model the latent. In comparison, we approach the problem from the perspective of preserving the distance between distributions different dimensions and propose a 2-stage training scheme involving an encoder.
BourGAN 84]
There are GANs that incorporate an Autoencoder structure. VAEGAN 47 90 combines a VAE by collapsing the decoder and the generator into one. (2) utilized a standard autoencoder to embed data into the latent space to address the mismatch of the continuous neural network and the discontinuous optimal transport (OT) map. (3) later proposed to utilize semi-discrete OT maps to sample from the latent and train GAN models. VQGAN 87 uses vector- quaatt-qa latent codes and transformers to model the tokenized latent distribution. VIT-VQGAN [8] later made improvements over VQGAN. Besides using vision transformers 20 as the backbone, also introduced an l2 regularization term to the latent code. However, the aforementioned works did not characterize the optimal latent either and did not point out the necessity of the decoupled 2-stage training. Our DAE approach can also be applied to them for further improvement.
In the literature of VAE, (77) introduced new priors consisting of mixture distribution with components given by variational posteriors conditioned on learnable pseudo-inputs. [17] proposed a two-stage learning scheme amend the problems associated with the Gaussian encoder/decoder assumptions. [1] stated that obtaining good ELBO is not enough for good representation learning and proposed to incorporate mutual information and rate-distortion curve to achieve better balance between the informativeness of the latent and the reconstruction quality. In comparison, our characterization of the optimal latent is from the perspective of matching distributions and minimizing complexities and is not restricted to Gaussian assumptions. It is also worth emphasizing that while the mutual information, closely related to VABs, can also serve as a "distance" between distributions different dimensions, it will not work for our benefit since we are considering deterministic encoders and
7

---

decoders. In the deterministic case, the conditional entropy would be zero and the mutual information would always be the entropy of the latent. Hence uniformly distributed latent is always ideal, which does not offer new insights.
Masked Autoencoder (MAE) 29 is generative-hased SSL method with the training objective of reconstructing masked images. In MAE, a high mask ratio and a relatively weak decoder have been verified as important factors for good discriminative performance, which is also corroborated by our perspective of preserving distributions in different dimensions. MAEs are designed for discriminative tasks and their generation capability is yet to be explored.
6 Experiments
In this section, we conduct empirical evaluations of our proposed DAE training scheme on variety of datasets and generative models, from toy Gaussian mixture data to DCGAN on CIFAR-10. to VQGAN and DiT on larger datasets. The detailed experiment settings can be found in Appendix B
6.1 Toy examples
To showcase the effectiveness of the proposed 2-stage training, we consider toy Gaussian mixture case with 8 clusters, whose centers are evenly placed on the unit circle, and the variances are 0.25. All samples in R² are then projected to R200 via linear orthogonal mapping. We employ the vanilla VAE 44 for demonstration with d2 = 2. In our implementation, both the encoder and the decoder are 3-layer fully connected networks with different hidden dimensions to control the size. Giving each cluster a label, we evaluate (1) the nearest neighbor classification accuracy from the 2D latent; (2) the nearest neighbor classification accuracy from the 10D reconstruction; and compare different configurations of the encoder and decoder pair. The average classification accuracy is reported in Table I which corroborates our analysis in Section [ i.e., a relatively weak decoder can learn a better latent and the following 2-stage training can further improve reconstruction.
Table 1: The mean (variance) of the classification accuracy w.r.t. difference hidden dimensions of the (encoder, decoder) over 10 replications.
Table 2:
The performance of DCGAN with dif-
ferent latent distributions.
Method
IS (t)
FID (L)
Hidden Dim
(64, 128)
(128, 64)
DCGAN (reproduced)
5.68
51.76
Latent Ace (%)
80.6 (7.0)
87.8 (5.8)
DCGAN-SimCLR
3.93
168.23
Hidden Dim
(128, 128)
DAE (128, 128)
VAEGAN
5.82
48.11
Rec Acc (%)
92.2 (6.1)
98.0 (1.3)
DAE-AAEGAN
6.16
46.12
6.2 GAN CIFAR-10
As more comprehensive proof-of-concept, we experiment with DCGAN 663 on the CIFAR- 10 45 dataset. Specifically, we consider vanilla baseline where the generator has 5 convolutional layers and the discriminator has 4 convolutional layers, followed by linear classification layer. The latent for DCGAN standard Gaussian. To corroborate our analysis in Section 4 we adapted the DCGAN with varying latent space configurations: (1) We utilize a latent space defined by CIFAR-10 pre-trained SimCLR model 16), denoting this adaptation as DCGAN-SinCLR. (2) We consider VAEGAN with a learnable latent space. The structures of the decoder and discriminator mirrored those of our baseline DCGAN while the encoder is implemented with a five-layer CNN. (3) We modified VAEGAN to first stage and then set back to its original structure in the second stage.
where the width of convolutional layers in the decoder is halved in the
We evaluated the aforementioned four methods using the metrics of Inception Score (IS) 70 and Frechlet Inception Distance (30). With the exception of DCGAN, which generates images from random noises, the other methods first encode an image into latent space, and then generate an image based on that encoding. To this end, we omit the latent space modeling and calculate their metrics from reconstructed images. The results are listed in Table 2 with two take-home messages: (1) Traditional SSI methods may not help with the latent for generative modeling; (2) Decoupling the encoder and decoder training can improve the overall performance of generative modeling.
8

---

6.3 VQGAN
VQGAN 21 consists two major components. The first is a discrete Autoencoder, where the latent is quantized by looking up nearest neighbors from learnable codebook. After the encoder and decoder are trained (objective contains adversarial loss), the second component, transformer model, learns to generate discrete codes autoregressively in the latent space, which are then mapped to images by the learned decoder.
To demonstrate the effectiveness of our DAE, we introduce the decoupled training strategy to the first component of the standard VQGAN. In the first stage, we implement the relatively weak auxiliary decoder by applying Dropout 73 to the decoder with ratio p. and train encoder and the dropped decoder jointly] Then in the second stage, we freeze the encoder and train the full decoder without Dropout. Training of the transformer component is the same as VQGAN. The modified model is denoted as DAE-VQGAN.
Setup. We evaluate DAE-VQGAN by comparing with the baseline VQGAN on the HacesHQ dataset, which is a combination of two face datasets CelebAHQ [51] and FFHQ 43 with 85k training images and 15k validation images in total. In all experiments, the input image size is 256x256. We use the official VQGAN implementation and model architectures for FacesH The Autoencoder is dubbed VQ-f16. The training process of stages, where in the first stage we apply 2D Dropout (channel-wisg to the decoder with ratio p = 0.5.
is divided equally into two
Table 3: Reconstruction FID over FacesHQ training and validation sets, and transformer generation FID on Cele- bAHQ and FFHQ training sets. f: Evaluated on the pub- licly available pre-trained model on FacesHQ. *: Our repro- duction is based the official VQGAN implementation.
Table 4: Class-conditional image gen- eration on ImageNet 256 x 256 using the DiT-L/4 model with different Au- toencoders. *: Our reproduction is based the official DiT implementa- tion, which is consistent with the re- sults reported in Fig.5 of [60].
Reconstruction
Generation
Method
Train
Val
CelebaHQ
FFHQ
VQGAN
4
6.27
10.2
9.6
Method
FID
sFID
IS
VQGAN*
4.23
5.83
9.97
10.44
DIT
35.16
7.33
39.25
DAE- VQGAN
2.01
3,82
8.58
8.36
DAE-DIT
32.29
6.90
41.71
Improved image reconstruction and generation. To quantitatively evaluate the encoder-decoder component, we train the model on FacesHQ and compute the reconstruction FID over the full training and validation splits. We further train a transformer separately for CelebAHQ and FFHQ with the same architecture in (21). and compute the generation FID with 50k generated samples against the training split of the respective dataset. As shown in Table 3 DAE- VQGAN achieves significantly improved FID compared to VQGAN, indicating that stronger encoder-decoder component can be unlocked for VQGAN by our DAE approach. We show qualitative generation results in Figure 2 reconstruct and generate images with high fidelity.
We see that models trained by DAE-VQGL Figure(3 3i the Appendix shows the reconstruction losses during training. It is worth noting that while our DAE-modified VQGAN has higher reconstruction losses in the first stage, it catches up fast in the second stage and converges to a lower level than the baseline. This is to be expected as the first stage of DAE focuses on learning a better latent, which can ease the training of the decoder in the second stage and result in better performance overall.
Decreased Complexity. Minimizing the model complexity is a key motivation for our DAE. To verify our claims, we evaluate the Lipschitz complexity (CLip defined in (B.I) of the encoder f and decoder in VQGAN. Since o of is trained to be approximately the identity map, the trends of (f and (gap(g) are expected to be the opposite, with the ideal number being close to 1 for both we want the overall complexity (f) + (g) ) to be minimized. We observed in Figure 4 that the encoder complexity of VQGAN and DAE- VQGAN are 1.40 and 0.88, respectively, with an increasing trend in terms of training steps; the decoder complexity of VQGAN and DAE-VQGAN
'We also implemented the auxiliary weak decoder by leveraging an extra decoder family with the number of convolution channels halved. The overall performance is similar and we put the results in Appendix B.I "aapss//ithubb.oom/oom Visttaiin transforrrrsss

---

ndu
(h)
Figure 2: (a) Top: original image; Middle: reconstruction of VQGAN; Bottom: reconstruction of DQEAN, (b) Generated samples of DAE-VQGAN
are 0.73 and 1.16, respectively, with decreasing trend in terms of training steps. Compared with the baseline, our DAE-modified version produces better latent distribution with significantly lower complexity while achieving a lower reconstruction error. See Appendix B.2 for more details.
6.4 Diffusion Transformers (DiT)
The encoder in VQGAN is discrete with codebook. In Appendix B.3 and investigate how different training strategies affect the behavior the learned latent. Our DAE can also apply continuous Autoencoders that usually regularized by KL divergence. Such KL-regularized AEs are adopted by latent diffusion models such as DiT [60]. Following similar settings as VQGAN, we also modified DiT DAE-DiT by employing the decoupled training scheme and experimented with the larger ImageNet dataset 18
we look into the codebook
The details can be found in Appendix B.4
For evaluation, train DiT on the ImageNet dataset with 256x resolution following the official implementatior Instead of the largest DiT-XL/2 model (675M Params), we select the DiT-L/4 model (458M Params) for higher training efficiency. The corresponding Autoencoder is dubbed KL-f8. Adam W (53] optimizer is employed with a constant learning rate of 10-1 and a weight decay 3 x 10-2 The batch size is 1024, and the number of epochs is 120. The trained model with Exponentially Moving Average (EMA) then used to generate 50k images of the 1000 categories equally via a 250-step DDPM sampling [32 and classifier-free guidance scale of 4. Table compares the image generation quality where we can see that our DAE-DiT achieves significant improvement, similar to what we observed for DAE-VQGAN.
Discussion
This work investigates the ideal latent distribution for generative models. We introduce a novel dis- tance between distributions to characterize the ideal P. that minimizes the required model complexity. Practically, we propose a two-stage training scheme called DAE that achieves practical improvements on various models such GAN, VQGAN, and DIT. Since many of the most powerful generative models are associated with latent space, the impact such investigations is potentially very high.
Nevertheless, there are many limitations of our work that call for further research along this line. First, our formulation of the optimal latent distribution is based on _ and D", which serve mainly illustrative purposes. The proposed distances cannot be effectively calculated. Second, the effectiveness of DAE is not proven mathematically and the resulting latent is not guaranteed to be closer to P Our work could be further strengthened if the aforementioned limitations can be addressed. It would also an interesting direction to explore how our method affects latent space disentanglement 152 26
10

---

References
[1] Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. In International conference machine learning, pages 159-168. PMLR, 2018.
[2] Dongsheng An, Yang Guo, Na Lei, Zhongxuan Luo, Shing-Tung Yau, and Xianfeng Gu. Ac-ot: new generative model based on extended semi-discrete optimal transport. ICLR 2020, 2019.
[3] Dongsheng An, Yang Guo, Min Zhang, Xin Qi, Na Lei, and Xianfang Gu. Ae-ot-gan: Training gans from data specific latent distribution, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part XXVI 16, pages 548-564. Springer, 2020.
[4] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances Neural Information Processing Systems, 32, 2019.
[5] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
[6] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. International conference on machine learning, pages 214-223. PMLR, 2017.
[7] Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
[8] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205. /1508, 2022.
[9] Yuhang Cai and Lek-Heng Lim. Distances between probability distributions of different dimensions. arXiv preprint arXiv:2011.00629, 2020.
[10] Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero So- riano. Instance-conditioned gan. Advances in Neural Information Processing Systems, 34: 27517-27529, 2021.
[11] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-1133 11325, 2022.
[12] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 2023.
[13] Tong Che, Ruixiang Zhang, Jascha Sohl- Dick-Diccst,in, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan secretly an energy-basee model and you should use discriminator driven latent sampling. Advances in Neural Information Processing Systems, 33:12275-12287, 2020.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[15] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[16] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint wwXiv:2003.04299) 2020.
[17] Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint 17X1v:190.005789, 2019.

---

[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 245. leee, 2009.
[19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wrxiv:1800048800 2018.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.11990 2020.
[21] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- reghtresouution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22873-12883 2021.
[22] Ruili Feng, Zhouchen Lin, Jiapeng Zhu, Deli Zhao, Jingren Zhou, and Zheng-Jun Zha. Uncer- tainty principles of encoding gans. In International Conference on Machine Learning, pages 3240-3251. PMLR, 2021.
[23] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[24] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[25] lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint 2xXi::2006.07733, 2020.
[26] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances neural information processing systems, 33:9841-9850, 2020.
[27] Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv: arXiv:1901.055: 1901.05544, 2019.
[28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv: 10t:://11.0677 2021.
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[31] Geoffrey Hinton and Sam T Roweis. Stochastic neighbor embedding. In NIPS, volume 15, pages 833-840. Citeseer, 2002.
[32] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:684006851, 2020.
[33] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 2022.
[34] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 8867-8887 PMLR, 2022.
12

---

[35] Tianyang Hu, Wenjia Wang, Cong Lin, and Guang Cheng. Regularization matters: A non- parametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics, pages 829-837. PMLR, 2021.
[36] Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learning is secretly doing stochastic neighbor embedding. In International Conference on Learning Representations, 2023.
[37] Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: theoretical analysis. arXiv preprint arX0v:2/10.02473, 2021.
[38] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021.
[39] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alce Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[40] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4401-4410, 2019.
[41] Tero Karras, Samuli Laine, Mïika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8110-81 8119, 2020.
[42] Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems,
2021.
[43] Vahid Kazemi and Josephine Sullivan. One millisecond face alignment with ensemble of regression trees. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1867-1874, 2014.
[44] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 13116:131 2013.
[45] Alex Krizhevsky, Learning multiple layers of features from tiny images. University of Toronto, 2009.
[46] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956-195 2020.
[47] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby. Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. In International conference on machine learning. pages 1558-1566 PMLR, 2016.
[48] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
[49] Yang Li, Yichuan Mo, Liangliang Shi, and Junchi Yan. Improving generative adversarial networks via adversarial learning latent space. Advances in Neural Information Processing Systems, 35:8868-888 8881, 2022.
[50] Tengyuan Liang. How well generative adversarial networks learn distributions. The Journal of Machine Learning Research,
2021.
[51] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
13

---

[52] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14- 4124, PMIR, 2019.
[53] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001, 2017.
[54] James Lucas, George Tucker, Roger В Grosse, and Mohammad Norouzi. Don't blame the elbo! a linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32, 2019.
[55] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders. arXiv preprint arXiv: 2015.
[56] Facundo Mémoli. Gromov- distances and the metric approach to object matching. Foundations of computational mathematiccs, 11(4):417-47, 2011.
[57] Ishan Misra and Laurens van der Maaten. Self-supervised learning of resentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
rep-
[58] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In NIPS, 2016.
[59] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609,0490, 2016.
[60] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[61] Pawet Piwek, Adam Klukowski, and Tianyang Hu. Exact count of boundary pieces of relu classifiers: Towards the proper complexity measure for classification. In Conference on Uncertainty in Artificial Intelligence, pages 1673-1683, PMLR, 2023.
[62] Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2 2021.
[63] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.066 2015.
[64] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[65] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero- text-to-image generation. In International Conference on Machine Learning, pages
PMLR, 2021.
[66] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
[67] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. arXiv preprint /tt/://90004444 2019.
[68] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695 2022.
[69] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic rext-to-image diffusion models with deep language understanding. arXiv preprint arXiv: 11487, 2022.
14

---

[70] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[71] Antoine Salmona, Julie Delon, and Agnès Desolneux. Gromov- wassersttsnn distances between gaussian distributions. arXiv preprint arXiv:2104.0 079700 2021.
[72] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[73] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfittigg The journal of machine learning research, 15(1):1929-199 2014.
[74] Sandeep Subramanian, Sai Rajeswar, Francis Dutil, Chris Pal, and Aaron C. Courville. Adver- sarial generation of natural language. In Rep4NLP@ACK, 2017.
[75] Akinori Tanaka. Discriminator optimal transport. Advances in Neural Information Processing Systems, 32, 2019.
[76] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. arXiv preprint #rXiv:/711.00556 2017.
[77] Jakub Tomczak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pages 1214-1223. PMLR, 2018.
[78] Ananya Uppal, Shashank Singh, and Barnabás Póczos. Nonparametric density estimation convergence rates for gans under besov ipm losses. Advances in neural information processing systems, 32, 2019.
[79] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 1287-11302, 2021.
[80] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[81] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008
[82] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[83] Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, 32, 2019.
[84] Chang Xiao, Peilin Zhong, and Changxi Zheng. Bourgan: Generative networks with metric embeddings. Advances in Neural Information Processing Systems, 31, 2018.
[85] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multirresolution spectrogram. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.
[86] Yunfei Yang, Zhen Li, and Yang Wang. On the capacity of deep generative networks for approximating distributions. Neural Networks, 145:144-154, 2022.
[87] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:2110.0460 2021.
[88] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10773 10789, 2022.
15

---

[89] Lantao Yu, Weinan Zhang, Jun Wang. and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. AAAI Conference on Artificial Intelligence. 2017.
[90] Xianwen Yu, Xiaoning Zhang, Yang Cao, and Min Xia. Vaegan: collaborative filtering framework based on adversarial variational autoencoders. In IJCAI, pages 4206 2019.
[91] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12104-12111 2022.
[92] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
16

---

A Technical details
A.1 Toy example
To illustrate the interplay between D and G, we devise a one-dimensional toy example where NO0,o²),
(0, 1), R R is monotonically increasing and C(g) is the Lipschitz constant upper bounded by c. In this case, the optimal transformation is g" = P whose derivative around 0 or 1 is exponentially dependent p//o. Due to the Lipschitz constraint, by the change of distribution formula, we have
(a(=)(T) i(g 1(2))(0.1) V/g- -
From the perspective of regression, the optimal lg fz-projection of g" to g. is
((2) 6 (z) if
c,
[o(== 2/) else.
Let 2. = /202 log(c//(// and it is clear that (z) = c, or equivalently, P=((e) = 1/c. As a result, the push-forward distribution is truncation of Ps supported on |x| S 2, (=))))), i... P3 = Pa if Pa 2 1/c and l/c otherwise, which is equivalent to minimizing the total variation (TV) between Pg and Pa- In this oversimplified example, it is easy to see the scaling between e and DE: ie., TV P. Pa) 2 Pa (-)),, which is polynomial with c in the Gaussian case.
A.2 Toy example
To illustrate the characterization of P and how it minimizes the complexity, consider toy example where dg 1, (0, Ma) and both and F are linear functions with bounded Lipschitz constant
e > 1. Denote the singular values of Eg as \ 2 2 Aa. In this case, we have C(P²) = 0 and P o²0, with a E (VAn, VAi).
Lemma A.1 (Example 6.1 in (9). If D is Wasserstein-2 metric.
VTn - ifo < V
D D=(N(0,)),
if Sos SosVi,
Vi ifa > Vx
where d 2- ...
Xa are the singular values E.
As a direct corollary of the above lemma, we have C(P:) = and P = -N(0,0) with o e (VAng VA1).
A.3
Discussion P and C(g)
The ideal P. is characterized as the one that minimizes (P,,,, and the capacity of the generator be made mathematically sound with extra assumption. Assumption A.2. For any Ps and P.. (G, P&) is continuous and monotonically decreasing with c.
P. Pa). The link between its definition
Assumption A.2 is not provable for general D(:,)) and C(.). Nevertheless, give proof that the ideal PK that minimizes (P,, Pa) will give rise to the minimal complexity generator if it holds.
Proof. First, recall that Ge {g G C(g) I c} where C(g) is defined some complexity measurement. It is easy to see that for any 02 2 e > 0, Ge, C Gez- Therefore, DG (Pa, Pa) S
D Pz).
Consider a generator family G with bounded complexity and we target to have the generated distribution close to the data at level measured by i.e.e D(Pz, Pa(s)) S for some P. and g E G.
For certain (non-degen)rate) P., there exists 0 such that DCa this latent Pz, we need at least complexity e to achieve the e goal. If P. not ideal, by definition we know that there exists p* that DG=(P= P2) e* By Assumption A.2 we know there exists e' < e such that D22 (Pr. Pa) = e. This means that the goal can be achieved using lower complexity generator if the latent is ideal.
that is. by using
D
17

---

A.4 Proof of claims in Section 3
Proposition 3.3 and 3.4 directly follow the definition of D
D is to D as D+ is to D Due to the enlarged function space, the equality D+ = D no longer holds. Directly following Lemma 2.1 in [9, have the following inequality: D (2,, Pa) S D' (Pz, Pa) where can be p- Wasserstein metric or an f-divergence.
Proposition A.3 (Classification guarantee for clustered data), Suppose Pa has m disjoint supports. Then the corresponding P. must preserve all the clusters.
Proof. Let DKL. be the KL divergence (for example). Suppose Pa. has disjoint supports. Then
m Σ
DKn Pa
(x) log
Pz dr.
P2 (x)
Xx
To minimize DK Pa), clearly (a(x) 0 implies (2) = 0. As Pa (1) increases, DKL (Pz. Pa)
(a(x) # 0. is possible to build frr such that pa (a) 2 (zr) for all x, and 1 = - k= Jx, Pa (x)dd > ExnJx, pa pa(x)da which implies there exists a positive measure such that (a p=))). This implies pa(x) cannot minimize D (P, Pa). Thus, we must have if (x) = O, then ps (x) O. This further implies that must have least disjoint supports. If has less than m disjoint supports, then & must have less than m disjoint supports, because the linear map is continuous. Then there must exists ps (z) = 0 but Pa (x) of
decreases, Since P3 can be selected according our will, if pa (z)
In 19]. it has been proved that inf DKL Pb) inf Dxz (Ps, Ps). Following the same logic, the number of supports of & must be smaller than the number of supports of which implies the number of supports 2 must be smaller than m. Combining these results with the previous one, we must have the number of supports of 2 is
A.5 Proof of claims in Section 4
Before we prove Theorem 4.1 we introduce the following lemma on linear auto-encoder.
Lemma A.4. Consider two optimization problems:
min WeRma
|X
WWW X
(A.1)
which corresponds to Principle Component Analysis (PCA), and
min Wi,We W.
|| - W2W,
(A.2)
which is linear auto-encoder. Let W be the solution to PCA and Wi, W2 be the solution to linear auto-encoder, and
1 = - W (W 12 = X w
(A.3)
Then, L1 = Lzz
Proof. Clearly, Lg S L,p, because we can set WT W2 = W which leads to
L2 = | - W < X W (W*)* XII: " = L1.
Therefore, suffices to show L1 S L2.
For fixed W2, By 77, the optimal solution with respect to W is
W (W, Wg) -WW
Therefore,
L2 = |
(W3) x
18

---

Let the singular value decomposition of W be
W = UEV,
where E Raxx and V E IR mmm are orthogonal matrices, and E Rexm a diagonal matrix with singular values X1 ... Am. Therefore,
W ( (wg) =UXIVVY -UN(E"))~ン''">
Note that ΣT is diagonal matrix. Therefore, we can set
W = -=-U(EEE)-1) 1/2
Clearly,
- (" 1/2 Ims
which implies
L2 =| WZ)
= =| X "X"""")~-"""""
=|| =|X X UE X
= =|XX X U
X (U,))
=I,
where E, diag(1, ... .......0.10...... 0) is rank diagonal matrix. This finishes the proof.
With the above lemma, Theorem 4.1 is a simple corollary.
Proof of Theorem 4.7 Recall the proof of Lemma A.4 where the linear encoder is W₁ and the linear decoder is W2. Then, in order to realize the PCA solution, we have
Wi ||2 =I (W, W2) ' [|2 || Σ) >>" Am1
W2112 -| -
(A.4)
Thus, as long as Am < 1, the encoder is larger in terms of 2-norm.
B More on Experiments
B.1 Experiment details for DAE-VQGAN
Let's recap the VQGAN basics with more details. VQGAN [21] consists two major components. In the first component, input image is encoded by encoder f into latent representation. f(x). quantization step is applied to obtain 2, = r(2), which contains nearest neighbors of entries of from a learnable codebook. Then decoder g reconstructs & g(2g) from the codes. After the encoder and decoder are trained (with adversarial loss), the second component, a transformer model, learns to generate discrete codes in the latent space, which are then mapped images by the learned decoder
We evaluate our DAE modifications VQGAN on the FacesHQ dataset, which is combination of two face datasets CelebAHQ and FFHQ, with 85k training images and 15k validation images total (Table 55. In all experiments, the input image size is 256x256.
We use the official VQGAN implementation and model architectures for FacesHQ, where both encoder f and decoder have 128 convolution channels. The spatial dimension of the latent representation is 16 16. See Table 7 and Table in (2 for details. All experiments are run on eight V100 GPUs. For training the encoder and decoder, the learning rate is 4.5 x 10 the batch
Sung::/ggiibbboommmmmmm////aaaap oaap:// transformers
19

---

Dataset
Training
Validation
Total
C
25k
5k
30k
FFHQ
60k
10k
70k
HacesHQ
85k
15k
100k
Table 5: Number of images in the face datasets.
size is 8 on each GPU (total batch size 64), and the number of training epochs is 80. For training the transformer the learning rate is 2 x 10-6 and the batch size is 12 each GPU. The FIDs in Table 3 are obtained by selecting the best results from three independent runs, for both VQGAN and DAE- VOGAN.
To implement our 2-stage DAE-VQGAN, we consider two ways to realize the auxiliary decoder family G during the first stage of training the encoder. Similar to the settings in Section 6. and 6.2 we first utilize an extra auxiliary decoder Jaur with half the number of channels as the actual decoder. Then, experimented with decoder Dropout 773, mentioned in Remark 4.3
f+g
0.9
VOGAN
0.8-
[+ Oau
DAE VQGAN
reconsructin conncrcctinnnn loss
0.7
[+ 9nuld
0.8
reconstruction
0.7
0.6-
0.5
0.3
0.3
0
50k
100k
150k
200k
250k
60k
step
stop
(a) Auxiliary Jauz
(b) Dropout
Figure 3: Training reconstruction loss trends for two implementations of DAE- VQGAN. (a) Halving the channels: while the reconstruction loss of our first stage (J + fauar) converges to higher value than the baseline (f + g). our second stage (f + gaua/ / converges to a lower level with much faster convergence speed; (b) Dropout: the switch between the 2 stages of DAE occurs at halfway (40 epoch, around step 53k), where we can see that our method trails behind before the switch and surpasses the baseline after the switch and converges to lower level of the reconstruction error.
Halving the channels. We use the same f and architectures as the VQGAN baseline, and Jaus is the same as g except for the number of channels set 64. To distinguish, the models from the baseline VQGAN, and use f + Youz and f + 9auxr /g to denote the first stage and second stage of DAE-VQGAN, respectively. For the DAE training, and the auxiliary decoder Douz E G in the first stage (first 40 epochs). Then the second stage (last 40 epochs), we replace Joaz with g. and train g from scratch with (and the trained codebook) fixed. For more thorough investigation, we extended the run for both VQGAN and DAE-VQGAN 200 epochs and the reconstruction eттor trends are shown in Figure a(), where we can see that our DAE-VQGAN achieves significantly better reconstruction. For the actual evaluation on the FacesHQ dataset, we stick the 80 epoch training scheme for both the baseline and our DAE- VQGAN, with the switch point between the two stages at epoch 40.
use f + g represent
jointly train encoder
Dropout. In the first stage, we implement the relatively weak auxiliary decoder by applying 2D Dropout (channel-wise) to the decoder. In particular, we replace the Dropout layer in each ResNet Block in the decoder of the original VQGAN with 2D Dropout and set ratio = 0.5, and train the encoder and the dropped decoder jointly. Then in the second stage, we freeze the encoder and train the full decoder without Dropout. There are in total 80 training epochs for both the baseline and DAE- VQGAN (the first epochs belong to the first DAE stage and the second DAE stage takes up the next 40 epochs). Figure 3)) shows the trends of the reconstruction loss during training, where
20

---

we can see that our DAE- DAE-VQGAN achieves significandly better reconstruction as well, despite the fact that the encoder is only updated in the first stage.
1.4
f+g
f+ Boux
3.5
complexity
Upschitz complexity
I+ 9auld
1.0
3.0
0.8
2.0
0.6
f+g
1.5
++ Saur
1.0
9lw/g
50k
100k
150k step
200k
250k
50k
100k
150k
200k
250k
scep
(a) Encoder
(b) Decoder
Figure 4: Lipschitz complexity of encoder and decoder during training. Our DAE second stage + 9ouz/9 utilizes the frozen encoder f from the first stage f + 9ouz at epoch 40 (around step 53k, where the blue line and the orange line intersect). Since o f is trained to be the identity map. the trends for (g) and (f) are approximately reciprocal and the ideal values for them are Compared with the baseline + g (blue line), our DAE stage one f + Jaus (orange line) has significantly lower Cfip(() that is closer to 1. Since the second stage f + 9auz 9 does not update the encoder, the green line stays flat.
B.2
Complexity of encoder and decoder
One benefit of our DAE is the ability to better exploit the capacity/complexity of the encoder and decoder. To this end, we investigate the Lipschitz complexity CLap of encoder and decoder defined as:
[|[(f())) (f ||21 22112 (1(((((<1))) - (((00)-((((((())))) [T((((=)) -f
Clip(f) = E E P2
(B.1)
Crap(g) = Em
Without loss of generality, choose the DAE-VQGAN implemented by introducing auxiliary Baua for demonstration. The conclusions for Dropout are similar. Figure 4 shows the Lipschitz complexity the encoder and decoder during 200 epochs of training, where f + represents the baseline VQGAN, f 9aaz represents the first stage of our method, and f + 9aws/ / represents the second stage of our method where encoder f is initialized with weights from the 40-th epoch (around training step 53k) of the first stage training. Since g f is trained to be approximately the identity map, the trends of the encoder and decoder are the opposite. As stated in Section 33 the Lipschitz complexity should be ideally close to for both the encoder and decoder. Compared with the baseline, our DAE- VQGAN produces better latent distribution with significantly lower complexity overall (closer to 1) while achieving smaller reconstruction errors. We also compute variant of CLip where f2 norms of difference between æi and 22, and that between their reconstructions, are replaced with perceptual similarity (LPIPS) [92]. This LPIPS complexity variant shows similar trends as Cr.pp (Figure 5)
B.3 VQ codebook
We look into the codebook and investigate how different training strategies affect the behavior of the learned latent. For both VQGAN and DAE- VQGAN, the codebook size 1024. For the codebook itself, we compute the pairwise cosine similarity among the learned codes and show the histogram of similarity scores in Figure For VQGAN, the cosine similarities are much more concentrated near and -1, indicating that a large portion of learned codes may be highly similar. In comparison, codes learned by DAE-VQGAN are more scattered in the latent space and therefore can be more efficient. The expressiveness of our codebook is more powerful in approximating the ideal latent
21

---

3,0024
800
650
220
450
8,0055
400
Ftdlx
xtep
(a) Encoder
(b) Decoder
Figure 5: Lipschitz complexity with perceptual similarity (LPIPS complexity) of encoder and decoder. The overall trends are similar to those in Cltp shown in Figure
distribution than the original one. We further plot the eigenvalues of the cosine distance matrix for learned codes in Figure where we can clearly see that the eigenvalue decay for the baseline is significantly faster, indicating more severe dimensional collapse.
10a
1200
VOGAN
DAEAYOGAN
13000
42003
10-4
20002
10-1
4000
10-4
100
6.7 2:20
0293 0770
1.50
(a)
(b)
(c)
Figure 6: (a) Histogram of cosine similarity of codes. (b) Top 20 eigenvalues of the cosine distance
matrix. (c) Number of appearances of codes in sorted order.
For the latent expressed using the codebook, we count and sort the number of appearances of codes the training set of CelebAHQ (Figure We that DAF- can learn less redundant codes and hence their appearances are more evenly distributed. Interestingly, all codes are utilized in the reconstruction process for both VQGAN and DAE-VQGAN perhaps due to the limited size of the codebook.
B.4 Experiment details for DAE-DiT
We note that while the auxiliary decoder Jour approach presents clear trends of complexity changing, requires manually changing the network between the two stages, and hence is less convenient to implement than the Dropout approach in practice. Therefore adopt Dropout as the default approach in our DAE-DiT experiments.
Similar to DAE- DQGAN in the first stage of DAE-DIT, we implement the relatively weak auxiliary decoder by applying Dropout 733 to g with ratio p = 0.2, and train encoder f and the dropped decoder jointly. Then in the second stage, we freeze encoder f and train the full decoder g without Dropout. We train on the OpenImages [46] dataset and adopt the KL-18 model as in (68). The encoder and decoder are trained with learning rate 4.5 x 10 and batch size 64.
For the diffusion training, we adjust the scale factor from the default 0.18215 0.8 such that the variance of features from the encoder is close to 1. Following the same setup the official implementation, the EMA rate 0.9999 and the classifier-free guidance scale The computation during both the training and the testing process was executed using FP16 precision except for the attention module for computational stability.
22


Reviews:
[{"review_id": "i65jAOqcTd", "reviewer": "Reviewer_ssMC", "summary": "This paper proposes an asymmetric training scheme for auto-encoders that double as image generator. Based on analytical insights that the decoder should have less capacity than the encoder for the encoder to capture correctly the data distribution, they propose a first training cycle where a strong encoder and a weak decoder are trained jointly. This produces a latent distribution able to better capture the data distribution. Then, in a second stage crucial for the end application, the encoder is frozen and a strong decoder is trained using the \"good\" latent distribution. Experiments are carried out on faces datasets with VQGAN and class conditional imagenet with a diffusion model and show the proposed training scheme is promising.", "questions": "l.146 the link between the ideal $P_z$ and a low capacity generator is tenuous at best. There is no formal proof of that statement as far as I understand the paper. I can see a structural risk argument (ie, among all generator that equally minimize $D(p_x, p_z)$ chose the lowest complexity one), but it is more a principle and not a formal proof. Correct?\nl. 190 $D(P_z, P_x)$ will be significantly larger in case of arbitrarily chosen $P_z$. This you don't know. If $P_x$ and $P_z$ have the same topology, then there exists a continuous mapping between the two that can be perfectly approximated by a sufficiently large neural network. In practice, maybe $D$ will be larger, but not necessarily in theory.\nl. 198 I do not agree that $C(F) = C(G)$ is intuitive. For optimal reconstruction, we have $f = g^{-1}$ (assuming all that is required) and the Lipschitz constant of $g^{-1}$ can be arbitrarily large before that of $g$. For example, with $g(x) = x^2$ and $f(x) = \\sqrt{x}$ on $(\\epsilon, 1)$, the Lipschitz constant of $f$ can be made arbitrarily large. And vice versa. I think this depends whether it is easier (ie, the mapping has lower Lipschitz) to map from $x$ to $z$ or from $z$ to $x$. But what does it depend on? The size of $z$?\nl. 312 this is a bit cheating. You should learn an encoder that inverse the (frozen) Gaussian space and evaluate on reconstruction for DCGAN.\nl 354 Why is the Lipschitz of the encoder lower than that of the decoder for the proposed method? Isn't it completely contradictory to all the story of the paper?\nA2. Fix notation errors.", "limitations": "The conclusion is a list of limitations that is very honest.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper are :\n- It is a simple method, yet effective. As such, it should be easy to reproduce. The improvements are maybe questionable (no error bars, test against the training set - as is traditional in this domain), but given that they are reported with widely different architecture and on different dataset, there is more confidence than usual that it actually works.\n- The analysis part is really nice. The linear example in particular sheds some light as to why under a constrained budget, the complexity of the encoder should exceed that of the decoder to properly minimize the projected distance (divergence) to the data distribution.", "weaknesses": "The main weakness is that most of the paper is about the analytical part, which is a bit handwavy (acknowledged in the conclusion), and lacks a good structure. The reading could be improved by having a clear outline of the work such that one does not wonder where the text is headed to after each section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments and questions. Below we address them separately:\n\n\n### 1. \"... most of the paper is about the analytical part ... lacks a good structure.\"\n\n> Thanks for the feedback. \nAs is acknowledged, the analytical part mainly serves illustrative purposes and the proposed DAE is not proven mathematically. \nHowever, the DAE approach is an outcome and empirical verification of our investigation of the ideal latent distribution for generative modeling. \n\n> Making practical improvements to existing latent generative models is an important goal of this work. However, models such as VQGAN and DiT are too complicated for rigorous analysis. Instead, we draw inspiration from simple (linear) cases and verify the effectiveness on real scenarios through experiments. We conducted a series of experiments starting from toy Gaussian mixture data, to DCGAN on CIFAR-10, to VQGAN and DiT on larger datasets. \n\n> To improve readability, we summarized the contributions of this work in the overall response and we will incorporate them at the end of the Introduction section to provide an overview.\n\n\n\n### 2. \"l.146 The link between the ideal $P_z$ and a low capacity generator is tenuous ...\"\n\n \n> The ideal $P_z^*$ is characterized as the one that minimizes $D^G(P_z,P_x)$. \nThe link between its definition and the capacity of the generator **can be made mathematically sound with an extra assumption**. \n\n> First, recall that $G_c:=\\{g\\in G: C(g)\\le c\\}$ where $C(g)$ is defined as some complexity measurement. It is easy to see that for any $c_2 \\ge c_1>0$, $G_{c_1} \\subset G_{c_2}$. Therefore, $D^{G_{c_2}}(P_z, P_x) \\le D^{G_{c_1}}(P_z, P_x)$. \n\n> Assumption 1: For any $P_z$ and $P_x$, $D^{G_c}(P_z, P_x)$ is continuous and monotonically decreasing with c. \n\n> Consider a generator family $G$ with bounded complexity and we target to have the generated distribution close to the data at level $\\epsilon$ measured by $D$, i.e., $D(P_x, P_{g(z)})\\le \\epsilon$ for some $P_z$ and $g\\in G$. \n\n> For a certain (non-degenerate) $P_z$, there exists $c>0$ such that $D^{G_c}(P_z, P_x) = \\epsilon$, that is, by using this latent $P_z$, we need at least complexity $c$ to achieve the $\\epsilon$ goal. \nIf $P_z$ is not ideal, by definition we know that there exists $P_z^*$ that $D^{G_c}(P_z^*, P_x)=\\epsilon^* < \\epsilon$. By Assumption 1, we know there exists $c^\\prime< c$ such that $D^{G_{c^\\prime}}(P_z^*, P_x) = \\epsilon$. \nThis means that the goal can be achieved using a lower complexity generator if the latent is ideal. \n\n> Assumption 1 is not provable for general $D(\\cdot, \\cdot)$ and $C(\\cdot)$. You are correct that this is more of a principle than a formal proof.\n\n\n### 3. \"l.190 $D(P_z,P_x)$ will be significantly larger in case of arbitrarily chosen $P_z$...\"\n\n> You are right that if $P_x$ and $P_z$ have the same topology with a continuous mapping between the two, $D^{ae}(P_z,P_x)$ may not be significantly larger. \n> We will modify the statement to $D^{ae}(P_z,P_x)$ will be significantly larger in case of poorly chosen $P_z$ (e.g., uni-modal $P_z$ to multi-modal $P_x$) to make it more clear. \n\n\n\n### 4. \"l.198 I do not agree that $C(F)=C(G)$ is intuitive... l.354 Why is the Lipschitz of the encoder lower than that of the decoder for the proposed method?\"\n\n> In the case of optimal reconstruction, we have $f=g^{-1}$ and the Lipschitz constant of $f$ can be arbitrarily large. This is related to our Remark 3.5 about scaling. \nLet $C(\\cdot)$ be the Lipschitz constant. \n$k\\cdot f(x)$ and $g(x/k)$ have the same reconstruction error. \n\n> To address the scaling problem, we considered the generalized $D^{ae}(\\cdot, \\cdot)$, where both $C(f)$ and $C(g)$ are considered and we want their sum to be small.\nIn this case, $C(f^*)=C(g^*)=1$ is the optimal choice since $C(f^*) + C(g^*) = C(f^*) + 1/C(f^*)\\ge 2$ is minimized when $C(f^*)=1$. \n\n\n> Going back to the line 198, it is worth emphasizing that, as stated in line 133, $C(\\cdot)$ can be as specific as the Lipschitz constant, or as general as the size (width, depth, etc.) of the network. \nWe will add remarks before line 198 to clarify that the complexity $C(\\cdot)$ here is the latter general case. \nIn practice, the encoder-decoder pair is often designed with symmetric architecture, with the same number of blocks and parameters, e.g., up-sampling vs. down-sampling, convolution vs. deconvolution, etc. \nThe $C(F)=C(G)$ statement is not mathematically rigorous and we apologize for the confusion. \n\n\n> For the line 198 argument, we refer to **Figure 4** of the Appendix, where we can see that our DAE-modified encoder (orange line) is closer to one than the baseline (blue line). The same is true for the decoder. \nGiven that the ideal case is that both are one, our DAE is effective at reducing the complexity. \n\n> As to whether the encoder or the decoder has a larger Lipschitz constant, we hypothesize that in the initial stage, it may closely relate to the mapping dimension. If $x_i$'s and $z_i$'s are of the same scale, $\\|x\\|_2/\\|z\\|_2\\approx \\sqrt{d/d_z}$. Hence, from high-dimensional $x$ to low-dimensional $z$, the Lipschitz constant might be smaller. This can be seen in the early stage of our Figure 4, where $C(f)\\approx 0.2$ in the beginning.   \n\n> However, after training, we do not know why the encoder has a larger Lipschitz constant.     \n\n\n\n### 5. \"l.312 this is a bit cheating.\"\n\n\n\n> Treating DCGAN as a baseline, we introduced 3 modifications and the results are shown in Table 2.\nBy DCGAN vs. DCGAN-SimCLR, we want to demonstrate that learning latent for generative modeling is different from standard self-supervised learning methods for discriminative tasks. \nBy DCGAN vs VAEGAN, we want to demonstrate that a learned latent from data can be better than data-agnostic Gaussian, and VAEGAN can be further improved by our DAE approach. \n\n\n### 6. \"A2. Fix notation.\"\n\n> Thanks for pointing it out. We have fixed the singular values notation to $\\lambda_i$ in line 576."}, {"review_id": "BAtxpdURoI", "reviewer": "Reviewer_2zRJ", "summary": "The paper first proposes a new framework to analyze latent spaces in the context of generative models.\nThis framework takes inspiration from prior results about GANs, which allowed interpreting the min-max training objective as computing a distance between distributions to be minimized, to define a similar distance between the latent distribution and the data distribution.\nFrom their analysis, they derive a simple two-step training for auto-encoders to learn better latents and reconstruction, in which they first train the encoder with a weak decoder to extract good latents, and then train a larger decoder to get better reconstructions.\n\nThey perform experiments in a simple toy case, and then with commonly used models such as GANs, VQGAN, and DiTs.", "questions": "One minor thing that was not clear to me is Remark 3.5. Could the authors explicit why scaling invariance would be a problem in this context?\n\nIn any case, despite its limitations, I still find the paper interesting for both its theoretical and practical contributions.\nI trust the authors will fix the related work section, and my current rating anticipates that the authors will demonstrate their willingness to do so.\n\n-----\nPost-rebuttal update:\nHaving taken into consideration the rebuttal, discussions and comments from the other reviewers, I updated my score from WA to Accept.", "limitations": "\nThe authors are very open regarding the limitations of their submission.\n\nPotential negative societal impacts of generative models are not. Widely acknowledge ones include amplification of biases, potential misuse for propagating fake information, and concerns about lack of attribution and copyright infringements.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is easy to follow\n- It proposes a novel view on latent codes, including an explanation of some different properties between SSL and generative latent codes, and a theoretical framework to describe why a powerful encoder/decoder pair can't learn a good latent code.\n- The proposed practical solution is simple and their experiments show that it can improve performances in a variety of generative settings.", "weaknesses": "- While the fresh view on latent codes is interesting, it doesn't provide any theoretical guarantees.\nAs far as I understand, the main conclusion is that in order to obtain a good latent code the encoder and decoder should not be too powerful but is not able to give an indication about how the correct balance. Note that in the context of Variational Autoencoders, a similar conclusion had been reached before: that a too powerful decoder is a good explanation for the phenomenon called *posterior collapse* that describes a state in which the latent code is completely uninformative [1].\nIt is good that it is formally extended to other auto-encoders, but unsurprising.\n\n- Because of this lack of quantification of optimal complexity, it is quite unclear whether the positive results obtained in VQGAN and DiT settings are actually related to the theoretical conclusions or not. It could very well be completely unrelated and just confirmation bias.\n\n- Related work present different generative models with loose links to the proposed method, but does not discuss any study about the latent spaces of generative models. In addition to links with posterior collapse in VAEs [1], I would also be very interested to read what the authors have to say about sparsity and disentanglement properties of VAE [2], and PCA directions in GAN space [3] among other things.\nMoreover, I have to point out that contrary to the paragraph in related works, Mask Autoencoders have been explored for their generative capabilities [4].\n\n\n--- \n[1] Fixing a Broken ELBO. ICML 2018.\nAlexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, Kevin Murphy.\n\n[2] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. ICML 2019.\nFrancesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem\n\n[3] GANSpace: Discovering Interpretable GAN Controls. NeurIPS 2020.\nErik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris\n\n[4] MaskGIT: Masked Generative Image Transformer. CVPR 2022.\nHuiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments. Below we address them separately:\n\n### 1. \"While the fresh view on latent codes is interesting, it doesn't provide any theoretical guarantees.\"\n\n> Thanks for the question.\nYou are correct that the main idea of our DAE approach is to balance the encoder and the decoder in different stages and our analysis did not offer precise algorithmic guidelines on how to achieve the optimal balance.\n\n> However, it is worth emphasizing that the DAE approach is an outcome and empirical verification of our investigation of the ideal latent distribution for generative modeling. \n\n> * Motivated by the GAN training objective, we first introduced $D^G$ to measure the closeness between the latent and the data in distribution. To the best of our knowledge, this work is the first to provide a characterization of the **optimal latent** distribution from the perspective of minimizing model complexity. Our proposed characterization has its own interest and may shed light on other applications as well.\n> * By utilizing an encoder, i.e., $P_z=P_{f(x)}$, we argue that minimizing $D^G(P_f(x), P_x)$ with respect to $f$ will give rise to the optimal latent. **Notice that here we are considering matching in distribution, rather than sample-wise reconstruction**. Therefore, we are advocating the use of adversarial training techniques as in VQGAN to train the encoder. \n> * We then identified the trade-off between the encoder and decoder, accompanied by a rigorous linear case analysis (Theorem 4.1). To address the trade-off, we proposed the decoupled approach DAE.     \n\n\n> Making practical improvements to existing latent generative models is an important goal of this work. However, models such as VQGAN and DiT are too complicated for rigorous analysis. Instead, we draw inspiration from simple (linear) cases and verify the effectiveness on real scenarios mainly through experiments.\n\n### 2. Relationships to VAE literature\n\n> This is a good point. As you pointed out, the fact that a decoder too powerful can result in posterior collapse is well known in VAE literature. Here we clarify the key differences to our work.   \n\n> * Different target: This work focuses on the **ideal latent** distribution for generative modeling from the perspective of **minimizing complexity**. We consider mainly **deterministic** encoder and decoder where the decoder is the generator, and the encoder is what we used to parameterize the latent distribution. In our work, a good latent distribution is the primary target while in VAE, the latent distribution is often chosen in advance to be data-agnostic priors.\n\n> * Deterministic vs stochastic: To characterize the ideal latent distribution, we introduced $D^G(P_z, P_x)$, which is closely connected to GANs (Integral Probability Metrics or $f$-divergence). In comparison, VAEs usually maximize likelihood or mutual information $I$. \nIt's worth noting that **while $I(x, z)$ can also serve as a \"distance\" between distributions in different dimensions, it will not work for our benefit** since we are considering deterministic encoders and decoders. \nIn the deterministic encoder case, we have the conditional entropy $H(f(x)|x)=0$ and $I(x, f(x))=H(f(x))-H(f(x)|x)=H(f(x))$, where $H$ denotes entropy. \nThat is, the mutual information would always be the entropy of the latent, hence uniformly distributed latent is always ideal. \n\n> In Fixing a Broken ELBO [1], the authors stated that obtaining a good ELBO is not enough for good representation learning. In practice, the authors proposed to target at a desired target rate $R^*$, such that a better balance between the informativeness of the latent and the reconstruction quality. \nAs a result of the last bullet point, this approach cannot extend to the deterministic case. \n\n> In comparison, we focus on the complexity of the generator and we provided characterization of the optimal latent. \n\n> [1] also discussed realizability and their optimality is hinted to be the tightest achievable sandwiched bound within the parametric family. However, no formal statement is given. \n\n> Nevertheless, the VAE literature is related and we will add discussions to it in our revision. \n\n\n### 3. \"Related work presents different generative models with loose links to the proposed method, but does not discuss any study about the latent spaces of generative models.\"\n\n> Thanks for the suggested related works [1-4]. We will add a paragraph in the related work section to discuss this line of study in our revision. \n\n> Here we briefly discuss their relationships to our work. \n\n\n> Representation disentanglement is discussed in [2], which is not covered in our work and would be an interesting future direction to explore. [3] also follows this line of work and investigates the disentangled or interpretable latent variables or directions in GAN generation to control different aspects of the image, e.g., viewpoint, aging, etc. \n\n\n> In MaskGIT, the authors investigated how to better model the tokenized images, rather than finding better latent spaces to do masked modeling. \nOur insights about the optimal latent distribution and the DAE approach can also be beneficial in their settings. \n\n\n### 4. \"It is quite unclear whether the positive results obtained in VQGAN and DiT settings are actually related to the theoretical conclusions or not.\"\n\n> To empirically evaluate our DAE approach, we conducted a series of experiments from Gaussian mixture data, to DCGAN on CIFAR-10, to VQGAN and DiT on larger datasets. The improvements are consistent across different settings. \n\n> In practice, we provide a general rule-of-thumb on how to weaken the decoder in the first stage (cut the channels in half or add Dropout at strength 0.5). \nSuch configurations work across different settings and we have not (and cannot afford to) done much fine-tuning, which supports the hypothesis that our theoretical insights are suitable and effective when applied to real-world model architectures."}, {"review_id": "DEdqQG9FV2", "reviewer": "Reviewer_14Q5", "summary": "This paper presents an approach with theoretical analysis to explore a more suitable latent distribution for generation. For this purpose, this paper proposes a novel distance between the latent and target distributions and tries to minimize it to obtain the optimal data-dependent latent distribution. In practice, a two-stage training strategy called Decoupled Autoencoder (DAE) is introduced to leverage a superior latent distribution to improve the generative performance. The experiments on VQGAN and DiT show the effectiveness of the proposed method.", "questions": "1.As discussed in [2], the natural image distribution lie upon multiple disjoint manifolds, which potentially requires a discontinuous latent distribution (Prop. 3.2). Does the proposed method adhere to this requirement?\n\n2.In Fig. 2 (a), actually it is hard to discern noticeable performance improvements in the comparison.\n\n3.I am curious why this method can be applied to diffusion models, for which the Gaussian is the stable distribution for noise addition. If the latent distribution is altered, how does the mechanism of diffusion models still function?\n\nPlease also see the discussion of weaknesses. If the issues are resolved, I would be inclined to reassess and potentially raise my rating.", "limitations": "Please see above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper is well-structured and easy to follow.\n\n2.The methodology is supported by theoretical analysis.", "weaknesses": "1.The proposed methodology is somehow similar to AE-OT-GAN [1], which also learns a latent distribution via an autoencoder and then utilizes the learned latent to train GAN. The authors should explicitly highlight the advantages of their approach over [1].\n\n[1] AE-OT-GAN: Training GANs from Data Specific Latent Distribution. ECCV 2020. \n\n2.Some important related works are missed, particularly the line of works that explore improved latent sampling in GANs, e.g., [1] [2] [3] [4]. In special, AdvLatGAN [2] seeks to adjust and discover a more suitable latent distribution via adversarial learning in both the post-training sampling and the training, and it has already introduced the concept of optimal latent distribution noted $p_z^{op}$ in its theory.\n\n[2] Improving Generative Adversarial Networks via Adversarial Learning in Latent Space. NeurIPS 2022.\n\n[3] Discriminator optimal transport. NeurIPS 2019.\n\n[4] Your gan is secretly an energy-based model and you should use discriminator-driven latent sampling. NeurIPS 2020.\n\n3.The experimental results are insufficient. The comparison basically focuses on the performance gain beyond the vanilla backbones. Authors should compare the other efforts on improving the latent sampling/distribution, e.g., the previously mentioned works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our work. There might be some misunderstanding and we have summarized and clarified the contributions of this work in the overall response. Hopefully, it can address some of your concerns.\n\nIt is worth noting that although our motivation involves vanilla GAN models, our proposed methodology has an encoder-decoder structure that can be potentially adopted by a wider range of generative models that utilize a low-dimensional latent space.\n\n\n### 1.\"The proposed methodology is somehow similar to AE-OT-GAN\"\n\n> **The focus of our work and AE-OT-GAN is different**. \n\n> To clarify, we first summarize the procedure of AE-OT-GAN:\n> - (1) Train an autoencoder to embed data into the latent space (with distribution $\\mu$) by minimizing the reconstruction loss;\n> - (2) Constructing $T$ with Semi-Discrete OT Map such that $T$ maps uniform $U[0, 1]^d$ to $\\mu$;\n> - (3) GAN Training from $\\mu$.\n\n> Our work shares the motivation of AE-OT-GAN but we investigate specifically the first step of AE-OT-GAN, i.e., the way to find a better latent. \nIn comparison, the AE part in AE-OT-GAN follows standard procedures. \nIn this sense, AE-OT-GAN can be seen as another example of a generative model that utilizes a latent distribution (induced by the encoder). \nHence, our DAE modifications can also be applied to AE-OT/AE-OT-GAN. In the first step, we can utilize an auxiliary decoder to train a better autoencoder, potentially benefiting the afterward OT/GAN modeling. \n\n> We will add discussions to the AE-OT-GAN work in our revision. \n\n\n###  2. \"Some important related works are missed, particularly the line of works that explore improved latent sampling in GANs\"\n\n> For the benefit of other viewers, let us first briefly summarize the related work [2-4]. \nBoth DOT [3] and DDLS [4] can be viewed as post-sampling latent space mining of GANs, by exploiting the trained discriminator (as discussed in the related work section of [4]). \nAdvLatGAN [2] further considered the GAN training process and proposed adding an implicit latent transform before the mapping function to improve the latent from its initial distribution. The implicit latent transform is trained by adversarial sample mining methods in a bi-level optimization fashion. \n\n\n> All aforementioned works can improve the performance of GAN models through latent space optimization. \nHowever, they do not characterize the **optimal latent distribution** and do not consider using an encoder network to parameterize/estimate it. \nAs an encoder-decoder structure is popularly used in large-scale text-to-image generation models, such as stable diffusion, DiT, Muse, Parti, etc, we believe our investigation could bring new insights to them. \n\n\n\n###  3.\"The experimental results are insufficient...Authors should compare the other efforts on improving the latent sampling/distribution...\"\n\n> Our work focuses on improving the latent distribution of generative models with a low-dimensional latent space. \nThe key practical contribution is the 2-stage DAE training of the autoencoders. \nFor the baseline autoencoders, the popular training methods, vector-quantized or KL-regularized are both experimented in this work, to demonstrate the effectiveness of our new understanding. \n\n> Decoder-only GAN models are not our main target and [1-4] do not consider the autoencoder structure. \nNevertheless, we have conducted DCGAN experiment on CIFAR-10 in Section 6.2 and the results are reported in Table 2.\n\n> Treating DCGAN as a baseline, we introduced 3 modifications.\nBy DCGAN vs. DCGAN-SimCLR, we want to demonstrate that learning latent for generative modeling is different from standard self-supervised learning methods for discriminative tasks. \nBy DCGAN vs VAEGAN, we want to demonstrate that a learned latent from data can be better than data-agnostic Gaussian, and VAEGAN can be further improved by our modified DAE-VAEGAN. \n\n> For larger-scale experiments, we considered VQ-GAN and DiT.\n\n\n\n### Q1: \"... natural image distribution lies upon multiple disjoint manifolds...Does the proposed method adhere to this requirement?\"\n\n> Yes. If the data distribution lies upon multiple disjoint manifolds, the continuous encoder is capable of mapping them to multiple disjoint manifolds in the latent space. The resulting $P_{f(x)}$ can be multi-modal as well. \n\n> This is not a special property of our method. In fact, any AE-type methods can handle the challenge, as discussed in AE-OT. Our work made further improvements over the standard AE training and can result in a better latent distribution. \n\n\n### Q2: \"In Fig. 2 (a), actually it is hard to discern noticeable performance improvements in the comparison.\"\n\n> The figures are highly subjective and serve as a complement to the reported FID. Please mainly refer to the FIDs for a quantitative evaluation.\nWe will move Fig 2(a) to the appendix and add more figures to showcase the difference. \n\n### Q3: \"I am curious why this method can be applied to diffusion models...\"\n\n\n> In our work, the latent distribution $P_z$ refers to the **low-dimensional** distribution to be mapped to the data distribution $P_x$. \nThe stable distribution of the forward process of diffusion models is sometimes referred to as the \"latent distribution\" as well. However, we do not call it the latent distribution in this work. \nWe apologize for the confusion and we will add a remark to clarify the difference. \n\n> As we reiterated, our work focuses on finding a better latent distribution for generative modeling. \nGiven the latent space, the generative modeling itself, whether it is autoregressive or diffusion, is not the focus of this work. \nFor latent diffusion models, such as Stable-Diffusion and Diffusion Transformers, the diffusion modeling is on the latent space $P_z$, instead of $P_x$. \nHence, our DAE modifications can be directly applied, not just to latent diffusion models, but potentially all generative models with an autoencoder structure. "}, {"review_id": "7jV9exybbG", "reviewer": "Reviewer_JCyg", "summary": "This work investigates what constitutes a good latent space for generative models, and proposes a new training paradigm for generative models – DAE. Simply put, with DAE generative models are trained as Autoencoder in two stages. First, a relatively weak decoder is employed, whose purpose is to aid the encoder in learning meaningful representations. Second, the weak decoder is replaced with the “actual” decoder and training is continued. The second decoder is the one eventually evaluated as a generative model.\n", "questions": "Could the authors please elaborate on “there are other algorithms to implement the\ndecoupled 2-stage training, and the evaluations in this work are not comprehensive”? Why would the authors not include such evaluations in the paper if they find them relevant?\n\n", "limitations": "Authors addressed limitations. One of them (non comprehensive evaluation) raises questions. I would appreciate a clarification on that.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is very well written, easy and interesting to follow.\n2. The questions of what constitutes a good latent space and how is one constructed are often overlooked. Most works follow longstanding paradigms of using predetermined distributions (e.g., Gaussian) or whatever is learned from an autoencoder. Posing these questions and formulating the setting, on its own, is impactful.\n3. The proposed method DAE is empirically proven effective, and since it is rather simple to implement, I conjecture it might have a strong impact on future generative modeling works.\n", "weaknesses": "While the mathematical formulation in the paper looks sound to me, I think it does not benefit the paper, and significant portions of it could be moved into the appendix. As the authors acknowledge in the Discussion section, the formulation“ serves mainly illustrative purposes” and is “not proven mathematically”. I don’t see it as an issue, as much of Generative Modeling research (and ML in general) is empiric in nature. The proposed method, DAE, could be introduced as an empirically-supported design, while some of the mathematical formulation could be described to serve as intuition (e.g., Theorem 4.1) . However, dedicating over three pages to it seems excessive to me. In my opinion, most readers would greatly benefit if the experiments in Appendix B were present in the main paper instead of some of the mathematical formulation. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments and questions. Below we address them separately:\n\n\n### 1. \"The proposed method, DAE, could be introduced as an empirically-supported design, while some of the mathematical formulation could be described to serve as intuition ... In my opinion, most readers would greatly benefit if the experiments in Appendix B were present in the main paper instead of some of the mathematical formulation.\"\n\n> Thanks for the advice. \n> You are correct that DAE could be introduced as an empirically-supported design. However, we want to bring more insights into the latent distribution for generative modeling. \n\n> It is worth emphasizing that the DAE approach is an outcome and empirical verification of our investigation of the ideal latent distribution for generative modeling. \n\n> * Motivated by the GAN training objective, we first introduced $D^G$ to measure the closeness between the latent and the data in distribution. To the best of our knowledge, this work is the first to provide a characterization of the optimal latent distribution from the perspective of minimizing model complexity. Our proposed characterization has its own interest and may shed light on other applications as well.\n> * By utilizing an encoder, i.e., $P_z=P_{f(x)}$, we argue that minimizing $D^G(P_f(x), P_x)$ with respect to $f$ will give rise to the optimal latent. Notice that here we are considering matching in distribution, rather than sample-wise reconstruction. Therefore, we are advocating the use of adversarial training techniques as in VQGAN to train the encoder. \n> * We then identified the trade-off between the encoder and decoder, accompanied by a rigorous linear case analysis (Theorem 4.1). To address the trade-off, we proposed the decoupled approach DAE.  \n\n\n> The experiments in Appendix B are certainly interesting. We will try to compress the mathematical formulation in the first part of the paper and put some of the results from Appendix B in the main text, to improve the reading experience.\n\n\n### 2. Elaboration of \"there are other algorithms to implement the decoupled 2-stage training, and the evaluations in this work are not comprehensive?\"\n\n> The gist of our DAE modification is to make the decoder relatively weak in the first stage when training the encoder. By other algorithms, we meant that there are various ways to make a weak decoder. We only explored reducing the number of channels and 2D Dropout. Other candidates include reducing the number of layers, adding $l_2$ regularization, etc. \nDue to the computational budget, we only experiment with two common ways of regularization. The improvement is consistent in our evaluations and we believe other forms will also work. "}]

Author rebuttal:
# Overall Response:

We thank all the reviewers for their time and efforts in reviewing our work. Before we address the questions and concerns of each reviewer, we would like to provide a summary of our work.

Our work aims to **characterize the ideal/optimal low-dimensional latent distribution for latent generative modeling**, from the perspective of minimizing the required **complexity** of the generator. The main contributions can be summarized as the following:

- Inspired by the training objective of GANs, we first propose a pseudo-distance $D^G$ between $P_x$ and $P_z$ and **characterize the optimal latent distribution $P_z^*$** by the one that is closest to $P_x$ in the terms of $D^G$. (Section 3.1)

- After characterizing $P_z^*$, we adopted the popular encoder parameterization of the latent distribution, i.e., $P_z = P_{f(x)}$ (Section 3.2). We then analyzed the interplay between the encoder and the decoder (generator) and identified the **trade-off between the quality/informativeness of the latent distribution and the capability of the decoder**. (Section 4.1)

- To address the trade-off, we proposed a **two-stage training scheme** called DAE that results in better latent spaces for more efficient generative modeling (Section 4.2). To verify our claims, we conducted experiments on various models such as GAN, VQGAN, and DiT, and achieved consistent improvement (Section 6). 


Although our motivation involves vanilla GAN models, our proposed methodology has an encoder-decoder structure that can be potentially adopted by a wider range of generative models that utilize a low-dimensional latent space.

To the best of our knowledge, this work is the first to provide a characterization of the optimal latent distribution from the perspective of minimizing model complexity. 
Our DAE approach is an outcome of our investigation along this line and an empirical verification of its practical effectiveness. 
The proposed characterization has its own interest and may shed light on other applications as well.



Meta-review: The paper makes a significant contribution to the field of generative modeling by providing a theoretical characterization of the optimal latent distribution for minimizing the required complexity of the generator. The authors propose a novel "distance" between the latent and data distributions, whose minimization coincides with that of the generator complexity. They then show that the minimizer of this distance can be parameterized by an encoder network, and they propose a two-stage training strategy called Decoupled Autoencoder (DAE) that can be used to learn this optimal latent distribution.

The authors conduct extensive experiments on various generative models, and they show that DAE consistently improves the quality of generated samples while also reducing the complexity of the generator. These results demonstrate the effectiveness of the proposed approach, and they make a significant contribution to the field of generative modeling.

The paper is well-written and easy to follow. The authors provide a clear and concise explanation of their approach, and they support their claims with strong experimental evidence. The paper is also well-referenced, and it makes a significant contribution to the existing literature.

I have no major concerns about the paper. The only minor issue is that the authors could provide more details about the implementation of DAE. However, this is a minor issue, and it does not detract from the overall quality of the paper.
Decision: Accept (spotlight)