On the Generative Utility of Cyclic Conditionals

Abstract:
We study whether and how can we model a joint distribution p(x, z) using two conditional models p(x|z) and q(z|x) that form a cycle. This is motivated by the observation that deep generative models, in addition to a likelihood model p(x|z), often also use an inference model q(z|x) for extracting representation, but they rely on a usually uninformative prior distribution p(z) to define a joint distribution, which may render problems like posterior collapse and manifold mismatch. To explore the possibility to model a joint distribution using only p(x|z) and q(z|x), we study their compatibility and determinacy, corresponding to the existence and uniqueness of a joint distribution whose conditional distributions coincide with them. We develop a general theory for operable equivalence criteria for compatibility, and sufficient conditions for determinacy. Based on the theory, we propose a novel generative modeling framework CyGen that only uses the two cyclic conditional models. We develop methods to achieve compatibility and determinacy, and to use the conditional models to fit and generate data. With the prior constraint removed, CyGen better fits data and captures more representative features, supported by both synthetic and real-world experiments.

heading: 1 Introduction
text: Deep generative models have achieved a remarkable success in the past decade for generating realistic complex data x and extracting useful representations through their latent variable z. Variational auto-encoders (VAEs) [45, 67, 14, 15, 46, 80] follow the Bayesian framework and specify a prior distribution p(z) and a likelihood model p(x|z), so that a joint distribution p(x, z) = p(z)p(x|z) is defined for generative modeling (the joint induces a distribution p(x) on data). An inference model q(z|x) is also used to approximate the posterior distribution p(z|x) (derived from the joint p(x, z)), which serves for extracting representations. Other frameworks like generative adversarial nets [30, 25, 27], flow-based models [24, 60, 44, 31] and diffusion-based models [74, 38, 76, 49] follow the same structure, with different choices of the conditional models p(x|z) and q(z|x) and training objectives. While for the prior p(z), there is often not much knowledge for complex data (like images, text, audio), and these models widely adopt an uninformative prior such as a standard Gaussian. This however, introduces some side effects: • Posterior collapse [15, 34, 64]: The standard Gaussian prior tends to squeeze q(z|x) towards
the origin for all x, which degrades the representativeness of the inferred z for x and hurts downstream tasks in the latent space like classification and clustering. • Manifold mismatch [22, 28, 41]: Typically the likelihood model is continuous (keeps topology), so the standard Gaussian prior would restrict the modeled data distribution to a simply-connected support, which limits the capacity for fitting data from a non-(simply) connected support.
While there are works trying to mitigate the two problems, they require either a strong domain knowledge [48, 41], or additional cost to learn a complicated prior model [52, 21, 79] sometimes even at the cost of inconvenient inference [59, 87]. ∗Correspondence to: Chang Liu <changliu@microsoft.com>. †Work done during an internship at Microsoft Research Asia.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
One question then naturally emerges: Can we model a joint distribution p(x, z) only using the likelihood p(x|z) and inference q(z|x) models? If we can, the limitations from specifying or learning a prior are then removed from the root. Also, the inference model q(z|x) is then no longer a struggling approximation to a predefined posterior but participates in defining the joint distribution (avoid “inner approximation”). Modeling conditionals is also argued to be much easier than modeling marginal or joint distributions directly [2, 9, 10]. In many cases, one may even have better knowledge on the conditionals than on the prior, e.g. shift/rotation invariance of image representations (CNNs [51] / SphereNet [20]), and rules to extract frequency/energy features for audio [65]. It is then more natural and effective to incorporate this knowledge into the conditionals than using an uninformative prior. In this paper, we explore such a possibility, and develop both a systematic theory and a novel generative modeling framework CyGen (Cyclic-conditional Generative model). (1) Theoretical analysis on the question amounts to two sub-problems: can two given cyclic conditionals correspond to a common joint, and if yes, can they determine the joint uniquely. We term them compatibility and determinacy of two conditionals, corresponding to the existence and uniqueness of a common joint. For this, we develop novel compatibility criteria and sufficient conditions for determinacy. Beyond existing results, ours are operable (vs. existential [11]) and self-contained (vs. need a marginal [9, 10, 50, 32]), and are general enough to cover both continuous and discrete cases. Our compatibility criteria are also equivalence (vs. unnecessary [1, 4–6]) conditions. The seminal book [6] makes extensive analysis for various parametric families. Besides the equivalence criteria, we also extend their general analysis beyond the product support case, and also cover the Dirac case. (2) In addition to its independent contribution, the theory also enables generative modeling using only the two cyclic conditional models, i.e. the CyGen framework. We develop methods for achieving compatibility and determinacy to make an eligible generative model, and for fitting and generating data to serve as a generative model. Efficient implementation techniques are designed. Note CyGen also determines a prior implicitly; it just does not need an explicit model for the prior (vs. [52, 21, 79, 59]). We show the practical utility of CyGen in both synthetic and real-world tasks. The improved performance in downstream classification and data generation demonstrates the advantage to mitigate the posterior collapse and manifold mismatch problems.

heading: 1.1 Related work
text: Dependency networks ([35]; similarly [39]) are perhaps the first to pursue the idea of modeling a joint by a set of conditionals. They use Gibbs sampling to determine the joint and are equivalent to undirected graphical models. They do not allow latent variables, so compatibility is not a key consideration as the data already specifies a joint as the common target of the conditionals. Beyond that, we introduce latent variables to better handle sensory data like images, for which we analyze the conditions for compatibility and determinacy and design novel methods to solve this different task. Denoising auto-encoders (DAEs). AEs [71, 7] aim to extract data features by enforcing reconstruction through its encoder and decoder, which are deterministic hence insufficient determinacy (see Sec. 2.2.2). DAEs [82, 9, 10] use a probabilistic encoder and decoder for robust reconstruction against random data corruption. Their utility as a generative model is first noted through the equivalence to score matching (implies modeling p(x)) for a Gaussian RBM [81] or an infinitesimal Gaussian corruption [2]. In more general cases, the utility to modeling the joint p(x, z) is studied via the Gibbs chain, i.e. the Markov chain with transition kernel p(x′|z′)q(z′|x). Under a global [9, 50, 32] or local [10] shared support condition, its stationary distribution π(x, z) exists uniquely. But this is not really determinacy: even incompatible conditionals can have this unique existence, in which case π(z|x) 6= q(z|x) [35, 9]. Moreover, the Gibbs chain does not give an explicit expression of π(x, z) (thus intractable likelihood evaluation), and requires many iterations to converge for data generation and even for training (Walkback [9], GibbsNet [50]), making the algorithms costly and unstable. As for compatibility, it is not really covered in DAEs. Existing results only consider the statistical consistency (unbiasedness under infinite data) of the p(x|z) estimator by fitting (x, z) data from p∗(x)q(z|x) [9, 10, 50, 32], where p∗(x) denotes the true data distribution. Particularly, they require a marginal p∗(x) in advance, so that the joint is already defined by p∗(x)q(z|x) regardless of p(x|z), while compatibility (as well as determinacy) is a matter only of the two conditionals. More crucially, the DAE loss is not proper for optimizing q(z|x) as it promotes a mode-collapse behavior. This hinders both compatibility and determinacy (Sec. 3.2): one may not use q(z|x) for inference, and data generation may depend on initialization. In contrast, CyGen explicitly enforces compatibility and guarantees determinacy, and enables likelihood evaluation and better generation.
Dual learning considers conversion between two modalities in both directions, e.g., machine translation [33, 85, 84] and image style transfer [42, 90, 88, 53]. Although we also consider both directions, the fundamental distinction is that in generative modeling there is no data of the latent variable z (not even unpaired). Technically, they did not consider determinacy: they require a marginal to determine a joint. We find their determinacy is actually insufficient (see Sec. 2.2.2). Their cycle-consistency loss [42, 90, 88] is a version of our compatibility criterion in the Dirac case (see Sec. 2.2.1), and we extend it to allow probabilistic conversion (see Sec. 3.1).

heading: 2 Compatibility and Determinacy Theory
text: To be a generative model, a system needs to determine a distribution on the data variable x. With latent variable z, this amounts to determining a joint distribution over (x, z), which calls for compatibility and determinacy analysis for cyclic conditionals. In this section we build a general theory on the conditions for compatibility and determinacy. We begin with formalizing the problems. Setup. Denote the measure spaces of the two random variables x and z as (X,X , ξ) and (Z,Z , ζ)3, where X , Z are the respective sigma-fields, and the base measures ξ, ζ (e.g., Lebesgue measure on Euclidean spaces, counting measure on finite/discrete spaces) are sigma-finite. We use X ∈X , Z ∈ Z to denote measurable sets, and use “ ξ=”, “⊆ξ” as the extensions of “=”, “⊆” up to a set of ξ-measure-zero (Def. A.1). Following the convention in machine learning, we call a “probability measure” as a “distribution”. We do not require any further structures such as topology, metric, or linearity, for the interest of the most general conclusions that unify Euclidean/manifold and finite/discrete spaces and allow X, Z to have different dimensions or types. Joint and conditional distributions are defined on the product measure space (X×Z,X ⊗Z , ξ⊗ζ), where “×” is the usual Cartesian product, X ⊗Z := σ(X ×Z ) is the sigma-field generated by measurable rectangles from X ×Z , and ξ⊗ζ is the product measure [13, Thm. 18.2]. Define the slice ofW ∈ X ⊗Z at z asWz := {x | (x, z) ∈ W} ∈ X [13, Thm. 18.1(i)], and its projection onto Z asWZ := {z | ∃x ∈ X s.t. (x, z) ∈ W} ∈ Z (Appx. A.3). In a similar style, denote the marginal of a joint π on Z as πZ(Z) := π(X×Z). To keep the same level of generality, we follow the general definition of conditionals ([13, p.457]; see also Appx. A.4): the conditional π(X|z) of a joint π is the density function (R-N derivative) of π(X×·) w.r.t πZ. We highlight the key characteristic under this generality that π(·|z) can be arbitrary on a set of πZ-measure-zero, particularly, outside the support of πZ. Appx. A and B provide more background details and our technical preparations that are also of independent interest. The goal of analysis can then be formalized below. Definition 2.1 (compatibility and determinacy). We say two conditionals µ(X|z), ν(Z|x) are compatible, if there exists a joint distribution π on (X×Z,X ⊗Z ) such that µ(X|z) and ν(Z|x) are its conditional distributions. We say two compatible conditionals have determinacy on a set S ∈X ⊗Z , if there is only one joint distribution concentrated on S that makes them compatible. To put the concept into practical use, the analysis aims at operable conditions for compatibility and determinacy. We consider two cases separately (still unifying continuous and discrete cases), as they correspond to different types of generative models, and lead to qualitatively different conclusions.

heading: 2.1 Absolutely Continuous Case
text: We first consider the case where for any z ∈ Z and any x ∈ X,4 the conditionals µ(·|z) and ν(·|x) are either absolutely continuous (w.r.t ξ and ζ, resp.) [13, p.448], or zero in the sense of a measure. Equivalently, they have density functions p(x|z) and q(z|x) (non-negative by definition; may integrate to zero). This case include “smooth” distributions on Euclidean spaces or manifolds, and all distributions on finite/discrete spaces. Many generative modeling frameworks use density models thus count for this case, including VAEs [45, 67, 66, 46, 80] and diffusion-based models [74, 38, 76].

heading: 2.1.1 Compatibility criterion in the absolutely continuous case
text: One may expect that when absolutely continuous conditionals p(x|z) and q(z|x) are compatible, their joint is also absolutely continuous (w.r.t ξ⊗ζ) with some density p(x, z). This intuition is verified by our Lem. C.1 in Appx. C.1. One could then safely apply density function formulae and get p(x|z)q(z|x) = p(x,z) p(z) / p(x,z) p(x) = p(x) p(z) factorizes into a function of x and a function of z. Conversely, if the ratio factorizes as such p(x|z)q(z|x) = a(x)b(z), one could get p(x|z) 1 Ab(z) = q(z|x) a(x) A where
3The symbol Z overwrites the symbol for the set of integers, which is not involved in this paper. 4There may be problems if absolute continuity holds only for ζ-a.e. z and ξ-a.e. x; see Appx. Example C.2.
A := ∫ X a(x)ξ(dx), which defines a joint density and compatibility is achieved. This intuition leads to the classical compatibility criterion [1; 4, Thm. 4.1; 5, Thm. 1; 6, Thm. 2.2.1].
However, the problem is more complicated than imagined. Berti et al. [11, Example 9] point out that the classical criterion is unfortunately not necessary. The subtlety is about on which region does this factorization have to hold. The classical criterion requires it to be the positive region of p(x|z) which also needs to coincide with that of q(z|x). But as mentioned, conditional µ(·|z) can be arbitrary outside the support of the marginal πZ (similarly for ν(·|x)), which may lead to additional positive regions that violate the requirement.5 To address the problem, Berti et al. [11] give an equivalence criterion (Thm. 8), but it is existential thus less useful as the definition of compatibility itself is existential. Moreover, these criteria are restricted to either Euclidean or discrete spaces.
Next we give our equivalence criterion that is operable. In addressing the subtlety with regions, we first introduce a related concept that helps identify appropriate regions.
Definition 2.2 (ξ⊗ζ-complete component). For a setW ∈X⊗Z , we say that a set S ∈X ⊗Z is a ξ⊗ζ-complete component ofW , if S] ∩W ξ⊗ζ= S , where S] := SX×Z∪X×SZ is the stretch of S .
Fig. 1 illustrates the concept. Roughly, the stretch S] of S represents the region where the conditionals are a.s. determined if S is the support6 of the joint. If S is a complete component ofW , it is complete under stretching and intersecting withW . Such a set S is an a.s. subset ofW (Lem. B.12), while has a.s. the same slice asW does for almost all z ∈ SZ and x ∈ SX (Lem. B.16). This is critical for the normalizedness of distributions in our criterion. Appx. B.3 shows more facts. With this concept, our compatibility criterion is presented below.
Theorem 2.3 (compatibility criterion, absolutely continuous). Let p(x|z) and q(z|x) be the density functions of two everywhere absolutely continuous (or zero) conditional distributions, and define: Pz := {x | p(x|z) > 0},Px := {z | p(x|z) > 0}, Qz := {x | q(z|x) > 0},Qx := {z | q(z|x) > 0}. Then they are compatible, if and only if they have a complete support S , defined as a (i) ξ⊗ζ-complete component of both
Wp,q := ⋃
z:Pz⊆ξQz
Pz×{z}, Wq,p := ⋃
x:Qx⊆ζPx
{x}×Qx,
such that: (ii) SX ⊆ξ WXq,p, SZ ⊆ζ WZp,q , (iii) (ξ⊗ ζ)(S) > 0, and (iv) p(x|z)q(z|x) factorizes as a(x)b(z), ξ⊗ζ-a.e. on S,7 where (v) a(x) is ξ-integrable on SX. For sufficiency,
π(W) := ∫ W∩S q(z|x)|a(x)|(ξ⊗ζ)(dxdz)∫
SX |a(x)|ξ(dx) , (1)
∀W ∈X ⊗Z , is a compatible joint of them.
Fig. 2 shows an illustration of the conditions. To understand the criterion, conditions (iv) and (v) stem from the starting inspiration, which also shows a hint for Eq. (1). Other conditions handle the subtlety to find a region S where (iv) and (v) must hold. This is essentially the support of a compatible joint π as there is no need and no way to control conditionals outside the support.
5The flexibility of p(x|z) on a ξ-measure-zero set for a given z (similarly for q(z|x)) is not a vital problem, as one can adjust the conditions to hold only a.e.
6While the typical definition of support requires a topological structure which is absent under our generality, Def. B.8 in Appx. B.1 defines such a concept for absolutely continuous distributions.
7Formally, there exist functions a on SX and b on SZ s.t. (ξ⊗ζ){(x, z) ∈ S | p(x|z) q(z|x) 6= a(x)b(z)} = 0.
For necessity, informally, if z is in the support of πZ, then p(x|z) determines the distribution on X×{z}; particularly, the joint π should be a.e. positive on Pz , which in turn asks q(z|x) to be so. This means Pz ⊆ξ Qz (unnecessary equal, since q(z|x) is “out of control” outside the joint support), which leads to the definitions ofWp,q andWq,p. The joint support should be contained within the two sets in order to avoid support conflict (e.g., although the bottom-left quadrant in Fig. 2 is part of the intersection of positive regions of the conditionals, a joint on it is required by p(x|z) to also cover the top-left, on which q(z|x) does not agree). Condition (i) indicates S ⊆ξ⊗ζ Wp,q and Wq,p so S satisfies this requirement and also makes the ratio in (iv) a.e. well-defined. The complete-component condition in (i) also makes the conditionals normalized on S: as mentioned, such an S has a.s. the same slice asWp,q does for a given z in support SZ, so the integral of p(x|z) on Sz is the same as that on (Wp,q)z = Pz which is 1 by construction; similarly for q(z|x). In contrast, Appx. Example C.3 shows S = Wp,q ∩Wq,p is inappropriate. Conditions (ii) and (iii) cannot be guaranteed by condition (i) (Appx. Example B.13), while are needed to rule out special cases (Appx. Lem. B.14, Example B.15). Appx. C.2 gives a formal proof. Finally, although the criterion relies on the existence of such a complete support, candidates are few (if any), so it is operable.

heading: 2.1.2 Determinacy in the absolutely continuous case
text: When compatible, absolutely continuous cyclic conditionals are very likely to have determinacy. Theorem 2.4 (determinacy, absolutely continuous). Let p(x|z) and q(z|x) be two compatible conditional densities, and S be a complete support that makes them compatible (necessarily exists due to Thm. 2.3). Suppose that Sz ξ = SX, for ζ-a.e. z on SZ, or Sx ζ = SZ, for ξ-a.e. x on SX. Then their compatible joint supported on S is unique, which is given by Eq. (1).
Proof is given in Appx. C.4. The condition in the theorem roughly means that the complete support S is “rectangular”. From the perspective of Markov chain, this corresponds to the irreducibility of the Gibbs chain for the unique existence of a stationary distribution. When the conditionals have multiple such complete supports, on each of which the compatible joint is unique, while globally on X×Z, they may have multiple compatible joints. In general, determinacy in the absolutely continuous case is sufficient, particularly we have the following strong conclusion in a common case (e.g., for VAEs). Corollary 2.5. We call two conditional densities have a.e.-full supports, if p(x|z) > 0, q(z|x) > 0 for ξ⊗ζ-a.e. (x, z). If they are compatible, then their compatible joint is unique, since X×Z is the ξ⊗ζ-unique complete support (Prop. C.4 in Appx. C.3), which satisfies the condition in Thm. 2.4.

heading: 2.2 Dirac Case
text: Many other prevailing generative models, including generative adversarial networks (GANs) [30] and flow-based models [24, 60, 44, 31], use a deterministic function x = f(z) as the likelihood model. In such cases, the conditional µ(X|z) = δf(z)(X ) := I[f(z) ∈ X ],∀X ∈ X is a Dirac measure. Note it does not have a density function when ξ assigns zero to all single-point sets, e.g. the Lebesgue measure on Euclidean spaces, so we keep the measure notion. This case is not exclusive to the absolutely continuous case: a Dirac conditional on a discrete space is also absolutely continuous.
2.2.1 Compatibility criterion in the Dirac case
Compatibility criterion is easier to imagine in this case. As illustrated in Fig. 3, it is roughly that the other-way conditional ν(·|x) could find a way to put its mass only on the curve; otherwise support conflict is rendered. Theorem 2.6 (compatibility criterion, Dirac). Suppose that X contains all the single-point sets: {x} ∈ X ,∀x ∈ X. Conditional distribution ν(Z|x) is compatible with µ(X|z) := δf(z)(X ) where function f : Z → X is X /Z -measurable8, if and only if there exists x0 ∈ X such that ν(f−1({x0})|x0) = 1.
See Appx. C.6 for proof. Note such x0 must be in the image set f(Z), otherwise ν(f−1({x0})|x0) = ν(∅|x0) = 0. For a typical GAN generator, the preimage set f−1({x0}) is discrete, so a compatible inference model must not be absolutely continuous. What may be counter-intuitive is that ν(·|x) is not required to concentrate on the curve for any x; one x0 is sufficient as δ(x0,f(x0)) is a compatible joint.
8For meaningful discussion, we require f to be X /Z -measurable, which includes any function between discrete sets and continuous functions when X and Z are the Borel sigma-fields.
Nevertheless, in practice one often desires the compatibility to hold over a set X to make a useful model. When ν(·|x) is also chosen in the Dirac form δg(x), this can be achieved by minimizing Ep(x)` ( x, f(g(x)) ) , where p(x) is a distribution on X and ` is a metric on X. This is the cycleconsistency loss used in dual learning [42, 90, 88, 53]. When f is invertible, minimizing the loss (i.e., g = f−1 a.e. on X ) is also necessary, as f−1(x) only has one element. Particularly, flow-based models are naturally compatible (so are their injective variants [3, 86, 16] by Thm. 2.6).

heading: 2.2.2 Determinacy in the Dirac case
text: As mentioned, for any x0 satisfying the condition, two compatible conditionals have the determinacy on this point {x0} with the unique joint δ(x0,f(x0)). But when such x0 is not unique, the distribution over these x0 values is not determined, so the two conditionals do not have the determinacy globally on X×Z. This is similar to the absolutely continuous case with multiple complete supports; particularly, each {(x0, f(x0))} is a complete support for discrete X and Z. This meets one’s intuition: compatible Dirac conditionals can only determine a curve in X×Z, but cannot determine a distribution on the curve. One exception is when f(z) ≡ x0 is constant, so this x0 is the only candidate. The joint then degenerates to a distribution on Z, which is fully determined by ν(·|x0). In general, determinacy in the Dirac case is insufficient, and this type of generative models (GANs, flow-based models) have to specify a prior to define a joint.

heading: 3 Generative Modeling using Cyclic Conditionals
text: The theory suggests it is possible that cyclic conditionals achieve compatibility and a sufficient determinacy, so that they can determine a useful joint without specifying a prior. Note a certain prior is implicitly determined by the conditionals; we find we just do not need an explicit model for it. This inspires CyGen, a novel framework that only uses Cyclic conditionals for Generative modeling. For the eligibility as a generative model, compatibility and a sufficient determinacy are required. For the latter, we just shown a deterministic likelihood or inference model is not suitable, so we use absolutely continuous conditionals as the theory suggests. The conditionals can then be modeled by parameterized densities pθ(x|z), qφ(z|x). We consider the common case where X = RdX , Z = RdZ , and pθ(x|z), qφ(z|x) have a.e.-full supports and are differentiable. Determinacy is then exactly guaranteed by Cor. 2.5. For compatibility, we develop an effective loss in Sec. 3.1 to enforce it.
For the usage as a generative model, we develop methods to fit the model-determined data distribution pθ,φ(x) to the true data distribution p∗(x) in Sec. 3.2, and to generate data from pθ,φ(x) in Sec. 3.3.

heading: 3.1 Enforcing Compatibility
text: In this a.e.-full support case, the entire product space X×Z is the only possible complete support (Prop. C.4 in Appx. C.3), so for compatibility, condition (iv) in Thm. 2.3 is the most critical one. For this, we do not have to find functions a(x), b(z) in Thm. 2.3, but only need to enforce such a factorization. So we propose the following loss function to enforce compatibility:
(min θ,φ
) C(θ, φ) := Eρ(x,z) ∥∥∇x∇>z rθ,φ(x, z)∥∥2F ,where rθ,φ(x, z) := log (pθ(x|z)/qφ(z|x)). (2)
Here, ρ is some absolutely continuous reference distribution on X×Z, which can be taken as p∗(x)qφ(z|x) in practice as it gives samples to estimate the expectation. When C(θ, φ) = 0, we have ∇x∇>z rθ,φ(x, z) = 0, ξ⊗ζ-a.e. [13, Thm. 15.2(ii)]. By integration, this means∇zrθ,φ(x, z) = V (z) hence rθ,φ(x, z) = v(z) + u(x), ξ⊗ζ-a.e., for some functions V (z), v(z), u(x) s.t. V (z) = ∇v(z). So the ratio pθ(x|z)/qφ(z|x) = exp{rθ,φ(x, z)} = exp{u(x)} exp{v(z)} factorizes, ξ⊗ζ-a.s. In the sense of enforcing compatibility, this loss generalizes the cycle-consistency loss to probabilistic conditionals. Also, the loss is different from the Jacobian-norm regularizers in contractive AE [68] and DAE [68, 2], and explains the “tied weights” trick for AEs [63, 82, 81, 68, 2] (see Appx. D.1).
Implication on Gaussian VAE which uses additive Gaussian conditional models, pθ(x|z) := N (x|fθ(z), σ2dIdX) and qφ(z|x) := N (z|gφ(x), σ2eIdZ). It is the vanilla and the most common form of VAE [45]. As its ELBO objective drives qφ(z|x) to meet the joint p(z)pθ(x|z), compatibility is enforced. Under our view, this amounts to minimizing the compatibility loss Eq. (2), which then enforces the match of Jacobians: (∇zf>θ (z))> = (σ2d/σ2e)∇xg>φ (x). As the two sides indicate the equation is constant of both x and z, it must be a constant, so fθ(z) and gφ(x) must be affine, and the joint is also a Gaussian [12; 6, Thm. 3.3.1]. This conclusion coincides with the theory on additive noise models in causality [89, 62], and explains the empirical observation that the latent space of such
VAEs is quite linear [73]. It is also the root of recent analyses that the latent space coordinates the data manifold [21], and the inference model learns an isometric embedding after a proper rescaling [58].
This finding reveals that the expectation to use deep neural networks for learning a flexible nonlinear representation will be disappointed in Gaussian VAE. So we use a non-additive-Gaussian model, e.g. a flow-based model [66, 46, 80, 31], for at least one of pθ(x|z) and qφ(z|x) (often the latter). Efficient implementation. Direct Jacobian evaluation for Eq. (2) is of complexity O(dXdZ), which is often prohibitively large. We thus propose a stochastic but unbiased and much cheaper method based on Hutchinson’s trace estimator [40]: tr(A) = Ep(η)[η>Aη], where η is any random vector with zero mean and identity covariance (e.g., a standard Gaussian). As the function within expectation is
∥∥∇x∇>z r∥∥2F = ∥∥∇z∇>x r∥∥2F = tr ((∇z∇>x r)>∇z∇>x r), applying the estimator yields a formulation that reduces gradient evaluation complexity to O(dX + dZ):
(min θ,φ
) C(θ, φ) = Eρ(x,z)Ep(ηx) ∥∥∇z(η>x ∇xrθ,φ(x, z))∥∥22,where E[ηx] = 0,Var[ηx] = IdX . (3)
As concluded from the above analysis on Gaussian VAE, we use a flow-based model for the inference model qφ(z|x). But in common instances evaluating the inverse of the flow is intractable [66, 46, 80] or costly [31]. This however, disables the use of automatic differentiation tools for estimating the gradients in the compatibility loss. Appx. D.2 explains this problem in detail and shows our solution.

heading: 3.2 Fitting Data
text: After achieving compatibility, Cor. 2.5 guarantees the a.e.-fully supported conditional models uniquely determine a joint, hence a data distribution pθ,φ(x). To fit pθ,φ(x) to the true data distribution p∗(x), an explicit expression is required. For this, Eq. (1) is not helpful as we do not have explicit expressions of a(x), b(z). But when compatibility is given, we can safely use density function formulae:
pθ,φ(x) = 1/ 1
pθ,φ(x) = 1/ ∫ Z pθ,φ(z ′) pθ,φ(x) ζ(dz′) = 1/ ∫ Z qφ(z ′|x) pθ(x|z′)ζ(dz ′) = 1/Eqφ(z′|x)[1/pθ(x|z′)],
which is an explicit expression in terms of the two conditionals. Although other expressions are possible, this one has a simple form, and the Monte-Carlo expectation estimation in Z has a lower variance than in X since usually dZ dX. We can thus fit data by maximum likelihood estimation:
(min θ,φ
) Ep∗(x)[− log pθ,φ(x)] = Ep∗(x)[logEqφ(z′|x)[1/pθ(x|z ′)]]. (4)
The loss function can be estimated using the reparameterization trick [45] to reduce variance, and the logsumexp trick is adopted for numerical stability. This expression can also serve for data likelihood evaluation. The final training process of CyGen is the joint optimization with the compatibility loss.
Comparison with DAE. We note that the DAE loss [82, 9] Ep∗(x)qφ(z′|x)[− log pθ(x|z′)] is a lower bound of Eq. (4) due to Jensen’s inequality, so it is not suitable for maximizing likelihood. In fact, the DAE loss minimizes Eqφ(z)KL(qφ(x|z)‖pθ(x|z)) for pθ(x|z) to match qφ(x|z), where qφ(z) and qφ(x|z) are induced from the joint p∗(x)qφ(z|x), but it is not a proper loss for qφ(z|x) as a mode-collapse behavior is promoted: the optimal qφ(z|x) only concentrates on the point(s) of argminz′ pθ(x|z′), and an additional entropy term −Eqφ(z)H[qφ(x|z)] is required to optimize the same KL loss. This behavior hurts determinacy, as qφ(z|x) tends to be a (mixture of) Dirac measure (Sec. 2.2.2). The resulting Gibbs chain may also converge differently depending on initialization, as ergodicity is broken. This behavior also hurts compatibility, as qφ(x|z) deviates from pθ(x|z) (not Dirac), and does not match the Gibbs stationary distribution [35, 9]. In contrast, CyGen follows a more fundamental logic: enforce compatibility explicitly and follow the maximum likelihood principle faithfully. It leads to a proper loss for both conditionals that does not hinder determinacy.

heading: 3.3 Data Generation
text: Generating samples from the learned data distribution pθ,φ(x) is not as straightforward as typical models that specify a prior, since ancestral sampling is not available. But it is still tractable via Markov chain Monte Carlo methods (MCMCs). We propose using dynamics-based MCMCs, which are often more efficient than Gibbs sampling (used in DAE [9] and GibbsNet [50]). They only require an unnormalized density function of the target distribution, which is readily available in CyGen when compatible: pθ,φ(x) =
pθ,φ(x) pθ,φ(z) pθ,φ(z) = pθ(x|z) qφ(z|x)pθ,φ(z) ∝ pθ(x|z) qφ(z|x) for any z ∈ Z. In practice, this z
can be taken as a sample from qφ(z|x) to lie in a high probability region for a confident estimate. Stochastic gradient Langevin dynamics (SGLD) [83] is a representative instance, which has been shown to produce complicated realistic samples in energy-based [26], score-based [75] and diffusion-
Figure 4: Generated data (DAE and CyGen use SGLD) and class-wise aggregated posteriors of DAE, VAE, BiGAN and CyGen. Also shows results of CyGen(PT) that is PreTrained as a VAE. (Best view in color.)
data DAE VAE BiGAN CyGen CyGen(PT)
class-wise aggregated
posterior
Figure 5: Generated data along the training process of CyGen after VAE pretraining (iteration 1000), using SGLD (rows 1,3) and Gibbs sampling (row 2) for generation, and with (rows 1,2) and without the compatibility loss (row 3) for training. See also Appx. Fig. 13.
iteration 1100 1200 1300 1400 30000
CyGen(PT), using SGLD
CyGen(PT), using Gibbs
compt. loss 7.0×103 5.4×103 7.7×103 6.2×103 4.6×103
CyGen(PT) w/o compt., using SGLD
compt. loss 1.1×105 1.6×105 2.6×105 8.6×105 1.2×108
based [38, 76] models. It gives the following transition:
x(t+1) = x(t) + ε∇x(t) log pθ(x (t)|z(t)) qφ(z(t)|x(t))
+ √
2ε η (t) x ,where z(t) ∼ qφ(z|x(t)), η(t)x ∼ N (0, IdX), (5)
and ε is a step size parameter. Method to draw z ∼ pθ,φ(z) can be developed symmetrically (see Appx. Eq. (25)). Also applicable are other dynamics-based MCMCs [19, 23, 57], and particle-based variational inference methods [56, 17, 54, 55, 77] which are more sample-efficient.

heading: 4 Experiments
text: We demonstrate the power of CyGen for data generation and representation learning. Baselines include DAE, and generative models using Gaussian prior e.g. VAE and BiGAN (Appx. E.1). For a fair comparison, all methods use the same architecture, which is an additive Gaussian pθ(x|z) and a Sylvester flow (Householder version) [80] for qφ(z|x) (Appx. E.2), as required by CyGen (Sec. 3.1). It is necessarily probabilistic for determinacy, so we exclude flow-based generative models and common BiGAN/GibbsNet architectures, which are deterministic. We also considered GibbsNet [50] which also aims at the prior issue, but it does not produce reasonable results using the same architecture, due to its unstable training process (see Appx. E.1). Codes: https://github.com/changliu00/cygen.

heading: 4.1 Synthetic Experiments
text: For visual verification of the claims, we first consider a 2D toy dataset (Fig. 4 top-left). Appx. E.3 shows more details and results, including the investigation on another similar dataset.
Data generation. The learned data distributions (as the histogram of generated data samples) are shown in Fig. 4 (row 1). We see the five clusters are blurred to overlap in VAE’s distribution and are still connected in BiGAN’s, due to the specified prior. In contrast, our CyGen fits this distribution much better; particularly it clearly separates the five non-connected clusters. This verifies the advantage to overcome the manifold mismatch problem. As for DAE, it cannot capture the data distribution due to collapsed inference model and insufficient determinacy (Sec. 3.2).
Representation. Class-wise aggregated posteriors (as the scatter plot of z samples from qφ(z|x)p∗(x|y) for each class/cluster y) in Fig. 4 (row 2) show that CyGen mitigates the poste-
rior collapse problem, as the learned inference model qφ(z|x) better separates the classes with a margin in the latent space. This more informative and representative feature would benefit downstream tasks like classification or clustering in the latent space. In contrast, the specified Gaussian prior squeezes the VAE latent clusters to touch, and the BiGAN latent clusters even to mix. The mode-collapsed inference model of DAE locates all latent clusters in the same place.
Incorporating knowledge into conditionals. CyGen alone (without pretraining) already performs well. When knowledge is available, we can further incorporate it into the conditional models. Fig. 4 shows pretraining CyGen’s likelihood model as in a VAE (CyGen(PT)) embodies VAE’s knowledge that the prior is centered and centrosymmetric, as the (all-class) aggregated posterior (≈ prior) is such. Note its data generation quality is not sacrificed. Appx. Fig. 14 verifies this directly via the priors.
Comparison of data generation methods. We then make more analysis on CyGen. Fig. 5 (rows 1,2) shows generated data of CyGen using SGLD and Gibbs sampling. We see SGLD better recovers the true distribution, and is more robust to slight incompatibility.
Impact of the compatibility loss. Fig. 5 (rows 1,3) also shows the comparison with training CyGen without the compatibility loss. We see the compatibility is then indeed out of control, which invalidates the likelihood estimation Eq. (4) for fitting data and the gradient estimation in Eq. (5) for data generation, leading to the failure in row 3. Along the training process of the normal CyGen, we also find a smaller compatibility loss makes better generation (esp. using Gibbs sampling).
4.2 Real-World Experiments DAE VAE CyGen(PT)
Figure 6: Generated data on the MNIST and SVHN datasets.
We test the performance of CyGen on real-world image datasets MNIST and SVHN. We consider the VAE-pretrained version, CyGen(PT), for more stable training. Appx. E.4 shows more details. On these datasets, even BiGAN cannot produce reasonable results using the same architecture, similar to GibbsNet.
Data generation. From Fig. 6, We see that CyGen(PT) generates both sharp and diverse samples, as a sign to mitigate manifold mismatch. DAE samples are mostly imperceptible, due to the mode-collapsed qφ(z|x) and the
subsequent lack of determinacy (Sec. 3.2). Table 7: Downstream classification accuracy (%) using
learned representation by various models. †: Results from [50] using a different, deterministic architecture (not suitable for CyGen).
DAE VAE BiGAN†GibbsNet† CyGen(PT)
MNIST 98.0±0.1 94.5±0.3 91.0 97.7 98.3±0.1 SVHN 74.5±1.0 30.8±0.2 66.7 79.6 75.8±0.5
VAE samples are a little blurry as a typical behavior due to the simply-connected prior. This observation is also quantitatively supported by the FID score [36, 72] on SVHN: CyGen achieves 102, while DAE 157 and VAE 128 (lower is better).
Representation. We then show in Table 7 that CyGen(PT)’s latent representation is more informative for the downstream classification task, as an indicator to avoid posterior collapse. BiGAN and GibbsNet make random guess using the same probabilistic flow architecture, and their reported results in [50] using a different, deterministic architecture (not suitable for CyGen due to insufficient determinacy) are still not always better, due to the prior constraint. We conclude that CyGen achieves both superior generation and representation learning performance.

heading: 5 Conclusions and Discussions
text: In this work we investigate the possibility of defining a joint distribution using two conditional distributions, under the motivation for generative modeling without an explicit prior. We develop a systematic theory with novel and operable equivalence criteria for compatibility and sufficient conditions for determinacy, and propose a novel generative modeling framework CyGen that only
uses cyclic conditional models. Methods for achieving compatibility and determinacy, fitting data and data generation are developed. Experiments show the benefits of CyGen over DAE and prevailing generative models that specify a prior in overcoming manifold mismatch and posterior collapse.
The novel CyGen framework broadens the starting point to build a generative model, and the general theory could also foster a deeper understanding of other machine learning paradigms, e.g., dual learning and self-supervised learning, and inspire more efficient algorithms.

Review summaries:
Notes:    -Model the joint p(x,z) by modeling p(x|z) and q(z|x) to form a cycle.     -Many existing techniques use an uninformative prior p(z).     -Compatibility and determinacy criterions.     -Use parameterized density models p(x|z) and q(z|x).     -Define r(x,z) = log(p(x|z) / q(z|x)) and make the loss the norm of the jacobian, computed over samples from p(x)*q(z|x) (real data / inferred latents).      -Jacobian is intractable, so instead approximate by taking gradient wrt. x, then multiplying by random vector eta, then taking gradient wrt z.     -Sample using stochastic gradient langevin dynamics.  

The work answers an interesting question: Given two cyclic conditional distributions, under which condition there exists a compatible joint distribution, and when the compatible joint distribution is unique.   From the theory the authors propose a generative framework called CyGen. CyGen is trained using maximum likelihood based on modelings of the conditional probabilities. Sampling from the marginal distribution modeled by the two conditional distributions is also not straightforward. Luckily this has been investigated before and there are practical MCMC based procedures that converge to the marginal distribution.  The authors also show some experimental results that the proposed CyGen is able to generate more realistic samples compared to VAE or DAE.