On the Generative Utility of Cyclic Conditionals
Authors: Chang Liu
Haoyue Tang
Tao Qin
Jintao Wang
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative model
variational auto-encoder
probability theory
conditionally specified distributions

Abstract: We study whether and how can we model a joint distribution $p(x,z)$ using two conditional models $p(x|z)$ and $q(z|x)$ that form a cycle. This is motivated by the observation that deep generative models, in addition to a likelihood model $p(x|z)$, often also use an inference model $q(z|x)$ for extracting representation, but they rely on a usually uninformative prior distribution $p(z)$ to define a joint distribution, which may render problems like posterior collapse and manifold mismatch. To explore the possibility to model a joint distribution using only $p(x|z)$ and $q(z|x)$, we study their compatibility and determinacy, corresponding to the existence and uniqueness of a joint distribution whose conditional distributions coincide with them. We develop a general theory for operable equivalence criteria for compatibility, and sufficient conditions for determinacy. Based on the theory, we propose a novel generative modeling framework CyGen that only uses the two cyclic conditional models. We develop methods to achieve compatibility and determinacy, and to use the conditional models to fit and generate data. With the prior constraint removed, CyGen better fits data and captures more representative features, supported by both synthetic and real-world experiments.

On the Generative Utility of Cyclic Conditionals
Chang ; Tang"! Tao Qin', Jintao Wang", Tie-Yan Liu' Microsoft Research Asia, Beijing, 100080. Tsinghua University, Beijing, 100084.
Abstract
We study whether and how conditional models p(æ[z) and (2(2z) that form a cycle. This is motivated by the observation that deep generative models, in addition to likelihood model p(z]z), often also use an inference model g(=[x) for extracting representation, but they rely usually uninformative prior distribution p(z) to define joint distribution, which may render problems like posterior collapse and manifold mismatch. To explore the possibility to model joint distribution using only p(x[2z) and (=[[x), we study their compatibility and determinacy, corresponding to the existence and uniqueness of joint distribution whose conditional distributions coincide with them. We develop a general theory for operable equivalence criteria for compatibility, and sufficient conditions for determinacy. Based on the theory, we propose a novel generative modeling framework CyGen that only uses the two cyclic conditional models. We develop methods achieve compatibility and determinacy, and to use the conditional models to fit and generate data. With the prior constraint removed, CyGen better fits data and captures more representative features, supported by both synthetic and real-world experiments.
model joint distribution p(x, 2) using two
1 Introduction
Deep generative models have achieved remarkable success in the past decade for generating realistic complex data x and extracting useful representations through their latent variable z. Variational
80 follow the Bayesian framework and specify prior distribution p(z) and likelihood model p(r[z), so that joint distribution p(x, 2) = p==)p(<x>> is defined for generative modeling (the joint induces a distribution p(z) on data). An inference model (==(x)) is also used approximate the posterior distribution p(2[ær) (derived from the joint p(x, =)), which serves for extracting representations. Other frameworks like generative adversarial
auto-encoders (VAEs) [5 67
14
15
46
nets 30 25 (27) flow-based models (24 60
44 BT] and diffusion- models [74 38 76 49)
follow the same structure, with different choices of the conditional models p(x|2) and g(2 x)) and training objectives. While for the prior p(2), there is often not much knowledge for complex data (like images, text, audio), and these models widely adopt uninformative prior such as a standard Gaussian. This however, introduces some side effects:
Posterior collapse [15] 34 [44] The standard Gaussian prior tends squeeze g(=[æz) towards the origin for all x, which degrades the representativeness of the inferred z for x and hurts downstream tasks in the latent space like classification and clustering.
Typically the likelihood model continuous (keeps topology), the standard Gaussian prior would restrict the modeled data distribution to simply-connedn support, which limits the capacity for fitting data from a non-(simply) connected support.
Manifold mismatch 22 28
While there are works trying mitigate the two problems, they require either strong domain
knowledge [48 41,, or additional cost to learn a complicated prior model [2 21 79] sometimes
even the cost of inconvenient inference [59 87].
"Correspondence to: Chang Work done during internship at Microsoft Research Asia.
com>.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

One question then naturally emerges: Can we model a joint distribution p(z, 2) only using the likelihood p(z|z) and inference ((((=) models? If we can, the limitations from specifying or learning prior are then removed from the root. Also, the inference model g(= (r)) is then no longer struggling approximation to predefined posterior but participates in defining the joint distribution (avoid "inner approximation") Modeling conditionals is also argued to be much easier than modeling marginal or joint distributions directly 12 I0]. In many cases, one may even have better knowledge on the conditionals than on the prior, e.g. shift/rotation invariance of image representations (CNNs [5]] / SphereNet 220), and rules to extract frequency/energy features for audio [65. is then more natural and effective to incorporate this knowledge into the conditionals than using uninformative prior.
In this paper, we explore such possibility, and develop both a systematic theory and a novel generative modeling framework CyGen (Cyclie-condtional Generative model).
(1) Theoretical analysis on the question amounts to two sub-problems, can two given cyclic condi- tionals correspond a common joint, and if yes, can they determine the joint uniquely. We term them compatibility and determinacy of two conditionals, corresponding to the existence and uniqueness of a common joint. For this, we develop novel compatibility criteria and sufficient conditions for determinacy. Beyond existing results, ours are operable (vs. existential [II] and self-contained (vs. need a marginal [9] TO, 50 32), and are general enough to cover both continuous and discrete cases. Our compatibility criteria are also equivalence (vs. unnecessary [ (6) conditions. The seminal book [6] makes extensive analysis for various parametric families. Besides the equivalence criteria, we also extend their general analysis beyond the product support case, and also cover the Dirac case.
(2) In addition to its independent contribution, the theory also enables generative modeling using only the two cyclic conditional models, i.e. the CyGen framework. We develop methods for achieving compatibility and determinacy to make an eligible generative model, and for fitting and generating data to serve as a generative model. Efficient implementation techniques are designed. Note CyGen also determines prior implicitly; it just does not need an explicit model for the prior (vs. 52
21 79 59 D. We show the practical utility of CyGen both synthetic and real-world tasks. The improved performance in downstream classification and data generation demonstrates the advantage to mitigate the posterior collapse and manifold mismatch problems.
1.1 Related work Dependency networks 35 similarly (39] are perhaps the first to pursue the idea of modeling a joint by set of conditionals. They use Gibbs sampling to determine the joint and are equivalent to undirected graphical models. They do not allow latent variables, so compatibility is not a key consideration as the data already specifies a joint as the common target of the conditionals. Beyond that, we introduce latent variables to better handle sensory data like images, for which we analyze the conditions for compatibility and determinacy and design novel methods to solve this different task.
Denoising auto-encoders (DAFs). AES [71 7 aim to extract data features by enforcing reconstruc- tion through its encoder and decoder, which are deterministic hence insufficient determinacy (see 10 use probabilistic encoder and decoder for robust reconstruction against random data corruption. Their utility as a generative model is first noted through the equivalence to score matching (implies modeling p(x)) for a Gaussian RBM [8]] or an infinitesimal Gaussian corruption (2). In more general cases, the utility to modeling the joint p(x, 2) is studied via the Gibbs chain, i.e. the Markov chain with transition kernel p(x') (=))g((((2>))) Under global [9] 50 32 or local [0] shared support condition, its stationary distribution x xxx, 2) exists uniquely. But this is not really determinacy: even incompatible conditionals can have this unique existence, in which case T (2(2x) # ((2(<)) 35.9). Moreover, the Gibbs chain does not give an explicit expression of (Ir, 2) (thus intractable likelihood evaluation), and requires many iterations to converge for data generation and even for training (Walkback [9], GibbsNet (50), making the algorithms costly and unstable.
Sec. 2.2.2). DAEs [2 9
As for compatibility, it is not really covered in DAES. Existing results only consider the statistical consistency (unbiasedness under infinite data) of the p(z|z) estimator by fitting (x., 2) data from p" (x)g (a)) [9 10 50, 32), where p'(x)) denotes the true data distribution. Particularly, they require marginal pz(x) in advance, so that the joint is already defined by p" (z)4(=(z) regardless of p(x)2), while compatibility (as well as determinacy) is a matter only of the two conditionals.
More crucially, the DAE loss is not proper for optimizing q(2 (x)) it promotes mode-collapse behavior. This hinders both compatibility and determinacy (Sec. 3.2c.3.2): one may not use g(>[x) for inference, and data generation may depend initialization. In contrast, CyGen explicitly enforces compatibility and guarantees determinacy, and enables likelihood evaluation and better generation.
2

---

Dual learning considers conversion between two modalities in both directions, e.g., machine translation (33
88 533. Although we also consider both directions, the fundamental distinction is that in generative modeling there is no data of the latent variable 2 (not even unpaired). Technically, they did not consider determinacy: they require a marginal to determine joint. We find their determinacy is actually insufficient (see sec.2.2.2). Their sycle-consistency loss [42] 90 [8) is a version of our compatibility criterion in the Dirac case (see Sec. 1.2.1). and we extend it to allow probabilistic conversion (see Sec. 3.Ib
85 84
and image style transfer 42
90
2 Compatibility and Determinacy Theory
To be generative model, a system needs determine distribution on the data variable x. With latent variable z, this amounts to determining a joint distribution over (æ, =), which calls for compatibility and determinacy analysis for cyclic conditionals. In this section we build a general theory on the conditions for compatibility and determinacy. We begin with formalizing the problems.
Setup. Denote the measure spaces of the two random variables x and as (X, 2 ( and (Z, 2, c where 2 are the respective sigma-fielde and the base measures ç. C (e.g.. Lebesgue measure on Euclidean spaces, counting measure on finite/discrete spaces) are aigma-finite. We use X E 2.
Z E 2 to denote measurable sets, and use "É", "C as the extensions of "_", "C" up to set of E-measure-zero (Def. A.I). Following the convention in machine learning, we call a "probability measure" as "distribution". We do not require any further structures such as topology, metric, or linearity, for the interest of the most general conclusions that unify Euclidean/manifol and finite/discrete spaces and allow X, Z to have different dimensions or types.
Joint and conditional distributions are defined on the product measure space (XxZ. 2. 69 ( where "x" is the usual Cartesian product, 2 (9 2 == o 2) is the sigma-field generated by measurable rectangles from 2' x 2, and ( is the product measure [13] Thm. 18.2]. Define the slice of W 2 2 as W₂ == {or (z, 2) E W) 2 [I] Thm. 18)] and its projection
onto Z as W = {= 3 E X s.t. (x, 2) WW E 2 (Appx. A.3). In a similar style, denote the marginal of a joint on Z as 2 (2) == 7: (Xx To keep the same level of generality, we follow the general definition of conditionals (133 p.457]: see also Appx. A..4p the conditional π (x|22) of a joint
is the density function (R-N derivative) of x (x)x)) w.r.t ~ We highlight the key characteristic under this generality that æ (z) can be arbitrary on a set of m=-measure-zero particularly, outside the support of π². Appx. A and B provide more background details and our technical preparations that are also of independent interest. The goal of analysis can then be formalized below.
Definition 2.1 (compatibility and determinacy). We say two conditionals (x('(z), v [ [z) are com- patible, if there exists a joint distribution π (X x E. 2' & 2)) such that ==x((=) and v(Z(z) are its conditional distributions. We say two compatible conditionals have determinacy on set S E 2 222 if there only one joint distribution concentrated on S that makes them compatible.
To put the concept into practical use, the analysis aims at operable conditions for compatibility and determinacy. We consider two cases separately (still unifying continuous and discrete cases), as they correspond to different types of generative models, and lead to qualitatively different conclusions.
2.1 A bsooutely Continuous Case
We first consider the case where for any z E E and any x E X the conditionals μd-1z) and v(-1z)) are either absolutely continuous (w.r. & and Ç, resp.) 13 p. 488, or zero in the sense of measure. Equivalently, they have density functions p(z|zz and (2(xx) (non-negative by definition; may integrate zero). This case include "smooth" distributions on Euclidean spaces or manifolds, and all distributions on finte/ddscreete spaces. Many generative modeling frameworks use density models
thus count for this case, including VAEs [45 67 66 46 80) and iiffusion-based models [74 38 76).
2.1.1 Compatibility criterion in the absolutely continuous case
One may expect that when absolutely continuous conditionals p(a[z) and q(2(2:) are compatible, their joint is also absolutely continuous (w.rt & with some density p(x, 2). This intuition is
verified by our Lem.
C.I in Appx. C.I One could then safely apply density function formulae and
get p(x1z) p(a.z)
p(..5) p(z)
p(2) factorizes into a function of x and function of 2. Conversely,
m(2(m) p(z)
p(z)
if the ratio factorizes as such p(z|=)
4(=[2) = axx)b(2), one could get p(æ|2) 1
Ab(²) = g(>|2= A where a(x)
"The symbol Z overwrites the symbol for the set of integers, which is not involved in this paper. 4 'There may be problems if absolute continuity holds only for Ç-a.e. and E-ac. x; see Appx. Example C.2
3

---

A:= A ((x))(dx)) which defines joint density and compatibility is achieved. This intuition leads to the classical compatibility criterion 4 Thm. 4.1: Thm. 1:6] Thm. 2.2.1].
However, the problem more complicated than imagined. Berti et al. [TT Example 9] point out that the classical criterion is unfortunately not necessary. The subtlety is about on which region does this factorization have to hold. The classical criterion requires it to be the positive region of p(z|z) which also needs to coincide with that of q(=|2r). But as mentioned, conditional su[-1=) can be arbitrary outside the support of the marginal T (similarly for v(-(z)), which may lead to additional positive regions that violate the requirement To address the problem, Berti et al. [T]] give an equivalence criterion (Thm. 8), but it is existential thus less useful as the definition of compatibility itself is existential. Moreover, these criteria are restricted to either Euclidean or discrete spaces.
Next we give our equivalence criterion that is operable. In addressing the subtlety with regions, we first introduce a related concept that helps identify appropriate regions.
Definition 2.2 (68(-compla) we say that a set S E 2 & 2' is a (CC-ompleete component of W. if S W < S, where S == 8 xZ Xx is the stretch of S.
component), For a set W E LøZ,
w
S
Fig. iilustrates the concept. Roughly, the stretch St of S repre- sents the region where the conditionals are a.s. determined if S is the suppor[of of the joint. If S is a complete component of W. it is complete under stretching and intersecting with W. Such a set is an a.s. subset of W (Lem. B.12). while has a.s. the same slice as W does for almost all E S and 8 (Lem. [6. This is critical for the normalizedness of distributions in our criterion. Appx. BB3 shows more facts. With this concept, our compatibility criterion is presented below.
5
8*
5
Figure 1: Illustration of a {
complete component S of W.
Theorem 2.3 (compatibility criterion, absolutely continuous). Let p(z|z) and g(zlxr) be the density functions of two everywhere absolutely continuous (or zero) conditional distributions, and define:
x
p(xle)
Pz
P. = {= p(x|=) > 0}, == (= p(u|=) 0}: O fx g()x)) > 0), 22 == {2 q(>[x)) > 0}-
gf
Wpa Wap s
Then they are compatible, if and only if they have a complete support S, defined as a (i) & &CCcoomleete component of both
21 Ø 1
22 5 W
Wp.:= : U P= WGP = U x x.P.
Illustration of our compatibility criterion in the absolutely continuous case (Thm. 2.3). The conditionals are uniform on the respective depicted slices. For condition (i), P. S Q2 is not satisfied on the left half, egg. #1, so Wp.g does not cover the left half; it is satisfied on the right half, e.g. 22, so Wp.g is composed of slices P:: on the right half, making the top-right quadrant (shaded). Similarly, W.. is the same region, and it is a E & (-complete component of itself. also satisfies other con- ditions thus is complete support S.
Figure 2:
2.P.
such that: (ii) S C Wx S c W.
W B.g (iii) (E0
gpp
()()) > O, and (iv)
p(at)) o(ala)
factorizes as alz)b(2),
{oC-uee on s Where (v) a(x) is E-integrable on
8x For sufficiency.
x(W)= Swns ((ela))a(t))(()
(1)
SSx ((d)) VW E 28 2r is a compatible joint of them.
Fig. 2 shows an illustration of the conditions. To understand the criterion, conditions (iv) and (v) stem from the starting inspiration, which also shows a hint for Eq. ( Other conditions handle the subtlety to find a region S where (iv) and (v) must hold. This is essentially the support of compatible joint as there is no need and no way to control conditionals outside the support.
'The flexibility of p(x|2) on a É-mcasure-zero set for given (similarly for q(=[z)) not a vital problem, as one can adjust the conditions to hold only a.c.
While the typical definition of support requires topological structure which is absent under our generality,
Def. B.8 in Appx. B.T defines such concept for absolutely continuous distributions.
Formally, there exist functions s and on S s.t. (( E E [(ا|=)
(()|)] + a(x))(=)) 0.
4

---

For necessity, informally, if 2 is in the support of x², then p(z[z) determines the distribution on X {=}: particularly, the joint т should be a.c. positive on Pz, which in turn asks q(2/2) to be so. This means P= C 2g (unnecessary equal, since g(=[z) is "out of control" outside the joint support), which leads to the definitions of Wp. and W..p. The joint support should be contained within the two sets in order to avoid support conflict (e.g. although the bottom-left quadrant in Fig. 2 is part of the intersection of positive regions of the conditionals, a joint on it is required by p(z|2) to also cover the top-left, on which g(= (x) does not agree). Condition (i) indicates S C Wp. and , S satisfies this requirement and also makes the ratio in (iv) a.e. well-defined. The complete-componet
condition in (i) also makes the conditionals normalized on S: as mentioned, such an S has a.s. the same slice as Wp.9 does for a given Z in support S so the integral of p(z|2z) on S₃ is the same as that on (Wp.g), = =P> which is by construction; similarly for (2|2z). In contrast, Appx. Example C.3 shows S Shoos guaranteed by condition (i) (Appx. Example B.13). while are needed to rule out special cases (Appx. Lem. 14 Example|...
Wp.q n Wg is inappropriate. Conditions (ii) and (iii) cannot be
Appx. C.2 gives a formal proof. Finally, although the criterion relies on the existence of such a complete support, candidates are few (if any), so it is operable.
2.1.2 Determinacy in the absolutely continuous case
When compatible, absolutely continuous cyclic conditionals are very likely to have determinacy.
Theorem 2.4 (determinacy, absolutely continuous), Let p(a|z) and q(2[x) be two compatible condi- tional densities, and S be a complete support that makes them compatible (necessarily exists due to Thm.. 2.3 Suppose that S. - S8", for Ç-a.e. z on S or S2 - S2. E-a.e. I on S% Then their compatible joint supported on S is unique, which is given by Eq. (T]
Proof is given in Appx. C.4 The condition in the theorem roughly means that the complete support S "rectangular". From the perspective of Markov chain, this corresponds to the irreducibility of the Gibbs chain for the unique existence of stationary distribution. When the conditionals have multiple such complete supports, on each of which the compatible joint is unique, while globally on X Z. they may have multiple compatible joints. In general, determinacy in the absolutely continuous case is sufficient, particularly we have the following strong conclusion in a common case (e.g., for VAEs).
Corollary 2.5. We call two conditional densities have a.c.-full supports, if p(z|2) > 0, g,((2(z) > 0 for && (xx 2). If they are compatible, then their compatible joint is unique, since X x is the É&C-unique complete support ( (Prop. C.4 in Appx. which satisfies the condition in Thm. 2.4
2.2 Dirac Case
Many other prevailing generative models, including generative adversarial networks (GANs) 30
and bosed models [24 60 44 31, use deterministic function x f(z) as the likelihood model. In such cases, the conditional =x((=)) = )x=)({)) := [=f(=) E XJ,VX' VX E 2 is a Dirac measure. Note it does not have a density function when & assigns zero to all single-point sets, e.g. the Lebesgue measure on Euclidean spaces, so we keep the measure notion. This case is not exclusive to the absolutely continuous case: a Dirac conditional on discrete space is also absolutely continuous.
2.2.1 Compatibility criterion in the Dirac case
Compatibility criterion is easier to imagine in this case. As illus- trated in Fig. 3 it is roughly that the other-way conditional v(- (x)) could find way to put its mass only on the curve; otherwise support conffict is rendered.
x f()
((x))
Theorem 2.6 (compatibility criterion. Dirac). Suppose that K contains all the single-poin point sets: {z} E 2,, Vx E X. Con- ditional distribution v v(Z(x)
compatible with 2(2(2)
(2) where function f Z- + X is 212 -measurable and only if there exists To X such that
R if
Figure 3: Illustration of our com- patibility criterion in the Dirac case (Thm. 2.6p.
See Appx. C.6 proof. Note such To must be in the image set (Z)), otherwise (())))))))(((()))))))) = (0) = O. For a typical GAN generator, the preimage set f ((xo)) is discrete, so compatible inference model must not be absolutely continuous. What may be counter-intuitive is that ((-1x:) is not required to concentrate on the curve for any x; one To is sufficient as )(20.f(20)) is a compatible joint.
For meaningful discussion, we require f to be 212 measurable, which includes any function between discrete sets and continuous functions when 2 and 2 are the Borel fields.
5

---

Nevertheless, in practice one often desires the compatibility to hold over a set A' to make a useful model. When v(-lz) is also chosen in the Dirac form 6g(z), this can be achieved by minimizing where p(z) is a distribution on X' and é is a metric on X. This is the cycle- consistency loss used in dual learning 42
Eptay f
88 531 When f is invertible, minimizing the loss (i.e. g = f a.e. on X¹) is also necessary, as f (x) only has one element. Particularly, flow-based models are naturally compatible (so are their injective variants
90.
3 86 [6] by Thm. 2.6
2.2.2 Determinacy in the Dirac case
As mentioned, for any To satisfying the condition, two compatible conditionals have the determinacy on this point { with the unique joint (zu))( But when such zi is not unique, the distribution over these To values not determined, so the two conditionals do not have the determinacy globally on E.Z. This is similar to the absolutely continuous case with multiple complete supports; particularly, each {(z0. { is a complete support for discrete X and ž. This meets one's intuition: compatible Dirac conditionals can only determine a curve in X xx 2. but cannot determine a distribution on the curve. One exception is when f(z) = Zi is constant, so this zy is the only candidate. The joint then degenerates to distribution on Σ. which is fully determined by (()o.)))
In general, determinacy in the Dirac case is insufficient, and this type of generative models (GANs, flow- how-based models) have to specify a prior to define a joint.
3 Generative Modeling using Cyclic Conditionals
The theory suggests it is possible that cyclic conditionals achieve compatibility and sufficient determinacy, so that they can determine a useful joint without specifying a prior. Note a certain prior is implicitly determined by the conditionals; we find we just do not need an explicit model for it. This inspires CyGen, a novel framework that only uses Cyclic conditionals for Generative modeling.
For the eligibility as a generative model, compatibility and a sufficient determinacy are required. For the latter, we just shown deterministic likelihood or inference model not suitable, so use absolutely continuous conditionals as the theory suggests. The conditionals can then be modeled by parameterized densities (x) go (z(zr). We consider the common case where X Rdx, Z Rdi and pe (x| 2). 4 (z) have a.cl-full supports and are differentiable. Determinacy is then exactly guaranteed by Cor. 2.5 For compatibility, we develop an effective loss in See. B. enforce
For the usage as a generative model, we develop methods to fit the model-determined data distribution pe (z) to the true data distribution p" (x) in Sec. 3.2 and to generate data from pe.o (2)) in Sec. 3.3
3.1 Enforeing Compatibility
In this a.e.-full support case, the entire product space Xx the only possible complete support (Prop. [C.4Jin in Appx. C.3). so for compatibility, condition (iv) in Thm. Thm..2.3]is is the most critical one. For this, we do not have to find functions a(x). b(=) in Thm. 2.3] but only need to enforce such a factorization. So we propose the following loss function to enforce compatibility:
(min) C(O, Ø) == Ep(..)) V. V. 2) }| where re. (z, 2) == log (po(=)2)/q0(()=))) (2) 0,0
Here, p is some absolutely continuous reference distribution on X x Z. which be taken as p" (z)))(2(z) in practice as it gives samples to estimate the expectation. When C(A, () = 0. we have 11 Thm. 15.)(ii). By integration, this means V Va x) (z)
V.V! ro.s (x, =) 0, (aC-a.e.
hence (2, 2) v(=) u(=). E (-a... for some functions V(z), v(z), u(x) s.t. V(2) Vo(z). So the ratio pa (=<=)=//oo==(>)
exp(ro,0(z, =}} = exp(u(z)) (x)))(z)) factorizes, E8C-a.s.
In the sense of enforcing compatibility, this loss generalizes the cycle-consistency loss to probabilistic conditionals. Also, the loss is different from the Jacobian-norm regularizers in contractive AE (68
and DAE 668 22. and explains the "tied weights" trick for AEs 63 82 81
68
(see Appx. D.I.
Implication on Gaussian VAE which uses additive Gaussian conditional models, po (x]=)) == fo) a2442) and go (2(x)) := (x) o is the vanilla and the most common form of VAE [45]. As its ELBO objective drives go (=(2)x) to meet the joint p(z) (z]2), compatibility is enforced. Under our view, this amounts to minimizing the compatibility loss Eq. (2). which then
enforces the match of Jacobians: (V:Jo (=)) = (o2/o2) 29 (x). As the two sides indicate the equation is constant of both x and z, it must be constant, so fo(=) and (x) must be affine, and the joint also a Gaussian 12 Thm. 3.. This conclusion coincides with the theory on additive
noise models in causality 89 62 and explains the empirical observation that the latent space of such

---

VAEs is quite linear [73]. is also the root of recent analyses that the latent space coordinates the data manifold (2T. and the inference model learns an isometric embedding after a proper rescaling [58].
This finding reveals that the expectation to use deep neural networks for learning a flexible nonlinear representation will be disappointed in Gaussian VAE. So we use a non-addiiive-Gauusian model, e.g. a flow-hased model [66 46 80 for at least one of (a(()(z) and (0(2)z)) (often the latter).
Direct Jacobian evaluation for Eq. (2) is of complexity (ddgdz) which is often prohibitively large. We thus propose a stochastic but unbiased and much cheaper method based on Hutchinson's 's trace estimator 40] tr(A) = Epnn [n An), where 7) is any random vector with zero mean and identity covariance (e.g., standard Gaussian). As the function within V
Efficient implementation.
applying the estimator yields a formulation that reduces gradient evaluation complexity to O(dx + dg):
expectation is V V rI
tr ((V: v.) V.Vīr)
(min) Ø) = Bp(n))) V= (n Vere,o 2)) I| where E(nz] 0, Var[na] Idx (3) 0,6
As concluded from the above analysis on Gaussian VAF, we use a flow-based model for the inference model qu (z|z). But in common instances evaluating the inverse of the flow intractable [66] 46 80] or costly [31]. This however, disables the use of automatic differentiation tools for estimating the gradients in the compatibility loss. Appx. D.7 explains this problem in detail and shows our solution.
3.2 Fitting Data
After achieving compatibilityy Cor. 2.5 guarantees the f.cl-fully supported conditional models uniquely determine a joint, hence a data distribution po.o (1)). To fit (o. to the true data distribution p" (xr). an explicit expression is required. For this, Eq. ①1 is not helpful as we do not have explicit expressions of a(x). b(z). But when compatibility is given, we can safely use density function formulae:
pe pe.a(z) (z) = 1/ po.at=) = J/ po.a(=)) po.a(0) ((dz') = 1/ Jz.
(=(='lrr
((dz') = (////,/t////////ooooaaa
which an explicit expression terms of the two conditionals. Although other expressions are possible, this one has a simple form, and the Monte-Caut Carlo expectation estimation in Z has lower variance than in X since usually dz < dg. We can thus fit data by maximum likelihood estimation:
(min)
(x) log pa po. (1x)
(x) [log (1 (1po (x]==))]
(4)
0.0
The loss function can be estimated using the reparameterization trick 45 to reduce variance, and the logsumexp trick is adopted for numerical stability. This expression can also serve for data likelihood evaluation. The final training process of CyGen is the joint optimization with the compatibility loss.
Comparison with DAE. We note that the DAE loss [2]
(e-"(iggteell po (x)())) is
lower bound of Eq. (4) due to Jensen's 's inequality, so it is not suitable for maximizing likelihood. In fact, the DAE loss minimizes (x) (go(<) 1z) Ilp (xx=)) for po(t) (z) to match (x]x))), where (2(2) and go (x(=)) are induced from the joint p" (xx (g (=|ar). but it is not a proper loss for (=(2)a)) as a mode-collapse behavior is promoted: the optimal go (z (r) only concentrates on the point(s) of argmin./ Po (ar)z), and an additional entropy term (=) (z]= required to optimize the same KL loss. This behavior hurts determinacy, as (=(x)) tends to be (mixture of) Dirac measure (Sec. (.2.2). The resulting Gibbs chain may also converge differently depending on initialization, as ergodicity is broken. This behavior also hurts compatibility, as qa (z(22 deviates from (0(2)2) (not Dirac), and does not match the Gibbs stationary distribution (35] 9]. In contrast, CyGen follows more fundamental logic: enforce compatibility explicitly and follow the maximum likelihood principle faithfully. It leads to proper loss for both conditionals that does not hinder determinacy.
3.3 Data Generation
Generating samples from the learned data distribution po.o (x) is not as straightforward as typical models that specify a prior, since ancestral sampling is not available. But it is still tractable via Markov chain Monte Carlo methods (MCMCs). We propose using dynamics-busee
MCMCs, which
often more efficient than Gibbs sampling (used in DAE [9] and GibbsNet (50)). They only require an unnormalized density function of the target distribution, which is readily available in CyGen when compatible: (x) = Pe.e (m)
pa(x|=) for any 2 E Z. In practice, this can be taken as sample from (2(x))) ) to lie in a high probability region for confident estimate.
(x) (2) (=(x)) Po. (2) x
(a(x
qa(s]=)
Stochastic gradient Langevin dynamics (SGLD) 83 is a representative instance, which has been shown to produce complicated realistic samples in energy-based [26). score-based (75 and diffusion-
7

---

Figure 4: Generated
data
DAE
VAE
BiGAN
CyGen
CyGen(PT)
data (DAE and CyGen use SGLD) and class-wise aggre- gated posteriors of DAE, VAE, BiGAN and CyGen. Also shows results of CyGen(PT) that is PreTrained as VAE. (Best view in color.)
7
class-wise aggregated posterior
Figure 5: Generated data along the training process of CyGen after VAE pretraining (iteration 1000), using SGLD (rows 1.3) and Gibbs sampling (row 2) for generation, and with (rows 1,2) and without the compat- ibility loss (row 3) for training. See also Appx. Fig. B
iteration
1100
1200
1300
1400
30000
P
CyG)n(PT), using SGLD
CyGen(PT). using Gibbs
compt. loss
7.0
5.4x 10a
x07
10
4.6x
CyGen(PT) w/o compt., using SGLD
compt. loss 1.1x 1
1.6x
2.6x
8.6x 10
1.2x10
based 38 76 models. gives the following transition:
x (++1) = + =V 200) log (1(20)]a0)) ((())
+ + (e)
where 2(1) ~ (0(22x(())) 7i ~ NN Iax), (5)
and e is a step size parameter. Method to draw z (2, can be developed symmetrically (see
Appx. Eq. (25)). Also applicable are other dynamics-base based MCMCs 19 23 57), and particle-based
variational inference methods [56 17
54 55 77 which are more sample-efficien.
Experiments
We demonstrate the power of CyGen for data generation and representation learning. Baselines include DAE, and generative models using Gaussian prior e.g. VAE and BiGAN (Appx.[..]) For fair comparison, all methods use the same architecture, which is an additive Gaussian [ [z| and Sylvester flow (Houscholder version) 80 for qs (2(2) (Appx. E.2 as required by CyGen (Sec. 3.1
is necessarily probabilistic for determinacy, so we exclude flow-based generative models and common BiGAN/GibbsNet architectures, which are deterministic. We also considered GibbsNet (50) which also aims at the prior issue, but it does not produce reasonable results using the same architecture, due its unstable training process (see Appx. E.T) Codes: htttp:://githar com coom/changllu0///
4.1 Synthetic Experiments
For visual verification the claims, we first consider 2D toy dataset (Fig 4 top-left). Appx. shows more details and results, including the investigation on another similar dataset.
Data generation. The learned data distributions (as the histogram of generated data samples) are shown Fig. 4 (row 1). We see the five clusters are blurred overlap in VAE's distribution and are still connected in BiGAN's, due to the specified prior. In contrast, our CyGen fits this distribution much better; particularly clearly separates the five non-connected clusters. This verifies the advantage to overcome the manifold mismalch problem. As for DAE, it cannot capture the data distribution due to collapsed inference model and insufficient determinacy (Sec. (3.2).
Representation. Class-wise aggregated posteriors (as the scatter plot of z samples from (2(x))pp (zly) for each class/cluster y) in Fig. (row 2) show that CyGen mitigates the poste-

---

4.2 World Experiments rior collapse problem, as the learned inference model (2(2)) better separates the classes with margin in the latent space. This more informative and representative feature would benefit down- stream tasks like classification or clustering in the latent space. In contrast, the specified Gaussian prior squeezes the VAE latent clusters to touch, and the BiGAN latent clusters even to mix. The mode-collapsed inference model of DAE locates all latent clusters in the same place.
Incorporating knowledge into conditionals. CyGen alone (without pretraining) already performs well. When knowledge is available, we can further incorporate into the conditional models. Fig. I shows pretraining CyGen's ss likelihood model as in VAE (CyGen(PT)) embodies VAE's knowledge that the prior centered and centrosymmetric, as the (all-class) aggregated posterior (Rs prior) such. Note its data generation quality is not sacrificed. Appx. Fig. [4 verifies this directly via the priors.
Comparison of data generation methods. We then make more analysis on CyGen. Fig. 5 (rows 1,2) shows generated data of CyGen using SGLD and Gibbs sampling. We see SGLD better recovers the true distribution, and is more robust slight incompatibility.
Impact of the compatibility loss. Fig. 5 (rows 1,3) also shows the comparison with training CyGen without the compatibility loss. We see the compatibility is then indeed out of control, which invalidates the likelihood estimation Eq. (4) for fitting data and the gradient estimation in Eq. (5) for data generation, leading to the failure in row 3. Along the training process of the normal CyGen, we also find a smaller compatibility loss makes better generation (esp. using Gibbs sampling).
DAE
VAE
CPG)
We test the performance of CyGen on real-world image datasets MNIST and SVHN. We consider the VAE-pretrained ver- sion, CyGen(PT), for more stable training. Appx. E.44 shows more details. On these datasets, even BiGAN cannot produce reason- able results using the same archi- tecture, similar to GibbsNet.
5
7
2
0
5 7
9
7
2
A A
8
9
4
%
7
9
4
1
4
6
5
7
5
9
2
2
4
0
5 7
4
d
2b
0
2
1
/
6
l
5
7 7
0
2
l
/
0
9
7
-21
$
O
3
O
3
3
9 13 2
7
4
I
4
6
8
9
2
7
y
6
6
3
9
2
2..
5
$
3
9
3
8
7
5
6
9
7
9
9
9
5
9
K 9 2
9
8
9
0
9
9
8
5
4
5
/
8
9
5
8
6
2
5
9
0
8
50
5
9
4
8
5
9
9
8
16
8
Data generation. From Fig. 6 We see that CyGen(PT) gener- ates both sharp and diverse sam- ples, as a sign to mitigate mani- fold mismatch. DAE samples are mostly imperceptible, due to the mode-collapsed yo (=(=)=) and the
Figure 6: Generated data on the MNIST and SVHN datasets.
subsequent lack of determinacy (Sec. 3.2 VAE samples are little blurry as typi- cal behavior due to the simply-connected prior. This observation is also quantita- tively supported by the FID score [36, on SVHN: CyGen achieves 102, while DAE 157 and VAE 128 (lower is better).
Table 7: Downstream classification accuracy (%) using learned representation by various models. i; Results from [50) using a different, deterministic ar- chitecture (not suitable for CyGen).
DAE VAE BIGAN GibbsNet CyGen(PT)
MNIST 98.0 94.5 91.0
97.7
98.3
Representation. We then show Table that CyGen(PT)'s latent representation more informative for the downstream classification task, as an indicator to avoid posterior collapse. BiGAN and GibbsNet make random guess using the same probabilistic flow architecture, and their reported results in [50) using a different, deterministic architecture (not suitable for CyGen due to insufficient determinacy) are still not always better, due to the prior constraint. We conclude that CyGen achieves both superior generation and representation learning performance.
SVHN 74.5 30.8-92
66.7
79.6
75.8
5 Conclusions and Discussions
In this work we investigate the possibility of defining joint distribution using two conditional distributions, under the motivation for generative modeling without an explicit prior. We develop systematic theory with novel and operable equivalence criteria for compatibility and sufficient conditions for determinacy, and propose a novel generative modeling framework CyGen that only

---

uses cyclic conditional models. Methods for achieving compatibility and determinacy, fitting data and data generation are developed. Experiments show the benefits of CyGen over DAE and prevailing generative models that specify a prior in overcoming manifold mismatch and posterior collapse.
The novel CyGen framework broadens the starting point to build a generative model, and the general theory could also foster a deeper understanding other machine learning paradigms, egg. dual learning and self-supervised learning, and inspire more efficient algorithms.
References
[1] Abrahams and J. B. Thomas. A note the characterization of bivariate densities by conditional densities. Communications in Statistics-Theory and Methods, 13(3):395-400, 1984.
[2] G. Alain and Y. Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15()):3563-3593, 2014.
[3] L. Ardizzone, J. Kruse, C. Rother, and U. Köthe. Analyzing inverse problems with invertible neural networks. In Proceedings of the International Conference on Learning Representations (ICLR 2019), 2019.
[4] B. C. Aroold and S. J. Press. Compatible conditional distributions. Journal of the American Statistical Association, 84(405):152-155 1989.
[5] B. C. Amold, E. Castillo, J. M. Sarabia, et al. Conditionally specified distributions: an introduction. Statistical Science, 16(3):249-274, 2001.
[6] B. C. Arnold, E. Castillo, and J.-M. S. Alegria. Conditionally specified distributions, volume 73. Springer Science & Business Media, 2012.
[7] Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):53-58, 1989.
[8] J. Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud. and J.-H. Jacobsen. Invertible residual networks. International Conference on Machine Learning, pages 573-582, PMLR, 2019.
[9] Y. Bengio, L. Yao, G. Alain, and P. Vincent. Generalized denoising auto-encoders as generative models. In Advances Neural Information Processing Systems, 2013.
[10] Y. Bengio, E. Laufer, G. Alain, and J. Yosinski. Deep generative stochastic networks trainable by backprop. In International Conference on Machine Learning, pages 226 234,234,222 2014.
[11] Berti, E. Dreassi, and P. Rigo. Compatibility results for conditional distributions. Journal of Multivariate Analysis, 125:190-200 1011999003,,000 2014.
[12] A. Bhattacharyya. On some sets of sufficient conditions leading to the normal bivariate distribution. Sankhyā: The Indian Journal of Statistics, pages 399-406, 1943.
[13] P. Billingsley. Probability and Measure. John Wiley & Sons, New Jersey, 2012. ISBN 978-1-118-1221 12237-2.
[14] J. Bornschein, S. Shabanian, A. Fischer, and Y. Bengio. Bidirectional Helmholtz machines. In International Conference on Machine Learning, pages 2511-2513 PMLR, 2016.
[15] Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from continuous space. In Proceedings The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, 2016.
[16] J. Brehmer and K. Cranmer. Flows for simultaneous manifold learning and density estimation. In Advances in Neural Information Processing Systems, volume 33, 2020.
[17] C. Chen, R. Zhang, W. Wang. B. Li, and L. Chen. unified particleoppimiizatinn framework for scalable Bayesian sampling. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI 2018), Monterey, California USA, 2018. Association for Uncertainty in Artificial Intelligence.
10

---

[18] R. T. Chen, J. Behrmann, D. Duvenaud, and J.-H. Jacobsen. Residual flows for invertible generative modeling. In Advances Neural Information Processing Systems, 2019.
[19] T. Chen, E. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pages 1683-1691, Beijing, China, 2014. IMLS.
[20] B. Coors, A. P. Condurache, and A. Geiger. SphereNet: Learning spherical representations for detection and classification in ommidirectional images. In Proceedings of the European Conference Computer Vision (ECCV), pages 518-533 2018.
[21] B. Dai and D. Wipf. Diagnosing and enhancing VAE models. In Proceedings of the International Conference Leurning Representations (ICLR 2019), 2019.
[22] T. R. Davidson, L. Falorsi, N. De Cao. T. Kipf, and M. Tomezak. Hyperspherical variational auto-encoders. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI 2018), 2018.
[23] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. In Advances in Neural Information Processing Systems, pages 3203-3211, Montréal, Canada, 2014. NIPS Foundation.
[24] L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-linear independent components estimation. In Workshop on the International Conference on Learning Representations, 2015.
[25] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. In Proceedings of the International Conference on Learning Representations (ICLR 2017), 2017.
[26] Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[27] V. Dumoulin, I. Belghazi, B. Poole, O. Mastropictro, A. Lamb, M. Arjovsky, and A. Courville. Adversarially learned inference. In Proceedings of the International Conference on Learning Representations (ICLR 2017.
2017.
(28) L. Falorsi, P. de Haan, T. R. Davidson, N. De Cao, M. Weiler, P. Forré, and T. S. Cohen. Explorations in homeomorphic variational auto-encoding. arXiv preprint h00.://00.466 2018.
[29] Galambos. Advanced probability theory, volume 10. CRC Press, 1995.
[30] I. Goodfellow, J. Pouget-Abadic M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, Montréal, Canada, 2014, NIPS Foundation.
[31] W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud. FFJORD: Free- form continuous dynamics for scalable reversible generative models. In Proceedings of the International Conference Learning Representations (ICLR 2019), 2019.
[32] A. Grover and S. Ermon. Uncertainty autoencoders: Learning compressed representations via variational information maximization. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2514-2524, PMLR, 2019.
[33] D. He, Y. Xia, T. Qin, I.. Wang, N. Yu, T.-Y Liu, and W.-Y. Ma. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820-828, 2016.
[34] J. Hc, D. Spokoyny, G. Neubig, and T. Borg-Kirkpatricke Lagging inference networks and posterior collapse in variational autoencoders. In Proceedings of the International Conference on Learning Representations (ICLR 2019), 2019.
[35] D. Heckerman, D. M. Chickering, C. Meek, R. Rounthwaite, and C. Kadie. Dependency networks for inference, collaborative filtering, and data visualization. Journal of Machine Learning Research, 1(Oct):49-55, 2000.

---

[36] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge a local Nash equilibrium. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[37] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerch- ner. Beta- VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of the International Conference on Learning Representations (ICLR 2017), 2017.
[38] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances Neural Information Processing Systems, 2020.
[39] R. Hofmann and V. Tresp. Nonlinear Markov networks for continuous variables. Advances in Neural Information Processing Systems, pages 521-527, 1998.
[40] M. Hutchinson. stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in -iissics-iimlatiinn and Computation, 18(3):1559 1076, 1989.
[41] D. Kalatzis, D. Eklund, G. Arvanitidis, and S. Hauberg. Variational autoencoders with ric- mannian brownian motion priors. In International Conference Machine Learning, pages 5053-5066. PMLR, 2020.
[42] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1857-1865. JMLR.org, 2017.
[43] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:I412 6980, 2014.
[44] D. P. Kingma and P. Dhariwal. Glow: generative flow with invertible Ix convolutions. In Advances in Neural Information Processing Systems, pages 10236-10244 2018.
[45] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR 2014), Banff, Canada, 2014. ICLR Committee.
[46] D. Kingma, T. Salimans, R. Jozefowiez, Chen, I. Sutskever, and M. Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pages 4751, Barcelona, Spain, 2016. NIPS Foundation.
[47] A. Klenke. Probability theory: a comprehensive course. Springer Science & Business Media, 2013.
[48] M. Kocaoglu, C. Snyder, A. G. Dimakis, and S. Vishwanath. CausalGAN: Learning causal implicit generative models with adversarial training. In Proceedings of the International Conference Learning Representations, 2018
[49] Z. Kong, W. Ping. J. Huang, Zhao, and B. Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09760 2020.
[50] A. M. Lamb, D. Hjelm, Y. Ganin, J. P. Cohen, A. C. Courville, and Y. Bengio. GibbsNet: Iterative adversarial inference for deep graphical models. In Advances in Neural Information Processing Systems, pages 5089-5098, 2017.
[51] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 144: 541-551, 1989.
[52] C. Li. M. Welling, J. Zhu, and B. Zhang. Graphical generative adversarial networks. In Advances in Neural Information Processing Systems, 2018.
[53] J. Lin, Z. Chen, Y. Xia, S. Liu, T. Qin, and J. Luo. Exploring explicit domain supervision for latent space disentanglement unpaired image-to-image translation. IEEE transactions on pattern analysis and machine intelligence, 2019.
12

---

[54] C. Liu, J. Zhuo, P. Cheng, R. Zhang. J. Zhu, and L. Carin. Understanding and accelerating particle-based variational inference. K. Chaudhuri and R. Salakhutdinov, editors, Proceedings Long
of the 36th International Conference on Machine Learning, volume 97, pages Beach, California USA, 2019. IMLS, PMLR.
[55] C. Liu, J. Zhuo, and J. Zhu. Understanding MCMC dynamics as flows on the Wasserstein space. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 4093-4103, Long Beach, California USA, 2019. IMLS, PMLR.
[56] Q. Liu and D. Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. In Advances in Neural Information Processing Systems, pages 2370-2378, Barcelona, Spain, 2016. NIPS Foundation.
[57] Y.-A. Ma, N. S. Chatterji. X. Cheng, N. Flammarion, P. L. Bartlett, and M. I. Jordan. there an analog of Nesterov acceleration for gradient-based MCMC? Bernoulli, 27(3):192-1192,, 2021.
[58] A. Nakagawa, K. Kato, and T. Suzuki. Quantitative understanding of VAE as non-linearly scaled isometric embedding. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[59] B. Pang, T. Han, E. Nijkamp, S.-C. Zhu, and Y. N. Wu. Learning latent space energy-based prior model. In Advances in Neural Information Processing Systems, volume 33, 2020.
[60] G. Papamakarios, T. Pavlakou, and I. Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2335-2344, 2017.
[61] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32:8026-8033 2019.
[62] J. Peters, J. M. Mooij, D. Janzing, and B. Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15()):2009-2553, 2014.
[63] M. Ranzato, Y.-.. Boureau, and Y. LeCun. Sparse feature learning for deep belief networks. Advances in Neural Information Processing Systems, 20: 185-1192, 2007.
[64] A. Razavi, A. v. d. Oord, B. Poole, and O. Vinyals. Preventing posterior collapse with delta- VAES. In Proceedings of the International Conference on Learning Representations (ICLR 2019), 2019.
[65] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.044 04558, 2020.
[66] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference Machine Learning (ICMI. 2015), pages 1530-1538, Lille, France, 2015, IMLS.
[67] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning. pages 1278-1288 2014.
[68] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the International Conference on Machine Learning, 2011.
[69] S. Rifai, Y. Bengio, Y.N. Dauphin, a. Vincent. A generative process for sampling contractive auto-encoders. In Proceedings of the International Conference on Machine Learning, pages 1811-, 2012.
[70] A. Rinaldo. Advanced probability, February 2018
[71] D. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back propppropggtting errors. Nature, 323(6089) 1986.
13

---

[72] M. Seitzer. pytorch-fid: FID Score for PyTorch. https: pytorch- fid August 2020. Version 0.2.1.
com www.shwwww.ooommeeeeeeeeeeeeeeee tzer
[73] H. Shao, A. Kumar, and P. Thomas Fletcher. The Riemannian geometry of deep generative models. In Pruceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pages 315-323, 2018.
[74] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMIR, 2015.
[75] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[76] Y. Song, J. Sohi-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based gen- crative modeling through stochastic differential equations. In Proceedings of the International Conference Learning Representations (ICLR 2021), 2021.
[77] A. Taghvaci and P. Mehta. Accelerated flow for probability distributions. In Proceedings of the 36th International Conference Machine Learning, pages 6076-6085, PMLR, 2019.
[78] T. Teshima, I. Ishikawa, K. Tojo, K. Oono, M. Ikeda, and M. Sugiyama. Coupling- based invertible neural networks universal diffeomorphism approximators. arXiv preprint arXiv:2006. / 1469, 2020.
[79] A. Vahdat and J. Kautz. NVAE: A deep hicrarchical variational autoencoder. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[80] R. Van Den Berg, L. Hasenclever, J. M. Tomezak, and M. Welling. Sylvester normalizing flows for variational inference. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pages 393 402. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
[81] P. Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, 23(7):1661-1674, 2011.
[82] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the International Conference on Machine Learning, pages 1096-1103, 2008
[83] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference Machine Learning (ICML 2011), pages 681 -688, Bellevue, Washington USA, 2011. IMLS.
[84] Y. Xia, Bian, T. Qin, N. Yu, and T.-Y. Liu. Dual inference for machine learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (LJCAI-17), pages 3112-3118, 2017.
[85] Y. Xia, T. Qin, W. Chen, J. Bian, N. Yu, and T.-Y. Liu. Dual supervised learning. In Proceedings of the 34th International Conference on Machine Learning- JMLR.org, 2017.
Learning-Volume Volume 70, pages 3789-3798.
[86] M. Xiao, S. Zheng, C. Liu, Y. Wang, D. He. G. Ke, Bian, Z. Lin, and T.-Y. Liu. Invertible image rescaling. In European Conference on Computer Vision, pages 126-144. Springer, 2020.
[87] Z. Xiao, K. Kreis, J. Kautz, and A. Vahdat. VAEBM: symbiosis between variational autoen- coders and energy-based models. In International Conference on Learning Representations (ICLR 2021), 2021.
[88] Z. Yi, H. Zhang, P. Tan, and M. Gong. DualGAN: Unsupervised dual learning for image-to- image translation. In Proceedings of the IEEE International Conference on Computer Vision pages 284992857,200
14

---

[89] K. Zhang and A. Hyvärinen. On the identifiability of the post-nonlinear causal model. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009), pages 647-655, AUAI Press, 2009.
[90] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages
2017.
15


Reviews:
[{"review_id": "KveDhYBsG1", "reviewer": "Reviewer_2j5F", "summary": "This paper studies the conditions under which two conditional densities are compatible, with a particular focus on cases relevant to generative modeling, establishing conditions for compatibility and uniqueness. Based on the theoretical results, an objective for regularizing towards compatibility is proposed, implemented, and fit to data. ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, this paper has been missing from the VAE literature for ~7 years. It fills an important hole in the literature. Importantly, the paper operationalizes the theory. \n\nThat being said, the proofs in the appendix are quite involved and time constraints did not permit me to go through them properly. Nothing obvious seems off, but I didn't check them carefully.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "KmbTNzmiWWD", "reviewer": "Reviewer_F92e", "summary": "The work answers an interesting question: Given two cyclic conditional distributions, under which condition there exists a compatible joint distribution, and when the compatible joint distribution is unique. \n\nFrom the theory the authors propose a generative framework called CyGen. CyGen is trained using maximum likelihood based on modelings of the conditional probabilities. Sampling from the marginal distribution modeled by the two conditional distributions is also not straightforward. Luckily this has been investigated before and there are practical MCMC based procedures that converge to the marginal distribution.\n\nThe authors also show some experimental results that the proposed CyGen is able to generate more realistic samples compared to VAE or DAE. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Problem setting of this paper is interesting. The compatibility and determinacy analysis are very clean. The proposed CyGen algorithm, so far as I know, is novel. \n\nI am a little worried about how useful these cyclic condition settings are. After all, most of the models simply assume some priors instead and they can still generate very realistic-looking samples. Besides the generative settings such as CyGen, is there any other use case for the Cyclic Conditionals?\n\nAnother concern is about the practicality of CyGen. Yeah the MCMC based methods are fine but compared to most of the generative models nowadays, it is not as efficient. In the experiment the authors first pretrain the model from VAE, and then use CyGen more like a “finetuning”. I am curious if we run a cold start from CyGen, will we get meaningful results? Will we run into issues such as mode collapse or some types of \"bad\" local optimum?  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " the authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "KZ2G0fjH35k", "reviewer": "Reviewer_F3Nz", "summary": "Notes: \n  -Model the joint p(x,z) by modeling p(x|z) and q(z|x) to form a cycle.  \n  -Many existing techniques use an uninformative prior p(z).  \n  -Compatibility and determinacy criterions.  \n  -Use parameterized density models p(x|z) and q(z|x).  \n  -Define r(x,z) = log(p(x|z) / q(z|x)) and make the loss the norm of the jacobian, computed over samples from p(x)*q(z|x) (real data / inferred latents).   \n  -Jacobian is intractable, so instead approximate by taking gradient wrt. x, then multiplying by random vector eta, then taking gradient wrt z.  \n  -Sample using stochastic gradient langevin dynamics.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-rebuttal: I appreciate the clarifications from the authors and keep my score.  \n\n--\n\nThis paper proposes a novel criteria for training a joint distribution described by p(x | z) and q(z | x), without the constrain that the z have a simple prior.  The technique clearly works in the sense of producing decent samples and decent latent variables, but results seem unlikely to be state of the art.  At the same time, the analysis and discussion of these cyclical generative models seems like a useful contribution, and gives a more theoretical grounding for this type of algorithm than previous papers in this general area.  \n\nComments: \n  -Overall model and objective seems reasonably easy to implement and is cleanly explained in the paper.  \n\n  -The experimental evaluation seems rather limited, and the classification accuracies are about the same as GibbsNet and DAE.  The generation quality seems decent, but is not quantitatively evaluated.  \n\n  -Is my understanding correct that the langevin dynamics sampling procedure (section 3.3) is only used at test time and does not need to be used at train time?  \n\n  -Out of curiosity, if you run a chain of z -> x -> z -> x -> ..., does this chain have good mixing properties in theory?  I.e. will the chain eventually be able to produce samples uniformly from p(x).  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitation is potential lack of disentanglement, but whether we want this is of course open to discussion.  ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues, work is purely technical.  ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper starts with the likelihood and inference model used in variational auto-encoders and studies if and when a joint distribution can be modeled using these two conditionals. The paper is a nice to read and brings to light many of the results on conditionally specified models to the VAE community. I also appreciated a practical implementation with cross-derivatives to encourage the two conditional distributions to be compatible with a joint distribution.  A couple of comments

- I don't think the full Arnold book was cited. It has more results and should be integrated into the narrative 

@book{arnold2012conditionally,
  title={Conditionally specified distributions},
  author={Arnold, Barry C and Castillo, Enrique and Alegria, Jose-Maria Sarabia},
  volume={73},
  year={2012},
  publisher={Springer Science \& Business Media}
}

- Edit the paper to frame and highlight the practical implications of the results 
Decision: Accept (Poster)