On the Generative Utility of Cyclic Conditionals

1 Introduction . Deep generative models have achieved a remarkable success in the past decade for generating realistic complex data x and extracting useful representations through their latent variable z. Variational auto-encoders ( VAEs ) [ 45 , 67 , 14 , 15 , 46 , 80 ] follow the Bayesian framework and specify a prior distribution p ( z ) and a likelihood model p ( x|z ) , so that a joint distribution p ( x , z ) = p ( z ) p ( x|z ) is defined for generative modeling ( the joint induces a distribution p ( x ) on data ) . An inference model q ( z|x ) is also used to approximate the posterior distribution p ( z|x ) ( derived from the joint p ( x , z ) ) , which serves for extracting representations . Other frameworks like generative adversarial nets [ 30 , 25 , 27 ] , flow-based models [ 24 , 60 , 44 , 31 ] and diffusion-based models [ 74 , 38 , 76 , 49 ] follow the same structure , with different choices of the conditional models p ( x|z ) and q ( z|x ) and training objectives . While for the prior p ( z ) , there is often not much knowledge for complex data ( like images , text , audio ) , and these models widely adopt an uninformative prior such as a standard Gaussian . This however , introduces some side effects : • Posterior collapse [ 15 , 34 , 64 ] : The standard Gaussian prior tends to squeeze q ( z|x ) towards the origin for all x , which degrades the representativeness of the inferred z for x and hurts downstream tasks in the latent space like classification and clustering . • Manifold mismatch [ 22 , 28 , 41 ] : Typically the likelihood model is continuous ( keeps topology ) , so the standard Gaussian prior would restrict the modeled data distribution to a simply-connected support , which limits the capacity for fitting data from a non- ( simply ) connected support . While there are works trying to mitigate the two problems , they require either a strong domain knowledge [ 48 , 41 ] , or additional cost to learn a complicated prior model [ 52 , 21 , 79 ] sometimes even at the cost of inconvenient inference [ 59 , 87 ] . ∗Correspondence to : Chang Liu < changliu @ microsoft.com > . †Work done during an internship at Microsoft Research Asia . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . One question then naturally emerges : Can we model a joint distribution p ( x , z ) only using the likelihood p ( x|z ) and inference q ( z|x ) models ? If we can , the limitations from specifying or learning a prior are then removed from the root . Also , the inference model q ( z|x ) is then no longer a struggling approximation to a predefined posterior but participates in defining the joint distribution ( avoid “ inner approximation ” ) . Modeling conditionals is also argued to be much easier than modeling marginal or joint distributions directly [ 2 , 9 , 10 ] . In many cases , one may even have better knowledge on the conditionals than on the prior , e.g . shift/rotation invariance of image representations ( CNNs [ 51 ] / SphereNet [ 20 ] ) , and rules to extract frequency/energy features for audio [ 65 ] . It is then more natural and effective to incorporate this knowledge into the conditionals than using an uninformative prior . In this paper , we explore such a possibility , and develop both a systematic theory and a novel generative modeling framework CyGen ( Cyclic-conditional Generative model ) . ( 1 ) Theoretical analysis on the question amounts to two sub-problems : can two given cyclic conditionals correspond to a common joint , and if yes , can they determine the joint uniquely . We term them compatibility and determinacy of two conditionals , corresponding to the existence and uniqueness of a common joint . For this , we develop novel compatibility criteria and sufficient conditions for determinacy . Beyond existing results , ours are operable ( vs. existential [ 11 ] ) and self-contained ( vs. need a marginal [ 9 , 10 , 50 , 32 ] ) , and are general enough to cover both continuous and discrete cases . Our compatibility criteria are also equivalence ( vs. unnecessary [ 1 , 4–6 ] ) conditions . The seminal book [ 6 ] makes extensive analysis for various parametric families . Besides the equivalence criteria , we also extend their general analysis beyond the product support case , and also cover the Dirac case . ( 2 ) In addition to its independent contribution , the theory also enables generative modeling using only the two cyclic conditional models , i.e . the CyGen framework . We develop methods for achieving compatibility and determinacy to make an eligible generative model , and for fitting and generating data to serve as a generative model . Efficient implementation techniques are designed . Note CyGen also determines a prior implicitly ; it just does not need an explicit model for the prior ( vs. [ 52 , 21 , 79 , 59 ] ) . We show the practical utility of CyGen in both synthetic and real-world tasks . The improved performance in downstream classification and data generation demonstrates the advantage to mitigate the posterior collapse and manifold mismatch problems . 1.1 Related work . Dependency networks ( [ 35 ] ; similarly [ 39 ] ) are perhaps the first to pursue the idea of modeling a joint by a set of conditionals . They use Gibbs sampling to determine the joint and are equivalent to undirected graphical models . They do not allow latent variables , so compatibility is not a key consideration as the data already specifies a joint as the common target of the conditionals . Beyond that , we introduce latent variables to better handle sensory data like images , for which we analyze the conditions for compatibility and determinacy and design novel methods to solve this different task . Denoising auto-encoders ( DAEs ) . AEs [ 71 , 7 ] aim to extract data features by enforcing reconstruction through its encoder and decoder , which are deterministic hence insufficient determinacy ( see Sec . 2.2.2 ) . DAEs [ 82 , 9 , 10 ] use a probabilistic encoder and decoder for robust reconstruction against random data corruption . Their utility as a generative model is first noted through the equivalence to score matching ( implies modeling p ( x ) ) for a Gaussian RBM [ 81 ] or an infinitesimal Gaussian corruption [ 2 ] . In more general cases , the utility to modeling the joint p ( x , z ) is studied via the Gibbs chain , i.e . the Markov chain with transition kernel p ( x′|z′ ) q ( z′|x ) . Under a global [ 9 , 50 , 32 ] or local [ 10 ] shared support condition , its stationary distribution π ( x , z ) exists uniquely . But this is not really determinacy : even incompatible conditionals can have this unique existence , in which case π ( z|x ) 6= q ( z|x ) [ 35 , 9 ] . Moreover , the Gibbs chain does not give an explicit expression of π ( x , z ) ( thus intractable likelihood evaluation ) , and requires many iterations to converge for data generation and even for training ( Walkback [ 9 ] , GibbsNet [ 50 ] ) , making the algorithms costly and unstable . As for compatibility , it is not really covered in DAEs . Existing results only consider the statistical consistency ( unbiasedness under infinite data ) of the p ( x|z ) estimator by fitting ( x , z ) data from p∗ ( x ) q ( z|x ) [ 9 , 10 , 50 , 32 ] , where p∗ ( x ) denotes the true data distribution . Particularly , they require a marginal p∗ ( x ) in advance , so that the joint is already defined by p∗ ( x ) q ( z|x ) regardless of p ( x|z ) , while compatibility ( as well as determinacy ) is a matter only of the two conditionals . More crucially , the DAE loss is not proper for optimizing q ( z|x ) as it promotes a mode-collapse behavior . This hinders both compatibility and determinacy ( Sec . 3.2 ) : one may not use q ( z|x ) for inference , and data generation may depend on initialization . In contrast , CyGen explicitly enforces compatibility and guarantees determinacy , and enables likelihood evaluation and better generation . Dual learning considers conversion between two modalities in both directions , e.g. , machine translation [ 33 , 85 , 84 ] and image style transfer [ 42 , 90 , 88 , 53 ] . Although we also consider both directions , the fundamental distinction is that in generative modeling there is no data of the latent variable z ( not even unpaired ) . Technically , they did not consider determinacy : they require a marginal to determine a joint . We find their determinacy is actually insufficient ( see Sec . 2.2.2 ) . Their cycle-consistency loss [ 42 , 90 , 88 ] is a version of our compatibility criterion in the Dirac case ( see Sec . 2.2.1 ) , and we extend it to allow probabilistic conversion ( see Sec . 3.1 ) . 2 Compatibility and Determinacy Theory . To be a generative model , a system needs to determine a distribution on the data variable x . With latent variable z , this amounts to determining a joint distribution over ( x , z ) , which calls for compatibility and determinacy analysis for cyclic conditionals . In this section we build a general theory on the conditions for compatibility and determinacy . We begin with formalizing the problems . Setup . Denote the measure spaces of the two random variables x and z as ( X , X , ξ ) and ( Z , Z , ζ ) 3 , where X , Z are the respective sigma-fields , and the base measures ξ , ζ ( e.g. , Lebesgue measure on Euclidean spaces , counting measure on finite/discrete spaces ) are sigma-finite . We use X ∈X , Z ∈ Z to denote measurable sets , and use “ ξ= ” , “ ⊆ξ ” as the extensions of “ = ” , “ ⊆ ” up to a set of ξ-measure-zero ( Def . A.1 ) . Following the convention in machine learning , we call a “ probability measure ” as a “ distribution ” . We do not require any further structures such as topology , metric , or linearity , for the interest of the most general conclusions that unify Euclidean/manifold and finite/discrete spaces and allow X , Z to have different dimensions or types . Joint and conditional distributions are defined on the product measure space ( X×Z , X ⊗Z , ξ⊗ζ ) , where “ × ” is the usual Cartesian product , X ⊗Z : = σ ( X ×Z ) is the sigma-field generated by measurable rectangles from X ×Z , and ξ⊗ζ is the product measure [ 13 , Thm . 18.2 ] . Define the slice ofW ∈ X ⊗Z at z asWz : = { x | ( x , z ) ∈ W } ∈ X [ 13 , Thm . 18.1 ( i ) ] , and its projection onto Z asWZ : = { z | ∃x ∈ X s.t . ( x , z ) ∈ W } ∈ Z ( Appx . A.3 ) . In a similar style , denote the marginal of a joint π on Z as πZ ( Z ) : = π ( X×Z ) . To keep the same level of generality , we follow the general definition of conditionals ( [ 13 , p.457 ] ; see also Appx . A.4 ) : the conditional π ( X|z ) of a joint π is the density function ( R-N derivative ) of π ( X×· ) w.r.t πZ . We highlight the key characteristic under this generality that π ( ·|z ) can be arbitrary on a set of πZ-measure-zero , particularly , outside the support of πZ . Appx . A and B provide more background details and our technical preparations that are also of independent interest . The goal of analysis can then be formalized below . Definition 2.1 ( compatibility and determinacy ) . We say two conditionals µ ( X|z ) , ν ( Z|x ) are compatible , if there exists a joint distribution π on ( X×Z , X ⊗Z ) such that µ ( X|z ) and ν ( Z|x ) are its conditional distributions . We say two compatible conditionals have determinacy on a set S ∈X ⊗Z , if there is only one joint distribution concentrated on S that makes them compatible . To put the concept into practical use , the analysis aims at operable conditions for compatibility and determinacy . We consider two cases separately ( still unifying continuous and discrete cases ) , as they correspond to different types of generative models , and lead to qualitatively different conclusions .

Summary: Notes:    -Model the joint p(x,z) by modeling p(x|z) and q(z|x) to form a cycle.     -Many existing techniques use an uninformative prior p(z).     -Compatibility and determinacy criterions.     -Use parameterized density models p(x|z) and q(z|x).     -Define r(x,z) = log(p(x|z) / q(z|x)) and make the loss the norm of the jacobian, computed over samples from p(x)*q(z|x) (real data / inferred latents).      -Jacobian is intractable, so instead approximate by taking gradient wrt. x, then multiplying by random vector eta, then taking gradient wrt z.     -Sample using stochastic gradient langevin dynamics.