On the Generative Utility of Cyclic Conditionals

1 Introduction . Deep generative models have achieved a remarkable success in the past decade for generating realistic complex data x and extracting useful representations through their latent variable z. Variational auto-encoders ( VAEs ) [ 45 , 67 , 14 , 15 , 46 , 80 ] follow the Bayesian framework and specify a prior distribution p ( z ) and a likelihood model p ( x|z ) , so that a joint distribution p ( x , z ) = p ( z ) p ( x|z ) is defined for generative modeling ( the joint induces a distribution p ( x ) on data ) . An inference model q ( z|x ) is also used to approximate the posterior distribution p ( z|x ) ( derived from the joint p ( x , z ) ) , which serves for extracting representations . Other frameworks like generative adversarial nets [ 30 , 25 , 27 ] , flow-based models [ 24 , 60 , 44 , 31 ] and diffusion-based models [ 74 , 38 , 76 , 49 ] follow the same structure , with different choices of the conditional models p ( x|z ) and q ( z|x ) and training objectives . While for the prior p ( z ) , there is often not much knowledge for complex data ( like images , text , audio ) , and these models widely adopt an uninformative prior such as a standard Gaussian . This however , introduces some side effects : • Posterior collapse [ 15 , 34 , 64 ] : The standard Gaussian prior tends to squeeze q ( z|x ) towards the origin for all x , which degrades the representativeness of the inferred z for x and hurts downstream tasks in the latent space like classification and clustering . • Manifold mismatch [ 22 , 28 , 41 ] : Typically the likelihood model is continuous ( keeps topology ) , so the standard Gaussian prior would restrict the modeled data distribution to a simply-connected support , which limits the capacity for fitting data from a non- ( simply ) connected support . While there are works trying to mitigate the two problems , they require either a strong domain knowledge [ 48 , 41 ] , or additional cost to learn a complicated prior model [ 52 , 21 , 79 ] sometimes even at the cost of inconvenient inference [ 59 , 87 ] . ∗Correspondence to : Chang Liu < changliu @ microsoft.com > . †Work done during an internship at Microsoft Research Asia . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . One question then naturally emerges : Can we model a joint distribution p ( x , z ) only using the likelihood p ( x|z ) and inference q ( z|x ) models ? If we can , the limitations from specifying or learning a prior are then removed from the root . Also , the inference model q ( z|x ) is then no longer a struggling approximation to a predefined posterior but participates in defining the joint distribution ( avoid “ inner approximation ” ) . Modeling conditionals is also argued to be much easier than modeling marginal or joint distributions directly [ 2 , 9 , 10 ] . In many cases , one may even have better knowledge on the conditionals than on the prior , e.g . shift/rotation invariance of image representations ( CNNs [ 51 ] / SphereNet [ 20 ] ) , and rules to extract frequency/energy features for audio [ 65 ] . It is then more natural and effective to incorporate this knowledge into the conditionals than using an uninformative prior . In this paper , we explore such a possibility , and develop both a systematic theory and a novel generative modeling framework CyGen ( Cyclic-conditional Generative model ) . ( 1 ) Theoretical analysis on the question amounts to two sub-problems : can two given cyclic conditionals correspond to a common joint , and if yes , can they determine the joint uniquely . We term them compatibility and determinacy of two conditionals , corresponding to the existence and uniqueness of a common joint . For this , we develop novel compatibility criteria and sufficient conditions for determinacy . Beyond existing results , ours are operable ( vs. existential [ 11 ] ) and self-contained ( vs. need a marginal [ 9 , 10 , 50 , 32 ] ) , and are general enough to cover both continuous and discrete cases . Our compatibility criteria are also equivalence ( vs. unnecessary [ 1 , 4–6 ] ) conditions . The seminal book [ 6 ] makes extensive analysis for various parametric families . Besides the equivalence criteria , we also extend their general analysis beyond the product support case , and also cover the Dirac case . ( 2 ) In addition to its independent contribution , the theory also enables generative modeling using only the two cyclic conditional models , i.e . the CyGen framework . We develop methods for achieving compatibility and determinacy to make an eligible generative model , and for fitting and generating data to serve as a generative model . Efficient implementation techniques are designed . Note CyGen also determines a prior implicitly ; it just does not need an explicit model for the prior ( vs. [ 52 , 21 , 79 , 59 ] ) . We show the practical utility of CyGen in both synthetic and real-world tasks . The improved performance in downstream classification and data generation demonstrates the advantage to mitigate the posterior collapse and manifold mismatch problems . 1.1 Related work . Dependency networks ( [ 35 ] ; similarly [ 39 ] ) are perhaps the first to pursue the idea of modeling a joint by a set of conditionals . They use Gibbs sampling to determine the joint and are equivalent to undirected graphical models . They do not allow latent variables , so compatibility is not a key consideration as the data already specifies a joint as the common target of the conditionals . Beyond that , we introduce latent variables to better handle sensory data like images , for which we analyze the conditions for compatibility and determinacy and design novel methods to solve this different task . Denoising auto-encoders ( DAEs ) . AEs [ 71 , 7 ] aim to extract data features by enforcing reconstruction through its encoder and decoder , which are deterministic hence insufficient determinacy ( see Sec . 2.2.2 ) . DAEs [ 82 , 9 , 10 ] use a probabilistic encoder and decoder for robust reconstruction against random data corruption . Their utility as a generative model is first noted through the equivalence to score matching ( implies modeling p ( x ) ) for a Gaussian RBM [ 81 ] or an infinitesimal Gaussian corruption [ 2 ] . In more general cases , the utility to modeling the joint p ( x , z ) is studied via the Gibbs chain , i.e . the Markov chain with transition kernel p ( x′|z′ ) q ( z′|x ) . Under a global [ 9 , 50 , 32 ] or local [ 10 ] shared support condition , its stationary distribution π ( x , z ) exists uniquely . But this is not really determinacy : even incompatible conditionals can have this unique existence , in which case π ( z|x ) 6= q ( z|x ) [ 35 , 9 ] . Moreover , the Gibbs chain does not give an explicit expression of π ( x , z ) ( thus intractable likelihood evaluation ) , and requires many iterations to converge for data generation and even for training ( Walkback [ 9 ] , GibbsNet [ 50 ] ) , making the algorithms costly and unstable . As for compatibility , it is not really covered in DAEs . Existing results only consider the statistical consistency ( unbiasedness under infinite data ) of the p ( x|z ) estimator by fitting ( x , z ) data from p∗ ( x ) q ( z|x ) [ 9 , 10 , 50 , 32 ] , where p∗ ( x ) denotes the true data distribution . Particularly , they require a marginal p∗ ( x ) in advance , so that the joint is already defined by p∗ ( x ) q ( z|x ) regardless of p ( x|z ) , while compatibility ( as well as determinacy ) is a matter only of the two conditionals . More crucially , the DAE loss is not proper for optimizing q ( z|x ) as it promotes a mode-collapse behavior . This hinders both compatibility and determinacy ( Sec . 3.2 ) : one may not use q ( z|x ) for inference , and data generation may depend on initialization . In contrast , CyGen explicitly enforces compatibility and guarantees determinacy , and enables likelihood evaluation and better generation . Dual learning considers conversion between two modalities in both directions , e.g. , machine translation [ 33 , 85 , 84 ] and image style transfer [ 42 , 90 , 88 , 53 ] . Although we also consider both directions , the fundamental distinction is that in generative modeling there is no data of the latent variable z ( not even unpaired ) . Technically , they did not consider determinacy : they require a marginal to determine a joint . We find their determinacy is actually insufficient ( see Sec . 2.2.2 ) . Their cycle-consistency loss [ 42 , 90 , 88 ] is a version of our compatibility criterion in the Dirac case ( see Sec . 2.2.1 ) , and we extend it to allow probabilistic conversion ( see Sec . 3.1 ) . 2 Compatibility and Determinacy Theory . To be a generative model , a system needs to determine a distribution on the data variable x . With latent variable z , this amounts to determining a joint distribution over ( x , z ) , which calls for compatibility and determinacy analysis for cyclic conditionals . In this section we build a general theory on the conditions for compatibility and determinacy . We begin with formalizing the problems . Setup . Denote the measure spaces of the two random variables x and z as ( X , X , ξ ) and ( Z , Z , ζ ) 3 , where X , Z are the respective sigma-fields , and the base measures ξ , ζ ( e.g. , Lebesgue measure on Euclidean spaces , counting measure on finite/discrete spaces ) are sigma-finite . We use X ∈X , Z ∈ Z to denote measurable sets , and use “ ξ= ” , “ ⊆ξ ” as the extensions of “ = ” , “ ⊆ ” up to a set of ξ-measure-zero ( Def . A.1 ) . Following the convention in machine learning , we call a “ probability measure ” as a “ distribution ” . We do not require any further structures such as topology , metric , or linearity , for the interest of the most general conclusions that unify Euclidean/manifold and finite/discrete spaces and allow X , Z to have different dimensions or types . Joint and conditional distributions are defined on the product measure space ( X×Z , X ⊗Z , ξ⊗ζ ) , where “ × ” is the usual Cartesian product , X ⊗Z : = σ ( X ×Z ) is the sigma-field generated by measurable rectangles from X ×Z , and ξ⊗ζ is the product measure [ 13 , Thm . 18.2 ] . Define the slice ofW ∈ X ⊗Z at z asWz : = { x | ( x , z ) ∈ W } ∈ X [ 13 , Thm . 18.1 ( i ) ] , and its projection onto Z asWZ : = { z | ∃x ∈ X s.t . ( x , z ) ∈ W } ∈ Z ( Appx . A.3 ) . In a similar style , denote the marginal of a joint π on Z as πZ ( Z ) : = π ( X×Z ) . To keep the same level of generality , we follow the general definition of conditionals ( [ 13 , p.457 ] ; see also Appx . A.4 ) : the conditional π ( X|z ) of a joint π is the density function ( R-N derivative ) of π ( X×· ) w.r.t πZ . We highlight the key characteristic under this generality that π ( ·|z ) can be arbitrary on a set of πZ-measure-zero , particularly , outside the support of πZ . Appx . A and B provide more background details and our technical preparations that are also of independent interest . The goal of analysis can then be formalized below . Definition 2.1 ( compatibility and determinacy ) . We say two conditionals µ ( X|z ) , ν ( Z|x ) are compatible , if there exists a joint distribution π on ( X×Z , X ⊗Z ) such that µ ( X|z ) and ν ( Z|x ) are its conditional distributions . We say two compatible conditionals have determinacy on a set S ∈X ⊗Z , if there is only one joint distribution concentrated on S that makes them compatible . To put the concept into practical use , the analysis aims at operable conditions for compatibility and determinacy . We consider two cases separately ( still unifying continuous and discrete cases ) , as they correspond to different types of generative models , and lead to qualitatively different conclusions .

Summary: The work answers an interesting question: Given two cyclic conditional distributions, under which condition there exists a compatible joint distribution, and when the compatible joint distribution is unique.   From the theory the authors propose a generative framework called CyGen. CyGen is trained using maximum likelihood based on modelings of the conditional probabilities. Sampling from the marginal distribution modeled by the two conditional distributions is also not straightforward. Luckily this has been investigated before and there are practical MCMC based procedures that converge to the marginal distribution.  The authors also show some experimental results that the proposed CyGen is able to generate more realistic samples compared to VAE or DAE.