Multimeasurement Generative Models

1 INTRODUCTION . Consider a collection of i.i.d . samples { xi } ni=1 , assumed to have been drawn from an unknown distribution with density pX in Rd . An important problem in probabilistic modeling is the task of drawing independent samples from pX , which has numerous potential applications . This problem is typically approached in two phases : approximating pX , and drawing samples from the approximated density . In unnormalized models the first phase is approached by learning the energy function fX associated with the Gibbs distribution pX ∝ exp ( −fX ) , and for the second phase one must resort to Markov chain Monte Carlo methods , such as Langevin MCMC , which are typically very slow to mix in high dimensions . MCMC sampling is considered an “ art ” and we do not have black box samplers that converge fast and are stable for complex ( natural ) distributions . The source of the problem is mainly attributed to the fact that the energy functions of interest are typically highly nonconvex . A broad sketch of our solution to this problem is to model a smoother density in an M-fold expanded space . The new density p ( y ) , called M-density , is defined in RMd , where the bold y is a shorthand for ( y1 , . . . , yM ) . M-density is smoother in the sense that its marginals pm ( ym ) are obtained by the convolution pm ( ym ) = ∫ pm ( ym|x ) p ( x ) dx with a smoothing kernel pm ( ym|x ) which for most of the paper we take to be the isotropic Gaussian : Ym = X +N ( 0 , σ 2 mId ) . Although we bypass learning p ( x ) , the new formalism allows for generating samples from p ( x ) since X can be estimated exactly given Y = y ( for large M ) . To give a physical picture , the approach here is based on “ taking apart ” the complex manifold where the random variable X is concentrated in Rd and mapping it to a smoother manifold in RMd where Y = ( Y1 , . . . , YM ) is concentrated . Smoothing a density with a kernel is a technique in nonparametric density estimation that goes back to Parzen ( 1962 ) . In kernel density estimation , the estimator of p ( x ) is obtained by convolving the empirical measure with a kernel . In that methodology , the kernel bandwidth ( σ , for Gaussian kernels ) is adjusted to estimate p ( x ) in Rd given a collection of independent samples { xi } ni=1 . This estimator , like most nonparametric estimators , suffers from a severe curse of dimensionality ( Wainwright , 2019 ) . But what if the kernel bandwidth is fixed : how much easier is the problem of estimating p ( y ) ? This question is answered in ( Goldfeld et al. , 2020 ) , where they obtained the rate of convergence eO ( d ) n−1/2 ( measured using various distances ) in remarkable contrast to the well-known n−1/d rate for estimating p ( x ) . This nonparametric estimation result is not directly relevant here , but it formalizes the intuition that learning p ( y ) = ∫ p ( y|x ) p ( x ) dx is a lot simpler than learning p ( x ) . With this motivation , we start with an introduction to the problem of learning unnormalized p ( y ) , based on independent samples from p ( x ) . This problem was formulated by Vincent ( 2011 ) using score matching ( Hyvärinen , 2005 ) . It was approached recently with the more fundamental methodology of empirical Bayes ( Saremi & Hyvärinen , 2019 ) . The idea is to use the Bayes estimator of X given Y = y , the study of which is at the root of the empirical Bayes approach to statistics ( Robbins , 1956 ) , in a least-squares objective . This machinery builds on the fact that the estimator x̂ ( y ) = E [ X|Y = y ] can be expressed in closed form in terms of unnormalized p ( y ) ( Sec . 3.1 ) . For Gaussian kernels , x̂ ( y ) is expressed in terms of the score function ∇ log p ( y ) ( Miyasawa , 1961 ) . For such kernels , the learning objective arrived at in ( Saremi & Hyvärinen , 2019 ) is identical to the denoising score matching formulation ( Vincent , 2011 ) , but with new insights rooted in empirical Bayes which is the statistical framework for denoising . The main problem with the empirical Bayes methodology is that p ( x|y ) remains unknown and can not be sampled from . The estimator x̂ ( y ) = E [ X|Y = y ] can be computed , but the concentration of the posterior p ( x|y ) around the mean is not in our control . Our solution to this problem starts with an observation that is very intuitive from a Bayesian perspective : one can sharpen the posterior by simply taking more independent noisy measurements . This scheme is formalized by replacing p ( y|x ) with the factorial kernel p ( y|x ) = M∏ m=1 pm ( ym|x ) , ( 1 ) which we name multimeasurement noise model ( MNM ) . Now , the object of interest is a different density which we call M-density obtained by convolving p ( x ) with the factorial kernel : p ( y ) = ∫ p ( y|x ) p ( x ) dx . ( 2 ) This formally maps the original problem of drawing samples from p ( x ) to drawing samples from p ( y ) for any fixed noise level since the estimator of X given Y = y is asymptotically exact . We quantify this for Gaussian MNMs using the plug-in estimator ( the empirical mean of measurements ) . Smooth & Symmetric ! Consider Gaussian MNMs with equal noise level σ in the regime of large σ , large M such that σ √ d/M is “ small ” .1 In that regime , the complex manifold associated with the data distribution is mapped to a very smooth symmetric manifold in a much higher dimensional space . The original manifold can be reconstructed via a single step by computing x̂ ( y ) . Due to equal noise levels , the manifold associated with M-density is symmetric under the permutation group : p ( y1 , . . . , yM ) = p ( yπ ( 1 ) , . . . , yπ ( M ) ) , ( 3 ) where π is a permutation of indices ( Fig . 1 ) . Although we develop a general methodology for studying M-densities , in the later part of the paper we focus on permutation invariant Gaussian M-densities . The paper is organized as follows . In Sec . 2 , we derive Bayes estimators for Poisson and Gaussian MNMs . In Sec . 3 , we present the least-squares objective for learning Gaussian M-densities . We also give a weaker formulation of the learning objective based on score matching . Sec . 4 is devoted to the important topic of parametrization , where we introduce multidenoising autoencoder ( MDAE ) in which we formally connect M-densities to the DAE literature . DAEs have never been studied for factorial kernels and the emergence of MDAE as a generative model should be of wide interest . In addition , we introduce metaencoder formulated in an unnormalized latent variable model , which is mainly left as a side contribution . In Sec . 5 , we present the sampling algorithm used in the paper . In Sec . 6 , we present our experiments on MNIST , CIFAR-10 , and FFHQ-256 datasets which were focused on permutation invariant M-densities . The experiments are mainly of qualitative nature demonstrating the effectiveness of this method in generating fast mixing Markov chains in high dimensions . Related works are discussed in Sec . 7 , and we finish with concluding remarks . 1The regime σ √ d/M 1 is obtained in our analysis of the highly suboptimal plug-in estimator ( Sec . 2.3 ) . Notation . The subscripts are dropped from densities and energy functions when it is clear from their arguments : p ( y ) = pY ( y ) , p ( y|x ) = pY|X=x ( y ) , f ( y ) = fY ( y ) , etc . Bold fonts are reserved for multimeasurement random variables : Y = ( Y1 , . . . , YM ) . The following are shorthand notations : [ M ] = { 1 , . . . , M } and∇m = ∇ym . Throughout , ∇ is the gradient with respect to inputs ( in RMd ) , not parameters . The following convention is used regarding parametric functions : fθ ( · ) = f ( · ; θ ) . Different parametrization schemes come with a different set of parameters , the collection of which we denote by θ . For all the datasets used in the paper , X takes values in the hypercube [ 0 , 1 ] d . 2 FORMALISM : MULTIMEASUREMENT BAYES ESTIMATORS . This work is based on generalizing the empirical Bayes methodology to MNMs . It is well known that the least-squares estimator of X given Y = y ( for any noise model ) is the Bayes estimator : x̂ ( y ) = ∫ xp ( y|x ) p ( x ) dx∫ p ( y|x ) p ( x ) dx . ( 4 ) Next we study this estimator à la Robbins ( 1956 ) for Poisson ( the Poisson kernel was the first example studied in 1956 ) and Gaussian MNMs . In both cases the estimator x̂ ( y ) is derived to be a functional of the joint density p ( y ) . In addition , x̂ ( y ) is invariant to scaling p ( y ) by a constant , therefore one can ignore the partition function in this estimation problem . This is the main appeal of this formalism . Poisson MNMs are included as a warm-up and to demonstrate the generality of the new formalism , but we will not pursue it as a generative model in our experiments for technical reasons due to the challenges regarding sampling discrete distributions in high dimenions ( see Remark 7 ) . 2.1 POISSON MNM . Let X be a random variable taking values in R+ . The Poisson MNM is defined by : p ( y|x ) = e−Mx M∏ l=1 xyl yl ! , yl ∈ N. The numerator in r.h.s . of Eq . 4 is computed next . The measurement index m below is an arbitrary index in [ M ] used for absorbing x such that xp ( y|x ) has the same functional form as p ( y|x ) : ∫ xp ( y|x ) p ( x ) dx = ∫ e−Mx ( ym + 1 ) x ( ym+1 ) ( ym + 1 ) ! ∏ l 6=m xyl yl ! p ( x ) dx = ( ym + 1 ) p ( y + 1m ) , where 1m is defined as a vector whose component l is δml . Using Eq . 4 , it immediately follows x̂ ( y ) = ( ym + 1 ) p ( y + 1m ) p ( y ) , m ∈ [ M ] . ( 5 ) We emphasize that the dependency of x̂ ( y ) on the noise channel ( measurement index ) m that appears on the right hand side of the expression above is only an artifact of the calculation ( we observe this again for Gaussian MNMs ) . The result above holds for any measurement index m ∈ [ M ] , therefore ( ym + 1 ) p ( y + 1m ) = ( ym′ + 1 ) p ( y + 1m′ ) for all m , m′ ∈ [ M ] . Example . We can derive the estimator x̂ ( y ) analytically for p ( x ) = e−x . We first derive p ( y ) :2 p ( y ) = ( ∑ l yl ) ! ∏ l yl ! ( M + 1 ) −1− ∑ l yl , where the sums/products are over the measurement indices l ∈ [ M ] . Using Eq . 5 , it follows x̂ ( y ) = ( ym + 1 ) p ( y + 1m ) p ( y ) = ( ym + 1 ) ∑ l yl + 1 ym + 1 1 M + 1 = ∑ l yl + 1 M + 1 As expected one arrives at the same result by computing Eq . 5 for any measurement index m .

Summary: This paper introduced an alternative sampling method, with an application on generative models, by convolving an unknown distribution $p_x$ with a factorial kernel called multi-measurement noise model (MNM). The resulting M-density $p_y$ is smoother (easier to sample from), and is permutation invariant. Two factorial kernels, Poisson and Gaussian MNMs, are introduced for the convolution, and can be connected to Bayes estimator, and the learning of parametric energy and score functions. Two parameterization schemes are proposed for modeling the energy and score functions of the Gaussian M-density respectively. Empirical results on FFHQ-256 dataset are very impressive. The main contribution is the usage of factorial kernels, which is a direct extension of smoothing a density in non-parametric density estimation, and is not used for generative models before.