Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators

1 Introduction . Deep generative model , as a powerful unsupervised framework for learning the distribution of highdimensional multi-modal data , has been extensively studied in recent literature . Typically , there are two types of generative models : explicit and implicit . Explicit models define a density function of the distribution [ 35 , 51 , 42 ] , while implicit models learn a mapping that generates samples by transforming an easy-to-sample random variable [ 15 , 39 , 2 , 4 ] . Both models have their own power and limitations . The density form in explicit models endows them with convenience to characterize data distribution and infer the sample likelihood . However , the unknown normalizing constant often causes computational intractability . On the other hand , implicit models including generative adversarial networks ( GANs ) can directly generate vivid samples in various application domains including images , natural languages , graphs , etc . Nevertheless , one important challenge is to design a training algorithm that do not suffer from instability and mode collapse . In view of this , it is natural to build a unified framework that takes full advantages of the two models and encourages them to compensate for each other . Intuitively , an explicit density estimator and a flexible implicit sampler could help each other ’ s training given effective information sharing . On the one hand , the density estimation given by explicit ⇤Part of the work was done when the two authors were visiting The Chinese University of Hong Kong , Shenzhen . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . models can be a good metric that measures quality of samples [ 6 ] , and thus can be used for scoring generated samples given by implicit model or detecting outliers as well as noises in input true samples [ 53 ] . On the other hand , the generated samples from implicit models could augment the dataset and help to alleviate mode collapse especially when true samples are insufficient that would possibly make explicit model fail to capture an accurate distribution . We refer to Appendix A for a more comprehensive literature review . Motivated by the discussions above , in this paper , we propose a joint learning framework that enables mutual calibration between explicit and implicit generative models . In our framework , an explicit model is used to estimate the unnormalized density ; in the meantime , an implicit generator model is exploited to minimize certain statistical distance ( such as the Wasserstein metric or Jensen-Shannon divergence ) between the distributions of the true and the generated samples . On top of these two models , a Stein discrepancy , acting as a bridge between generated samples and estimated densities , is introduced to push the two models to achieve a consensus . Unlike flow-based models [ 36 , 26 ] , our formulation does not impose invertibility constraints on the generative models and thus is flexible in utilizing general neural network architectures . Our main contributions are as follows : i ) Theoretically , we prove that our method allows the two generative models to impose novel mutual regularization on each other . Specifically , our formulation penalizes large kernel Sobolev norm of the critic in the implicit ( WGAN ) model , which ensures the critic not to change suddenly on the high-density regions and thus preventing the critic of the implicit model being too strong during training . In the mean time , our formulation also smooths the function given by the Stein discrepancy through Moreau-Yosida regularization , which encourages the explicit model to seek more modes in the data distribution and thus alleviates mode collapse . ii ) In addition , we show that our joint training helps to stabilize the training dynamics . Compared with other common regularization approaches for GAN models that may shift original optimum , our method can facilitate convergence to unbiased model distribution . iii ) We conduct comprehensive experiments to justify our theoretical findings and demonstrate that joint training can help two models achieve better performance . On the one hand , the energy model can detect complicated modes in data more accurately and distinguish out-of-distribution samples . On the other hand , the implicit model can generate higher-quality samples . 2 Background . Energy Model . The energy model assigns each data x 2 Rd with a scalar energy value E ( x ) , where E ( · ) is called energy function and parameterized by . The model is expected to assign low energy to true samples according to a Gibbs distribution p ( x ) = exp { E ( x ) } /Z , where Z is a normalizing constant dependent of . The term Z is often hard to compute , making optimization intractable , and various methods are proposed to detour such term ( see Appendix A ) . Stein Discrepancy . Stein discrepancy [ 16 , 30 , 5 , 38 , 3 ] is a measure of closeness between two probability distributions and does not require knowledge for the normalizing constant of one of the compared distributions . Let P and Q be two probability distributions on X ⇢ Rd , and assume Q has a ( unnormalized ) density q . The Stein discrepancy S ( P , Q ) is defined as S ( P , Q ) : = sup f2F Ex⇠P [ AQf ( x ) ] = sup f2F ( Ex⇠P [ rx log q ( x ) f ( x ) > +rxf ( x ) ] ) , ( 1 ) where F is often chosen to be a Stein class ( see , e.g. , Definition 2.1 in [ 30 ] ) , f : Rd ! Rd0 is a vector-valued function called Stein critic and is an operator that transforms a d⇥ d0 matrix into a scalar value . One common choice of is trace operator when d0 = d. One can also use other forms for , like matrix norm when d0 6= d [ 30 ] . If F is a unit ball in some reproducing kernel Hilbert space ( RKHS ) with a positive definite kernel k , it induces Kernel Stein Discrepancy ( KSD ) [ 17 ] . More details are in Appendix B. Wasserstein Metric . Wasserstein metric is suitable for measuring distances between two distributions with non-overlapping supports [ 2 ] . The Wasserstein-1 metric between P and Q is W ( P , Q ) : = min E ( x , y ) ⇠ [ kx yk ] , T e Sa e Sa e Ge e a De i E i a Wa e ei C i ic S ei C i ic N i e Fa e Sa e T e De i Fa e De iS eiC i ic Implici Model E plici Model S ein B idge Da a F Wi h BP U da e & I ac Figure 1 : Model framework for Stein Bridging . Table 1 : Comparison of objectives between different generative models , where D1 : = D1 ( Preal , PG ) , D2 : = D2 ( Preal , PE ) and D3 : = D3 ( PG , PE ) denote general statistical distances between two distributions . Model Objective GAN [ 15 ] D1 Energy Model [ 35 ] D2 Energy-based GAN [ 55 ] D1 Contrastive Divergence [ 25 ] D2 Cooperative Learning [ 50 ] D2 + D3 Two Generator Game [ 7 ] D2 + D3 Stein Bridging ( ours ) D1 + D2 + D3 where the minimization with respect to is over all joint distributions with marginals P and Q . By Kantorovich-Rubinstein duality , W ( P , Q ) has a dual representation W ( P , Q ) : = max D { Ex⇠P [ D ( x ) ] Ey⇠Q [ D ( y ) ] } , ( 2 ) where the maximization is over all 1-Lipschitz continuous functions . Sobolev space and Sobolev dual norm . Let L2 ( P ) be the Hilbert space on Rd equipped with an inner product hu , viL2 ( P ) : = R Rd uvdP ( x ) . The ( weighted ) Sobolev space H 1 is defined as the closure of C10 , a set of smooth functions on Rd with compact support , with respect to norm kuk H1 : = R Rd ( u 2 + kruk22 ) dP ( x ) 1/2 , where P has a density . For v 2 L2 , its Sobolev dual norm kvk H 1 is defined by [ 10 ] kvk H 1 : = sup u2H1 ⇢ hv , uiL2 : Z Rd kruk22 dP ( x ) 1 , Z Rd u ( x ) dP ( x ) = 0 . The constraint R Rd u ( x ) dx = 0 is necessary to guarantee the finiteness of the supremum , and the supermum can be equivalently taken over C10 . 3 Proposed Model : Stein Bridging . In this section , we formulate our model Stein Bridging . A scheme of our framework is illustrated in Figure 1 . Denote by Preal the underlying real distribution from which the data { x } are sampled . The formulation simultaneously learns two generative models – one explicit and one implicit – that represent estimates of Preal . The explicit generative model has a distribution PE on X with explicit probability density proportional to exp ( E ( x ) ) , x 2 X , where E is referred to as an energy function . We focus on energy-based explicit model in model formulation as it does not enforce any constraints or assume specific density forms . For specifications , one can also consider other explicit models , like autoregressive models or directly using some density forms such as Gaussian distribution with given domain knowledge . The implicit model transforms an easy-to-sample random noise z with distribution P0 via a generator G to a sample ex = G ( z ) with distribution PG . Note that for distribution PE , we have its explicit density without normalizing term , while for PG and Preal , we have samples from two distributions . Hence , we can use the Stein discrepancy ( that does not require the normalizing constant ) as a measure of closeness between the explicit distribution PE and the real distribution Preal , and use the Wasserstein metric ( that only requires only samples from two distributions ) as a measure of closeness between the implicit distribution PG and the real data distribution Preal . To jointly learn the two generative models PG and PE , arguably a straightforward way is to minimize the sum of the Stein discrepancy and the Wasserstein metric : min E , G W ( Preal , PG ) + S ( Preal , PE ) , where 0 . However , this approach appears no different than learning the two generative models separately . To achieve information sharing between two models , we incorporate another term S ( PG , PE ) – called the Stein bridge – that measures the closeness between the explicit distribution PE and the implicit distribution PG : min E , G W ( Preal , PG ) + 1S ( Preal , PE ) + 2S ( PG , PE ) , ( 3 ) where 1 , 2 0 . The Stein bridge term in ( 3 ) pushes the two models to achieve a consensus . Remark 1 . Our formulation is flexible in choosing both the implicit and explicit models . In ( 3 ) , we can choose statistical distances other than the Wasserstein metric W ( Preal , PG ) to measure closeness between Preal and PG , such as Jensen-Shannon divergence , as long as its computation requires only samples from the involved two distributions . Hence , one can use GAN architectures other than WGAN to parametrize the implicit model . In addition , one can replace the first Stein discrepancy term S ( Preal , PE ) in ( 3 ) by other statistical distances as long as its computation is efficient and hence other explicit models can be used . For instance , if the normalizing constant of PE is known or easy to calculate , one can use Kullback-Leibler ( KL ) divergence . Remark 2 . The choice of the Stein discrepancy for the bridging term S ( PG , PE ) is crucial and can not be replaced by other statistical distances such as KL divergence , since the data-generating distribution does not have an explicit density form ( not even up to a normalizing constant ) . This is exactly one important reason why Stein bridging was proposed , which requires only samples from the data distribution and only the log-density of the explicit model without the knowledge of normalizing constant as estimated in MCMC or other methods . In our implementation , we parametrize the generator in implicit model and the density estimator in explicit model as G✓ ( z ) and p ( x ) , respectively . The Wasserstein term in ( 3 ) is implemented using its equivalent dual representation in ( 2 ) with a parametrized critic D ( x ) . The two Stein terms in ( 3 ) can be implemented using ( 1 ) with either a Stein critic ( parametrized as a neural network , i.e. , fw ( x ) ) , or the non-parametric Kernel Stein Discrepancy . Our implementation iteratively updates the explicit and implicit models . Details for model specifications and optimization are in Appendix E.2 . Comparison with Existing Works . There are several studies that attempt to combine explicit and implicit generative models from different ways , e.g . by energy-based GAN [ 55 ] , contrastive divergence [ 25 , 6 ] , cooperative learning [ 50 ] or two generator game [ 7 ] . Here we provide a high-level comparison in Table 1 where we note that the formulations of existing works only consider one-side discrepancy or at most two discrepancy terms . Such formulations can not address the respective issues for both models and , even worse , the training for two models would constrain rather than exactly compensate each other ( more discussions are in Appendix A.3 ) . Differently , our model considers three discrepancies simultaneously as a triangle to jointly optimize two generative models . In the following , we will show that such new simple formulation enables two models to compensate each other via mutual regularization effects and stabilize the training dynamics .

Summary: ===== After Rebuttal =====  I have bumped up my score for the enhanced empirical results author(s) have provided in the discussion round. I remain conservative in my final recommendation because I am not fully convinced the gains are actually from the Stein energy penalty (i.e., some other regularization may also offer similar results) and the cost in modeling complexity may not be justified by the gains.   =====================  This paper proposes to improve generative modeling via joining the strength of unnormalized density estimation (based on Stein discrepancy) and Wasserstein GAN. Numerical experiments verify the proposed solution enjoys better convergence on both synthetic and real-world datasets. Some theories are developed to justify the use of training loss.