Stein Self-Repulsive Dynamics: Benefits from Past Samples

We propose a new Stein self-repulsive dynamics for obtaining diversified samples from intractable un-normalized distributions . Our idea is to introduce Stein variational gradient as a repulsive force to push the samples of Langevin dynamics away from the past trajectories . This simple idea allows us to significantly decrease the auto-correlation in Langevin dynamics and hence increase the effective sample size . Importantly , as we establish in our theoretical analysis , the asymptotic stationary distribution remains correct even with the addition of the repulsive force , thanks to the special properties of the Stein variational gradient . We perform extensive empirical studies of our new algorithm , showing that our method yields much higher sample efficiency and better uncertainty estimation than vanilla Langevin dynamics . 1 INTRODUCTION . Drawing samples from complex un-normalized distributions is one of the most basic problems in statistics and machine learning , with broad applications to enormous research fields that rely on probabilistic modeling . Over the past decades , large amounts of methods have been proposed for approximate sampling , including both Markov Chain Monte Carlo ( MCMC ) ( e.g. , Brooks et al. , 2011 ) and variational inference ( e.g. , Wainwright et al. , 2008 ) . MCMC works by simulating Markov chains whose stationary distributions match the distributions of interest . Despite nice asymptotic theoretical properties , MCMC is widely criticized for its slow convergence rate in practice . In difficult problems , the samples drawn from MCMC are often found to have high auto-correlation across time , meaning that the Markov chains explore very slowly in the configuration space . When this happens , the samples returned by MCMC only approximate a small local region , and under-estimate the probability of the regions un-explored by the chain . Stein variational gradient descent ( SVGD ) ( Liu & Wang , 2016 ) is another type of approximation sampling methods designed to overcome the limitation of MCMC . Instead of drawing random samples sequentially , SVGD evolves a pre-defined number of particles ( or sample points ) in parallel with a special interacting particle system to match the distribution of interest by minimizing the KL divergence . In SVGD , the particles interact with each other to simultaneously move towards the high probability regions following the gradient direction , and also move away from each other due to a special repulsive force . As a result , SVGD allows us to obtain diversified samples that correctly represent the variation of the distribution of interest . SVGD has been found a promising tool for solving difficult sampling problems in which diversity promotion is critical ( e.g. , Feng et al. , 2017 ; Haarnoja et al. , 2017 ; Pu et al. , 2017 ; Liu et al. , 2017 ; Gong et al. , 2019 ) . Various extensions have been developed ( e.g. , Han & Liu , 2018 ; Chen et al. , 2018 ; Liu et al. , 2019 ; Wang et al. , 2019a ) . However , one problem of SVGD is that it theoretically requires to run an infinite number of chains in parallel in order to approximate the target distribution asymptotically ( Liu , 2017 ) . With a finite number of particles , the fixed point of SVGD does still provide a prioritized , partial approximation to the distribution in terms of the expectation of a special case of functions ( Liu & Wang , 2018 ) . Nevertheless , it is still desirable to develop a variant of “ single-chain SVGD ” , which only requires to run a single chain sequentially like MCMC to achieve the correct stationary distribution asymptotically in time , with no need to take the limit of infinite number of parallel particles . In this work , we propose an example of single-chain SVGD by integrating the special repulsive mechanism of SVGD with gradient-based MCMC such as Langevin dynamics . Our idea is to use repulsive term of SVGD to enforce the samples in MCMC away from the past samples visited at previous iterations . Such a new self-repulsive dynamics allows us to decrease the auto-correlation in MCMC and hence increase the mixing rate , but still obtain the same stationary distribution thanks to the special property of the SVGD repulsive mechanism . We provide throughout theoretically analysis of our new method , establishing it asymptotic convergence to the target distribution . As we show in the work , the analysis is highly non-trivial , because our new self-repulsive dynamics is a non-linear , high-order Markov process . Empirically , we extensively evaluate our methods on an array of challenging sampling tasks , showing that our method yields much better uncertainty estimation and larger effective sample size . 2 BACKGROUND : LANGEVIN DYNAMICS AND SVGD . In this section , we give a brief introduction on Langevin dynamics ( Rossky et al. , 1978 ) and Stein Variational Gradient Descent ( SVGD ) ( Liu & Wang , 2016 ) , which we integrate teogehter to develop our new self-repulsive dynamics for more efficient sampling . Langevin dynamics Langevin dynamics is a basic gradient based MCMC method . For some target distribution on Rd with density function ρ∗ ( θ ) ∝ exp ( −V ( θ ) ) , where V : Rd 7→ R is the potential function , the ( Euler-discrerized ) Langevin dynamics simulates a Markov chain with the following rule : θk+1 = θk − η∇V ( θk ) + √ 2ηek , ek ∼ N ( 0 , I ) , where k denotes the number of iterations , { ek } are independent standard Gaussian noise , and η is a step size parameter . It is well known that the limiting distribution of θk when k →∞ approximates the target distribution when η is sufficiently small . Because the updates in Langevin dynamics are local and incremental , new points generated by the dynamics is highly correlated to the past sample . As a result , we need to run Langevin dynamics sufficiently long in order to obtain diverse samples . Stein Variatinal Gradient Descent ( SVGD ) Different from Langevin dynamics , SVGD iteratively evolves a pre-defined number of particles in parallel . Starting from an initial set of particles { θi0 : i = 1 , ... , M } , SVGD updates the M particles in parallel by θik+1 = θ i k + ηg ( θ i k ; δ̂ M k ) ∀i = 1 , . . . , M , where the velocity field , which we denote by g ( θik ; δ̂ M k ) , depends the empirical distribution of the current set of particles δ̂Mk : = 1 M ∑M j=1 δθjk in the following way , g ( θik ; δ̂ M k ) = Eθ∼δ̂Mk −K ( θ , θik ) ∇V ( θ ) ︸ ︷︷ ︸ Confining Term + ∇θK ( θ , θik ) ︸ ︷︷ ︸ Repulsive Term . Here δθ is the Dirac measure centered at θ , and hence Eθ∼δ̂Mk [ · ] denotes averaging on the particles . The K ( · , · ) is a positive definite kernel , such as the Gaussian RBF kernel , specified by users . Note that g ( θik ; δ̂ M k ) consists of a confining term and repulsive term : the confining term pushes particles to move towards high density region , and the repulsive term prevents the particles from colliding with each other . It is the balance of these two terms that allows us to asymptotically approximate the target distribution ρ∗ ( θ ) ∝ exp ( −V ( θ ) ) at the fixed point , when the number of particles goes to infinite . We refer the readers to Liu & Wang ( 2016 ) ; Liu ( 2017 ) ; Liu & Wang ( 2018 ) for throughout theoretical justifications of SVGD . But a quick , informal way to justify the SVGD update is through the Stein ’ s identity , which shows that the velocity field g ( θ ; ρ ) equals zero when ρ equals the true distribution ρ∗ , that is , g ( θ′ ; ρ∗ ) = Eθ∼ρ∗ [ −K ( θ , θ′ ) ∇V ( θ ) +∇θK ( θ , θ′ ) ] = 0 , ∀θ′ ∈ Rd . ( 1 ) This shows that SVGD would converge if the particle distribution already forms a closed approximation to the target distribution p , meaning that the target distributions forms an ( approximate ) fixed point of the update . 3 STEIN SELF-REPULSIVE DYNAMICS . In this work , we propose to integrate Langevin dynamics and SVGD to simultaneously decrease the auto-correlation of Langevin dynamics and eliminate the need for running parallel chains in SVGD . The idea is to use Stein repulsive force between the the current sample and the past samples , hence forming a new self-avoiding dynamics with fast convergence speed . Specifically , assume we run a single Markov chain like Langevin dynamics , where θk denotes the particle at the k-th iteration . Denote by δ̃Mk the empirical distribution of M samples taken from the past iterations , i.e. , δ̃Mk : = 1 M M∑ j=1 δθk−jcη , cη = c/η , where cη is a thinning factor , which scales inversely with the step size η , introduced to slim the sequence of past samples . Compared with the δ̂Mk in SVGD , which is averaged over M parallel particles , δ̃Mk is averaged across time over M past samples . Given this , our Stein self-repulsive dynamics updates the sample via θk+1 ← θk + ( −ηV ( θk ) + √ 2ηek ) ︸ ︷︷ ︸ Langevin + ηαg ( θk ; δ̃ M k ) ︸ ︷︷ ︸ Stein Repulsive , ( 2 ) in which the particle is updated with the typical Langevin gradient , plus a Stein repulsive force against the samples from the previous iterations . α ≥ 0 is a parameter that controls the magnitude of the Stein repulsive term . In this way , the particles are pushed away from the past samples , and hence admits lower auto-correlation and faster convergence speed . Importantly , the addition of the repulsive force does not impact the asymptotic stationary distribution , thanks to Stein ’ s identity in ( 1 ) . This is because when if the self-repulsive dynamics have converged to the target distribution ρ∗ , such that θk ∼ ρ∗ for all k , the Stein self-repulsive term would equal to zero in expectation due to Stein ’ s identity and hence does not introduce additional bias over Langevin dynamics . Rigorous theoretical analysis of this idea is developed in Section 4 . Practical Algorithm Because δ̃Mk is averaged across the past samples , it is necessary to introduce a burn-in phase with the repulsive dynamics . Therefore , our overall procedure works as follows , θk+1 = { θk − η∇V ( θk ) + √ 2ηek , k < Mcη θk + η [ −∇V ( θk ) + αg ( θk ; δ̃Mk ) ] + √ 2ηek , k ≥Mcη . ( 3 ) It includes two phases . The first phase is the same as the Langevin dynamics which collects the initial M samples used in the second phase while serves as a warm start . The repulsive gradient update is introduced in the second phase to encourage the dynamics to visit the under-explored density region . We call this particular instance of our algorithm Self-Repulsive Langevin dynamics ( SRLD ) , self-repulsive variants of more general dynamics is discussed in Section 5 . Remark Notice that , the first phase is introduced to collect the initial M samples . However , it ’ s not really necessary to generate the initial M samples with Langevin dynamics . We can simply use some other initialization distribution and get M initial samples from that distribution . In practice , we find using Langevin dynamics to collect the initial samples is natural and it can also be viewed as the burn-in phase before sampling , so we use ( 3 ) in all of the other experiments . Remark The general idea of introducing self-repulsive terms inside MCMC or other iterative algorithms is not new itself . For example , in molecular dynamics simulations , an algorithm called metadynamics ( Laio & Parrinello , 2002 ) has been widely used , in which the particles are repelled away from the past samples in a way similar to our method , but with a typical repulsive function , such as ∑ j D ( θk , θk−jcη ) , where D ( · , · ) is any notation of dis-similarity . However , introducing an arbitrary repulsive force would alter the stationary distribution of the dynamics , introducing a harmful bias into the algorithm . The key highlight of our approach , as reflected by our theoretical results in Section 4 , is the unique property of the Stein repulsive term , that allows us to obtain the correct stationary distribution even with the addition of the repulsive force . Remark Recently , ( Zhang et al. , 2018 ) proposed a different combination of SVGD and Langevin dynamics , in which the Langevin force is directly added to a set of particles that evolve in parallel with SVGD . Using our terminology , their system is θik+1 = θ i k + ( −ηV ( θik ) + √ 2ηeik ) + ηαg ( θ i k ; δ̂ M k ) , ek ∼ N ( 0 , I ) ∀i = 1 , . . . , M. This is significantly different from our method on both motivation and practical algorithm . Their algorithm still requires to simulate M chains of particles in parallel like SVGD , and was proposed to obtain easier theoretical analysis than the deterministic dynamics of SVGD . Our work is instead motivated by the practical need of decreasing the auto-correlation in Langevin dynamics , and avoiding the need of running multiple chains in SVGD , and hence must be based on self-repulsion against past samples along a single chain . In another recent work , ( Chen et al. , 2018 ) proposed a π-SGLD method , which simulates the linear combination of the evalutionary partial differential equations of SVGD and Langevin dynamics using discrete gradient flow with blob-based method . Their method is again motivated by the theoretical interest of discovering new categories of algorithms , and does not involve self-repulsive on a single chain like our method . An Illustrative Example Here we give an illustrative example to show the key advantage of our self-repulsive dynamics . Assume that we want to sample from a bi-variate Normal distribution shown in Figure 1 . Unlike standard settings , we assume that we have already obtained some initial samples ( yellow dots in Figure 1 ) before running the dynamics . The initial samples are assumed to concentrate on the left part of the target distribution as shown in Figure 1 . In this extreme case , since the left part of the distribution is over-explored by the initial samples , it is desirable to have the subsequent new samples to concentrate more on the un-explored part of the distribution . However , standard Langevin dynamics does not take this into account , and hence yielding a bias overall representation of the true distribution ( see the left panel ) . With our self-repulsive dynamics , the new samples are forced to explore the un-explored region more frequently , allowing us to obtain a much more accurate approximation when combining the new and initial samples .

Summary: This paper proposed another variant of Langevin dynamics, called “Stein self-repulsive dynamics,” which simultaneously decreases the auto-correlation of Langevin dynamics and eliminates the need for running parallel chains in SVGD. They combined Langevin dynamics with Stein variational gradient descent and theoretically justified that the proposed method successfully converges to the stationary distribution with only a single chain, unlike SVGD. The proposed method decreases the auto-correlation of Langevin dynamics, so the proposed method increases the sample efficiency.