Entropy-based Training Methods for Scalable Neural Implicit Samplers
Authors: Weijian Luo
Boya Zhang
Zhihua Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: implicit sampler
learning to sample
generative models
TL;DR: In this work, we propose two novel training methods for scalable and efficient implicit neural samplers.

Abstract: Efficiently sampling from un-normalized target distributions is a fundamental problem in scientific computing and machine learning. Traditional approaches such as Markov Chain Monte Carlo (MCMC) guarantee asymptotically unbiased samples from such distributions but suffer from computational inefficiency, particularly when dealing with high-dimensional targets, as they require numerous iterations to generate a batch of samples. In this paper, we introduce an efficient and scalable neural implicit sampler that overcomes these limitations. The implicit sampler can generate large batches of samples with low computational costs by leveraging a neural transformation that directly maps easily sampled latent vectors to target samples without the need for iterative procedures. To train the neural implicit samplers, we introduce two novel methods: the KL training method and the Fisher training method. The former method minimizes the Kullback-Leibler divergence, while the latter minimizes the Fisher divergence between the sampler and the target distributions. By employing the two training methods, we effectively optimize the neural implicit samplers to learn and generate from the desired target distribution. To demonstrate the effectiveness, efficiency, and scalability of our proposed samplers, we evaluate them on three sampling benchmarks with different scales. These benchmarks include sampling from 2D targets, Bayesian inference, and sampling from high-dimensional energy-based models (EBMs). Notably, in the experiment involving high-dimensional EBMs, our sampler produces samples that are comparable to those generated by MCMC-based methods while being more than 100 times more efficient, showcasing the efficiency of our neural sampler. Besides the theoretical contributions and strong empirical performances, the proposed neural samplers and corresponding training methods will shed light on further research on developing efficient samplers for various applications beyond the ones explored in this study.

Entropy-based Training Methods for Scalable Neural Implicit Sampler
Weijian Luo
Boya Zhang
Zhihua Zhang 1
Abstract
Efficiently sampling from un-normalized target distributions is fundamental problem in scientific computing and machine learning. Traditional approaches such as Markov Chain Monte Carlo (MCMC) guarantee asymptotically unbiased samples from such distributions but suffer from computational inefficiency, par- ticularly when dealing with high-dimensional targets, they require numerous iterations to generate a batch of samples. In this paper, we introduce an efficient and scalable neural implicit sampler that overcomes these limitations. The im- plicit sampler can generate large batches of samples with low computational costs by leveraging neural transformation that directly maps easily sampled latent vec- tors to target samples without the need for iterative procedures. To train the neural implicit samplers, we introduce
novel methods: the KL training method and the Fisher training method. The former method minimizes the Kullback- Leibler divergence, while the latter minimizes the Fisher divergence between the sampler and the target distributions. By employing the two training methods, we effec- tively optimize the neural implicit samplers learn and generate from the desired target distribution. To demonstrate the effectiveness, efficiency, and scalability of our proposed samplers, we evaluate them on three sampling benchmarks with dif- ferent scales. These benchmarks include sampling from 2D targets, Bayesian in- ference, and sampling from high-dimensional energy-based models (EBMs). No- tably, in the experiment involving high-dimensional EBMs, our sampler produces samples that are comparable to those generated by MCMC-based methods while being more than 100 times more efficient, showcasing the efficiency of our neural sampler. Besides the theoretical contributions and strong empirical performances, the proposed neural samplers and corresponding training methods will shed light on further research on developing efficient samplers for various applications be- yond the ones explored in this study.
Introduction
Efficiently sampling from un-normalized distributions is fundamental challenge that arises in var-
ious research domains, including Bayesian statistics I. biology and physies simulations [2] 3.
as well as generative modeling and machine learning 4 5). The task at hand involves generating
batches of samples from a target distribution defined by a differentiable un un-normalized potential function, denoted log g(æ). This problem seeks effective techniques for obtaining samples that accurately represent the underlying target distribution while minimizing computational costs.
Two main classes of methods exist for addressing this challenge. The first class comprises Markov Chain Monte Carlo (MCMC) algorithms 9.. MCMC methods involve the design of Markov Chains with stationary distributions that match the target distribution. By simulating these Markov
School of Mathematical Sciences; Peking University: poneeijiannetu... pku. edu cn;
Academy for Advanced Interdisciplinary Studies; Peking University; zhangboyadpku.. edu. en; I School of Mathematical Sciences; Peking University; zhzhang@math, pku. edu. en;
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Chains, it becomes possible to transition a batch of initial samples towards samples approximately distributed following the target distribution. While MCMC methods offer asymptotically unbiased samples, they often suffer from computational inefficiency, particularly when dealing with high- dimensional target distributions. This inefficiency stems from the need to perform a significant number of iterations to generate a single batch of samples.
The second class of methods, known as learning to sample (L2S) models [10188. focuses on train- ing neural components, such as generative models, to generate samples following the target dis- tribution. L2S models aim to improve the quality and efficiency of sample generation compared to traditional training- free algorithms like MCMCs. Among the L2S models, the neural implicit sampler stands out due to its remarkable efficiency. The neural implicit sampler is an L2S model that incorporates a neural transformation a = (z),,, where & is an casy-to-sample latent vector from known distribution (=))). This transformation allows the sampler to generate samples di- rectly from the target distribution without the need for iterations or gradient operations, which are commonly required in MCMCs. Such a sampler offers significant advantages in terms of efficiency and scalability, particularly when dealing with high-dimensional distributions such as those found in image spaces. However, training a neural implicit sampler poses technical challenges due to the intractability of sampler distributions that are caused by complex neural architectures of go- Achieving effective training requires careful consideration of various factors and the development of appropriate training strategies.
In this paper, we introduce two innovative training approaches, namely the KL training methods and the Fisher training methods, for training implicit samplers. We theoretically demonstrate that our proposed KI. and Fisher training methods are equivalent to minimizing the Kullhack-Leibler divergence and the Fisher divergence [19], respectively, between the sampler and the target distribu- tion. To evaluate the effectiveness, efficiency, and scalability our proposed training approach and sampler, we conduct experiments on three different sampling benchmarks that span various scales. These benchmarks include sampling from 2D targets with low dimensions, Bayesian inference with moderate dimensions, and sampling from high-dimensional energy- models (EBMs) trained on the MNIST dataset. The results of experiments consistently demonstrate the superior performance of our proposed method and the quality of the generated samples. Notably, in the EBM experi- ment, our sampler trained with the KL training method achieves sample quality comparable to that of the EBM model while being more than 100 times more efficient than traditional MCMC meth- ods. These findings highlight the effectiveness, efficiency, and scalability of the proposed approach across different sampling scenarios.
2 Backgrounds
2.1 Generative Models for Sampling
Generative models have emerged as powerful tools in various domains, demonstrating their ability to produce diverse, high-quality samples. They have been successfully applied in tasks such as text-to- image generation (20)28), audio generation [29], video and 3D (reation[30 and even molecule Recently, there has been a growing interest in leveraging generative models for sampling from target distributions.
design 334 35
Specifically, consider the problem that we have access to an un-normalized target distribution ((x), or its logarithm log ((x). Our objective is to train generative models that can effectively generate samples following the target distribution. By learning the underlying structure of the target distri- bution, these generative models can generate samples that capture the characteristics of the target distribution, facilitating tasks such as posterior inference in Bayesian statistics.
Three primary classes of generative models have been extensively studied for sampling tasks. The first class comprises normalizing flows (NFs) [36), while the second class consists of diffusion mod- els (DMs) (37]. NFs employ invertible neural transformations to map Gaussian latent vectors z to obtain samples æ. The strict invertibility of NF transformations enables the availability of likeli- hood values for generated samples, which are differentiable with respect to the model's 's parameters. Training NFs often involves minimizing the KL divergence between the NF and the target distri- bution [12 On the other hand, DMs employ neural score networks to model the marginal score functions of a data-initialized diffusion process. The score functions are learned using techniques related to score-matching 388 DMs have been successfully employed to enhance the sampler per-

---

formance of the annealed importance sampling algorithm [3]] widely recognized MCMC method for various sampling benchmarks. Despite the successes of NFs and DMs, both models have their limitations. The invertibility of NFs restricts their expressiveness, which can hinder their ability to effectively model high-dimensional targets. Moreover, DMs still require a considerable number of iterations for sample generation, resulting in computational inefficiency.
The third class is the implicit generative model. An implicit generative model (IGM) uses flexible neural transform (o).) to push forward easy-to-sample latent vectors 2 Ps to obtain samples = go (z). The main difference between IGMs and NFs that IGMs' neural transformation is not required to be strictly invertible, which unlocks both the flexibility and the modeling power of deep neural networks. For instance, Hu al. [T] proposed train implicit samplers by minimizing the Stein discrepancy between the sampler and the target distribution. The Stein Discrepancy (SD) [40] between p and y defined as
E d=1 dxa f
DSD == sup Ep (V. feF
+
The notation D represents the dimension of data space and fa(.) is the d-th component of the vector- valued function f. The calculation of Stein's 's discrepancy relies on solving a maximization problem w.r.t. the test function f. When the function class JF is carefully chosen, the optimal f may have an explicit solution or easier formulation. For instance, Hu al. [[]] found that if F is taken be F (s: Be (() 0}, the SD is equivalent to a regularized representation
Σ
θ fa(x) {11f(=))13
DsD(P. 9) max Eapp V log ((x),, f(z))
dz
They used two neural networks: ge to parametrize an implicit sampler and f, to parametrize the test function. Let (a denote the implicit sampler distribution induced by (z) with ~ p. (z))). They solved a bi-level mini-max problem on parameter pair (0, v7) to obtain a sampler that tries to minimize the SD with
min m 0
L(O, =Ep (V logg(t))
+ da. fa(x) Al)
Hu et al. opened the door to training implicit samplers by minimizing divergences which are implemented with a two-networks bi-level optimization problem. Since the general Stein's dis- crepancy does not have explicit formulas, in our paper we study training approaches of an implicit sampler by minimizing the KI. divergence and the Fisher divergences. For the implementation of minimization of Fisher divergence, some backgrounds of score function estimation are needed as we put in Section 2.2
2.2 Score Function Estimation
Since the implicit sampler does not have an explicit log-likelihood function or score function, train- ing with score-based divergence requires inevitably estimating the score function (or equivalent component). Score matching [4] and its variants provided practical approaches to estimating score functions. Assume one only has available samples mated distribution 4s (x) to approximate p. Such an approximation can be made by minimizing the Fisher Divergence between p and qo with the definition
p and wants use a parametric approxi-
DFD(P, 96) E<wwp V 1ogp(=)113 V log (x())l 2(V logp(x), V la log qo (x)))
Under certain conditions, the equality
Expp V log p(x), Va log (x)) = -E Exw log (x)
holds (usually referred to as Stein's Identity Here De log (x) E, / d log qo (œ) denotes
the Laplacian operator applied log (z). Combining this equality and noting that the first term
3

---

of FD Eæ-pp VV p())()) does not rely on the parameter o, minimizing (PD(p: 96) is equivalent minimizing the following objective
SS (0) = Eapp V log qo(2))1 + 2 log qs (x)
This objective can be estimated only through samples from thus is tractable when is well- defined. Moreover, one only needs to define a score network (x(x): RD - IR instead of density model represent the parametric score function. This technique was named after Score Matching. Other variants of score matching were also studied 48). Score Matching related techniques have been widely used in the domain of generative modeling, so we use them to estimate the score function of the sampler's distribution.
3 Training Approaches of Neural Implicit Samplers
The neural implicit sampler has the advantage of sampling efficiency and scalability. But the training approach for such samplers still requires to explore. In this section, start by introducing the proposed KL and Fisher training methods for implicit samplers.
3.1 Training Neural Implicit Sampler by Minimizing the KL Divergence
Let g-(-): P2 - R Px be an implicit sampler (i.e., neural transform), the latent distribution, Po the sampler induced distribution z (o(z), and ((x) the un-normalized target. Our goal in this section is to train the implicit sampler by minimizing the KI. divergence between the sampler distribution and the un-normalized target. For a neural implicit sampler, we can efficiently sample from it but the density po has no explicit expression. Our goal is to minimize the KI. divergence between pa and target q in order to train the sampler ga- The KL divergence between pe and is defined as
D (K (pe. EEwpo log (o - log [(x))] = Ez-p, (gggo((g(2)) - log a(ge (z))].
(2)
In order update the sampler ge. we need calculate the gradient of for the divergence equation 2 So we take the gradient of 0 with respect to the KL divergence equation 2 and obtain
① D KIKI (po. 9)
00
(g)(z) òe
[
O log (e
Ezap,
V log logp)(go(z)) V logg(90
+ Ezup.
[x)]
The second term vanishes under loose conditions (we put detailed discussions in the Appendix A.l
1 log Pa (x)
i log pe (x)
Öpe (æ) 09
E Ez-p.
xx) (x)] = Exnpo pp
dx
00
00
/
0
ae po dx)
08
So the gradient of the KL term only remains the first term and the gradient can be calculated with
Ogo (2)
Grad(KL) (0) = Ezmp, [sp(90(2)) sy (9)(
00
where 8, := Ve log ((x) is the score function of target distribution so we have the explicit expres- sion. The 8, = Væ log (x) is the score function of the implicit sampler but do not have the explicit expression. However, thanks to advanced techniques of neural score function estimation as we introduced in 2.22 we can another score neural network to estimate the score function sp(x) x (s(x) with samples consistently drawn from Pa- With the estimated (z)x,, the formula equation 3 provides practical method for updating the implicit sampler that is shown to minimize the KL divergence.
More precisely, we can alternate between score-estimation phase and a KL minimization phase in order minimize the KI. divergence between the sampler and the target distribution. The former phase uses score estimation techniques to train score network ss approximate the sampler's

---

score function with samples consistently generated from the implicit sampler. In the latter phase, we can minimize the KL divergence with gradient-basee optimization algorithms according to gradient formula equation 3
By alternating the optimization of score network and implicit sampler go, one can eventually train an implicit sampler that can approximately generate samples that are distributed according to the target distribution. We name such method the Kl. training method.
3.2 Training Neural Implicit Sampler by Minimizing Fisher Divergence
In the previous section, we proposed KL training method for implicit samplers. In this section, proposed an alternative training method motivated by minimizing the Fisher divergence instead of the KL. We use the same notation as the section 3.1 Recall the definition of the Fisher divergence,
1 DIP (ps. 9) Eye マ V lo
V log (()) Ex-ps 2
11 (o(()(())) s,(ge(z))
The score functions 8e and 8, remain the same meaning as in the equation For an implicit sampler, do not know the explicit expression of sg. In order to minimize the Fisher divergence, take the 0 gradient of equation and obtain
a D (F)((( (pe. 9) E m=)g(=), 00
(s)(2) Ox
(z Ōx
[sp(x)) s,(x)]
dx
00
(8p(2))
(5)
Rzzp, (=(())=)) [g)0
x=00(()
00
Grad F. (0) + (F22(
We 8qr represent a function that does differentiate with respect parameter 0, the first term gradient equation 5 equivalent to taking the gradient of an equivalent objective
((F = Ez-p= || (gol (z)) Sy (=(002)))
(6)
As for the second term of equation
5
we notice that under regularity conditions
Os (x)
a
0 log (o))
log log
00
a0 dæ
8x 00
So the second term of equation
5 equals to
Grad (F:2) (0) Eu-wo
sx)
Ø
logps(x) = / (sa(x)) sy (x) (o a
dx de log)o(z)dz
00 dx
00 Epo
Epo
T (x) (x) (x) + (=( (x) s,(x))
(7)
With the expression of the Grad (F:2)()), minimizing the second part of (0) is equivalent to minimizing an equivalent objective
We put detailed derivation and required conditions in Appendix A.2]
(F:2)
I (0) - Ea-ps %6r (s0 8, (g0()))) + Va(sar (go(z)) no (00[2))]]
(8)
Combining equivalent loss function equation 0 and equation g we obtain a final objective function for training implicit sampler that minimizes the Fisher divergence
((F) (0) (F.1)) (0) + C (F)
(9)
E ag (x) z~ ps
s 11 ||sor (a) 2V [s sor (0))
And we formally define the gradient operator
Grad (0) = 2 (0)
(10)
00
Similar the KL divergence case as discussed in Section 3.1 our overall training approach of such a Fisher divergence minimization consists of two phases: the ecore-estimation phase, and the Fisher divergence minimization phase. The score-estimation phase the same as the one used for KI. training, while the Fisher divergence minimization updates the implicit sampler by minimizing objective equation Since the introduced method aims to minimize the Fisher divergence, we name training the Fisher training for implicit sampler.
5

---

Architecture choice of score network. Notice that for Fisher training, the objective equation needs the calculation of the gradient of the scoring network (z)) so requires the scoring network to be point-wisely differentiable. Commonly used activation functions such as ReI.U or LeakyRel.U do not satisfy the differentiability property. In our further experiments, we use differ- entiable activation functions for Fisher training.
We formally give unified Algorithm for the training of implicit samplers with KL, Fisher, and Combine training in Algorithm ] We use the standard score-matching objective as an example of score estimation phases, other estimation methods also suitable. The notation { and 2d represent the d-th component of the score network so(.) and the data æ, and D means the data dimension. The notation sg(.) represents stopping the parameter dependence of the input.
Algorithm 1: Training Algorithm for Implicit Samplers
Input: un-normalized target log (ax). latent distribution (z)),, implicit sampler go(.), score network (-).), mini-batch size B, max iteration M.
Randomly initialize (0("), ((0)).
for 0:M do
update score network parameter
Get mini-batch without parameter dependence
F; sgigew> (z:)), 2; ~ Pz (z). 1..., B. Calculate score matching related objective:
(m.))
(()) = - Σ
[x [12 + 2
800
Minimize (Ø) to get 1(++1). // update sampler parameter
Get mini-batch samples with parameter-dependeene
2;
gote) (zi), z; P= (=)) 1, B.
Calculate Gradient grad with formula 10
Update parameter 0 with grad and gradient- based optimization algorithms to get ((++1)
end
return (0, ().
3.3
Connection Fisher-Stein's Sampler
In this section, we provide an analysis of the connection between our proposed Fisher training method and Fisher-Stein's sampler proposed in Hu training method used in Fisher-Stein 's sampler is implicitly equivalent to our proposed Fisher train- ing. To establish this connection, we adopt the notations introduced in the introduction of Fisher- Stein's sampler in Section 2.1. We summarize our findings in theorem, and for comprehensive understanding of the proof and analysis, we provide detailed explanations Appendix A..5
[I]] Surprisingly, we discover that the
Proposition Estimating the sampler's score function sa(.) with score matching is equivalent the maximization of test function f of Fisher-Stein 's Discrepancy objective (75). More specially, the optimal score estimation S and Fisher-Stein' 's optimal lest function f satisfy
1
fx(
V log (x:) -5" s=(x))
2X
With Proposition 2 we can prove a Theorem that states that the Fisher-Stein training coincides with our proposed Fisher training.
Theorem 1. Assume f* the optimal test function that maximizes Fisher-Stein's objective 15 Then the minimization part of objective for sampler Go with the gradient-based algorithm is equivalent to the Fisher training, i.e. the minimization objective of shares the same gradient with Grad (F) (0).
Theorem states that Fisher-Stein's sampler and its training method are equivalent to our proposed Fisher training. The two methods differ only the parametrization the score network (and the test function f) and the motivation that they are proposed. It is one of our contributions to point out

---

this equivalence. Our analysis reveals the underlying connection between the two training methods, shedding light on the similarities and shared principles between them. By establishing this con- nection, we provide a deeper understanding of the theoretical foundations of our proposed Fisher training approach and its relationship existing techniques in the field.
Experiments
In previous sections, we have established the KL and Fisher training methods for implicit samplers. Both methods only require access to an un-normalized potential function of the target distribution. In this section, we aim apply our introduced training methods on three sampling benchmarks whose scales vary across both low (2-dimensional targets) to high (784-dimensions image targets), We compare both the implicit sampler and our training methods to other competitor methods and samplers. All experiments demonstrate the significant advantage of the implicit sampler and the proposed training methods on both the sampling efficiency and the sample quality.
4.1 2D Synthetic Target Sampling
Sample quality on 2D targets. In this section, refer to the open-source implementation of Sharrock and Nemeth [49 s1 and train samplers on eight 2D target distributions. We compare our neural samplers with MCMC baselines: Stein variational gradient descent (SVGD) [50], Langevin dynamics (LD) [1, and Hamiltonian Monte Carlo (HMC) [52]; explicit baseline: coupling nor- malizing flow [53]; and 2 implicit samplers: KSD neural sampler (KSD-NS) (II] and SteinGan [54]. All implicit samplers have the same neural architectures, i.c. four layer MLP with 400 hidden units at each layer and ELU activation functions for sampler and score network (if necessary).
The kernelized Stein's discrepancy (KSD) [55] a popular metric for evaluating the sample quality of Monte Carlo samplers [55 40 We evaluate the KSD with IMQ kernel (implemented by an open-source package, the sgmemejax on all target distributions as the metric reported Table
5
Table 1: KSD Comparison of Samplers. If not especially emphasized, we set the Stepsize=0.01, 5te0 num particles=500, num chains=20.
SAMPLER
GAUSSIAN
MOG2
ROSENBROCK
DONUT
FUNNEL
SQUIGGLE
MCMC
SVGD(500)
0,013 + 0.001
0.011 = 0,006
0.053 + 0.002
0.057 + 0.001
0.052 0.001
0.024 + 0.002
LD(500)
007 +: 0.025
0.090 0.100
: 0.008
0.152 + 0.030
0.107
4 0.020
0.1 = 0.029
0.139 + 0.030
HMC(500)
0.0911 # 0.020
0.020
0.134 0.034
0.113
0.020
0.135 0.010
0.135 + 0.033
COUP COUP-FLOW FLOW
0.102 1 0.028
0.158 1 0.019
0.150
0.026
0.239 0.013
0.269 1 0.019
0.130 1 0.026
KSD-NS
0.206 1: 0.043
1.129 0.197
1,531
0.058
0.341 0.030
0.396 = 0.221
0.462 1 0.065
STEINGAN
0.091 + 0.013
0.131 0.011
0.121 + 0.022
0.104 0.013
0.129 0.020
0.124 + 0.018
FISHER-NS
0.013
0.157
0.030
0.179
T.837
KL-NS
0.099 1 0.015
0.104 1 0.015
0.123
1 0.021
0.109 1: 0.015
0.115 0.012
0.118 1 0.024
Settings. For all MCMC samplers, we set the number of iterations to 500, which we find is enough for convergence. For SVGD and LD, we set the sampling step size 0.01. For the HMC sampler, we optimize and find the step size to be 0.1, and LeapFrog updates to 10 work the best. For Cou- pling Flow, we follow Dinh et al. (33] and use 3 invertible blocks, with each block containing a 4-layer MLP with 200 hidden units and Gaussian Error Linear Units (GELU) activations. The total parameters of flow are significantly larger than neural samplers. We find that adding more coupling blocks does not lead to better performance. For all targets, we train each neural sampler with the Adam optimizer with the same learning rate of 2e-5 and default bete. We use the same batch size of 5000 for 10k iterations when training all neural samplers. We evaluate the KSD for every 200 iterations with 500 samples with 20 repeats for each time. We pick the lowest mean KSD among 10k training iterations as final results. Since proposed Fisher neural sampler and KL neural sampler require learning the score network, we find that using 5-step updates of the score network for each update of the neural sampler works well.

---

Performance. Table I shows the numerical comparison between the mentioned samplers. The SVGD performs significanlly the best among all samplers. However, the SVGD has more heavy computational cost when the number of particles grows because its update requires matrix compu- tation for a large matrix. The LDD and HMC perform almost the same. The KL.NNS performs the best across almost all targets, slightly better than LD and HMC on each target. The SteinGAN performs second and is closely comparable to KL-NS. In theory, both the KL-NS and the SteinGAN aim to minimize the KL divergence between the sampler and the target distribution in different ways, so we are not surprised by their similar performances. The Coupling Flow performs overall third, but fails to correctly capture the Rosenbrock target. We believe more powerful flows, such stochastic variants or flows with more complex blocks will lead to better performance, but these enhancements will inevitably bring in more computational complexity. The Fisher-NS performs the fourth, with failure case on the Funnel target. We find that the KSD-NS is hard to tune in practice, which has two failure cases. Besides, the KSDD-NS has relatively high computational cost because it re- quires differentiation through the empirical KSD, which needs large matrix computation when the batch size is large. Overall, the one-shot KL-NS has shown strong performance, outperforming LD and HMC with multiple iterations. We also visualize the sample results on five distributions with characteristics such as multi-modality and periodicity, as shown in Figure 2
Comparison of Computational Costs. To demonstrate the significant advantage of our proposed implicit samplers over other samplers, evaluate the efficiency of each sampler under the same environment to measure the wall-clock inference time. In below Table we summarize the compu- tational costs (wall-clock time) of SVGD, HMC, and LD together with our KL-NS neural sampler to compare the computational costs of each method. The eight targets are analytic targets while the EBM is neural target. Each sampler can generate samples with comparable qualities (i.e. their KSD values are comparable). Table 2 records the wall-clock inference time (seconds) for each sampler when generating 1 samples.
Table 2: Inference Time Comparison of MCMC Samplers and Neural Samplers (seconds). The 2D experiment is conducted on an 8-CPU cluster with PyTorch of 1.8.1, while the EBM experiment is on 1 Nvidia Titan RTX GPU with PyTorch 1.8.1. Unless especially emphasizing, we set the stepsize=0.01, itrratoos000,,, num particles=1000, num repeats=100.
SAMPLER
GAUSSIAN
MOG2
ROSENBROCK
DONUT
FUNNEL
SVGD(500)
26.4224 = 0,6000
26.7897 = 0.5712
26.1088 + 0.4666
26.3632 + 0.
26.0555 = 0.4232
LD(500)
0.1035 0.0003
0.2168 0.0008
0.1284 = 0.0002
0.1100 + 0.0003
0.1339 + 0.0032
HMC(500)
0.3438 1 0.0017
1.7125 1 0.0154
0.6725 1 0.0044
0.3854 1: 0.0015
0.6837 0.0018
KI.-N
0.0014 1 0.0000
0.0014 = 0.0000
0.0014 1 0.0000
0.0014 1 0.0000
0.0014 + 0.0000
The LD and HMC with 500 iterations have comparable (slightly worse) performance than the Neural Sampler, however, their wall-clock time for obtaining Ik samples is significantly larger than Neu- ral samplers. Let's take the simple mixture-of-2 -aatee-of---gauuse --gauusiinn (MOG2) target (with analytic potential and scores) as an example, the neural sampler is 771550000014:122 n.2168/0.0014=55: times faster than LD. If one wants to keep the same inference time for LD and HMC to be comparable with the neural sampler, the LD will only have 500/154=3.2 iterations, while the HMC will only have 500/1223=0A iterations. It is impossible to obtain promising samples with only 3.2 or 0.4 MCMC iterations.
times faster than HMC and
In conclusion, for both analytic and neural targets, the neural samplers show a significant efficiency advantage over MCMC samplers with better (comparable) performances. This shows that neural samplers have potentially wider use on many sampling tasks for which high efficiency and low computational costs are demanded.
4.2 Bayesian Regression
Test accuracy compared with Stein sampler. In the previous section, we validate the sampler on low-dimensional 2D target distributions. The Bayesian logistic regression provides a source of real-world target distributions with medium dimensions (10-100), In this section, we compare our sampler with the most related sampler, Stein's 's sampler that is proposed in Hu et al. [II. We compare the test accuracy of our proposed sampler against Stein's 's samplers and MCMC samplers.
8

---

6
6
3
6
A 5
5
0
6 8
7
0
6
3
2
8
2
8 0 0
6
0
2
J
5 A
6
2
8
B 0
3
7 r 4
5
9
0
6
0007
F 0
7 6
5
Sampler 1NFE
Ebm 5ONFE
Ebm 100NFE
Ebm 150NFE
Ebm 200NFE
Figure| Samples from implicit samplers and energy-based models trained on MNIST datasets.
Our trained sampler only requires one NFE, which is comparable to EBM samples with 100 NFEs.
|1 O
Figure 2: Samples from implicit samplers trained with KL training method on five target distribu- tions. Up: Visualization of target un-normalized density; Below: samples from our trained sampler.
Experiment settings. We follow the same setting Hu et al. [ the Bayesian logistic gression tasks. The Covertype data set [11 has 54 features and 581,01 20122 observations. It has been widely used benchmark for Bayesian inference. More precisely, following the Hu et al. (II), we assume prior of the weights is p(w[o) = A N(w; 0, a a=1) and p(a) = Gamma(o; 1, 0.01). The (80%) and the testing set (20%). Our goal is train
data is randomly split into the training an implicit sampler that can sample from posterior distribution efficiently. The sampler trained with Fisher training the same as the Fisher-Neural Sampler, which achieves the best accuracy, as we have proved in previous sections. The sampler trained with the KL method gives the secondary test accuracy in Table 3 This experiment shows that our proposed training method capable of handling real-world Bayesian inference with medium dimension. After training, the implicit sam- pler can result in the best accuracy while achieving hundreds of times faster than MCMC methods. For detailed settings, please refer to Appendix B
Table 3: Test Accuracies for Bayesian Logistic Regression on Covertype Dataset. The Fisher-NS is equivalent to sampler with our proposed Fisher training, i.e. Fisher-IS.
SGLD
DSVI 73.46% 4.52%
75.09%
0.20%
SVGD 74.76%
STEINGAN 75.37% 0.19%
0.47%
FISHER-NS 76.22% 0.43%
KL-IS(OURS) 75.95% = 0.002%
4.3
Sampling from Energy-based Models
main advantage of an implicit sampler its inference efficiency. This advantage benefits applica- tions for which the default sampling method is inefficient, such as sampling from high-dimensional energy-based models. In this section, we use our proposed KI. training method for an application of learning efficient implicit sampler to sample from pre-trained energy- model.

---

Table 4: Comparison of Sampling Efficiency and Performance.
Model
NFE
Wall-clock Time
FLOPS
FID
KID
EBM
250
0.8455
0.58Gx250
20.95
0.0097
EBM
200
0.6658
0.58Gx200
20.92
0.0111
EBM
150
0.5115
0.58Gx150
21.32
0.0169
EBM
100
0.3455
0.58Gx100
30.35
0.0742
EBM
50
0.1692
0.58Gx50
52.55
0.2061
KL Sampler (ours)
1
0.0012
1.11G
22.29
0.0045
Experiment settings. An energy-based model (EBM) [56] uses a neural network to parametrize negative energy function, i.e. the logarithm of some un-normalized distribution. After training, obtaining samples from an EBM usually requires running an annealed Markov Chain Monte Carlo pre-train deep EBM [57] the MNIST
4 which is computationally inefficient. In this section, dataset and apply our proposed KL. training method for learning a neural implicit sampler to learn sample from the pre-trained EBM efficiently. To take the inductive biases of image data, parametrize our sampler via the stacking of several convolutional neural blocks. The neural sampler takes random Gaussian input with dimension of 128 and outputs a 32x32 tensor which has the same size as the resolution on which the EBM has been trained. We visualize the samples drawn from EBM with 50+ evaluations of the forward and backward pass EBMs with annealed Langevin dynamics and samples from our sampler which only requires single forward pass of the sampler's neural network. We train an MNIST classifier and compute the Frechet Inception Distance (FID) [58 and the KID [59] as an evaluation metric.
Performance. As is shown in Table 4 our sampler generates indistinguishable samples from the EBM but is about 100+ more efficient than annealed Langevin dynamics. To demonstrate the sam- pling efficiency of our sampler, we compare the efficiency and computational costs of EBM and our sampler, by listing some major metrics on computational performance in Table 4 The EBM has a total amount of 9. 9.06M parameters and base FLOPS of 0.58G, while the sampler has 12.59M parameters and a base FLOPS of 1.11G. Generating samples from EBM requires running MCMC chains, which consist of up to 200 neural functional evaluations (NFEs) of the EBM. We com- pare the efficiency and performance of EBM with different NFEs and find that with less than 100 NFEs the EBM can not generate realistic samples, while our sampler only needs one single NFE for good samples. This high-ding nigh-dimensional EBM sampling experiment demonstrates the scalability of KL training methods and their potential for accelcrating large-scale mergg--bseed model sampling.
Limitations and Future Works
We have presented two novel approaches for training an implicit sampler to sampler from un- normalized density. We show in theory that our training methods are equivalent to minimizing the KI. or the Fisher divergences. We systematically compare the proposed training methods against each other and against other samplers (NFs and MCMC). Besides, we conduct an experiment that trains a convolutional neural sampler to learn to sample from a pre-trained EBM. This experiment shows that our implicit sampler and corresponding training have great potential to improve the effi- ciency of modern machine learning models such as energy-based models or diffusion models.
However, there are still limitations to our proposed methods. First, the score estimation is not computationally cheap, so developing more efficient training algorithm by eliminating the score estimation phase is an important research direction. Besides, now the sampler is only limited to sam- pling problems. How to extend the methodology for other applications such as generative modeling is another interesting direction.
Acknowledgments and Disclosure of Funding
This work has been supported by the National Natural Science Foundation of China (No. 12271011) and the Beijing Natural Science Foundation (Z.10001).
'The Fisher training method fails when learning to sample from EBM. Besides, the Fisher training method requires backpropagating through models multiple times, therefore is computationally inefficient.
10

---

References
[1] J. Green, "Reversible jump markov chain monte carlo computation and bayesian model determination," Biometrika, vol. 82, no. 4, pp. 711-732, 1995.
[2] C. Schütte, A. Fischer, W. Huisinga, and P. Deufthard, "A direct approach conformational dynamics based on hybrid monte carlo," Journal of Computational Physics, vol. 151, no. 1, pp. 146-168, 1999.
[3] Olsson, "Two phase transitions in the fully frustrated xy model," " Physical review letters, vol. 75, no. 14. 2758. 1995.
[4] J. Xie, Y. Lu, S.-C. Zhu, and Y. Wu, "A theory of generative convnet," in International Con- ference on Machine Learning. PMLR, 2016, pp. 2635-2644.
[5] C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan, "An introduction to meme for machine learning"" Machine learning. vol. 50, no. I, pp. 43, 2003.
[6] W. Hastings, "Monte carlo sampling methods using markov chains and their applications." Biometrika, vol. 57, no. 1, pp. 97-109, 1970.
[7] G. O. Roberts and J. S. Rosenthal, "Optimal scaling of discrete approximations to langevin dif- fusions." " Journal of the Royal Statistical Sociery: Series B (Statistical Methodology), vol. 60, no. 1, pp. 255-268, 1998.
[8] T. Xifara, C. Sherlock, S. Livingstone, S. Byrne, and M. Girolami, "Langevin diffusions and the metropolis-adjusted langevin algorithm," Statistics & Probability Letters, vol. 91, pp. 14- 19, 2014.
[9] R. M. Neal, "Meme using hamiltonian dynamics," in Handbook of Markov Chain Monte Carlo. Chapman and Hall/CRC, 2011, pp. 139-188.
[10] D. Levy, M. D. Hoffiman, and J. Sohl-Dickster "Generalizing hamiltonian monte carlo with neural networks," in International Conference on Learning Representations, 2018.
[11] T. Hu, Z. Chen, Sun, J. Bai, M. Ye, and G. Cheng. "Stein neural sampler," arXiv preprint arXiv: 1810.03545, 2018.
[12] H. J. Köhler, and F. Noé, "Stochastic normalizing flows," Advances in Neural Information Processing Systems, vol. 33, pp. 5933-5944, 2020.
[13] L. L. di Langosco, V. Fortuin, and H. Strathmann, "Neural variational gradient descent," arXiv preprint arXiv:2107.10731, 2021.
[14] M. Arbel, A. Matthews, and A. Doucet, "Annealed flow transport monte carlo," International Conference on Machine Learning. PMLR, 2021, pp. 318-330.
[15] Q Zhang and Y. Chen, "Path integral sampler: A stochastic control approach for sampling," in International Conference on Learning Representations, 2021.
[16] A. Matthews, M. Arbel, D. J. Rezende, and A. Doucet, "Continual repeated annealed flow transport monte carlo," in International Conference on Machine Learning. PMLR, 2022, pp- 15 196-15 219.
[17] Vargas, W. S. Grathwohl, and A. Doucet, "Denoising diffusion samplers," " in The Eleventh International Conference Learning Representations, 2022.
[18] S. Lahlou, T. Deleu, Lemos, D. Zhang, A. Volokhova, A. Heráández-Gariaa, L. N. Ezzine, Y. Bengio, and N. Malkin, "A theory of continuous generative flow networks," in International Conference Machine Learning. PMLR, 2023, pp. 18 269-18 300.
[19] O. Johnson, "Information theory and the central limit theorem," 2004.
[20] A. Brock, Donahue, and K. Simonyan, "Large scale gan training for high fidelity natural image synthesis," in International Conference on Learning Representations, 2018.
11

---

[21] T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adver- sarial networks." " in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4401-4410.
[22] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, "Analyzing and improving the image quality of stylegan," in Proceedings the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8110-8119.
the
[23] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila, "Alias-free generative adversarial networks," Advances in Neural Information Processing Systems, vol. 34, pp.
2021.
[24] A. Q- Nichol and P. Dhariwal, "Improved denoising diffusion probabilistic models," in Inter- national Conference on Machine Learning. PMLR, 2021, pp. 8162-8171.
[25] P. Dhariwal and A. Nichol, "Diffusion models beat gans image synthesis," Advances in neural information processing systems, vol. 34, pp. 8780-8794, 2021.
[26] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with clip latents."
[27] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans et al., "Photorealistic text-to-image diffusion mod- cls with deep language understanding." Advances in Neural Information Processing Systems, vol. 35, pp. 36 479-36 494,, 2022.
[28] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image syn- thesis with latent diffusion models." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 10 684
[29] R. Huang. J. Huang. D. Yang. Y. Ren, L. Liu, M. Li, Z. Ye, J. Liu, X. Yin, and Z. Zhao, "Make- an-audio: Text-to-audio generation with prompt-enhanced diffusion models," arXiv preprint arXiv:2301. /2661, 2023.
[30] A. Clark, J. Donahue, and Simonyan, "Adversarial video generation on complex datasets," 2019.
arXiv preprint
[3]] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet et al., "Imagen video: High definition video generation with diffu- sion models," arXiv preprint arXiv:22/0.023303 2022.
[32] E. Molad, E. Horwitz, D. Valevski, A. R. Acha, Y. Matias, Y. Pritch, Y. Leviathan, and Y. Hoshen, "Dreamix: Video diffusion models are general video editors," arXiv preprint arXiv:2302.01329, 2023.
[33] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, "Dreamfusion: Text-to-3d using 2d diffu- sion," " in The Eleventh International Conference on Learning Representations, 2022.
[34] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, "Glide: Towards photorealistic image generation and editing with text-guided diffu- sion models," in International Conference on Machine Learning, 2021.
[35] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, "Video diffusion models," arXiv preprint 2:X4v:2204.033458 2022.
[36] D. Rezende and S. Mohamed, "Variational inference with normalizing flows," in International conference machine learning. PMLR, 2015, pp. 1530-1538.
[37] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," in International Conference on Learning Representations, 2021.
[38] A. Doucet, S. Grathwohl, A. G. D. G. Matthews, and H. Strathmann, "Score-based diffusion meets annealed importance sampling," " in Advances in Neural Information Processing Systems, 2022.
12

---

[39] R. M. Neal, "Annealed importance sampling," Statistics and computing, vol. 11, no. 2, pp- 125-139, 2001.
[40] Gorham and L. Mackey, "Measuring sample quality with stein's method," Advances in Neu- ral Information Processing Systems, vol. 28, 2015.
[41] A. Hyvärinen and P. Dayan, "Estimation of non-normalized statistical models by score match- ing." " Journal of Machine Learning Research, vol. 6, no. 4, 2005.
[42] C. M. Stein, "Estimation of the mean of a multivariate normal distribution," " The annals of Statistics, pp. 1135-1151, 1981.
[43] Y. Song, S. Garg, J. Shi, and S. Enmon, "Sliced score matching: A scalable approach den- sity and score estimation," in Proceedings of the Thirty- Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, 2019, p. 204.
[44] P. Vincent, "A connection between putation, vol. 23, no. 7, pp. 1661-1674, 2011.
matching and denoising autoencoders," Neural com-
[45] T. Pang, K. Xu, C. Li, Y. Song, S. Ermon, and J. Zhu, "Efficient learning of generative models via finiteddifference score matching," Advances in Neural Information Processing Systems, vol. 33, pp. 19 175-19 188, 2020.
[46] C. Meng, L. Yu, Y. Song. J. Song, and S. Ermon, "Autoregressive score matching," Advances in Neural Information Processing Systems, vol. 33, pp. 6673-6683, 2020.
[47] C. Lu, K. Zheng, Bao, Chen, C. Li. and Zhu, "Maximum likelihood training for score- based diffusion odes by high order denoising score matching," in International Conference on Machine Learning. PMLR, 2022, pp. 44429-14 460.
[48] Bao, C. Li, K. Xu, H. Su, J. Zhu, and B. Zhang, "Bi-level score matching for learning ameryy-based latent variable models," Advances in Neural Information Processing Systems, vol. 33, pp. 18 110-18 122, 2020.
[49] L. Sharrock and C. Nemeth, "Coin sampling: Gradient- bayesian inference without learn- ing rates," arXiv preprint 23Xi:23301.12290 2023.
[50] Q. Liu and D. Wang, "Stein variational gradient descent: A general purpose bayesian inference algorithm," " Advances in neural information processing systems, vol. 29, 2016.
[51] M. Welling and Y. W. Teh, "Bayesian learning via stochastic gradient langevin dynamics," " in Proceedings of the 28th international conference machine learning (ICML-11), 2011, pp- 681-688.
[52] R. M. Neal et al., "Meme using hamiltonian dynamics," Handbook of markov chain monte carlo, vol. no. 11, p. 2, 2011.
[53] L. Dinh, J. N. Sohl-Dickstein, and S. Bengio, "Density estimation using real nvp," ArXiv, vol. abs/1605.088
2016.
[54] D. Wang and Q. Liu, "Learning to draw samples: With application to amortized mle for gen- erative adversarial learning," arXiv preprint 1rXiv:1611.01722 2016.
[55] Q. Liu, Lee, and M. I. Jordan, "A kernelized stein discrepancy for goodness-off tests." in International Conference on Machine Learning, 2016.
[56] Y. LeCun, S. Chopra, R. Hadsell, A. Ranzato, and F. J. Huang, "A tutorial on energy- learning." 2006.
[57] Li, Y. Chen, and F. T. Sommer, "Learning energy-based models in high-dimensional spaces with multi-scale denoising score matching." arXiv preprint arXiv:/910.07760 2019.
[58] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "GANs trained by a two time-scale update rule converge to local Nash equilibrium," in Advances in Neural Information Processing Systems, 2017, pp. 6626-6637.
13

---

[59] M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton, "Demystifying mmd gans," in International Conference on Learning Representations, 2018.
[60] L. K. Wenliang and H. Kanagawa, "Blindness of score-based methods to isolated components and mixing proportions," arXiv preprini arXiv:2008. 10087, 2020.
[61] W. Gong and Y. Li, "Interpreting diffusion score matching using normalizing flow," in ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
[62] A. Blackard and D. Dean, "Comparative accuracies of artificial neural networks and dis- criminant analysis in predicting forest cover types from cartographic variables," Computers and Electronics in Agricultur,, vol. 24, pp. 131-151, 1999.
[63] M. Titsias and L.ázaro-Greeiilla, "Doubly stochastic variational bayes for non-conjugate inference," in International Conference Machine Learning, 2014.
[64] D. Hendrycks and K. Gimpel, "Gaussian enror linear units (gelus)," arXiv: Learning, 2016.
[65] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," " 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
[66] S. Zagoruyko and N. Komodakis, "Wide residual networks," arXiv preprint arXiv:1605.07146, 2016.
[67] M. F. Hutchinson, "A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines," Communications in Statistics Simulation and Computation, vol. 18, Pp- 1059-1076, 1989.
14

---

A Theory
A.1 Detailed Derivations of Vanishing Term of KL Training Method
Recall the vanishing term
3 log po l==go(x))
E
00
log (a(
Opu(a) (x) da
/ po(x))da 00 00
0 00
00
The equality holds if pe (a( satisfies the conditions (1). pe (x) Lebesgue integrable for x with each 0; (2). For almost all æ E R the partial derivative Öpp ( (z) /0 exists for all E e. (3) there exists an integrable function g(.) R >> R. such that po(x)) S g(ær) for all in its domain. Then the derivative w.r.t & can be exchanged with the integral over æ, i.e.
I a po(x)d=== 0e ʃ po(x))dx.
A.2 Detailed Derivations of Fisher Training Methods
Recall the definition of the Fisher divergence,
1 D(F) D(F(pp, = Eyps 2 Va log)o(x)) Va log (1=) = Be-p, (1100(0(())) (1-0051100(00 - 8y (g(00()))
1
(11)
The score functions and s remain the same meaning as in the equation 3 For an implicit sampler, we do not know the explicit expression of sg. order minimize the Fisher divergence, we take the 0 gradient of equation 4 and obtain
i D [F)
Osp(x) Fx
(s,(2)) Ox
(po, 9) = E m=zo((s) aaps
(so(x)) s,(x)]
00
dx
00
@s (xx
(12)
E Expp. (so(go(2)) (g)(2)))
00
x=
(rad((:)) + (rad(F:2) (8))
We use Sa to represent a function that does not differentiate with respect to parameter 6, so the first term gradient equation is equivalent to taking the gradient of an equivalent objective
((F.)()) = 12z-p / ||s (go(z)) 8y (g(o0(2))13
(13)
As for the second term of equation 5
we notice that under regularity conditions
Oso (x:
a
log pe(x)
log (x))
00 09 dæ
Ox de
15

---

So the second term of equation 5 equals to
Grad (P.2) (0)
= Ex Ex (0(()) - s,(x]]
0 0
log (x)
00 0x
0 a
- Be-py (ss(x) =x)
log (x)
Oz 00
f (so(2)) [x(x)]] po
log pa (a))da
du 00
① log po (a: (mo( 9g (=))))o((=))am) f (185))(/ao [x(2))]
-( 00
+ V (x) [x]]] po
Ø
-0- Eyg 8/ (<)))<())) [x) V=(sa(x)) s,(x0)
(og)((x)
00
/
o/ (z))(a(()) sx(x)) (=))) (c)))) (x)))
Po (x))d
00
0
/
00 Sot (5t) (2) - (x)) + Valsot (z)- sx)) p0(2)
θ Ep
T Agt
x (80
(,)) + V ( - 8g (x))
00
(14)
The notation E Ω means the boundary of distribution density of (2)) and the value pe vanishes on the boundary. Usually, the integral over the boundary Ω of the support vanishes if the density po decays fast enough and the logarithm of density has finite parameter derivatives || 5o log po || for all E O, then the first term vanishes. In practice, we find that the first term of the above expression does not bother the performance of Fisher training, however, for complex circumstances, the first term can not be guaranteed to be exactly O. Besides, the above derivations are for the Fisher training, our proposed KL training method is not bothered by them.
A.3 Additional Discussions on Proposed Algorithms
The training algorithm D consists of two alternative phases for training implicit sampler: the score estimation phase and the generator update phase. The former phase uses score-matching-relatee techniques update the score network to match the score function of the implicit distribution. The latter phase uses the score function of the target distribution and the learned score network to update the generator's 's parameter in order to minimize the KL divergence between the implicit distribution and the target distribution.
Concerns on blindness and ill-landscape. Our proposed methods incorporate a score estimation phase, where a score neural network is used to estimate the score function of the sampler. However, two concerns arise regarding this score estimation step: the blindness issue and the presence of an ill-landscape in score estimation.
Blindness. The blindness (60] a pervasive practical issue of score-based methods that states that two multi-modal distributions could potentially have very small Fisher divergence even if they are highly dissimilar. For our proposed training method that minimizes the Fisher divergence, there exist concerns that blindness would cause the training to fail. However, our intensive study shows that, with the introduced annealing techniques that train samplers a progressive way, the blindness issue does occur.
Ill-landscape. As is pointed out in Gong and Li 61 the parameter landscape of the Fisher diver- gence to heavy-tail distribution can be ill ill-posed, having a unique minimizer but hard to be caught by gradient-basse based optimization algorithms. For our proposed Fisher training, we find that the annealing technique and suitable parametrization of the score network overcome the parameter landscape of training. For straight demonstration, we train implicit sampler to capture the ID student-t distribution, a heavy-tail distribution that has been studied in Gong and Li [6 and find that the sampler is capable of learning such heavy-tail distribution correctly.
16

---

A.4 Details on Ill-Lansscape
Ill-aansscape. As pointed out in Gong and Li 61 the parameter landscape of the Fisher diver- gence to heavy-tail distribution can be iol-posed, having a unique minimizer but hard to be caught by gradient-based optimization algorithms. The student-t distribution, a heavy-tail distribution, is representative distribution that has an ill landscape as is shown in Gong and Li [6]]. Since our Fisher training aims to minimize the Fisher divergence, there is a concern about the ill landscape of the Fisher training. However, for both our proposed KI. and Fisher training, we find that suit- able parametrization of the score network overcomes the ill parameter landscape of training. For straight demonstration, we train an implicit sampler to capture the ID student-t distribution, heavy-tail distribution that has been studied in Gong and Li [61] and finds that the sampler capa- ble of learning such a heavy-tail distribution correctly. More specifically, We plot comparison of the student-t distribution and our learned sampler with Fisher training in Figure
Figure 3: Illustration of proposed Implicit Sampler heavy-tail student-t target distribution.
A.5 Proofs of Equivalence to Fisher-Stein Sampler
We provide proof of Proposition I here.
Proposition 2. Estimating the sampler's 's score function so(.) with score matching is equivalent the maximization of test function f of Fisher-Stein's Discrepancy objective. More specially, the optimal score estimation S and Fisher-Stein' 's optimal test function f* satisfy
(x))
V log -5*
2x
Proof. With fixed p and known target q, the optimal test function f" has representation
f* arg min L(f)
Where functional [(f) has integral representation
[(f) - EaNp ( fogg(z)) + (V., f(x)) (V,, ([))))))
/ (o P(<) f(x)) - 1p (z)
/ 1(x.f, vf)dæ.
Here 1(x, Vf) = f (V log (()) + p(z) - (x)) By Euler-
Lagrange equation, the optimal function f satisfies
Ø1
d
01
0²
0l
+
0.
Of
dx
0r
dx2
By caleulation, have
01
(x) p(x) logg(x) 2xp(x)f(xx)
d
01
dæ 0 (m) = v=p(w))
al
(x) = 0.
Of
17

---

So the optimal f* satisfies the Euler-Lagrange equation as p()) v log ((w)- - 2Ap( Vxp(2:) - 0. Divide the both side with p(a) and note that (ap(a)/p(at) - V log logp(x)) the equation turns to
1 (x) 2A (V - V logp(x)).
Next, consider optimal S The S is obtained by minimizing the Score Matching objective, which equivalent to minimizing the Fisher divergence between p and S induced family, thus the optimal S (ax) V log l(x). Substitute ve logp(æ) with S into f formula, we have
f (x) 2A
V log ((x) - -500)55()))) (²))
Next, we prove Theorem Recall the definition of Fisher-Stein objective min max I(B,)) 7
(15)
L( m) Eps (V= logg + Σ a Ozd
- d=1
(16)
Theorem. Assume f is the optimal test function that maximizes Fisher-Stein's objective 75 Then the minimization part of objective [55] sampler Ge with the gradient-bute algorithm is equiv- alent to the Fisher training, i.e. the minimization objective of [5 shares the same gradient with Grad (F))
Proof. Notice that by Proposition alent to learning score function
the maximization of equation 15 to get an optimal f* is equiv- with the score- matching related techniques, i.e.
(x) = 2X V log ((x) - (x)),
(17)
(²xx) = so (2), Va
(18)
Here (x(x) denotes the unknown score function of the implicit sampler's distribution. Here we use Stein's identity which states
E
falw) -Epo (s0 f(x))
(19)
Oxd
Put the equation
19 and f in objective 15 with S* according to the relation 17
we obtain
min (())
(20)
((0) = Ep Vv logg()), - (8,, fz))) -
(21)
Ep V (x)) (x)) - (x))3
(22)
Ep V log (x))
-X
(23)
Ep, V æ log (xx)
A (x)
(24)
Ep, 2X
(8"
so(x), (x)) so)))))))))ff)))))
(x)
(25)
Ep, 2| (z) (5) s=)
(26)
Taking the 0 gradient of the above objective, we have
00 I(0) x Ep, ff (z) + Epu f' (x)² (s.(
(27)
00
00
1
1
0sp(æ)
Ep V log - S (x))l
Bnd (V., logg(x)) S (x))
4A
00
2X
00
(28)
1
1
En (5*(x) - V. log (xx)) 0sp(æ)
E |S (z)
Va log ( +
4X 20
2X
08
(29)
18

---

In the above equation, some gradient terms vanish because (*(x) sp(x) for all æ. Since the implemented with the implicit sampler, i.c. I po means Gø(z), 2 Pz(z). So above gradient term can be further simplified as
1 a
1 - Ep (5* 2X
V log logg( (sp(x)
Be 1l
1| (x) Ve logg
(30)
4A 00
00
1
E smpp
(Gg()() - V log (G
- Ep (5 (5" - V logg())))
1
(s)(ar)
4A 00
2
00
(31)
1
Ex=),(s)
(0(x)) -x((z))
(a)))))
(s)(a))
dr
(32)
2X smps
Ox
x
08
1 21
(s)(z)
+
pp (5' v V q(x))²
(33)
00
Here (,(x) = V log q So the gradient is the same as the Fisher training (equation (5)) in the paper.
B Experiments
B.1 Experiment Details on 2D Synthetic Sampling
Model architectures. For toy 2-dimensional data experiments, we use 4-layer MLP neural net- work with 200 hidden units in each layer as the sampler. The activation of the sampler chosen as LeakyReLU non- con-linearity with a 0.2 coefficient. The score network is 4-layer MLP with 200 hidden units in each layer. The activation of the score network GELU non- non-linearity.
Evaluation metrics. We calculate the Kemelized Stein Discrepancy with multi-scale RBM kernel as is implemented in codebase /
We set the bandwidth of the kernel to be 0.25.
Comparison of KL and Fisher training methods. In this work, we propose the KL training method and the Fisher training method for training neural implicit samplers. Both methods require two phases which update a neural score network to match the score function of implicit distribution and update the implicit sampler's parameter in order to minimize either the KL divergence or the Fisher divergence. To this end, we give some comparisons of the proposed two training methods. Empirically, we find that the KI. training methods work better than Fisher training under the KSD for 2D targets. We also observe that the Fisher training tends to learn an implicit sampler that collapses on certain modes of the target distribution. Besides, since the Fisher training requires calculating the data derivative of the score function, i.c. V (x) Søt (z)]. thus requires the score function be differentiable. The non-differentiable activate functions such ReLU and LeakyReLU are not suitable for implementing Fisher training. However, the KL training does not suffer from such
issues. Besides, the computational costs of Fisher training are also more expensive than KL training for each iteration.
B.2 Experiment Details on Bayesian Regression
Experiment settings. We use the same setting as [ where the prior of the weights is p(w a) N (w; a)) and p(a)- = Gammaa(; 1, 0.01). The Covertype dataset [62] (581,012 samples with 54 features) is randomly split into the training set (80%) and testing set (20%). Our methods are compared with Stein GAN, SVGD, SGLD, doubly stochastic variational inference (DSVI) (63) II on this data set. For SGLDD, DSVI, and SVGD, the model trained with 3 epochs of the training set (about 15k iterations), while for neural network-based methods (Fisher-NS, KSD-NS, and Stein GAN), the training is run until convergence. Since we have shown the equivalence of Fisher-NS M. we only train the implicit sampler with the KL training method. We set the training learning rate for both the score network and the sampler to be 0.0002. The target distribution (posterior distribution) is computed with 500 random data samples. We set the number of score estimation phases as 2. We evaluate the test accuracy of the logistic regression model with 100 samples from the trained sampler.
KSD-NS, and Fisher-NS
SisceepDiscrpaaccc.o .it
19

---

Average Training Accuracy
0.75
accuracy (%)
0.60
0.55
0.50
20
80
100
number iterations
Figure 4: Test accuracy of KI. sampler with the different number of iterations.
We use the same settings for Bayesian inference as Hu al. [1]. The neural samplers are imple- mented via a 4-layer MLP with 1024 hidden units in each layer and GELU activation functions. The output dimension of sampler 55 and the input dimension is set to be 55x 550, following the same setting as in [1]. The score network has the same neural architecture as the sampler, but the input dimension is set to 55 which matches the output dimension. We use Adam optimizers with learning rate of 0,0002 and default beta values for both sampler and score networks. We use batch size of 100 for training the sampler and use 2 updates for score network for each update the sampler network. We the standard score matching for learning the score network. We train the sampler for 10k iterations for each repeat and use 30 independent repeats to compute the mean and std of the test accuracy. The learning rate of SGLD is chosen to be 0.1/(t 1)0.00 as suggested in Welling and Teh [51). and the average of the last 100 points is used for evaluation. For DSVI, the learning rate le 07, and 100 iterations with bandwidth h calculated by the median trick as Liu and Wang (50), and 100 particles are used for evaluation with step size being 0.05.
used for each stage. For SVGD,
RBF kernel
Model architectures. We use 4-layer MLP neural networks as the sampler and score network, respectively. The activation of the sampler is chosen Gaussian Error Linear Units (GELU) [64] non-linearity for both the score network and the sampler. We set the hidden dimension to 1024. The input dimension the sampler is set to 128.
B.3 Experiment Details Sampling from EBMs
Datasets and model architectures. We pre-train a deep (multi-scale) EBM [57 denoted as Ea(.) with 12 residual layers [65] as is used in DeepEBM S We set the starting noise level and ending noise level to be o minn = 0.3 and mea 3.0. The energy for different noises parametrized with Ex) f The training learning rate is set 0.001. We pre-train the EBM for 200k iterations. Samples from deep EBM are obtained by simulating annealed Langevin dynamics algorithm that is the same as the used in Li et al. [57]. Our implicit sampler is also neural network. The network consists of four inverse convolutional layers which hidden dimensions to be 1024, 512, 256, and 3. For each inverse convolutional layer, we 2D Normaiooraonooatoonn to be 0.2. The prior distribution of the
and LeakyReL.U non-linearity with a leak implicit generator the standard Multi-variate Gaussian distribution. The score network follows
btunss/ggthubboom//engggLLLADDD
20

---

UNet architecture that is adapted from the codebase In order to match the design of the multi-scale EBM, we parametrize the score function in the same way as Sz)x) == S
Training details. We pre-train deep EBM on the MNIST dataset in the house. Then the pre-trained EBM viewed as a multi-scale un-normalized target distribution. We then use the KL training method to update the score network and the generator in order to let the generator match the target distribution. We initialize the generator and the score network randomly. For each training iteration, we randomly select noise E [in, Amar]] We then generate a sample from the generator and add a Gaussian noise with variance a². Then we update the score network with generated samples according denoising score matching. Then we generate a batch of samples and add Gaussian noise with the same variance from the generator and use the gradient estimation in Algorithm [ to update the generator's parameter. We use the Adam optimizer with learning rate of 0.0001 for both the score network and the generator. For the score network, we set the hyper-parametee of the optimizer to be 30 0.9 and 31 0.99. For the generator, set fo and 31 0.99.
Evaluation metrics. To qualitatively evaluate the quality of generated samples, we adapt the def- inition of the Frechet Inception Distance (FID) [58]. We pre-train MNIST image classifier with architecture the WideResNet (66) model with a depth of 16 and widen factor of We then calculate the Wasserstein distance in the feature space of the pre-trained classifier with the same method as the FID.
Additional discussion. In this experiment, we the KL training method to train generator to sample from pre-trained multi-scale EBM. We also try the Fisher training method. But the Fisher training fails to train implicit samplers to sample from EBMs. One of the reasons is that the Fisher training requires calculating the trace of the Jacobian matrix of the score network. For high-dimensional data such image data, such trace term is very expensive. We also try the stochas- tic trace estimation method such as the Skilling-Hutchise the Fisher training, we find that stochastic trace estimation does make the Fisher training work for high-dimensional samplers.
trace estimation technique [67]. But for
'https: www.eeawwww..eteeeeoooooeoeeeeee
ipynb
21


Reviews:
[{"review_id": "MvIOyrMXaJ", "reviewer": "Reviewer_Dhq1", "summary": "The paper is the area of approximate inference. The goal is to use approximating distributions where the density is unknown but samples can be drawn. The advantage of this approximating class is that it is potentially more expressive and easier to work with ones where the density is known/tractable.  The paper considers approximating distributions based on two distortion measures - namely the Fisher Divergence and the reverse KL divergence.  In both cases the gradient updates are written in terms of the score of the approximating distribution. This unknown score is then approximated using score matching using samples from the distribution. An equivalence is derived with the Fisher Stein Sampler of Hu et al. for one variant of the algorithm under optimality assumptions. Experiments are carried out including a comparison to EBM inference on MNIST.", "questions": "I have no questions.", "limitations": "I see no negative societal impact.\n\nThere is some discussion of limitations but I already mentioned some things I would have liked to see more discussion of.  Please see the weaknesses section above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I think the basic idea is interesting. Certainly it is true that implicit distributions offer some advantages over more tractable approximating classes if one can deal with the resulting challenges and although there is work in this area (see below) it is less explored than for example normalizing flows. Also I think there is room for improvement on the existing approaches and with more work this approach might provide such an improvement. \n\nWhilst it is not a big jump in methodology conceptually I’m not aware of this exact approach in the prior literature and the smaller details can make a big difference in terms of improved performance.\n\nI think equivalence described with the Fisher Stein sampler and the Fisher divergence variant of the proposed method is interesting. I also think it is not a dead end in terms of the potential of this proposed algorithm variant. Since the equivalence relies on idealized optimality of the witness function the two methods may well differ in practice and this one might be better. So I would encourage the authors to explore this further. \n\nThere are quite a few baselines compared to in the Bayesian logistic regression section although the details are somewhat sparse.", "weaknesses": "**Unfortunately, the literature review is missing quite a few relevant references.** The papers are significant enough to the proposed method that this is a significant limitation of the submission as it stands. Here are is a non-exhaustive list of examples:\n\n*Unbiased Implicit Variational Inference. Titsias and Ruiz. AISTATS 2019.*\n\n*Semi-implicit variational inference. Yin and Zhou. ICML 2018.*\n\n*Semi-Implicit Variational Inference via Score Matching. Yu and Zhang. ICLR 2023.*\n\nThis group of papers is on the topic of variational inference with implicit distributions. The methods are somewhat distinct from the proposed submission but they are close that they should be discussed in the text and possibly compared to.\n\n*Operator Variational Inference. Ranganath, Tran and Blei. NeurIPS 2016.* This highly cited work from NeurIPS is significantly earlier than some of the references in the submitted text. The paper uses Stein divergences and highlights the benefit of using implicit distributions. \n\n**Existence of approximating density** One theoretical concern is as to whether the implicit approximating distributions actually have a density at all. For example the experiment in Section 4.3 the dimension of the inputs random variables is 128 but the output space is of dimension 32 x 32. So the samples will fall on a lower dimensional manifold of the output space and no have density with respect to the full Lebesgue measure. Note that this is a different issue from the density existing but being unknown/intractable. I think with thought and effort this issue could probably be tamed but it should be considered and discussed in a paper with this topic. \n\n**The submitted experiments are not entirely reproducible.** Take for example the toy experiment in Section 4.1. No details are given for the baselines in either the main text or the supplement. Hamiltonian Monte Carlo is a gold standard asymptotically exact method which should work well for unimodal distributions in low dimensions. It is therefore not credible for the banana experiment that the proposed algorithm could be better unless the computational budget is constrained or the algorithm is mistuned. No details are given of the tuning or computational budget.  Similarly coupling flows are in principle capable of modelling simple low dimensional distributions. There are many possible reasons why they did not in this case but since there are few details given it is not possible to know why. For Section 4.2 the details of the baselines are lacking. The tuning of these algorithms is challenging and can substantially affect their performance. Submitting the code would have helped a lot here.\n\n**The experiments do not cover all relevant questions.** In Section 4.3 whilst I can see that the sampling is faster than annealed Langevin dynamics there are several methods that would plausibly be able to learn a one shot sampler and these are not compared to.\n\nWhilst I acknowledge some discussion of the computational bottlenecks of the method, I would have liked more. For example the training time in the EBM method must be large and is not included in the comparison to the annealed Langevin dynamics. \n\nSince learning point estimates of parameters along side inference is a big advantage of variational approaches I would have liked to see an example of this. For example, learning the parameters of the EBM rather than just using a pretrained one. \n\n**Smaller points:**\n\nEquation following Line 146: The result you are proving here is that the expectation of the score is zero. This is a very well known result. For instance it often comes up when discussing the Fisher Information matrix. So you could spend less of your precious space in the main text proving it. \n\nLine 305: \"Besides, now the sampling is only limited for sampling problems\" The meaning was unclear for this sentence. This has not affected my review score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback, we will address your concerns one by one.\n\n**Q1**. missing quite a few relevant references.\n\n**A1**. Thank you for the reminder. We are sorry for the loss of discussion of VI methods with implicit distributions. We agree that these VI methods have strong relations to neural samplers, and have significant contributions to research of implicit distributions. We will incorporate the discussion of them in the revision. \n\n**Q2**. Existence of approximating density\n\n**A2**. We agree that the intrinsic low-dimensional manifold for an implicit distribution is a common issue when using implicit distributions that have different input and output dimensions. However, in practice, the low-dimensional manifold issue could be overcome by adding slight noise to samples from implicit distributions. Overall, we thank your reminder and will incorporate a discussion on such an issue in the revision. \n\n**Q3**. Experiments are not entirely reproducible. Details of Bayesian inference experiment.\n\n**A3**. Thank you for your suggestions. In order to make a stronger comparison, we refs the open-source implementation of [4] and run a new comparison experiment on 2D targets. In this experiment, we compare our KL-NS and Fisher-NS with 3 MCM baselines: SVGD, LD, and HMC; 1 explicit baseline: coupling flow; and 2 implicit samplers: KSD-NS and SteinGan. Due to the limitation of words, we put the results and details of the new experiment in **Table 1** in the global author rebuttal cell part. Below we (1) analyze the results of the new experiment to compare our sampler with both MCMCs and other neural samplers and (2) give details on Bayesian inference experiments in Section 4.2.\n\n(1) The result shows that when compared with MCMC samplers, our KL-NS performs comparable with (slightly better than) LD and HMC samplers, while the sampling speed is over 500 times faster than MCMC samplers (1 versus 500 iterations). Compared with other one-shot neural samplers, the KL-NS performs significantly the best across all targets. This shows that the KL-NS is a strong one-shot neural sampler with high efficiency. Besides, we are willing to compare more neural-enhanced MCMC samplers to get a more thorough understanding of our one-shot neural samplers.\n\n(2) Details on Bayesian inference: In section 4.2, we use the same settings for Bayesian inference as in [1]. The neural samplers are implemented via a 4-layer MLP with 1024 hidden units in each layer and GELU activation functions. The output dimension of the sampler is 55 and the input dimension is set to be 55x10 = 550, following the same setting as in [1]. The score network has the same neural architecture as the sampler, but the input dimension is set to 55 which matches the output dimension. We use Adam optimizers with a learning rate of 0.0002 and default beta values for both sampler and score networks. We use a batch size of 100 for training the sampler and use 2 updates for score network for each update of the sampler network. We use the standard score matching for learning the score network. We train the sampler for 10k iterations for each repeat and use 30 independent repeats to compute the mean and std of the test accuracy. The learning rate of SGLD is chosen to be $0.1/(t + 1)^{0.55}$ as suggested in [2], and the average of the last 100 points is used for evaluation. For DSVI, the learning rate is 1e − 07, and 100 iterations are used for each stage. For SVGD[3], we use RBF kernel with bandwidth h calculated by the \"median trick\" as in [3], and 100 particles are used for evaluation with step size being 0.05. \n\n**Q4**. experiments do not cover all relevant questions\n\n**A4**. To explore KL-NS and Fisher-NS more thoroughly, we tried to implement an experiment to compare KSD-NS and SteinGAN on EBM experiments in Section 4.3 of the main text. We adapt the open-source Tensorflow implementation of KSD-NS and use it to the MNIST image. For the implementation of SteinGAN, we adapt the open-source Theano implementation. We find that the Fisher-NS and KSD-NS do not converge for high-dimensional MNIST targets, so the methods are not scalable for high dimensions in practice. This limitation of scalability has also been discussed in [1]. Besides, we find that SteinGAN shows strong mode-collapse behavior, for which the generator collapse to certain modes of EBM distribution such as number \"5\" or number \"1\". We implement the SteinGAN with pixel-space RBM kernel, by viewing each image as a vector. Maybe a more complex \"image space\" kernel can resolve the mode-collapse issue, and we plan to continue to explore the comparison of SteinGAN and KL-NS.\n\n**Q5**. The neural sampler requires additional training. \n\n**A5**. We acknowledge that neural samplers' additional pre-training phase is a limitation. This issue is faced by all other neural samples. However, from another perspective, the computational costs for pre-training will be amortized to each inference of the neural sampler. So if the trained neural sampler is used for sampling infinitely times, the costs for its pre-training can be ignored.\n\n**Q6**. Experiment for training EBM.\n\n**A6**. The goal of this work is to train a neural sampler to learn to sample from target distribution, so we think that training an EBM from scratch is not proper for evaluating the proposed method, because multiple aspects such as EBM architecture and hyper-parameters could potentially influence the EBM's convergence. \n\n**Q7-8**. Small points.\n\n**A7-8**. Thanks for the suggestion, we consider re-organizing the representation in the revision. \n\nThank you for your valuable comments. We hope our answers have resolved your concerns, and if you still have any concerns, please do let us know.\n\n[1] Stein Neural Samplers\n\n[2] Bayesian Learning via Stochastic Gradient Langevin Dynamics\n\n[3] Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm\n\n[4] Coin Sampling"}, {"review_id": "tW2wflPs87", "reviewer": "Reviewer_MaqB", "summary": "This submission proposes two training algorithms for implicit samplers, which are based on KL divergence and Fisher divergence, respectively. Tractable objective estimator are derived and practical training algorithms are demonstrated. Numerical experiments are conduct in several different settings.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The writing is clear for most of the parts. The idea of implicit sampler has been studied before but has not been studied under the perspective of this article, especially with Fisher divergence. The derivation is sound mathematically. The experiments are done in a proper way.", "weaknesses": "The reviewer has the following issues.\n\nFor the second part of gradient estimator of Fisher divergence, can we directly use\n${sg}\\[s_\\theta(g_\\theta(z) - s_q(g_\\theta(z))\\]^{T}s_\\theta(x)$? Here \"sg\" means stop_gradient.\n\nThere are a bunch of missing baselines and benchmarks (at least citations). To name a few:\n- PIS http://arxiv.org/abs/2111.15141\n- AFT http://arxiv.org/abs/2102.07501\n- CRAFT http://arxiv.org/abs/2201.13117\n- DDS http://arxiv.org/abs/2302.13834\n- GFlowNet https://arxiv.org/abs/2301.12594\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful feedback. We will address your concerns one by one in the following paragraphs.\n\n**A1**. Thank you for your reminder. The notation $sg$ does mean the stop-gradient. We feel sorry for the confusion and will refine the notations in the revision. \n\n**A2**. We agree that there are many existing methods that also use neural networks for sampling. We will incorporate a discussion of them in the revision. To better explore the pros and cons of our proposed neural samplers, we conduct a new experiment on 2D target distributions to compare our proposed KL-NS and Fisher-NS with 2 one-shot neural samplers, the KSD-NS and SteinGAN, 1 explicit neural sampler, the RealNVP coupling flow [3], and 3 MCMC samplers, the SVGD, LD and the HMC. We put the results and analysis in **Table 1** in the global author rebuttal cell. \n\nThe result shows that when compared with MCMC samplers, our KL-NS performs comparable with (slightly better than) LD and HMC samplers, while the sampling speed is over 500 times faster than MCMC samplers (1 versus 500 iterations). Compared with other one-shot neural samplers, the KL-NS performs significantly the best across all targets. This shows that the KL-NS is a strong one-shot neural sampler with high efficiency. Besides, we are willing to compare more neural-enhanced MCMC samplers to get a more thorough understanding of our one-shot neural samplers.\n\nThank you for your valuable comments. We hope our answers have resolved your concerns, and if you still have any concerns, please do let us know."}, {"review_id": "hYVF2vPBUb", "reviewer": "Reviewer_cgAz", "summary": "Efficient sampling from unnormalized target distribution is of crucial interest in Bayesian inference. Classic methods such as Markov Chains Monte Carlo sampler provide unbiased samples but could be computationally expensive. This paper proposes a novel approach called neural implicit sampler to employ a neural transformation and leverage generative models to sample from target distribution. The authors have developed two innovative inference methods: the KL training method, which minimizes the Kullback-Leibler divergence and the Fisher Training method, which minimizes the Fisher divergence. The methods are evaluated on three benchmark cases including 2D target sampling, Bayesian inference and high-dimensional energy-based models. Using metrics Frechet Inception Distance, the proposed method is over 100 times more efficient than EBM's sampler. ", "questions": "1. In Table 1, the authors use HMC as a competing method. Since HMC's performance is very sensitive to the two tuning parameters epsilon and L, have the authors picked these two parameters to optimize the performance of HMC?\n\n2. Can the authors make their code repository publicly open if the paper is accepted?\n\n3. In [1] , a metric KID similar to FID is proposed, which shows to converge quickly to its presumed true value than FID. Can the authors evaluate their method using KID as well to see if the conclusions are consistent with FID? \n\n[1] Bińkowski, Mikołaj, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. \"Demystifying mmd gans\", 2018 ICLR. ", "limitations": "Currently, the sampling approach is only limited for sampling problems. It is interesting to see if the method can be applied for generative modeling or image translation problems. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has derived two novel training approaches for neural implicit sampler via KL divergence training and Fisher divergence training. The authors have also provided a theoretical analysis of the connection between their proposed Fisher training method and Fisher-Stein's sampler proposed in Hu. The authors also have evaluated their methods via three empirical benchmark studies and it is impressive that the proposed sampler can produce samplers with similar quality while being 100 times more efficient, which demonstrates the efficiency of the proposed method.  ", "weaknesses": "I do not find clear weakness of the paper. I just have some minor comments:\n\n1. In Table 1, the authors use HMC as a competing method. Since HMC's performance is very sensitive to the two tuning parameters epsilon and L, have the authors picked these two parameters to optimize the performance of HMC?\n\n2. Can the authors make their code repository publicly open if the paper is accepted?\n\n3. In [1] , a metric KID similar to FID is proposed, which shows to converge quickly to its presumed true value than FID. Can the authors evaluate their method using KID as well to see if the conclusions are consistent with FID? \n\n[1] Bińkowski, Mikołaj, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. \"Demystifying mmd gans\", 2018 ICLR. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful feedback. We will address your concerns one by one in the following paragraphs.\n\n**A1**. In the experiment of Section 4.1, we optimize the HMC to get the step size and LeapFrog iterations. Besides, in the rebuttal period, we conduct a new experiment on 6 more 2D targets. In this experiment, we compare our proposed KL-NS and Fisher-NS with 2 one-shot neural samplers, the KSD-NS and SteinGAN, 1 explicit neural sampler, the RealNVP coupling flow, and 3 MCMC samplers, the SVGD, LD, and the HMC. We put the results and analysis in **Table 1** in the global author rebuttal cell. \n\nThe result shows that when compared with MCMC samplers, our KL-NS performs comparable with (slightly better than) LD and HMC samplers, while the sampling speed is over 500 times faster than MCMC samplers (1 versus 500 iterations). Compared with other one-shot neural samplers, the KL-NS performs significantly the best across all targets. This shows that the KL-NS is a strong one-shot neural sampler with high efficiency. Besides, we are willing to compare more neural-enhanced MCMC samplers to get a more thorough understanding of our one-shot neural samplers.\n\n**A2**. We promise to release our code if the paper is accepted.\n\n**A3**. Thank you for the useful suggestion. We will add the comparison of KID and FID values in the revision. \n\nThank you for your valuable comments."}, {"review_id": "xwOFOX9qyF", "reviewer": "Reviewer_wbpF", "summary": "This article studies the problem of learning implicit samplers, for which the score function is not available, and therefore the KL (or the Fisher) divergence cannot be explicitly computed. They introduce an alternating procedure, where, for a fixed sampler parameter $\\theta$ they first learn the surrogate score $s_{\\phi}$ of the implicit sampler, and then use this score to minimize the divergence with respect to the target. An equivalent between Fisher divergence training and the min-max approach to minimizing Stein's discrepancy is proven. The attractivity of the method is illustrated on standard benchmarks.\n\n", "questions": "### What you can do to improve my score\nThe empirical evaluation is, in my opinion, the weakest part of this, otherwise, well-written and motivated article. Effectively comparing with the relevant sampling literature would improve the empirical soundness and make for a stronger acceptance case.\n\n### Some questions/remarks\n- I am very confused by the paragraph on performance l.287 onwards. Can you please reformulate it?\n- It would be good to explain how you mimic the FID exactly.\n\n#### Minor comments\n- l43-44: please add a specific reference when introducing the neural implicit sampler.\n- [24] is by Hyvärinen only. Dayan was the editor of the paper. This seems to come from Google Scholar. Please do verify the rest of the citations for similar issues with Google Scholar referencing.\n- In some places, you use \"sg\" vs. the dagger superscript to denote the stop-gradient operator, it would be better to stick to one only.\n- I don't really see the point of A.1, this is well-known (for instance from the log-derivative trick) and should be replaced by a citation in this effect. Also, what is the point of putting in the conditions for Lebesgue's theorem to apply if you don't verify them for your models?\n- Please use \\eqref rather than \\ref for equation referencing.\n\n\n### Typos\nA didn't notice many\n- Algorithm 1: dependecne\n- Capitalization in citations is off (people's names are often written without upper case: stein -> Stein, markov -> Markov, etc)\n\n\n", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This article is an original application of score-matching methods to implicit samplers. The procedure is well-motivated, grounded, and, to the best of my knowledge novel. The method is also fairly significant in its somewhat general applicability. The sampling application is relevant to a large part of the community. The article is further pretty well written and only requires minimal changes from a presentation/clarity standpoint.", "weaknesses": "The performance of the method is evaluated only in terms of sampling accuracy, never in terms of the total cost *including training*. As a consequence, I don't believe that HMC and SGLD are fair competitors given that they provide samples immediately, with no slow training procedure. \n\nMore related are the works considering warping the target distribution, such as https://arxiv.org/abs/1903.03704, https://arxiv.org/abs/2107.08001, https://invertibleworkshop.github.io/INNF_2020/accepted_papers/pdfs/24.pdf, https://arxiv.org/abs/2210.10644.\n\nUnderstanding the competitiveness of the proposed approach with respect to these more related methods both in terms of ease of training and in terms of samples quality post training would make the article a lot stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. We will address your concerns one by one in the following paragraphs. \n\n**Q1-2**. Stronger evaluation. Compared to other baselines.\n\n**A1 and A2**. \n(1) It is truly a limitation that neural samplers require additional training, so we think it necessary to compare our neural samplers with other neural samplers other than MCMCs.  We also agree that your mentioned work, i.e. neural-enhanced MCMC samplers, are strong competitors which use (invertible) neural mapping to enhance MCMC algorithms for faster mixing or improved performances. However, due to the time limitations of the rebuttal period, we choose to first compare our one-shot neural samplers to other alternative one-shot neural samplers such as KSD-NS and SteinGAN to show the pros and cons of our neural samplers. We are willing to compare more neural-enhanced MCMC samplers in the revision. To this end, we conduct one more experiment on 6 more 2D target distributions. The results, details, and analysis are summarized in **Table 1** in the global author rebuttal cell. In the new experiment, we compare our neural samplers with 3 MCM baselines: SVGD, LD, and HMC as MCMC samplers; 1 explicit baseline: coupling flow model; and 2 implicit samplers: KSD-NS and SteinGan. \n\nThis experiment shows that the trained neural samplers perform comparable with (slightly better than) LD and HMC samplers, while the sampling speed of neural samplers is 500 times faster than MCMC samplers (1 versus 500 iterations). Compared with other one-shot neural samplers, the KL-NS performs significantly better across all targets. We are willing to compare more neural-enhanced MCMC samplers to get a more thorough understanding of our one-shot neural samplers in the revision. \n\n(2) Besides the experiments on low-dimensional 2D targets, we also run another new experiment that tried to compare KL-NS and Fisher-NS with alternative KSD-NS and SteinGAN on high-dimensional EBM targets as in Section 4.2 in the main text. However, we find that the Fisher-NS and KSD-NS do not converge for high-dimensional MNIST targets, so they are considered not scalable for high dimensions targets in practice. This limitation of scalability has also been discussed in KSD-NS's original paper [1]. Besides, we find that SteinGAN shows strong mode-collapse behavior, that the generated samples collapse to a certain model of the target distribution, such as the number \"5\" or \"1\" in the MNIST target. We implement the SteinGAN with pixel-space RBM kernel, by viewing each image as a vector. Maybe a more complex \"image space\" kernel can resolve the mode-collapse issue, and we plan to continue to explore the comparison of SteinGAN and KL-NS. Overall, we find that KL-NS is not bothered by mode-collapse behavior when scaled to image space, while the SteinGAN currently is bothered by mode-collapse behavior.\n\n**Q3**. paragraph on performance l.287 onwards\n\n**A3**. In line 287 we mean to show the computational costs of sampling by running a one-shot neural sampler and running annealed Langevin dynamics MCMC to sample from the neural EBM. The Langevin dynamic requires the score function in each iteration, so we need to forward pass the neural EBM and then backward through it to obtain the score function. The FLOPs account for the computational cost (the number of operations) of the forward pass of a neural network. In Table 3 in the main text, we show that the trained one-shot neural sampler has a total FLOP of 1.11G, which is about 80 times more efficient than using annealed Langevin MCMC to sample from the EBM (0.58G×150). \n\n**Q4**. It would be good to explain how you mimic the FID exactly.\n\n**A4**. We mimic the concepts of FID by pre-training a wide-resnet classifier [4] on MNIST data with 99+ pct accuracy. We then extract the features (features before feeding into the final linear layers) from the pre-trained classifier and compute the Wasserstein distance between generated samples and the training data.\n\n**Q5**. Writing and references\n\n**A5**. We thank you for your suggestions on the writing and references of our work. We will revise them in the revision. \n\nThank you for your valuable comments. We hope our answers have resolved your concerns, and if you still have any concerns, please do let us know.\n\n[1] Stein Neural Samplers\n\n[2] Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning\n\n[3] Coin Sampling: Gradient-Based Bayesian Inference without Learning Rates\n\n[4] Wide Residual Networks "}]

Author rebuttal:
Thank all reviewers for your valuable feedback. In the rebuttal period, we run a new comparison experiment on 2D targets with a reference of [1], and report the results in **Table 1**. 

In this new experiment, we compare our neural samplers with 3 MCM baselines: SVGD[2], LD, and HMC; 1 explicit baseline: coupling flow; and 2 implicit samplers: KSD-NS[3] and SteinGan[4]. All implicit samplers have the same neural architectures, i.e. four layer MLP with 400 hidden units at each layer and ELU activation functions for sampler and score network (if necessary). We evaluate the KSD with IMQ kernel (implemented by an open-source package sgmcmcjax) on all target distributions as the performance metric reported in **Table 1**. 

**Table1**. KSD Comparison of Samplers

stepsize=0.01, iterations=500, num particles=500, num chains=20
| Sampler   | Gaussian          | MOG2              | Rosenbrock        | Donut             | Funnel            | Squiggle          |
| :------:  | :------:          | :------:          | :------:          | :------:          | :------:          | :------:          |
| MCMC Samplers | | | | | | |
| svgd(500) | 0.013 $\pm$ 0.001 | 0.044 $\pm$ 0.006 | 0.053 $\pm$ 0.002 | 0.057 $\pm$ 0.004 | 0.052 $\pm$ 0.001 | 0.024 $\pm$ 0.002 |
| ld(500)   | 0.107 $\pm$ 0.025 | 0.099 $\pm$ 0.008 | 0.152 $\pm$ 0.030 | 0.107 $\pm$ 0.020 | 0.116 $\pm$ 0.029 | 0.139 $\pm$ 0.030 |
| hmc(500)  | 0.094 $\pm$ 0.020 | 0.106 $\pm$ 0.020 | 0.134 $\pm$ 0.034 | 0.113 $\pm$ 0.020 | 0.135 $\pm$ 0.010 | 0.135 $\pm$ 0.033 |
| Neural Samplers | | | | | | |
| Coup Flow	    | 0.102 $\pm$ 0.028 | 0.158 $\pm$ 0.019 | 0.150 $\pm$ 0.026 | 0.239 $\pm$ 0.013 | 0.269 $\pm$ 0.019 | 0.130 $\pm$ 0.026 |
| KSD-NS    | 0.206 $\pm$ 0.043 | 1.129 $\pm$ 0.197 | 1.531 $\pm$ 0.058 | 0.341 $\pm$ 0.039 | 0.396 $\pm$ 0.221 | 0.462 $\pm$ 0.065 |
| SteinGAN  | 0.091 $\pm$ 0.013 | 0.131 $\pm$ 0.011 | 0.121 $\pm$ 0.022 | 0.104 $\pm$ 0.013 | 0.129 $\pm$ 0.020 | 0.124 $\pm$ 0.018 |
| **Fisher-NS(ours)** | 0.095 $\pm$ 0.016 | 0.118 $\pm$ 0.013 | 0.157 $\pm$ 0.030 | 0.179 $\pm$ 0.028 | 7.837 $\pm$ 1.614 | 0.202 $\pm$ 0.037 |
| **KL-NS(ours)**     | 0.099 $\pm$ 0.015 | 0.104 $\pm$ 0.015 | 0.123 $\pm$ 0.021 | 0.109 $\pm$ 0.015 | 0.115 $\pm$ 0.012 | 0.118 $\pm$ 0.024 |

**Settings**: 
For all MCMC samplers, we set the number of iterations to 500, which we find is enough for convergence. For SVGD and LD, we set the sampling step size to 0.01. For the HMC sampler, we optimize and find the step size to be 0.1, and LeapFrog updates to 10 work the best. For Coupling Flow, we follow [3], and use 3 invertible blocks, with each block containing a 4-layer MLP with 200 hidden units and Gaussian Error Linear Units (GELU) activations. The total parameters of flow are significantly larger than neural samplers. We find that adding more coupling blocks does not lead to better performance. For all targets, we train each neural sampler with the Adam optimizer with the same learning rate of 2e-5 and default bete. We use the same batch size of 5000 for 10k iterations when training all neural samplers. We evaluate the KSD for every 200 iterations with 500 samples with 20 repeats for each time. We pick the lowest mean KSD among 10k training iterations as our final results. Since our proposed Fisher neural sampler and KL neural sampler requires learning the score network, we find that using 5-step updates of the score network for each update of the neural sampler works well. 

**Analysis**: 
The SVGD performs significantly the best among all samplers. However, the SVGD has a more heavy computational cost ($\mathcal{O}(n^2)$) when the number of particles grows because its update requires matrix computation for a large matrix. The LD and HMC perform almost the same. The KL-NS performs the best across almost all targets, slightly better than LD and HMC on each target. The SteinGAN performs second and is closely comparable to KL-NS. In theory, both the KL-NS and the SteinGAN aim to minimize the KL divergence between the sampler and the target distribution in different ways, so we are not surprised by their similar performances. The Coupling Flow performs overall third, but it fails to correctly capture the Rosenbrock target. We believe more powerful flows, such as stochastic variants or flows with more complex blocks will lead to better performance, but these enhancements will inevitably bring in more computational complexity. The Fisher-NS performs the fourth, with 1 failure case on the Funnel target. We find that the KSD-NS is hard to tune in practice, which has two failure cases. Besides, the KSD-NS has a relatively high computational cost because it requires differentiation through the empirical KSD, which needs large matrix computation when the batch size is large. Overall, the one-shot KL-NS has shown strong performance, outperforming LD and HMC with requires multiple iterations.

[1] Coin Sampling: Gradient-Based Bayesian Inference without Learning Rates

[2] Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

[3] Stein Neural Samplers

[4] Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning

Meta-review: This paper proposes a novel and promising new method for training Monte-Carlo samplers implicitly defined by neural networks.  The biggest concerns raised in the reviews were regarding citation and comparison to related work, as well as some details of the experimental results.  Following author rebuttals and some discussion with reviewers, the consensus is that while some minor concerns remain, overall there is compelling evidence that the proposed methods are effective.  Please do carefully revise your manuscript to incorporate the additional results, clarifications, and citations that arose during the review process.
Decision: Accept (poster)