Local Curvature Smoothing with Stein's Identity for Efficient Score Matching
Authors: GENKI OSADA
Makoto Shing
Takashi Nishide
Venue: NeurIPS 2024 poster (2024)
Keywords: Score matching
score-based generative models
diffusion models
Primary area: diffusion_based_models

Abstract: The training of score-based diffusion models (SDMs) is based on score matching. The challenge of score matching is that it includes a computationally expensive Jacobian trace. While several methods have been proposed to avoid this computation, each has drawbacks, such as instability during training and approximating the learning as learning a denoising vector field rather than a true score.
We propose a novel score matching variant, local curvature smoothing with Stein's identity (LCSS). The LCSS bypasses the Jacobian trace by applying Stein's identity, enabling regularization effectiveness and efficient computation. We show that LCSS surpasses existing methods in sample generation performance and matches the performance of denoising score matching, widely adopted by most SDMs, in evaluations such as FID, Inception score, and bits per dimension. Furthermore, we show that LCSS enables realistic image generation even at a high resolution of $1024 \times 1024$.

Local Curvature Smoothing with Stein's Identity for Efficient Score Matching
Genki Osada
Makoto Shing Sakana AI, Japan
LY Corporation, Japan
genki oswda@1ycorp..ooommmmo co.pp.co.rpoo....... co. jp
mkshing@sakana.co
Takashi Nishide University of Tsukuba, Japan nishideerisk. asukuba.ac...
Abstract
The training of score-based diffusion models (SDMs) is based on score matching. The challenge of score matching is that it includes computationally expensive Ja- cobian trace. While several methods have been proposed to avoid this computation, each has drawbacks, such instability during training and approximating the learn- ing as learning a denoising vector field rather than a true score. We propose novel matching variant, local curvature smoothing with Stein's 's identity (LCSS). The LCSS bypasses the Jacobian trace by applying Stein's 's identity, enabling regu- larization effectiveness and efficient computation. We show that LCSS surpasses existing methods in sample generation performance and matches the performance of denoising score matching, widely adopted by most SDMs, in evaluations such as FID, Inception score, and bits per dimension. Furthermore, we show that LCSS enables realistic image generation even high resolution of 1024 1024.
1 Introduction
Score-based diffusion models (SDMs) 335 30 31 12 have emerged as powerful generative models that have achieved remarkable results in various fields (27 26 24
While ikelihood-basee models
learn the density of observed (i.c., training) data as points 225 5 28 38 37 14]. SDMs learn the gradient of logarithm density called the score vector field pointing toward increasing data density, The sample generation process of SDMs has two steps: 1) learning the score for a given dataset and 2) generating samples by guiding random noise vector toward high-density regions based on the learned score using stochastic differential equation (SDE).
Score matching used for learning the score includes computationally expensive Jacobian trace, making it challenging to apply to high-dimensional data. While some methods have been proposed to avoid computing the Jacobian trace, each has its drawbacks. Denoising score matching (DSM) (39, ubiquitously employed in SDMs, learns not the ground truth score but its approximation and imposes constraints on the design of SDE. On the other hand, sliced score matching (SSM) (33) and its variant, finite-difference SSM (FD-SSM) 23), suffer from high variance due to using random projection.
In this paper. we propose novel score matching variant, local curvature smoothing with Stein's identity (LCSS). The key idea of LCSS is to use Stein's 's identity to bypass the expensive computation of Jacobian trace. To apply Stein's identity, we take the expectation over a Gaussian distribution
38th Conference on Neural Information Processing Systems (NeurIPS 2024).
The code will be released after the paper is published.

---

centered on imput data points, which is indeed equivalent to the regularization with local curvature smoothing. Exploiting this equivalence, we propose score matching method that offers both regularization benefits and faster computation.
We first establish method conditional version for its application to SDMs. We present the experimental results using synthetic data and several popular datasets. Our proposed method is highly efficient compared to existing score matching methods and enables the generation of high-resolution images with a size of 1024. We show that LCSS outperforms SSM, FD-SSM, and DSM in the quality of generated images and is comparable to DSM in the qualitative evaluation of the FID score, Inception score, and the negative likelihood measured in bits per dimension. While DSM requires the drift and diffusion coefficients of an SDE to be affine, our LCSS has no such constraint, allowing for a more flexible SDE design (Sec. 2.4p. Hence, this paper contributes to opening up new directions in SDMs' research based on more fexibble SDEs.
independent
matching technique, then propose time-
Related works. Liu al. [20 proposed method for directly estimating scores using Stein's identity without using score matching. Shi et al. (29) further enhanced that method by applying spectral decomposition to the function in Stein's identity. However, Song al. (33] reported that
these methods underperform compared to SSM. In our approach, we use Stein's 's identity specifically avoid computing the Jacobian trace in score matching. The regularization effect attained by adding noise to data has been recognized for long time 3 which our method utilizes. The relationship between noise-adding regularization and curvature smoothing the least square function is elucidated in Bishop [2]. The previous studies of score matching variants are described in the next section. In efforts remove the affine constraint of SDE in SDMs, Kim al. [3 proposed running SDE in the latent space of normalizing flows. This constraint stems from using DSM for score matching, and we propose a score matching method free from such constraint.
2 Preliminary
2.1 Score-based diffusion models
Score-based diffusion models (SDMs) [35 31 define stochastic differential equation (SDE) for
X2 E Rd in continuous time E [0, T as
dx, f(x Odlt + g(()dwe:
(1)
Figure 1: Samples generated from models trained on CelebA-HQ (1024 x 1024) using our proposed score matching method, LCSS. The rightmost images in each row are generated by DDPM+ with su SDE, while the rest are by NCSN+ with VE SDE.

---

where f(:,t): Rd + Rd is the drift coefficient, g(t) E R is the diffusion coefficient, and w, denotes standard Wiener process. Eq. ① known the forward process, has corresponding reverse process from time 7 to 0 [I]
dx. [(x,,)) V Pxx)) dt g(t)dw:
(2)
where w, a standard Wiener process in reverse-time and pr (x(x)) denotes the ground truth marginal density of x, following the forward process. Samples from dataset are represented Xo po- while initial vectors for sample generation with Eq. (2) are XT ~ pr. In Eq. 2), the only unknown term is Vx x log Pi (x)), referred to as the score of density (x().. To estimate Vx log): SDMs train a score network Sy parametrized by by score matching.
2.2 Score matching
A score network Sy that estimates the score of the ground truth density is trained through score matching [ Score matching, technique independent of SDMs and SDE, has no concept of time. So, as long as our discussion is focused on score matching, we use the notation of x and p. without the subscript of t, and treat a score network without conditioning on t. iee, denote it as sa (x( instead of t)), Score matching is defined as the minimization of [(0) Expp [>s{()) Vx log p(x)) Calculating [(0) is generally impractical since it re- logp(x), p(x), but Hyvärinen has shown that J is equivalent to the following (f) up constant:
quires knowing the ground truth Vx log
Jsm( == Ex-p [JsM x)]
where [0, x) is the version of (8) for a single data point x, defined as
1 Js, x) == Tr (x(x))(()) + 2 |ss
(4)
Score matching in SDMs and its problem. SDMs train a time-conditional score function sa (a(x,, !) using score matching. The loss function of SDMs is defined as the integral of (0) over time I E (0, T] as
CSM( = J/" (((t) Ext)ExiNp Exaspo Tr(V x (x,, t)) +
2 ||50 (x,, t)1[3 dt.
1
(5)
The weight function A(L) is determined by the form of the SDE, and X(() used for typical SDEs can be found in Table in (3.. The Pt is obtained from the SDE Eq. ① with its mean dependent on Xo. and its specific form in typical SDEs is given as Eq. (29) in Song al. (35]. The problem is that
since sæ (x: has the same dimension as input xt, computing its Jacobian trace, Tr(VxSy SX t)), is
costly, It renders training with score matching impractical in high-dimensional data.
2.3 Existing score matching variants
To avoid the computation of the Jacobian trace, the following scalable variants of JsM have been developed. While any score matching method can be used to train Sg. SDMs predominantly employ DSM due to its empirical performance 31 35
Sliced score matching (SSM) and finite-difference sliced score matching (FD-SSM). SSM approximates Tr(V Sx(())) with Hutchinson's trick [8] and minimizes the following:
33
(0) = Evepy Excup
Xx)e + 2d [1s0(
(6)
where is small scaler value and Pv is d-dimensional random vector such that Ev-p, [vv") T- objective function
21 To enhance the efficiency of SSM further, FD-SSM 23 adopts finite difference to Eq. (6). The
1
(B) : Evaup Expu
(v s0(x + v) - s0(x - v))
2c
1
||00 + v) so - v)ll2
+
8d
3

---

The drawback of these two methods is the high variance induced by random projection with v. In particular, the error between the true trace of matrix A, (r))), and the estimate by Hutchinson's trick, Ta. is (r) TAl - JM I| where II-II is the Frobenius norm and M is the sampling times from Py 22 Typically, M = setting is employed in these methods, potentially making the error magnitude non-negligi and causing instability in training process, as we see in See 4.2.3
Denoising score matching (DSM). DSM (39 cireumvents the computation of Tr (T(Vx8((X) by perturbing x with a Gaussian noise distribution go (x[x) with noise scale and then estimating the score of the perturbed distribution qo (x) := ʃ go (x xx (x))xxxx The DSM minimizes
JOSM( (0) Ex (Rxx) Exup
|so |so({)) (x)
V log 9o (x(x)]
(8)
In SDMs, the following condiconllli version is used:
JOSM (0, 1) == Exg (8|xa)
E Exo-po
|| Xx,, 1) - Vx log go, (Xt xo)1
where is designed to increase / progresses from 0 to T. Almost all SDMs use DSM for score matching because it performs faster and is more stable than SSM and FD-SSM. However, DSM has three drawbacks. 1) Approximation: in DSM, Se learns Vx log lo (xx(xo) rather than the ground true score, Vx log Pe (xx). 2) Constraining the design of SDE: DSM constrains SDE coefficients to be affine. We will describe this in Sec. 224
3) The dilemma regarding o₂ Only when or + does Vx log 9o. (5x Xb) match Vx log pe (xx).. However, as a - 0, both the numerator and denominator of xa xi approach 0, leading to potential numerical instability 19
of
2.4 DSM restricts SDE to affine
The design of SDEs directly influences the performance of SDMs, as demonstrated in previous studies [12.. The benefits of non-linear SDE, particularly highlighted in [33, enable more accurate alignment of scores with the ground-tree truth data distributions than affine SDE and thus enhance the quality of generated samples. (Fig. 2 [3 illustrates this.) However, unless specific modifications are made as proposed in these studies, the general SDEs (35 used in almost all existing SDMs must be affine. This constraint comes from the fact that the SDMs, consciously or unconsciously. select DSM for their score matching methods. The loss function of DSM requires Vx log 4o, (&, Xx) as Eq. ② Thus, to compute Eq. (9) at every training iteration, Vx log 9o, (x, xo) needs to be in closed form. DSM models Yo,
(&el xo) as a Gaussian distribution, for which this requirement is satisfied - However, this Gaussian modeling comes at the cost of imposing of Xt,
as Vx log 9o, (xo)
constraint on the SDE design: the drift and diffusion terms of SDE, i.c., f(x,, t) and gt) in Eq. ( need to be affine. The existing SDMs are DSM- so the SDEs used in these SDMs, including the VE SDE and subVP SDE we use in our experiments, are designed to adhere to this constraint. The details of the same discussion and the specific form of the Gaussian distribution q., (3,, (x)) for the typical SDEs can be found in Sec. 3.3 in Song et al. 35 Unlike DSM, SSM and FD-SSM do not have this limitation, allowing for more flexible SDE design and thus removing the requirement to limit the forward process's 's convergence destination to Gaussian distributions. Unfortunately. as we will see later, SSM and FD-SSM cannot handle high-dimensional data due to the high-variance they cause, Our proposed method uniquely satisfies both the flexible design of SDEs and compatibility with high-dimensional data.
3 Our Method
We propose novel score matching variant that avoids the expensive computation of the Jacobian trace. The crux of our method is using Stein's 's identity to bypass Jacobian computation. Our approach comprises three steps: 1) introducing local curvature smoothing regularization into score matching (Definition I. 2) treating the regularization of 1) as taking an expectation over a Gaussian distribution (Lemmal 1. and 3) applying Stein's identity (Corollary 2 While introducing regularization may appear to cause extra computational costs, it enables faster computation by the use of Stein's 's identity trick. We begin by discussing our method separately from SDMs, without involving the time variable t, and then explain its incorporation into SDMs at the end of this section.

---

3.1 Score matching with Local Curvature Smoothing with Stein's identity (LCSS)
We first introduce some lemmas and corollaries that constitute our method.
Definition 1 (Score matching with local curvature smoothing (15I). Regularizing the score matching objective JSM at a data point x E Idd with local curvature smoothing (LCS) is defined as:
[0,s(0,x,0) == (0, + =²² [
(10)
2
Given (x) approximating the Hessian of log)(x), minimizing the regularization term 2 V|V (x) acts local curvature smoothing where the square of the curvature of the surface of the log-density
are penalized. Curvature smoothing one of the commonly employed regularizations in machine learning H
Lemma (Kingma and LeCun (5).. ) Score matching with local curvature smoothing (Definition equivalent to the expectation (0, x) over Gaussian distribution centered at x, i.e., x' N N(x, T));:
7
Ji, Ex-Nixx [T +0)
(11)
where E : |x' -xx12 x112-
Lemma [ states that taking the expectation of score matching objective with respect to Gaussian distribution centered around x yields an effect equivalent to a curvature smoothing regularization.
Definition 2 (Stein class
(36].. Assume that (z) is continuous differentiable probability density
supported on Z C RR Then, function Z + R is the Stein class of if satisfies
/ (z(((2) ZEZ
(12)
The condition for Eq. 12) to hold is
lim f(z)Q(z)
(13)
x +00
Lemma (Stein's 's identity, Liu et al. (20), Gorham and Mackey (6). Leth: - Rd be a smooth (i.e., continuous and differentiable) vector valued function h(z) [h (z), har (z)]². Then,
if (z)xvi = 1...., d' the Stein class of a smooth density (z), the following identity holds:
ExrQ [h log Q(z)* Veh(z)] Owrdd
(14)
In Eq. 14 V, log Q(z) aale d matrix, Vzh(z) d' matrix, and O d is a d' x d zero matrix.
Corollary (Li and Turner 18) When Q(z) N(z; L we have
21 Mi
ExnQ
hz(z))
Exrq (2),(z))
(15)
x²
Eq. (15) holds for the i-th element of the vector h. The condition Eq. 13 holds for Gaussian
distribution Q, since Q(z) + as ||z||| + co. Then, (z(z)Vi = I,.... d' are the Stein class of Q, and
thus Lemma 2 is valid for a Gaussian distribution Q. As we also know V, log ((z)
- 11),
by substituting it into Lemma 2] we obtain Corollary [
Corollary (Bypassing Jacobian trace computation). Let Rdx1, so(x) gdx1, and Q(x')
(x x, o With Corollary i and fe assumptions, we have the followin.::
x'
Ex [Ir(Vxss (x'))) = Ex-NNix so(x))
(16)
o²
The Sp, which represents score network in our context, corresponds to h in Lemma and Corollary The derivation of Eq. (T6) is presented in Appendix [ in which we assume the interchangeability between the expectation and summation regarding (xx).

---

Objective function of LCSS. We propose a variant of score matching method, local curvature smoothing with Stein's 's identity (LCSS). The development of the objective function of LCSS, Tess-
begins with the curvature smoothing regularization of Eq. (T0), followed by the application of Lemma D and Corollary 2 Since Tics(A (0, x, 0) in Eq. (TO) involves computationally expensive Vx80(x), (x), alongside the original challenge of Tr(V,x (x)xx))
JSM training with Jigs is impractical
for high-dimensional data. However, by inserting the transformation of Lemma [ it enables the application of Corollary to Jics- By substituting Eq. (4) into Eq. (11 and ignoring (E²), we have
(0,(6,,5 0.) = -(x
Tr Tr (xx)) + 2 || (1s((2))
(17)
and by applying Eq. 16 to the first term, we obtain Jicss as:
x' -
1
Ress(O, = (xx so(x')
+ 2 |1s0 (x)1
(18)
o²
In Tesss Tr Tr(Vxsn(x)) is replaced with the inner product, sx(x' efficiently, thereby bypassing the issue of high computational cost.
which is computed
Comparing LCSS with existing score matching methods. Unlike SSM and FD-SSM, LCSS does
not use random projection, eliminating the high variance issue. While DSM learns the approximation of ground truth score Vx log go (& x), LCSS learns the ground truth score Vx log p(x). Furthermore, unlike DSM, Ticss does not require Vx log 9o x)) thus eliminating the need for affine restrictions on the SDE coefficients. The original score matching, i.e., minimizing JSM- involves the following two: (1) Increasing the first term Tr (rVx5)(() ~ Vx Vx log p(x), the divergence of the score, in the negative direction promotes Sy to learn the vector field flowing into points where x exists.
(2) Minimizing the second term 1/2 ||sa(x)]] promotes Se to learn that its length approaches 0 at points where x exists, The LCSS also performs both (1) and (2), but instead of at a single point x, it considers a Gaussian cloud centered around x. By applying Stein's identity, LCSS bypasses the challenge of (I), thereby making score matching feasible even for high-dimensional data.
3.2 Score-based difflusion models with LCSS
We define time-conditional version of LCSS for training SDMs as:
x xo
Ticss (6, Xo, f, o²) := (x0.N() so sx(x
t)²
+ ||s0 (5((x(,)) 1)
(19)
of
and formulate the loss function of SDMs based on LCSS as:
LiSss (0) = 1-² X(t) Exo [Tesss (0, xo, at) dt.
(20)
We replace a in Eq. (18) with time-varying o, By making o, take on a wide range of values depending on i, we aim to facilitate robust learning of score vectors even in low-density regions in Po. mirroring the original motivation of NCSN (31). With Eq. (19), Sy learns a vector in the direction of -xx; - Xo) to minimize the inner product of the first term, weighted by J. while minimizing its L2 norm, ||so tx Sampling x/ in the expectation in Eq. 19) only once yields satisfactory performance, as evidenced by our experimentation.
oz
SDEs for LCSS-based SDMs can be designed flexibly without restricting the drift and diffusion coefficients to be affine. However, devising a new SDE beyond the scope of this paper and is left for future work, and our experiments use existing SDEs designed for use with DSM: the Variance Exploding (VE) SDE and the sub Variance Preserving (subVP) SDE [35]. Taking advantage of the fact that Pe is Gaussian distribution in both SDEs, we employ the standard deviation of p. as the value ot o; in each SDE in our experiments. For example, for VE SDE, o, = g().. For both SDEs, o, increases as t goes from O to T, but the way it increases is different for each SDE.
Following Song and Ermon [3I, we set X(t) g(t) With this setting, X(t) becomes A(t) g(t)² = of effectively cancelling out of in the denominator of Eq. (19) and avoiding unstable situations where the denominator could become zero. For other SDE types (VP and sub VP), X(1) is more elaborate but similarly cancels out of in the denominator. For fairness, we note that, similarly, in training SDMs with DSM, applying the coefficient X(1) = (t)) allows for the cancellation of of in the denominator, thus circumventing the weakness of DSM.

---

Score matching method
Dataset
Model
SSM
FD-SSM
DSM
LCSS
Checkerbourd
MLP
497
445
419
Training data
SSM
FD-SSM
DSM
LCSS
FFHQ
NCSNv2
1838
1367
1381
1075
Table Estimated densities on Checkerboard.
Table 2: Elapsed time for model training (ms)
4
Experiments
In this section, we demonstrate that our LCSS enables fast model training and high-quality image generation several commonly used image datasets.
4.1 Setup
We use five SDMs: NCSNv2 32 as a discrete-time model, NCSN+ and DDPM++ and their extensive version, NCSN++ deep and DDPM+ deep, continuous-time models. Only for synthetic dataset, Checkerboard, we use multilayer perceptron (MLP) with publicly available code based Song and Ermon [3I. In continuous-time models, NCSN++ deep and subVP SDE for DDPM++ and DDPM++ deep as per Song et al. [35] The same SDEs applied all the score matching methods we evaluate, including
VE SDE for NCSN+ and
LCSS. We use the official codes from the original papers, and the hyperparameters are kept as in the official code, unless stated otherwise. Fl For LCSS, perform only one sampling iteration calculate the expectation in Tesss (Eq. (19). We set og g() in each SDEs. All experiments are performed on a server with 128 GB RAM, 32 Intel Xeon, Silver 4316 CPUs, and eight NVIDIA A100 SXM GPUs.
4.2 Results
We evaluate the proposed LCSS against existing score matching methods, SSM, FD-SSM, and DSM, in density estimation, training efficiency, and qualitative and quantitative sample generation evaluation.
4.2.1 Density estimation
We first compare LCSS to SSM, FD-SSM, and DSM in matching performance. We visualize estimated densities on Checkerboard dataset, whose density is multi-modal. The details of the experiments, including the training loss eurve, are presented in Appendix B Table depicts the density distribution learned by the model. Compared to SSM and FD-SSM, LCSS demonstrates higher accuracy in density estimation with faster convergence and stability in loss reduction. DSM exhibits similar accuracy in density estimation and stability in loss reduction to LCSS. However, LCSS shows slightly better consistency estimating high-density regions (bright-colored areas) and maintains stable loss.
4.2.2 Training efficiency
We compare LCSS with the existing score matching methods for training efficiency. We measure the time taken for model training on Checkerboard and FFHQ dataset [1]] resized to 256 256. Table 2 shows the average elapsed time over 100 epochs for Checkerboard and 1000 iterations for FFHQ. respectively. It shows that LCSS is the most efficient.
4.2.3 Sample quality
We show generated samples on CIFAR-10 using NCSN+ deep and DDPM++ deep trained with LCSS in Appendix [.T In this subsection, we qualitatively compare the sample generation capability of LCSS with existing methods.
g Sccora///n///ooroetciiiiggngg Sato:/ graddogg settooe__d__petrrhhhheeeeeeeeeeh

---

SSM
FD-SSM LCSS (ours)
SSM
FD-SSM
L.CSS (ours)
Figure 2: Comparison of sample quality in the early stages of training. The model is NCSNv2 trained on CIFAR-10. The left three panels show generated samples 5k steps training, while the right three show generated samples at 90k steps training.
SSM
FD-SSM
LCSS (ours)
SSM
FD-SSM
LCSS (ours)
Figure 3: Comparison of generated samples CelebA (64 64). The left three show samples from models trained for 10k steps. In the right three, FD-SSM and LCSS images are from models trained for 210k steps, whereas SSM images are from model trained for 60k steps.
Comparison with SSM and FD-SSM. We first focus on comparing LCSS with SSM and FDSSM We generate samples using NCSNv2 on CIFAR- 10 (32 x 32) [[6], CelebA (64 x 64) [21] and FFHQ (256 x 256) [II] The results show that LCSS demonstrates stable long-term training and faster convergence compared to the other two methods. This can explained by LCSS not using random projection, unlike SSM and FD-SSM. Details are provided below.
On CIFAR-10 (32 32), unlike LCSS, SSM and FD-SSM, when reaching 95k and 495k training
steps, respectively, are unable to continue generating meaningful images and produce only entirely black images. Fig. 2 displays generated images 5k and 90k training steps for each method. The faster convergence of LCSS compared to SSM and FD-SSM is exhibited from the differences in the image quality. On CelebA (64 64), Fig. 3 (left) displays images generated by each method 10k steps, highlighting LCSS's s faster learning. Fig. 33 (right) presents the generated images of LCSS and FD-SSM at the 210k training steps. For SSM, after 65k training steps, only generated completely black images, so the displayed SSM images are from the model trained for 60k iteration. On FFHQ (256 x 256), LCSS can generate decent images, while SSM and FD-SSM failed, as shown in Fig.
Comparison with DSM. In the previous experiments, we saw that LCSS significantly outperforms SSM and FD-SSM image generation. In this subsection, we compare LCSS with DSM, widely adopted as the objective function in score-based diffusion models. The results show that LCSS surpasses DSM in qualitative evaluation, and achieves performance on par with DSM quantitative evaluation on CIFAR-10 10 using Fréchet inception distance (FID), Inception score (IS), and negative log likelihood measured in bits per dimension (BPD). The details are below.
We compare generated samples on FFHQ AFHQ, and FFHQ + AFHQ. The size of images the three datasets is (256x250) and we train NCSNv2 for 600k with batch size 16 on each of them. On FFHQ LCSS can generate more realistic images than DSM, shown in Fig. We note that during the training with DSM, around 21 training steps, sharp decline in the quality of generated images was observed. On AFHQ 4. Fig. 5 shows that LCSS generates realistic samples, but DSM does not. We also create and examine with a dataset FFHQ + AFHQ, a fusion of FFHQ and AFHQ, designed to increase learning difficulty by diversifying data modalities. On FFHQ + AFHQ, Fig. 6 shows LCSS's superior capability in generating realistic images over DSM.
DSM
LCSS (ours)
Figure 6: Samples on FFHQ + AFHQ.
Althoogh FD- DD-DSM has also been proposed. it was excluded from comparative evaluation due to reported performance below DSM in Pang et al. (23) and failure generate images appropriately in experiments.

---

SSM FD-SSM
DSM
LCSS (ours)
DSM
LCSS (ours)
Samples on FFHQ (256 x 256). Models are trained for 600k steps with batch size 16. SSM and FD- SSM fail to produce face images.
Figure 4:
Figure 5: Samples on AFHQ (256 256). Models are trained in the same setting as those on FFHQ.
Table 3: Sample quality evaluation on CIFAR-IO. FID (4), IS (t), and BPD (1).
Model
NCSN+
NCSN + deep
+DPM
DDPM + deep
Score matching
FID
IS
BPD
FID
15
BPD
FID
1S
BPD
FID
15
BPD
DSM
4.45
9.86
3,62
4.29
9.86
3,38
4.81
9.62
2.64
4.49
9.58
2.64
LCSS (ours)
4.90
9.88
4.17
4.72
9.95
3.61
5.06
9.63
2.47
4.61
9.80
2.58
Table 3 shows the qualitative results on CIFAR-10. 10. Regardless of SDMs, LCSS tends to surpass DSM IS but underperform in FID. Compared to the values in Song al. 35 our experimental results generally exhibit higher (better) IS values and higher (worse) FID values 3
In BPD, LCSS surpasses DSM in DDPM++ variants but underperforms in NCSN++ variants. Overall, qualitative evaluation on CIFAR-10 suggests no decisive superiority between LCSS and DSM, hinting at distinct characteristics.
Summary. Table 4 illlstraaes a highly simplified comparison of the relative performance between LCSS and DSM. Model training more complex in Case #2 than in Case #1. was observed that in challenging conditions like Case #2, DSM suffered from performance degradation. We regularly monitored the quality of generated images during model training. In the experiments of Case #2 with DSM, as noted above, although the quality of generated images was improving up to a certain stage (around 210k iterations, for example). suddenly deteriorated. Also, frequent spikes in loss values were observed during training with DSM, which appeared to be a trigger for the deterioration. Unlike DSM, LCSS retained superior performance without suffering from such instability.
4.3 High resolution image generation
We demonstrate that learning with LCSS enables models to generate high-resolution images. We train NCSN++ and DDPM++ on CelebA HQ (1024 1024) (10), using hyperparameters consistent with those used to train DSM-based models in Song et al. [35]. In Fig. [ we show generated images: NCSN++ images are from the model trained for around 1.3M iterations, and DDPM++ ones are trained for around 0.3M iterations, with batch size 16 for both. The figures show that LCSS is promising as score matching method. More generated samples are presented in Appendix C.2
4.4 Ablation study
The loss function of LCSS, similar to the original score matching, consists two terms. To study the roles of each term, using the modified version of Teess in Eq. (19) with a balancing coefficient y as
t)T (5se(X ((0
- X0
|| (x(, +)12
(21)
we train NCSNv2 on FFHQ. Images generated from the models trained with different are shown in Fig.7 7 When 7 = 0.5, only noisy images akin to those at time t = T. Xr, are produced. With 7 < 1, the force to minimize the second term, (x(, t) [[ is more emphasized than when using the original Tuess. leading to shorter score vector lengths. The score vector length is directly linked the spatial movement distance of X, during the reverse process for sample generation. Since the score vector is
Although we use the official code from Song et al. 335 in our experiments, the difference between JAX version in Song al. 35 and PyTorch version in experiments is considered to be the cause.

---

Table 4: Simplified performance comparison between LCSS and DSM.
Score matching method
Case
Model capacity
Image resolution
Corresponding results
DSM
Large (NCSN++, DDPM++ etc.)
32 x 32
Figures.
X
Small (NCSNv2)
Table
0.5, iter 300k
2, iter=t0000 300k
10, iter= 10k
10, iter=3000 300k
10, itek-600k
Figure 7: Generated samples on FFHQ (256 x 256) by the model trained with LCSS (ours) with different y. The notation ifer signifies the training iterations.
forced to be short, the noise generated at the start of the sample generation process cannot reach the regions corresponding to realistic images with high density as it traces back from time T to On the other hand, when y > 1, particularly for 7 10, it is observed that as the number of training iterations increased, images with emphasized contours but lost textures are generated. suggests that the involvement of the first term of TLess in contour formation. The object contours in an image are characterized by rapid changes in pixel values, which can be associated with high curvature or
changes in second-order derivatives. Since so (x(: t) x in Eq. 21 (99) corresponds to the Hessian trace of log-density, this observation can interpreted as natural. It implied that the first and second terms in the loss function of LCSS are dedicated to the formation of contours and texture, respectively.
Conclusion
Linitation. While LCSS, unlike DSM, can design SDEs flexibly without restricting them to affine forms, we used existing affine SDEs designed for use together with DSM, i.e., VE SDE and sub SDE, in this work. Proposing more flexible SDEs leveraging LCSS is left for future work.
proposed local curvature smoothing with Stein's identity (LCSS), regularized score matching method expressed in simple form, enabling fast computation. We demonstrated LCSS's ' effectiveness in training on high-dimensional data and showed that CSS-bbaeed SDMs enable high-resolution image generation. Currently, SDMs primarily rely DSM, constraining the design of SDE. LCSS offers an alternative to DSM, opening avenues for SDM research based on more flexible SDEs.
10

---

References
[1] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313 -261333-6,,62 1982.
[2] Chris M Bishop. Training with noise equivalent tikhonov regularization. Neural compula- tion, 7(1):108-116, 1995.
[3] Christopher M Bishop. Neural networks for pattern recognition. Oxford university press, 1995.
[4] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[5] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In International Conference on Learning Representations, 2015.
[6] Jackson Gorham and Lester Mackey. Measuring sample quality with stein's method. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[7] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6855 Curran Associates, Inc., 2020.
[8] M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Simulation and Computation, 18(3):055) 1076, 1989. doi:
[9] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-709, 2005.
[10] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representa- lions, 2018.
[11] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for genera- tive adversarial networks. In 20/9 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396 4405, 2019. doi: 10.119//VPP.2209.00455
[12] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, Curran Associates, Inc., 2022.
pages
[13] Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-chul Moon. Maximum likelihood training of implicit nonlinear diffusion model. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32270-32284. Curran Associates, Inc., 2022.
[14] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[15] Durk P Kingma and Yann LeCun. Regularized estimation of image statistics by score matching. In J. Lafferty, C. Williams, J. Shawe-Taylos R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[16] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[17] Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Fp-diffusion: Improving based diffusion models by enforcing the underlying score -kkker-planck equation. Proceedings of the 40th International Conference on Machine Learning, 2023,

---

[18] Yingzhen Li and Richard E. Turner. Gradient estimators for implicit models. In International Conference on Learning Representations, 2018.
[19] Jae Hyun Lim, Aaron Courville, Christopher Pal, and Chin- Wei Huang. Ar-dae: towards unbi- ased neural entropy gradient estimation. In Proceedings of the 37th International Conference on Machine Learning, ICML'20 '2. JMLR.org. 2020.
[20] Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-fe tests. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 276-284, New York, New York, USA, 20-22 Jun 2016. PMLR.
[21] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[22] Raphael A. Meyer, Cameron Musco, Christopher Musco, and David P. Woodruff. Hutch++: Optimal Stochastic Trace Estimation, pages 145-155. doi:
[23] Tianyu Pang, Kun Xu, Chongxuan LI, Yang Song, Stefano Ermon, and Jun Zhu, Efficient learning of generative models via finite-difference score matching. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages
Curran Associates, Inc., 2020.
[24] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
[25] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Pro- ceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
[26] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[27] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 36479-36494. Curran Associates, Inc., 2022.
[28] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixeICNN++: Improving the PixcICNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017.
[29] Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit distributions. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4644-4653. PMLR, 10-15 Jul 2018.
[30] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265, Lille, France, 07-09 Jul 2015.
PMLR.
[31] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
12

---

[32] Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12438- Curran Associates, Inc., 2020.
[33] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach density and score estimation. In Proceedings of the Thirty-Fifth Conference Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, page 204, 2019.
[34] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34,
pages 1415-1428. Curran Associates, Inc., 2021.
[35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representutions, 2021.
[36] Charles Stein. A bound for the error in the normal approximation to the distribution of sum of dependent random variables. In Proceedings of the sixth Berkeley symposium on mathematical statistics and probability, volume 2: Probability theory, volume 6, pages 583-603 University of California Press, 1972.
[37] Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. In Advances in Neural Information Processing Systems, volume 26, pages 2 2175-2183. Curran Associates, Inc., 2013.
[38] Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, koray kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems, volume 29, pages 4790-4799 Curran Associates, Inc., 2016.
[39] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674 2011.
13

---

A Applying Stein Identity to Jacobian Trace
To derive Corollary 2 first introduce the following two assumptions.
Assumption 1. Let be real-valued vector, [=,,...,,aa" x.l"". Considering the derivative of function Se IR + Rd, we assume that the expectation and summation are interchangeable as follows:
d
Oso, (x')
Ooe, (x') Or.
Ex
(22)
Oz:
Assumption 2. For the same function Sy in Assumption
we assume the following interchangeabillty:
d
Exx~N(Xx00P))
Σ
(so. (x')(4 -x:)) =
Ext Es
x:)]-
(23)
1=1
The interchangeability holds when (score) function Sy is integrable and differentiable. As So is implemented by neural networks in our case, we assume these conditions are met. We noted that if there are correlations between the dimensions of x' over which expectations are taken, interchangeability does not always hold. However, because x' is a sample from a diagonal Gaussian, (x, La), there are no correlations between dimensions, thus fulfilling this condition.
Derivation of Corollary 2
Using these assumptions, Eq. (16) is derived as follows:
Üso, (x')
(x) [Tr Tr(V2S06 (e(V>s(x')))
Bx
(24)
ax,
Ose; (ss;(xx')
Bx
(25)
Ox:
Oso. (x') Ox!
Bx,NN(x,011))
(26)
Oa'
0x.
Oso, (x') 0vh
Bx
(27)
Ox! Dz.
E N
80. (x')
1/
-2
x/.
(28)
0²
Exl-N(x,
Σ
d
80, (x')'')
2/
24
(29)
σ²
= -E
sq (x
x'
x
(30)
Assumption is applied in transitioning from Eq. 24 to Eq.
The transition from Eq. 25
to Eq. 26 achieved by applying the chain rule, with x!! n; + where ~0, 1), due to
dzd z N N(x o²), and thus we have dez in Eq. (7 We use Corollary in the transition from
Eq.
27 to Eq. (28) We finally apply Assumption 2 in the transition from Eq. (88 to Eq. 29
B Experiments on Checkerboard
We describe the setup for the experiments on Checkerboard dataset. We use the publicly available c The generation of the Checkerboard dataset can be found, for example, in Appendix D. 1 in Lai et [I7]. In the data space X' ?R? we train function fo X parameterized by 0 to
"github.com/EEddingg0055000 gradloop
14

---

estimate density directly. The score, calculated as Vxfe. corresponds to Sa in the main text. We apply the score matching methods we examine estimate Vxfo. resulting in the trained density estimation function fo. Sampling is performed using Langevin dynamics based on VxJo as:
X xx (x) +
(31)
where (0, clly) and c 0.1. The function fg is implemented simple multilayer perceptron (MLP) with two hidden layers, each with 300 units, which is the same architecture as the one used in Song and Ermon [31. The number of Langevin dynamics steps is 1000, with the initial vector, X1000. being randomly sampled from a uniform distribution. We use stochastic gradient descent with batch size of 10k, learning rate of Le 3, and train for 200 epochs. Fig. shows the sampling results of 250M points. The loss curve during training on Checkerboard is shown in Fig.
FD-55M
-25
-10000
-50
11500
-6
-20000
cooch
opach
opech
cooch
Figure 8: Training loss curve on checkerboard, corresponding to Fig. DSM, and LCSS (ours).
From left to right: SSM,
C More results
C.1
Generated Samples on CIFAR-10
Fig. 9 shows generated samples from the models trained on CIFAR-10 using LCSSS. The models are NCSN ++ deep with VE SDE and DDPM+ deep with subVP SDE.
C.2
Generated Samples on CelebA-HQ
Fig. 10 shows generated samples from models trained on CelebA-HQ (1024 x 1024) using LCSS. The model is NCSN++ with VE SDE.
15

---

Figure 9: Generated samples on CIFAR-10 with LCSS. The model for the top is NCSN+ deep with VE SDE, and the one for the bottom is DDPM+ deep with subVP SDE.
16

---

OV
NA
WAR
Figure 10: Samples generated from NCSN+ with VE SDE trained on CelebA-HQ (1024 x 1024) using LCSS.
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We outline what is demonstrated to support the claim made in the paper's ' title at the level of both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section contaans paragraph explaining the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The derivation of Corollary ii presented in the appendix, and the lemmas we used are referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our main results can be expressed in a very simple form. By replacing the loss function of the existing SDMs with Eq. (20). the experiment can be reproduced. addition, the code and trained models utilized in the experiments are based on the official release of Song et al. 355 as mentioned the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
19

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code used in experiments is based on Song al. [35] 's already published code. We plan to release our code after the publication of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
CodeSubmii ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ys]]
Justification: The details of the experiments of observing SDM likelihood behavior are in the main text. Almost all settings the same Song al. 35 or (11]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For the qualitative evaluation using FID, IS, and BPD, we calculated the mean over all samples in the CIFAR- 10 test set, and in line with common practice, eтооr bars are not shown, as Song et al. 335 for example.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We presented in the second paragraph of Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We read and confirmed that this study is in compliance with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our proposal enables the development of score- booee generative models based nonlinear SDEs, potentially significantly improving generation quality over existing methods. As noted in the guidelines below, this could lead to the misuse in creating more sophisticated deepfakes.
Guidelines:
21

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We provided proper citations for the code, trained models, and datasets used in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "1k0yc04OhR", "reviewer": "Reviewer_AvFd", "summary": "This manuscript proposes a new score matching method that bypasses the Jacobian trace by applying Stein’s identity, enabling effective regularization and efficient computation.", "questions": "Please refer to weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method is computationally efficient compared to other SSM variants.\n2. Experimental results demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The advantage of the proposed method compared to denoising score matching (DSM) is unclear. The manuscript mentions that it restricts the SDE to be affine, but it does not clarify the benefit of using a non-affine SDE. Furthermore, the influence of the SDE on the generative model needs to be elaborated.\n2. The experimental results do not show significant improvements over DSM. The proposed method achieves comparable sample quality, as shown in Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer taking the time to read our paper thoroughly. \nWe hope the following responses clarify our contribution. Additionally, our response to Reviewer tfBV below provides an argument about the advantage over SSM and FD-SSM, which we would like the reviewer to examine.\n\n-----------------\nResponse to Q.1\n-----------------\nThe design of SDE directly influences the performance of score-based diffusion models, as demonstrated in [1][2]. The benefits of non-linear SDE, particularly highlighted in [3], enable more accurate alignment of scores with the ground-truth data distributions than affine SDE and thus enhance the quality of generated samples. (Fig. 2 in [3] illustrates this.)\n\nOur LCSS allows for the engineering of non-linear SDEs with DSM-equivalent performance. \nThis paper does not cover the creation of new non-linear SDE based on LCSS, which is noted as a limitation, but we focus on describing LCSS itself as a novel score-matching method in this work.\n\n- [1] Dockhorn, T., Vahdat, A., & Kreis, K. Score-based generative modeling with critically-damped langevin diffusion. (ICLR 2022)\n- [2] Karras, T., Aittala, M., Aila, T., & Laine, S. Elucidating the design space of diffusion-based generative models. (NeurIPS 2022)\n- [3] Kim, D., Na, B., Kwon, S. J., Lee, D., Kang, W., & Moon, I. C. Maximum likelihood training of implicit nonlinear diffusion model. (NeurIPS 2022)\n\n-----------------\nResponse to Q.2\n-----------------\nThe evaluations in Table 3 utilize large models, such as DDPM++, at a low resolution of $ 32 \\times 32$, where the results show that LCSS performs comparably to DSM, as the reviewer stated.\nHowever, qualitative evaluations presented in Figs. 4-6 conducted with the smaller model NCSNv2 at $256 \\times 256$ resolution demonstrate noticeable differences between LCSS and DSM; DSM exhibits poor image generation performance under these conditions, as suggested in [4], whereas LCSS can still produce high-quality images.\nThe stable performance of LCSS with limited model capacity is its advantage over DSM.\n\n- [4] Song, Y., & Ermon, S. (2020). Improved techniques for training score-based generative models. (NeurIPS 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gCX2eoETr7", "reviewer": "Reviewer_tfBV", "summary": "The paper proposes to use Stein's lemma to obtain a computationally efficient way in implementing a local-curvature regularized variant of the score matching objective. The main idea is to rewrite the Jacobian-trace term in a way that requires no Jacobian evaluations. In numerical experiments, the effectiveness of this approach is clearly demonstrated.", "questions": "Can there be a formal argument that the proposed estimator has lower variance than random projections? Essentially, the gradient is estimated through random (zero-order) sampling, which is not exactly low-variance?", "limitations": "All limitations are addressed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and the main idea is clear and easy to understand. \n- Other works which the paper builds upon are referenced and fairly attributed. \n- Experiments on small-scale data clearly demonstrate the effectiveness of the approach. \n- Also on larger datasets, the method appears to give strong empirical results.", "weaknesses": "- Approximating Jacobian trace through Stein's identity potentially leads to an estimator with large variance -- I found the claims that it solves Hutchinson's high variance problem to be a bit misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's meaningful question; addressing it will clarify our paper's contributions. For reader comprehension, we plan to include the following argument in the camera-ready version (maybe in the appendix).\n\n----\n\n## Question: a formal argument that the proposed estimator has lower variance than random projections?\n\n\nThe expectation in Eq.(16) is approximated by a single sampling, leading to an error, as the reviewer asks.\nHowever, the key lies in taking the expectation over Gaussian samples, resulting in sufficiently small errors. \nBelow, we show the comparison between the error in a single sampling approximation of Stain's identiy with the error caused in random projections (Hutchinson's trick error).\n\n----\n### 1. Error in single sampling approximation\n\nLet $x$ be a $d$-dimension vector, $S(x)$ be a $L$-Lipschitz (score) function,  $S: \\mathbb{R}^{d} \\rightarrow \\mathbb{R}$.\nIn our implementation, we approximate $M :=  \\mathbb{E} _ { x' \\sim N(x,  \\sigma^{2} I_d) }  \\left[ \\mathcal{J} _ \\text{SM}^{s} (\\theta, x')  \\right]$ \nby its single sampling, \n$M' := \\mathcal{J} _ \\text{SM}^{s} (\\theta, x')$.\nThen, Chernoff bound for Gaussian variable tells us that\n$\n\t\\Pr[ |M - M'| \\geq \\delta] \\leq 2 \\exp \\left(- \\frac{\\delta^{2}}{2 L^{2} \\sigma^{2}}\\right),\n\t \\forall  \\delta \\geq 0.\n$\nLetting $p = 2 \\exp \\left(- \\frac{\\delta^{2}}{2 L^{2} \\sigma^{2}} \\right)$, we obtain\n$\n\t|M - M'| \\leq \\delta = \\sqrt{2 L^{2} \\sigma^{2} \\log \\left(\\frac{2}{p}\\right)}\n$\nwith probability at least $1 - p$.\n\n(See Thm. 2.4 in [5], for example.)\n- [5] Wainwright, M. (2015). \"Mathematical Statistics: Chapter 2  Basic tail and concentration bounds.\"\n\n----\n### 2. Upper bound of Hutchinson's trick error\nWe denote the Jacobian matrix of $S(x)$, $\\nabla _ {x} S(x)$, by $A$.\nThe error between the true trace of $A$, $\\text{Tr}(A)$, and the estimate by Hutchinson's trick, $\\tilde{T}$, is bounded as\n$|\\text{Tr}(A) - \\tilde{T}| \\leq |A| _ {F} $ where $ |\\cdot|_{F}$ is the Frobenius norm. \nThe upper bound of Hutchinson's trick error is thus $|A| _ {F}$. \n\n(See line 94-96 of our paper.)\n\n----\n### 3. Comparison\nWe compare the upper bound of $|M - M'| $ to $|A _ {F}|$.\nBy substituting $|A _ {F}|$ into $\\delta$ in the above, we know that \n$\n|M - M'| \\leq |A _ {F}|\n$\nholds with probability at least $q := 1 - p = 1 - 2 \\exp \\left( - \\frac{|A| _ {F}^{2}}{2 L^{2} \\sigma^{2}} \\right)$.\nIn the case of small $\\sigma$ (a region particularly crucial for SDM training), $q$ is nearly 1, indicating that a single sampling approximation of the expected value on a Gaussian distribution almost consistently yields smaller errors compared to Hutchinson's trick.\n\n---\nWe note that empirically, the stability of the loss curve in Fig. 8 in Appendix B indicates that the proposed LCSS has lower variance than SSM (random projection) and even DSM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VM8OIYL26Y", "reviewer": "Reviewer_qUJ6", "summary": "This paper provides a new way for score matching with the purpose of resolving some of the limitations of the existing methods such as high variance of sliced score matching and Gaussian constraints of denoising score matching (DSM). The new method is based on the local curvature smoothing proposed in [15]. A new score matching objective function is proposed by combining the Stein's Identity with the local curvature smoothing. The authors empirically show that the new method is more efficient in training than DSM and also has comparable performance to DSM.", "questions": "- In section 2.4, the authors criticize the DSM method for having a Gaussian constraint. However, later there is no clarification showing how the new method is different from DSM in this regard. Can you please clarify this?\n- In line 108, the authors criticize the DSM for having 0 numerator and denominator. However, in the final LCSS (equation (16)), the denominator can also be 0 and be problematic. Can the authors provide more discussion on why the new method is better in this regard?\n- In Corollary 2, there is an assumption that an integral must be 0. How restrictive is this assumption? It seems to me that later on when designing LCSS objective, formula (14) is directly used without any further discussion on this assumption. Can the authors explain why this assumption can be dropped?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Although DSM is the default method used nowadays for score matching, the authors provide a nice novel alternative which may have some advantages over DSM. I'm interested to see more theoretical study in the future of this new method.", "weaknesses": "I think some parts of the paper are not stated clearly and further clarification is needed. See questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The detailed questions from the reviewer reflect her/his careful reading of our paper. \nWe are grateful for the constructive questions and hope our responses address their concerns.\n\n-----------------\nResponse to Q.1\n-----------------\nThe loss function of DSM includes $\\nabla _ {x} \\log q _ {\\sigma _ {t}} ( \\tilde{x} _ {t} | x _ {0} )$.\n To compute it for any $0 \\leq t \\leq T$, $\\nabla _ {x} \\log q _ {\\sigma _ {t}} ( \\tilde{x} _ {t} | x _ {0} )$ must be Gaussian, necessitating a Gaussian prior and constraining the SDE to be linear. \nIn constrast, our LCSS\ndoes not include $\\nabla _ {x} \\log q _ {\\sigma _ {t}} ( \\tilde{x} _ {t} | x _ {0} )$ in the loss function, allowing for a non-Gaussian prior and the design of nonlinear SDEs. \n\nWe note the following two:\n- The LCSS loss function involves sampling from a Gaussian,  $x' \\sim \\mathcal{N}(x, \\sigma^2 \\mathbb{I} _ {d})$,\nbut this stems from curvature smoothing regularization and does not relate to or constrain the prior distribution form or SDE design. \n- Like our LCSS, SSM and FD-SSM do not impose Gaussian constraints, yet, as shown in the paper, SSM and FD-SSM fail to learn effectively at resolutions larger than $32 \\times 32$.\n\n-----------------\nResponse to Q.2\n-----------------\nActual model training of SDMs with our method minimizes Eq.(18), which incorporates Eq.(16).\nIn Eq.(18), the coefficient $\\lambda(t)=g^2(t)$ and for the VE SDE, $\\lambda(t)=g^2(t)=\\sigma_{t}^2$, effectively canceling out $\\sigma_{t}^2$ in the denominator of Eq. (18) and avoiding unstable situations where the denominator could become zero.\nFor other SDE types (VP and sub VP), $\\lambda(t)$ is more elaborate but similarly cancels out $\\sigma_{t}^2$ in the denominator. \n\nWe note that, similarly, in training SDMs with DSM, applying the coefficient $\\lambda(t)=g^2(t)$ allows for the cancellation of $\\sigma_{t}^2$ in the denominator, thus circumventing the weakness of DSM the authors mentioned in line 108. \nFor fairness, we plan to add this point to the camera-ready version.\n\n-----------------\nResponse to Q.3\n-----------------\nThe assumption in Corollary 2 holds almost surely. \nThe assumption can be confirmed if $\\lim _ {\\lVert x \\rVert  \\rightarrow \\infty} s_{\\theta_{i}}(x) Q(x) = 0$, as noted in [6].\nBecause $Q$ is defined as a probability density function of Gaussian in Eq. (14), $Q(x)$ tending towards zero at $x=\\pm \\infty$, this condition is satisfied. \nFor clarity, let us also include this explanation in the camera-ready version.\n\n- [6] Liu, Q., Lee, J., & Jordan, M. A kernelized Stein discrepancy for goodness-of-fit tests. (ICML 2016)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GhCfUMRLMl", "reviewer": "Reviewer_j6aU", "summary": "The paper proposes a novel score matching variant called Local Curvature Smoothing with Stein’s Identity (LCSS). This method addresses the computational challenges associated with the Jacobian trace in score matching, particularly for high-dimensional data, by leveraging Stein’s identity. LCSS aims to bypass the expensive computation of the Jacobian trace, offering both regularization benefits and efficient computation. The method is validated through experiments on synthetic and real datasets.", "questions": "see weakness", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  the idea of LCSS is novel\n2. Jacobian is not computed directly, but implicitly respected.\n3. Experiments on high and low resolution are performed.", "weaknesses": "1. In lines 161-162, interchangeability is assumed. However, in the analysis, interchangeability requires some properties of the interested function. The reason why the assumption holds is missing.\n2. This paper does not approximate the Jacobian but instead circumvents the Jacobian. The empirical and theoretical differences against the method using Jacobian should be discussed, such as the difference in the estimated error bound. \n3. In Tab. 3, the improvement seems to be marginal, while in figures, such as Fig. 4, the selected picture is much better under LCSS. The discrepancy should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for thoroughly reading our paper and asking important questions, which we believe will clarify the contributions of our work.\n\n-----------------\nResponse to Q.1\n-----------------\nInterchangeability holds when the score function $S_{\\theta}$ is both integrable and differentiable. \n$S_{\\theta}$ is implemented by a neural network, and we assume these conditions are met. \n\nWe noted that if there are correlations between the dimensions of $x'$ over which expectations are taken, interchangeability does not always hold.\nHowever, because $x'$ is a sample from a diagonal Gaussian, $x' \\sim \\mathcal{N}(x, \\sigma^2 \\mathbb{I} _ {d})$,  in our case, there are no correlations between dimensions, thus fulfilling this condition.\nWe plan to include an extra sentence in the camera-ready version for better reader comprehension.\n\n-----------------\nResponse to Q.2\n-----------------\nThe loss curve in Fig. 8 (Appendix B) empirically demonstrates that our LCSS has a lower variance than SSM (and even DSM). \nThe theoretical explanation is provided below, (essentially identical to the response given to Reviewer tfBV's question).\n\nWe replaced the Jacobian computation with Stain's identity, approximating the expectation computation with a single sampling. \nCompared to Hutchinson's trick, which approximates the Jacobian by random projection in the existing methods (SSM and FD-SSM), we show that our approximation error is smaller with high probability when the variance of Gaussian, $\\sigma^{2}$, is small.\n\n---\n\n### 2-1. Error in single sampling approximation\nLet $x$ be a $d$-dimension vector, $S(x)$ be a $L$-Lipschitz (score) function, $S: \\mathbb{R}^{d} \\rightarrow \\mathbb{R}$.\nWe approximate $M :=  \\mathbb{E} _ { x' \\sim N(x,  \\sigma^{2} I_d) }  \\left[ \\mathcal{J} _ \\text{SM}^{s} (\\theta, x')  \\right]$ \nby its single sampling, \n$M' := \\mathcal{J} _ \\text{SM}^{s} (\\theta, x')$.\nThen, Chernoff bound for Gaussian variable tells us that\n$\n\t\\Pr[ |M - M'| \\geq \\delta] \\leq 2 \\exp \\left(- \\frac{\\delta^{2}}{2 L^{2} \\sigma^{2}}\\right),\n\t \\forall  \\delta \\geq 0.\n$\nLetting $p = 2 \\exp \\left(- \\frac{\\delta^{2}}{2 L^{2} \\sigma^{2}} \\right)$, we obtain\n$\n\t|M - M'| \\leq \\delta = \\sqrt{2 L^{2} \\sigma^{2} \\log \\left(\\frac{2}{p}\\right)}\n$\nwith probability at least $1 - p$.\n\n(See Thm. 2.4 in [5], for example.)\n\n- [5] Wainwright, M. (2015). \"Mathematical Statistics: Chapter 2  Basic tail and concentration bounds.\"\n\n---\n### 2-2. Upper bound of Hutchinson's trick error\nWe denote the Jacobian matrix of $S(x)$, $\\nabla _ {x} S(x)$, by $A$.\nThe error between the true trace of $A$, $\\text{Tr}(A)$, and the estimate by Hutchinson's trick, $\\tilde{T}$, is bounded as\n$|\\text{Tr}(A) - \\tilde{T}| \\leq |A| _ {F} $ where $ |\\cdot|_{F}$ is the Frobenius norm. \nThe upper bound of Hutchinson's trick error is thus $|A| _ {F}$. \n\n(See line 94-96 of our paper.)\n\n---\n### 2-3. Comparison\nWe compare the upper bound of $|M - M'| $ to $|A _ {F}|$.\nBy substituting $|A _ {F}|$ into $\\delta$ in the above, we know that \n$\n|M - M'| \\leq |A _ {F}|\n$\nholds with probability at least $q := 1 - p = 1 - 2 \\exp \\left( - \\frac{|A| _ {F}^{2}}{2 L^{2} \\sigma^{2}} \\right)$.\nIn the case of small $\\sigma$ (a region particularly crucial for SDM training), $q$ is nearly 1, indicating that a single sampling approximation of the expected value on a Gaussian distribution almost consistently yields smaller errors compared to Hutchinson's trick.\n\n-----------------\nResponse to Q.3\n-----------------\nEvaluation in Table 3 was conducted using large models like DDPM++ at a low resolution of $32 \\times 32$, where LCSS and DSM showed comparable performance.\nIn contrast, the experiments in Figs. 4-6 were conducted with the smaller NCSNv2 model at $256 \\times 256$ resolution, presented more challenging conditions. \nThe qualitative evaluation of Figs. 4-6 indicates that LCSS significantly outperforms DSM under such severe conditions. \nThe results demonstrate that LCSS operates stably even when the model capacity is small.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel score matching variant called Local Curvature Smoothing with Stein’s Identity (LCSS), designed to address the computational challenges associated with calculating the Jacobian trace in score matching, particularly in high-dimensional settings. By leveraging Stein’s identity, LCSS bypasses the need for direct Jacobian evaluations, offering both computational efficiency and regularization benefits. Reviewers generally found the idea to be innovative and the method effective, as demonstrated through experiments on both synthetic and real datasets. However, concerns were raised about the paper's clarity, particularly regarding the advantages of LCSS over existing methods like Denoising Score Matching (DSM). Additionally, the paper could benefit from more detailed explanations of certain assumptions and a deeper comparison with methods that directly compute the Jacobian. Despite these issues, the contribution is deemed valuable, with promising potential for future exploration. I recommend accepting the paper.
Decision: Accept (poster)