NOTE

Communicated by Aapo Hyvärinen

A Connection Between Score Matching and
Denoising Autoencoders
Pascal Vincent
vincentp@iro.umontreal.ca
Département d’Informatique, Université de Montréal,Montréal (QC) H3C 3J7, Canada

Denoising autoencoders have been previously shown to be competitive
alternatives to restricted Boltzmann machines for unsupervised pretraining of each layer of a deep architecture. We show that a simple denoising autoencoder training criterion is equivalent to matching the score(with respect to the data) of a specific energy-based model to that of a
nonparametric Parzen density estimator of the data. This yields several
useful insights. It defines a proper probabilistic model for the denoising
autoencoder technique, which makes it in principle possible to sample from them or rank examples by their energy. It suggests a different
way to apply score matching that is related to learning to denoise anddoes not require computing second derivatives. It justifies the use of tiedweights between the encoder and decoder and suggests ways to extend
the success of denoising autoencoders to a larger family of energy-basedmodels.

1 Introduction
This note uncovers an unsuspected link between the score matching technique (Hyvärinen, 2005, 2008) for learning the parameters of unnormalized
density models over continuous-valued data, and the training of denoising autoencoders (Vincent, Larochelle, Bengio, & Manzagol, 2008; Vincent,Larochelle, Lajoie, Bengio, & Manzagol, 2010).
Score matching (SM) is an alternative to the maximum likelihood principle suitable for unnormalized probability density models whose partition
function is intractable. Its relationship to maximum likelihood has been
investigated by Lyu (2010), who formally relates the Fisher divergencethat yields score matching and the Kullback-Leibler divergence that yields
maximum likelihood. Interestingly, his formal analysis indicates that SM
searches for parameters that are more robust to small-noise perturbationsof the training data. Score matching has also been recast as a special caseunder the more general frameworks of generalized score matching (Lyu,

Neural Computation 23, 1661–1674 (2011)
C 2011 Massachusetts Institute of Technology

1662

P. Vincent

2010; Marlin, Swersky, Chen, & de Freitas, 2010) and minimum probability
flow (Sohl-Dickstein et al., 2009), allowing generalizations of SM to discrete
distributions (Hyvärinen, 2007b; Lyu, 2010; Sohl-Dickstein et al., 2009). The
minimum probability flow paradigm is particularly interesting as it unifies
several recent alternative parameter estimation methods, for both continuous and discrete data, under a single unified view.1 Recently, Kingma and
LeCun (2010) investigated a regularized form of SM that adds a specific regularization term to the original SM objective. Its relationship to this work is
discussed in detail in section 5.
Denoising autoencoders (DAE) were proposed by Vincent et al. (2008)as a simple and competitive alternative to the contrastive-divergencetrained restricted Boltzmann Machines (RBM) used by Hinton, Osindero, and Teh (2006) for pretraining deep networks (Erhan et al., 2010;
Vincent et al., 2010). Previous studies have already pointed out connections between SM and contrastive divergence (Hyvärinen, 2007a; SohlDickstein et al., 2009), have connected SM to optimal denoising for
gaussian noise with infinitesimal variance (Hyvärinen, 2008), and haveshown that training gaussian binary RBM with SM is equivalent totraining a regular (nondenoising) autoencoder with an additional regularization term (Swersky, 2010). This note, however, is the first torecast the training of a DAE as a form of regularized SM. This connection yields insights relevant to both research directions and suggests
a novel parameter estimation technique that has its roots in both DAE
and SM.
We begin with a brief presentation of the DAE architecture forcontinuous-valued inputs in section 2 and the SM technique in section 3.
This allows us to introduce our formalism and precise terminology. In section 4, we connect the denoising autoencoder objective to SM. We conclude
by a discussion on how our findings advance our understanding of both
approaches.
1.1 Notation. We are interested in techniques that learn the parameters
θ of a model by minimizing some objective function J (θ ). For uniformity ofnotation, all distributions will be represented by their probability density
functions (pdf) on Rd . The pdf for discrete distributions will be expressedwith Dirac-deltas δ.

1 Specifically SM (Hyvärinen, 2005), minimum velocity learning (Movellan, 2008), andcertain forms of contrastive divergence (Hinton, 2002; Welling & Hinton, 2002) are allrecast as minimizing the Kullback-Leibler divergence between the data distribution andthe distribution obtained after running, for infinitesimal time, a dynamic that would
transform it into the model distribution (Sohl-Dickstein et al., 2009).

A Connection Between Score Matching and Denoising Autoencoders

q (x)Dn = {x(1) , . . . , x(n) }nδ(x − x(i) )q 0 (x) = n1 i=1
1
2q σ (x̃|x) = (2π )1d/2 σ d e − 2σ 2 x̃−xq σ (x̃, x) = q σ (x̃|x)q 0 (x)q σ (x̃) = n1 nt=1 q σ (x̃|x(t) )p(x; θ )J1  J2
Eq (x) [g(x)] =x q (x)g(x)dxu, v =√ i ui vi
u = u, usoftplus(x) = log(1 + e x )sigmoid(x) = 1+e1 −x = softplus (x)
I
WT
Wi
W·, j

1663

Unknown true pdf. x ∈ Rd
Training set: i.i.d. sample from q
Empirical pdf associated with Dn
Smoothing kernel or noise model:
isotropic gaussian of variance σ 2
Joint pdf
Parzen density estimate based on Dn
obtainable by marginalizingq σ (x̃, x)Density model with parameters θMeans J 1 (θ ) and J 2 (θ ) are equivalent
optimization objectives2
Expectation with respect todistribution q
Dot product between two vectors
Euclidean norm of vector uWill be applied elementwise tovectors
Will be applied elementwise tovectors
Identity matrix
Transpose of matrix W
Vector for ith row of W
Vector for jth column of W

2 Denoising Autoencoders
Denoising autoencoders (DAEs) are a simple modification of classical autoencoder neural networks that are trained not to reconstruct their input
but rather to denoise an artificially corrupted version of their input (Vincent et al., 2008, 2010). Whereas an overcomplete regular autoencoder caneasily learn a useless identity mapping, a DAE must extract more usefulfeatures in order to solve the much harder denoising problem. DAEs have
proven to be an empirically successful alternative to restricted Boltzmann
machines (RBM) for pretraining deep networks (Vincent et al., 2008, 2010;
Erhan et al., 2010). Denoising autoencoders have also been used in different
contexts in the earlier works of LeCun (1987); Gallinari, LeCun, Thiria, andFogelman-Soulie (1987); and Seung (1998).

2 Equivalence will be asserted when J = α J + β with α > 0, β ∈ R. Indeed, a2
1
gradient-based optimization algorithm, when starting from some initial θ value, should
land in the exact same minimum whether optimizing J 1 or J 2 (this may, however, requirelearning rate adjustment to compensate for scaling factor α).

1664

P. Vincent

In this study, we consider the denoising version of a simple classicalautoencoder that uses a single sigmoidal hidden layer. Since data points
originate from a continuous real valued distribution, it is natural to use alinear decoder with a squared reconstruction loss.3 We will be using tiedweights whereby encoder and decoder share the same linear transformation
parameters. The considered corruption is additive isotropic gaussian noise.
A detailed description of the architecture follows:r A training input x ∈ Dn is first corrupted by additive gaussian noiseof covariance σ 2 I, yielding corrupted input x̃ = x + ,∼ N (0, σ 2 I).

This corresponds to conditional density q σ (x̃|x) = (2π )1d/2 σ d e − 2σ 2 x̃−x .
r The corrupted version x̃ is encoded into a hidden
representationh ∈ Rdh through an affine mapping followed by a nonlinearity:h = encode(x̃) = sigmoid(Wx̃ + b), where x̃ ∈ Rd , h ∈ (0, 1)dh , W is adh × d matrix and b ∈ Rdh .
r The hidden representation h is decoded into reconstruction xr throughaffine mapping: xr = decode(h) = WT h + c, where c ∈ Rd .
r The parameters θ = {W, b, c} are optimized so that the expectedsquared reconstruction error xr − x2 is minimized, that is, the objective function being minimized by such a DAE is1

2

J DAEσ (θ ) = Eqσ (x̃,x) [decode(encode(x̃)) − x2 ]= Eqσ (x̃,x) [WT sigmoid(Wx̃ + b) + c − x2 ].

(2.1)

3 Score Matching
3.1 Explicit Score Matching. Score matching was introduced byHyvärinen (2005) as a technique to learn the parameters θ of probability density models p(x; θ ) with intractable partition function Z(θ ), where p
can be written asp(x; θ ) =1
exp(−E(x; θ )).
Z(θ )E is called the energy function. Following Hyvärinen (2005), we willcall score the gradient of the log density with respect to the data vector:∂ log p(x;θ ). Beware that this use differs slightly from traditional
ψ(x; θ ) =∂x
statistics terminology where score usually refers to the derivative of the log
likelihood with respect to parameters, whereas here we are talking about ascore with respect to the data. The core principle of SM (Hyvärinen, 2005) is∂ log p(x;θ )to learn θ so that ψ(x; θ ) =best matches the corresponding score∂x
3 As opposed to a linear+sigmoid decoder with a Bernoulli cross-entropy loss, which
would be the preferred choice for binary input.

A Connection Between Score Matching and Denoising Autoencoders

1665∂ log q (x)of the true distribution: ∂ x . The corresponding objective function to beminimized is the expected squared error between these two vectors: 
2 ∂ log q (x) 1 .
ψ(x; θ ) −J E SMq (θ ) = Eq (x)2∂x
We refer to this formulation as explicit score matching (ESM).
Note that the score ψ(x; θ ) does not depend on troublesome Z(θ ). But∂ log q (x)since q is unknown, we do not have explicit regression targets.
∂xHyvärinen (2005) mentions in passing that nonparametric methods might
be used to estimate those, and we shall later pay closer attention to thispossibility.
3.2 Implicit Score Matching. Hyvärinen (2005) instead proceeds byproving the following remarkable property:
 
2 

1
∂
log
q
(x)
ψ(x; θ ) −Eq (x)


2
∂x



= Eq (x)J E SMq (θ )d∂ψi (x; θ )1ψ(x; θ )2 ++ C1 ,
2
∂xi
i=1


(3.1)

J I SMq (θ )∂ log p(x;θ ), and C1 is a constant that does notwhere ψi (x; θ ) = ψ(x; θ )i =∂xi
depend on θ . This yields an implicit SM objective J I SMq that no longer
requires having an explicit score target for q but is nevertheless equivalent
to J E SMq . Hyvärinen (2005) formally shows that provided q (x) and ψ(x; θ )satisfy some weak regularity conditions,4 we haveJ E SMq  J I SMq .

(3.2)

3.3 Finite Sample Version of Implicit Score Matching. Since we only
have samples Dn from q , Hyvärinen proposes to optimize the finite sample
version of J I SMq which, following our notation, we shall write as J I SMq0 :


d
2
1
∂ψ
(x;
θ
)
i
ψ(x; θ ) +J I SMq0 (θ ) = Eq0 (x)
2
∂xi
i=1

=

1
n

n

t=1


1
ψ(x(t) ; θ )2 +
2

d

i=1

∂ψi (x(t) ; θ )
.
∂xi

(3.3)

∂ log q (x) 2
4 Namely, q (x) and ψ(x; θ ) are differentiable, E
 ] is finite, and for any θ,
q (x) [∂x2
Eq (x) [ψ(x; θ ) ] is finite and limx→∞ q (x)ψ(x; θ ) = 0.

1666

P. Vincent

J I SMq0 is asymptotically equivalent to J I SMq when n → ∞ and hence
asymptotically equivalent to objective J E SMq . This can be summarized as
J E SMq  J I SMq  lim J I SMq0 .

(3.4)

n→∞

What happens in the transition from J I SMq to finite-sample version J I SMq0
is, however, not entirely clear. Concerns regarding the stability of the resulting criterion were raised by Kingma and LeCun (2010), who instead
propose optimizing a regularized version of J I SMq0 ,J I SMreg (θ ) = J I SMq0 (θ ) + λ

1
n

n

d

t=1 i=1∂ψi (x(t) ; θ )
∂xi

2
,

(3.5)

where the strength of the additional regularization term is controlled byhyperparameter λ. The relationship between this criterion and the criteria
we propose below are further discussed in section 5.
4 Linking Score Matching to the Denoising
Autoencoder Objective
4.1 Matching the Score of a Nonparametric Estimator. The possibility of matching the score ψ(x; θ ) with an explicit target score for q obtained through nonparametric estimation was mentioned but not pursued
in Hyvärinen (2005). We now examine this possibility more closely. Explicitly matching ψ(x; θ ) with the score of Parzen windows density estimatorq σ (x̃) yields the following objective:
 
2 
1
∂ log q σ (x̃)  .
ψ(x̃; θ ) −J E SMqσ (θ ) = Eqσ (x̃)

2
∂ x̃

(4.1)

For σ > 0, q σ is differentiable, decreases to 0 at infinity, and∂ log q σ (x̃) 2
 ] is finite. All regularity conditions are satisfied, so the∂ x̃same equivalence with ISM as in equation 3.2 holds:
Eq σ (x̃) [J E SMqσ  J I SMqσ .

(4.2)

Note that this equivalence breaks in the limit σ → 0 because q σ no longer
satisfies these regularity conditions and J E SMq σ can no longer be computed(whereas J I SMqσ remains well behaved).
4.2 Denoising Score Matching. Let us now consider a slightly different
objective, which is inspired by both the SM principle and the DAE approachof using pairs of clean and corrupted examples (x, x̃). For joint density

A Connection Between Score Matching and Denoising Autoencoders

1667

q σ (x̃, x) = q σ (x̃|x)q 0 (x), we define the following denoising score matching(DSM) objective:
 
2 

∂
log
q
1
(x̃|x)σψ(x̃; θ ) − .
J DSMqσ (θ ) = Eqσ (x,x̃)


2
∂ x̃

(4.3)

The underlying intuition is that following the gradient ψ of the log
density at some corrupted point, x̃ should ideally move us toward the clean
sample x. Note that with the considered gaussian kernel, we have
∂ log q σ (x̃|x)1= 2 (x − x̃).
∂ x̃
σ

(4.4)

Direction σ12 (x − x̃) clearly corresponds to moving from x̃ back toward clean
sample x, and we want ψ to match that as best it can.
This objective, inspired by denoising autoencoders, is equivalent to explicit SM. Formally,J E SMqσ  J DSMqσ .

(4.5)

The proof is in the appendix and does not depend on the particular formof q σ (x̃|x) as long as log q σ (x̃|x) is differentiable with respect to x̃.
4.3 An Energy Function That Yields the Denoising Autoencoder
Objective. Let us now choose for model p the form
1
exp(−E(x; θ )),
Z(θ )



h
c, x − 12 x2 + dj=1
softplus W j , x + b j. (4.6)E(x; W, b, c) = − σ2p(x; θ ) =θWe then haveψi (x; θ ) =∂ log p(x; θ )
∂xi

∂E
∂xi
⎛
⎞


dh


 ∂ Wj, x + bj
1 ⎝
⎠
= 2 ci − xi +softplus W j , x + b j
σ
∂xi
=−

j=1

1668

P. Vincent

⎛
⎞
dh



1 ⎝
= 2 ci − xi +sigmoid W j , x + b j W ji ⎠
σ
j=1

=



1 
ci − xi + W·i , sigmoid(Wx + b) ,σ2which we can write as the single equation
ψ(x; θ ) =
1  TW sigmoid(Wx + b) + c − x .
2
σ

(4.7)

Substituting equations 4.4 and 4.7 in the expression for J DSMqσ ,equation 4.3, we get, for this choice of Parzen kernel and density model, 2 ∂ log q σ (x̃|x) 1ψ(x̃; θ ) −J DSMqσ (θ ) = Eqσ (x,x̃)

2
∂ x̃
 
1 1
= Eqσ (x,x̃) (WT sigmoid(Wx̃ + b) + c − x̃)
2σ2
2 

1
− 2 (x − x̃)

σ
2 

1
Eq σ (x,x̃) WT sigmoid(Wx̃ + b) + c − x
42σ
1
= 4 J DAEσ (θ ).
2σ=We have thus shown thatJ DSMqσ  J DAEσ .

(4.8)

5 Discussion
Putting together equations 4.2, 4.5, and 4.8, we can write, for σ > 0,J I SMqσ  J E SMqσ  J DSMqσ  J DAEσ .

(5.1)

In summary, training the denoising autoencoder defined in section 2 isequivalent to performing SM (explicit or implicit) with the energy functionof equation 4.6 on Parzen density estimate q σ . Such training would typicallyuse stochastic gradient descent, whereby samples from q σ are obtained by

A Connection Between Score Matching and Denoising Autoencoders

1669

corrupting samples from Dn . And it may be carried out with any of thesefour optimization objective formulations.5
We introduced the kernel-smoothed empirical distribution q σ to show
a connection between SM and a simple DAE. Interestingly, the regularized SM criterion J I SMreg (see equation 3.5) that Kingma and LeCun (2010)
recently introduced with the very different motivation of curing possible
instabilities, was derived by approximating what amounts to J I SMqσ . 6 From
this perspective, our four q σ -based criteria in equation 5.1, including theDAE, may be seen as alternative approximation-free forms of regularized
score matching. A key difference is that, as is done with DAE training, we
would optimize stochastic versions of these approximation-free regularized
criteria by corrupting training examples (i.e., sampling form q σ ), whereas
Kingma and LeCun (2010) optimize an approximation of J I SMqσ , centered on
the training samples only (i.e., sampling from q 0 ). Also, whereas J I SMreg , likethe other ISM criteria, requires computing second derivatives, the stochastic version of our novel J DSMqσ criterion does not, and thus appears much
simpler to implement.
Note that the energy function in equation 4.6 is particular in that its
scaling, which we may call its temperature, is chosen to match the corrupting noise level σ 2 . This is required only to establish the last equivalencewith the specific DAE we considered. But regarding the generic objectives
J I SMqσ  J E SMqσ  J DSMqσ , their σ may in principle be chosen regardlessof the form or temperature of whatever energy function is to be learned.
Interestingly, the energy function in equation 4.6, which we designed toyield the equivalence with our denoising autoencoder objective, happens
to be very similar to the free energy of a restricted Boltzmann machine withbinary hidden units and gaussian visible units (Welling, Rosen-Zvi, & Hinton, 2005; Bengio, Lamblin, Popovici, & Larochelle, 2007; Swersky, 2010).
The major difference is that this latter free energy does not have a global
temperature scaling of the whole expression.7 We designed equation 4.6 toexactly yield the denoising version of the classic autoencoder described in
section 2. But with tied weights, it may be preferable to allow an extra positive scaling parameter α for the reconstruction, so that there at least exists
an equivalent reparameterization of the model for scaled input data.8 This
5 Note, however, that while these q -based objectives are formally equivalent, theirσstochastic gradient descent optimization, based on sampling a limited number of corrupted examples, is likely to behave differently for each objective.
6 A first-order Taylor expansion and a diagonal Hessian approximation are used.
7 Specifically, in the free energy of a gaussian-binary RBM, the softplus terms are notdivided by σ 2 or scaled in any way.
8 If, for example, one multiplies the input values by 100, one can obtain the same
hidden representation as before by dividing W by 100. But because of the tied weights,
this means that the reconstruction would also be divided by 100 (i.e., there is no equivalent reparameterization), unless it can be compensated by an additional scaling of thereconstruction by a parameter α.

1670

P. Vincent

is easily obtained in the energy function by multiplying the sum of softplus
terms in equation 4.6 by α. We may even allow an arbitrary rescaling factor α j for each hidden-layer dimension independently by multiplying eachsoftplus term by its own rescaling parameter α j , which yields the following
more flexible energy function:
E(x; W, b, c, α, σm )


θ

⎛
1
1
= − 2 ⎝c, x − x2 +
σm
2

dh

⎞

αj softplus W j , x + b j ⎠ .




j=1

Here we have also included, as model parameter, a σm (where m stands formodel) distinct from the noise σ of the training objective.9
Our q σ -based objectives J I SMqσ , J E SMqσ , or J DSMqσ can be used as alternatives to the finite sample objective J I SMq0 (see equation 3.3) advocated in
Hyvärinen (2005) for learning unnormalized densities. Note that J I SMq0 is a
special case of J I SMqσ obtained in the limit of σ → 0. Also, since Kingma and
LeCun (2010) showed that it may be preferable to use a regularized criterion(that they derived from smoothed empirical distribution q σ ), it is likely that
our q σ -based criteria may, for σ > 0, yield better generalization performance
than the J I SMq0 advocated in Hyvärinen (2005).10 It seems that σ could allow one to choose an optimal bias-variance trade-off for the finite-sample
estimation of the true SM gradient of interest ∇θ J E SMq = ∇θ J I SMq . While∇θ J I SMq0 is an unbiased estimator of it, ∇θ J I SMqσ = ∇θ J E SMqσ = ∇θ J DSMqσwill generally be biased when σ > 0 but are also likely to have a lower
variance.
Among the three equivalent SM objectives based on q σ , objective J DSMqσappears particularly interesting as a novel alternative formulation. It was
motivated by both the SM and the DAE principles. From DAE, it borrows theidea of learning to denoise artificially corrupted samples, and from SM, itborrows the idea of learning a score function derived from an unnormalized
density. J DSMqσ may prove simpler and more efficient in practice than themathematically equivalent J I SMqσ because it does not require computing
second derivatives.
Our result is also a significant advance for DAEs. First, we have defineda proper energy function for the considered DAE through equation 4.6.
9 We would, however, have to set σ = σ to recover a recognizable denoising autoenm
coder objective.
10 It is also noteworthy that the experimental results of Vincent et al. (2008, 2010)
on DAE showed that the best models, judged by their ability to extract useful features,
were obtained for nonnegligible values of the noise parameters. Moreover, this way ofcontrolling the model’s capacity worked much better than either reducing the hidden
layer size or than traditional weight decay.

A Connection Between Score Matching and Denoising Autoencoders

1671

This will enable many previously impossible or ill-defined operations on a
trained DAE, for example, deciding which is the more likely among several
inputs, or sampling from a trained DAE using hybrid Monte Carlo (Duane,
Kennedy, Pendleton, & Roweth, 1987). Second, whereas using the same
weight matrix (“tied weights”) for the encoder and decoder is justified for
RBMs, the encoder-decoder framework does not constrain that choice. Previous work on DAEs (Vincent et al., 2008, 2010; Erhan et al., 2010) exploredboth options, often finding tied weights to yield better empirical results.
Within the SM framework presented here, using tied weights between encoder and decoder now has a proper justification, since it follows naturally
from differentiating the energy. Third, this framework opens the door to
new variants that would naturally fall out from other choices of the energyfunction.
Appendix: Proof That J E SMqσ  J DSMqσ Equation 4.5
The explicit score matching criterion using the Parzen density estimator isdefined in equation 4.1 as
 
2 
1
∂ log q σ (x̃)  ,J E SMqσ (θ ) = Eqσ (x̃)
ψ(x̃; θ ) −

2
∂ x̃
which we can develop as
J E SMqσ (θ ) = Eqσ (x̃)

1ψ(x̃; θ )2 − S(θ ) + C2 ,
2

(A.1)

  ∂ log q (x̃) 2 
where C2 = Eqσ (x̃) 12  ∂ x̃σ  is a constant that does not depend on θ ,
and


∂ log q σ (x̃)
S(θ ) = Eqσ (x̃) ψ(x̃; θ ),
∂ x̃



∂ log q σ (x̃)
d x̃= q σ (x̃) ψ(x̃; θ ),
∂ x̃
x̃


∂
q σ (x̃)= q σ (x̃) ψ(x̃; θ ), ∂ x̃d x̃q σ (x̃)x̃ ∂ψ(x̃; θ ), q σ (x̃) d x̃
=
∂ x̃
x̃


 
∂
ψ(x̃; θ ),q 0 (x)q σ (x̃|x)dx d x̃=∂ x̃ xx̃

1672

P. Vincent ∂q σ (x̃|x)=ψ(x̃; θ ), q 0 (x)dx d x̃
∂ x̃
x̃
x


 
∂ log q σ (x̃|x)ψ(x̃; θ ), q 0 (x)q σ (x̃|x)dx d x̃
=
∂ x̃
x̃
x


 
∂ log q σ (x̃|x)
dxd x̃
q 0 (x)q σ (x̃|x) ψ(x̃; θ ),
=
∂ x̃
x̃ x


 
∂ log q σ (x̃|x)=
dxd x̃
q σ (x̃, x) ψ(x̃; θ ),
∂ x̃
x̃ x


∂ log q σ (x̃|x)= Eqσ (x̃,x) ψ(x̃; θ ),.
∂ x̃Substituting this expression for S(θ ) in equation A.1 yields




1
ψ(x̃; θ )22∂ log q σ (x̃|x)+ C2 .
−Eqσ (x,x̃) ψ(x̃; θ ),∂ x̃J E SMqσ (θ ) = Eqσ (x̃)(A.2)

We also have defined in equation 4.3, 
2 ∂ log q σ (x̃|x) 1 ,ψ(x̃; θ ) −J DSMqσ (θ ) = Eqσ (x,x̃)

2
∂ x̃
which we can develop as



2
1


ψ(x̃; θ )J DSMqσ (θ ) = Eqσ (x̃)
2∂ log q σ (x̃|x)+ C3 ,−Eqσ (x,x̃) ψ(x̃; θ ),
∂ x̃

(A.3)

  ∂ log qσ (x̃|x) 2  is a constant that does not depend
where C3 = Eqσ (x,x̃) 12 
∂ x̃on θ .
Looking at equations A.2 and A.3, we see that J E SMqσ (θ ) = J DSMqσ (θ ) +C2 − C3 . We have thus shown that the two optimization objectives areequivalent.
Acknowledgments
I thank Yoshua Bengio, Olivier Delalleau, and the other members of the Lisa
Lab who provided timely feedback, as well as two anonymous referees

A Connection Between Score Matching and Denoising Autoencoders

1673

whose thoughtful comments and suggestions helped improve this note.
This research was supported by NSERC, MITACS, and FQRNT.
References
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances inneural information processing systems, 19 (pp. 153–160). Cambridge, MA: MIT Press.
Duane, S., Kennedy, A., Pendleton, B., & Roweth, D. (1987). Hybrid Monte Carlo.
Phys. Lett. B, 195, 216–222.
Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010).
Why does unsupervised pre-training help deep learning? Journal of Machine
Learning Research, 11, 625–660.
Gallinari, P., LeCun, Y., Thiria, S., & Fogelman-Soulie, F. (1987). Memoires
associatives distribués. In Proceedings of COGNITIVA 87. Paris: Cesta-Afcet.
Hinton, G. E. (2002). Training products of experts by minimizing contrastive
divergence. Neural Computation, 14, 1771–1800.
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep
belief nets. Neural Computation, 18, 1527–1554.
Hyvärinen, A. (2005). Estimation of non-normalized statistical models using scorematching. Journal of Machine Learning Research, 6, 695–709.
Hyvärinen, A. (2007a). Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables. IEEE Transactions
on Neural Networks, 18, 1529–1531.
Hyvärinen, A. (2007b). Some extensions of score matching. Computational Statistics
and Data Analysis, 51, 2499–2512.
Hyvärinen, A. (2008). Optimal approximation of signal priors. Neural Computation,
20(12), 3087–3110.
Kingma, D., & LeCun, Y. (2010). Regularized estimation of image statistics by scorematching. In J. Lafferty, C.K.I. Williams, J. Shawe-Taylor, R. Zemel, & A. Culotta(Eds.), Advances in neural information processing systems, 23 (pp. 1126–1134).
Cambridge, MA: MIT Press.
LeCun, Y. (1987). Modèles connexionistes de l’apprentissage. Unpublished doctoral
dissertation, Université de Paris VI.
Lyu, S. (2010). Interpretation and generalization of score matching. In Proceedings ofthe 25th Conference in Uncertainty in Artificial Intelligence (UAI’09). Corvallis, OR:
AUAI Press.
Marlin, B., Swersky, K., Chen, B., & de Freitas, N. (2010). Inductive principles
for restricted Boltzmann machine learning. In Proceedings of The ThirteenthInternational Conference on Artificial Intelligence and Statistics (AISTATS’10) (Vol. 9,
pp. 509–516). N.p.: JMLR.
Movellan, J. R. (2008). Contrastive divergence in gaussian diffusions. NeuralComputation, 20(9), 2238–2252.
Seung, S. H. (1998). Learning continuous attractors in recurrent networks. In
M. Jordan, M. Kearns, & S. Solla (Eds.), Advances in neural information processing
systems, 10 (pp. 654–660). Cambridge, MA: MIT Press.

1674

P. Vincent

Sohl-Dickstein, J., Battaglino, P., & DeWeese, M. R. (2009). Minimum probability flow
learning (Tech. Rep.). arXiv:0906.4779.
Swersky, K. (2010). Inductive principles for learning restricted Boltzmann machines.
Unpublished master’s thesis, University of British Columbia.
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting andcomposing robust features with denoising autoencoders. In W. W. Cohen,
A. McCallum, & S. T. Roweis (Eds.), Proceedings of the 25th Annual International
Conference on Machine Learning (pp. 1096–1103). New York: ACM.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P.-A. (2010). Stacked
denoising autoencoders: Learning useful representations in a deep network witha local denoising criterion. Journal of Machine Learning Research, 11, 3371–3408.
Welling, M., & Hinton, G. E. (2002). A new learning algorithm for mean fieldBoltzmann machines. In ICANN ’02: Proceedings of the International Conference on
Artificial Neural Networks (pp. 351–357). Berlin: Springer-Verlag.
Welling, M., Rosen-Zvi, M., & Hinton, G. E. (2005). Exponential family harmoniums
with an application to information retrieval. In L. Saul, Y. Weiss, & L. Bottou(Eds.), Advances in neural information processing systems, 17 (pp. 1481–1488).
Cambridge, MA: MIT Press.

Received September 30, 2010; accepted December 15, 2010.