2014 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 21–24, 2014, REIMS, FRANCEMAXIMUM RECONSTRUCTION PROBABILITY TRAINING OF RESTRICTEDBOLTZMANN MACHINES WITH AUXILIARY FUNCTION APPROACH
Norihiro Takamune1) and Hirokazu Kameoka1),2)
1)

2)

Graduate School of Information Science and Technology, The University of Tokyo,
NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation{takamune,kameoka}@hil.t.u-tokyo.ac.jp
ABSTRACT

Restricted Boltzmann machines (RBMs) are stochastic neural networks that can be used to learn features from raw data.
They have attracted particular attention recently after being
proposed as building blocks for deep belief network (DBN)
and have been applied with notable success in a range ofproblems including speech recognition and object recognition. The success of these models raises the issue of how
best to train them. At present, the most popular training algorithm for RBMs is the Contrastive Divergence (CD) learning algorithm. We propose deriving a new training algorithm
based on an auxiliary function approach for RBMs using thereconstruction probability of observations as the optimization
criterion. Through an experiment on parameter training of anRBM, we confirmed that the present algorithm outperformed
the CD algorithm in terms of the convergence speed and thereconstruction error when used as an autoencoder.
Index Terms— Deep learning, deep belief networks, restricted Boltzmann machine, auxiliary function approach
1. INTRODUCTION
In recent years, Deep Belief Networks (DBN) [1] have been
applied with notable success to a wide range of applications
including image recognition and speech recognition [2]. DBN
is composed of multiple layers, each of which can be viewed
as a restricted Boltzmann machine (RBM) [3]. Layerwise
unsupervised pre-training of RBM has been found effective
for the training of the entire DBN. At present, the most popular training algorithm for RBMs is the Contrastive Divergence (CD) learning algorithm [4]. If we can develop a fasterand better-behaved training algorithm for RBMs, the computational time for learning DBN can be reduced accordingly.
For many nonlinear optimization problems, parameter estimation algorithms constructed using an auxiliary function
have proven to be very effective. The general principle forthe parameter estimation scheme using an auxiliary function
is referred to as the “auxiliary function approach” (or alternatively the “minorization-maximization (MM) approach” [5]).
Note that the auxiliary function approach itself is not an algorithm, but a description of how to construct an optimization
algorithm. When applying an auxiliary function approachThis work was supported by JSPS KAKENHI Grant Number 26730100.

c
978-1-4799-3694-6/14/$31.00 ⃝2014IEEE

to a certain optimization problem, the first step is to design
an auxiliary function that upper-bounds or lower-bounds theobjective function. An algorithm that consists of iteratively
minimizing/maximizing the auxiliary function is guaranteed
to converge to a stationary point of the objective function. Itshould be noted that this concept is adopted in many existing algorithms. For example, the expectation-maximization(EM) algorithm [6] builds a surrogate for a likelihood function of latent variable models by using Jensen’s inequality. It
is also well known for its use in devising an algorithm for
non-negative matrix factorization [7, 8]. In general, if we
can build a tight upper/lower bound function for the objective function of a specific optimization problem, we expect toobtain a fast-converging algorithm. In fact, the authors andcolleagues have thus far proposed deriving parameter estimation algorithms based on the auxiliary function approachfor various optimization problems, some of which have been
proven to be significantly faster than gradient-based methods(e.g., [8–15]). In [16], we have proposed deriving a new algorithm based on the auxiliary function approach for maximum
likelihood training of RBMs. By contrast, this paper considers the problem of maximizing the reconstruction probabilityof training samples (that can be interpreted as an autoencoder
training criterion), and proposes an optimization algorithm
for that problem based on the auxiliary function approach.
2. RESTRICTED BOLTZMANN MACHINES
2.1. Objective functions for training RBM
An RBM has a structure called complete bipartite graph (seeFig. 1). One side of an RBM is called a visible layer corresponding to the observations and the other side is called ahidden layer corresponding to the features. Let us denote theIunits in the visible and hidden layers by v = {vi } ∈ {0, 1}
Jand h = {hj } ∈ {0, 1} , respectively, where I and J are
the numbers of the units in the visible and hidden layers. The
joint probability distribution of h and v is defined asp (v, h|Θ) =where
E(v, h; Θ) = −∑iexp (−E (v, h; Θ))
,
Z (Θ)

bV
i vi −∑
jbH
j hj −∑i,j(1)

Wij vi hj , (2)to a computational complexity which is in general exponential
in the number of variables. As for the first term, we can use
the fact that the hidden units are conditionally independent
given the state of the visible variables and vice versa:∏p (v|h, Θ) =p (vi |h, Θ) ,(10)iFig. 1. ∑Graph∑structure of RBM.
Z (Θ) =
exp (−E (v, h; Θ)) ,v(3)and Θ =
model
∑parameters of the RBM. In this paper, we will use v
and h to mean the sums over all possible patterns of v
and h, respectively. The objective of RBM learning involves
estimating Θ using N training samples, v (1) , . . . , v (N ) .
A frequently used objective function for RBM learning is
the mean log-likelihood given as follows:1 ∑log p(v (n) |Θ)
N n
∑
1 ∑
log
p(v (n) , h|Θ).
N n

=

(4)

h

Since an RBM can be used as an autoencoder [17], we canalso define and utilize the reconstruction probability of inputs
as an alternative objective function for RBM learning:
Jr (Θ) =

1 ∑
N

log

n

p (hj |v, Θ) .

(11)

j

h

H
(bV
the
i , bj , Wij ) denotes the set consisting of ∑J(Θ) =

p (h|v, Θ) =∏∑p(v (n) |h, Θ)p(h|v (n) , Θ),(5)hThe conditional distributions p(hj |v, Θ) and p(vi |h, Θ) canbe interpreted as the firing rate of each neuron, given by()∑
p(vi = 1|h, Θ) = σ bV
Wij hj ,(12)i +(p(hj = 1|v, Θ) = σ

j

bH
j +

∑

)
Wij vi ,

(13)

i
−x
where σ denotes the logistic∑ sigmoid:∑ σ(x) = 1/(1 + e ).
Hence, we can replace h with j , thus allowing us to calculate the expectation of the first term in (8) without the exponential complexity. However, it is still difficult to computethe second term in (8). One convenient way is to approximate
the expectation by samples from the joint distribution. Thesesamples can be obtained with Gibbs sampling using

hd−1∼ p(hj |v d−1 , Θ),j(14)

vid ∼ p(vi |hd−1 , Θ),(15)

where d denotes the step in the Gibbs sampling chain.

wherep(v, h|Θ),′v ′ p(v , h|Θ)p(v, h|Θ).
p(h|v, Θ) = ∑′h′ p(v, h |Θ)p(v|h, Θ) = ∑

(6)
(7)

2.2. Contrastive Divergence Learning [1, 4]
Since it is not possible to find the parameters maximizing themean log-likelihood analytically, one convenient way would
be to apply the gradient ascent approach.
The partial derivative of the mean log-likelihood with respect to Θ is given by∂J
1 ∑∑∂E (n)(Θ) = −(v , h; Θ)p(h|v (n) , Θ)
∂Θ
N n
∂Θ
h
(8)
∑∑
∂E
+
p(v, h|Θ)(v, h; Θ).
∂ΘvhThis leads to a simple update rule based on steepest ascent in
the log-likelihood:Θ ← Θold + ϵ∆cd Θ,(9)where, ϵ is the learning rate and ∆cd Θ = ∂J/∂Θ. Directly
calculating the sums that run over all values of v and h leads

3. RBM LEARNING WITH AUXILIARY FUNCTION
APPROACH
3.1. Auxiliary function approach
Here we introduce the general principle of the auxiliary function approach (a.k.a the MM approach). Let us use G(θ) to
denote an objective function that we want to maximize withrespect to θ. G+ (θ, θ̃) is defined as an auxiliary function forG(θ) if it satisfies
G(θ) = max G+ (θ, θ̃).

(16)

θ̃

We call θ̃ an auxiliary variable. By using G+ (θ, θ̃), G(θ) canbe iteratively increased viaθ ← argmax G+ (θ, θ̃),

(17)

θ

θ̃ ← argmax G+ (θ, θ̃).

(18)

θ̃

Since it is preferrable to use a function that can be maximized
analytically as an auxiliary function, here we aim to design an
auxiliary function such that the parameters are separated into
individual terms:∑
G+ (θ, θ̃) =gk (θk , θ̃) + C,(19)k

where θk is the k-th element of θ, gk is a function that depends
only on θk and θ̃, and C is a term that does not depend on
θ. If we can design such a lower bound function, the global
maximization of G+ can be achieved by separately solving
the problems of maximizing g1 , . . . , gK .
3.2. Auxiliary function approach for ML training [16]In [16], we have proposed deriving a new algorithm based on
the auxiliary function approach for maximizing J(Θ), givenby (4). The update rules are given as
∑ (n)
V,old
n vi
∑
∑
←
b
+
β
log
bV
,
(20)
i
i
i
p
v
h (v, h|Θ) vi
)
∑ (
(n)
,Θ
H,old
n p hj = 1|v
H, (21)bj ← bj+ β(I+j) log ∑ ∑v
h p (v, h|Θ) hj
Wij ← Wijold(22))∑ (n) ((n),Θ
n vi p hj = 1|v+ β(I+J+(i−1)J+j) log ∑ ∑,v
h p (v, h|Θ) vi hj

where, βi , β(I+j) and β(I+J+(i−1)J+j) (which we hereafter
denoteby βk ) are arbitrary values subject to βk ∈ [0, 1] and
∑
β
= 1.
k
k
It is interesting to compare these update rules with theupdate rules of the CD algorithm. While each step of the CD
algorithm moves the parameters in the direction given by thedifference of the two expectations, the proposed update rulemoves the parameters in the direction given by the differenceof the logarithms of the two expectations.
3.3. Auxiliary function approach for maximizing reconstruction probability
This paper focuses on maximizing (5) and proposes a new
optimization algorithm based on the auxiliary function approach. Instead of directly using (5) as the objective function,
here we propose to approximate it by:
1 ∑
Jr (Θ) ≈log p(v (n) |h(n) , Θ)p(h(n) |v (n) , Θ)
N n
(23)
˜
≡ Jr (Θ) ,(n)where h indicates a sample drawn from the following conditional distribution
h(n) ∼ p(h|v (n) , Θ).

(24)

By using (10), (11), (12) and (13), J˜r (Θ) can be written as{()
1 ∑ ∑ (n)V˜
Jr = −
vi log 1 + exp(−fni
)
N n
i
∑
(n)
V
+
(1 − vi ) log(1 + exp fni
)
i

+

∑
j

(n)

hj

(
)
H
log 1 + exp(−fnj
)

}
∑
(n)
H
+
(1 − hj ) log(1 + exp fnj ) ,
j

(25)

∑
∑
(n)
(n)
V
H
H
where fni
= bVi +j Wij hj , fnj = bj +i Wij vi .
Difficulty in solving the optimization problem of maximizing (25) lies in the fact that the terms depending on theparameters Θ are the arguments of the logarithm functions.
Since a negative logarithm function is a convex function, we
can use its tangent line as a lower bound function:− log (1 + exp f ) ≥ −1 + exp f− log ζ + 1,
ζ

(26)

V
V
H
H
where f represents fni, −fni, fnj
or −fnj, that is,

f = b∗ +∑Wl∗ xl ,(27)

land ζ is an auxiliary variable. Equality of (26) holds when
ζ = 1 + exp f.

(28)

Next, since a negative exponential function is a concave
function, we can design a lower bound function by using
Jensen’s inequality(∗− exp b +∑( ∗)b − α0≥ −β0 exp
β0()∑
Wl∗ xl − αl−βl exp, (29)βlWl∗ xll)lwhere α0 , αl , β0 and βl are auxiliary variables that must satisfy
α0 +∑αl = 0,(30)

β0 , βl ∈ [0, 1],∑βl = 1.
β0 +

(31)

l

(32)

l

Equality of (29) holds when()∑
α0 = b∗ − β0 b∗ +Wk∗ xk ,

(33)

k

(
)
∑
αl = Wl∗ xl − βl b∗ +Wk∗ xk .

(34)

kwe can construct a lower bound function(Therefore,)J˜r+ Θ, Θ̄ of J˜r (Θ) by using (25), (26) and (29). Now,
let us consider performing the updates (17) and (18). The
maximizers of the lower bound function with respect to theauxiliary variables ζ, α0 and αl are given by (28), (33) and(34). By substituting (28), (33) and (34) into the lower bound



i

+

∑
j

+

(35)

) H
(n) (
H
hj 1 − q̂nj
ξnj

∑(

(n)
1 − hj

)

}
H H
q̂nj
ηnj

( )
+ C Θ̄ ,j((n)
vi = 1|h(n) , Θold

(

(n)

H
q̂nj
= p hj

= 0.

)
)

,

= 1|v (n) , Θold ,
∑
V (n)
hj
V
V −b̂VV −Ŵij
ξni
= βi0
e i +
βij
e
,

(36)

∑

,

(38)
(39)

j
H
H −b̂j
ξnj
= β0j
e
+
H

∑

H (n)

H −Ŵij vi
βij
e
,

(40)

i

H

H b̂j
H
e +
= β0j
ηnj

∑

H (n)

H Ŵij vi
e
,
βij

(41)

i

where bV,old, bH,oldand Wijold correspond to the parameijter estimates obtained at the previous iteration, and b̂V
i =
V,old
bV
i −bi
,
V
βi0
old
Wij −Wij
H
βij

b̂H
j

H,old
bH
j −bj
,
H
β0j

old
Wij −Wij
,
V
βij

ŴijV

ŴijH

=
=
=
( )
, C Θ̄ is a constant term about Θ and auxiliary

V
V
H
H
variables Θ̄ are bV,old, bH,old, Wijold , βi0
, βij
, β0j
and βij
.
i
j
V
V
H
H
Note that βi0 , βij , β0j and βij are arbitrary variables thatmust satisfy
H
V
V
H
, β0j
, βij
∈ [0, 1],
, βij
βi0
∑
V
V
βij = 1, ∀i,
βi0 +

(42)

j

H
β0j
+

∑

H
βij
= 1,

∀j.

i

We notice that in this lower bound function, the parameters
are separated in individual terms. Thus, the auxiliary function
can be maximized with respect to Θ by maximizing each termH˜+separately. By solving ∂ J˜r+ /∂bV
i = 0 and ∂ Jr /∂bj = 0, we
obtain closed-form update rules:)∑ (n) (
V
V
v
1
−
q̂
β
V,old
)ni  ,bV
+ i0 log  ∑ n (i(43)
i = bi
(n)
2
1−v
q̂ V
n

i

ni

ij

(37)

(n)

V
V Ŵijβij
e hj∂ 2 J˜+However, ∂W r2 are always negative:

j

V
V b̂V
ηni
= βi0
ei +

(44)

Unfortunately, it is difficult to represent ∂ J˜r+ /∂Wij = 0 in
closed form:{) (n) −Ŵ V h(n)∂ J˜r+1 ∑(n) (V
1 − q̂ni
hj e ij j
=−
− vi
∂Wij
N n
(
)
V (n)
(n)
V (n) Ŵij
+ 1 − viq̂ni
hj e hj) (n) −Ŵ H v(n) (45)(n) (Hvi e ij i− hj 1 − q̂nj
}
)
(
H (n)
(n)
vi
H (n) Ŵij
q̂nj vi e+ 1 − hj

with
V
q̂ni
=p



)
∑ (n) (
H
H
1 − q̂nj
β0j
n hj
H,old
H

.
(
)
log ∑
bj = bj
+
(n)
2
H
1
−
h
q̂
j
nj
n

function, we have()
J˜r+ Θ, Θ̄{) V
1 ∑ ∑ (n) (V=−vi
1 − q̂ni
ξni
N n
i
)
∑(
(n)
V V
+
1 − viq̂ni
ηni∂ 2 J˜r+∂Wij2{) ( (n) )2 −Ŵ V h(n)1 ∑ 1 (n) (
V
v
1
−
q̂
hj
e ij j=−niiV
N n βij
)
(
)2
V (n)
1 (
(n)
(n)
V
+ V 1 − viq̂nihjeŴij hj
(46)
βij
(
)
2
(
)
(n)
H
1 (n)
(n)
H
vi
e−Ŵij vi+ H hj 1 − q̂nj
βij
}
)
(
)2
H (n)
1 (
(n)
(n)
H
Ŵij
vi
+ H 1 − hjq̂nj vi
e
βij
< 0,
(n)

(n)

V
H
where vi , hj ∈ {0, 1}, q̂ni
, q̂nj∈ [0, 1], therefore, the+
˜solution of ∂ Jr /∂Wij = 0 can be efficiently obtained with
Newton’s method.
It is interesting to compare the updates (43) and (44) withV
those of the CD algorithm (9). From this perspective, βi0
and
H
V
H
β0j as well as βij and βij can be regarded as the learning
H
V
V
H
, β0j
and βij
must satisfy the sum-to-unity, βij
rate. Since βi0
constraints, the larger the number of the hidden and visible
V
V
H
H
variables becomes, the smaller βi0
, βij
, β0j
and βij
becomein average, thus slowing the convergence speed of the algorithm. We can thus expect to accelerate the algorithm by replacing β ′ with β γ (where β is intended to be an abbreviationVV
H
H
for βi0, βij, β0j
and βij), setting γ ∈ [0, 1] at a reasonably
small value at the early stage of the algorithm and moving it
towards 1 as the iteration proceeds.

4. EXPERIMENT
We conducted an experiment to compare the performance ofthe present algorithm against the maximum likelihood training algorithm based on the auxiliary function approach described in [16] (hereafter referred to as “auxiliary function1”) and the CD algorithm. In order to evaluate the values

of the likelihoods, we used a small scale RBM with I = 10
visible units and J = 8 hidden units. We used N = 2000
randomly-generated binary data as the training examples. All
the algorithms were run for T = 500 iterations. The initial
parameters were set at the same values for all the algorithms,
which were generated randomly.
The arbitrary auxiliary variables of the present method
V
V
H
H
βi0
, βij
, β0j
and βij
were set uniformly:
4

(47)

(

)
h̄ = argmax p h|v , Θ ,
h
(
)
(n)
(n)
v̄ = argmax p v|h̄ , Θ .
(n)

−1.35
−1.4
−1.45

Contrastive Divergence
Auxiliary Function 1
Auxiliary Function 1 with gamma
Auxiliary Function 2
Auxiliary Function 2 with gamma

−1.5

(49)

−1.55

(50)

respectively, where we set ϵinit = 1, ϵend = 0.1, γinit = 0.1
and γend = 1. The Gibbs sampling and Newton’s method
were both run for 1 iteration at each iterative step. We implemented all the learning algorithms in MATLAB.
In addition to the log-likelihood (4), we used the reconstruct error rate Ereconst for the measure for comparisons:)2
1 ∑ ∑ ( (n)(n)
Ereconst =vi − v̄i,(51)
NI n i(n)log−likelihood

−1.3

(

where

x 10

−1.25

(48)

This was also the case for “auxiliary function 1.” The learning
rate ϵ of the CD algorithm and the value of γ were scheduledin the following way: At step t of the algorithms, we updated
ϵ and γ at
) t−1
ϵend T −1
ϵ (t) = ϵinit
,
ϵinit
(
) t−1
γend T −1
γ (t) = γinit
γinit

−1.2

(52)
(53)

v

Fig. 2 shows the evolution of the log-likelihoods (4) andthe reconstruction error rate (51) obtained with all the algorithms. With the MATLAB implementation, the execution
time per iteration for the present algorithm and “auxiliary
function 1” was about the same. For the present algorithmand “auxiliary function 1,” the convergence in terms of of thelikelihoods was faster when updating γ than when fixing it at
1, as expected. It is interesting to note that the present algorithm tended to increase the likelihood score even though it
was designed to optimize a criterion different from the likelihood.
Next, we investigated the case where the number of thestates of RBM was set at I = 1000 and J = 800. All otherconditions were the same as the previous experiment. As forthe learning rate, we set ϵinit = 0.1 and ϵend = 0.01 for the
CD algorithm, γinit = 0.5 and γend = 1 for the present algorithm, and γinit = 0.3 and γend = 0.5 for “auxiliary function 1,” respectively. Since under this setup, computing the
log-likelihood required considerable time, we used only the

−1.6
0

100

200
300
loop count

400

500

(a) Evolution of log-likelihood

0.5

Contrastive Divergence
Auxiliary Function 1
Auxiliary Function 1 with gamma
Auxiliary Function 2
Auxiliary Function 2 with gamma

0.45
reconstruct error rate

1
,
1+J
1
H
H
= βij
=
β0j
.
1+I

V
V
βi0
= βij
=

0.4
0.35
0.3
0.25
0.2
0.15
0.1
0

100

200
300
loop count

400

500

(b) Evolution reconstruction error rate

Fig. 2. Evolution of the log-likelihood (a) and the reconstruction error rate (b) during training of a small scale RBM using
the CD algorithm (black solid line), the present algorithm and“auxiliary function 1” (red and blue) with and without the γscheduling (solid and dashed line), respectively.

0.6

Contrastive Divergence
Auxiliary Function 1
Auxiliary Function 1 with gamma
Auxiliary Function 2
Auxiliary Function 2 with gamma

reconstruct error rate

0.5
0.4
0.3
0.2
0.1
0

100

200
300
loop count

400

500

Fig. 3. Evolution of the reconstruction error rate during thetraining of a larger scale RBM using the CD algorithm (blacksolid line), the present algorithm and “auxiliary function 1”(red and blue) with and without the γ scheduling (solid anddashed line), respectively.
reconstruction error rate (51) as the measure for the comparison.
Fig. 3 shows the evolution of the reconstruction error rate(51) during the training using the three algorithms. The effectof the γ scheduling appeared to be more significant under theV
V
H
H
large I and J settings, since βi0
, βij
, β0j
and βij
becomemuch smaller in average than under the small I and J settings. Furthermore, the present algorithm converged fasterthan “auxiliary function 1.” This was simply because the objective function employed in the present algorithm reflects thereconstruction error rate better than the likelihood. These results encourage us to hope that the convergence of the present
algorithm and “auxiliary function 1” can further be accelerated by seeking for a better way of γ scheduling.
5. CONCLUSION
This paper proposed deriving a new training algorithm basedon an auxiliary function approach for RBMs using the reconstruction probability of observations as the optimization criterion. Through an experiment on parameter training of anRBM, we confirmed that the present algorithm outperformed
the CD algorithm in terms of the convergence speed and thereconstruction error when used as an autoencoder. Future
work includes using the present algorithm as pre-training of
DBN and investigating its behavior.
6. REFERENCES[1] G. E. Hinton, S. Osindero, and Y. W. Teh, “A fast learningalgorithm for deep belief nets,” Neural Computation, vol. 18,
no. 7, 2006.
[2] G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed,
N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath,and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition: The shared views of four researchgroups,” Signal Processing Magazine, IEEE, vol. 29, no. 6, pp.
82–97, 2012.

[3] P. Smolensky, “Information processing in dynamical systems:
Foundations of harmony theory,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D.E.
Rumelhart and J.L McClelland, Eds. MIT Press, 1986.
[4] G. E. Hinton, “Training products of experts by minimizing
contrastive divergence,” Neural Computation, vol. 14, no. 8,
2002.
[5] J. De Leeuw and W. J. Heiser, “Convergence of correction
matrix algorithms for multidimensional scaling,” in Geometricrepresentations of relational data, J. C. Lingoes, E. E. Roskam,and I. Borg, Eds. Ann Arbor, MI: Mathesis Press, 1977.
[6] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum
likelihood from incomplete data via the EM algorithm,” J. ofRoyal Statistical Society Series B, vol. 39, 1977.
[7] D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factorization,” in Adv. Neural Information and Processing
Systems (NIPS). 2001, pp. 556–562, MIT Press.
[8] M. Nakano, H. Kameoka, J. Le Roux, Y. Kitano, N. Ono,and S. Sagayama, “Convergence-guaranteed multiplicativealgorithms for non-negative matrix factorization with betadivergence,” in 2010 IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2010), 2010, pp.
283–288.
[9] H. Kameoka, Statistical Approach to Multipitch Analysis,
Ph.D. thesis, The University of Tokyo, 2007.
[10] H. Kameoka, N. Ono, and S. Sagayama, “Auxiliary function
approach to parameter estimation of constrained sinusoidal
model for monaural speech separation,” in 2008 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP2008), 2008, pp. 29–32.
[11] H. Kameoka, N. Ono, K. Kashino, and S. Sagayama, “Complex NMF: A new sparse representation for acoustic signals,”in Proceedings of the 2009 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP 2009),
2009, pp. 3437–3440.
[12] H. Kameoka, T. Nakatani, and T. Yoshioka, “Robust speech
dereverberation based on non-negativity and sparse nature ofspeech spectrograms,” in Proceedings of the 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2009), 2009, pp. 45–48.
[13] N. Ono, “Stable and fast update rules for independent vector
analysis based on auxiliary function technique,” in 2011 IEEE
Workshop on Applications of Signal Processing to Audio andAcoustics (WASPAA2011), 2011, pp. 261–264.
[14] N. Yasuraoka, H. Kameoka, T. Yoshioka, and H. G. Okuno, “Idivergence-based dereverberation method with auxiliary function approach,” in Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP 2011), 2011, pp. 369–372.
[15] H. Sawada, H. Kameoka, S. Araki, and N. Ueda, “Efficient algorithms for multichannel extensions of Itakura-Saito
nonnegative matrix factorization,” in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP2012), 2012, pp. 261–264.
[16] H. Kameoka and N. Takamune, “Training restricted Boltzmann machines with auxiliary function approach,” in 2014
IEEE International Workshop on Machine Learning for Signal
Processing (MLSP 2014), 2014, submitted.
[17] L. Deng, M. Seltzer, D. Yu, A. Acero, A. Mohamed, and G. E.
Hinton, “Binary coding of speech spectrograms using a deep
auto-encoder,” in Proceedings of the International Conferenceon Spoken Language Processing (Interspeech’2010 - ICSLP),
2010, pp. 1692–1695.