2014 22nd International Conference on Pattern Recognition

To be Bernoulli or to be Gaussian, for a Restricted
Boltzmann Machine
Takayoshi Yamashita∗ , Masayuki Tanaka† , Eiji Yoshida‡ , Yuji Yamauchi‡ and Hironobu Fujiyoshi‡∗ Chubu University† Tokyo Institute of Technology‡ Tome R&D

Email: yamashita@cs.chubu.ac.jp
Abstract—We introduce a method that automatically selects
appropriate RBM types according to the visible unit distribution.
The distribution of a visible unit strongly depends on a dataset.
For example, binary data can be considered as pseudo binary
distribution with high peaks at 0 and 1. For real-value data,
the distribution can be modeled by single Gaussian model or
Gaussian mixture model. Our proposed method selects appropriate RBM according to the distribution of each unit. We employ
the Gaussian mixture model to determine whether the visible
unit distribution is the pseudo binary or the Gaussian mixture.
According to this distribution, we can select a Bernoulli-Bernoulli
RBM(BBRBM) or a Gaussian-Bernoulli RBM(GBRBM). Furthermore, we employ normalization process to obtain a smoothed
Gaussian mixture distribution. This allowed us to reduce variations such as illumination changes in the input data. After
experimentation with MNIST, CBCL and our own dataset, ourproposed method obtained the best recognition performance andfurther shortened the convergence time of the learning process.

I.

I NTRODUCTION

other layers.[2]. On the other hand, Cho used the GBRBM
for all stacked layers[9][11]. In those approaches, one needs
to heuristically pre-determine which RBM is appropriate foreach layer. We propose a method to automatically select theappropriate RBM type according to the distribution of visible
unit for each layer. In the stacked RBM, the conditional
probability or the inference of the hidden unit is used as
the input of the next layer of the RBM, even though the
distribution of hidden unit is assumed as binary in both the
BBRBM and the GBRBM. In this sense, we need to selectRBM for each layer according to the distribution of actual
input data. For the ﬁrst layer, we can select RBM according to
the distribution of the given data. For example, the GBRBM
should be selected for the real-valued data such as image.
However, it is unclear which type of the RBM is suitable
for the second layer. Therefore, we need to evaluate the
distribution of the inferred hidden state of the ﬁrst layer. When
the GBRBM is selected, we employ the normalization process
where it is possible to suppress variations in the distributionof the input data to obtain the Gaussian distribution. We refer
this GBRBM with normalization process to nGBRBM.

A restricted Boltzmann machine (RBM)[1] is becoming
more and more popular stochastic graphical model in recent
years. iFollowing the RBM proposed, various RBM and thestacked RBM methods have proposed as new machine learning
methods [3][4][5][7]. These methods are applied to image
recognition, speech recognition, etc., and show state-of-theart performance in benchmark tests [6][8][10]. A conventional
RBM assume that the visible unit is of binary data, it becomesa critical limitation to the various applications. The method
such as softmax which approximate the binary data from realvalue, is employed to address this issue. Alternatively, Hinton
also proposed a method to apply real-value data through aformulation of the RBM by assuming the input data to beof Gaussian distribution[2]. These existing approaches used tohandle real-value data are quite effective when the distributionof the input data is known. However, it is difﬁcult to determine
the data distribution for general purpose, i.e. whether the input
is from a binary image or a gray scale image. In this paper,
a Bernoulli-Bernoulli RBM (BBRBM) refers to the RBM
assuming that the distribution of both visible and hidden units
is binary. A Gaussian-Bernoulli RBM (GBRBM) refers to the
RBM assuming that the distribution of the visible unit is theGaussian distribution and that the distribution of the hidden
unit is binary.

A. Bernoulli-Bernoulli RBM

The Stacked RBM can achieve higher recognition performance compared with the single RBM. In order to apply togray scale image, Hinton proposed the stacked RBM whichcontain the GBRBM in the ﬁrst layer and the BBRBM in

The restricted Boltzmann machine (RBM) consists of mvisible units v = (v1 , ..., vm ) and n hidden units h =(h1 , ..., hn ), with fully connecting between them. But there
are no visible to visible and hidden to hidden connections. The

1051-4651/14 $31.00 © 2014 IEEE
DOI 10.1109/ICPR.2014.270

The key contributions of our proposed method are, 1) the
proposed automatic RBM selection improves the recognition
performance. In particular, it is no longer necessary to heuristically pre-determine the RBM type of each layer; 2) the normalization for the GBRBM improves the performance of thestacked RBM which includes the GBRBM. This normalization
is not only successful in the ﬁrst layer, but also obtains better
results in later layers.
The rest of paper, we review the BBRBM and the GBRBM
in Section 2. Then, we propose the automatic RBM typeselection and the normalization for the GBRBM in Section
3. The comparison results for predeﬁned multiple layers of
the RBM and the GBRBM in various dataset, such as binary
data and real-value data, are shown in Section 4. Finally, we
will discuss about the effectiveness of our proposed methodwith analysis of the distribution of each unit in Section 5, andpresent our conclusions in Section 6.
II.

1520

R ESTRICTED B OLTZMANN M ACHINE

visible units in the ﬁrst layer correspond to the measurements.
In computer vision tasks, one visible unit often corresponds
to one pixel value. The hidden units h are independent conditionally to the measurements. Because of this, each hidden unit
becomes an independent speciﬁc feature. The RBM has been
mainly developed to model binary variables (v, h) which takethe binary values of (v, h) ∈ {0, 1}m+n . A joint probabilityof (v, h) can be expressed with the BBRBM as:1(1)p(v, h) = e−E(v,h) ,Z
where Z is the normalizing constant and the energy function
E,
m 
m
n
n



E(v, h) = −wij vi hj −bi vi −
cj hj , (2)i=1 j=1i=1In terms of probability, the hidden units are independent
from the visible units and vice versa, as shown in Eqs. (3) and(4). When binary data are given in visible units, the conditional
probability is estimated from the neural network propagation
rule by Eqs. (5) and (6).
p(vi |h),(3)p(v|h) =i=1


p(hj |v),

(4)

i=1

p(vi = 1|h) = f (bi +

hj wij ),







(5)

Fig. 1.
Distribution of visible unit. (a) is MNIST, (b) is the intensity
distribution of certain unit for MNIST images, (c) is CBCL, and (d) is theintensity distribution of certain unit for CBCL images.

where N (·|μ, σ 2 ) denotes the Gaussian probability density
function with mean μ and standard deviation σ.
In the parameter updating process, a contrastive divergence(CD) learning is highly successful and is becoming the standard learning method to train the RBM parameters[1]. The CD
learning only samples for k steps to approximate the second
term in the log-likelihood gradient from a sample from theRBM distribution. In the CD learning, k is usually set to 1.
The update rules for the parameters are the following,j=1

p(hj = 1|v) = f (cj +



j=1

for all i ∈ {1, ..., n} and j ∈ {1, ..., m}, wij is a real valued
weight associated with the edge between units vi and hj and bi
and cj are real valued bias terms associated with the i-th visible
and the j-th hidden variable, respectively. From Eqs. (1) and(2), E(v, h) with a low energy are given a high probability.

p(h|v) =vi wij ),1
1
v i hj > d − < 2 v i hj > m ,
σi2
σi
1
1
∇bi =< 2 vi >d − < 2 vi >m ,σiσi∇cj =< hj >d − < hj >m ,∇wij =<(6)i=1

where f (·) is the sigmoid activation function.
The model with the energy function has been developed tomodel the random binary variables. Therefore, this model is
not suitable to model a continuous value data. To address thisissue, the GBRBM has proposed.

(10)
(11)
(12)

where shorthand notations < · >d and < · >m denote theexpectation computed over the data and model distributions
accordingly.

B. Gaussian-Bernoulli RBM
The Gaussian-Bernoulli RBM (GBRBM) has visible units
with real-value vm and binary hidden units hn . Based on thesame idea as the BBRBM, the energy function of the GBRBM
is deﬁned as
m 
m
n
n

vi  (vi − bi )2 E(v, h) = −wij hj −−c j hj ,σi i=12σi2i=1 j=1
j=1(7)
where bi and cj are biases corresponding respectively to visible
and hidden units, wij are the connecting weights between thevisible and hidden units and σi is the standard deviation associated with Gaussian visible units vi . Conditional probabilities
for visible and hidden units are
hj wij , σi2 ),(8)p(vi = v|h) = N (v|bi +jp(hj = 1|v) = f (cj +


i

wij

vi
),
σi2

(9)

III.

P ROPOSED METHODAs shown in Eqs.(5) and (6), the random binary variables
are assumed for the inputs of the BBRBM. This assumption
becomes a critical issue in considering real applications suchas image processing. Hand writing digit data as shown in Fig.1(a) can be considered as pseudo binary variables whose density
functions condense at two values as shown in Fig.1 (b). In
this case, the BBRBM can obtain an efﬁcient classiﬁcation
feature in the hidden units because most of the inputs are ofbinary value. On the other hand, if the input data is real-value
like grayscale images in Fig.1(c), the density function of each
input is distributed in wide range as shown in Fig.1(d). The
GBRBM may obtain efﬁcient features for these distribution
input data. In this sense, the performance of the RBM can be
improved by selecting the GBRBM or the BBRBM dependingon the distribution of the inputs. First, we propose the RBM
selection method that determines whether to apply either the1521

BBRBM or the GBRBM according to the distribution of thevisible unit.
Training of the GBRBM is a challenging task. One of
the reasons for that the data distribution of each visible unit
has many variations. The distribution of each visible unit isassumed to be single Gaussian distribution, but it becomes an
obscure and vague Gaussian distribution. We therefore propose
a normalized Gaussian-Bernoulli RBM (nGBRBM) where
each visible unit is normalized in order to obtain a distinct
Gaussian distribution from the Gaussian mixture distribution.





Fig. 2. Normalized image and distribution of visible unit after input datanormalization. (a) is the original image in CBCL, (b) is normalized images, (c)is the distribution of visible unit using normalized images (original distribution
is Fig.1(d)).

A. Proposed RBM type selection
We assume that the distribution of each input unit belongs
to a pseudo binary distribution, single Gaussian distribution, orGaussian mixture distribution. The pseudo binary distribution
has two sharp peaks at two values like 0 and 1. The pseudo
binary distribution can be considered the special case of theGaussian mixture distribution. We model distribution of each
inputs by the Gaussian mixture model (GMM). Then, thedistribution is classiﬁed by parameters of the GMM. The
distribution of visible unit vi is modeled byp(vi ) =Kπk N (vi |μk , Σk ).

(13)

k=1

Note that K is the number of Gaussian components, πk is the
mixture weight of k-th Gaussian, μk and Σk are respectively
the mean and variance. We estimated the parameters π, μ andΣ via a maximum likelihood estimation, which matches the
distribution of the training data. In the maximum likelihood
estimation, the parameters can be obtained iteratively by using
an EM algorithm. On the EM iteration, we compute a posteriori
probability for component k as E-step, given by
πk N (vi |μk , Σk )pr(k|vi , μk , Σk ) = K.
l=1 πl N (vi |μk , Σl )In M-step, the parameters are updated as follows,
T
(t)
(t)
t=1 pr(k|vi , μk , Σk )vi,
μk = 
(t)T
t=1 pr(k|vi , μk , Σk )
T
(t)
(t)2
t=1 pr(k|vi , μk , Σk )viΣk = 
− μ2k ,(t)T
pr(k|v
,
μ
,
Σ
)
k
k
i
t=1
1
pr(k|vi , μk , Σk ).
T t=1

(14)

(15)

(16)

T

πk =

(17)

We estimate the parameters for each k. Then, we ﬁndbest K with minimum ﬁtting error between the distributionof visual unit and the distribution with estimated parameters.
If K is equal or greater than 3, the distribution of visible
unit is identiﬁed as the Gaussian mixture, and if K equals
1, the distribution is identiﬁed as the single Gaussian. On theother hand, if K equals 2, it must be identiﬁed as either thepseudo binary distribution or the Gaussian mixture. The pseudo
distribution has two Gaussians with small standard variations
around 0 and 1. This mean that two μk are separated bysigniﬁcant distance T hm . Moreover, both standard deviations
Σk are smaller than the threshold T hd . If these two conditions
are satisﬁed, we are able to identify the distribution as thepseudo binary. After identiﬁcation of the distribution for all
visible unit using the above method, we determine the RBM
types by taking a vote of the distribution of them. For example,
if pseudo binary distribution is majority, the BBRBM is
selected. On the other hand, if major distribution is single or
the Gaussian mixture, then the nGBRBM is selected.
B. Normalization of Visible Unit
We normalize each training data to obtain a distinct Gaussian mixture distribution for visible units. Each pixel of the
training data is normalized with the zero-mean and the unit
variance of each training data. Figure 2 shows the normalized
images and distribution of visible unit using them. Advantages
to this normalization are that 1) the training data becomes uniform with no condition variations, such as illumination change,2) the distribution of each visible unit becomes smoother.
C. Stacked selected RBMs
The stacked RBM is sometimes called a deep Boltzmann
machine (DBM) or a deep belief network (DBN). It has
simple layer-wise structure where the RBM is stacked on top
repeatedly multiple times. The hidden units of the learned
RBM can be used as visible unit for learning a following RBM.
It can be characterized as follows: 1) high representation canbe built from a large amount of unlabeled input data, and justa few minimal labeled data can be used to slightly ﬁne-tune
the model. 2) better propagation of ambiguous inputs can beobtained with approximate inference procedure of ﬁne-tuning
feedback. In our proposed method, the RBM type is identiﬁedaccording to the distribution of the visible unit in the learningof the RBM in each layer. Accordingly, the appropriate RBM
type for each layer can be decided without the pre-deﬁnitions
required in the conventional methods.
IV.

E XPERIMENTS

A. Implementation
We evaluate the reconstruction error of the a single layer
RBM and the recognition performance with stacked layers invarious datasets. For the comparison, we use the standard handwritten dataset MNIST[13] and face image dataset CBCL[12].
MNIST and CBCL are famous databases but they include
strictly controlled data for their background. In order to evaluate under various real-scene situations, we use our original
hand shape dataset including 6 hand shape classes in clutter
background. We deﬁne the following learning parameters of
the BBRBM, the GBRBM and our proposed method. The size

1522

TABLE I.

TABLE II.

S TRUCTURE OF PROPOSED METHOD IN EACH DATASET.
Layer
1
2
3
4

MNIST
BBRBM
nGBRBMBBRBM
BBRBM

RBM typeCBCLnGBRBM
BBRBM
BBRBM
BBRBM

Hand shapenGBRBMnGBRBM
BBRBM
BBRBM

of minibach is 100, while the initial learning rate and its upperbound are set to 0.001 for pre-training. The weight-updating
ratio is set to 0.1. As parameters of the RBM selection, T hm
is 0.99 and T hd is 0.01.
We learn for all methods with 4-layer structure for eachdataset. We use the 60000 handwritten images for learningand 10000 for testing in MNIST. The image size is 28 × 28
and the number of visible unit is 784. All methods are of 4layer structure with respectively 400, 255, 100 and 10 units ineach layer. The 10 units in the ﬁnal layer are the number oflabels that are connected with the previous layer. In CBCL, welearn for all methods with 2000 face images, and we use 1000
images for testing with a 19 ×19 image size. The number ofvisible units is 361, and later layers respectively has 400, 255,
100 and 2 units. In this network, the output layer discriminates
between faces and non-faces for given image. As a third dataset
for comparison, we use a hand shape dataset with a 40 × 40sized images. The number for learning data is 28000 and fortest data is 10000. The number of visible units is 1600, while
hidden units are 1024, 400, 100 and 7.
We also employ ﬁne tuning to ﬁnd optimized parameters
for whole of network with dropout. Fine tuning is basedon gradient decent on a supervised training criterion[3][4].
Dropout ratio is set to 0.5, iteration of ﬁne tuning is 1000.
All methods apply same parameters for all dataset.
B. RBM Selection
The selected RBM type in each layer of the proposed
method is shown in Table I. For MNIST, since the distributionof visible unit is close to the pseudo binary distribution,
the BBRBM is selected for the ﬁrst layer. The nGBRBM is
selected for the second layer because the distribution is near theGaussian mixture distribution. For the later layers, the BBRBM
is selected since most of units have the pseudo binary distribution. As shown in Fig.1, CBCL data has Gaussian distribution.
This Gaussian distribution data is given as visible unit andtherefore, the nGBRBM is selected for the ﬁrst layer. For thesecond and following layers, the BBRBM is selected because
many units have pseudo binary distribution. Like CBCL, thenGBRBM is selected in the ﬁrst layer for hand shape database.
The nGBRBM is also selected since the Gaussian distribution
still remained in the second layer. For the later layers, the
BBRBM is selected since most of units have a pseudo binary
distribution. As mentioned above, the appropriate RBM type
depends on the dataset and the layer. In MNIST, most ofregion is black, and hand written pixels are white like thebinary image. It is possible to learn with high discriminationeven when we apply the BBRBM in ﬁrst layer. However,
CBCL and hand shape dataset are grayscale images and thatdata represents Gaussian distribution. When the nGBRBM is
selected ﬁrst, the best suitable RBM type depending on the
difﬁculty of the datasets will be selected for the following

R ECOGNITION PERFORMANCE IN EACH DATASET.
DatasetMNIST
CBCL
Hand Shape

BBRBM
96.02
93.59
86.21

GBRBM
96.39
96.39
87.77

Ours
97.10
97.60
89.99

layers. In addition, the normalization process of the nGBRBM
is efﬁcient in order to obtain the pseudo binary distribution
in earlier layers. This normalization reduce condition variationsuch as illumination change and distribution of unit becomessmoother. Therefore, the nGBRBM can learn more easily evenfrom datasets including difﬁcult data. We describe more details
the distribution in each layer in the following section.
C. Reconstruction Result
We reconstruct of images to compare each RBM type. The
reconstructed images in MNIST are shown in Fig.3. Whilemost regions are correctly reconstructed with all methods
in MNIST, the details in handwritten images can not be
reproduced with the BBRBM or the GBRBM. For example, for
the ’7’ in the third column, the BBRBM can not reconstruct
the upper region with enough thickness. The ’3’ in the 5thcolumn also does not have enough thickness. The nGBRBM
can reconstruct much more correctly not only shapes but also
thickness. In CBCL, while each pixel is real-value data, theirimages are uniform since the face size and background are notcontained in the image. Therefore, even if it is grayscale image,
the reconstruction is not such a difﬁcult task for all methods, as
shown in Fig.4. All methods can reconstruct correctly various
faces. However, as shown in Fig.5, the nGBRBM had significantly better reconstruction performance compared with the
BBRBM and the GBRBM. The BBRBM and the GBRBM can
not correctly reconstruct both the hand shape and backgroundregion. While most of reconstructed images with them areblurry, the reconstruction with the nGBRBM is clearer thanthat of others.. Moreover, the nGBRBM reconstructs the handregion correctly even with a cluttered background. It had thesurprising ability to reconstruct not only hand shape but also
the background.
D. Recognition Performance
The comparison results of the recognition performance oneach dataset are shown in Table II. Our proposed method can
improve the recognition performance in all dataset. In MNIST,
the recognition rate of our proposed method reaches 97.10%,
an increase of more than 1% compared to the BBRBM.
Even for binary images in MNIST, our method performs
the best accuracy. Our proposed method also has the highest
recognition rate among comparison methods in CBCL and thehand shape dataset. With these results, we can say that our
proposed method offers signiﬁcant improvement for real-value
data. In other words, our proposed method is able to obtaineffective initial parameters throughout the whole network.
Furthermore, as shown in Fig.6, convergence comes muchearlier in pre-training with our proposed method than with theother methods.
V.

D ISCUSSIONIn Fig.7, we show the distributions of typical units in eachdataset. The visible units of the ﬁrst layer in MNIST receives

1523



    

 
 
 











 


Fig. 3. Reconstructed images in MNIST. (a)original image, (b)BBRBM result,(c)GBRBM result and (d)our proposed method.
Fig. 6.











 





Reconstruction error of each iteration in CBCL.



















Fig. 4. Reconstructed images in CBCL. (a)original image, (b)BBRBM result,(c)GBRBM result and (d)our proposed method.






Fig. 7. Distribution of hidden units. (a) is the 80th unit in the ﬁrst layer,and (b) is the 205th unit in the second layer, in MNIST. (c) is the 15th unit
in the ﬁrst layer, and (d) is the 57th unit in the second layer, in CBCL. (e) is
the 71th unit in the ﬁrst layer, and (f) is the 95th unit in the second layer, inthe hand shape data set.

Fig. 5.
Reconstructed images in hand shape data. (a)original image,(b)BBRBM result, (c)GBRBM result and (d)our proposed method.

the data that have the distribution as shown in Fig.1(b). Those
distributions have two peaks at 0 and 1 because most of the
pixels in MNIST consist of black and white. These distribution
can be seen as the pseudo binary, and our method selects the
BBRBM in the ﬁrst layer. The output distribution of the typicalunits in the ﬁrst layer has two peaks, as shown with the yellow
line in Fig.7(a). Although the distribution has high peak at

0 and 1, it is identiﬁed the Gaussian mixture instead of thepseudo binary. Because the most of distributions have large
variance than threshold T Hm . Therefore, the nGBRBM is
selected as the 2nd layer. As a result, the proposed method
outputs the distribution with high peak at 0 and 1, while the
BBRBM and the GBRBM output like Gaussian distribution asshown in Fig.7(b). The BBRBM is selected in later layers,
because the most of distributions are identiﬁed the pseudo
binary like yellow line in Fig.7(b).

1524process for the GBRBM to suppress variations of the input
data is incorporated. With the proposed method, one canselect the appropriate RBM types automatically. As a results,
the proposed method improve the performance compared toconventional methods.





R EFERENCES



[1]
[2]

Fig. 8. Weight Visualization. (a) the distribution has a peak in 1, it is common
feature to discriminate several classes, (b) the distribution has two peaks near
0 and 1, it is speciﬁc feature to discriminate speciﬁc class, in this case it seemis speciﬁed to ’1’ in MNIST, (c) the distribution is uniform, it does not focus
on speciﬁc classes.

[3]
[4]

In CBCL, our proposed method selects the nGBRBM byreason that many distribution of visible units are Gaussian asshown in Fig.1(c). The distribution has high peaks at 0 and 1
and it is identiﬁed the pseudo binary distribution. Therefore,
the BBRBM is selected as the 2nd layer. While the BBRBM
has two peaks near 0 or 1, the distance between them is not
far enough than threshold T Hd like red line in Fig.1(c). In thelater layer, our proposed method selects the BBRBM because
the distribution is the pseudo binary distribution like yellow
line in Fig.1(d).
The hand shape dataset has wide variation due to poseand illumination changes. Moreover, since the background iscomplex, the distribution of visible unit is a Gaussian mixture.
The proposed method selects the nGBRBM in the ﬁrst layer
as same as CBCL. As shown in Fig.7(e)(f), the distributions
have two peaks. However, the proposed method selected thenGBRBM as the second and third layers by reason that it has
not enough distance between them and variance.
We show the visualization of typical units of MNIST in
Fig.8. We ﬁnd the characteristics of the unit through thisobservation of the distribution and Fig.8 as follows. 1)if theunit has a high peak near 1, it seems common features for the
training data. As shown in Fig.8, white region that has highweights, commons to several class such as ’2’, ’6’, ’8’, etc.
It means that some weights connected with this unit are highfor several classes and other weights are low to discard restof classes. 2)if the unit has two high peak near 0 and 1, it
has a high discrimination for the speciﬁc class like Fig.8(b).
In this case it attention to speciﬁc class like ’1’. 3)if the unit
has uniform distribution, it has not enough discrimination for
training data. It means that the weights are not high to speciﬁcclass like Fig.8(c).

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]
[13]

We show the distribution of the units for the BBRBM, the
GBRBM and the proposed methods in each layer. We illustrated the effect of RBM type selection through the distributionof the visible units. We could obtain effective parameters withhigh discrimination with our proposed RBM type selection.
VI.

C ONCLUSION

In this paper, we have proposed the method to selectthe RBM types according to the distribution of the visible
unit. By doing so, it is no longer necessary to pre-determine
the type of RBM in advance. Furthermore, the normalization

1525

G. E. Hinton, “Training products of experts by minimizing contrastive
divergence”, Neural Computation, Vol.14, No.8, pp.1771–1800, 2002.
G. E. Hinton and R. Salakhutdinov, “Reducing the Dimensionality ofData with Neural Networks”, Science, Vol.313, No.5786, pp.504–507,
2006.
G. E. Hinton and S. Osindero, “A fast learning algorithm for deep beliefnets”, Neural Computation,Vol.18, No.7, pp.1527–1554, 2006.
Y. Bengio, P. Lamblin, V. Popovici, and H. Larochelle, “Greedy layerwise training of deep networks”, In Advances in Neural Information
Processing Systems 19 (NIPS’06), pp.153–160, 2007.
R. Salakhutdinov and G. E. Hinton, “Deep Boltzmann machines”, In
Proceedings of the International Conference on Artiﬁcial Intelligenceand Statistics, pp.448–455, 2009.
A. Krizhevsky, “Learning multiple layers of features from tiny images”,
Technical report, Computer Science Department, University of Toronto,
2009.
V. Nair and G. E. Hinton, “Rectiﬁed linear units improve restricted
Boltzmann machines”, In Proceedings of the 27th International Conference on Machine Learning (ICML10), pp. 807-814, 2010.
B. Lake, R. Salakhudinov, J. Gross, and J. Tenenbaum, V. Ramesh, P.
Meer, “One shot learning of simple visual concepts”, Proceedings ofthe 33rd Annual Conference of the Cognitive Science Society, 2011.
K. Cho, A. Ilin, and T. Raiko, “Improved Learning of GaussianBernoulli Restricted Boltzmann Machines”, Artiﬁcial Neural Networks
and Machine Learning(ICANN 2011), Springer Berlin Heidelberg,
pp.10–17, 2011.
A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classiﬁcation
with deep convolutional neural networks”, In Advances in Neural
Information Processing Systems 25 (NIPS’12). pp.1106-1114, 2012.
K. Cho, T. Raiko, and A. Ilin, “Gaussian-Bernoulli deep Boltzmann
machine”, In NIPS 2011 Workshop on Deep Learning and Unsupervised
Feature Learning, 2011.
CBCL Face Database MIT Center For Biological and Computation
Learning, http://www.ai.mit.edu/projects/cbcl
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. ”Gradient-basedlearning applied to document recognition.” Proceedings of the IEEE,
Vol.86, No.11, pp.2278–2324, 1998.