Generative learning for deep networks

Boris Flach
flachbor@cmp.felk.cvut.cz

Alexander Shekhovtsov
shekhovtsov@gmail.com

Ondrej Fikar
fikarond@fel.cvut.cz

arXiv:1709.08524v1 [cs.LG] 25 Sep 2017

Czech Technical University in Prague

Abstract
Learning, taking into account full distribution of the data, referred to as generative,
is not feasible with deep neural networks (DNNs) because they model only the
conditional distribution of the outputs given the inputs. Current solutions are either
based on joint probability models facing difficult estimation problems or learn
two separate networks, mapping inputs to outputs (recognition) and vice-versa
(generation). We propose an intermediate approach. First, we show that forward
computation in DNNs with logistic sigmoid activations corresponds to a simplified
approximate Bayesian inference in a directed probabilistic multi-layer model. This
connection allows to interpret DNN as a probabilistic model of the output and all
hidden units given the input. Second, we propose that in order for the recognition
and generation networks to be more consistent with the joint model of the data,
weights of the recognition and generator network should be related by transposition.
We demonstrate in a tentative experiment that such a coupled pair can be learned
generatively, modelling the full distribution of the data, and has enough capacity to
perform well in both recognition and generation.

1

Introduction

Neural networks with logistic sigmoid activations may look very similar to Bayesian networks of the
same structure with logistic conditional distributions, aka sigmoid belief networks [11]. However,
hidden units in NNs are deterministic and take on real values while hidden units in Bayesian networks
are binary random variables with an associated distribution. Given enough capacity and training
data, both models can estimate the posterior distribution of their output arbitrary well. Besides
somewhat different modelling properties, there is a principled difference that with stochastic models
it is possible to pose, at least theoretically, a number of inference and learning problems with missing
data by marginalizing over latent and unobserved variables. Unfortunately, even forward inference in
Bayesian networks requires methods such as sampling or optimization of a variational approximation.
Likewise, in more tightly coupled graphical models such as deep Boltzmann machine (DBM) [16] or
deep belief networks (DBN) [5, 9] practically all computations needed e.g. for computing marginal
and posterior probabilities are not tractable in the sense that approximations typically involve sampling
or optimization.
In this paper we propose stacked (deep) conditional independent model (DCIM). There are two views
how the model can be defined. One, as a Bayesian network with logistic conditional distributions.
The other, just assuming that conditional probabilities of a general Bayesian network factor over the
parent nodes up to a normalising factor. With binary units this necessary implies that conditional
probabilities are logistic. It is noteworthy that we find the same form of conditional probabilities in
most of the neural probabilistic models: restricted Boltzmann machine, DBM [16], DBN [5], etc.
When the units are arranged in layers, as in a typical DNN, the layered Bayesian network can be
viewed as a Markov chain with a state space of all variables in a layer. In this view, all necessary
assumptions can be summarized in one property, termed strong conditional independence that the
forward conditional transition probabilities between the layers factorize over the dimensions of

both the input and output state spaces of the transition up to a normalising factor. Making only
this assumption, we show that a simple approximate Bayesian inference in DCIM recovers main
constructive elements of DNNs: sigmoid and softmax activations with its real-valued variables
corresponding to expectations of binary random variables. With this interpretation we can view DNN
as a way of performing approximate inference very efficiently. To our knowledge, this relationship
has not been established before. Under this approximation conditional likelihood learning of a DCIM
is equivalent to that of DNN and can be performed by back-propagation.
Our second objective is to propose an alternative, generative learning approach for DNNs. In a
number of recent works [6, 13, 15, 10] and in the prior work [4, 5] a pair of deterministic recognition
(encoder) and a stochastic generator (decoder) networks is trained. Let us denote x0 the input of
the recognition network, to be specific, an image, and xd (random variables at layer d) the output
of the recognition network, the latent state. Although the two networks often are taken to have a
symmetric structure [4, 6], their parameters are decoupled. The stochastic generator network can
typically only generate samples, but cannot directly evaluate its posterior distribution or the gradient
thereof, requiring variational or sampling-based approximations. The methods proposed in [6, 13]
assume that samples generated from the same latent state xd must fall close together in the image
space. This prohibits using categorical latent spaces such as digit class in MNIST because digits of
the same class naturally may look differently. Instead, a model’s own continuous latent space is used
such that fixing a point in it defines both the class and the shape of the digit. Works [6, 13] are thus
restricted to unsupervised learning.
Given full training data pairs of (x0 , xd ), the recognition network could learn a distribution p(xd |
x0 ) and the generator network could in principle learn a distribution q(x0 | xd ). With our link
between DNN and DCIM, we ask the question when the two DCIMs, modelling the two conditional
distributions p(x0 | xd ) and q(xd | x0 ), are consistent, i.e., correspond to some implicitly modelled
joint distribution p∗ (x0,...,d ) of the data and all hidden layers. It is the case when p(x1,...d |
x0 )/q(x0,...d−1 | xd ) = A(x0 )B(xd ) for some functions A, B. While this cannot be strictly
satisfied with our strong conditional independence assumptions, we observe that most of the terms
in the ratio cancel when we set the weights of recognition network p to be transposed of those of
network q. Both models therefore can be efficiently represented by one and the same DNN, share its
parameters and can be learned simultaneously by using an estimator similar to pseudo-likelihood.
We further use the link between DNN and DCIM to approximately compute the posterior distribution
in the generator network q(xd | xk ) given a sample of an inner layer xk . The approximation is
reasonable when this posterior is expected to have a single mode, such as when reconstructing an
image from lower level features. We thus can fully or partially avoid sampling in the generator model.
We demonstrate in a tentative experiment that such a coupled pair can be learned simultaneously,
modelling the full distribution of the data, and has enough capacity to perform well in both recognition
and generation.
Outline In Section 2.1 we consider strongly conditional independent model with two layers and
derive the NN from it. This model will serve as a building block of DCIM formally introduced
in Section 2.2. In Section 2.3 we consider coupled pairs of DCIMs and propose a novel approach
for generative learning. In Section 2.4 we discuss more connections with related work. In Section 3
we propose our proof-of-concept experiments of generative learning using only DNN inference or
sampling only a single layer.

2
2.1

Deep conditional independent models
An approximation for conditional independent models

Let x = (x1 , . . . , xn ) and y = (y1 , . . . , ym ) be two collections of discrete random variables with
a joint distribution p(x, y) = p(y | x) p(x). The conditional
distribution p(y | x) is strongly
Q
1
conditional independent if it factors as p(y | x) = Z(x)
g
(x
ij
i , yj ). Without loss of generality,
i,j
it can be written as
p(y | x) =

m
m
m
X
Y
Y
X
1
1
exp
uij (xi , yj ) =
p(yj | x) =
exp
uij (xi , yj ).
Z(x)
Z (x)
i,j
j=1
j=1 j
i=1

2

(1)

The functions uij are arbitrary and Zj (x) denote the corresponding normalization constants. While
sampling from p(y | x) is easy, computing p(y) by marginalizing over x is not tractable even if p(x)
factorizes over the components of x.
We consider the following approximation for the marginal distribution p(y):
XX
1
p(y) ≈ exp
p(xi )uij (xi , yj ).
Z
ij x

(2)

i

One of the possible ways to derive it, is to consider Taylor expansions
for moments of functions of
P
random variables. By using a linear embedding Φ such that i,j uij (xi , yj ) = hΦ(x, y), ui, where
u is a vector with components ui,j (k, l), we can write p(y | x) in the form
exp hΦ(x, y), ui
.
0
y 0 exp hΦ(x, y ), ui

p(y | x) = P

(3)

The marginal distribution p(y) is obtained from
exp hΦ(x, y), ui
.
0
y 0 exp hΦ(x, y ), ui

Ex∼p(x) p(y | x) = Ex∼p(x) P

(4)

Taking the first order Taylor expansion w.r.t. the random variable Φ(x, y) for fixed y around
Φ(y) = Ex∼p(x) Φ(x, y)

(5)

and computing its expectation w.r.t. x gives (2) as the constant term, while the first order term
vanishes.
Example 1. Assume that random variables in the two collections x = (x1 , . . . , xn ) and y =
(y1 , . . . , ym ) are {0, 1}-valued. Any function uij (xi , yj ) of binary variables can be written as
uij (xi , yj ) = yj Wji xi + bj yj + ci xi + d. Terms ci xi + d cancel in
Qthe normalization of p(y | x)
and thus can be omitted. The approximation (2) reads then p̂(y) = j p̂(yj ) with
j
e(hw ,x̄i+bj )yj
p̂(yj ) = hwj ,x̄i+b
,
j + 1
e

(6)

where wj denotes the j-th row of the matrix W and x̄ = Ex∼p(x) x. This in turn leads to the
following approximation for the expectation of y:
ȳj = p̂(yj =1) =

1
1+

j
e−hw ,xi−bj

= S(hwj , x̄i + bj ),

(7)

where S is the logistic (sigmoid) function.
If ±1 encoding is used instead to represent states of x and y, then the corresponding expressions read


exp yj wj , x̄ + bj

 ,
p̂(yj ) =
(8)
2 cosh hwj , x̄i + bj
and

ȳj = tanh[ wj , x̄ + bj ].

(9)

Remark 1. Let p(x, y) be a joint model such that p(y | x) is strongly conditional independent as
above. Then p(x | y) is not in general conditional independent. If both conditional distributions are
strongly conditional independent, then the joint model is a restricted Boltzmann machine or, what is
the same, an MRF on a bipartite graph.
2.2

Deep conditional independent models

Let X = x0 , x1 , . . . , xd be a sequence of collections of binary valued random variables

xk = xki = ±1 i = 1, . . . , nk .

(10)

The next observation highlights the difficulties that we run into if considering the deep Boltzmann
machine model.
3

Observation 1. Let us assume that the joint distribution p(X ) is a deep Boltzmann machine, i.e. an
MRF on a layered, d-partite graph. Then neither the forward conditionals p(xk | xk−1 ) nor the
backward conditionals p(xk−1 | xk ) are (conditionally) independent. This can be seen as follows.
The joint distribution of such a model can be written w.l.o.g. as
p(X ) =

d
d D
E X
i
hX
1
exp
bk , xk +
xk , W k xk−1
Z

(11)

k=0

k=0

and it follows that
p(xk | xk−1 ) =

E
i
hD
F (xk )
exp bk , xk + xk , W k xk−1
k−1
Z(x
)

(12)

where F (xk ) results from marginalisation over xk+1 , . . . , xd and can be arbitrarily complex.
We will therefore consider a similar, but different class, for which the forward conditionals are
(conditionally) independent.
Definition 1. A joint distribution p(X ) for a sequence of collections of binary valued random
variables (10) is a deep conditional independent model (DCIM) if it has the form
p(x0 , . . . , xn ) = p(x0 )

d
Y

p(xk | xk−1 )

(13)

k=1

and its forward conditionals p(xk | xk−1 ) are strongly conditional independent, i.e. have the form
(1) for all k = 1, 2, . . . , d.
A DCIM can be thus seen as an inhomogeneous Markov chain model with high dimensional state
spaces xk ∈ X k , however with an yet unspecified distribution for x0 . Given such a model and a
realisation of x0 , the posterior conditional distribution p(xd | x0 ) can be computed by applying the
approximation (2) recursively (see Example. 1). This leads to the recursion

x̄ki = tanh wki , x̄k−1 + bki , k = 1, 2 . . . , d − 1,
(14)
where x̄k denotes the expectation of xk w.r.t. p(xk | x0 ). Finally, we obtain
p(xdi | x0 ) =

exp(xdi adi )
with adi = wdi , x̄d−1 + bdi .
2 cosh(adi )

(15)

It is obvious that the (approximate) computation of p(xd | x0 ) for a DCIM is exactly the forward
computation for a corresponding DNN. Hence, discriminative learning of a DCIM by maximizing
the conditional likelihood p(xd | x0 ) of training samples (xd , x0 ) means to learn the corresponding
DNN with the same loss.
With 0-1 state variables the model is equivalent to a Bayesian network (a directed graphical model),
in which the set of parents of a variable xkj is the preceding layer xk−1 (or its subset, depending on
the structure of the weights) and the conditional distributions are of the form (1), i.e., logistic. Such
networks were proposed by Neal [11] under the name sigmoid belief networks as a simpler alternative
to Boltzmann machines. When we derive the model from the strong conditional independence
assumption, it establishes a link to deep Boltzmann machines [16], which are graphical models
factorizing over a d-partite graph.
2.3

Generative learning of deep networks

To learn a DCIM generatively from a given i.i.d. training data T , it is necessary to specify the
joint model p(x0 , . . . , xd ) and to choose an appropriate estimator. In theory, the DCIM model can
be completed to a joint model by specifying a prior distribution p(x0 ) over images and then the
maximum likelihood estimator can be applied. It is however not realistic to model this complex
multi-modal distribution in a closed form. We propose to circumvent this problem by applying the
following bidirectional conditional likelihood estimator:
X 

1
log pθ (xd | x0 ) + log pθ (x0 | xd ) → max,
(16)
θ
|T | 0 d
(x ,x )∈T

4

because it avoids to model p(x0 ) and p(xd ) explicitly.
From the definition of a DCIM, and particularly from the fact that a DCIM is also a Markov chain
model, follows that its reverse conditional distribution factorises similar as (13), i.e.
p(x0 , . . . , xd−1 | xd ) =

d−1
Y

p(xk | xk+1 ).

(17)

k=0

This holds for any p(x0 ) completing the DCIM to a joint distribution. Unfortunately, however, the
reverse conditionals p(xk | xk+1 ) of a DCIM are not (conditionally) independent. This follows from
an argument similar to the one used in Observation 1. In order to resolve the problem, we propose to
consider pairs of tightly connected DCIMs: a forward model and a reverse (i.e. backward) model
p(xd , . . . , x1 | x0 ) =

d
Y

p(xk | xk−1 ) and

q(x0 , . . . , xd−1 | xd ) =

k=1

d−1
Y

q(xk | xk+1 ) (18)

k=0

such that the forward conditionals of p and the backward conditionals of q are strongly (conditional)
independent. Such a pair would give rise to a single joint distribution if
A(xd )
p(xd , . . . , x1 | x0 )
=
0
d−1
d
q(x , . . . , x
|x )
B(x0 )

(19)

holds for any realisation of X for some functions A, B. We already know that this is impossible to
achieve while retaining conditional independence of both, the forward and backward conditionals.
Therefore, we propose to choose the parameters for two models such that the equation is fulfilled as
close as possible. Substituting (1) for the binary case in the lhs of (19), gives
hP
D
Ei
d
k
Qd−1
k
k k−1
k+1 exp
x
,
W
x
+
b
d
k=1
)
? A(x )
k=0 Z̃(x
hP
i =
,
(20)
Qd
d−1
k−1 )
B(x0 )
exp
hxk , V k xk+1 + ak i
k=1 Z(x
k=0

where Zk and Z̃k denote the partition functions of p(xk | xk−1 ) and q(xk | xk+1 ) respectively. It is
readily seen that all terms in the exponentials cancel out if (V k−1 )T = W k holds for the weights and
bk = ak−1 holds for the biases. The remaining, not cancelling terms in the lhs of (19) are then the
partition functions of the conditional distributions.
Summarising, a pair of such DCIMs share the same structure as well as their weights and biases.
They are therefore represented by a single DNN and can be learned simultaneously by the estimator
(16), which now reads as
X 

1
log pθ (xd | x0 ) + log qθ (x0 | xd ) → max .
(21)
θ
|T | 0 d
(x ,x )∈T

Since both, the forward and backward conditionals are strongly independent, approximation (2) can
be applied for computing the probabilities p(xd | x0 ) and q(x0 | xd ).
Remark 2. If the model consists of one layer only (d = 1), then the pair of p(x1 | x0 ) and q(x0 | x1 )
define a single joint distribution, which is an RBM, i.e. an MRF on a bipartite graph. The estimator
(16) becomes the pseudo-likelihood estimator.
2.4

Relations to other models

We briefly discuss relations between the proposed model and other models not mentioned in the
introduction.
The connection between Bayesian networks and neural networks with injected noise, used in [7, 13]
is different from ours. It relates two equivalent representations of stochastic models, while we relate
stochastic and deterministic models.
Similar to our work, Hinton et al. [5] uses a constraint that generator and recognition networks are
related by transposition of weights in the unsupervised, layer-by-layer pre-training phase, motivated
5

by their study of deep Boltzmann machine. For the supervised fine-tuning the two models are
decoupled.
The proposed model is of course related to classical auto-encoders [14] at least technically. Our
learning approach is generative, but in contrast to auto-encoders it is supervised. Moreover, the
“decoding” part (the reverse model) is in our case constrained to have the same parameters as the
“encoding” part (the forward model).
Our model and the proposed generative learning approach are undoubtedly related to generative
adversarial networks (GAN), [3],[12] proposed by Goodfellow et al. Similar to them, the reverse
part of our model aims at generating data. But in contrast to them our model uses no specific “noise
space” and conditions the image distribution on the class. The randomness in the generating part
comes solely through the reverse Markov chain. We believe that this should suffice for modelling rich
distributions in the input space. It is however to expect that the approximation (2) used for learning
might impose limitations on the expressive power of this model part as compared with GANs (see
experiments). Another difference is that our model is learned jointly with a “twin”, i.e. the forward
part, rather than in competition with an “adversarial”.

3

Experiments

3.1

MNIST dense

The first experiment is a simple proof of concept. We trained a small network (three fully connected
layers with 512, 512, 10 nodes) on the standard MNIST dataset [8] for both learning approaches:
the discriminative and the generative one. Sigmoid activation was chosen for all but the last layer,
for which we used soft-max activation, i.e. considering its nodes as categorical variable. A standard
optimiser without drop-out and any other regularisers was used for searching the maximum of
the objective function. Fig. 1 shows the training and validation accuracies for both approaches as
functions of the iterations. It is clearly seen that the combined objective (16) of the generative learning
imposes only a rather insignificant drop of the classification accuracy. Fig. 1 also shows the training
and validation loss of the the backward model.
To further asses the reverse model learned by the generative approach, we sampled 10 images for each
of the classes from both models. The results are shown in Fig. 2. For each sampled image we also
applied the forward model to classify it. The classes with the highest probability along with values
of that probability are shown on top of each sampled image. It is clearly seen that the generative
learning approach has found a set of model parameters that yields simultaneously good classification
accuracy and the ability to generate images of average digits. On the other hand, it is also visible
from Fig. 2 that the generative model part learned only the mean shapes of the digits.
0.26

1.00

0.25

loss

accuracy

0.95

0.90

0.85
0

0.24
0.23

10

20

epochs

30

generative
discriminative
40
50

0.22
0

10

20

epochs

30

40

50

Figure 1: Learning simple network on MNIST. Left: accuracy for discriminative and generative
learning (dashed - training, solid: validation). Right: loss of the reverse model in generative learning
(dashed - training, solid: validation).

6

0:99

0:99

0:99

0:99

0:99

0:99

0:99

0:99

0:99

0:99

1:53

1:93

6:74

1:99

1:98

1:86

1:69

1:97

8:65

1:89

2:99

2:99

2:97

2:99

2:99

2:99

2:99

2:98

2:99

2:99

3:99

3:99

3:99

3:99

3:99

3:99

3:99

3:99

3:99

3:99

4:99

4:91

4:99

4:99

4:97

4:99

4:99

4:99

4:99

4:99

8:94

5:99

5:80

5:99

5:96

5:99

5:99

5:99

5:99

5:99

6:99

6:89

6:99

6:99

2:94

6:99

6:99

6:99

6:99

6:99

9:94

7:99

9:99

7:99

7:99

7:99

7:99

7:99

7:99

7:99

8:99

8:99

8:99

8:99

7:49

8:99

8:99

8:99

8:99

8:99

9:99

4:99

5:73

9:99

7:93

9:99

9:99

9:99

9:99

9:99

Figure 2: Images sampled form the reverse model part along with their classification and probability
values (see text). Each row shows images sampled from one class. Left: images obtained by recursive
sampling through the net starting from the class layer till the image layer. Right: images obtained by
sampling through one layer only and then recursively computing probabilities till the image layer.

3.2

MNIST CNN

The second experiment intends to analyse the behaviour of the proposed generative learning approach
for convolutional neural networks. We learned a deep CNN ( eight layers) with architecture ((3,1,32),
(3,1,32), (2,2,32), (3,1,64), (3,1,64), (2,2,64), (50), (10)) on MNIST data. The components of the
triplets denote the window size, stride and kernel number for convolutional layers. The singletons
denote the size of dense layers. Sigmoid activation was used for all but the last layer. When using the
objective (21), the net achieves 0.989 validation accuracy for the forward part and 0.23 loss for the
reverse model part. We observed neither over-fitting nor “vanishing gradient” effects. We conjecture
that the reverse learning task serves as a strong regulariser. The left tableaux in Fig. 3 shows images
sampled from the reverse model part. Again, for better visibility, we sampled from the the distribution
q(xd−1 | xd ) and then recursively computed the probabilities down to the image layer by applying
the approximation (2). It is clearly seen that the learned generative model part is not able to capture
the multi-modal image distribution. We conjecture that one of the possible reasons is the simplicity
of the approximation (2), especially when applied for learning of the reverse model part.
To analyse the problem, we considered a somewhat different learning objective for this model part
1
|T |

X



log pθ (xd | x0 ) + log qθ (x0 | xd−1 ) → max .
θ

(x0 ,xd )∈T

(22)

where xd−1 is sampled from pθ (xd−1 | x0 ) for the current θ. The model learned with this objective
achieved 0.990 validation accuracy and 0.14 validation loss for the reverse model part. Images
sampled from this model in the same way as described above, are shown in the right tableaux of
Fig. 3. It is clearly seen that this model captured modes in the image distribution, i.e. different writing
styles of the digits.
All experiments presented in this paper were carried out by using Keras [2] on top of Tensorflow [1].
7

Figure 3: Images sampled from generatively learned CNNs on MNIST data. For better visibility
the images were obtained by sampling through one layer only and then recursively computing
probabilities till the image layer. Left: learning method as described in Sec. 2.3. Right: alternative
learning method described in Sec. 3.2.

4

Conclusions

We proposed a class of probabilistic models that leads to a new statistical interpretation of DNNs
in which all neurons of the net become random variables. In contrast to Boltzmann machines, this
class allows for an efficient approximation when computing the distribution of the output variables
conditioned on a realisation of the input variables. The computation itself becomes identical to the
recursive forward computation for a DNN.
The second objective of the paper, to design a generative learning approach for DCIMs, has been
reached only partially. The proposed approach and its variants do allow to learn the forward and
backward model parts simultaneously. However, the achievable expressive power of the backward
part of the model is still inferior to GANs.
Notwithstanding this, we believe that the presented approach opens interesting research directions.
One of them is to analyse the proposed approximation and to search for better ones, leading to new
and better activation functions. Another question in that direction is a statistical interpretation of the
ReLu activation as the expectation of a possibly continuous random variable.
A different direction calls for generalising DCIMs by introducing interactions between neurons in
layers. This would lead to deep CRFs. We envision to learn such models, at least discriminatively, by
using e.g. recent results for efficient marginal approximations of log-supermodular CRF.
Acknowledgements
A. Shekhovtsov was supported by Toyota Motor Europe HS, O. Fikar was supported by Czech
Technical University in Prague under grant SGS17/185/OHK3/3T/13 and B. Flach was supported by
Czech Science Foundation under grant 16-05872S.
8

References
[1] Abadi, M. et al. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems.
Software available from tensorflow.org.
[2] Chollet, F. et al. (2015). Keras. https://github.com/fchollet/keras.
[3] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville,
A. C., and Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information
Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014,
December 8-13 2014, Montreal, Quebec, Canada, pages 2672–2680.
[4] Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995). The wake-sleep algorithm for
unsupervised neural networks. Science, 268:1158.
[5] Hinton, G. E., Osindero, S., and Teh, Y. W. (2006). A fast learning algorithm for deep belief
networks. Neural Computation, 18(7):1527–1554.
[6] Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. CoRR, abs/1312.6114.
[7] Kingma, D. P. and Welling, M. (2014). Efficient gradient-based inference through transformations
between bayes nets and neural nets. In Proceedings of the 31st International Conference on
International Conference on Machine Learning - Volume 32, ICML’14, pages II–1782–II–1790.
JMLR.org.
[8] LeCun, Y. and Cortes, C. (2010). MNIST handwritten digit database.
[9] Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. (2009). Convolutional deep belief networks for
scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual
International Conference on Machine Learning, ICML ’09, pages 609–616, New York, NY, USA.
ACM.
[10] Mnih, A. and Gregor, K. (2014). Neural Variational Inference and Learning in Belief Networks.
In Proceedings of the 31st International Conference on Machine Learning, Cycle 2, volume 32 of
JMLR Proceedings, pages 1791–1799. JMLR.org.
[11] Neal, R. M. (1992). Connectionist learning of belief networks. Artif. Intell., 56(1):71–113.
[12] Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised representation learning with deep
convolutional generative adversarial networks. In ICLR.
[13] Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and
approximate inference in deep generative models. In Jebara, T. and Xing, E. P., editors, Proceedings
of the 31st International Conference on Machine Learning (ICML-14), pages 1278–1286. JMLR
Workshop and Conference Proceedings.
[14] Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by
back-propagating errors. Nature, 323:533–536.
[15] Salakhutdinov, R. (2015). Learning deep generative models. Annual Review of Statistics and Its
Application, 2:361–385.
[16] Salakhutdinov, R. and Hinton, G. E. (2009). Deep Boltzmann Machines. In AISTATS, pages
448–455.

9

