Deep Predictive Coding Networks
Authors: Rakesh Chalasani, Jose C. Principe
ICLR 2013

Abstract: The quality of data representation in deep learning methods is directly related to the prior model imposed on the representations; however, generally used fixed priors are not capable of adjusting to the context in the data. To address this issue, we propose deep predictive coding networks, a hierarchical generative model that empirically alters priors on the latent representations in a dynamic and context-sensitive manner. This model captures the temporal dependencies in time-varying signals and uses top-down information to modulate the representation in lower layers. The centerpiece of our model is a novel procedure to infer sparse states of a dynamic model; which is used for feature extraction. We also extend this feature extraction block to introduce a pooling function that captures locally invariant representations. When applied on a natural video data, we show that our method is able to learn high-level visual features. We also demonstrate the role of the top-down connections by showing the robustness of the proposed model to structured noise.

arXiv:1301.3541v3  [cs.LG]  15 Mar 2013
Deep Predictive Coding Networks
Rakesh Chalasani Jose C. Principe
Department of Electrical and Computer Engineering
University of Florida, Gainesville, FL 32611
rakeshch@ufl.edu, principe@cnel.ufl.edu
Abstract
The quality of data representation in deep learning methods is directly related to
the prior model imposed on the representations; however, ge nerally used ﬁxed
priors are not capable of adjusting to the context in the data. To address this issue,
we propose deep predictive coding networks, a hierarchical generative model that
empirically alters priors on the latent representations in a dynamic and context-
sensitive manner. This model captures the temporal dependencies in time-varying
signals and uses top-down information to modulate the repre sentation in lower
layers. The centerpiece of our model is a novel procedure to infer sparse states of a
dynamic network which is used for feature extraction. We also extend this feature
extraction block to introduce a pooling function that captu res locally invariant
representations. When applied on a natural video data, we sh ow that our method
is able to learn high-level visual features. We also demonstrate the role of the top-
down connections by showing the robustness of the proposed model to structured
noise.
1 Introduction
The performance of machine learning algorithms is dependen t on how the data is represented. In
most methods, the quality of a data representation is itselfdependent on prior knowledge imposed on
the representation. Such prior knowledge can be imposed usi ng domain speciﬁc information, as in
SIFT [1], HOG [2], etc., or in learning representations usin g ﬁxed priors like sparsity [3], temporal
coherence [4], etc. The use of ﬁxed priors became particularly popular while training deep networks
[5–8]. In spite of the success of these general purpose prior s, they are not capable of adjusting to
the context in the data. On the other hand, there are several a dvantages to having a model that can
“actively” adapt to the context in the data. One way of achiev ing this is to empirically alter the
priors in a dynamic and context-sensitive manner. This will be the m ain focus of this work, with
emphasis on visual perception.
Here we propose a predictive coding framework, where a deep locally-connected generative model
uses “top-down” information to empirically alter the prior s used in the lower layers to perform
“bottom-up” inference. The centerpiece of the proposed mod el is extracting sparse features from
time-varying observations using a linear dynamical model . To this end, we propose a novel proce-
dure to infer sparse states (or features) of a dynamical system. We then extend this feature extraction
block to introduce a pooling strategy to learn invariant feature representations from the data. In line
with other “deep learning” methods, we use these basic build ing blocks to construct a hierarchical
model using greedy layer-wise unsupervised learning. The h ierarchical model is built such that the
output from one layer acts as an input to the layer above. In other words, the layers are arranged in a
Markov chain such that the states at any layer are only dependent on the representations in the layer
below and above, and are independent of the rest of the model. The overall goal of the dynamical
system at any layer is to make the best prediction of the representation in the layer below using the
top-down information from the layers above and the temporal information from the previous states.
Hence, the name deep predictive coding networks (DPCN).
1
1.1 Related W ork
The DPCN proposed here is closely related to models proposed in [9, 10], where predictive cod-
ing is used as a statistical model to explain cortical functi ons in the mammalian brain. Similar to
the proposed model, they construct hierarchical generative models that seek to infer the underlying
causes of the sensory inputs. While Rao and Ballard [9] use an update rule similar to Kalman ﬁlter
for inference, Friston [10] proposed a general framework considering all the higher-order moments
in a continuous time dynamic model. However, neither of the m odels is capable of extracting dis-
criminative information, namely a sparse and invariant representation, from an image sequence that
is helpful for high-level tasks like object recognition. Un like these models, here we propose an
efﬁcient inference procedure to extract locally invariant representation from image sequences and
progressively extract more abstract information at higher levels in the model.
Other methods used for building deep models, like restricted Boltzmann machine (RBM) [11], auto-
encoders [8, 12] and predictive sparse decomposition [13], are also related to the model proposed
here. All these models are constructed on similar underlyin g principles: (1) like ours, they also use
greedy layer-wise unsupervised learning to construct a hierarchical model and (2) each layer consists
of an encoder and a decoder. The key to these models is to learn both encoding and decoding
concurrently (with some regularization like sparsity [13] , denoising [8] or weight sharing [11]),
while building the deep network as a feed forward model using only the encoder. The idea is
to approximate the latent representation using only the fee d-forward encoder, while avoiding the
decoder which typically requires a more expensive inference procedure. However in DPCN there is
no encoder. Instead, DPCN relies on an efﬁcient inference pr ocedure to get a more accurate latent
representation. As we will show below, the use of reciprocal top-down and bottom-up connections
make the proposed model more robust to structured noise duri ng recognition and also allows it to
perform low-level tasks like image denoising.
To scale to large images, several convolutional models are also proposed in a similar deep learning
paradigm [5–7]. Inference in these models is applied over anentire image, rather than small parts of
the input. DPCN can also be extended to form a convolutional network, but this will not be discussed
here.
2 Model
In this section, we begin with a brief description of the gene ral predictive coding framework and
proceed to discuss the details of the architecture used in this work. The basic block of the proposed
model that is pervasive across all layers is a generalized state-space model of the form:
˜ yt = F(xt) + nt
xt = G(xt−1, ut) + vt (1)
where ˜ yt is the data and F and G are some functions that can be parameterized, say byθ. The terms
ut are called the unknown causes . Since we are usually interested in obtaining abstract information
from the observations, the causes are encouraged to have a no n-linear relationship with the obser-
vations. The hidden states, xt, then “mediate the inﬂuence of the cause on the output and end ow
the system with memory” [10]. The terms vt and nt are stochastic and model uncertainty. Several
such state-space models can now be stacked, with the output from one acting as an input to the layer
above, to form a hierarchy. Such an L-layered hierarchical model at any time ’ t’ can be described
as1:
u(l−1 )
t = F(x(l)
t ) + n(l)
t ∀l ∈ { 1, 2, ..., L}
x(l)
t = G(x(l)
t−1, u(l)
t ) + v(l)
t (2)
The terms v(l)
t and n(l)
t form stochastic ﬂuctuations at the higher layers and enter e ach layer in-
dependently. In other words, this model forms a Markov chain across the layers, simplifying the
inference procedure. Notice how the causes at the lower laye r form the “observations” to the layer
above — the causes form the link between the layers, and the st ates link the dynamics over time.
The important point in this design is that the higher-level p redictions inﬂuence the lower levels’
1When l = 1, i.e., at the bottom layer, u(i− 1)
t = yt, where yt the input data.
2
Causes (ut)
States (xt)
Observations (yt)
(a) Shows a single layered dynamic network
depicting a basic computational block.
- States (xt)
- Causes (ut)
(Invariant Units)
{ {Layer 1 Layer 2
(b) Shows the distributive hierarchical model formed by
stacking several basic blocks.
Figure 1: (a) Shows a single layered network on a group of smal l overlapping patches of the input
video. The green bubbles indicate a group of inputs ( y(n)
t , ∀n), red bubbles indicate their corre-
sponding states ( x(n)
t ) and the blue bubbles indicate the causes ( ut) that pool all the states within
the group. (b) Shows a two-layered hierarchical model const ructed by stacking several such basic
blocks. For visualization no overlapping is shown between the image patches here, but overlapping
patches are considered during actual implementation.
inference. The predictions from a higher layer non-linearl y enter into the state space model by em-
pirically altering the prior on the causes. In summary, the t op-down connections and the temporal
dependencies in the state space inﬂuence the latent representation at any layer.
In the following sections, we will ﬁrst describe a basic comp utational network, as in (1) with a
particular form of the functions F and G. Speciﬁcally, we will consider a linear dynamical model
with sparse states for encoding the inputs and the state transitions, followed by the non-linear pooling
function to infer the causes. Next, we will discuss how to stack and learn a hierarchical model using
several of these basic networks. Also, we will discuss how to incorporate the top-down information
during inference in the hierarchical model.
2.1 Dynamic network
To begin with, we consider a dynamic network to extract featu res from a small part of a video
sequence. Let {y1, y2, ..., yt, ...} ∈ RP be a P -dimensional sequence of a patch extracted from
the same location across all the frames in a video 2 . To process this, our network consists of two
distinctive parts (see Figure.1a): feature extraction (inferring states) and pooling (inferring causes).
For the ﬁrst part, sparse coding is used in conjunction with a linear state space model to map the
inputs yt at time t onto an over-complete dictionary of K-ﬁlters, C ∈ RP ×K (K > P ), to get
sparse states xt ∈ RK . To keep track of the dynamics in the latent states we use a lin ear function
with state-transition matrix A ∈ RK×K . More formally, inference of the features xt is performed
by ﬁnding a representation that minimizes the energy function:
E1(xt, yt, C, A) = ∥yt − Cxt∥2
2 + λ∥xt − Axt−1∥1 + γ ∥xt∥1 (3)
Notice that the second term involving the state-transition is also constrained to be sparse to make
the state-space representation consistent.
Now, to take advantage of the spatial relationships in a loca l neighborhood, a small group of states
x(n)
t , where n ∈ { 1, 2, ...N } represents a set of contiguous patches w.r.t. the position i n the image
space, are added (orsum pooled ) together. Such pooling of the states may be lead to local translation
invariance. On top this, a D-dimensional causes ut ∈ RD are inferred from the pooled states to
obtain representation that is invariant to more complex loc al transformations like rotation, spatial
frequency, etc. In line with [14], this invariant function i s learned such that it can capture the
dependencies between the components in the pooled states. S peciﬁcally, the causes ut are inferred
2Here yt is a vectorized form of
√
P ×
√
P square patch extracted from a frame at time t.
3
by minimizing the energy function:
E2(ut, xt, B) =
N∑
n=1
( K∑
k=1
|γk · x(n)
t,k |
)
+ β∥ut∥1 (4)
γk = γ0
[ 1 + exp(−[But]k)
2
]
where γ0 > 0 is some constant. Notice that here ut multiplicatively interacts with the accumulated
states through B, modeling the shape of the sparse prior on the states. Essent ially, the invariant
matrix B is adapted such that each component ut connects to a group of components in the ac-
cumulated states that co-occur frequently. In other words, whenever a component in ut is active
it lowers the coefﬁcient of a set of components in x(n)
t , ∀n, making them more likely to be active.
Since co-occurring components typically share some common statistical regularity, such activity of
ut typically leads to locally invariant representation [14].
Though the two cost functions are presented separately abov e, we can combine both to devise a
uniﬁed energy function of the form:
E(xt, ut, θ) =
N∑
n=1
(1
2∥y(n)
t − Cx(n)
t ∥2
2 + λ∥x(n)
t − Ax(n)
t−1∥1 +
K∑
k=1
|γt,k · x(n)
t,k |
)
+ β∥ut∥1
(5)
γt,k =γ0
[ 1 + exp(−[But]k)
2
]
where θ = {A, B, C}. As we will discuss next, both xt and ut can be inferred concurrently from
(5) by alternatively updating one while keeping the other ﬁx ed using an efﬁcient proximal gradient
method.
2.2 Learning
To learn the parameters in (5), we alternatively minimize E(xt, ut, θ) using a procedure similar to
block co-ordinate descent. We ﬁrst infer the latent variabl es (xt, ut) while keeping the parameters
ﬁxed and then update the parameters θ while keeping the variables ﬁxed. This is done until the
parameters converge. We now discuss separately the inferen ce procedure and how we update the
parameters using a gradient descent method with the ﬁxed variables.
2.2.1 Inference
We jointly infer bothxt and ut from (5) using proximal gradient methods, taking alternative gradient
descent steps to update one while holding the other ﬁxed. In o ther words, we alternate between
updating xt and ut using a single update step to minimize E1 and E2, respectively. However,
updating xt is relatively more involved. So, keeping aside the causes, w e ﬁrst focus on inferring
sparse states alone from E1, and then go back to discuss the joint inference of both the st ates and
the causes.
Inferring States: Inferring sparse states, given the parameters, from a linea r dynamical system
forms the crux of our model. This is performed by ﬁnding the so lution that minimizes the energy
function E1 in (3) with respect to the states xt (while keeping the sparsity parameter γ ﬁxed).
Here there are two priors of the states: the temporal depende nce and the sparsity term. Although
this energy function E1 is convex in xt, the presence of two non-smooth terms makes it hard to
use standard optimization techniques used for sparse codin g alone. A similar problem is solved
using dynamic programming [15], homotopy [16] and Bayesian sparse coding [17]; however, the
optimization used in these models is computationally expensive for use in large scale problems like
object recognition.
To overcome this, inspired by the method proposed in [18] for structured sparsity, we propose an
approximate solution that is consistent and able to use efﬁc ient solvers like fast iterative shrinkage
thresholding alogorithm (FISTA) [19]. The key to our approach is to ﬁrst use Nestrov’s smoothness
method [18, 20] to approximate the non-smooth state transition term. The resulting energy function
4
is a convex and continuously differentiable function in xt with a sparsity constraint, and hence, can
be efﬁciently solved using proximal methods like FISTA.
To begin, letΩ( xt) = ∥et∥1 where et = ( xt − Axt−1). The idea is to ﬁnd a smooth approximation
to this function Ω( xt) in et. Notice that, since et is a linear function on xt, the approximation will
also be smooth w.r.t. xt. Now, we can re-write Ω( xt) using the dual norm of ℓ1 as
Ω( xt) = arg max
∥α ∥∞ ≤1
α T et
where α ∈ Rk. Using the smoothing approximation from Nesterov [20] on Ω( xt):
Ω( xt) ≈ fµ (et) = arg max
∥α ∥∞ ≤1
[α T et − µd(α )] (6)
where d(·) = 1
2 ∥α ∥2
2 is a smoothing function and µ is a smoothness parameter. From Nestrov’s
theorem [20], it can be shown that fµ (et) is convex and continuously differentiable in et and the
gradient of fµ (et) with respect to et takes the form
∇et fµ (et) = α ∗ (7)
where α ∗ is the optimal solution to fµ (et) = arg max
∥α ∥∞ ≤1
[α T et − µd(α )] 3. This implies, by using
the chain rule, that fµ (et) is also convex and continuously differentiable in xt and with the same
gradient.
With this smoothing approximation, the overall cost function from (3) can now be re-written as
xt = arg min
xt
1
2∥yt − Cxt∥2
2 + λfµ (et) + γ∥xt∥1 (8)
with the smooth part h(xt) = 1
2 ∥yt − Cxt∥2
2 + λfµ (et) whose gradient with respect to xt is given
by
∇xt h(xt) = CT (yt − Cxt) + λα ∗ (9)
Using the gradient information in (9), we solve for xt from (8) using FISTA [19].
Inferring Causes: Given a group of state vectors, ut can be inferred by minimizing E2, where we
deﬁne a generative model that modulates the sparsity of thepooled state vector, ∑
n |x(n)|. Here we
observe that FISTA can be readily applied to infer ut, as the smooth part of the function E2:
h(ut) =
K∑
k=1
(
γ0
[ 1 + exp( −[But]k)
2
]
·
N∑
n=1
|x(n)
t,k |
)
(10)
is convex, continuously differentiable and Lipschitz inut [21] 4. Following [19], it is easy to obtain
a bound on the convergence rate of the solution.
Joint Inference: We showed thus far that bothxt and ut can be inferred from their respective energy
functions using a ﬁrst-order proximal method called FISTA. However, for joint inference we have
to minimize the combined energy function in (5) over both xt and ut. We do this by alternately
updating xt and ut while holding the other ﬁxed and using a single FISTA update step at each
iteration. It is important to point out that the internal FIS TA step size parameters are maintained
between iterations. This procedure is equivalent to alternating minimization using gradient descent.
Although this procedure no longer guarantees convergence of both xt and ut to the optimal solution,
in all of our simulations it lead to a reasonably good solutio n. Please refer to Algorithm. 1 (in the
supplementary material) for details. Note that, with the alternating update procedure, each xt is now
inﬂuenced by the feed-forward observations, temporal pred ictions and the feedback connections
from the causes.
3Please refer to the supplementary material for the exact form of α ∗.
4The matrix B is initialized with non-negative entries and continues to b e non-negative without any addi-
tional constraints [21].
5
2.2.2 Parameter Updates
With xt and ut ﬁxed, we update the parameters by minimizing E in (5) with respect to θ. Since the
inputs here are a time-varying sequence, the parameters are updated using dual estimation ﬁltering
[22]; i.e., we put an additional constraint on the parameter s such that they follow a state space
equation of the form:
θt = θt−1 + zt (11)
where zt is Gaussian transition noise over the parameters. This keep s track of their temporal rela-
tionships. Along with this constraint, we update the parame ters using gradient descent. Notice that
with a ﬁxed xt and ut, each of the parameter matrices can be updated independentl y. Matrices C
and B are column normalized after the update to avoid any trivial solution.
Mini-Batch Update: To get faster convergence, the parameters are updated after performing infer-
ence over a large sequence of inputs instead of at every time instance. With this “batch” of signals,
more sophisticated gradient methods, like conjugate gradi ent, can be used and, hence, can lead to
more accurate and faster convergence.
2.3 Building a hierarchy
So far the discussion is focused on encoding a small part of a v ideo frame using a single stage
network. To build a hierarchical model, we use this single st age network as a basic building block
and arrange them up to form a tree structure (see Figure.1b). To learn this hierarchical mode l, we
adopt a greedy layer-wise procedure like many other deep learning methods [6, 8, 11]. Speciﬁcally,
we use the following strategy to learn the hierarchical model.
For the ﬁrst (or bottom) layer, we learn a dynamic network as d escribed above over a group of
small patches from a video. We then take this learned network and replicate it at several places
on a larger part of the input frames (similar to weight sharin g in a convolutional network [23]).
The outputs (causes) from each of these replicated networks are considered as inputs to the layer
above. Similarly, in the second layer the inputs are again grouped together (depending on the spatial
proximity in the image space) and are used to train another dynamic network. Similar procedure can
be followed to build more higher layers.
We again emphasis that the model is learned in a layer-wise ma nner, i.e., there is no top-down
information while learning the network parameters. Also no te that, because of the pooling of the
states at each layers, the receptive ﬁeld of the causes becomes progressively larger with the depth of
the model.
2.4 Inference with top-down information
With the parameters ﬁxed, we now shift our focus to inference in the hierarchical model with the
top-down information. As we discussed above, the layers in the hierarchy are arranged in a Markov
chain, i.e., the variables at any layer are only inﬂuenced by the variables in the layer below and the
layer above. Speciﬁcally, the states x(l)
t and the causes u(l)
t at layer l are inferred from u(l−1)
t and
are inﬂuenced by x(l+1)
t (through the prediction term C(l+1)x(l+1)
t ) 5. Ideally, to perform inference
in this hierarchical model, all the states and the causes have to be updated simultaneously depending
on the present state of all the other layers until the model re aches equilibrium [10]. However, such
a procedure can be very slow in practice. Instead, we propose an approximate inference procedure
that only requires a single top-down ﬂow of information and then a single bottom-up inference using
this top-down information.
5The sufﬁxes n indicating the group are considered implicit here to simplify the notation.
6
For this we consider that at any layer l a group of input u(l−1,n )
t , ∀n ∈ { 1, 2, ..., N } are encoded
using a group of states x(l,n )
t , ∀n and the causes u(l)
t by minimizing the following energy function:
El(x(l)
t , u(l)
t , θ(l)) =
N∑
n=1
(1
2∥u(l−1,n )
t − C(l)x(l,n )
t ∥2
2 + λ∥x(l,n )
t − A(l)x(l,n )
t−1 ∥1
+
K∑
k=1
|γ(l)
t,k · x(l,n )
t,k |
)
+ β∥u(l)
t ∥1 + 1
2 ∥u(l)
t − ˆ u(l+1)
t ∥2
2 (12)
γ(l)
t,k = γ0
[ 1 + exp(−[B(l)u(l)
t ]k)
2
]
where θ(l) = {A(l), B(l), C(l)}. Notice the additional term involving ˆ u(l+1)
t when compared to (5).
This comes from the top-down information, where we call ˆ u(l+1)
t as the top-down prediction of the
causes of layer (l) using the previous states in layer (l + 1) . Speciﬁcally, before the “arrival” of a
new observation at time t, at each layer (l) (starting from the top-layer) we ﬁrst propagate the most
likely causes to the layer below using the state at the previous time instance x(l)
t−1 and the predicted
causes ˆ u(l+1)
t . More formally, the top-down prediction at layer l is obtained as
ˆ u(l)
t = C(l)ˆ x(l)
t
where ˆ x(l)
t = arg min
x(l)
t
λ(l)∥x(l)
t − A(l)x(l)
t−1∥1 + γ0
K∑
k=1
|ˆγt,k · x(l)
t,k | (13)
and ˆγt,k = (exp( −[B(l)ˆ u(l+1)
t ]k))/2
At the top most layer, L, a “bias” is set such that ˆ u(L)
t = ˆ u(L)
t−1, i.e., the top-layer induces some
temporal coherence on the ﬁnal outputs. From (13), it is easy to show that the predicted states for
layer l can be obtained as
ˆx(l)
t,k =
{
[A(l)x(l)
t−1]k, γ 0γt,k < λ (l)
0, γ 0γt,k ≥ λ(l) (14)
These predicted causesˆ u(l)
t , ∀l ∈ { 1, 2, ..., L} are substituted in (12) and a single layer-wise bottom-
up inference is performed as described in section 2.2.1 6. The combined prior now imposed on the
causes, β∥u(l)
t ∥1 + 1
2 ∥u(l)
t − ˆ u(l+1)
t ∥2
2, is similar to the elastic net prior discussed in [24], leading to
a smoother and biased estimate of the causes.
3 Experiments
3.1 Receptive ﬁelds of causes in the hierarchical model
Firstly, we would like to test the ability of the proposed mod el to learn complex features in the
higher-layers of the model. For this we train a two layered ne twork from a natural video. Each
frame in the video was ﬁrst contrast normalized as described in [13]. Then, we train the ﬁrst layer
of the model on 4 overlapping contiguous 15 × 15 pixel patches from this video; this layer has
400 dimensional states and 100 dimensional causes. The caus es pool the states related to all the
4 patches. The separation between the overlapping patches he re was 2 pixels, implying that the
receptive ﬁeld of the causes in the ﬁrst layer is 17 × 17 pixels. Similarly, the second layer is trained
on 4 causes from the ﬁrst layer obtained from 4 overlapping 17 × 17 pixel patches from the video.
The separation between the patches here is 3 pixels, implying that the receptive ﬁeld of the causes
in the second layer is 20 × 20 pixels. The second layer contains 200 dimensional states an d 50
dimensional causes that pools the states related to all the 4 patches.
Figure 2 shows the visualization of the receptive ﬁelds of th e invariant units (columns of matrix
B) at each layer. We observe that each dimension of causes in th e ﬁrst layer represents a group of
6Note that the additional term 1
2 ∥u(l)
t − ˆ u(l+1)
t ∥2
2 in the energy function only leads to a minor modiﬁcation
in the inference procedure, namely this has to be added to h(ut) in (10).
7
(a) Layer 1 invariant matrix, B(1)
 (b) Layer 2 invariant matrix, B(2)
Figure 2: Visualization of the receptive ﬁelds of the invariant units learned in (a) layer 1 and (b) layer
2 when trained on natural videos. The receptive ﬁelds are constructed as a weighted combination of
the dictionary of ﬁlters at the bottom layer.
primitive features (like inclined lines) which are localized in orientation or position 7. Whereas, the
causes in the second layer represent more complex features, like corners, angles, etc. These ﬁlters
are consistent with the previously proposed methods like Lee et al. [5] and Zeiler et al. [7].
3.2 Role of top-down information
In this section, we show the role of the top-down information during inference, particularly in the
presence of structured noise. Video sequences consisting of objects of three different shapes (Refer
to Figure 3) were constructed. The objective is to classify e ach frame as coming from one of the
three different classes. For this, several 32 × 32 pixel 100 frame long sequences were made using
two objects of the same shape bouncing off each other and the “walls”. Several such sequences were
then concatenated to form a 30,000 long sequence. We train a two layer network using this sequence.
First, we divided each frame into12 × 12 patches with neighboring patches overlapping by 4 pixels;
each frame is divided into 16 patches. The bottom layer was tr ained such the 12 × 12 patches were
used as inputs and were encoded using a 100 dimensional state vector. A 4 contiguous neighboring
patches were pooled to infer the causes that have 40 dimensions. The second layer was trained with
4 ﬁrst layer causes as inputs, which were itself inferred from20 × 20 contiguous overlapping blocks
of the video frames. The states here are 60 dimensional long and the causes have only 3 dimensions.
It is important to note here that the receptive ﬁeld of the second layer causes encompasses the entire
frame.
We test the performance of the DPCN in two conditions. The ﬁrs t case is with 300 frames of clean
video, with 100 frames per shape, constructed as described above. We consider this as a single video
without considering any discontinuities. In the second case, we corrupt the clean video with “struc-
tured” noise, where we randomly pick a number of objects from same three shapes with a Poisson
distribution (with mean 1.5) and add them to each frame independently at a random locations. There
is no correlation between any two consecutive frames regarding where the “noisy objects” are added
(see Figure.3b).
First we consider the clean video and perform inference with only bottom-up inference, i.e., during
inference we consider ˆ u(l)
t = 0 , ∀l ∈ { 1, 2}. Figure 4a shows the scatter plot of the three dimen-
sional causes at the top layer. Clearly, there are 3 clusters recognizing three different shape in the
video sequence. Figure 4b shows the scatter plot when the sam e procedure is applied on the noisy
video. We observe that 3 shapes here can not be clearly distin guished. Finally, we use top-down
information along with the bottom-up inference as describe d in section 2.4 on the noisy data. We
argue that, since the second layer learned class speciﬁc inf ormation, the top-down information can
help the bottom layer units to disambiguate the noisy objects from the true objects. Figure 4c shows
the scatter plot for this case. Clearly, with the top-down in formation, in spite of largely corrupted
sequence, the DPCN is able to separate the frames belonging to the three shapes (the trace from one
cluster to the other is because of the temporal coherence imposed on the causes at the top layer.).
7Please refer to supplementary material for more results.
8
(a) Clear Sequences (b) Corrupted Sequences
Figure 3: Shows part of the (a) clean and (b) corrupted video s equences constructed using three
different shapes. Each row indicates one sequence.
0
5
10
0
2
4
0
2
4
6
Object 1
Object 2
Object 3
(a)
0
5
10
0
2
4
6
0
2
4
6
Object 1
Object 2
Object 3
(b)
0
2
4
6
0
1
2
3
0
2
4
6
Object 1
Object 2
Object 3
(c)
Figure 4: Shows the scatter plot of the 3 dimensional causes a t the top-layer for (a) clean video
with only bottom-up inference, (b) corrupted video with only bottom-up inference and (c) corrupted
video with top-down ﬂow along with bottom-up inference. At e ach point, the shape of the marker
indicates the true shape of the object in the frame.
4 Conclusion
In this paper we proposed the deep predictive coding network , a generative model that empirically
alters the priors in a dynamic and context sensitive manner.This model composes to two main com-
ponents: (a) linear dynamical models with sparse states used for feature extraction, and (b) top-down
information to adapt the empirical priors. The dynamic mode l captures the temporal dependencies
and reduces the instability usually associated with sparse coding 8, while the task speciﬁc informa-
tion from the top layers helps to resolve ambiguities in the lower-layer improving data representation
in the presence of noise. We believe that our approach can be extended with convolutional methods,
paving the way for implementation of high-level tasks like o bject recognition, etc., on large scale
videos or images.
Acknowledgments
This work is supported by the Ofﬁce of Naval Research (ONR) gr ant #N000141010375. We thank
Austin J. Brockmeier and Matthew Emigh for their comments and suggestions.
References
[1] David G. Lowe. Object recognition from local scale-inva riant features. In Proceedings of the
International Conference on Computer V ision-V olume 2 - V ol ume 2 , ICCV ’99, pages 1150–,
1999. ISBN 0-7695-0164-8.
[2] Navneet Dalal and Bill Triggs. Histograms of oriented gr adients for human detection. In
Proceedings of the 2005 IEEE Computer Society Conference on Computer V ision and P attern
Recognition (CVPR’05) - V olume 1 - V olume 01 , CVPR ’05, pages 886–893, 2005. ISBN
0-7695-2372-2.
[3] B. A. Olshausen and D. J. Field. Emergence of simple-cellreceptive ﬁeld properties by learning
a sparse code for natural images. Nature, 381(6583):607–609, June 1996. ISSN 0028-0836.
[4] L. Wiskott and T.J. Sejnowski. Slow feature analysis: Un supervised learning of invariances.
Neural computation , 14(4):715–770, 2002.
8Please refer to the supplementary material for more details.
9
[5] Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y . Ng. Convolutional deep belief
networks for scalable unsupervised learning of hierarchical representations. In Proceedings of
the 26th Annual International Conference on Machine Learni ng, ICML ’09, pages 609–616,
2009. ISBN 978-1-60558-516-1.
[6] K. Kavukcuoglu, P. Sermanet, Y .L. Boureau, K. Gregor, M. Mathieu, and Y . LeCun. Learn-
ing convolutional feature hierarchies for visual recognit ion. Advances in Neural Information
Processing Systems , pages 1090–1098, 2010.
[7] M.D. Zeiler, D. Krishnan, G.W. Taylor, and R. Fergus. Deconvolutional networks. InComputer
V ision and P attern Recognition (CVPR), 2010 IEEE Conferenc e on , pages 2528–2535. IEEE,
2010.
[8] P. Vincent, H. Larochelle, I. Lajoie, Y . Bengio, and P.A.Manzagol. Stacked denoising autoen-
coders: Learning useful representations in a deep network with a local denoising criterion. The
Journal of Machine Learning Research , 11:3371–3408, 2010.
[9] Rajesh P. N. Rao and Dana H. Ballard. Dynamic model of visu al recognition predicts neural
response properties in the visual cortex. Neural Computation , 9:721–763, 1997.
[10] Karl Friston. Hierarchical models in the brain. PLoS Comput Biol , 4(11):e1000211, 11 2008.
[11] Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. AFast Learning Algorithm for Deep
Belief Nets. Neural Comp. , (7):1527–1554, July .
[12] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise train-
ing of deep networks. In In NIPS , 2007.
[13] Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. Fast inference in sparse coding
algorithms with applications to object recognition. CoRR, abs/1010.3467, 2010.
[14] Yan Karklin and Michael S. Lewicki. A hierarchical baye sian model for learning nonlinear
statistical regularities in nonstationary natural signals. Neural Computation , 17:397–423, 2005.
[15] D. Angelosante, G.B. Giannakis, and E. Grossi. Compres sed sensing of time-varying signals.
In Digital Signal Processing, 2009 16th International Confer ence on , pages 1 –8, july 2009.
[16] A. Charles, M.S. Asif, J. Romberg, and C. Rozell. Sparsi ty penalties in dynamical system
estimation. In Information Sciences and Systems (CISS), 2011 45th Annual C onference on ,
pages 1 –6, march 2011.
[17] D. Sejdinovic, C. Andrieu, and R. Piechocki. Bayesian sequential compressed sensing in sparse
dynamical systems. In Communication, Control, and Computing (Allerton), 2010 48 th Annual
Allerton Conference on , pages 1730 –1736, 29 2010-oct. 1 2010. doi: 10.1109/ALLERT ON.
2010.5707125.
[18] X. Chen, Q. Lin, S. Kim, J.G. Carbonell, and E.P. Xing. Sm oothing proximal gradient method
for general structured sparse regression. The Annals of Applied Statistics , 6(2):719–752, 2012.
[19] Amir Beck and Marc Teboulle. A Fast Iterative Shrinkage -Thresholding Algorithm for Linear
Inverse Problems. SIAM Journal on Imaging Sciences , (1):183–202, March . ISSN 19364954.
doi: 10.1137/080716542.
[20] Y . Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming , 103
(1):127–152, 2005.
[21] Karol Gregor and Yann LeCun. Efﬁcient Learning of Sparse Invariant Representations. CoRR,
abs/1105.5307, 2011.
[22] Alex Nelson. Nonlinear estimation and modeling of noisy time-series by d ual Kalman ﬁltering
methods. PhD thesis, 2000.
[23] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howa rd, W. Hubbard, and L. D.
Jackel. Backpropagation applied to handwritten zip code re cognition. Neural Comput. , 1
(4):541–551, December 1989. ISSN 0899-7667. doi: 10.1162/ neco.1989.1.4.541. URL
http://dx.doi.org/10.1162/neco.1989.1.4.541.
[24] H. Zou and T. Hastie. Regularization and variable selec tion via the elastic net. Journal of the
Royal Statistical Society: Series B (Statistical Methodol ogy), 67(2):301–320, 2005.
10
A Supplementary material for Deep Predictive Coding Networ ks
A.1 From section 2.2.1, computing α ∗
The optimal solution of α in (6) is given by
α ∗ = arg max
∥α ∥∞ ≤1
[α T et − µ
2 ∥α ∥2]
= arg min
∥α ∥∞ ≤1


α − et
µ



2
=S
(et
µ
)
(15)
where S(.) is a function projecting
(
et
µ
)
onto an ℓ∞-ball. This is of the form:
S(x) =



x, −1 ≤ x ≤ 1
1, x > 1
−1, x < −1
A.2 Algorithm for joint inference of the states and the cause s.
Algorithm 1 Updating xt,ut simultaneously using FISTA-like procedure [19].
Require: Take Lx
0,n > 0 ∀n ∈ { 1, 2, ..., N }, Lu
0 > 0 and some η > 1.
1: Initialize x0,n ∈ RK ∀n ∈ { 1, 2, ..., N }, u0 ∈ RD and set ξ1 = u0, z1,n = x0,n .
2: Set step-size parameters: τ1 = 1 .
3: while no convergence do
4: Update
γ = γ0(1 + exp( −[Bui])/2
.
5: for n ∈ { 1, 2, ..., N } do
6: Line search: Find the best step size Lx
k,n .
7: Compute α ∗ from (15)
8: Update xi,n using the gradient from (9) with a soft-thresholding function.
9: Update internal variables zi+1 with step size parameter τi as in [19].
10: end for
11: Compute ∑ N
n=1 |xi,n |
12: Line search: Find the best step size Lu
k .
13: Update ui,n using the gradient of (10) with a soft-thresholding function.
14: Update internal variables ξi+1 with step size parameter τi as in [19].
15: Update
τi+1 =
(
1 +
√
(4τ2
i + 1)
)
/2
.
16: Check for convergence.
17: i = i + 1
18: end while
19: return xi,n ∀n ∈ { 1, 2, ..., N } and ui
11
A.3 Inferring sparse states with known parameters
20 40 60 80 1000
0.5
1
1.5
2
2.5
3
Observation Dimensions
steady state rMSE
 
 
Kalman Filter
Proposed
Sparse Coding [20]
Figure 5: Shows the performance of the inference algorithm w ith ﬁxed parameters when compared
with sparse coding and Kalman ﬁltering. For this we ﬁrst simu late a state sequence with only 20
non-zero elements in a 500-dimensional state vector evolvi ng with a permutation matrix, which is
different for every time instant, followed by a scaling matrix to generate a sequence of observations.
We consider that both the permutation and the scaling matrices are known apriori. The observation
noise is Gaussian zero mean and variance σ2 = 0 .01. We consider sparse state-transition noise,
which is simulated by choosing a subset of active elements in the state vector (number of elements
is chosen randomly via a Poisson distribution with mean 2) an d switching each of them with a
randomly chosen element (with uniform probability over the state vector). This resemble a sparse
innovation in the states. We use these generated observation sequences as inputs and use the apriori
know parameters to infer the states from the dynamic model. F igure 5 shows the results obtained,
where we compare the inferred states from different methodswith the true states in terms of relative
mean squared error (rMSE) (deﬁned as ∥xest
t − xtrue
t ∥/∥xtrue
t ∥). The steady state error (rMSE)
after 50 time instances is plotted versus with the dimension ality of the observation sequence. Each
point is obtained after averaging over 50 runs. We observe th at our model is able to converge to
the true solution even for low dimensional observation, when other methods like sparse coding fail.
We argue that the temporal dependencies considered in our model is able to drive the solution to the
right attractor basin, insulating it from instabilities typically associated with sparse coding [24].
12
A.4 Visualizing ﬁrst layer of the learned model
(a) Observation matrix (Bases)
Active state
element at (t-1)
Corresponding,  
predicted states at (t)
(b) State-transition matrix
Figure 6: Visualization of the parameters. C and A, of the model described in section 3.1. (A)
Shows the learned observation matrix C. Each square block indicates a column of the matrix,
reshaped as √p × √p pixel block. (B) Shows the state transition matrix A using its connections
strength with the observation matrix C. On the left are the basis corresponding to the single active
element in the state at time (t − 1) and on the right are the basis corresponding to the ﬁve most
“active” elements in the predicted state (ordered in decreasing order of the magnitude).
(a) Connections
 (b) Centers and Orientations (c) Orientations and Frequencies
Figure 7: Connections between the invariant units and the ba sis functions. (A) Shows the connec-
tions between the basis and columns of B. Each row indicates an invariant unit. Here the set of
basis that a strongly correlated to an invariant unit are sho wn, arranged in the decreasing order of
the magnitude. (B) Shows spatially localized grouping of the invariant units. Firstly, we ﬁt a Gabor
function to each of the basis functions. Each subplot here is then obtained by plotting a line indicat-
ing the center and the orientation of the Gabor function. Thecolors indicate the connections strength
with an invariant unit; red indicating stronger connection s and blue indicate almost zero strength.
We randomly select a subset of 25 invariant units here. We obs erve that the invariant unit group
the basis that are local in spatial centers and orientations . (C) Similarly, we show the correspond-
ing orientation and spatial frequency selectivity of the in variant units. Here each plot indicates the
orientation and frequency of each Gabor function color coded according to the connection strengths
with the invariant units. Each subplot is a half-polar plot with the orientation plotted along the angle
ranging from 0 to π and the distance from the center indicating the frequency. A gain, we observe
that the invariant units group the basis that have similar orientation.
13


Peer review:
[Review]:
A brief summary of the paper's contributions, in the context of prior work.
The paper proposes a hierarchical sparse generative model in the context of a dynamical system. The model can capture temporal dependencies in time-varying data, and top-down information (from high-level contextual/causal units) can modulate the states and observations in lower layers. 

Experiments were conducted on a natural video dataset, and on a synthetic video dataset with moving geometric shapes. On the natural video dataset, the learned receptive fields represent edge detectors in the first layer, and higher-level concepts such as corners and junctions in the second layer. In the synthetic sequence dataset, hierarchical top-down inference is used to robustly infer about “causal” units associated with object shapes.


An assessment of novelty and quality.
This work can be viewed as a novel extension of hierarchical sparse coding to temporal data. Specifically, it is interesting to see how to incorporate dynamical systems into sparse hierarchical models (that alternate between state units and causal units), and how the model can perform bottom-up/top-down inference. The use of Nestrov’s method to approximate the non-smooth state transition terms in equation 5 is interesting.

The clarity of the paper needs to be improved. For example, it will be helpful to motivate more clearly about the specific formulation of the model (also, see comments below). 

The experimental results (identifying high-level causes from corrupted temporal data) seem quite reasonable on the synthetic dataset. However, the results are all too qualitative. The empirical evaluation of the model could be strengthened by directly comparing the DPCN to related works on non-synthetic datasets.


Other questions and comments:
- In the beginning of the section 2.1, please define P, D, K to improve clarity.
- In section 2.2, little explanation about the pooling matrix B is given. Also, more explanations about equation 4 would be desirable.
- What is z_{t} in Equation 11?
- In Section 2.2, it’s not clear how u_hat is computed. 


A list of pros and cons (reasons to accept/reject).
Pros:
- The formulation and the proposed solution are technically interesting. 
- Experimental results on a synthetic video data set provide a proof-of-concept demonstration.

Cons:
- The significance of the experiments is quite limited. There is no empirical comparison to other models on real tasks.
- Inference seems to be complicated and computationally expensive. 
- Unclear presentation