Effect of top-down connections in Hierarchical Sparse Coding
Authors: Victor Boutin
Angelo Franciosini
Franck Ruffier
Laurent Perrinet
Venue:  (2020)
Keywords: Hierarchical Sparse Coding
Convolutional Sparse Coding
Top-down connections

Abstract: Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images. The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems. However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers. In this study, a new model called Sparse Deep Predictive Coding (SDPC) is introduced to assess the impact of this inter-layer feedback connection. In particular, the SDPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers. A 2-layered SDPC and a Hi-La networks are trained on 3 different databases and with different sparsity parameters on each layer. First, we show that the overall prediction error generated by SDPC is lower thanks to the feedback mechanism as it transfers prediction error between layers. Second, we demonstrate that the inference stage of the SDPC is faster to converge than for the Hi-La model. Third, we show that the SDPC also accelerates the learning process. Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the SDPC features are more generic and informative.

Under review as a conference paper at ICLR 2020
EFFECT OF TOP-DOWN CONNECTIONS IN HIERARCHICAL SPARSE CODING
Anonymous authors Paper under double-blind review
ABSTRACT
Hierarchical Sparse Coding (HSC) is powerful model to efficiently represent muuli-dimensional, structured data such as images. The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems. However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers. In this study, a new model called 2- Layers Sparse Predictive Coding (2L-SPC) is introduced assess the impact of this inter-layer feedback connection. In particular, the 2L-SPC compared with Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers. 2L-SPC and a 2-layers Hi-La networks are trained on 4 different databases and with different sparsity parameters each layer. First, prediction error generated by 2L-SPC is lower thanks to the feedback mechanism it transfers prediction error between layers. Second, we demonstrate that the inference stage of the 2L-SPC is faster to converge than for the Hi-La model. Third, we show that the 2L-SPC also accelerates the learning process. Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the 2L-SPC features are more generic and informative.
show that the overall
INTRODUCTION
Finding a "efficient" representation to model a given signal in a concise and efficient manner is an inverse problem that has always been central the machine learning community. Sparse Coding (SC) has proven to be one of the most successful methods to achieve this goal. SC holds the idea that signals (e.g. images) can be encoded as linear combination of few features (called atoms) drawn from a bigger set called the dictionary (Elad) 2010). The pursuit of optimal coding is usually decom- posed into two complementary subproblems: inference (coding) and dictionary learning. Inference consists in finding an accurate sparse representation of the input data considering the dictionaries are fixed, could performed using algorithms like ISTA & FISTA (Beck & Teboulle 2009), Match- ing Pursuit (Mallat & Zhang 1993). Coordinate Descent (Li & Osher 2009). or ADMM (Heide jet al.) 2015). Once the representation is inferred, one can learn the atoms from the data using methods like gradient descent (Rubinstein et al.) 2018), or online dictionary learning (Mairal et al.
2010 Kreutz-Delaado et al. 2003 Sulam et al. 2009a). Consequently, SC offers an unsupervised framework to learn simultaneously basis vectors (e.g. atoms) and the corresponding input repre- sentation. SC has been applied with success to image restoration (Mairal et al.] (2099)), feature extraction (Szlam et al.) 2010), and classification (Yang al. 2011 Perrinet & Bednar (215)). Inter- estingly, SC is also a field of interest for computational neuroscientists. Olshausen & Field (1977)
first demonstrated that adding a sparse prior to a shallow neural network was sufficient to account for the emergence of neurons whose Receptive Fields (RFs) are spatially localized, band-pass and oriented filters, analogous to those found in the primary visual cortex (V1) of mammals (Hubel & Wiesel 1962). Because most of the SC algorithms are limited to single-layer network, they cannot model the hierarchical structure of the visual cortex. However, few solutions have been proposed to tackle Hierarchical Sparse Coding (HSC) as a global optimization problem (Sulam et al.) 2018 Makhzani & Frey| 2013; 2015 Aberdam et al.) 2019 Sulam et al. 2019). These methods are looking for optimal solution of HSC without considering their plausibility in term of neuronal implementation. Consequently, the quest for reliable HSC formulation that is compatible with a neural implementation remains open.

---

Under review as a conference paper at ICLR 2020
Rao & Ballard (999) introduce the Predictive Coding (PC) to model the effect of the interaction of cortical areas in the visual cortex. PC intends to solve the inverse problem of vision by combining feedforward and feedback connections. In PC, feedback connection carries prediction of the neural activity of the lower cortical area while feedforward pass prediction error to the higher cortical area. In such framework, neural population are updated to minimize the unexpected component of the neural signal (Friston) 2010). PC has been applied for supervised object recognition (Wen et al.) 2018 Han et al. 2018 Spratling 2017) or unsupervised prediction of future video frames (Lotter et al., 2016). Interestingly. PC flexible enough to introduce a sparse prior to each layer. Therefore, one can consider PC as bio-plausible formulation of the HSC problem. This formulation is to confront with the other bio-plausible HSC formulation that consists of a stack of independent Lasso problems (Sun et al.) 2017). . To the best of our mnwleedge, no study has compared these two mathe- matically different formulations of the same problem of optimizing the Hierarchical Sparse Coding of images. What is the effect of top-down connection of PC? What are the consequences in term of computations and convergence? What are the qualitative differences concerning the learned atoms?
The objective of this study is to experimentally answer these questions and to show that the PC framework could be successfully used for improving solutions to HSC problems. We start study by defining the two different mathematical formulations to solve the HSC problem: the Hierarchical Lasso (Hi-La) that consists in stacking Lasso sub-problems, and the 2-Layers Sparse Predictive Coding (2L-SPC) that leverages PC into a deep and sparse network of bi-directionally connected layers. To experimentally compare both models, we train the 2L-SPC and Hi-La networks on different databases and we vary the sparsity of each layer. First, we compare the overall prediction error of the two models and we break it down to understand its distribution among layers. Second, we analyze the number of iterations needed for the state variables of each network to reach their stability. Third, we compare the convergence of both models during the dictionary learning stage. Finally, we discuss the qualitative differences between the features learned by both networks in light of their activation probability.
METHODS
2.1 BACKGROUND
In our mathematical description, italic letters are used as symbols for scalars, bold lower case letters for column vectors, bold uppercase letters for MATRICES and v.L denotes the gradient of L w.r.t. to r. The core objective of Hierarchical Sparse Coding (HSC) is to infer the internal state variables {7/ Ji= (also called sparse map) for each input image ((k)) and to learn the parameters ( that solved the inverse problem formulated in Eq. |
x (k) DJ (A)
(()
s.t. 117 (1) an and
y (R) 0
7
71
(k) D (k)
+ 62
(k)
s.t. 1172
( 1lo < 02
and
72 > 0
(k)
Vh-1 = DE (8)
(1)
+ +
(k)
s.t. ll
(1)) l1o < az and Yi.
(k) 0
L is the number of layers and c(*) is the layer prediction-eree error corresponding to the input x: (4). The map 7: could be viewed as the projection of 701 in the basis described by the atoms (denoted also
basis vector) composing D., e(() is historically called pprdictioo error' but it is actually quantifying the local reconstruction error between DJ (k) and (k) The sparsity of the internal state variables, specified by the lo noru,o-norm, is constrained by the scalar O;- In practice we use 4-dimensional
tensors to represent both vectors and matrices. (*(k) is a tensor of size [1, Cz. Way ha] with e₂ being the number of channels of the image, wa and hr the width and height of the image respectively. In our mathematical description we raveled (A) as a vector of size (Ca x Wr x hz]. Furthermore, we impose 2-dimensional convolutional structure to the parameters {D.) the dictionary as a 4-dimensional tensor of size [na, Cd, Wa, fug), one can derive D,
one describes matrix of nh local features (size: Ca Wa X ha) that cover every possible location of the input (*) (Sulam et al. 2018). In other words, D; is Toeplitz matrix. For the sake of concision in our mathematical descriptions, we matrix/vector multiplication in place of convolution it is mathematically strictly equivalent. Replaced in a biological context D; could be interpreted as the synaptic weights between neural populations whose activity is represented by 7 and 7 respectively.
2

---

Under review as a conference paper at ICLR 2020
2.2 FROM HIERARCHICAL LASSO...
One possibility to solve Eq. [q.[wwiii while keeping the locality of the processing required by neural imple- mentation, is to minimize loss for each layer corresponding the addition of the squared fo-nrrm of the prediction error with a sparsity penalty. To obtain a convex cost, we relax the to constraint into a f1 E1-penalty. defines, therefore, a loss function for each layer the form of standard Lasso problem (Eq. 2) that could be minimized using gradient- gradient-based methods:
F F(D 7: (0)) - Il ( DJ 7 /h 1 1h
In particular, we use the Iterative Shrinkage Thresholding Algorithm (ISTA) to minimize F w.r.t Yi (Eq. 3) as it proven to be computationally cheap (Beck & Teboulle 2009). In practice, we use an accelerated version of the ISTA algorithm called FISTA. In convolutional case in which the proximal operator has a closed-form, FISTA has the advantage to converge faster than other sparse coding algorithms (e.g. Coordinate Descent) (Chalasani et al. 2013). Note that in Eq. have removed image indexation to keep concise notation.
7: [+1 = (7) Me, V F) = (7) + ne, -D,"))
(3)
In Eq. 3 (-( denotes the non-negative soft-thresholding operator, 1)c, is the learning rate of the inference process and Y is the state variable 7r at time f. Interestingly, one can interpret Eq. 3 as one doop of a recurrent layer that we will call the Lasso layer (Gregor & LeCun (010). Following Eq. D is a decoding dictionary that back-projects 7 into the space of the (i 1)-th layer. This back nack-projection is used to elicit an error with Y that will be encoded by D; to update the state variables %:- Finally, Lasso layers can be stacked together to form a Hierarchical Lasso (Hi-La) network (see Fig. without blue arrow). The inference of the overall Hi-La network consists in updating recursively all the sparse maps until they have reached a stable point.
Inter-Layer Feedback
connections (Only for SDPC)
7
72
Foodforward connections
Internal Feedback connections
D
D
Input Image
SPC Layer
SPC Layer
Figure 1: Inference update scheme for the 2L-SPC network. I is the input image, and A; tunes the sparseness level of Y: The encoding and decoding dictionaries (D; and D respectively) are reciprocal. The sparse maps (7)) are updated through a bi-directional dynamical process (plain and empty arrows). This recursive process alone describes the Hi-La network. If we add the top-down influence, called inter-layer feedback connection (blue arrow), it then becomes a SP-SPC network.
2.3 TO HIERARCHICAL. PREDICTIVE CODING
Another alternative to solve Eq. is to use the Predictive Coding (PC) theory. Unlike the Lasso loss function, PC is not only minimizing the bottom-up prediction error, but it also adds top-down prediction error that takes into consideration the influence of the upper-layer on the current layer (see Eq. 4.. In other words, finding the 7 that minimizes consists finding a trade-off between representation that best predicts the lower level activity and another one that is best predicted by the upper-layer.
1 (k) DD. (k) c(D.,> (1)) = =F(D,, x0)) + 1,
i+1 7--111
(4)
3

---

Under review as a conference paper at ICLR 2020
For consistency, also the ISTA algorithm to minimize L w.r.t %v- The update scheme is described in Eq. (without image indexation for concision):
7tt1 Tri, A. (ni Te, x) T (7) + ne. Di x.) De, (of D ((+))) (5)
Fig. shows how we can interpret this update scheme loop of recurrent layer. This recurrent layer, called Sparse Predictive Coding (SPC) layer, forms the building block of the 2-Layers Sparse Predictive Coding (2L-SPC) network (see Algorithm 2 in Appendix for the detailed implementa- tion of the 2L-SPC inference). The only difference with the Hi-La architecture is that the 2L-SPC includes inter-layer feedback connection materialize the influence coming from upper-layers (see the blue arrow in Fig.
2.4 CODING STOPPING CRITERION AND UNSUPERVISED LEARNING
For both networks, the inference process is finalized once the relative variation of 7 w.r.t to 7/-1 is below a threshold denoted Tatab. In practice, the number of iterations needed to reach the stopping criterion between 30 100 (see Fig for more details). Once the convergence achieved, we update the dictionaries using gradient descent (see Algorithm. D It was demonstrated by [Sulam] al. (2018) that this alternation of inference and learning offers reasonable convergence guarantee. The learning of both Hi-La and 2L-SPC consists in minimizing the problem defined in Eq. 0 in which is the number of images in the dataset. The learning occurs during the training phase only. Conversely, the inference process is the same during both training and testing phases.
min
(8) F(D:, Yi
(6)
k=1==1
For both models. dictionaries are randomly initialized using the standard normal distribution (mean 0 and variance 1) and all the sparse maps are initialized to zero at the beginning of the inference process. After every dictionary update, we fg-normalize each atom of the dictionary avoid any redundant solution. Interestingly, although the inference update scheme is different for the two models, the dictionary learning loss is the same in both cases since the top-down prediction error term in L does not depend on D; (see Eq. 6 This loss is then a good evaluation point to assess the impact both 2L-SPC and Hi-La inference process on the layer prediction error 6, used PyTorch 1.0 to implement, train, and test all the models described above. The code of the two models and the simulations of this paper are available at www. ggtphbbbcomXXXXXXXXXXX
Algorithm 1: Alternation of inference and learning for training and testing
for (k) in the training set do # initialization
Vi, Vi.75 O while convergence not reached do for i to L do
(k)
Tmc (1r
Ne, V. v,(
# inference
for
do
D. t D, 1O V C
learning (only for training)
EXPERIMENTAL SETTINGS: DATASETS AND PARAMETERS
We use 4 different databases to train and test both networks.
STL-110 The STL-10 database (Coates et al. 2011 is made of 100000 colored images of size 96 x 96 pixels (px) representing 10 classes of objects (airplane, bird..... STL-10 presents high diversity of objects view-points and background. This set is partitioned into a training set composed of 90000 images, and testing set of 10000 images.
A The AT&T database (ATT 1994) is made of 400 grayscale images of size 92 x 112 pixels (px) representing faces of 40 distinct subjects with different lighting conditions, facial expressions, and details. This set is partitioned into batches of 20 images. The training set composed of 330 images (33 subjects) and the testing set id composed of 70 images (7 subjects).
4

---

Under review as a conference paper at ICLR 2020
CFD. The Chicago Face Database (CFD) (Ma al. 2015) consists of 1, 804 high-resolution (2,444 1, 718 px), color, standardized photographs of male and female faces of varying ethnicity between the ages of 18 and 40 years. We re-sized the pictures 170 120 px keep reasonable computational time. The CFD database is partitioned into batches of 10 images. This dataset is split into a training set composed of 721 images and testing of 486 images.
MNIST. MNIST (LeCun 1998) is composed of 28 x 28 px, 70, 000 grayscale images representing handwritten digits. We decomposed this dataset into batches of 32 images. This dataset is split into a training set composed of 60, 000 digits and testing set of 10, 000 digits.
All these databases are pre-processed using Local Contrast Normalization (LCN) and whitening. LCN is inspired by neuroscience and consists in a local subtractive and divisive normalization (Jar-
rett et al., 2009). In addition, we use whitening to reduce dependency between pixels.
To draw fair comparison between the 2L-SPC and Hi-La models, we train both models using the
same set of parameters. All these parameters are summarized in Table | for STL-10, MNIST and CFD databases and in Appendix C for ATT database. Note that the parameter 7g is omitted in the table because it is computed as the inverse of the largest eigenvalue of D.D, (Beck & Teboulle, 2009). To learn the dictionary D;, we use stochastic gradient descent on the training set only, with learning rate V[, and a momentum equal to 0.9. this study, we consider only 2-layered networks and we vary the sparsity parameters of each layer (A1 and A2) to assess their effect on both 2L-SPC and Hi-La networks.
Table 1: Network architectures, training and simulation parameters. The size of the convolutional kernels are shown in the format: [# features, channels, width, height] (stride). To describe the range of explored parameters during simulations, use the format [0.3 0.7 0.1, 0.5] which means that we vary d from 0.3 to 0.7 by step of 0.1 while X2 is fixed to 0.5.
DataBases
STL-10
CFD
MNIST
Dj size
[64, I, 8, 8] (2)
[64, 3, 9, 9] (3)
[32, 1, 5, 5] (2) [64, 32, 5, 5] (1)
network param.
D2 size
[128, 64, 8) (1)
[128, 64, 9, 9] (1)
Tetab
Ic-4
5e-3
5e-4
epochs
10
250
100
training
le-4
1e-4
5e-2
param.
l]
5e-3
5e-3
1c-3
simu.
A range
[0.2 : 0.6 : 0.1, 1, 1.6]
[0.3 0.7 :: 0.1, 1.8]
[0.1 : 0.3 :: 0.05, 0.3]
param.
X2 range
[0.4, 1.4 1.8 0.1]
[0.5, 1 1.8 :: 0.2]
[0.2, 0.2 0.4 0.05]
4 RESULTS
times all the simulations presented in this section, each time with a different random dictionary initialization. We define the central tendency of our curves by the median of the runs, and its variation by the Median Absolute Deviation (MAD) (Pham-Gia Hung 2001). We prefer this measure to the classical mean + standard deviation because a few measures did not exhibit normal distribution. All presented curved are obtained on the testing set.
For cross-validation,
4.1 2L-SPC CONVERGES TO A LOWER PREDICTION ERROR
As a first analysis we report the global prediction error, as computed as the sum of the prediction error (c:) over all layers (see Fi..3). and its decomposition among layer (see Fig Fig-2land Appendix [ fro AT&T database) for different value of sparsity. For scaling reasons, and because the error bars small we cannot display them on Fig. 2. we thus include them in Appendix Fig. 7 For all the
simulations shown in Fig. 2 and Fig. 3 we observed that the global prediction error is lower for the 2P-SPC than for the Hi-La model. As expected, in both models the prediction errors increase similarly when we increase X or A2. For all databases and sparsity parameters, Fig, [ shows that the first layer prediction error of the 21. is always higher, and the second layer prediction error
5

---

Under review as a conference paper at ICLR 2020
(a)
Distribution of the prediction error among layers when varying Å1-
STSTL10
C)-CFD
MNST
pro0kdio etre
prodicon 0rodidionn enor
2000
pred
0.3
0.4
0.5
0.6
0.3
0.4
0.5
0.6
0.7
24
Layert SCPC
(b) Distribution of the prediction error among layers when varying λ2-
LayerO HiLa
Layer HiLa
CF
8000
4500
120
predition e/o
prediction eney
4000
3000
80
rrediction
2000
1500
0
1.5
1.7
0.20
0.75
0.90
0.35
0.40
Figure 2: Evolution of all layers prediction error, evaluated on the testing set, for both 2L-SPC and Hi-La networks and trained on STL-10, CFD and MNIST databases. We vary the first layer sparsity in the top 3 graphs (a) and the second layer sparsity in the bottom graphs (b).
always lower than the corresponding Hi-La prediction error. This is expected: while the Hi-La first layer is fully specialized in minimizing the prediction error with the lower level, the 2L-SPC finds a trade-off between lower and higher level prediction errors. addition, when A₁ increased, the Hi-La first layer prediction error increases faster (+ 139% for STL-10, +105% for CFD, +157% for MNIST and +933 for AT&T) than the 2L-SPC first layer prediction error (+103% for STL- 10, +90% for CFD, +117% for MNIST and 76% for AT&T), This suggests that part of the penalty induced by the increase of A is absorbed by the second layer in the case of the 2L-SPC This supported by the fact that the second layer prediction error of the 2L-SPC is in general increasing slightly faster than the Hi-La second layer error. When X2 is increased, the prediction error of the first layer of the Hi-La model not varying whereas the 2L-SPC first layer prediction error is slightly increasing (+1% for STL-10, +3% for CFD, +7.1% for MNIST and +3.3% for AT&T), The explanation here is straightforward: while the first-layer loss of the 2L-SPC includes the influence of the upper-layer, the Hi-La doesn't 't have such mechanism. It suggests that the inter-layer feedback connection of the 2L-SPC transfers a part of the extra- extra-penalty coming from the when
increase of λ in the first layer. Fig and ii) show the mapping of the global prediction we vary the sparsity of each layer for the 2L-SPC and Hi-La, respectively. These heatmaps confirm what has been observed in Fig 2|nd extend it larger range of sparsity values: both models' losses are more sensitive to a variation of X than to a change in A2. Fig 3 jii) is a heatmap of the relative difference between the 2L-SPC and the Hi-La global losses. It shows that the minimum relative difference between 2L-SPC and Hi-La (10.6%) (0.))))))))) is reached when A₁ is maximal and A2 is minimal, and the maximum relative difference (19.99%) suggests that the previously observed mitigation mechanism originated by the feedback connection more efficient when the sparsity of the first layer is lower. All these observations point in the same direction: the 2L-SPC framework mitigates the global prediction eπror thanks to a better distribution of the prediction error among layers. This mechanism is even more pronounced when the sparsity of the first layer is lower. Surprisingly, while the inter-layer feedback connection of the 2L-SPC imposes more constraints on the state variables, it also happens to generate less global prediction error.
reached when both X and X2 are minimal.
4.2 SP-SPC HAS A FASTER INFERENCE PROCESS
One may wonder if this lower prediction error is not achieved at the cost a slower inference process. To address this concern, we report for both models the number of iterations needed by the

---

Under review as a conference paper at ICLR 2020
() SDPC loss when varying layer's
[i] Hi-La loss when varying layer's sparsity
III) - SDPCC Hi-La rolative loss difference when varying layer's sparsity
sparsity
1.8
1.8
Joor iesptpa leooj0
20%
Aelative loos difference
18%
1.6
3500
1.6
1.6
16%
1.4
1.4
< 1.4
3000
3000
14%
1.2
2500
1.2
2500
12%
1.0
0.3
0.4
0.5
0.6 0.7
2000
1.0
0.3
0.4
0.5
0.6
0.7
2000
0.3 0.4
0.5
0.6 0.7
10%
A1
A1
Figure 3: Heatmaps of the global loss enror when varying layers' sparsity for 2L-SPC (i) and Hi-La (ii) on CFD database. (iii) shows the heatmap the relative difference between the Hi-La and the 2L-SPC global losses when varying the layers' sparsity
inference process to converge towards a stable state on the testing set. Fig. shows the evolution of this quantity, for STL-10, 10, CFD and MNIST databases (see Appendix E for AT&T database), when varying both layers' sparsity. For all the simulations, the 2L-SPC needs less iteration than the Hi-La model to converge towards stable state. We also observe that the data dispersion is, in general, more pronounced for the Hi-La model. addition to converging to lower prediction error, the 2L- SPC is also decreasing the number of iterations in the inference process to converge towards stable state.
(a) Number of iterations of the inference when varying À STL10
CFO
230
140
140
namber of karadens
number of toratonn
torations
200
330
70
70
100
25
35
0.2
0.3
0.4
0.5
0.8
0.3
0.4
0.5
0.6
0.7
0.10
0.15
0.20
0.25
0:30
SDPC
(b) Number of iterations of the inference when varying A2
Hi-La
i1 STL10
cro
ii) NNIST
rumber darrrtttnss
number of Raratooss
205
105
110
70
70
80
50
Az
Figure 4: Evolution of the number of iterations needed reach stability criterium for both 2L- and Hi-La networks on the testing set of STL-10, CFD and MNIST databases. We vary the first layer sparsity in the top 3 graphs (a) and the second layer sparsity in the bottom 3 graphs (b). Shaded areas correspond to mean absolute deviation on runs. Sometimes the dispersion is so small that it looks like there is shade.
4.3 2L-SPC LEARNS FASTER
shows the evolution of the global prediction error during the dictionary learning stage and evaluated on the testing set (see Appendix. Foro AT&T database). For the all databases, the 2P-SPC model reaches its minimal prediction error before the Hi-La model. The convergence rate both models is comparable, but the 2L-SPP has much lower prediction error in the very first epochs.
Fig.

---

Under review as a conference paper at ICLR 2020
The inter-layer feedback connection of the 2L-SPC pushes the network towards lower prediction error since the very beginning of the learning.
glbbi prediction crnor
10
glcet gloel predicioo eroo
3x10
globel predictin
2x
1x10' 8k10'
3x.10"
307
10"
307
10
107
30'
10
307
10'
10*
opochs
opochs
SDPC
Hi-La
Figure 5: Evolution of the global prediction error during the training evaluated on the STL-10, 10, CFD and MNIST testing sets. Shaded areas correspond to mean absolute deviation 7 runs. All graphs have a logarithmic scale both x and y-axis.
4.4 QUALITATIVE ANALYSIS OF THE FEATURES
2L-SPP
Hi-La
1s layer features
2nd layer
features
0.05
0.05
Activation of 2nd layer atoms
0.04
acteatter 118g
0.04
0.0
0.01
0.00
64
128
0.00
64
128
Figure 6: Hi-La and 21-SPP RFs obtained on the CFD database (with X 0.3, X2 1.8) and their associated second layer activation probability histogram. The first and second layer RFs have a size of x 9 and 33 x 33 px respectively. For the first layer RFs, we randomly selected 12 out of 64 atoms. For the second layer RFs, we sub-sampled 32 out 128 atoms ranked by their activation probability in descending order. For readability, we removed the most activated filter (RF framed black) in 2L-SPC and Hi-La second layer activation histogram. The activation probability of the RFs framed in red shown red bar in the corresponding histogram.
Another way grasp the impact of the inter-layer feedback connection is visualize its effect on the dictionaries. To make human-readable visualizations of the learned dictionaries, we back- pooj-etoecct them into the image space using cascade of transposed convolution (see Appendix Fig IT.. These back-projection are called Receptive Fields (RFs). Fig. O shows some of the RFs of the 2 layers and the second layer activation probability histogram for both models when they are trained on the CFD database. In general, first layer RFs are oriented Gabor-like filters, and second layer RF' are more specific and represent more abstract concepts (curvatures, eyes, mouth, nose...). Second layer RFs present longer curvatures in the 2L-SPC than in the Hi-La model: they cover bigger part of the input image, and include more contextual and informative details. some extreme cases, the Hi-La second layer RFs are over-fitted to specific faces and do not describe the generality of the
8

---

Under review as a conference paper at ICLR 2020
concept of face. The red-framed RFs highlights one of these cases: the corresponding activation probabilities are 0.25% and 0.69% for Hi-La and 2L-SPC respectively. This is supported by the fact that the lowest activation probability of the second layer's atoms is higher for the 2L-SPC than for the Hi-laa (0.30% versus 0.16%). This phenomenon is even more striking when we sort all the features by activation probabilities in descending order (see Appendix Figures [3]. We filter the highest activation probability (corresponding to the low-frequency filters highlighted by black square) of both Hi-La and 2L-SPC keep good readability of the histograms. All the filters are displayed in Appendix Fig. I2 Fig. [3] Fig. 14 and Fig. T5 for STL-10, CFD, MNIST and AT&T RFs respectively. The atoms' activation probability confirm the qualitative analysis of the RFs: the features learned by the 2L-SPC are more generic and informative as they describe a wider range of images.
CONCLUSION
What are the computational advantages of inter-layer feedback connections in hierarchical sparse coding algorithms? We answered this question by comparing the Hierarchical Lasso (Hi-La) and the 2-Layers Sparse Predictive Coding (2L-SPC) models. Both are identical in every respect, except that the 2L-SPC brings inter-layer feedback connections. This extra-connection forces the inter- nal state variables of the 2L-SPC to converge toward a trade-off between on one hand an accurate prediction passed by the lower-layer and on the other hand facilitated predictability by the upper- layer. Experimentally, we demonstrated on 4 different databases and for 2-layered network that the inter-layer feedback top-down connection (i) mitigates the overall prediction error by distributing among layers, (ii) accelerates the convergence towards a stable internal state and (iii) accelerates the learning process. Besides, we qualitatively observed that top-down connections bring contextual information that helps to extract more informative and less over-fitted features.
The 2L-SPC holds the novelty to consider Hierarchical Sparse Coding as combination of local sub-prohe cub-problems that are tightly related. This crucial difference with CNNs that are trained by back- propagating gradients from a global loss. To the best of our knowledge the 2L-SPC is the first one that leverage local sparse coding into a hierarchical and unsupervised algorithms (the MICCSC from (Sulam
2018) is equivalent to one layer sparse coding algorithm (Aberdam al. 2019). and the ML-ISTA from (Sulam al. 2019 is trained using supervised learning).
Moreover, even if our results are robust as they hold for different databases and with a large spectrum of first and second layer sparsity, further work will be conducted to generalize our results to deeper networks and different sparse coding algorithms such as Coordinate Descent or ADMM. Further studies will show that our SP- framework could be used for practical applications like image inpainting, denoising, or image super-resolution.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Aviad Aberdam, Jeremias Sulam, and Michael Elad. Multi-layer sparse coding: The holistic way. SIAM Journal on Mathematics of Data Science, 1(1):46-77, 2019.
ATT. Face Database, AT&T Laboratories, Cambridge, 1994.
Amir Beck and Mare Teboulle. A fast iterative thiekoage algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.
Rakesh Chalasani, Jose C Principe, and Naveen Ramakrishnan. A fast proximal method for con- volutional sparse coding. In Neural Networks (JJCNN,, The 2013 International Joins Conference on, pp. 1-5. IEEE, 2013.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-22522222 2011.
Michael Elad. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. Springer, New York, 2010 edition edition, August 2010. ISBN 978-1 1-419- 7010-7. 00000.
Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, (2):177,
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings the 27th International Conference on International Conference on Machine Learning, pp. 399-406, Omnipress, 2010.
Kuan Han, Haiguang Wen, Yizhen Zhang, Di Fu, Eugenio Culurciello, and Zhongming Liu. Deep predictive coding network with local recurrent processing for object recognition. Advances in Neural Information Processing Systems, pp. 9221-9233, 2018.
Felix Heide, Wolfgang Heidrich, and Gordon Wetzstein. Fast and flexible convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 5135-5143, 2015.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional archi- tecture in the cat's visual cortex. The Journal of physiology, 160(1):106-1555) 1962.
Kevin Jarrett, Koray Kavukcuoglu, Yann LeCun, et al. What is the best multi-stage architecture for object recognition? In Computer Vision, 2009 IEEE 12th International Conference on, pp- 2146-2153. IEEF, 2009.
Kenneth Kreutz Delgado, Joseph F Murray, Bhaskar D Rao, Kjersti Engan, Te- Ron Lee, and Ter- rence J Sejnowski. Dictionary learning algorithms for sparse representation. Neural computation, 15(2):349-396 2003.
Yann LeCun. The MNIST database of handwritten digits. http://yann. lecun. comfexdb/mmist// 1998.
Yingying Li and Stanley Osher. Coordinate descent optimization for 11 minimization with applica- tion to compressed sensing; greedy algorithm. Inverse Problems and Imaging, 3(3):487-503, 2009.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning, arXiv preprint 1005v:160..88/11 2016.
Debbie Ma, Joshua Correll, and Bernd Wittenbrink. The chicago face database: free stimulus set faces and norming data. Behavior research methods, 47(4): 122-11222.... 2015.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In Proceedings of the 26th anmual international conference on machine learning, pp. 689-696. ACM, 2009a.
10

---

Under review as a conference paper at ICLR 2020
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Non-local sparse models for image restoration. 2009 IEEE 12th International Conference on Computer Vision (ICCV), pp. 2272-2277 2279. IEEE, 2009b.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprini arXiv: 2013.
Alireza Makhzani and Brendan Frey. Winner-take-all autoencoders. In Advances in neural infor- mation processing systems, pp. 2791-2792 2015.
Stéphane Mallat and Zhifeng Zhang. Matching pursuit with time- fine-freyenney dictionaries. Technical report, Courant Institute of Mathematical Sciences New York United States, 1993.
Bruno A Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):331133325 1997.
Laurent Perrinet and James A Bednar. Edge co-oceurrences can account for rapid categorization of natural versus animal images. Scientific Reports. 5:10400, 2015. doi: 10. 10.1/e8/s 1400. URL mnttp:/wwwe nature...e..... www.eottaoleeeeeeeeoooooooeeeeee wtepiiieeesseepel10e//eep
Pham-Gia and TL Hung. The mean and median absolute deviations. Mathematical and Computer Modelllnng, 34(7-8):921-99) 1229999,,,222 2001.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpreta- tion of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79, 1999.
Ron Rubinstein, Alfred M Bruckstein, and Michael Elad. Dictionaries for sparse representation modeling. Proceedings of the IEFF, 1455-10555555 2010.
Michael W Spratling. A hierarchical predictive coding model of object recognition in natural images. Cognitive computation, 9()):111-677, 2017.
Jeremias Sulam, Vardan Papyan, Yaniv Romano, and Michael Elad. Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Transactions on Signal Processing, 66(15); 4090-4104, 2018.
Jeremias Sulam, Aviad Aberdam, Amir Beck, and Michael Elad. On multi-layer basis pursuit, efficient algorithms and convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 2019.
Xiaoxia Sun, Nasser M. Nasrabadi, and Trac D. Tran. Supervised multilayer sparse coding networks for image classification. CoRR, abs/1701 088349 2017. URL http://axxiv.rrg//
17C1.08349
Arthur Szlam, Koray Kavukcuoglu, and Yann LeCun. Convolutional matching pursuit and dictio- nary training, arXiv preprint 101.::10 2010.
Haiguang Wen, Kuan Han, Junxing Shi, Yizhen Zhang, Fugenio Culurciello, and Zhongming Liu. Deep predictive coding network for object recognition. arXiv preprint arXiv: 1002.0//02.0476) 2018.
Meng Yang, Lei Zhang, Jian Yang, and David Zhang. Robust sparse coding for face recognition. In CVPR 2011, pp. 625-632. IEBE, 2011.

---

Under review as a conference paper at ICLR 2020
A 2L-SPC PSEUDO CODE
Algorithm 2: 2L-SPC inference algorithm
input image: æ, dictionaries: (D:)ez Ei-penalty parameters: (Me stability threshold: Tttb
% = (T--==- 0, (mm, Ji=1 ## Initializing layer state variables and FISTA momentum
a 1# Initializing momentum strength
lle maz(eigen_value (D) D,))
Stable False Initializing the stability criterion
while Stable = False do
++
2+1 1+ I 4(at)²
2
for i 1o L do
# +Update lower-layer error D. Y'..
T77 ,
# Update the upper-layer enror if * L then
GUL Mi, D Mm
else
EUL =0
7 = Tne, ( + nc, DIELL - -7,, EUL) Update layer state variables
a² -1
n = To (3t +
@ -
# Update the FISTA momentum
|1 - -//21/2
if
Tatas) then
|| l2
Stable = True Update the stability creterion
return b
Note: (-) denotes the element-wise non-negative soft-thresholding operator. A fortiori, To(-) is rectified linear unit operator, comments are comments.
12

---

Under review as a conference paper at ICLR 2020
B EVOLUTION OF THE GLOBAL PREDICTION ERROR WITH ERROR BAR
(a) Global prediction error when varying Å1-
S-STL1O
CFD
@l0bal prodictoo enor
200cal prediction eror
5000
3800
110
predection
4000
1200
70
2000
0.5
2000
0.5
50
0.10
0.35
0.20
0.25
0.30
SDPC
(b) Global prediction error when varying X2-
Hi-La
5200
II STL110
4400
130
II) MNIST
@lbba predicion arer
globa predicton arer
X200
3200
90
3700
2000
144
1.5
1.5
1.7
1.0
1.0
1.4
1.6
50 0.25
0.35
0.40
Figure 7: Evolution of the global prediction error evaluated on the testing set for both 2L-SPC and Hi-La networks. We vary the first layer sparsity in the top 3 graphs (a) and the second layer sparsity in the bottom 3 graphs (b). Experiments have been conducted on STL-10, CFD and MNIST- databases. Shaded areas correspond to mean absolute deviation on runs. Sometimes the dispersion is small that it looks like there is no shade.
C 2L-SPC PARAMETERS ON ATT
Table Network architectures, training and simulation parameters on AT&T database. The size of the convolutional kernlls are shown in the format: [# features, # channels, width, height] (stride). To describe the range of explored parameters during simulations, we use the format [0.3 0.7 0.1, 0.5]. which means that we vary A from 0.3 to 0.7 by step of 0.1 while X2 is fixed to 0.5.
ATT Database
Di size
[64, 1, 9] (3)
network param.
D2 size
[128, 64, 9, 9] (1)
Tstab
5e-4
epochs
1000
training
le-4
param.
5e-3
simu.
A1 range
[0.3 0.7 :: 01
param.
Ag range
[0.5, 0.6::.6::0 0.21
13

---

Under review as a conference paper at ICLR 2020
D PREDICTION ERRORS DISTRIBUTION ON AT&T
(a) Distribution of the prediction error among layers when varying A1. AT&T
(b) Distribution of the prediction error among layers when varying X2.
AT&T
2000
2000
prediction error
1500
prediction
1000
500
LayerO SDPC
SDPC
0.3
0.4
0.5
0.6
0.7
LayerO
0.6
0.8
1.2 1.4
1.6
Layer1 Hila
Figure 8: Evolution of all layers prediction error, evaluated on the testing set, for both 2L-SPC and Hi-La networks and trained on AT&T. We vary the first layer sparsity in (a) and the second layer sparsity in (b).
E NUMBER OF ITERATION OF THE INFERENCE ON AT&T
(a) Number of iterations of the inference when varying A1-
(b) Number of iterations of the inference
when varying A2.
140
140
number iterations
number of iterations
105
105
70
35
35
SDPC
0.3
0.4
0.5
0.6
0.7
0.6
0.8
1.0
1.2
1.4
1.6
Hi-La
Figure 9: Evolution of the number of iterations needed reach stability criterium for both 2L. and Hi-La networks on the AT&T testing set. We vary the first layer sparsity in (a) and the second layer sparsity in (b). Shaded areas correspond mean absolute deviation on runs. Sometimes the dispersion so small that it looks like there is no shade.
14

---

Under review as a conference paper at ICLR 2020
F EVOLUTION OF PREDICTION ERROR DURING TRAINING
AT&T
4x
global prediction error
1x10
SDPC
Hi-La
5 10
2x
10f
10
102
10
Figure 10: Evolution of the global prediction error during the training for the ATT testing set. Shaded areas correspond to mean absolute deviation on runs. The graph have logarithmic scale both and y-axis.
G ILLUSTRATION OF THE BACK-PROJECTION MECHANISM
Dictionary (D+)
Dictionary (D,)
Effective dictionary (D
Figure 11: Generation of the second-layer effective dictionary. The result of this back proj-prooection called effective dictionary and could be assimilate to the notion of preferred stimulus in neuroscience. In general case, the effective dictionary at layer is computed as follow: D. D D'' DJ' SSlam et al. 2018).
15

---

Under review as a conference paper at ICLR 2020
H 2L-SPC AND HI-LA RFS ON STL10
2L-SPC first layer RFs
(b) Hi-La first layer RFs
2L-SPC second layer RFs
(d) Hi-La second layer RFs
Figure 12: 2L-SPC (a & e) and Hi-La (b & d) effective dictionaries obtained the STL-10 database, with sparsity parameter: (x1=0.5.\2=1). All other parameters are those described in Ta- ble for the STL-10 database. Atoms sorted activation probabilities in descending order. The visualization shown here is the projection of the dictionaries into the input space. First layer effective dictionaries have size of 8 px & b) and second layer RFs have size of 22 x 22 (c & d) px respectively.
16

---

Under review as a conference paper at ICLR 2020
I 2L-SPC AND HI-LA RFS ON CFD
(a) 2L-SPC first layer RFs
(b) Hi-La first layer RFs
2L-SPC second layer RFs
(d) Hi-La second layer RFs
Figure 13: 2L-SPC (a & e) and Hi-La (b & d) effective dictionaries obtained the CFD database, with sparsity parameter: (x1=0.3.2211..
All other parameters are those described in Table for the CFD database. Atoms sorted by activation probabilities in descending order. The visualization shown here is the projection of the dictionaries into the input space. First layer effective dictionaries have size of px (a & and second layer RFs have of 33 33 & d) px respectively.
17

---

Under review as a conference paper at ICLR 2020
J 2L-SPC AND HI-LA RFS ON MNIST
2L-SPC first layer RFs
Hi-La first layer RFs
CP-
second layer RFs
+
(d) Hi-La second layer RFs
Figure 14: 2L-SPC (a & c) and Hi-La (b & d) effective dictionaries obtained on the MNIST database, with sparsity parameter: (2x=0.2.);=0.3), Atoms are sorted by activation probabilities in descending order. All other parameters are those deseribed in Table the MNIST database. The visualization shown here is the projection the dictionaries into the input space. First layer effective dictionaries have size of 5 x px (a & b) and second layer RFs have size of 14 x 14 (c & d) px respectively.
18

---

Under review as a conference paper at ICLR 2020
K 2L-SPC AND HI-LA RFS ON AT&T
(a) SP- first layer RFs
(b) Hi-La first layer RFs
(c) 2L-SPC second layer RFs
(d) Hi-La second layer RFs
Figure 15: 2L-SPC (a & e) and Hi-La (b & d) effective dictionaries obtained on the AT&T database, with sparsity parameter: (0,=0.5.A:=1) All other parameters are those described Table 2 for the AT&T database Atoms are sorted by activation probabilities in descending order. The visualization shown here is the projection the dictionaries into the input space. First layer effective dictionaries have size of 9 x 9 px (a & b) and second layer RFs have size of 26 x 26 (c & d) px respectively.
19


Reviews:
[{"review_id": "rygVHXratH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Sparse Deep Predictive Coding (SDPC) to access the impact of the inter layer feedback, which is suggested by neuro-scientific evidence. The SDPC model is compared with HILA on 2 different databases, and the experimental results show that SDPC achieved lower prediction error, faster converge rate. \n\nAlthough the paper is well-motivated and well-written, the potential impact of this work is limited in the community. Moreover, the experiments are not sufficient to valid the advantage of this model. For example, there are only 3 dataset adopted in the experiments and the all these three datasets are quite small. Moreover, there is only one baseline, which seems too weak to show the advantage of the model.\n\nBased on the above points, I tend to reject the paper.\n\n**updated comments**\nThanks for the response and I really appreciate the hard work during the rebuttal. However, the STL-10 is still a small scale dataset, and the baselines provided in the rebuttal are still limited. I will not change the original score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyxh8GFTtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a study that compares two techniques for Hierarchical Sparse Coding. This is the problem of learning sparse representations in multi-layer (but not necessarily deep) models. The first method applies Lasso at each layer in a bottom up fashion, while the second method — introduced in the submitted work — adds an additional term which propagates information in a top-down manner. It is found that the top-down term is beneficial in terms of reducing predictive error and can learn faster. In terms of novelty the new term, the paper does not make a breakthrough contribution, but I consider this to be sufficient. Moreover, the paper is well written and presents some interesting results. However I am not entirely sure that the impact of these results will attract the interest of a broad audience since the experiments presented are on “small datasets” and with some rather shallow neural nets. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJg036X15S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed a new model called Sparse Deep Predictive Coding, which introduced top-down connections between consecutive layers, to improve the solutions to hierarchical sparse coding (HSC) problems. Instead of decomposing the HSC problem into independent subproblems, the proposed model added a new term to the loss function, which represents the influence of the latter-layer on the current layer.\n\n#Pros:\n-- The proposed model adopted the idea from predictive coding and came up with a relatively novel idea for HSC problems.\n-- The experiments are solid. The experiments evaluated the proposed methods with different hyper-parameter settings and three real-world datasets.\n-- The figures in the result sections are well designed and concise.\n\n#Cons:\n-- The mathematical description of the main problem and the proposed model is not clear. For example, the dimensionality for the variables in Eq.(1) is not clarified.\n-- The test procedure is not clear. How the internal state variables are obtained for the test set is not clarified.\n-- The proposed model was only compared with a basic Hierarchical Lasso network. There are not any state-of-art methods included as baseline methods.\n\n#Detailed comments:\n\n(1) The proposed model is named as sparse DEEP predictive coding, however, the experiments only considered the SDPC and Hi-La networks with 2 layers. I am wondering if a deeper structure will improve the performance?\n\n(2) For the structure shown in Fig.1, the decoding dictionaries are $D^T_i$, but I am confused why the encoding dictionaries are reciprocal to encoding dictionaries. Does it come from the optimization updates shown in Eq.(3)?\n\n(3) According to Eq.(1), $x$ is a vector and $D$ is a 2d matrix. However, the real inputs in the experiments are images and $D$ is a convolutional filter with 4 dimensions. How are the matrices reshaped?\n\n(4) For section 2.2 and 2.3, the number/index of samples is not shown in the loss function for training. The loss should be over the whole training set. Besides, the test procedure is not clarified.\n\n(4) The number of iterations using FISTA of SDPC and Hi-La networks is shown to compare the rate of convergence. However, considering both models are solving a lasso-type regression problem, I would suggest using coordinate descent for optimization.\n\n(5) For the main result of prediction error, why is the “global prediction error” more important than the reconstruction error? Is the first-layer prediction error the reconstruction error? If yes, Fig.2 shows that Hi-La has a lower prediction error compared to SDPC for the first layer.\n\n(6) Two minor comments on writing:\n(a) It would be better to have a separate section for 2.5 since it describes the dataset and is not related to the proposed model.\n(b) A typo of “neuronal implementation” exists in the introduction section. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new architecture for sparse coding.

The reviewers gave long and constructive feedback that the authors in turn responded at length on. There is consensus among the reviewers that despite contributions this paper in its current form is not ready for acceptance.

Rejection is therefore recommended with encouragement to make updated version for next conference.  


Decision: Reject