Title: EFFECT OF TOP-DOWN CONNECTIONS IN HIERARCHICAL SPARSE CODING Anonymous

Abstract: Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images. The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems. However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers. In this study, a new model called 2-Layers Sparse Predictive Coding (2L-SPC) is introduced to assess the impact of this inter-layer feedback connection. In particular, the 2L-SPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers. 2L-SPC and a 2-layers Hi-La networks are trained on 4 different databases and with different sparsity parameters on each layer. First, we show that the overall prediction error generated by 2L-SPC is lower thanks to the feedback mechanism as it transfers prediction error between layers. Second, we demonstrate that the inference stage of the 2L-SPC is faster to converge than for the Hi-La model. Third, we show that the 2L-SPC also accelerates the learning process. Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the 2L-SPC features are more generic and informative.

1. INTRODUCTION:

Finding a "efficient" representation to model a given signal in a concise and efficient manner is an inverse problem that has always been central to the machine learning community. Sparse Coding (SC) has proven to be one of the most successful methods to achieve this goal. SC holds the idea that signals (e.g. images) can be encoded as a linear combination of few features (called atoms) drawn from a bigger set called the dictionary (Elad, 2010) . The pursuit of optimal coding is usually decomposed into two complementary subproblems: inference (coding) and dictionary learning. Inference consists in finding an accurate sparse representation of the input data considering the dictionaries are fixed, it could be performed using algorithms like ISTA & FISTA (Beck & Teboulle, 2009) , Matching Pursuit (Mallat & Zhang, 1993) , Coordinate Descent (Li & Osher, 2009) , or ADMM (Heide et al., 2015) . Once the representation is inferred, one can learn the atoms from the data using methods like gradient descent (Rubinstein et al., 2010; Kreutz-Delgado et al., 2003; Sulam et al., 2018) , or online dictionary learning (Mairal et al., 2009a) . Consequently, SC offers an unsupervised framework to learn simultaneously basis vectors (e.g. atoms) and the corresponding input representation. SC has been applied with success to image restoration (Mairal et al., 2009b) , feature extraction (Szlam et al., 2010) and classification (Yang et al., 2011; Perrinet & Bednar, 2015) . Interestingly, SC is also a field of interest for computational neuroscientists. Olshausen & Field (1997) first demonstrated that adding a sparse prior to a shallow neural network was sufficient to account for the emergence of neurons whose Receptive Fields (RFs) are spatially localized, band-pass and oriented filters, analogous to those found in the primary visual cortex (V1) of mammals (Hubel & Wiesel, 1962) . Because most of the SC algorithms are limited to single-layer network, they cannot model the hierarchical structure of the visual cortex. However, few solutions have been proposed to tackle Hierarchical Sparse Coding (HSC) as a global optimization problem (Sulam et al., 2018; Makhzani & Frey, 2013; 2015; Aberdam et al., 2019; Sulam et al., 2019) . These methods are looking for an optimal solution of HSC without considering their plausibility in term of neuronal implementation. Consequently, the quest for reliable HSC formulation that is compatible with a neural implementation remains open. Rao & Ballard (1999) introduce the Predictive Coding (PC) to model the effect of the interaction of cortical areas in the visual cortex. PC intends to solve the inverse problem of vision by combining feedforward and feedback connections. In PC, feedback connection carries prediction of the neural activity of the lower cortical area while feedforward pass prediction error to the higher cortical area. In such a framework, neural population are updated to minimize the unexpected component of the neural signal (Friston, 2010) . PC has been applied for supervised object recognition (Wen et al., 2018; Han et al., 2018; Spratling, 2017) or unsupervised prediction of future video frames (Lotter et al., 2016) . Interestingly, PC is flexible enough to introduce a sparse prior to each layer. Therefore, one can consider PC as a bio-plausible formulation of the HSC problem. This formulation is to confront with the other bio-plausible HSC formulation that consists of a stack of independent Lasso problems (Sun et al., 2017) . To the best of our knowledge, no study has compared these two mathematically different formulations of the same problem of optimizing the Hierarchical Sparse Coding of images. What is the effect of top-down connection of PC? What are the consequences in term of computations and convergence? What are the qualitative differences concerning the learned atoms?

The objective of this study is to experimentally answer these questions and to show that the PC framework could be successfully used for improving solutions to HSC problems. We start our study by defining the two different mathematical formulations to solve the HSC problem: the Hierarchical Lasso (Hi-La) that consists in stacking Lasso sub-problems, and the 2-Layers Sparse Predictive Coding (2L-SPC) that leverages PC into a deep and sparse network of bi-directionally connected layers. To experimentally compare both models, we train the 2L-SPC and Hi-La networks on 4 different databases and we vary the sparsity of each layer. First, we compare the overall prediction error of the two models and we break it down to understand its distribution among layers. Second, we analyze the number of iterations needed for the state variables of each network to reach their stability. Third, we compare the convergence of both models during the dictionary learning stage. Finally, we discuss the qualitative differences between the features learned by both networks in light of their activation probability.

2. METHODS:



2.1. BACKGROUND:

In our mathematical description, italic letters are used as symbols for scalars, bold lower case letters for column vectors, bold uppercase letters for MATRICES and ∇ x L denotes the gradient of L w.r.t. to x. The core objective of Hierarchical Sparse Coding (HSC) is to infer the internal state variables {γ

(k) i } L

i=1 (also called sparse map) for each input image x (k) and to learn the parameters {D i } L i=1 that solved the inverse problem formulated in Eq. 1.

EQUATION

L is the number of layers and

(k)

i is the layer i prediction-error corresponding to the input x (k) . The map γ (k) i could be viewed as the projection of γ (k) i-1 in the basis described by the atoms (denoted also basis vector) composing D i .

(k):

i is historically called 'prediction error' but it is actually quantifying the local reconstruction error between D T i γ

(k) i and γ

(k)

i-1 . The sparsity of the internal state variables, specified by the 0 pseudo-norm, is constrained by the scalar α i . In practice we use 4-dimensional tensors to represent both vectors and matrices. x (k) is a tensor of size [1, c x , w x , h x ] with c x being the number of channels of the image, w x and h x the width and height of the image respectively. In our mathematical description we raveled x (k) as a vector of size [c x × w x × h x ]. Furthermore, we impose a 2-dimensional convolutional structure to the parameters {D i } L i=1 . If one describes the dictionary as a 4-dimensional tensor of size [n d , c d , w d , h d ], one can derive D i as a matrix of n d local features (size: c d × w d × h d ) that cover every possible location of the input x (k) (Sulam et al., 2018) . In other words, D i is a Toeplitz matrix. For the sake of concision in our mathematical descriptions, we use matrix/vector multiplication in place of convolution as it is mathematically strictly equivalent. Replaced in a biological context D i could be interpreted as the synaptic weights between two neural populations whose activity is represented by γ i-1 and γ i respectively.

2.2 FROM HIERARCHICAL LASSO ... One possibility to solve Eq. 1 while keeping the locality of the processing required by neural implementation, is to minimize a loss for each layer corresponding to the addition of the squared 2 -norm of the prediction error with a sparsity penalty. To obtain a convex cost, we relax the 0 constraint into a 1 -penalty. It defines, therefore, a loss function for each layer in the form of a standard Lasso problem (Eq. 2), that could be minimized using gradient-based methods:

EQUATION

In particular, we use the Iterative Shrinkage Thresholding Algorithm (ISTA) to minimize F w.r.t. γ

(k) i

(Eq. 3) as it is proven to be computationally cheap (Beck & Teboulle, 2009) . In practice, we use an accelerated version of the ISTA algorithm called FISTA. In a convolutional case in which the proximal operator has a closed-form, FISTA has the advantage to converge faster than other sparse coding algorithms (e.g. Coordinate Descent) (Chalasani et al., 2013) . Note that in Eq. 3 we have removed image indexation to keep a concise notation.

EQUATION

In Eq. 3, T α (•) denotes the non-negative soft-thresholding operator, η ci is the learning rate of the inference process and γ t i is the state variable γ i at time t. Interestingly, one can interpret Eq. 3 as one loop of a recurrent layer that we will call the Lasso layer (Gregor & LeCun, 2010) . Following Eq. 3, D T i is a decoding dictionary that back-projects γ i into the space of the (i -1)-th layer. This back-projection is used to elicit an error with γ i-1 that will be encoded by D i to update the state variables γ i . Finally, Lasso layers can be stacked together to form a Hierarchical Lasso (Hi-La) network (see Fig. 1 without blue arrow). The inference of the overall Hi-La network consists in updating recursively all the sparse maps until they have reached a stable point.

Figure 1 : Inference update scheme for the 2L-SPC network. x is the input image, and λ i tunes the sparseness level of γ i . The encoding and decoding dictionaries (D i and D T i , respectively) are reciprocal. The sparse maps (γ i ) are updated through a bi-directional dynamical process (plain and empty arrows). This recursive process alone describes the Hi-La network. If we add the top-down influence, called inter-layer feedback connection (blue arrow), it then becomes a 2L-SPC network.

2.3. ... TO HIERARCHICAL PREDICTIVE CODING:

Another alternative to solve Eq. 1 is to use the Predictive Coding (PC) theory. Unlike the Lasso loss function, PC is not only minimizing the bottom-up prediction error, but it also adds a top-down prediction error that takes into consideration the influence of the upper-layer on the current layer (see Eq. 4). In other words, finding the γ i that minimizes L consists in finding a trade-off between a representation that best predicts the lower level activity and another one that is best predicted by the upper-layer.

L D i , γ (k) i = F D i , γ (k) i + 1 2 γ (k) i -D T i+1 γ (k) i+1 2 2 (4)

For consistency, we also use the ISTA algorithm to minimize L w.r.t γ i . The update scheme is described in Eq. 5 (without image indexation for concision):

γ t+1 i = T ηc i λi γ t i -η ci ∇ γ t i L = T ηc i λi γ t i + η ci D i (γ t i-1 -D T i γ t i ) -η ci (γ t i -D T i+1 γ t i+1 )

(5) Fig. 1 shows how we can interpret this update scheme as a loop of a recurrent layer. This recurrent layer, called Sparse Predictive Coding (SPC) layer, forms the building block of the 2-Layers Sparse Predictive Coding (2L-SPC) network (see Algorithm 2 in Appendix for the detailed implementation of the 2L-SPC inference). The only difference with the Hi-La architecture is that the 2L-SPC includes an inter-layer feedback connection to materialize the influence coming from upper-layers (see the blue arrow in Fig. 1 ).

2.4. CODING STOPPING CRITERION AND UNSUPERVISED LEARNING:

For both networks, the inference process is finalized once the relative variation of γ t i w.r.t to γ t-1 i is below a threshold denoted T stab . In practice, the number of iterations needed to reach the stopping criterion is between 30 to 100 (see Fig. 4 for more details). Once the convergence is achieved, we update the dictionaries using gradient descent (see Algorithm. 1). It was demonstrated by Sulam et al. (2018) that this alternation of inference and learning offers reasonable convergence guarantee. The learning of both Hi-La and 2L-SPC consists in minimizing the problem defined in Eq. 6 in which N is the number of images in the dataset. The learning occurs during the training phase only. Conversely, the inference process is the same during both training and testing phases.

EQUATION

For both models, dictionaries are randomly initialized using the standard normal distribution (mean 0 and variance 1) and all the sparse maps are initialized to zero at the beginning of the inference process. After every dictionary update, we 2 -normalize each atom of the dictionary to avoid any redundant solution. Interestingly, although the inference update scheme is different for the two models, the dictionary learning loss is the same in both cases since the top-down prediction error term in L does not depend on D i (see Eq. 6). This loss is then a good evaluation point to assess the impact of both 2L-SPC and Hi-La inference process on the layer prediction error i . We used PyTorch 1.0 to implement, train, and test all the models described above. The code of the two models and the simulations of this paper are available at www.github.com/XXX/XXX. Algorithm 1: Alternation of inference and learning for training and testing CFD. The Chicago Face Database (CFD) (Ma et al., 2015) consists of 1, 804 high-resolution (2, 444 × 1, 718 px), color, standardized photographs of male and female faces of varying ethnicity between the ages of 18 and 40 years. We re-sized the pictures to 170 × 120 px to keep reasonable computational time. The CFD database is partitioned into batches of 10 images. This dataset is split into a training set composed of 721 images and a testing set of 486 images.

for x (k) in the training set do ∀i, γ (k) i = 0 # initialization while convergence not reached do for i = 1 to L do γ (k) i ← T ηc i λi γ (k) i -η ci ∇ γ (k) i L # inference for i = 1 to L do D i ← D i -η i ∇ Di L # learning (

MNIST. MNIST (LeCun, 1998) is composed of 28 × 28 px, 70, 000 grayscale images representing handwritten digits. We decomposed this dataset into batches of 32 images. This dataset is split into a training set composed of 60, 000 digits and a testing set of 10, 000 digits.

All these databases are pre-processed using Local Contrast Normalization (LCN) and whitening.

LCN is inspired by neuroscience and consists in a local subtractive and divisive normalization (Jarrett et al., 2009) . In addition, we use whitening to reduce dependency between pixels.

To draw a fair comparison between the 2L-SPC and Hi-La models, we train both models using the same set of parameters. All these parameters are summarized in Table 1 for STL-10, MNIST and CFD databases and in Appendix C for ATT database. Note that the parameter η ci is omitted in the table because it is computed as the inverse of the largest eigenvalue of D T i D i (Beck & Teboulle, 2009) . To learn the dictionary D i , we use stochastic gradient descent on the training set only, with a learning rate η Li and a momentum equal to 0.9. In this study, we consider only 2-layered networks and we vary the sparsity parameters of each layer (λ 1 and λ 2 ) to assess their effect on both 2L-SPC and Hi-La networks. 

4. RESULTS:

For cross-validation, we run 7 times all the simulations presented in this section, each time with a different random dictionary initialization. We define the central tendency of our curves by the median of the runs, and its variation by the Median Absolute Deviation (MAD) (Pham-Gia & Hung, 2001) . We prefer this measure to the classical mean ± standard deviation because a few measures did not exhibit a normal distribution. All presented curved are obtained on the testing set.

4.1. 2L-SPC CONVERGES TO A LOWER PREDICTION ERROR:

As a first analysis we report the global prediction error, as computed as the sum of the prediction error ( i ) over all layers (see Fig. 3 ), and its decomposition among layer (see Fig. 2 and Appendix D fro AT&T database) for different value of sparsity. For scaling reasons, and because the error bars are small we cannot display them on Fig. 2 , we thus include them in Appendix Fig. 7 . For all the simulations shown in Fig. 2 and Fig. 3 , we observed that the global prediction error is lower for the 2L-SPC than for the Hi-La model. As expected, in both models the prediction errors increase similarly when we increase λ 1 or λ 2 . For all databases and sparsity parameters, Fig. 2 shows that the first layer prediction error of the 2L-SPC is always higher, and the second layer prediction error is always lower than the corresponding Hi-La prediction error. This is expected: while the Hi-La first layer is fully specialized in minimizing the prediction error with the lower level, the 2L-SPC finds a trade-off between lower and higher level prediction errors. In addition, when λ 1 is increased, the Hi-La first layer prediction error increases faster (+139% for STL-10, +105% for CFD, +157% for MNIST and +93% for AT&T ) than the 2L-SPC first layer prediction error (+103% for STL-10, +90% for CFD, +117% for MNIST and +76% for AT&T). This suggests that a part of the penalty induced by the increase of λ 1 is absorbed by the second layer in the case of the 2L-SPC. This is supported by the fact that the second layer prediction error of the 2L-SPC is in general increasing slightly faster than the Hi-La second layer error. When λ 2 is increased, the prediction error of the first layer of the Hi-La model is not varying whereas the 2L-SPC first layer prediction error is slightly increasing (+1% for STL-10, +3% for CFD, +7.1% for MNIST and +3.3% for AT&T). The explanation here is straightforward: while the first-layer loss of the 2L-SPC includes the influence of the upper-layer, the Hi-La doesn't have such a mechanism. It suggests that the inter-layer feedback connection of the 2L-SPC transfers a part of the extra-penalty coming from the increase of λ 2 in the first layer. Fig. 3 i) and ii) show the mapping of the global prediction error when we vary the sparsity of each layer for the 2L-SPC and Hi-La, respectively. These heatmaps confirm what has been observed in Fig. 2 and extend it to a larger range of sparsity values: both models' losses are more sensitive to a variation of λ 1 than to a change in λ 2 . Fig. 3 iii) is a heatmap of the relative difference between the 2L-SPC and the Hi-La global losses. It shows that the minimum relative difference between 2L-SPC and Hi-La (10.6%) is reached when λ 1 is maximal and λ 2 is minimal, and the maximum relative difference (19.99%) is reached when both λ 1 and λ 2 are minimal. It suggests that the previously observed mitigation mechanism originated by the feedback connection is more efficient when the sparsity of the first layer is lower. All these observations point in the same direction: the 2L-SPC framework mitigates the global prediction error thanks to a better distribution of the prediction error among layers. This mechanism is even more pronounced when the sparsity of the first layer is lower. Surprisingly, while the inter-layer feedback connection of the 2L-SPC imposes more constraints on the state variables, it also happens to generate less global prediction error.

4.2. 2L-SPC HAS A FASTER INFERENCE PROCESS:

One may wonder if this lower prediction error is not achieved at the cost of a slower inference process. To address this concern, we report for both models the number of iterations needed by the inference process to converge towards a stable state on the testing set. Fig. 4 shows the evolution of this quantity, for STL-10, CFD and MNIST databases (see Appendix E for AT&T database), when varying both layers' sparsity. For all the simulations, the 2L-SPC needs less iteration than the Hi-La model to converge towards a stable state. We also observe that the data dispersion is, in general, more pronounced for the Hi-La model. In addition to converging to lower prediction error, the 2L-SPC is also decreasing the number of iterations in the inference process to converge towards a stable state. . Shaded areas correspond to mean absolute deviation on 7 runs. Sometimes the dispersion is so small that it looks like there is no shade.

4.3. 2L-SPC LEARNS FASTER:

Fig. 5 shows the evolution of the global prediction error during the dictionary learning stage and evaluated on the testing set (see Appendix. F for AT&T database). For the all databases, the 2L-SPC model reaches its minimal prediction error before the Hi-La model. The convergence rate of both models is comparable, but the 2L-SPC has a much lower prediction error in the very first epochs.

The inter-layer feedback connection of the 2L-SPC pushes the network towards lower prediction error since the very beginning of the learning. Another way to grasp the impact of the inter-layer feedback connection is to visualize its effect on the dictionaries. To make human-readable visualizations of the learned dictionaries, we back-project them into the image space using a cascade of transposed convolution (see Appendix Fig. 11 ). These back-projection are called Receptive Fields (RFs). Fig. 6 shows some of the RFs of the 2 layers and the second layer activation probability histogram for both models when they are trained on the CFD database. In general, first layer RFs are oriented Gabor-like filters, and second layer RFs are more specific and represent more abstract concepts (curvatures, eyes, mouth, nose...). Second layer RFs present longer curvatures in the 2L-SPC than in the Hi-La model: they cover a bigger part of the input image, and include more contextual and informative details. In some extreme cases, the Hi-La second layer RFs are over-fitted to specific faces and do not describe the generality of the concept of face. The red-framed RFs highlights one of these cases: the corresponding activation probabilities are 0.25% and 0.69% for Hi-La and 2L-SPC respectively. This is supported by the fact that the lowest activation probability of the second layer's atoms is higher for the 2L-SPC than for the Hi-La (0.30% versus 0.16%). This phenomenon is even more striking when we sort all the features by activation probabilities in descending order (see Appendix Figures 13). We filter out the highest activation probability (corresponding to the low-frequency filters highlighted by black square) of both Hi-La and 2L-SPC to keep good readability of the histograms. All the filters are displayed in Appendix Fig. 12 , Fig. 13 , Fig. 14 and Fig. 15 , for STL-10, CFD, MNIST and AT&T RFs respectively. The atoms' activation probability confirm the qualitative analysis of the RFs: the features learned by the 2L-SPC are more generic and informative as they describe a wider range of images.

5. CONCLUSION:

What are the computational advantages of inter-layer feedback connections in hierarchical sparse coding algorithms? We answered this question by comparing the Hierarchical Lasso (Hi-La) and the 2-Layers Sparse Predictive Coding (2L-SPC) models. Both are identical in every respect, except that the 2L-SPC brings inter-layer feedback connections. This extra-connection forces the internal state variables of the 2L-SPC to converge toward a trade-off between on one hand an accurate prediction passed by the lower-layer and on the other hand a facilitated predictability by the upperlayer. Experimentally, we demonstrated on 4 different databases and for a 2-layered network that the inter-layer feedback top-down connection (i) mitigates the overall prediction error by distributing it among layers, (ii) accelerates the convergence towards a stable internal state and (iii) accelerates the learning process. Besides, we qualitatively observed that top-down connections bring contextual information that helps to extract more informative and less over-fitted features.

The 2L-SPC holds the novelty to consider Hierarchical Sparse Coding as a combination of local sub-problems that are tightly related. This a crucial difference with CNNs that are trained by backpropagating gradients from a global loss. To the best of our knowledge the 2L-SPC is the first one that leverage local sparse coding into a hierarchical and unsupervised algorithms (the ML-CSC from (Sulam et al., 2018) is equivalent to a one layer sparse coding algorithm (Aberdam et al., 2019) , and the ML-ISTA from (Sulam et al., 2019) is trained using supervised learning).

Moreover, even if our results are robust as they hold for 4 different databases and with a large spectrum of first and second layer sparsity, further work will be conducted to generalize our results to deeper networks and different sparse coding algorithms such as Coordinate Descent or ADMM. Further studies will show that our 2L-SPC framework could be used for practical applications like image inpainting, denoising, or image super-resolution.

A 2L-SPC PSEUDO CODE:

Algorithm 2: 2L-SPC inference algorithm input : image: x, dictionaries: {D i } L i=1 , 1 -penalty parameters: {λ i } L i=1 , stability threshold: 

T stab γ t 0 = x {γ 0 i } L i=1 = 0, {γ 1 mi } L i=1 = 0 # Initializing layer state variables and FISTA momentum α 1 = 1 # Initializing momentum strength η ci = 1 max(eigen value(D T i D i )) Stable = F alse # Initializing the stability criterion t = 0 while Stable == F alse do t += 1 α t+1 = 1 + 1 + 4(α t ) 2 2 for i = 1 to L do # Update lower-layer error LL = γ t mi-1 -D T i γ t mi # Update the upper-layer error if i = L then U L = γ t mi -D T i+1 γ t mi+1 else U L = 0 γ t i = T ηc i λi γ t mi + η ci D i LL -η ci U L # Update layer state variables γ t+1 mi = T 0 γ t i + α t -1 α t+1 γ t i -γ t-1 i # Update the FISTA momentum if L i=1 γ t i -γ t-1 i 2 γ t i 2 < T stab then Stable = T

:



Figures and Tables:

only for training) 3 EXPERIMENTAL SETTINGS: DATASETS AND PARAMETERS We use 4 different databases to train and test both networks.STL-10. The STL-10 database(Coates et al., 2011) is made of 100000 colored images of size 96 × 96 pixels (px) representing 10 classes of objects (airplane, bird...). STL-10 presents a high diversity of objects view-points and background. This set is partitioned into a training set composed of 90000 images, and a testing set of 10000 images.AT&T. The AT&T database(ATT, 1994)  is made of 400 grayscale images of size 92 × 112 pixels (px) representing faces of 40 distinct subjects with different lighting conditions, facial expressions, and details. This set is partitioned into batches of 20 images. The training set composed of 330 images (33 subjects) and the testing set id composed of 70 images (7 subjects).

Distribution of the prediction error among layers when varying λ 1 . (b) Distribution of the prediction error among layers when varying λ 2 .

Figure 2: Evolution of all layers prediction error, evaluated on the testing set, for both 2L-SPC and Hi-La networks and trained on STL-10, CFD and MNIST databases. We vary the first layer sparsity in the top 3 graphs (a) and the second layer sparsity in the bottom 3 graphs (b).

Figure 3: Heatmaps of the global loss error when varying layers' sparsity for 2L-SPC (i) and Hi-La (ii) on CFD database. (iii) shows the heatmap of the relative difference between the Hi-La and the 2L-SPC global losses when varying the layers' sparsity

Figure4: Evolution of the number of iterations needed to reach stability criterium for both 2L-SPC and Hi-La networks on the testing set of STL-10, CFD and MNIST databases. We vary the first layer sparsity in the top 3 graphs (a) and the second layer sparsity in the bottom 3 graphs (b). Shaded areas correspond to mean absolute deviation on 7 runs. Sometimes the dispersion is so small that it looks like there is no shade.

Figure 5: Evolution of the global prediction error during the training evaluated on the STL-10, CFD and MNIST testing sets. Shaded areas correspond to mean absolute deviation on 7 runs. All graphs have a logarithmic scale in both x and y-axis.

rue # Update the stability creterion return {γ t i } L i=1 Note: Tα(•) denotes the element-wise non-negative soft-thresholding operator. A fortiori, T0(•) is a rectified linear unit operator. # comments are comments. B EVOLUTION OF THE GLOBAL PREDICTION ERROR WITH ERROR BAR (a) Global prediction error when varying λ 1 . (b) Global prediction error when varying λ 2 .

Figure7: Evolution of the global prediction error evaluated on the testing set for both 2L-SPC and Hi-La networks. We vary the first layer sparsity in the top 3 graphs (a) and the second layer sparsity in the bottom 3 graphs (b). Experiments have been conducted on STL-10, CFD and MNISTdatabases. Shaded areas correspond to mean absolute deviation on 7 runs. Sometimes the dispersion is so small that it looks like there is no shade.

PREDICTION ERRORS DISTRIBUTION ON AT&T (a) Distribution of the prediction error among layers when varying λ 1 . (b) Distribution of the prediction error among layers when varying λ 2 .

Figure 8: Evolution of all layers prediction error, evaluated on the testing set, for both 2L-SPC and Hi-La networks and trained on AT&T. We vary the first layer sparsity in (a) and the second layer sparsity in (b).

Figure9: Evolution of the number of iterations needed to reach stability criterium for both 2L-SPC and Hi-La networks on the AT&T testing set. We vary the first layer sparsity in (a) and the second layer sparsity in (b). Shaded areas correspond to mean absolute deviation on 7 runs. Sometimes the dispersion is so small that it looks like there is no shade.

Network architectures, training and simulation parameters. The size of the convolutional kernels are shown in the format: [# features, # channels, width, height] (stride). To describe the range of explored parameters during simulations, we use the format [0.3 : 0.7 :: 0.1, 0.5] ,which means that we vary λ 1 from 0.3 to 0.7 by step of 0.1 while λ 2 is fixed to 0.5.

Table content: <table><tr><td/><td/><td/><td>DataBases</td><td/></tr><tr><td/><td/><td>STL-10</td><td>CFD</td><td>MNIST</td></tr><tr><td>network param.</td><td>D 1 size D 2 size T stab</td><td>[64, 1, 8, 8] (2) [128, 64, 8, 8] (1) 1e-4</td><td>[64, 3, 9, 9] (3) [128, 64, 9, 9] (1) 5e-3</td><td>[32, 1, 5, 5] (2) [64, 32, 5, 5] (1) 5e-4</td></tr><tr><td>training param.</td><td># epochs η L1 η L2</td><td>10 1e-4 5e-3</td><td>250 1e-4 5e-3</td><td>100 5e-2 1e-3</td></tr><tr><td>simu.</td><td>λ 1 range</td><td>[0.2 : 0.6 :: 0.1, 1.6]</td><td>[0.3 : 0.7 :: 0.1, 1.8]</td><td>[0.1 : 0.3 :: 0.05, 0.3]</td></tr><tr><td>param.</td><td>λ 2 range</td><td>[0.4, 1.4 : 1.8 :: 0.1]</td><td>[0.5, 1 : 1.8 :: 0.2]</td><td>[0.2, 0.2 : 0.4 :: 0.05]</td></tr></table>

Network architectures, training and simulation parameters on AT&T database. The size of the convolutional kernels are shown in the format: [# features, # channels, width, height] (stride). To describe the range of explored parameters during simulations, we use the format [0.3 : 0.7 :: 0.1, 0.5] ,which means that we vary λ 1 from 0.3 to 0.7 by step of 0.1 while λ 2 is fixed to 0.5.

Table content: <table><tr><td/><td/><td>ATT Database</td></tr><tr><td>network param.</td><td>D 1 size D 2 size T stab</td><td>[64, 1, 9, 9] (3) [128, 64, 9, 9] (1) 5e-4</td></tr><tr><td>training param.</td><td># epochs η L1 η L2</td><td>1000 1e-4 5e-3</td></tr><tr><td>simu.</td><td>λ 1 range</td><td>[0.3 : 0.7 :: 0.1, 1]</td></tr><tr><td>param.</td><td>λ 2 range</td><td>[0.5, 0.6 : 1.6 :: 0.2]</td></tr></table>

Peer review (iclr2020):
This work proposed a new model called Sparse Deep Predictive Coding, which introduced top-down connections between consecutive layers, to improve the solutions to hierarchical sparse coding (HSC) problems. Instead of decomposing the HSC problem into independent subproblems, the proposed model added a new term to the loss function, which represents the influence of the latter-layer on the current layer. #Pros: -- The proposed model adopted the idea from predictive coding and came up with a relatively novel idea for HSC problems. -- The experiments are solid. The experiments evaluated the proposed methods with different hyper-parameter settings and three real-world datasets. -- The figures in the result sections are well designed and concise. #Cons: -- The mathematical description of the main problem and the proposed model is not clear. For example, the dimensionality for the variables in Eq.(1) is not clarified. -- The test procedure is not clear. How the internal state variables are obtained for the test set is not clarified. -- The proposed model was only compared with a basic Hierarchical Lasso network. There are not any state-of-art methods included as baseline methods. #Detailed comments: (1) The proposed model is named as sparse DEEP predictive coding, however, the experiments only considered the SDPC and Hi-La networks with 2 layers. I am wondering if a deeper structure will improve the performance? (2) For the structure shown in Fig.1, the decoding dictionaries are , but I am confused why the encoding dictionaries are reciprocal to encoding dictionaries. Does it come from the optimization updates shown in Eq.(3)? (3) According to Eq.(1), is a vector and is a 2d matrix. However, the real inputs in the experiments are images and is a convolutional filter with 4 dimensions. How are the matrices reshaped? (4) For section 2.2 and 2.3, the number/index of samples is not shown in the loss function for training. The loss should be over the whole training set. Besides, the test procedure is not clarified. (4) The number of iterations using FISTA of SDPC and Hi-La networks is shown to compare the rate of convergence. However, considering both models are solving a lasso-type regression problem, I would suggest using coordinate descent for optimization. (5) For the main result of prediction error, why is the “global prediction error” more important than the reconstruction error? Is the first-layer prediction error the reconstruction error? If yes, Fig.2 shows that Hi-La has a lower prediction error compared to SDPC for the first layer. (6) Two minor comments on writing: (a) It would be better to have a separate section for 2.5 since it describes the dataset and is not related to the proposed model. (b) A typo of “neuronal implementation” exists in the introduction section.