Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work proposed a new model called Sparse Deep Predictive Coding, which introduced top-down connections between consecutive layers, to improve the solutions to hierarchical sparse coding (HSC) problems. Instead of decomposing the HSC problem into independent subproblems, the proposed model added a new term to the loss function, which represents the influence of the latter-layer on the current layer.

#Pros:
-- The proposed model adopted the idea from predictive coding and came up with a relatively novel idea for HSC problems.
-- The experiments are solid. The experiments evaluated the proposed methods with different hyper-parameter settings and three real-world datasets.
-- The figures in the result sections are well designed and concise.

#Cons:
-- The mathematical description of the main problem and the proposed model is not clear. For example, the dimensionality for the variables in Eq.(1) is not clarified.
-- The test procedure is not clear. How the internal state variables are obtained for the test set is not clarified.
-- The proposed model was only compared with a basic Hierarchical Lasso network. There are not any state-of-art methods included as baseline methods.

#Detailed comments:

(1) The proposed model is named as sparse DEEP predictive coding, however, the experiments only considered the SDPC and Hi-La networks with 2 layers. I am wondering if a deeper structure will improve the performance?

(2) For the structure shown in Fig.1, the decoding dictionaries are $D^T_i$, but I am confused why the encoding dictionaries are reciprocal to encoding dictionaries. Does it come from the optimization updates shown in Eq.(3)?

(3) According to Eq.(1), $x$ is a vector and $D$ is a 2d matrix. However, the real inputs in the experiments are images and $D$ is a convolutional filter with 4 dimensions. How are the matrices reshaped?

(4) For section 2.2 and 2.3, the number/index of samples is not shown in the loss function for training. The loss should be over the whole training set. Besides, the test procedure is not clarified.

(4) The number of iterations using FISTA of SDPC and Hi-La networks is shown to compare the rate of convergence. However, considering both models are solving a lasso-type regression problem, I would suggest using coordinate descent for optimization.

(5) For the main result of prediction error, why is the “global prediction error” more important than the reconstruction error? Is the first-layer prediction error the reconstruction error? If yes, Fig.2 shows that Hi-La has a lower prediction error compared to SDPC for the first layer.

(6) Two minor comments on writing:
(a) It would be better to have a separate section for 2.5 since it describes the dataset and is not related to the proposed model.
(b) A typo of “neuronal implementation” exists in the introduction section. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents a study that compares two techniques for Hierarchical Sparse Coding. This is the problem of learning sparse representations in multi-layer (but not necessarily deep) models. The first method applies Lasso at each layer in a bottom up fashion, while the second method — introduced in the submitted work — adds an additional term which propagates information in a top-down manner. It is found that the top-down term is beneficial in terms of reducing predictive error and can learn faster. In terms of novelty the new term, the paper does not make a breakthrough contribution, but I consider this to be sufficient. Moreover, the paper is well written and presents some interesting results. However I am not entirely sure that the impact of these results will attract the interest of a broad audience since the experiments presented are on “small datasets” and with some rather shallow neural nets. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes Sparse Deep Predictive Coding (SDPC) to access the impact of the inter layer feedback, which is suggested by neuro-scientific evidence. The SDPC model is compared with HILA on 2 different databases, and the experimental results show that SDPC achieved lower prediction error, faster converge rate. 

Although the paper is well-motivated and well-written, the potential impact of this work is limited in the community. Moreover, the experiments are not sufficient to valid the advantage of this model. For example, there are only 3 dataset adopted in the experiments and the all these three datasets are quite small. Moreover, there is only one baseline, which seems too weak to show the advantage of the model.

Based on the above points, I tend to reject the paper.

**updated comments**
Thanks for the response and I really appreciate the hard work during the rebuttal. However, the STL-10 is still a small scale dataset, and the baselines provided in the rebuttal are still limited. I will not change the original score.
review_title: Official Blind Review #2

Meta-review: This paper introduces a new architecture for sparse coding.

The reviewers gave long and constructive feedback that the authors in turn responded at length on. There is consensus among the reviewers that despite contributions this paper in its current form is not ready for acceptance.

Rejection is therefore recommended with encouragement to make updated version for next conference.  


Decision: Reject