Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: It is not clear how the problem of irrelevant features mentioned in the introduction is solved by the proposed solution.  It would be helpful to provide ablative study on how the class information during the dictionary learning can improve the classification performance compared to unsupervised dictionary learning version of the proposed formulation.


Eq 5 needs more elaboration. The last step seems to be coming from the fact that ||d_k||_1 <= \sqrt{D} ||d_k||_2 = \sqrt{D} and ||z_k||_1 <= \sqrt{D} ||z_k||_2. 


Eq 6 would benefit from clarification. I understand that the last step is obtained from Eq 5 and the fact that E_code(x,z) is would linearly scale with z. It would be helpful to make them explicit in the text.

review_title: questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes sparse coding problem with cosine-loss and integrated it as a feed-forward layer in a neural network as an energy based learning approach. The bi-directional extension makes the proximal operator equivalent to a certain non-linearity (CReLu, although unnecessary). The experiments do not show significant improvement against baselines. 

Pros: 
- Minimizing the cosine-distance seems useful in many settings where compute inner-product between features are required. 
- The findings that the bidirectional sparse coding is corresponding to a feed-forward net with CReLu non-linearity. 

Cons:
- Unrolling sparse coding inference as a feed-foward network is not new. 
- The class-wise encoding makes the algorithm unpractical in multi-class cases, due to the requirement of sparse coding net for each class. 
- It does not show the proposed method could outperform baseslines in real-world tasks.
review_title: null

final_score: rating: null
review_content: If I understand correctly, the main point of the paper is that by using this energy based model, with a smart sparse-coding-type formulation, one can involve a feedback strategy into a DNN without the need of a iterative procedure, just running a feed-forward pass. I find the idea interesting.

The optimisation in the single layer setting is clearly explained, however, the multi-layer case is not, in my opinion. Is the global optimisation still feed forward? 

There are approaches that attempt the same objective with recurrent settings (such as Cao et al. cited in the paper) The authors say that the advantage of the proposed formulation is that it is feed forward rather than iterative. However, the forward pass of the EB-SSC network needs to be performed as many times as distinct labels are considered. Conditioning acts not only the last layer, but the full network, if I understand correctly from Table 1. This can be expensive and, with many classes, comparable to an iterative procedure. Please clarify. Maybe I'm missing something.

In my opinion it would be good to include a section describing the multi layer setting. Please, let us know if results are updated in the PDF with the larger model (max pooled version of the model).
review_title: Question on multi-layer setting and setting

final_score: rating: null
review_content: Sec. 2.2 The motivation of splitting the sparse code to positive and negative part is not clear. The author claimed it brings flexibility, but there are many ways to build a over-parameterized representation of a vector. One can even come up with a more over-complete representations, such as a matrix that the sum of the column equals to the z. It seems to me that the main reason is that the proximal operator of this way of encoding is coupled with CReLU. But it is not necessary design in this way. I am wondering whether this is necessary but there is no ablation study to prove that. Please justify.


Sec. 3. Equ. 13 you need to solve K spherical sparse coding problem in this Equation, where K is the number of classes. That is to say, you need K feed-forward sparse coding net in order to evaluating the loss. If K is large it is quite computational expensive. Could you please comment this?

Some related work should be discussed. Karol Gregor and Yann LeCun "Learning Fast Approximations of Sparse Coding" ICML10. 
review_title: Using cosine loss is interesting. Energy based sparse coding is quite standard. The paper requires some justification. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces an efficient variant of sparse coding and uses it as a building block in CNNs for image classification. The coding method incorporates both the input signal reconstruction objective as well as top down information from a class label. The proposed block is evaluated against the recently proposed CReLU activation block.

Positives:
The proposed method seems technically sound, and it introduces a new way to efficiently train a CNN layer-wise by combining reconstruction and discriminative objectives.

Negatives:
The performance gain (in terms of classification accuracy) over the previous state-of-the-art is not clear. Using only one dataset (CIFAR-10), the proposed method performs slightly better than the CRelu baseline, but the improvement is quite small (0.5% in the test set). 

The paper can be strengthened if the authors can demonstrate that the proposed method can be generally applicable to various CNN architectures and datasets with clear and consistent performance gains over strong CNN baselines. Without such results, the practical significance of this work seems unclear.


review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
First, I'd like to thank the authors for their answers and clarifications.
I find, the presentation of the multi-stage version of the model much clearer now.

Pros:

+ The paper states a sparse coding problem using cosine loss, which allows to solve the problem in a single pass.

+ The energy-based formulation allows bi-directional coding that incorporates top-down and bottom-up information in the feature extraction process. 

Cons:

+ The cost of running the evaluation could be large in the  multi-class setting, rendering the approach less attractive and the computational cost comparable to recurrent architectures.

+ While the model is competitive and improves over the baseline, the paper would be more convincing with other comparisons (see text). The experimental evaluation is limited (a single database and a single baseline)

------

The motivation of the sparse coding scheme is to perform inference in a feed forward manner. This property does not hold in the multi stage setting, thus optimization would be required (as clarified by the authors).

Having an efficient way of performing a bi-directional coding scheme is very interesting. As the authors clarified, this could not necessarily be the case, as the model needs to be evaluated many times for performing a classification.

Maybe an interesting combination would be to run the model without any class-specific bias, and evaluation only the top K predictions with the energy-based setting.

Having said this, it would be good to include a discussion (if not direct comparisons) of the trade-offs of using a model as the one proposed by Cao et al. Eg. computational costs, performance.

Using the bidirectional coding only on the top layers seems reasonable: one can get a good low level representation in a class agnostic way. This, however could be studied in more detail, for instance showing empirically the trade offs. If I understand correctly, now only one setting is being reported.

Finally, the authors mention that one benefit of using the architecture derived from the proposed coding method is the spherical normalization scheme, which can lead to smoother optimization dynamics. Does the baseline (or model) use batch-normalization? If not, seems relevant to test.


Minor comments:

I find figure 2 (d) confusing. I would not plot this setting as it does not lead to a function (as the authors state in the text).



review_title: Review

Meta-review: This paper proposes a variant of convolutional sparse coding with unit norm code vectors using cosine distance to evaluate reconstructions. The performance gains over baseline networks are quite minimal and demonstrated on limited datasets, therefore this work fails to demonstrate practical usefulness, while the novelty of the contribution is too slight to stand on its own merit.
Decision: Reject