Title: Energy-Based Spherical Sparse Coding
Authors: Bailey Kong
Charless C. Fowlkes
Venue: ICLR 2017

Abstract: In this paper, we explore an efficient variant of convolutional sparse coding with unit norm code vectors and reconstructions are evaluated using an inner product (cosine distance). To use these codes for discriminative classification, we describe a model we term Energy-Based Spherical Sparse Coding (EB-SSC) in which the hypothesized class label introduces a learned linear bias into the coding step. We evaluate and visualize performance of stacking this encoder to make a deep layered model for image classification.


Review discussion:
If I understand correctly, the main point of the paper is that by using this energy based model, with a smart sparse-coding-type formulation, one can involve a feedback strategy into a DNN without the need of a iterative procedure, just running a feed-forward pass. I find the idea interesting.
The optimisation in the single layer setting is clearly explained, however, the multi-layer case is not, in my opinion. Is the global optimisation still feed forward?
There are approaches that attempt the same objective with recurrent settings (such as Cao et al. cited in the paper) The authors say that the advantage of the proposed formulation is that it is feed forward rather than iterative.
However, the forward pass of the EB-SSC network needs to be performed as many times as distinct labels are considered. Conditioning acts not only the last layer, but the full network, if I understand correctly from Table 1. This can be expensive and, with many classes, comparable to an iterative procedure. Please clarify. Maybe I'm missing something.
In my opinion it would be good to include a section describing the multi layer setting. Please, let us know if results are updated in the PDF with the larger model (max pooled version of the model).
The coding for a single layer is non-iterative but running the model for multiple class hypotheses in the energy-based setting requires running the forward model multiple times, one for each hypothesized class y. In the experiments, we run a simpler version in which we compute non-class conditional codes and then simply apply a standard classifier to the codes or only apply the energy-based coding to the top few layers.
You're correct, that the optimal codes cannot be found in a feed-forward manner in the multi-layer setting. Empirically, however, we have found this to be good-enough. We have added a section explaining the multi-layer setting in more detail with an experiment doing block-coordinate descent (unrolled) optimization of the codes, thanks for your suggestion.
We found that max-pooling does work better and have updated the table of results.
Sec. 2.2 The motivation of splitting the sparse code to positive and negative part is not clear. The author claimed it brings flexibility, but there are many ways to build a over-parameterized representation of a vector. One can even come up with a more over-complete representations, such as a matrix that the sum of the column equals to the z. It seems to me that the main reason is that the proximal operator of this way of encoding is coupled with CReLU.
But it is not necessary design in this way. I am wondering whether this is necessary but there is no ablation study to prove that. Please justify.
Sec. 3. Equ. 13 you need to solve K spherical sparse coding problem in this Equation, where K is the number of classes.
That is to say, you need K feed-forward sparse coding net in order to evaluating the loss. If K is large it is quite computational expensive. Could you please comment this?
Some related work should be discussed. Karol Gregor and Yann LeCun "Learning Fast Approximations of Sparse Coding" ICML10.
While the splitting is not absolutely necessary for our formulation, as the term with z- can be dropped in the case of ReLU, Shang et al. ICML 2016 found that CReLU outperforms ReLU. We have repeated this experiment and have added a row to the table showing this.
Your understanding is correct. It is for this reason that in the experiments we consider a simpler version in which we compute non-class conditional codes and then simply apply a standard classifier to the result or only apply the energy-based coding to the top few layers.
Thanks for your suggestion, we have added a citation and discussion.
It is not clear how the problem of irrelevant features mentioned in the introduction is solved by the proposed solution.  It would be helpful to provide ablative study on how the class information during the dictionary learning can improve the classification performance compared to unsupervised dictionary learning version of the proposed formulation.
Eq 5 needs more elaboration. The last step seems to be coming from the fact that ||d_k||_1 <= \sqrt{D} ||d_k||_2 = \sqrt{D} and ||z_k||_1 <= \sqrt{D} ||z_k||_2.
Eq 6 would benefit from clarification. I understand that the last step is obtained from Eq 5 and the fact that E_code(x,z) is would linearly scale with z. It would be helpful to make them explicit in the text.
You're correct in that our formulation does not solve the problem of irrelevant features. We have updated the text accordingly. We tried training our energy-based model on top of dictionaries learned in the unsupervised setting and was not able to improve the error rate beyond 0.79.
Thanks for your feedback on Eq 5 & 6, we have edited text to make this more clear.
First, I'd like to thank the authors for their answers and clarifications.
I find, the presentation of the multi-stage version of the model much clearer now.
Pros:
+ The paper states a sparse coding problem using cosine loss, which allows to solve the problem in a single pass.
+ The energy-based formulation allows bi-directional coding that incorporates top-down and bottom-up information in the feature extraction process.
Cons:
+ The cost of running the evaluation could be large in the  multi-class setting, rendering the approach less attractive and the computational cost comparable to recurrent architectures.
+ While the model is competitive and improves over the baseline, the paper would be more convincing with other comparisons (see text). The experimental evaluation is limited (a single database and a single baseline)
The motivation of the sparse coding scheme is to perform inference in a feed forward manner. This property does not hold in the multi stage setting, thus optimization would be required (as clarified by the authors).
Having an efficient way of performing a bi-directional coding scheme is very interesting. As the authors clarified, this could not necessarily be the case, as the model needs to be evaluated many times for performing a classification.
Maybe an interesting combination would be to run the model without any class-specific bias, and evaluation only the top K predictions with the energy-based setting.
Having said this, it would be good to include a discussion (if not direct comparisons) of the trade-offs of using a model as the one proposed by Cao et al. Eg. computational costs, performance.
Using the bidirectional coding only on the top layers seems reasonable: one can get a good low level representation in a class agnostic way. This, however could be studied in more detail, for instance showing empirically the trade offs. If I understand correctly, now only one setting is being reported.
Finally, the authors mention that one benefit of using the architecture derived from the proposed coding method is the spherical normalization scheme, which can lead to smoother optimization dynamics. Does the baseline (or model) use batch-normalization? If not, seems relevant to test.
Minor comments:
I find figure 2 (d) confusing. I would not plot this setting as it does not lead to a function (as the authors state in the text).
Thanks for your review.
Cao et al. proposed a binarized feedback network that propagates semantic information back to the image representation to maximize the target score, which is obtained from an existing pretrained network. The main focus of their work is object localization, and so is not comparable to ours as we focus on object classification.
Another reason the work is not comparable is because their feedback model is not tied to the bottom-up model like ours. In our work top-down (feedback) and bottom-up are tied mathematically, unlike Cao et al.'s ad-hoc feedback model.
In fact, there is no learning involved with their feedback model as they optimize the activations to maximize the target score of interest.
We tried evaluating multiple blocks on top of class-agnostic features. In the table below, we show the number of multiplication operations and test error rates when using multiple energy-based classifiers. The number of blocks for the entire network is always fixed at 7, so if 3 energy-based classifiers are stacked at the very top, then 4 class-agnostic feature extraction blocks are stacked below them. Blocks=2 is SSC+EBC_{6-7} in the paper. Performance at blocks=3 is the same as blocks=2, but at 53% more computation. The increased computational cost makes training much more difficult, while the increased network capacity increases the likelihood of the model overfitting the training data (as seen with blocks=4).
Further decrease in test error will likely require more training data.
Enery-Based     Computation Cost            Performance
Blocks          (#Multiply Ops)     (Train Error)   (Test Error)
1               647M                0.0085          0.1019
2               713M                0.0021          0.0923
3               1,096M              0.0014          0.0923
4               2,624M              0.0013          0.0954
5               4,153M
6               4,918M
7               6,446M
This paper proposes sparse coding problem with cosine-loss and integrated it as a feed-forward layer in a neural network as an energy based learning approach. The bi-directional extension makes the proximal operator equivalent to a certain non-linearity (CReLu, although unnecessary). The experiments do not show significant improvement against baselines.
Pros:
- Minimizing the cosine-distance seems useful in many settings where compute inner-product between features are required.
- The findings that the bidirectional sparse coding is corresponding to a feed-forward net with CReLu non-linearity.
Cons:
- Unrolling sparse coding inference as a feed-foward network is not new.
- The class-wise encoding makes the algorithm unpractical in multi-class cases, due to the requirement of sparse coding net for each class.
- It does not show the proposed method could outperform baseslines in real-world tasks.
Thanks for your review.
We evaluated our method on a pollen grain classification problem. The dataset contains electron microscopic image scans of the pollen grain surface. There are 10 species of pollens in the dataset, with 2052 training images and 108 testing images per class. The problem can be seen as a texture classification problem.
To save a little bit of time, we chopped the network down to 6 blocks from 7, but the network architecture is otherwise the same as the one used for CIFAR-10.
We see that with this dataset as well, our proposed method outperform the baseline, as we saw with CIFAR-10.
Model               Train Err(%)    Test Err(%)
CReLU+LC_6          3.51            12.77
CReLU(SN)+LC_6      3.48            10.46
SSC+LC_6            2.63            10.27
SSC+EBC_6           1.16            9.58
SSC+EBC_{5-6}       0.86            9.33
The paper introduces an efficient variant of sparse coding and uses it as a building block in CNNs for image classification. The coding method incorporates both the input signal reconstruction objective as well as top down information from a class label. The proposed block is evaluated against the recently proposed CReLU activation block.
Positives:
The proposed method seems technically sound, and it introduces a new way to efficiently train a CNN layer-wise by combining reconstruction and discriminative objectives.
Negatives:
The performance gain (in terms of classification accuracy) over the previous state-of-the-art is not clear. Using only one dataset (CIFAR-10), the proposed method performs slightly better than the CRelu baseline, but the improvement is quite small (0.5% in the test set).
The paper can be strengthened if the authors can demonstrate that the proposed method can be generally applicable to various CNN architectures and datasets with clear and consistent performance gains over strong CNN baselines. Without such results, the practical significance of this work seems unclear.
Thanks for your review.
We evaluated our method on a pollen grain classification problem. The dataset contains electron microscopic image scans of the pollen grain surface. There are 10 species of pollens in the dataset, with 2052 training images and 108 testing images per class. The problem can be seen as a texture classification problem.
To save a little bit of time, we chopped the network down to 6 blocks from 7, but the network architecture is otherwise the same as the one used for CIFAR-10.
We see that with this dataset as well, our proposed method outperform the baseline, as we saw with CIFAR-10.
Model               Train Err(%)    Test Err(%)
CReLU+LC_6          3.51            12.77
CReLU(SN)+LC_6      3.48            10.46
SSC+LC_6            2.63            10.27
SSC+EBC_6           1.16            9.58
SSC+EBC_{5-6}       0.86            9.33

Meta-review:
role: PC
data: comment: This paper proposes a variant of convolutional sparse coding with unit norm code vectors using cosine distance to evaluate reconstructions. The performance gains over baseline networks are quite minimal and demonstrated on limited datasets, therefore this work fails to demonstrate practical usefulness, while the novelty of the contribution is too slight to stand on its own merit.

Decision: Reject