Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed the convolutional sparse coding layer (CSC-layer) to obtain performance comparable to standard ConvNets with better interpretability and stability.  By conducting extensive experiments on CIFAR-10, CIFAR-100, and ImageNet, the model with CSC-layer has shown to achieve comparable performance as standard ConvNets and be more robust to data perturbation than the standard one.
strengths_and_weaknesses: Strengths:
This paper proposed an interesting and effective idea for robust inference with the proposed convolutional sparse coding layer and achieved impressive performance on both image corruption and adversarial attack. 

The weaknesses mainly lay in the cost of the training and robustness inference: 
1. This paper claims the fast training speed as one of its contributions. However, this needs careful comparison with its baselines. 
2. Although this method does not require modifying the training procedure like the existing methods to obtain robustness, it has additional costs to get the optimal $\lambda$ for robustness inference.
questions: 1. In Table 1, the model size of SDNet-18 and SDNNet-34 on CIFAR-100 are much smaller than on CIFAR-10, which seems wired. 
2. With similar performance, the proposed method is much faster than its baselines. In Table 1, the proposed SDNet only replaces the first convolutional layer with CSC-layer while SCN is a multilayer sparse coding network. For a fair comparison, this paper may compare the time and memory consumption of a single sparse coding layer between those methods. 
3. Each layer of CSC-layer of SDNet-18 and SDNet34 needs unrolling two iterations of FISTA and more iterations will only slightly improve the performance. As SDNet-18 and SDNet-34 have only one CSC-layer for the input images, I’m curious whether it is this low dimension (3 channels) of input that make two iterations sufficient. On SDNet-18-All and SDNet-34-All, could you list the dimension of the input and output of each CSC-layers and their corresponding iterations used? 
4. What is the cost to find the optimal $\lambda$ and calculate the residual in Algorithm 1? 
5. I would like to confirm whether it is $z$ of a CSC-layer that will be the input of the next layers. 
6. Does FISTA algorithm always randomly initialize $z$ for any CSC-layer in any iteration during the training? If it is, is it possible to initialize it with the previous leaned values for an image when the model sees it again? This may reduce the number of FISTA iterations. 

limitations: This paper introduced an appealing method to apply CSC-layer for robustness inference. It is great if this paper has a more detailed comparison of its time and memory for both training and inference with the other methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach to incorporate convolutional sparse coding into deep neural networks. In particular, a convolutional layer of a ResNet is replaced with an implicit layer, referred to as convolutional sparse coding layer (CSC-layer), which outputs a sparse representation (a feature map) of its input given a (learned) dictionary of convolutional filters. The sparse representation for an input is computed using the FISTA algorithm. The experiments show that ResNet models whose first convolutional layer is replaced by such a CSC-layer can achieve similar or better classification performance on the CIFAR-10, CIFAR-100, and ImageNet datasets. Additionally, experiments suggest that ResNet models with a CSC-layer are more robust to different kinds of additive noise applied to the input when compared to a regular ResNet model.
strengths_and_weaknesses: Strengths

- The proposed method which combines sparse coding with deep neural networks is scalable to datasets such as ImageNet.

- With only 2 iterations of FISTA, the proposed modification of the ResNet model is more robust to additive noise in the input than the vanilla ResNet architecture when evaluated on the task of classification.

- The paper is organized and written clearly. 

questions: 1. Have the authors considered visualizing the learned sparse dictionary convolutional kernels common in related literature (eg. [1])? I believe this would help with interpretability and understanding what the dictionary of convolutional kernels encodes.

2. Typically, the FISTA algorithm requires hundreds of iterations to converge so my expectation is that the reconstructions $\tilde{x} = \mathcal{A}(z_*)$ with only 2 iterations are not high fidelity (e.g., terms of PSNR). This is supported by the visualization in Appendix B2 which shows that feature maps only encode contours or high-level information about the input. The authors mention that increasing the number of FISTA iterations can boost the classification performance a bit. Have the authors’ studied how increasing the number of FISTA iterations affects the model’s robustness to noise or can they provide intuition about it?

3. My understanding is that only the first convolutional layer of ResNet-18 and ResNet-34 (the one closest to the input) is replaced by a CSC-layer. Is this correct or does “the first convolutional layers” (line 235) refer to the first convolutional layer of each ResNet block? 

4. How is the value of $\lambda = 0.1$ used during training selected? What is the size of $C$ used in experiments, i.e. the number of sparse feature maps in $z$ (line 125)? How sparse on average are the feature maps output by FISTA when only 2 iterations are used with regularization coefficient $\lambda = 0.1$?

5. What magnitudes do levels 0-6 in Figure 2 correspond to for each type of noise? E.g. for Gaussian noise, what levels of noise are considered? Same for Tables 2 and 3.

6. Have the authors studied how robust SD SCN model (from [3]) is to additive noise? Would be insightful to add it as a baseline in Tables 2 and 3.

7. Are the authors planning to provide an implementation of the proposed framework?

8. Nitpicks
- Figures 2 and 3 would be easier to read if the font size in is bigger. 

[1] Kavukcuoglu, K., Sermanet, P., Boureau, Y.L., Gregor, K., Mathieu, M. and LeCun, Y., 2010. Learning convolutional feature hierarchies for visual recognition. Advances in neural information processing systems, 23.

[2] Beck, A. and Teboulle, M., 2009. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1), pp.183-202.

[3] Sun, X., Nasrabadi, N.M. and Tran, T.D., 2018, October. Supervised deep sparse coding networks. In 2018 25th IEEE International Conference on Image Processing (ICIP) (pp. 346-350). IEEE.

limitations: The authors are upfront about the limitations of higher memory and inference costs of unrolling FISTA for a higher number of iterations. They could elaborate on the potential negative societal impacts, for example, the fact that the model can propagate any bias present in the dataset.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes to replace the traditional convolutional layers with convolutional sparse coding (CSC) layers, claiming that such substitution adds interpretability and robustness to neural networks.
strengths_and_weaknesses: The writing is generally good and easy to follow; however, the technical claims in the paper are not well supported by empirical evidence.
questions: I have three main concerns about the claims in the paper.

**1. Interpretability**. The traditional convolutional layer performs a forward computation (the output is a linear combination of the inputs). In contrast, the convolutional sparse coding (CSC) layer performs a backward computation (the input is a linear combination of the outputs). It is not apparent why a backward computation is more interpretable than a forward one. In my opinion, it is not an individual layer that makes a neural network hard to interpret but the stack of these layers. While convolution layer and convolutional sparse coding are easy to interpret individually, using them in deep networks (with nonlinearities, normalization, etc.) is not.

**2. Robustness**. The explanation of why the CSC layer is more robust is insufficient --- no math derivation is used to explain this concept. From the writing, it seems the Lipschitz constant decreases if the regularizer increases. In this case, the comparison to standard convolutional networks is not fair. Maybe the authors would also like to enforce the Lipschitzness of traditional convolutional networks (e.g., following https://arxiv.org/abs/1804.04368).

**3. Computational Complexities**. It looks to me that the proposed layer is quite expensive. In the experiment, only one layer in ResNet is replaced by the proposed layer, and only two iterations are used in unrolling. And this already decreases the speed from 1000 to 900. I think a more comprehensive study on the relationship between accuracy, complexity, and iterations is needed when all layers are replaced.
limitations: Not applicable.
review_title: null

Meta-review: metareview: In this paper, the authors introduce a convolutional sparse coding layer, which is intended as a replacement for a convolutional layer that is has greater interpretability and stability.  Experiments show that a ResNet modified with this CSC-layer can achieve comparable performance on standard datasets as convolutional networks and are more robust to noise.  The strength of this paper is that the novel layer it proposes is faster than previous space coding networks and that it has comparable accuracy and speed to ResNets while being more robust.  A weakness of the paper is that the claims of improved interpretability do not transfer to the network as a whole.  The strengths of the paper outweighs the weaknesses, and the authors should clarify in the camera ready that interpretability is only intended layer-wise.
confidence: Less certain
Decision: Accept