Revisiting Sparse Convolutional Model for Visual Recognition
Authors: Xili Dai
Mingyang Li
Pengyuan Zhai
Shengbang Tong
Xingjian Gao
Shao-Lun Huang
Zhihui Zhu
Chong You
Yi Ma
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sparse Dictionary Learning
inverse models
image classification

Abstract: Despite strong empirical performance for image classification, deep neural networks are often regarded as ``black boxes'' and they are difficult to interpret. On the other hand, sparse convolutional models, which assume that a signal can be expressed by a linear combination of a few elements from a convolutional dictionary, are powerful tools for analyzing natural images with good theoretical interpretability and biological plausibility. However, such principled models have not demonstrated competitive performance when compared with empirically designed deep networks. This paper revisits the sparse convolutional modeling for image classification and bridges the gap between good empirical performance (of deep learning) and good interpretability (of sparse convolutional models). Our method uses differentiable optimization layers that are defined from convolutional sparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks. We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100 and ImageNet datasets when compared to conventional neural networks. By leveraging stable recovery property of sparse modeling, we further show that such models can be much more robust to input corruptions as well as adversarial perturbations in testing through a simple proper trade-off between sparse regularization and data reconstruction terms. 

Revisiting Sparse Convolutional Model for Visual Recognition
Xili Dai Mingyang Li Pengyuan Zhai
Shengbang Tong' Xingjian Gaot Chong You" Yi Ma
Shao-Lun Huang Zhihui Zhu
'The Hong Kong University of Science and Technology (Guangzhou)
Eisinghua-Berkelee Shenzhen Institute (TBSI), Tsinghua University
Harvard University
UUniversity of California, Berkeley Ohio State University
Abstract
Despite strong empirical performance for image classification, deep neural net- works are often regarded as "black boxes" and they are difficult to interpret. On the other hand, sparse convolutional models, which assume that a signal can be expressed by linear combination of few elements from convolutional dictionary, are powerful tools for analyzing natural images with good theoretical in- terpretability and biological plausibility. However, such principled models have not demonstrated competitive performance when compared with empirically designed deep networks. This paper revisits the sparse convolutional modeling for image classification and bridges the gap between good empirical performance (of deep learning) and good interpretability (of sparse convolutional models). Our method uses differentiable optimization layers that are defined from convolutional sparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks. We show that such models have equally strong empirical performance CIFAR-IO, CIFAR-100 and ImageNet datasets when compared conventional neural networks. By leveraging stable recovery property of sparse modeling, we further show that such models can be much more robust input corruptions as well as adversarial perturbations in testing through simple proper trade-off between sparse regularization and data reconstruction terms. Source code can be found at //gp::///itthbl com/Delay Xili/SDNet.
Introduction
In recent years, deep learning has been a dominant approach for image classification and has significantly advanced the performance over previous shallow models. Despite the phenomenal empirical success, it has been increasingly realized as well as criticized that deep convolutional networks (ConvNets) are "black boxes" for which we are yet to develop clear understanding [I]. The layer operations such as convolution, nonlinearity and normalization are geared towards minimizing an end-to-end training loss and do not have much data-specific meaning. As such, the functionality of each intermediate layer in a trained ConvNets is mostly unclear and the feature maps that they produce are hard to interpret. The lack interpretability also contributes to the notorious difficulty in enhancing such learning systems for practical data which are usually corrupted by various forms of perturbation.
This paper presents a visual recognition framework by introducing layers that have explicit data modeling tackle shortcomings of current deep learning systems. We work under the assumption that the layer input can be represented by few atoms from a dictionary shared by all data points. This
Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

is the classical sparse data modeling that, as shown in pioneering work of (2), can easily discover meaningful structures from natural image patches. Backed by its ability in learning interpretable representations and strong theoretical guarantees E
4
5
6
g
(e.g. for handling corrupted
data), sparse modeling has been used broadly in many signal and image processing applications
[9] However, the empirical performance of sparse methods have been surpassed by deep learning
methods for classification of modern image datasets.
Because of the complementary benefits of sparse modeling and deep learning, there exist many efforts that leverage sparse modeling to gain theoretical insights into ConvNets and/or to develop computational methods that further improve upon existing ConvNets. One of the pioneering works is
[10] which interpreted a ConvNet as approximately solving a multi-layer convolutional sparse coding
model. Based on this interpretation, the work [10] and its follow-ups [TT 12 [3] [1] presented alternative algorithms and models in order to further enhance the practical performance of such learning systems. However, there has been no empirical evidence that such systems can handle modern image datasets such as ImageNet and obtain comparable performance to deep learning. The only exception to the best of our knowledge is the work of TS 16 which exhibited a performance on par to (on ImageNet) or better than (on CIFAR- 10) ResNet. However, the method in [15] 1) requires a dedicated design of network architecture that may limit its applicability. 2) is computationally orders of magnitude slower to train, and 3) does not demonstrate benefits in terms of interpretability and robustness. In nutshell, sparse modeling is yet to demonstrate practicality that enables its broad
applications.
Paper contributions. In this paper, we revisit sparse modeling for image classification and demon- strate through a simple design that sparse modeling can be combined with deep learning to obtain performance on par with standard ConvNets but with better layer-wise interpretability and stability. Our method encapsulates the sparse modeling into an implicit layer [7 [8 [9] and uses it as a drop-in replacement for any convolutional layer in standard ConvNets. The layer implements the convolutional sparse coding (CSC) model of [20), and is referred to as a CSC-layer, where the input signal is approximated by sparse linear combination of atoms from convolutional dictionary. Such a convolutional dictionary is treated as the parameters of the CSC-layer that are amenable to training via back-propageeee propagation. Then, the overall network with the CSC-layers may be trained in end-to-end fashion from labeled data by minimizing the cross cross-entropy loss as usual. This paper demonstrates that such learning framework has the following benefits:
Performance on standard datasets. We demonstrate that our network obtains better (on CIFAR- 100) or on par (on CIFAR- 10 and ImageNet) performance with similar training time compared with standard architectures such ResNet (21). This provides the first evidence on the strong empirical performance of sparse modeling for deep learning to the best of our knowledge. Compared previous sparse methods [15] that obtained similar performance, our method is of orders of magnitude faster.
Robustness to input perturbations. The stable recovery property of sparse convolution model equips the CSC-layers with the ability to remove perturbation in the layer input and to recover clean sparse code. As a result, our networks with CSC-layers are more robust to perturbations in the input images compared with classical neural networks. Unlike existing approaches for obtaining robustness that require heavy data augmentation [22] or additional training techniques 23 our method is
and does not require modifying the training procedure at all.
2 Related Work
Implicit layers. The idea of trainable layers defined from implicit functions can be traced back at least to the work of (24). Recently, there is a revival of interests in implicit layers [17] [8 19 (55
26 27 28 29] as an attractive alternative to explicit layers in existing neural networks. However, a majority of the cited works above define an implicit layer by a fixed point iteration, typically motivated from existing explicit layers such as residual layers, therefore they do not have clear interpretation in terms of modeling of the layer input. Consequently, such models do not have the ability to deal with input perturbations. The only exceptions are differentiable optimization layers 30 [8 31
that incorporate complex dependencies between hidden layers through the formulation of convex optimization. Nevertheless, most of the above works focus on differentiating through the convex optimization layers (such as disciplined parametrized programming [18) without specializing in any
2

---

particular signal models such as the sparse models considered in this paper nor demonstrating their performance when encapsulated in multi-layer neural networks.
Sparse prior in deep learning. Aside from image classification, sparse modeling has been intro-
duced to deep learning for many image processing tasks such as super-resolution 22
denoising (
and so on 34 that are motivated by (but are not the same as) an unrolled sparse coding algorithm LISTA [38]. In sharp contrast to ours, there is no guarantee that such architectures perform sparse encoding with respect a particular (convolution) dictionary all. As result, they lack the capability of handling input perturbations as in our method. A notable exception is the work of [15] where each layer performs precise sparse encoding and exhibits on par or better performance for image classification over ResNet. However, the practical benefit of the sparse modeling in terms of robustness is not demonstrated. Moreover, [15 adopts a patch-based sparse coding model for images and has large computational burden.
35 36 37.. These works incorporate sparse modeling by using network architectures
Robustness. It is known that modern neural networks are extremely vulnerable to small perturbations in the input data. A plethora of techniques have been proposed to address this instability issue, including stability training [2]] adversarial training 39 40 41, data augmentation 42 43 (2].
etc. Nevertheless, these techniques either need a computational and memory overhead, or require selection of appropriate augmentation strategies cover all possible corruptions. With standard training only, our model can be made robust to input perturbations in test data by simply adapting sparse modeling account for noise. Closely related our work are 4+ 45 46 which use sparse modeling to improve adversarial robustness. However, they either only demonstrate performance on very simple networks
455] 46 or sacrifice natural accuracy for robustness
44 In contrast, our
method is tested on realistic networks and does not affect natural accuracy.
3 Neural Networks with Sparse Modeling
In this section, we show how sparse modeling is incorporated into a deep network via a specific type of network layer that we refer to as the convolutional sparse coding (CSC) layer. We describe the CSC-layer in Sec. 3.T and and explain how we use them for deep learning in Sec. 3.2 Finally, Sec. 3.3 explains how CSC enables robust inference with corrupted test data.
Notations. Given a single-channel image & E I # XW represented as a matrix, we may treat it as a [1...., W] and extended Z by padding 2D signal defined
2D signal defined on the discrete domain [1...., H] , x
the discrete domain [- ... ko] x [-kg..... , ko] with & = 2kg + 1 and extended to E x by padding zeros. Then, for convenience, we use "*" and "x" to denote the convolution and correlation operators, respectively, between two 2D signals:
zeros. Given 2D kernel a e & we may treat it
* E]i,,
Σ
Σ
Eli - p.j pjj- g] a gl.
(1)
(a * Eii.j]]
Eli + p.j + alp.g)
3.1 Convolutional Sparse Coding (CSC) Layer
Sparse modeling is introduced in the form of an implicit layer of a neural network. Unlike classical fully-connected convolutional layers in which input-output relations are defined by an explicit function, implicit layers are defined from implicit functions. For our case, in particular, we introduce an implicit layer that is defined from an optimization problem involving the input to the layer as well as a weight parameter, where the output of the layer is the solution to the optimization problem.
A generative model via sparse convolution. Concretely, given a multi collii-dimessoonal input signal M to the layer where H, W are spatial dimensions and M is the number of channels for ar. We assume the signal x is generated by a multi-channel sparse code z E C convoluting with dimei-dilm kernel A E MR xxxxxk which is referred to as a convolution dictionary. Here C is the number of channels for and the convolution kernel A. To be more precise, denote
3

---

= (S1..... ,c) where each So E "x (presumably sparse), and denote the kernel A as
011
012
Q13
an
Q21
022
agg
02c
E ER MXCXXXX
amc
where each aij R is kernel of size lo x k. Then the signal is generated via the following operator A(-) defined by the kernel A as:
= A(z) = al * (aMe. Se) E R
aMe aMe
(3)
layer convolutional sparse coding. Given multi-dimensional input signal RMx#xx we define that the function of layer" is to perform an (inverse) mapping to a preferably sparse output 2. I CC where C' is the number of output channels. Under the above sparse generative model, we can seek the optimal sparse solution z by solving the following Lasso type optimization problem:
2, = arg min >||
1 + =11100 A(2)11
E RRCW
(4)
The optimization problem in 4 is based on the convolutional sparse coding (CSC) model [20 Hence, we refer to the im- plicit layer defined by (4) as a CSC-layer. The goal of the CSC model is to reconstruct the input via A(z), where the feature map z specifies the locations and magni- tudes of the convolutional filters in to be linearly combined (see Figure [ for an illus- tration). The reconstruction not required to be exact in order to tolerate modeling discrepancies, and the difference between and A(x) is penalized by its entry-wiss 2z-norm (i.c., the lg norm of x - -A2z) flat- tened into vector). Sparse modeling is introduced by the entry-wise & -norm of z in the objective function, which enforces to be sparse. The parameter À > 0 controls the tradeoff between the sparsity and the magnitude of the residual As we will show in Sec. 3.3 A can be used to improve the performance of our model in the test phase when the input is corrupted.
Oup Parc
Illustration of the operator A in the convolu- tional sparse coding model for the CSC-layer.
Figure 1:
A(2),, and is treated as hyper-paramen that subjects tuning via cross-validand valsssaaii.tion..
Based on the onput-output mapping of the CSC-layer given in 4 one may perform forward propa- gation by solving the associated optimization, and perform backward propagation by deriving the gradient of 2, with respect to the input æ and parameter A. In this paper, we adopt the fast iterative shrinkage thresholding algorithm (FISTA) [7] for the forward propagation, which also produces an unrolled network architecture that can carry out automatic differentiation for backward propagation. We defer discussion of the implementation details of the CSC-layer the Appendix.
3.2 Sparse Dictionary Learning Network Architecture and Training
Convolution layers are basic ingredients of ConvNets that appear in many common network archi- tectures such LeNet 48 and ResNet [49]. In this paper, incorporate sparse modeling into given existing/baseline network architecture by replacing certain / all convolution layers with the CSC-layer. Mcanwhile, all other layers such normalization, nonlinear, and fully connected layers
TTypically, convolution operators "4" are used in the detinition of the operator A (see 3 3), rather than the
correlation operators "+". We adopt the definition in 3 to be consistent with the convention of modern deep learning packages.

---

are retained. This simple design choice allows the CSC-laaees to be broadly applicable. We will refer our network with CSC-llyers as Sparse Dictionary learning Network (SDNet).
Give a set of training data { y, Ji=1 where a; denotes an image and y; is the corresponding label, train network with CSC-layers by solving the following optimization problem:
N
Σ lcE (f(m; 0): S.t. A EN ASES, N
1
min
(5)
where iCE denotes the cross-entropy loss. In above, we use f(: 0) to denote the mapping that is performed by the neural network. where 0 is set of learnable parameters containing a subset of kernel parameters {As) associated with the set S of CSC-layers. Following the convention the sparse dictionary learning literature, add the constraint A, E N for the dictionaries in CSC-layers, where N denotes the set of normalized dictionaries:
{ A E R MXCxIxx
N =
Σ
am = 1, [C]
To handle such constraint, we use the projected stochastic gradient descent (SGD) for solving the problem (5) That is, after each gradient update step for the parameters and {As)ses as in regular SGD, an extra step is taken to project each A. onto the constraint S..
3.3 Robust Inference
The fundamental difference of the CSC-layer vis-a-vis a classical explicit layer (e.g., convolutional layer) is that the CSC-layer imposes an assumption on the input feature map. That is, it assumes that the input feature map (or image) can be approximated by a superposition of few atoms of dictionary, which is the layer parameter that is learnable and is shared across all data. In this section, we show that CSC-layers enables us to design a robust inference strategy to obtain robustness to corruptions in ways that cannot be achieved by classical explicit layers.
We leverage an attractive property of the CSC model is that admits stable recovery of the sparse signals with respect to input noise.
In particular, Theorem shows that any bounded perturbation to the input of a CSC-layer produces bounded perturbation to its output and does not change the support of the output, if one uses a À in (4) that is proportional to the norm of the perturbation in the layer input, provided that certain technical conditions are satisfied.
Theorem 1. (Informal version of (50] Theorem 19]) Suppose Ty has a representation A(z₂) as in
3 and that it is contaminated by noise to create the input æ w. + Then as long 2 is sufficiently sparse, the solution z, to (4) with X = O([eel2)) satisfies (i) the support of z, is contained
in that of 2 and (ti) ||z. 24 21112- (1
Intuitively, the parameter À in (4) controls a balance between the sparsity regularization ||=|| || and the residual x A(z), the latter which accounts for modeling discrepancies and increases when æ is noisy. Therefore, using larger value of À helps the model to handle a larger residual.
We thus present a very simple approach to obtain model robustness. We consider the setting where a model is trained on an uncorrupted dataset but is tested on data that is corrupted by random noise. Corruptions in the input image may propagate into deeper feature maps (hence corrupting the input of all CSC-layers) during forward propagation. Therefore, instead of directly using the CSC-layers that are obtained from the training phase, our method is to adjust the trade-off parameter À in the CSC-layers for the test data. As we show in the experiments (see Sec. 4.2 optimal value of À indeed increases with the variance of the noise in the input image, which is well-aligned with the result in Theorem I
Choosing the optimal A. practical applications, the amount of noise in given test dataset is often unknown. Hence, the optimal choice of parameter X becomes nontrivial task. Here we present practical technique for determining a proper choice of value based on the simple observation that the amount of noise in the test data correlates with the magnitude of the residual æ A(z). That is, for test data corrupted by a larger amount of noise, we expect the magnitude of the residual in CSC-layers to become larger with such data fed into the network. Since the residual for any test data
5

---

Algorithm 1 Robust inference with neural networks constructed from CSC-layers
Input: A network architecture with CSC-layers f(: 0, Ao). (clean) training data Tiain a (corrupted)
test data Trest corruption type T, set C of corruption levels, set A of values for X.
1: # Training the network
2: Train the network f(: 0, Au)) on Tuain as described in Sec. 3.2 to obtain learned parameters 0..
3: # Fitting relationship between optimal and the residual from CSC-layers using Timin
4: for each noise level C do
5: Generate corrupted data Train by injecting random noise of type T with level e to Tirain-
6: Apply :.0..)))) on T and compute averaged residual from all CSC-layers as Te.
7: for each parameter E A do
8: Apply f(;; X) on Tuain and compute averaged accuracy ax.
9: end for
10: Set Xe arg maxxEA ax.
11: end for
12: Fit a function À X(r) from (eeere via linear least squares.
13: # Computing the residual from CSC-layers on Trea
14: Apply 0.: Xo) on Tres and compute averaged residual from all CSC-layers as Tiest- Output: Predicted labels on Test with the network f(; 0., X(rea))-
can always be computed, the key question here is how we can find a relationship between an optimal value Ả and the magnitude of the residual.
Our technique for addressing this challenge is to learn such a relationship on the training set, by injecting synthetic data corruptions. For simplicity we summarize our technique in Algorithm ! and explain it in details below. We assume that although the amount of noise in a test dataset Trea is unknown, the type of the noise (e.g., Gaussian noise, shot noise, etc.) is known. Given the noise type, we first determine a set C which contains a set of values specifying the potential amount of noise in test data. For example, if we consider Gaussian noise, then C contains set of values specifying the variance the noise. We also specify a set A of potential values for to be used for inference. After training the network specified in Step 2 we use procedure described in between Step 4 and Step 12 to fit a function that maps a residual value r to an optimal choice of The idea for fitting such a relation is to generate synthetic noise of varying magnitudes in C' on the training set, and for each noise magnitude we sweep the parameter À E A to find a that produces the best accuracy on training set. Once such relationship is learned, we feed in the residual value computed on the test dataset Tres to predict an optimal À to be used for inference on test data, as described in Step 14
Experiments
In this section, we provide experimental evidence for neural networks with CSC-lavers as discussed in Sec. Through experiments on CIFAR-10, CIFAR- and ImageNe Sec. 4.1 shows that our networks have equally competitive classification performance as mainstream architectures such as ResNet. Furthermore, we show in Sec. 4.2 that our network is able to handle input perturbations with the robust inference technique. Finally, we demonstrate in Sec. [4.3 that our network is able to handle adversarial perturbations as well. More details about implementation are given in the Appendix.
Datasets. We test the performance of our method using the CIFAR-I0 and CIFAR- 100 (5T] datasets. Each dataset contains 50,000 training images and 10,000 testing images where each image is of size 32 x 32 with RGB channels. We also use the ImageNet dataset [52) that contains 1,000 classes and a total number of around I million images.
Network architecture. We use the network architectures with the first convolutional layers of ResNet-18 and ResNet-34 [21 [2 replaced by CSC-layers, and refer to these networks as SDNet-18
and SDNet-34, respectively. We use À = 0.1 as the trade-off parameter in (4) for all CSC-layers
3 CCIAR--1 10 and CIFAR- 100 are available at bestps://www.com tww.nto..oootooooooooooooo edu/ paaeronttooduu/kriieeee "kriiciiiiii html
'ImageNet is a publicly available dataset:
'We use the implementation at https:: / wtw.hhwwww.oo.ouggggggg//tttttre cifar which is under the
MIT License with Copyright (c) 2017 liukuang.

---

Table 1:
Comparison of different network archtectures, including ResNet, Multi-scale Deep Equalib- rium (MDEQ), Sparse Coding Network (SCN, SCN-first), and our SDNet, for image classification tasks. We report the number of model parameters (i.e., Model Size), accuracy on test data (i.e., Top-1 Acc), GPU memory consumption during training (i.e., Memory), and the number of images that are handled per second (n/s) during training (i.e., Speed).
Dataset
Architecture
Model Size
Top-1 Acc
Memory
Speed
ResNet-18 21
11.2M
95.54%
1.0 GB
1600 n/s
ResNet-34 21
21.1M
95.57%
2.0 GB
1000 n/s
MDEQ 27
11.1M
93.80%
2.0 GB
90 n/s
CIFAR-10
SCN [5]
0.7M
94.36%
10.0GB
39 n/s
SCN-18
11.2M
95.12%
3.
158 n/s
SDNet- 18 (ours)
11.2M
95.20%
GB
1500 n/s
SDNet-34 (ours)
21.1M
95.57%
2.4 GB
900 n/s
ResNet-18 21
11.2M
77.82%
GB
1600 n/s
ResNet-34 [2
21.1M
78.39%
GB
1000 n/s
MDEQ 27
11.2M
74.12%
GB
90 n/s
CIFAR-100
SCN 15
0.7M
80.07%
10.0GB
39 n/s
SCN-18
11.2M
78.59%
3.5 GB
158 n/s
SDNet- 18 (ours)
11.3M
78.31%
1.2
1500 n/s
SDNet 34 (ours)
21.2M
78.48%
2,4 GB
900 n/s
ResNet-18 21
11.7M
68.98%
24.1 GB
2100 n/s
ResNet-34 21
21.5M
72.83%
32.3 GB
1400 n/s
ImageNet
SCN [TS
9.8M
70.42%
95.1 GB
51 n/s
SDNet-18 18 (ours)
11.7M
69.47%
37.6 GB
1800 n/s
SDNet-34 (ours)
21.5M
72.67%
46.4 GB
1200 n/s
unless specified otherwise. Forward propagation through each CSC-layer is performed via unrolling two iterations of FISTA.
a cosine learning rate decay schedule with an initial learning rate of 0. and train the model for 220 epochs. We use the SGD optimizer with 0.9 momentum and Nestrov. The weight decay is 5 x 10-4, and batch size set to 128. All the experiments are conducted on a single NVIDIA GTX 2080Ti GPU. For ImageNet, we use multi-step learning rate decay schedule with an initial learning rate of 0. decayed by factor of 0.1 at the 30th, 60th, and 90th epochs, and train the model for 100 epochs. The batch size is 512, and the optimizer SGD with 0.9 momentum and Nestrov, All experiments on ImageNet are conducted on NVIDIA RTX 3090 GPUs.
Network training. For CIFAR-10 and CIFAR-100,
4.1 Performance for Image Classification
We compare our method with standard network architectures 18 and ResNet-34 [21]. In addition, we compare architectures with implicit layers (i.e., MDEQ (27)) and architectures with sparse modeling (i.e., SCN (55). For ResNet-18 18 and ResNet-34, we train the model using the same setup as our SDNet models. For MDEG and SCN, we train the models using the settings as stated in their respective papers. SCN has both different sparse coding layer and different network architecture compared to our SDNet. Hence, we also include a baseline referred to as SCN-18, which constructed by replacing the first convolutional layer of ResNet-18 with the sparse coding layer from SCN (hence has the same architecture as SDNet-18 18 but a different sparse coding layer).
The results are reported in Table [ T accuracy that closely matches (for CIFAR-10 and ImageNet) or surpasses (for CIFAR-100) that of ResNet-18/34 while having comparable speed. This shows the potential of our network with modeling based layers as a powerful alternative to existing data-driven models, since our model has the additional benefit of handling corruptions.
We see that with similar model size. SDNet- 18/34 produces
7

---

CFAR 10 Causian Mosse
CIRAR
Shot hobee
CIRAR 10 Speciee N652
CIFAR 10 impulse Rese
Leve Lovel
Test
Test
Lamidd
Lamrida
Larrbda
Lartoda
(a) Gaussian Noise
(b) Shot Noise
(c) Speckle Noise
(d) Impulse Noise
Figure 2: Test accuracy of SD trained on CIFAR-10 dataset with A = 0.1 and evaluated on 4
types of additive noise from CIFAR- 10-C (53] 5 severity levels each with varying values X. For each corruption type, optimal value of À for testing increases monotonically with the severity level.
We also compare our SDNet-18 model with the MDEQ model, which has a similar model size, and that SDNet-18 is not only more accurate than MDEQ but is much (> 7 times) faster. Note that MDEQ cannot handle corrupted data as in our method as well.
The SCN network, which also uses sparse modeling, obtains Top-1 accuracy that is highly competi- tive to all methods. However, a significant drawback of SCN is that it is very slow to train. This is true even with SCN-18, where only one convolutional layer is replaced by the sparse coding layer. The reason may be that SCN uses a patch-based sparse coding model for images, in contrast to a convolutional sparse coding model in method, which requires solving many sparse coding problems in each forward propagation that cannot benefit from parallel computing.
4.2 Handling Input Perturbations
To test the robustness of our method to input perturbations, we use the CIFAR-10-C dataset 153
which contains test set for CIFAR-10 that corrupted with different types of synthetic noise and 5 severity levels for each type. Because the CSC model in (4) penalizes the entry-wise difference between input and reconstructed signals, it is more suited for handling additive noises. Hence, we focus on four types of additive noises in CIFAR-IO-
namely, Gaussian noise, shot noise, speckle noise, and impulse noise. We evaluate the accuracy of our SDNet-18 and compare its performance with ResNet-18.
Robustness as a function of X. As discussed in Sec. 3.3 model to noisy test data by using values of À that are different from the training phase. Therefore, we report the performance our method with varying À in the range of [0.1, 1.5] Figure 2 (recall that À = 0.1 is used for training). It can be seen that for all types of noises and all severity levels (except for impulse noise with levels O, 1, and 2), properly choosing value of À that is different from that used during training helps improve the test performance. In particular, the accuracy curves as a function of X exhibit a unimodal shape where the performance first increases and then decreases. Moreover, within each corruption type the values of where peak performance is achieved increase monotonically with the severity level of the corruption. Such an observation is well-aligned with our discussion in Sec. 3.3
we may improve the performance of our
Choosing an optimal X. While Figure 2 demonstrates that one may improve the performance on corrupted data via proper choice of X. it does not show how to choose the best À in practice. Here we show that the technique presented in Algorithm Can be used to select for robust inference. Specifically, we apply Algorithm with [(: 0, Ao) being SDNet- with λo = 0.1, Tiain being the CIFAR-10 10 training set, Test being LIFAR-I0-CA data with a particular type of corruption under a particular severity level, and report the performance of the algorithm output in Table Lable and Table 3
Table 2 shows the results with varying severity levels of Gaussian noise. We can see that if we use the same À = 0.1 as in training (i.e., SDNet- 18 w/ X == 0.1), then the performance of SDNet- 18 already significantly better ResNet-18. This may be attributed to the fact that the CSC-layer is intrinsically more robust to input perturbations than convolution layers owning to its sparse modeling. However, if we use an adaptive À computed from Algorithm (i.e.. SDNet- w/ adaptive A), then the performance is further significantly improved at all noise levels. To demonstrate that our method works beyond Gaussian noise, we report results for each of the four types of additive noises averaged over all severity levels in Table 3 for CIFAR- 10-C well ImageNet-C, The results demonstrate
8

---

Table 2: Classification result under varying severity levels of Gaussian noise on CIFAR-1O-C with
0.1 (i.e., same as training) and with an adaptive computed from Algorithm
Severity Level
Level-0
Level-1
Level-2
Level-3
Level-4
ResNet-18 [21
79.43%
56.17%
34.86%
28.23%
23.45%
SCN [T
80.89%
60.21%
44.97%
37.79%
30.11%
SDNet-18 w/ = 0.1
81.78%
63.50%
43.86%
35.84%
27.92%
SDNet-18 w/ adaptive À
84.76%
74.87%
61.38%
54.77%
48.84%
À from linear fitting
0.49
0.60
0.75
0.84
0.94
Table 3: Classification result under varying corruption types (averaged over all severity levels for each type) on CIFAR-IO-C and ImageNet-C with À 0.1 (i.e., same as training) and with adaptive À computed from Algorithm
CIFAR-10-C
ImageNet-C
Noise Type
Gaussian
Shot
Speckle
Impulse
Gaussian
Shot
Impulse
ResNet- 18 [21]
44.43%
57.88%
62.16%
51.72%
22.73%
21.78%
17.38%
SCN [15
50.79%
62.97%
67.45%
54.19%
SDNet-18 w/ À = 0.1
50.58%
63.29%
67.11%
54.13%
24.98%
23.97%
19.12% 22.01% %
SDNet-18 w/ adaptive
64.92%
71.13%
71.42%
57.48%
29.16%
27.59%
that sparse modeling enables us to effectively handle various types of additive noises test data very easily with the procedure in Algorithm
Coursiinn
Koice
Speckle Nobse
Impuite Moice
Recorametion Erw
Retorstuction Ertrr
Recordraction Entrr
Reconctruttinn Entor
(a) Gaussian Noise
(b) Shot Noise
(c) Speckle Noise
(d) Impulse Noisc
Figure 3: Relation between the optimal choice of À and the magnitude of residual in CSC-layers (i.e.. (Ac, rabeec in Algorithm I drawn red crosses) as well as the linear fitting (i.e., À = X(c) in Algorithm drawn as black lines) for four corruption types T E (Gaussian, Shot, Speckle, Impulse].
Finally, in Figure 3 we plot the relationship between the optimal choice of X (in y-axis) and the magnitude of residual in CSC-layers (in x-axis) learned from training data according Algorithm We also plot the linear fitting of such a relation, which can be seen to provide a good quality approximation for each of the four corruption types. On the other hand, note that the fitted linear relations differ across different types of corruption. Hence, it is important that the relationship is estimated for each corruption type separately.
4.3 Handling Adversarial Perturbations
We show that our method also exhibits robustness to adversarial perturbations. In this experiment, we generate adversarial perturbations on the CIFAR- 10 test dataset using PGD attack on our SDNet (with
A = 0.1), with Lx norm of the perturbation being e = 8/ 8/255 and L2 norm of the perturbation being e = 0.5, respectively. The robust accuracy of our method reported in Table and compared with that of ResNet-18. We can see that while SDNet does not perform much better than ResNet with 0. we may tune the parameter À to drastically improve the robust accuracy.
4.4 Analysis
Effect of number of iterations. Recall from Sec. 3.1 that forward propagation through CSC-layer performed by running a few iterative steps of the FISTA algorithm. Here we provide study on how the number of iterations affects model performance on ImageNet and ImageNet-C using 18. The results are shown Table With the increasing number of FISTA iterations, the model performance on both natural accuracy and robust accuracy improves.

---

Table 4: Robust accuracy on CIFAR-I0 with adversarial perturbation using PGD attack.
Robust Accuracy
Robust Accuracy
Model
(Loe 8/255)
(L2 0.5)
ResNet-18 18
21
0.01%
29.47%
SDNet-18 w/ X 0.1
0.1 11
29.95% 62.80%
SDNet-18 (After tuning A)
35.18%
Table 5: Effect of number of FISTA iteration on natural and robust accuracy (evaluated with ImageNet- C) for SDNet-18 trained on ImageNet.
# of FISTA Iterations
Natural Accuracy
Gaussian
Shot
Impulse
2
69.47%
29.16%
27.5 27.59%
22.01%
4
69.51%
29.69%
28.15%
24.15%
8
69.79%
30.91%
29.87%
25.69%
Replacing all convolution layer by the CSC-layer. We also train version of SDNet obtained from replacing all convolution layers of a ResNet with the CSC-layer (as opposed to only the first convolution layer), and refer to such model as SDNet-18-All and SDNet-34-All. On ImageNet we observe that SDNet-18-AN and SDNer-34-AR 34lAll obtain 69.37% and 72.54% Top-I accuracy, respectively, Comparing such results with those of SDNet-18/34 reported in Table [ is not significantly affected by replacing more convolution layers with CSC-layers (see Table D. in Appendix for more results). Moreover, 1NN34-18/44AA11 enables us to develop a visualization technique as described in the Appendix B.
that the performance
Conclusion and Discussion
This paper revisits the classical sparse modeling and provides a simple way of using to guide the design of interpretable deep networks. Despite multiple prior attempts, work is the first to demonstrate that such design can produce performance (in terms accuracy, model size, and memory) that is par with standard ConvNets modern image datasets such ImageNet. The success in combining sparse modeling with deep learning provides a means of borrowing and utilizing the rich results in the well-developed field of sparse modeling 54 55 for network design and analysis. While it is not the purpose of this work to fully explore all potentials, our experiments already demonstrate clear advantages of so-designed networks in handling various forms of data corruption. Looking forward, other fundamental principles, algorithms, and techniques in sparse modeling may be introduced to further enhance the capability our presented framework. Along this line, we provide some preliminary evidence on how sparse modeling enables interpretability in the Appendix, and leave further study to future work.
Acknowledgments and Disclosure of Funding
Zhihui Zhu acknowledges support from NSF grants CCF-2240708, Shao-Lun Huang acknowledges support from Shenzhen Science and Technology Program under Grant CQTD2017081015000 146, National Key R&D Program of China under Grant 2021 YFA07 15222 and High-end Foreign Expert Talent Introduction Plan under Grant G2021032013L. Yi Ma acknowledges support from ONR grants N00014-200 1-2022 and 10001422--1-2000 the joint Simons Foundation-NSF DMS grant #2031 1899, as well as partial support from Berkeley FHL Vive Center for Enhanced Reality and Berkeley Center for Augmented Cognition, Teingeua-Berkeley Shenzhen Institute (TBSI) Research Fund, and Berkeley AI Research (BAIR).
References
[1] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
10

---

[2] Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381(6583):607-606 1996.
[3] Daniel A Spielman, Huan Wang. and John Wright. Exact recovery of sparsely- dictionaries. In Conference on Learning Theory, pages 37-1. JMLR Workshop and Conference Proceedings, 2012.
[4] Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2):853-884 2016.
[5] Yuqian Zhang, Han-Wen Kuo, and John Wright. Structured Local Optima in Sparse Blind Deconvolution. IEFE Transactions on Information Theory, (6(1))419945 2019.
[6] Qing Qu, Yuexiang Zhai, Xiao Li, Yuqian Zhang, and Zhihui Zhu. Geometric Analysis of Nonconvex Optimization Landscapes for Overcomplete Learning, In International Conference on Learning Representations, 2019.
[7] Yuexiang Zhai, Zitong Yang. Zhenyu Liao, John Wright, and Yi Ma. Complete Dictionary Learn- ing via N Maximization over the Orthogonal Group. J. Mach. Learn. Res., 21(165):1-68, 2020.
[8] Yuexiang Zhai, Hermish Mehta, Zhengyuan Zhou, and Yi Ma. Understanding ba-ba Dic- tionary Learning: Interpretation, Stability, and Robustness. In International Conference on Learning Representations, 2019.
[9] Julien Mairal, Francis Bach, and Jean Ponce. Sparse modeling for image and vision processing. arXiv preprint arXiv: /4/ 1.3230, 2014.
[10] Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research, 18(1):8887-2938 2017.
[11] Jeremias Sulam, Vardan Papyan, Yaniv Romano, and Michael Elad. Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Trunsactions on Signal Processing, 66(15):409044104, 2018
[12] Aviad Aberdam, Jeremias Sulam, and Michael Elad. Multi-layer sparse coding: The holistic way. SIAM Journal on Mathematics of Data Science, 1(1):46-77, 2019.
[13] Zhiyang Zhang and Shihua Zhang. Towards understanding residual and dilated dense neural networks via convolutional sparse coding. National Science Review, 8(3):nwaal 1.wal159, 159, 2021.
[14] George Cazenavette, Calvin Murdock, and Simon Lucey. Architectural adversarial robustness: The case for deep pursuit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 71 7150-7158, 2021.
[15) Xiaoxia Sun, Nasser M Nasrabadi, and Trac D Tran. Supervised deep sparse coding networks. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 346-350. IEEE, 2018.
[16] Xiaoxia Sun, Nasser M Nasrabadi, and Trac D Tran. Supervised deep sparse coding networks for image classification. IEEE Transactions on Image Processing, 29:405-4133 2019.
[17] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference Machine Learning, pages 136-145. PMLR, 2017.
[18] Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. arXiv preprint arXiv:1910.120 12430, 2019.
[19] Stephen Gould, Richard Hartley, and Dylan Camphell. Deep declarative networks: A new hope. 2019.
arXiv preprint arXiv:1909.00
[20] Mattew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional net- works. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition, pages 2528-2535 IEEE, 2010.
11

---

[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006. 16241, 2020.
[23] Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 4480-4488, 2016.
[24] Julien Mairal, Francis Bach, and Jean Ponce. Task-driven dictionary learning. IEEE transactions on pattern analysis and machine intelligence, 34(4))791-804, 2011.
[25] Laurent EI Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit deep learning. arXiv preprini arXiv: 1908 2, 2019.
[26] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32:690-701, 2019.
[27] Shaojie Bai, Vladlen Koltun, and Zico Kolter. Multiscale deep equilibrium models. arXiv preprint arXiv:2006. 08656, 2020.
[28] Tiancai Wang, Xiangyu Zhang, and Jian Sun. Implicit feature pyramid network for object detection. arXiv preprint arXiv:2012.1 13563, 2020.
[29] Sheng Liu, Xiao Li, Yuexiang Zhai, Chong You, Zhihui Zhu, Carlos Fernandez- Granda, and Qing Qu Convolutional normalization: Improving deep convolutional network robustness and training. Advances in Neural Information Processing Systems, 34:28919-28928, 2021.
[30] Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. Advances in Neural Information Processing Systems, 30:1013- 2017.
[31] Brandon Amos. Differentiable
satiiiizzatio-bbaedd modeling for machine learning. PhD) thesis, PhD thesis. Carnegie Mellon University, 2019.
[32] Zhaowen Wang, Ding Liu, Jianchao Yang. Wei Han, and Thomas Huang. Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE international conference on computer vision, pages 370-378, 2015.
[33] Meyer Scetbon, Michael Elad, and Peyman Milanfar, Deep k-svd denoising. arXiv preprint ArXiv: 13164, 2019.
[34] Hillel Sreter and Raja Giryes. Learned convolutional sparse coding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2191-2195, IEEE, 2018.
[35] Bruno Lecouat, Jean Ponce, and Julien Mairal. Fully trainable and interpretable non-local sparse models for image restoration. European Conference on Computer Vision (ECCV) 2020. Springer, 2020.
[36] Bruno Lecouat, Jean Ponce, and Julien Mairal. A flexible framework for designing trainable priors with adaptive smoothing and game encoding. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
[37] Tianlin Liu, Anadi Chaman, David Belius, and Ivan Dokmanié. Interpreting u-nets via task- driven multiscale dictionary learning. arXiv preprint 1:Xiv:2011.1281 2020.
[38] Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. Proceedings of the 27th international conference on international conference on machine learning, pages 399-406, 2010.
[39] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016.
12

---

[40] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint (Xiv:1705.002044 2017.
[41] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pages 274-283. PMLR. 2018.
[42] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 12231, 2018.
[43] Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. arXiv preprint arXiv: 08988, 2019.
[44] Soorya Gopalakrishnan, Zhinus Marzi, Upamanyu Madhow, and Ramtin Pedarsani. Robust adversarial learning via sparsifying front ends. arXiv preprint arXiv: 10100.05000 2018.
[45] Yaniv Romano, Aviad Aberdam, Jeremias Sulam, and Michael Elad. Adversarial noise attacks of deep learning architectures: Stability analysis via sparse-modeled signals. Journal of Mathematical Imaging and Vision, 62(3):313-322 2020.
[46] Jeremias Sulam, Ramchandran Muthukumar, and Raman Arora. Adversarial robustness of supervised sparse coding. Advances in Neural Information Processing Systems, 33:21 10-2121, 2020.
[47] Amir Beck and Mare Teboulle. A fast iterative serrnkagg--hreshooddi algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.
[48] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 36(11):2278-2324 1998.
[49] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[50] Vardan Papyan, Jeremias Sulam, and Michael Elad. Working locally thinking globally: Theo- retical guarantees for convolutional sparse coding. IEEE Transactions on Signal Processing, 65(21):5687-5701 2017.
[51] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[52] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 Icce, 2009.
[53] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[54] Simon Foucart and Holger Rauhut. A mathematical introduction compressive sensing. Bull. Am. Math
2017.
[55] John Wright and Yi Ma. High- Dimennsiin Data Analysis with Low-Dimensional Models: Principles, Computation, and Applications. Cambridge University Press, 2021.
13


Reviews:
[{"review_id": "e_vbTgs8Jp1", "reviewer": "Reviewer_nUHw", "summary": "This paper proposed the convolutional sparse coding layer (CSC-layer) to obtain performance comparable to standard ConvNets with better interpretability and stability.  By conducting extensive experiments on CIFAR-10, CIFAR-100, and ImageNet, the model with CSC-layer has shown to achieve comparable performance as standard ConvNets and be more robust to data perturbation than the standard one.", "questions": "1. In Table 1, the model size of SDNet-18 and SDNNet-34 on CIFAR-100 are much smaller than on CIFAR-10, which seems wired. \n2. With similar performance, the proposed method is much faster than its baselines. In Table 1, the proposed SDNet only replaces the first convolutional layer with CSC-layer while SCN is a multilayer sparse coding network. For a fair comparison, this paper may compare the time and memory consumption of a single sparse coding layer between those methods. \n3. Each layer of CSC-layer of SDNet-18 and SDNet34 needs unrolling two iterations of FISTA and more iterations will only slightly improve the performance. As SDNet-18 and SDNet-34 have only one CSC-layer for the input images, I’m curious whether it is this low dimension (3 channels) of input that make two iterations sufficient. On SDNet-18-All and SDNet-34-All, could you list the dimension of the input and output of each CSC-layers and their corresponding iterations used? \n4. What is the cost to find the optimal $\\lambda$ and calculate the residual in Algorithm 1? \n5. I would like to confirm whether it is $z$ of a CSC-layer that will be the input of the next layers. \n6. Does FISTA algorithm always randomly initialize $z$ for any CSC-layer in any iteration during the training? If it is, is it possible to initialize it with the previous leaned values for an image when the model sees it again? This may reduce the number of FISTA iterations. \n", "limitations": "This paper introduced an appealing method to apply CSC-layer for robustness inference. It is great if this paper has a more detailed comparison of its time and memory for both training and inference with the other methods. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper proposed an interesting and effective idea for robust inference with the proposed convolutional sparse coding layer and achieved impressive performance on both image corruption and adversarial attack. \n\nThe weaknesses mainly lay in the cost of the training and robustness inference: \n1. This paper claims the fast training speed as one of its contributions. However, this needs careful comparison with its baselines. \n2. Although this method does not require modifying the training procedure like the existing methods to obtain robustness, it has additional costs to get the optimal $\\lambda$ for robustness inference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JlPDGbyfO0j", "reviewer": "Reviewer_WxVJ", "summary": "The paper proposes to replace the traditional convolutional layers with convolutional sparse coding (CSC) layers, claiming that such substitution adds interpretability and robustness to neural networks.", "questions": "I have three main concerns about the claims in the paper.\n\n**1. Interpretability**. The traditional convolutional layer performs a forward computation (the output is a linear combination of the inputs). In contrast, the convolutional sparse coding (CSC) layer performs a backward computation (the input is a linear combination of the outputs). It is not apparent why a backward computation is more interpretable than a forward one. In my opinion, it is not an individual layer that makes a neural network hard to interpret but the stack of these layers. While convolution layer and convolutional sparse coding are easy to interpret individually, using them in deep networks (with nonlinearities, normalization, etc.) is not.\n\n**2. Robustness**. The explanation of why the CSC layer is more robust is insufficient --- no math derivation is used to explain this concept. From the writing, it seems the Lipschitz constant decreases if the regularizer increases. In this case, the comparison to standard convolutional networks is not fair. Maybe the authors would also like to enforce the Lipschitzness of traditional convolutional networks (e.g., following https://arxiv.org/abs/1804.04368).\n\n**3. Computational Complexities**. It looks to me that the proposed layer is quite expensive. In the experiment, only one layer in ResNet is replaced by the proposed layer, and only two iterations are used in unrolling. And this already decreases the speed from 1000 to 900. I think a more comprehensive study on the relationship between accuracy, complexity, and iterations is needed when all layers are replaced.", "limitations": "Not applicable.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The writing is generally good and easy to follow; however, the technical claims in the paper are not well supported by empirical evidence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7CxjdvC29S", "reviewer": "Reviewer_EVjq", "summary": "This paper proposes an approach to incorporate convolutional sparse coding into deep neural networks. In particular, a convolutional layer of a ResNet is replaced with an implicit layer, referred to as convolutional sparse coding layer (CSC-layer), which outputs a sparse representation (a feature map) of its input given a (learned) dictionary of convolutional filters. The sparse representation for an input is computed using the FISTA algorithm. The experiments show that ResNet models whose first convolutional layer is replaced by such a CSC-layer can achieve similar or better classification performance on the CIFAR-10, CIFAR-100, and ImageNet datasets. Additionally, experiments suggest that ResNet models with a CSC-layer are more robust to different kinds of additive noise applied to the input when compared to a regular ResNet model.", "questions": "1. Have the authors considered visualizing the learned sparse dictionary convolutional kernels common in related literature (eg. [1])? I believe this would help with interpretability and understanding what the dictionary of convolutional kernels encodes.\n\n2. Typically, the FISTA algorithm requires hundreds of iterations to converge so my expectation is that the reconstructions $\\tilde{x} = \\mathcal{A}(z_*)$ with only 2 iterations are not high fidelity (e.g., terms of PSNR). This is supported by the visualization in Appendix B2 which shows that feature maps only encode contours or high-level information about the input. The authors mention that increasing the number of FISTA iterations can boost the classification performance a bit. Have the authors’ studied how increasing the number of FISTA iterations affects the model’s robustness to noise or can they provide intuition about it?\n\n3. My understanding is that only the first convolutional layer of ResNet-18 and ResNet-34 (the one closest to the input) is replaced by a CSC-layer. Is this correct or does “the first convolutional layers” (line 235) refer to the first convolutional layer of each ResNet block? \n\n4. How is the value of $\\lambda = 0.1$ used during training selected? What is the size of $C$ used in experiments, i.e. the number of sparse feature maps in $z$ (line 125)? How sparse on average are the feature maps output by FISTA when only 2 iterations are used with regularization coefficient $\\lambda = 0.1$?\n\n5. What magnitudes do levels 0-6 in Figure 2 correspond to for each type of noise? E.g. for Gaussian noise, what levels of noise are considered? Same for Tables 2 and 3.\n\n6. Have the authors studied how robust SD SCN model (from [3]) is to additive noise? Would be insightful to add it as a baseline in Tables 2 and 3.\n\n7. Are the authors planning to provide an implementation of the proposed framework?\n\n8. Nitpicks\n- Figures 2 and 3 would be easier to read if the font size in is bigger. \n\n[1] Kavukcuoglu, K., Sermanet, P., Boureau, Y.L., Gregor, K., Mathieu, M. and LeCun, Y., 2010. Learning convolutional feature hierarchies for visual recognition. Advances in neural information processing systems, 23.\n\n[2] Beck, A. and Teboulle, M., 2009. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1), pp.183-202.\n\n[3] Sun, X., Nasrabadi, N.M. and Tran, T.D., 2018, October. Supervised deep sparse coding networks. In 2018 25th IEEE International Conference on Image Processing (ICIP) (pp. 346-350). IEEE.\n", "limitations": "The authors are upfront about the limitations of higher memory and inference costs of unrolling FISTA for a higher number of iterations. They could elaborate on the potential negative societal impacts, for example, the fact that the model can propagate any bias present in the dataset.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The proposed method which combines sparse coding with deep neural networks is scalable to datasets such as ImageNet.\n\n- With only 2 iterations of FISTA, the proposed modification of the ResNet model is more robust to additive noise in the input than the vanilla ResNet architecture when evaluated on the task of classification.\n\n- The paper is organized and written clearly. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept