A Simple Sparse Denoising Layer for Robust Deep Learning
Authors: Yueming Lyu
Xingrui Yu
Ivor Tsang
Venue:  (2021)
Keywords: Sparse Coding
Deep Learning

Abstract: Deep models have achieved great success in many applications. However, vanilla deep models are not well-designed against the input perturbation.   In this work, we take an initial step to designing a simple robust layer as a lightweight plug-in for vanilla deep models.  To achieve this goal, we first propose a fast sparse coding and dictionary learning algorithm for sparse coding problem with an exact $k$-sparse constraint or  $l_0$ norm regularization. Our method comes with a closed-form approximation for the sparse coding phase by taking advantage of a novel structured dictionary.   With this handy approximation,  we propose a simple sparse denoising layer (SDL) as a lightweight robust plug-in.   Extensive experiments on both classification and reinforcement learning tasks manifest the effectiveness of our methods.

Under review as a conference paper at ICLR 2021
A SIMPLE SPARSE DENOISING LAYER FOR ROBUST DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep models have achieved great success in many applications. However, vanilla deep models are not well-designed against the input perturbation. In this work, we take an initial step to design a simple robust layer as a lightweight plug-in for vanilla deep models. To achieve this goal, we first propose a fast sparse cod- ing and dictionary learning algorithm for sparse coding problem with an exact k-sparse constraint or lo norm regularization. Our method comes with closed- form approximation for the sparse coding phase by taking advantage of a novel structured dictionary. With this handy approximation, we propose simple sparse denoising layer (SDL) as a lightweight robust plug-in. Extensive experiments on both classification and reinforcement learning tasks manifest the effectiveness of our methods.
INTRODUCTION
Deep neural networks have obtained a great success in many applications, including computer vi- sion, reinforcement learning (RL) and natural language processing, etc. However, vanilla deep models are not robust to noise perturbations of the input. Even a small perturbation of input data would dramatically harm the prediction performance (Goodfellow al. 2013).
To address this issue, there are three mainstreams of strategies: data argumentation based learning methods (Zheng al., 2016; Ratner et al., 2017 Madry al.,|2201.1 2018 Cubuk et al. (020), loss func- tions/regllarizatoon techniques (Flsayed et 2018
et al. 2019),, and importance weighting of network architecture against noisy input perturbation. (Su al. (2018) empirically investigated 18 deep classification models. Their studies found that model architecture is a more critical factor robustness than the model size. Most recently, Guo al. (2020) employed neural architecture search (NAS) method to investigate the robust architectures. However, the NAS-basee methods are still very computationally expensive. Furthermore, their resultant model cannot be easily adopted as a plug-in for other vanilla deep models. A handy robust plug-in for backbone models remains highly demanding.
In this work, we take initial step to design simple robust layer lightweight plug-in for the vanilla deep models. To achieve this goal, we first propose a novel fast sparse coding and dictionary learning algorithm. Our algorithm has closed-form approximation for the sparse coding phase, which is cheap to compute compared with iterative methods in the literature. The closed- form update is handy for the situation that needs fast computation, especially in the deep learning- Based on this, we design a very simple sparse denoising layer for deep models. Our SDL is very flexible, and it enables end-to-end training. Our SDL can be used lightweight plug-in for many modern architecture of deep models (c.g., ResNet and DenseDet for classification and deep PPO models for RL). Our contributions are summarized as follows:
We propose simple sparse coding and dictionary learning algorithms for both k-sparse con- strained sparse coding problem and fo-mrmm regularized problem. Our algorithms have simple approximation form for the sparse coding phase.
We introduce a simple sparse denoising layer (SDL) based on our handy update. Our SDL. involves simple operations only, which is fast plug-in layer for end-to-end training.
Extensive experiments on both classification tasks and reinforcement learning tasks show the effectiveness of our SDL..

---

Under review as a conference paper at ICLR 2021
2 RELATED WORKS
Sparse Coding and Dictionary Learning: Sparse coding and dictionary learning are widely stud- ied in computer vision and image processing. One related popular method is K-SVD (Elad & Aharon, 2006, Rubinstein et al.) 2008), it jointly learns an over-complete dictionary and the sparse representations by minimizing lo-norm regularized reconstruction problem. Specifically, K-SVD alternatively iterates between the sparse coding phase and dictionary updating phase. The both steps based on heuristic greedy methods. Despite its good performance, K-SVD is very computation- ally demanding. Moreover, as pointed out by [aao et al. (2013), both the sparse coding phase and dictonary updating of K-SVD use some greedy approaches that lack rigorous theoretical guarantee on its optimality and convergence. |Bao et al.] (2013) proposed to learn an orthogonal dictionary instead of the over-complete one. The idea to concatenate the free parameters with predefined filters to form an orthogomal dictionary. This trick reduces the time complexity compared with K- SVD. However, their algorithm relies on the predefined filters. Furthermore, the alternative descent method heavily relies on SVD, which is not easy to extend to deep models.
In contrast, our method learns a structured over-complete dictionary, which has a simple form as a layer for deep learning. Recently, some works (Venkatakrishnan et al., (013) employed deep neural networks to approximate alternating direction method of multipliers (ADMM) or other proximal algorithms for image denoising tasks. In (Wei et al. 2020). reinforcement learning is used to learn the hyperparameters of these deep iterative models. However, this kind of method itself needs to train complex deep model. Thus, they are computationally expensive, which is too heavy or inflexible as a plug-in layer for backbone models in other tasks instead of image denoising tasks, e.g., reinforcement learning and multi-clase classification, etc. An illustration of number of parameters of SDL, DnCNN (Zhang et al. 2017) and PnP (Wei et al., 2020) are shown in Table [ SDL has much less parameters and simpler structure compared with DrCNN and PnP, and serve lightweight plug-in in for other tasks, e.g., RL.
Table 1: Number of parameters of different models
SDL DrCNN (Zhang et al. 2017) PnP (Wei et al. 2020)
#parameters 630
667,008
296.640
Robust Deep Learning: the literature of robust deep learning, several robust losses have been studied. To achieve better generalization ability. Elsayed et al. (2018) proposed loss function to impose large margin of any chosen layers of deep network. Barron (2019) proposed a general loss with a shape parameter to cover several robust losses as special cases. For the problems with noisy input perturbation, several data argumentation-
algorithms and regularization techniques proposed (Zheng et al. [016 Ratner et al. 20177; Cubuk et al. 2020; Elsayed et al., [2018] Zhang eet al. 2019). However, the network architecture remains less explored to address the robustness of the input perturbation. Guo et al. (2020) employed NAS methods search the robust architectures. However, the searching- method is very computationally expensive. The resultant architectures cannot be easily used as a plug-in for other popular networks. In contrast, our SDL is based on closed-form of sparse coding, which can be used as a handy plug-in for many backbone models.
FAST SPARSE CODING AND DICTIONARY LEARNING
In this section, we present our fast sparse coding and dictionary learning algorithm for the k-sparse problem and the fo-norm regularized problem in Section 3.1 and Section 3.2, respectively. Both algorithms belong to the alternative descent optimization framework.
3.1 K- S-SAASSE CODING
We first introduce the optimization problem for sparse coding with a k-sparse constraint. Mathe- matically, we aim at optimizing the following objective
min YD
X
1
subject to
||yello k. E (.....,.)
(1)
p (D)= <
|| = 1, (1.....M
2

---

Under review as a conference paper at ICLR 2021
where D R is the dictionary, and d; denotes the tth column of matrix D. y/ denotes the ith column of the matrix Y E IR MgM and fu(-) denotes the mutual coherence that defined as
d d d. 1d11
" D) = max
(2)
The optimization problem T discrete and non-convex, which very difficult to optimize. To alleviate this problem, we employ structured dictionary as
D
We require that R La and BB = La, and each column vector of matrix B has constant 12-norm, i.e., | = c. The benefit of the structured dictionary is that it enables a fast update algorithm with closed-form approximation for the sparse coding phase.
3.1.1 CONSTRUCTION OF STRUCTURED MATRIX B
Now, we show how design structured matrix B that satisfies the requirements. First, we con- struct B by concatenating the real and imaginary parts of rows of a discrete Fourier matrix. The proof of the following theorems regarding the properties of B can be found in Appendix.
Without loss of generality, we assume that d 2m, M = 2n. Let F E C" be an n X zedk?
discrete Fourier matrix. Fk. ( k2,,.. km C {I,..., it - }} be subset of indexes.
is the (k, e of F, where i --1. Let A
The structured matrix B can be constructed as Eq.(4).
B
ReFA
IImAA
RdxN
(4)
ImFA
ReFa
where Re and Im denote the real and imaginary parts of complex number, and FA in Eq. the matrix constructed by m rows of F
FA-
E
CmR
(5)
Proposition Suppose d 2m, M 2n. Construct matrix as in Eq. Then BBT and |b,1|2 / Vj (1,...,.. M...
Id
Theorem shows that the structured construction B satisfies the orthogonal constraint and constant norm constraint. One thing remaining is how construct B to achieve small mutual coherence.
To achieve this goal, we can leverage the coordinate descent method in (Lyu 2017) to construct the index A. For prime number such that divides 1, i.e., [n 1), can employ closed-form construction. Let g denote primitive root modulo n. We construct the index A =
{k1, h2, ... }m} as
A [g g
1Xn-1 mod
(6)
The resulted structured matrix B has bounded mutual coherence, which is shown in Theorem J Theorem 1. Suppose 2m, M 2n, and n is prime such that ml - 1). Construct matrix B as in Eq. with index set as Eq.66) Let mutual coherence μ( BB) = maxiys
bb, byl Then
b. b,
( <
v
Remark: The bound of mutual coherence in Theorem is non-trivial when m² For the case n 2 m², we can use the coordinate descent method LYu 2017 to minimize the mutual coherence.
Now, we show that the structured dictionary D = R B satisfies the constant norm constraint and has bounded mutual coherence. The results summarized in Theorem
3

---

Under review as a conference paper at ICLR 2021
Corollary 1. Let D = B with R R RR Ig. Construct matrix B in Eq. with index set as Eq. Then = I v m
and | 212 = | 022 v Vj E fl.... M).
T that, for any orthogonal matrix R. each column vector of the structured dictionary D has a constant lo-norm. Moreover, it remains a constant mutual coherence (u(D) (B). Thus, given fixed matrix B, we only need learn matrix R for the dictionary learning without undermining the low mutual coherence property.
3.1.2 JOINT OPTIMIZATION FOR DICTIONARY LEARNING AND SPARSE CODING
With the structured matrix B, we can jointly optimize and for the optimization problem 7
min X X - R BY
subject to llv/llo S k, E {1, , NN R RR = Ia
This problem can be solved by the alternative descent method. For fixed we show the sparse representation Y has a closed-fore approximation thanks to the structured dictionary. For the fixed sparse codes Y, dictionary parameter R has a closed-form solution.
Fix R, optimize Y: Since the constraints of Y is column separable, i.e., l|vello S k, and the objective (8) also decomposable,
N Σ i=1
X R BY
x R By
(8)
It is sufficient to optimize the sparse code y: RM for each point æ, Rd separately.
Without loss of generality, for any input x E Rd, we aim at finding the optimal sparse code y E RM
such that | k. Since R R RR = Ia and BB
II we have
| R Byll
Rx RR Byl
Ra Byll2
|BB T Rx Byl
B Rx - -
B y) 12-
(9)
where h T Ræ is dense code. Case 1: can rewrite (9. into a summation form as
When m (the columns of B are orthogonal), we
M 1,,
[Bh 2)1l
Σ (h,
(10)
Case When m < n, we have an error-bounded approximation using R.H.S. in Eq.(10). Let
h - y. we have
M
M
Bz
1
Σ
Σ
2,2, by b,
(11)
1=1
1=1
f= 1,
M
M
1242y) ||11:111110 by (1
(12)
f=1 j=1Øfi
M
M
Σ
|z,zz|
(BB)
(13)
1=1 1ー
4

---

Under review as a conference paper at ICLR 2021
It is worth note that the error bound is small when the mutual coherence (BB) is small. When employ the structural matrix in Theorem It follows that
M
M
M
Bz 1 - Σ
m
min
7 1)
(14)
j=1
j=1
M
M
(15)
i=
M
Σ
[4 - villhy V3l
(16)
where C' = min( 1 ). In 14). we use (B S v from Theorem
Considering the sparse constraint Iolllo I k. the error bound is minimized when all the non-zero term My h to get My hyl O. Let S denote the set of index of non-zero element yj of y. Now the problem is to find the index set S to minimize
M
M
Σ プ=ر
M4
y: villi;
Myl
Σ
Σ
(17)
iES JESSjj
where S denotes the complement set of S. We can see that Eq. (17 is minimized when S consists of the index of the largest (in absolute value) elements of h.
Now, we consider Σ/ 2311b, 11.. Note that b,, = m follows that
M
Σ
11,
n (hy 01)
Σ
(18)
Because each term (h:- )3 ) 2 0 is minimized when yj h3. we know that Eq. (9 under sparse
constraints minimized when all the non-zero term setting as yj = hg. Otherwise we can set a non-zero term Yj to yy h, to further reduce term (hy yソ )2 zero.
Now, the problem is to find the index set of the non-zero term to minimize Eq. 19
M
Σ
(h v2)
h
(19)
SE
where S = fily + 0}. We can see that Eq. 19 is minimized when S consists of the index of the k largest (in absolute value elements of h.
Remark: Both the approximation Σ 2) bbbb1 and the error bound minimized by the same solution.
Fix Y, Optimize R : For a fixed Y, we know that
| R BY = | + BY 2tr R BYX
(20)
This is the nearest orthogonal matrix problem, which has a closed-form solution as shown in (Schönemann, 1966, Gong et al. 2012). Let BY X T UTV obtained by singular value decomposition (SVD), where U, V are orthgonal matrix. Then, Eq. 20 is minimized by
R R-UVT
(21)
3.2 40 40-NORM REGULARIZATION
We employ the structured dictionary D R B same as Section 3.1 with norm regularization is defined as
The optimization problem
min X
R BY A
subject to R R R..
(22)
5

---

Under review as a conference paper at ICLR 2021
SDL Z f(B'RZ)
X
Z
1(0; X)
+
||z 2|1
Figure 1: Illustration of the SDL Plug-in
This problem be solved by the alternative descent method. For fixed R, we show Y has a closed-form approximation thanks to the structured dictionary. For fixed the sparse codes Y, dictionary parameter R also has closed-form solution.
Fix R, optimize Y: Since the objective can be rewritten as Eq.23
N
| - R 1|
Σ
a R Byill? + Allyello-
(23)
is sufficient to optimize Y, for each point X; separately. Without loss of generality, for any input E Rdd we aim at finding optimal sparse code y RRM Since R R RRR'
Ia and
BB = Id. when m = n, following the derivation Section 3.1.2 we have
x Byll + Al = B y) + All3/llo-
(24)
where h B Ra is a dense code. Note that ||b mn together with Eq, 24 it follows that
nd
[B y)ll Ally/lo
Σ (hg v3))
1y + 0]
(25)
m
where 1[:] is indicator function which is if its argument is true and 0 otherwise.
This problem separable for each variable yy, and each term is minimized by setting
h4 if h nd otherwise
(26)
Fix Y, update R: For a fixed Y. minimizing the objective leads the same nearest orthogonal matrix problem as shown in Section 3.1.2 Let BY UTV obtained by SVD, where U, V are orthogonal matrix. Then, the reconstruction problem is minimized by R UVT.
Remark: Problems with other separable regularization terms can be solved in similar way. The key difference is how achieve sparse codes y. For example, for noom regularized problems, y can be obtained by soft thresholding function, i.e., y sign(y) @ max (0, yy nn/(2m)).
4 SPARSE DENOISING LAYER
One benefit of our fast sparse coding algorithm is that it enables simple closed-form reconstruction, which be used plug-in layer for deep neural networks. Specifically, given orthogonal matrix R and input vector the optimal reconstruction of our method can be expressed as
R. BJ Ra)
(27)
where f(-) is a non non-linear mapping function. For the k-sparse constrained problem, f(-) is k-max pooling function (w.r.t the absolute value) as Eq. (8)
f(h)) { h if |hy is one of the k-highest values of |h| M
(28)
otherwise

---

Under review as a conference paper at ICLR 2021
For the lu no-no regularization problem, f(-) is hard thresholding function as Eq
29
f(hz) = { hj
if otherwise
1
Z2
m
(29)
For the norm regularization problem, f(-) a soft thresholding function as Eq.
f(hy) sign(h,) max (0. (0.1hy|
nd
(30)
2m
where sign(-) denotes the Sign function.
The reconstruction in Eq. (27) can be used as simple plug-in layer for deep networks, we named as sparse denoising layer (SDL). It is worth noting that only the orthogonal matrix is needed to
leam. The structured matrix B is constructed as in Section 3.1.1 and fixed.
The orthogonal matrix R can be parameterized by exponential mapping or Cayley mapping (Hel- friih et al., 2018)) of a skew-symmetric matrix. In this work, we employ the Cayley mapping to enable gradient update using deep learning tools. Specifically, the orthogonal matrix R can be obtained by the Cayley mapping of a skew-symmn sememmyeete matrix as
R= (I - W)-1,
(31)
WT E Rdddd For a skew-symmetric matrix W, only the upper triangular matrix (without main diagonal) are free parameters. Thus, the number of free parameters of SDL is d(d - 1)/2, which is much smaller compared with the number of parameters of backbone deep networks.
where W a skew-symmetric matrix, i.e., W = =-
For training a network with SDL, we add a reconstruction loss term as regularization. The optimization problem is defined
min WERoX
((X: W. θ) + 3|| 2
(32)
where W is a skew-symmetrie matrix parameter of SDL. θ is the parameter of the backbone net- work. Z is the reconstruction of the latent representation Z via SDL (Eq,(7)). An illustration of the SDL plug-in shown in Figure When SDL is used as the first layer, then Z X, and = X. In this case, Z is the reconstruction of the input data X.
is worth noting that the shape of the input and output of SDL are same. Thus, SDL can be used as plug-in for any backbone models without changing the input/output shape of different layers in the backbone network. With the simple SDL plug-in, backbone models can be trained from scratches.
SCL
Twa Aecurecy
Tas Recrurie
Coods
Toach
Eeoth
(b) CIFARIOD
(c)
Resne134-CIFAROO CIFARIO
(d) Resnet34 CIFARI00
Figure 2: Mean test accuracy + std over 5 independent runs on CIFAR 10/CIFAR100 dataset under FGSM adversarial attack for Densenet and Resnet with or without SDL
5 EXPERIMENTS
We evaluate the performance of our SDL on both classification tasks and RI. tasks, For classification, we employ both DenseNet-100 100 (Huang al. (017) and ResNet-34 (He et al.) (016) as backbone. For RL tasks, employ deep PPO (Schullma we test the performance of backbone models with and without our SDL when adding Gaussian noise
al. 2017 as backbone. For all the tasks,
or Laplace noise. In all the experiments, plug SDL as the first layer of deep models. We set the

---

Under review as a conference paper at ICLR 2021
Twst Faturae
(a) Densener-Clean
(b) Denseoatt-Guusian-O..
Demsanc--Guusian-022
(d)
Teal Acc
Teat
Taad
Coooth
Fhom
Ehom
Enocth
(c) Resact-Clean
(1)
Resnet-Gausian-O..
(g)
Resnet-Gausian-0.2
(h)
Resee--Guusian-033
Figure 3: Mean test accuracy + std over 5 independent runs on CIFARIO dataset with Gaussian noise for Densenet and Resnet with or without SDL
Trc Aocureey
Tec
Cooch
Bouch
Eooch
Eboch
(a) Densenet-Clean
(b) Densenntt-Gausian-O..
(c) Densane--Gunsian-022
(d) 0.
decurey
Stamardd
Standerd
Test
Cooch
Eboch
Ebach
Eooch
(c) Resnet- Clean
Resnet- Resnet-Gamssinn-00. Gaussian-O.
g
Resnet- Gaussian-0.2
(h) Resnet-Gaussinn 0.3
Figure 4: Mean test accuracy + std over 5 independent runs on CIFAR100 dataset with Gaussian noise for Densenet and Resnet with or without SDL.
standard deviation of input noise (0, 011, 0.2, 0.3), respectively. (The input noise is added after input normalization) We keep all the hyperparameters of the backbone models same, the only difference is whether plugging SDL. The parameter 3 for the reconstruction loss is fixed as 3 100 in all the experiments.
Classification Tasks: We test SDL CIFARIO and CIFAR100 dataset. We construct the struc- tured matrix B E R12x14 by (4. In this setting, the orthogonal matrix R corresponds to the convolution parameter of Conv2d( with kernelsize=2 2. We the sparse parameter of our k-sparse SDL as k = 3 in all the classification experiments. The average test accuracy over five independent runs on CIFARIO and CIFARI00 with Gaussian noise shown in Fig, 3 and Fig, respectively. We can observe that models with SDL obtain a similar performance compared with vanilla model on the clean input case. With an increasing variance of imput noise, models with SDL outperform the vanilla models more and more significantly. The experimental results with Laplace noise are presented Fig. and Fig. T0 the supplementary. The results on Laplace noise cases show the similar trends with Gaussian noise cases. We further test the performance of SDL under the fast gradient sign method (FGSM) attack (Goodfellow et al., 2015). The perturbation parame- ter epsilon is set to 8/256. Experimental results are shown in Fig. 2 We observe that adding SDL plug-in in can improve the adversarial robustness of the backbone models. More experiments on liny-Imagenet dataset can be found in Appendix G
RL Tasks: We test deep PPO model with SDL on Atari games: KungFuMaster, Tennis and Seaquest. The deep PPO model concatenates four frames the input state. The size of input
8

---

Under review as a conference paper at ICLR 2021
Aneraee terua
Boom careeee Betum
kungfumaster (b) kungfuraster kungfumaster Gaussian
(d)
kungfuraster
Koerage Serum
Aoorag Retum
Aeerog
Tennis-Clean
Tennis- Gaussian-OO
( Teamis-Gassiin-O-.
(h) Ceamis-Gaussian-e
Figure 5: Average Return + std over independent on KungfuMaster and Tennis game with Gaussian noise with or without SDL
Aperage Momrpe-eeumm
(a) KungfuMaster-CCeen Lunglau (c) SangruuMaster-Lapacce-
(d)
Geeraga Return
merrog Retum
10000 9vc m
Tennis-Clean
Teanis-Laplace-O.1
Temii-Laplacc-022
Temais-Lpplac-0.3
Figure 6: Average Return + std over independent KungfuMaster and Tennis game with Laplace noise with or without SDL
is 84 84 We construct the structured matrix B R 36 by Eq. this setting, the orthogonal matrix R corresponds to the convolution parameter of Cono2d(-) with kernelsize=3 x 3. The number of free parameters is 630. We the sparse parameter of k-sparse SDL all the RL experiments.
in
The return of one episode is the sum of rewards over all steps during the whole episode. We present the average return over five independent runs on KungFuMaster and Tennis game with Gaussian noise and Laplace noise Fig. 5 and Fig. 6 respectively. Results on Seaquest game are shown in Fig. 16 in the supplement due to the space Timitation. We can see that models with SDL achieve competitive average return on the clean cases. Moreover, models with SDL obtain higher average return than vanilla models when the input state is perturbed with noise.
CONCLUSION
We proposed fast sparse coding algorithms for both k-sparse problem and lo-norm regularization problems. Our algorithms have a simple closed-form update. We proposed a sparse denoising layer lightweight plug-in for backbone models against noisy input perturbation based this handy closed-form. Experiments on both ResNet/DenseNet classification model and deep PPO RL model showed the effeteness of our SDL against noisy input perturbation and adversarial perturbation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Chenglong Bao, Jian-Feng Cai, and Hui Ji. Fast aparsity-based orthogonal dictionary learning for image restoration. In Proceedings of the IEEE International Conference on Computer Vision, pp- 2013.
Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 4331-4339, 2019.
Jean Bourgain, Alexey A Glibichuk, and SERGEI VLADIMIROVICH KONYAGIN. Estimates for the number of sums and products and for exponential sums fields of prime order. Journal of the London Mathematical Society, 73(2):380-392 2006.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing, 15(12):3736-3745 2006.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi. Kevin Regan, and Samy Bengio. Large margin deep networks for classification. Advances neural information processing systems, pp. 842-852, 2018.
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE transactions on pattern analysis and machine intelligence, 35(12):9916-222) 2012.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 631-640, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform. In International Conference on Machine Learning, pp. 1969-1978. PMLR, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Yueming Lyu. Spherical structured feature maps for kernel approximation. In International Confer- ence on Machine Learning, pp. 2256-2264, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose comain-specifii transformations for data augmentation. In Advances in neural information processing systems, pp. 3236-3246, 2017.
Ron Rubinstein, Michael Zibulevsky, and Michael Elad. Efficient implementation of the k-svd algo- rithm using batch orthogonal matching pursuit. Technical report, Computer Science Department, Technion, 2008.
Peter H Schönemann. generalized solution the orthogonal procrustes problem. Psychometrika, 3101):1-10, 1966.
10

---

Under review as a conference paper at ICLR 2021
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
Singanallur Venkatakrishnan, Charles Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 945-948. IEEE, 2013.
Kaixuan Wei, Angelica Rivero, Jingwei Liang, Ying Fu, Carola-Bibiane Schnlieb, and Hua Huang. Tuning-free free plug-and-play proximal algorithm for inverse imaging problems. ICML, 2020.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing. Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. ICML, 2019.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26 17):314233155, 2017.
Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480-4488, 2016.
11

---

Under review as a conference paper at ICLR 2021
A PROOF OF PROPOSITION 1
Proof. Let C, E Cl be the hh row of matrix FA E C Eq. Let os E RR122n be the ith row of matrix B IR 2m222 in Eq. 44. For I i S m, i + I. we know that
0:00 1+m 0,
(33)
Mi+m 11 j+m
; Re(c,
(34)
Vitm J 0: jmm = im(c;c)),
(35)
where * denotes the complex conjugate, Re(-) and Im(-) denote the real and imaginary parts of the input complex number.
For discrete Fourier matrix F, we know that
Siki
1,
if otherwise
c,
(36)
k=0
When * j. from 36, we know c.₁ Thus, we have
Vitm 2j T jmm 2 = Re(c,c)) = 0,
(37)
V/+m Vj - U.U0+m Im(c,c)) O,
(38)
When = J, we know that U,-m V Vitm = i.Ns = c, 1.
Put two cases together, also note that = 2m, we have BB Id- The no-m of the column vector of B is given
2Tka
2 771
b,b
sin
cos
= 72
(39)
71
Thus, we have |1bيll2 = The for E [1, M}
B PROOF OF THEOREM
Proof. Let e; E Cm be the ith column of matrix FA Cmx in 5 Let b R22x1 be the fth column of matrix B E 2m2mx2 in Eq. (4. For 1 S i,j & n, i * j. we know that
6 bitn = 0, b T Djtn 6, bj Re(c,ej),
(40)
(41)
bin b. bitn lm(c)cy),
(42)
where * denotes the complex conjugate, Re(-) and Im(-) denote the real and imaginary parts of the input complex number.
It follows that
(BB) == max
b b
max
1otel = (F)))
(43)
From the definition of FA in Eq. we know that
(FFA)
max
1c/ =
max
Σ
2rizis-1)
(44)
i
ini m
ZEA
max
Σ e
2riak
(45)
15k5n-1
EE
12

---

Under review as a conference paper at ICLR 2021
Because A {g 9 n- multiplicative group {g]
9
12
mod n. we know that A is subgroup of the
mod n. From Bourgain et al. (006), we know that
max IskEn-1
Σ
2rdsk
S Vn
(46)
.EEA
Finally, we know that
1(B) S (FFA) < Vn
(47)
C PROOF OF COROLLARY
Proof. Since R R = RR = La and D = R B, we know that || ||6 (byll2. From Theorem
!
we know that |bbll2 V Vj E f1,...,M It follows that |d |bb un for
Vj E , .M..
From the definition of mutual coherence p(-), we know it is rotation invariant. Since D R B with R RR =I we know (uD) = n B). From Theorem we have ((B) < //n Thus, we obtain (DD) = (B V22
D EMPIRICAL CONVERGENCE OF OBJECTIVE FUNCTIONS
We test our fast dictionary learning algorithms Lena with image patches (size 12x 12). We present the empirical convergence result of our fast algorithms Figure shows that the objective tends to converge less than fifty iterations.
Dagothe Value
Oijostive
Chjjetion
Herraon
teration
(a) k-sparse constrained objective
(b) 16-norm regularized objective
(c) 1--norm regularized objective
Figure 7: Decreasing of the objective functions
13

---

Under review as a conference paper at ICLR 2021
E DEMO OF DENOISED IMAGES
We show the denoised results of our fast sparse coding algorithm on some widely used testing im- ages. The input images are perturbed by Gaussian noise with std 100. The denoised results presented in Figure It shows that our algorithms can reduce the influence of the noisy perturbation of the images.
(a) Noisy Lena
(b) Noisy peppers
(c) Noisy boat
(d) Noisy barbara
Je-sparse Denoised Leny
(1) sparee Demoised peppers
&-sparse Denoised hoat
(h) sespprre Denoised barhara
fo-morm Denoised Lena
1o-momm Denoised peppers
(k) 4--ormm Denoised boat
(I) 4--norm Denoised barbara
(m) [y-norm Denoised Lena
(n) Iy-nomm Denoised peppers (o) 1 !--orrm Denoised boat
(p) Iy-norm Denoised barbara
Figure 8: Demo of denoised results of our fast sparse coding algorithm
14

---

Under review as a conference paper at ICLR 2021
F EXPERIMENTAL RESULTS ON CLASSIFICATION WITH LAPLACE NOISE
Stnder
Densenet- Clean
(b) Lamsentt-Laplac--0..
(c) Densenet- Densenet--apllac-0...
Densenet- Lannenet-Laplace-...
Text Pettecureyy
Sanker
Sander
Sander
Resnet-Clean
(I)
Resnet-Laplace-O.
Laplac-La
(h) 0.
Figure 9: Mean test accuracy + std 5 independent CIFAR10 dataset with Laplace noise for Densenet and Resnet with or without SDL
BDS
booch
touch
bo0
tooch
(3) Densener-Clean
(b) De
(c) Oaplac.-
(d) Densenet-Laplec-00..
Wander
Mander
Tea
(e) Resnet- Clean
(f) Resnet-Laplace 0.1
g Resnet- Laplece 0.2
(h) Resnet- Resnet-Laplaced Laplace-O.3 0.3
Figure 10: Mean test accuracy std over independent runs on CIFAR100 dataset with Laplace noise for Densenet and Resnet with or without SDL
15

---

Under review as a conference paper at ICLR 2021
G
EXPERIMENTAL RESULTS ON TINY-IMAGENET DATASET
Accuracy
Test Accuracy
Accuracy
103
Eooch
(a) Densenet-Clees
(b) Densenet Gaussian-d. 0..1
Densenet-Gaussiant
(c) Densenet Caussian-B 0..2
Demsenet-Gaussian
200
Accuracc
Teat Accursccy
Test Accuracy axy
(d) Densenet-Goussian--0
(c) Densenct-Gmmssin--0.4
(f) Densenet-Gaussian-OO
Figure 11: Mean test accuracy std over independent runs on Tiny- Imagenet dataset with
Gaussian noise for Densenet with or without SDL
Test Accurecy
Fest Accuracy
SDL
SDL
SCL
Stande
Standand
Standard
Eboch
(b) Eooch
Eooch
(a)
Gaussian
Test Test.Accuree Accuracy
Test Accuracy
Test Accuracy
4111 Skandund
SDI
Standard
Fonch
Fooch
Enach
(d) Resnet. Gaussian 0.3
(e) Resnet Gaussian 0.4
(f) Resnet Gaussian 0.5
Figure 12: Mean test accuracy + std over 5 independent Gaussian noise for Resnet with or without SDL
Tiny-nm dataset with
16

---

Under review as a conference paper at ICLR 2021
Accuracy
Teat Accuracy 9
Accuracy
100
(a) Densenet Clean
(b)
Densenet Luplace 0.1
(c) Densenet Laplace 0.2
Accuracy
Text Accuracy
Test Accuracy
Standard
(d) Densenct-Laplac-00.
(c) Dansenct-Laplacc-04
(f) Densenct--aplaas-0..
Figure 13: Mean accuracy std over 5 independent runs Tiny-Imagenet dataset with Laplace noise for Densenet with or without SDL
Test Accuracy
Accuracy
Standard
220
(a) Resnet-Clean
(b) Resnet-Laplace-0..
(c) Resnet-Laplace-02
Accuracy
Test Accuracy
Teat Accuracy
40
(d) Resnet-Lapplac-0.3
(c) Resnet-Laplace-O.4
(f) Resnet-Laplace-O0
Figure 14: Mean accuracy + std over 5 independent Timy-Im dataset with Laplace noise for Resnet with or without SDL.
17

---

Under review as a conference paper at ICLR 2021
Test Accuracy
Test Accuracy
18
16
55
12
13
10
Standard
Standard
EDoch
EDoch
(a) FGSM
(b) Resnet-
Figure 15: Mean test accuracy + std over 5 independent runs on Tiny-Imagent dataset under
FGSM attack for Densenet and Resnet with or without SDL
H
RESULTS OF RL ON SEAQUEST GAME
Zandard
Uporaber
Updates
Updatee
Upiaatex
(a) Clean
(b) Gaussian-O.1
(c) Gaussian-0.2
Gaussian-0
50t
Destage felum
Mpprates
Npronnes
Lpdatee
Upsatees
(e) Clean
(1)
Laplaco-0.1
(g)
Laplaco-0.2
(h) Laplacc-0.3
Figure 16: Average Return std over 5 independent runs on Seaquest game with Gaussian/Laplace noise with or without SDL
18

---

Under review as a conference paper at ICLR 2021
1 PYTORCH IMPLEMENTATION OF THE SDL LAYER
class Sparseteenoisingiayee (nn. Medule)
def _init__sell___ (self, sparsek, B,n)
super (wt.epppaeeeeoisinniiioooo self.ksize 2
self) self)._inite init (nit_()
kernelsize of Conv2d channel of input
self.channel
cutplanes self channel saizeeself.ksia.. self.B torch. from numpy (B). float () cuda
self.n =
outplanes
sclf.sparseK sparseK
relf.register_parrae (name-
param-torch, nn. Parammete (torch. randn (outplanes, outplanes). cuda ()
def forward (self, x)
# Cayley Mapping to compute orthogonal matrix KA torch.triu (self. diagenal-1
tmpA = KA ().t()
wwwf.eelllffffeeeee... outplanes) cuda -tmpA KU torch.mm! (torch. (sclf. outplanes, sclf.outplanes) .udda ()
tmp2 torch.eye (self.cutplaaeee self.
tmpA corch.invers tmpB
} orthogenal matrix R
weight KU).view (sel wtwpllffottaaaaaaeeoe sel fea.f.ccaane channel, selel,sellll ksize, self.
ksize) #Reshape into the Conv2d parameter
out conv stride=1, padding self.ksize-1]
out out. permute (0, 3, (out, self.B)
out
(function k-max pooling w.r.t the absolute value index corch.abs (out) .tek (self, sparresp dim 3)
mask = torch. zeros (out. shape) cuda
mask basktecatter_f (3, Andex[1], 1.) out out mask
#
out marchl (out, wtw.te.ttaee..oooooootooooootttt transpose (self.H, 0, 1))
out out permute 3,,,, out c.conv teansstrasppseeddccc (cut, weight, stride=1, padding self ksize-1 /
ksize)
return out reconstruction of the inpat
19


Reviews:
[{"review_id": "srnKoMtplhI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is generally well presented. However, a main issue is that the optimization algorithms for the l0-norm regularized problems (Section 3.1.2 and Section 3.2) are not correctly presented. Specifically, in the algorithm development to solve the \"Fix $\\boldsymbol{R}$, optimize $\\boldsymbol{Y}$\" subproblem, it overlooks the coupling/interaction between the variables $y_1, y_2, \\dots,y_M$ and mistakenly obtains a closed-form solution. See Comment 1 for details.\n\nThis paper proposes two L0 regularized sparse coding and dictionary learning models: one in bound form (with L0-norm explicitly bounded as a constraint in the formulated problem) and the other in Lagrangian form (with L0-norm term appearing in the objective function), and uses it as a robust layer for vanilla deep models. Experiments demonstrate that using the proposed model as a plug-in layer improves noise-robustness of both ResNet/DenseNet for classification and deep PPO models for reinforcement learning tasks.\n\nBelow are specific comments.\n\n1. Main comments:\n\n1.1 In Section 3.1.2, the \"Fix $\\boldsymbol{R}$, optimize $\\boldsymbol{Y}$\" subproblem is not correctly solved.\n\nSpecifically, in Eq. (10), the $\\left\\|\\boldsymbol{B} (\\boldsymbol{h}-\\boldsymbol{y})\\right\\|_F^2$ (written as $\\left\\|\\boldsymbol{B} (\\boldsymbol{h}-\\boldsymbol{y})\\right\\|_2^2$ below) should be\n\\[\n\\begin{aligned}\n\\left\\|\\boldsymbol{B} (\\boldsymbol{h}-\\boldsymbol{y})\\right\\|_2^2\n&= \\left\\| \\sum_{j=1}^M (h_j-y_j) \\boldsymbol{B}_j \\right\\|_2^2 \\\\\n&= \\sum_{i=1}^M \\sum_{j=1}^M (h_i-y_i) (h_j-y_j) \\boldsymbol{B}_i^{\\rm T} \\boldsymbol{B}_j.\n\\end{aligned}\n\\]\n\nSince $\\boldsymbol{B}$ is a wide matrix (with much more columns than rows: $M=2n \\gg d=2m$), its columns are not (all) orthogonal. Actually, as implied by Theorem 2, $|\\boldsymbol{B}_i^{\\rm T} \\boldsymbol{B}_j|$ can be very close to $\\|\\boldsymbol{B}_j\\|_2^2 = \\|\\boldsymbol{B}_j\\|_2^2 = \\frac{m}{n}$ when $n$ is comparable to or greater than $m^2$. Therefore, $\\left\\|\\boldsymbol{B} (\\boldsymbol{h}-\\boldsymbol{y})\\right\\|_2^2$ can not be expressed as $\\sum_{j=1}^M (h_j-y_j)^2 \\|\\boldsymbol{B}_j\\|_2^2$, i.e., as a sum of $M$ functions, each depending on only $y_j$, and each $y_j$ appearing in only one of these functions, thereby each $y_j$ can be independently solved in closed form.\n\nInstead, the variables $y_1, y_2, \\dots,y_M$ are coupled in the objective and numerical optimization methods need to be used be solve them.\n\n1.2 Similarly, in Section 3.2, the algorithm to solve the \"Fix $\\boldsymbol{R}$, optimize $\\boldsymbol{Y}$\" subproblem is also not correctly presented, due to the invalid equality in Eq. (18) which ignores the coupling/interaction between the variables $y_1, y_2, \\dots,y_M$.\n\n\n2. Minor comments (notation inconsistencies/abuse, typos, etc.):\n\nIn Section 3.1.1, \"a $n \\times n$ discrete Fourier matrix\": The article `a' should be `an'.\n\n\"Eq.(4)\" etc. is missing a space between \"Eq.\" and equation number.\n\nIn Section 3.1.1 and Section 3.2, to make it clearer, please change the Frobenius norm operator $\\|\\cdot\\|_F$ to the Euclidean norm operator $\\|\\cdot\\|_2$ whenever it is applied to a vector, e.g., in Eq. (8) on the right-hand side of equality, and in Eqs. (9-11).\n\nThroughout the paper, symbols denoting scalars are not in bold typeface, vectors are denoted as boldface lowercase letters and matrices as boldface capitals. For clarity and following the convention, these notation rules could also be applied to the lower-order parts, e.g., the $j$-th elements of vectors $\\boldsymbol{h},\\boldsymbol{y}$ are denoted as $h_j,y_j$ (rather than $\\boldsymbol{h}_j,\\boldsymbol{y}_j$), and the $i$-th columns of matrices $\\boldsymbol{X},\\boldsymbol{D},\\boldsymbol{Y}$ are denoted as $\\boldsymbol{x}_i,\\boldsymbol{d}_i,\\boldsymbol{y}_i$ (rather than $\\boldsymbol{X}_i,\\boldsymbol{D}_i,\\boldsymbol{Y}_i$).\n\nOn page 5, the sentence in the first two lines should read as \"We can see that Eq. (12) is minimized when $S$ consists of the indices of the $k$ largest (in absolute value) elements of $\\boldsymbol{h}$\".\n\nIn Eq. (18), $\\|\\boldsymbol{y}_j\\|_0$ should be written as $1[y_j \\neq 0]$, where $1[\\cdot]$ is an indicator function which is 1 if its argument is true and 0 otherwise.\n\n==================== After rebuttal =======================\nThank the authors for their detailed responses, which have addressed my concerns regarding Eqs. (10) and (18). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "pg8H72Q_x6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for the efforts of the authors. Some of my concerns have been addressed. However, I think that  the paper should test SDL layer by plugging in any hidden layer. So I keep my score.\n_____________________________________________\nSummary: \nThe paper provides a simple sparse denoising layer for robust deep learning. By employing a structured dictionary, the sparse coding phase of the l_0-norm based sparse coding has a closed-form solution. So it can be solved very fast when testing. Experiments show that the deep network with the presented sparse denoising layer outperforms the counterpart.\n\nPros: \nThe idea of using a structured dictionary is interesting, which produces a fast sparse coding. Also, the sparse coding layer can be integrated into a deep network and trained end-to-end.\n\nCons: \n1.\tThe paper does not distinguish the corruption (e.g., additive Gaussian noise and blur) and the adversarial robustness. The paper considers the image corruption robustness. However, the statements in Section 1 and 2 are on adversarial robustness.  \n2.\tThe experiments do not test the superiority of the proposed approach. It only shows that using denoised images rather than the noisy ones are better for the classification task. It is especially when the denoising phase can be trained end-to-end with  deep networks. The paper only shows the effectiveness of SDL layer used as the first layer. It is better to verify the SDL layer used as a hidden layer. Moreover, I think the image denosing task [Zhang, 2017 in the references] is more appropriate for testing the superiority of SDL layer. Also, a two-stage approach should be included for comparison, i.e., it first denoises the noisy images with traditional sparse coding and then performs classification with deep networks without using SDL layer. \n3.\tEq. (20) is the one-step sparse coding, i.e., it solves problem (7) or (15) for one-step update. What about mimicking the optimization of sparse coding, i.e., solves problem (7) or (15) for several updates?\n4.\tFigure 1 and Section 5 are not clear for me. The images in CIFAR10 are with a size of 32x32x3. However, sparse coding algorithms are image patch-based. Does the paper use 12x12x3 patches for classification task on CIFAR10? It is better to detail this issue.\n\nMinor comments:\n1.\tThe claim ”the alternative descent method heavily relies on SVD, which is not easy to extend to deep models” is not right. Actually, these are several papers shown that SVD can be extended to deep networks.\n2.\tJust above Eq. (8): ||Y_i|| -> ||Y_i||_0", "extra_scores": {}, "extra_text": {}}, {"review_id": "BmWCj-5owu6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nThis paper addresses the problem of Deep Learning models' output instability to small perturbations in the input, first pointed out by Goodfellow et al 2015. They propose the usage of a denoising layer that can be applied as a first step to any deep CNN structure.\n\nThe proposed denoising layer consists in a sparse coding and dictionary learning algorithm:\n-  whose sparse coding phase has a closed form solution (unlike the iterative methods in the literature). [both steps have independently closed form solutions, that need to be iterated]\n- that allows end-to-end learning with low computational cost since it implements over-complete dictionary learning that translates easily as a deep learning layer.\n\n\nIn the numerical section they show how the method outperforms other state of the art methods when adding Gaussian noise to image classification and Reinforcement Learning\n\n\n*Positive points* \n- The paper tackles an important problem for ML with an interesting approach\n- Numerical testing on very different scenarios such as classification tasks and reinforcement learning.\n- The authors numerically show the robustness of their method to Gaussian noise perturbation of images\n- The paper gives many details that allows reproducibility and is well written\n\n*Negative points*\n- The usage of denoising layers is not new in the context of image processing for restauration. It is clear that the paper would like to tackle the implications of denoising for image classification, but it is difficult to argue that Gaussian noise is representative of realistic perturbations that can trick an image classifier.\n\n- The algorithms are tested on 'theoretical' data sets, meaning that the authors add synthetic noise to the CIFAR images and then train a classifier (idem for the RL tests). With no added noise the use of SDL does not improve the results (check Fig 2,3,4,5(a)) which might imply that the method cannot tackle real-life scenarios.\n\nSome typos:\n- sentence 'The optimization problem (1) is non-convex and non-smooth, which is very difficult to optimize' after eq 2 might need a bit more explaining or a reference since people have been using intensively numerical solvers to minimize non-convex and non-smooth energies. \n- dimension of X is missing after eq 1 (although it is easy to understand from the dimension of the other vector/matrices)\n- last sentence of the first page: ‘effeteness’, should be ‘effectiveness’?\n- second page: ‘The both two steps are based on heuristic’, either 'both' or 'two'\n\n*Conclusion*\nThe paper presents a new DL denoising layer that should allow to make classifiers and RL models more robust to added noise on the input. The instability of DL models to small perturbations is important and needs to be addressed, but the paper does not show that it can tackle realistic scenarios. Instead it focus on simulated perturbations of Gaussian noise on the CIFAR data set.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7hN_idIz_zP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros: The general idea of the paper to introduce a simple single layer that does not really hurt the performance on clean data but is able to provide significantly more robust results on noisy data is interesting on valuable. Moreover, the numerical results demonstrate that classical networks are unable to use some of their expressive power to denoise and classify at the same time, such that the proposed approach indeed has significant benefits over the vanilla networks. \n\nCons: Despite the nice general idea, the paper has several weaknesses.\n- Most significantly, I believe the derivation of the updates is not correct. Denoting $z = h-y$ for a moment, eq. (10) states that\n$$ \\|Bz\\|^2 = \\sum_j (z_j)^2 \\|B_j\\|^2. $$\nHowever, we have\n$$ \\|Bz\\|^2 = \\langle \\sum_j z_j B_j, \\sum_i z_i B_i \\rangle = \\sum_{i,j} z_i z_j \\langle B_i, B_j \\rangle, $$\nand the above equality holds if and only if $\\langle B_i, B_j \\rangle=0$ for $i\\neq j$. The $B_i$, however, denote the columns of $B$ and the only property we know about $B$ is that $BB^T = I$, i.e., the rows of B are orthogonal. Therefore, I do not believe the result is correct. The same holds for the argument in eq. (18). Please correct me if I am overlooking something here. \n- Secondly, the paper does not compare to any baseline other than training the plain networks (that were not really designed to handle significant noise). Is there absolutely no other approach in the literature to tackle this problem? And even if so, wouldn't applying a fixed denoiser as preprocessing be a better baseline?\n\nMinor things:\n- The motivation of the approach was not clear to me at the beginning. The introduction  refers to a missing robustness of neural networks and cites a paper on adversarial attacks. These attacks are, however, not addressed in the numerical experiments. \n- It was a little unusual to me to avoid the problems of minimizing (1) by just designing parts of the matrix $D$. In which cases is the specific design with Fourier coefficients justified?\n- Parts of the theory seem a little overstated to me - Theorem 1 follows from the orthogonality of the Fourier transform and Theorem 3 is a very simple corrolary. Maybe the term \"Theorem\" is a little too strong for these results. Theorem 2 is interesting but unfortunately seems far from the practical case. \n- The final structure of the SDL frequently appears in other contexts where minimization methods are used to motivate network architectures or unrolling is used, e.g. in Schmidt and Roth, \"Shrinkage Fields for Effective Image Restoration\", or Kobler et al. \"Variational Networks: Connecting Variational Methods and Deep Learning\". \n\nCurrently, I clearly do not recommend the acceptance of this paper. If I am overlooking something in my first concern, and if a baseline comparison is added, I am happy to reconsider my score. \n\n-----------\nAfter the rebuttal: I would like to thank the authors for their response and clarification. The argument that the proposed strategy minimizes an upper bound is, however, somewhat weak, because $\\|B(h - y)\\|\\leq \\|B\\| \\|h-y\\| $ is a possible upper bound for any matrix $B$, but using the right-hand side as a replacement for $\\|B(y-y)\\|$ within an optimization problem can make a large difference. I would expect to at least verify a marginal difference in the results numerically. Moreover, I do not think that the argument that some dictionary learning algorithms are slow is valid for omitting a comparison to a simple baseline denoiser at all: Your denoising baseline could be as simple as thresholding DCT coefficients or applying a median filter - operations that are surely as fast as the proposed approach (and still represent a sequential pipeline of first denoising and then applying a network, which one can expect to work worse than the proposed approach). Therefore, I keep my score and do not recommend this paper for acceptance in its current form. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper motivates the need for robustness, citing a paper on adversarial attacks. The type of perturbations are quite different (and of greater concern) than those originally included in the work, namely additive Gaussian or Laplace noise. This was raised by reviewers 1, 2 and 3.

The authors provided a detailed rebuttal in which they addressed many concerns of the reviewers. In particular, they included experiments with adversarial attacks. While these initial results seem interesting, the AC believes that they are too preliminary and it is not possible to evaluate their significance. The work should incorporate stronger baselines (other than plain networks) and consider more challenging forms of attacks (only FGSM attacks were studying). In particular, the paper should discuss where these ideas sit in the context of the literature and compare against conceptually related works that have been published in the area, such as

Xie, Cihang, et al. "Feature denoising for improving adversarial robustness." CVPR. 2019.

Without the adversarial perturbations, the impact of the work reduces significantly, as highlighted by reviewer 1, 2 and 3. In such case, it would be important to include stronger baselines in that setting as well. Please see the suggestions made by reviewer 1.

The authors satisfactorily addressed the problems in the derivation of the updates raised by Reviewers 1 and 4. However, new questions arise that would require careful consideration, as mention by reviewer 4 (which the authors could not answer as they were posted after the discussion period ended). This alone would not imply rejection, but suggests that the paper would be stronger after incorporating further feedback.

While this did not play a role in the decision, the AC suggests to also look at the literature around LISTA, in particular the work:
Liu, Jialin, and Xiaohan Chen. "ALISTA: Analytic weights are as good as learned weights in LISTA." ICLR. 2019.

All four reviewers recommend rejecting the paper and did not change their position after reading the author's response. The AC agrees with this assessment.

Decision: Reject