A Simple Sparse Denoising Layer for Robust Deep Learning

1 INTRODUCTION . Deep neural networks have obtained a great success in many applications , including computer vision , reinforcement learning ( RL ) and natural language processing , etc . However , vanilla deep models are not robust to noise perturbations of the input . Even a small perturbation of input data would dramatically harm the prediction performance ( Goodfellow et al. , 2015 ) . To address this issue , there are three mainstreams of strategies : data argumentation based learning methods ( Zheng et al. , 2016 ; Ratner et al. , 2017 ; Madry et al. , 2018 ; Cubuk et al. , 2020 ) , loss functions/regularization techniques ( Elsayed et al. , 2018 ; Zhang et al. , 2019 ) , and importance weighting of network architecture against noisy input perturbation . Su et al . ( 2018 ) empirically investigated 18 deep classification models . Their studies found that model architecture is a more critical factor to robustness than the model size . Most recently , Guo et al . ( 2020 ) employed a neural architecture search ( NAS ) method to investigate the robust architectures . However , the NAS-based methods are still very computationally expensive . Furthermore , their resultant model can not be easily adopted as a plug-in for other vanilla deep models . A handy robust plug-in for backbone models remains highly demanding . In this work , we take an initial step to design a simple robust layer as a lightweight plug-in for the vanilla deep models . To achieve this goal , we first propose a novel fast sparse coding and dictionary learning algorithm . Our algorithm has a closed-form approximation for the sparse coding phase , which is cheap to compute compared with iterative methods in the literature . The closedform update is handy for the situation that needs fast computation , especially in the deep learning . Based on this , we design a very simple sparse denoising layer for deep models . Our SDL is very flexible , and it enables an end-to-end training . Our SDL can be used as a lightweight plug-in for many modern architecture of deep models ( e.g. , ResNet and DenseDet for classification and deep PPO models for RL ) . Our contributions are summarized as follows : • We propose simple sparse coding and dictionary learning algorithms for both k-sparse constrained sparse coding problem and l0-norm regularized problem . Our algorithms have simple approximation form for the sparse coding phase . • We introduce a simple sparse denoising layer ( SDL ) based on our handy update . Our SDL involves simple operations only , which is a fast plug-in layer for end-to-end training . • Extensive experiments on both classification tasks and reinforcement learning tasks show the effectiveness of our SDL . 2 RELATED WORKS . Sparse Coding and Dictionary Learning : Sparse coding and dictionary learning are widely studied in computer vision and image processing . One related popular method is K-SVD ( Elad & Aharon , 2006 ; Rubinstein et al. , 2008 ) , it jointly learns an over-complete dictionary and the sparse representations by minimizing a l0-norm regularized reconstruction problem . Specifically , K-SVD alternatively iterates between the sparse coding phase and dictionary updating phase . The both steps are based on heuristic greedy methods . Despite its good performance , K-SVD is very computationally demanding . Moreover , as pointed out by Bao et al . ( 2013 ) , both the sparse coding phase and dictonary updating of K-SVD use some greedy approaches that lack rigorous theoretical guarantee on its optimality and convergence . Bao et al . ( 2013 ) proposed to learn an orthogonal dictionary instead of the over-complete one . The idea is to concatenate the free parameters with predefined filters to form an orthogonal dictionary . This trick reduces the time complexity compared with KSVD . However , their algorithm relies on the predefined filters . Furthermore , the alternative descent method heavily relies on SVD , which is not easy to extend to deep models . In contrast , our method learns a structured over-complete dictionary , which has a simple form as a layer for deep learning . Recently , some works ( Venkatakrishnan et al. , 2013 ) employed deep neural networks to approximate alternating direction method of multipliers ( ADMM ) or other proximal algorithms for image denoising tasks . In ( Wei et al. , 2020 ) , reinforcement learning is used to learn the hyperparameters of these deep iterative models . However , this kind of method itself needs to train a complex deep model . Thus , they are computationally expensive , which is too heavy or inflexible as a plug-in layer for backbone models in other tasks instead of image denoising tasks , e.g. , reinforcement learning and multi-class classification , etc . An illustration of number of parameters of SDL , DnCNN ( Zhang et al. , 2017 ) and PnP ( Wei et al. , 2020 ) are shown in Table 1 . SDL has much less parameters and simpler structure compared with DnCNN and PnP , and it can serve as a lightweight plug-in for other tasks , e.g. , RL . Robust Deep Learning : In the literature of robust deep learning , several robust losses have been studied . To achieve better generalization ability , Elsayed et al . ( 2018 ) proposed a loss function to impose a large margin of any chosen layers of a deep network . Barron ( 2019 ) proposed a general loss with a shape parameter to cover several robust losses as special cases . For the problems with noisy input perturbation , several data argumentation-based algorithms and regularization techniques are proposed ( Zheng et al. , 2016 ; Ratner et al. , 2017 ; Cubuk et al. , 2020 ; Elsayed et al. , 2018 ; Zhang et al. , 2019 ) . However , the network architecture remains less explored to address the robustness of the input perturbation . Guo et al . ( 2020 ) employed NAS methods to search the robust architectures . However , the searching-based method is very computationally expensive . The resultant architectures can not be easily used as a plug-in for other popular networks . In contrast , our SDL is based on a closed-form of sparse coding , which can be used as a handy plug-in for many backbone models . 3 FAST SPARSE CODING AND DICTIONARY LEARNING . In this section , we present our fast sparse coding and dictionary learning algorithm for the k-sparse problem and the l0-norm regularized problem in Section 3.1 and Section 3.2 , respectively . Both algorithms belong to the alternative descent optimization framework . 3.1 K-SPARSE CODING . We first introduce the optimization problem for sparse coding with a k-sparse constraint . Mathematically , we aim at optimizing the following objective min Y , D ‖X −DY ‖2F subject to ‖yi‖0 ≤ k , ∀i ∈ { 1 , · · · , N } ( 1 ) µ ( D ) ≤ λ ‖dj‖2 = 1 , ∀j ∈ { 1 , · · · , M } , where D ∈ Rd×M is the dictionary , and di denotes the ith column of matrix D. yi denotes the ith column of the matrix Y ∈ RM×N , and µ ( · ) denotes the mutual coherence that is defined as µ ( D ) = max i 6=j |d > i dj | ‖di‖2‖dj‖2 . ( 2 ) The optimization problem ( 1 ) is discrete and non-convex , which is very difficult to optimize . To alleviate this problem , we employ a structured dictionary as D = R > B . ( 3 ) We require that R > R = RR > = Id and BB > = Id , and each column vector of matrix B has a constant l2-norm , i.e. , ‖bi‖2 = c. The benefit of the structured dictionary is that it enables a fast update algorithm with a closed-form approximation for the sparse coding phase . 3.1.1 CONSTRUCTION OF STRUCTURED MATRIX B . Now , we show how to design a structured matrix B that satisfies the requirements . First , we construct B by concatenating the real and imaginary parts of rows of a discrete Fourier matrix . The proof of the following theorems regarding the properties of B can be found in Appendix . Without loss of generality , we assume that d = 2m , M = 2n . Let F ∈ Cn×n be an n × n discrete Fourier matrix . Fk , j = e 2πikj n is the ( k , j ) thentry of F , where i = √ −1 . Let Λ = { k1 , k2 , ... , km } ⊂ { 1 , ... , n− 1 } be a subset of indexes . The structured matrix B can be constructed as Eq. ( 4 ) . B = 1√ n [ ReFΛ −ImFΛ ImFΛ ReFΛ ] ∈ Rd×N ( 4 ) where Re and Im denote the real and imaginary parts of a complex number , and FΛ in Eq . ( 5 ) is the matrix constructed by m rows of F FΛ= e 2πik11 n · · · e 2πik1n n ... . . . ... e 2πikm1 n · · · e 2πikmn n ∈ Cm×n . ( 5 ) Proposition 1 . Suppose d = 2m , M = 2n . Construct matrix B as in Eq. ( 4 ) . Then BB > = Id and ‖bj‖2 = √ m n , ∀j ∈ { 1 , · · · , M } . Theorem 1 shows that the structured construction B satisfies the orthogonal constraint and constant norm constraint . One thing remaining is how to construct B to achieve a small mutual coherence . To achieve this goal , we can leverage the coordinate descent method in ( Lyu , 2017 ) to construct the index set Λ . For a prime number n such that m divides n−1 , i.e. , m| ( n − 1 ) , we can employ a closed-form construction . Let g denote a primitive root modulo n. We construct the index Λ = { k1 , k2 , ... , km } as Λ = { g0 , g n−1 m , g 2 ( n−1 ) m , · · · , g ( m−1 ) ( n−1 ) m } mod n. ( 6 ) The resulted structured matrix B has a bounded mutual coherence , which is shown in Theorem 1 . Theorem 1 . Suppose d = 2m , M = 2n , and n is a prime such that m| ( n − 1 ) . Construct matrix B as in Eq . ( 4 ) with index set Λ as Eq. ( 6 ) . Let mutual coherence µ ( B ) : = maxi 6=j |b > i bj | ‖bi‖2‖bj‖2 . Then µ ( B ) ≤ √ n m . Remark : The bound of mutual coherence in Theorem 1 is non-trivial when n < m2 . For the case n ≥ m2 , we can use the coordinate descent method in ( Lyu , 2017 ) to minimize the mutual coherence . Now , we show that the structured dictionary D = R > B satisfies the constant norm constraint and has a bounded mutual coherence . The results are summarized in Theorem 1 . Corollary 1 . Let D = R > B with R > R = RR > = Id . Construct matrix B as in Eq . ( 4 ) with index set Λ as Eq. ( 6 ) . Then µ ( D ) = µ ( B ) ≤ √ n m and ‖dj‖2 = ‖bj‖2 = √ m n , ∀j ∈ { 1 , · · · , M } . Corollary 1 shows that , for any orthogonal matrix R , each column vector of the structured dictionary D has a constant l2-norm . Moreover , it remains a constant mutual coherence µ ( D ) = µ ( B ) . Thus , given a fixed matrix B , we only need to learn matrix R for the dictionary learning without undermining the low mutual coherence property .

Summary: The paper is generally well presented. However, a main issue is that the optimization algorithms for the l0-norm regularized problems (Section 3.1.2 and Section 3.2) are not correctly presented. Specifically, in the algorithm development to solve the "Fix $\boldsymbol{R}$, optimize $\boldsymbol{Y}$" subproblem, it overlooks the coupling/interaction between the variables $y_1, y_2, \dots,y_M$ and mistakenly obtains a closed-form solution. See Comment 1 for details.