Understanding l4-based Dictionary Learning: Interpretation, Stability, and Robustness

1 INTRODUCTION . The explosion of massive amounts of high-dimensional data has become the modern-day norm for a large number of scientific and engineering disciplines and hence presents a daunting challenge for both computation and learning . Rising to this challenge , sparse dictionary learning ( SDL ) provides a potent framework in representation learning that exploits the blessing of dimensionality : real data tends to lie in or near some low-dimensional subspaces or manifolds , even though the ambient dimension is often extremely large ( e.g . the number of raw pixels in an image ) . More specifically , SDL ( Olshausen & Field ( 1997 ) ; Mairal et al . ( 2008 ; 2012 ; 2014 ) ; Spielman et al . ( 2012 ) ; Sun et al . ( 2015 ) ; Bai et al . ( 2018 ) ; Qu et al . ( 2019 ) ) concerns the problem of learning a compact , sparse representation from raw , unlabelled data : given a data matrix Y = [ y1 , y2 , . . . , yp ] ∈ Rn×p that contains p n-dimensional samples , one aims to find a linear transformation ( i.e . a dictionary ) D ∈ Rn×m and an associated maximally sparse representation X = [ x1 , x2 , . . . , xp ] ∈ Rm×p that satisfies Y = DX . ( 1 ) As the data matrix Y can represent a variety of signals ( e.g . images , audios , languages , and genetics etc ) in practical applications , SDL provides a versatile structure-seeking formulation that has found widespread applications in computational neuroscience , image processing , computer vision , and machine learning at large ( Olshausen & Field , 1996 ; 1997 ; Argyriou et al. , 2008 ; Ranzato et al. , 2007 ; Elad & Aharon , 2006 ; Wright et al. , 2008 ; Yang et al. , 2010 ; Zhang et al. , 2013 ; Mairal et al. , 2014 ; Zhang et al. , 2014 ; 2019 ) . Related Works . Motivated by this practical significance , there has been a growing surge of interest recently ( e.g . Rambhatla et al . ( 2019 ) ; Bai et al . ( 2018 ) ; Gilboa et al . ( 2018 ) ; Nguyen et al . ( 2018 ) ; Chatterji & Bartlett ( 2017 ) ; Mensch et al . ( 2016 ) ) that aims to tackle SDL . In attempts to recover the sparse signals X , these existing work adopt an ` 0- or ` 1-penalty function to promote the underlying sparsity and give various optimization algorithms for the resulting objectives ( some of those are heuristics while a few others have theoretical convergence guarantees ) . Although these penalty ∗Work done when interning at ByteDance . YZ would like to thank Chong You at UC Berkeley for meaningful discussions . Correspondence to : Yuexiang Zhai , ysz @ berkeley.edu or Yi Ma , yima @ eecs.berkeley.edu . Codes are available at https : //github.com/hermish/ZMZM-ICLR-2020 . functions are indeed sparsity-promoting , the resulting optimization problems must be solved one row at a time , hence resulting as many optimization problems as the ambient dimension n. Consequently , ` 0- or ` 1-based objectives result only in local methods ( i.e . can not yield the entire solution at once ) and hence entail prohibitive computational burden . Another prominent approach in SDL is Sumof-Squares ( SOS ) , proposed by and articulated in a series of recent work ( Barak et al . ( 2015 ) ; Ma et al . ( 2016 ) ; Schramm & Steurer ( 2017 ) ) . The key idea there is to utilize the properties of higher order SOS polynomials to correctly recover one column of the dictionary at a time , for which there are m columns in total . However , the computational complexity of these recovery methods are quasi-polynomial , thus again resulting in large computational expense . Very recently , in the complete dictionary learning1 setting , a novel global approach has been suggested in Zhai et al . ( 2019a ; b ) that presents a formulation which can efficiently recover the sparse signal matrix X once and for all . In particular , Zhai et al . ( 2019b ) has shown that if the generative model for Y = DoXo ∈ Rn×p satisfies that Do ∈ O ( n ; R ) is orthonormal and Xo ∈ Rn×p is Bernoulli-Gaussian sparse,2 then maximizing the ` 4-norm3 of AY over O ( n ; R ) : max A 1 4 ‖AY ‖44 subject to A ∈ O ( n ; R ) ( or AA ∗ = I ) , ( 2 ) is able to find the ground truth dictionary Do up to an arbitrary signed permutation . Moreover , Zhai et al . ( 2019b ) has proposed the simple “ Matching , Stretching , and Projection ” ( MSP ) algorithm , which is shown to be experimentally efficient and effective , for solving the program in equation 2 : MSP : At+1 = PO ( n ; R ) [ ( AtY ) ◦3Y ∗ ] = UtV ∗ t , ( 3 ) where UtV ∗t are from the singular value decomposition : UtΣtV ∗ t = SVD [ ( AtY ) ◦3Y ∗ ] . In this paper , we here give an alternative ( arguably simpler and more revealing ) derivation of the MSP algorithm ( 3 ) . Consider the Lagrangian formulation of the constrained optimization problem given in equation 2 . The necessary condition of critical points , ∇A 14 ‖AY ‖ 4 4 = ∇A 〈Λ , AA∗ − I〉 for some Lagrangian multipliers Λ , implies : ( AY ) ◦3Y ∗ = ( Λ + Λ∗ ) A . ( 4 ) As the optimization is over the orthogonal group O ( n ; R ) , restricting the condition in equation 4 onto the orthogonal group yields a necessary condition for any critical point A:4 PO ( n ; R ) [ ( AY ) ◦3Y ∗ ] = A . ( 5 ) Hence the critical point A can be viewed as a “ fixed point ” of the map : PO ( n ; R ) [ ( ( · ) Y ) ◦3Y ∗ ] from O ( n ; R ) to itself . The MSP algorithm in equation 3 is to find the fixed point ( s ) of this map . Notice that the orthonormal constraint A ∈ O ( n ; R ) in equation 2 can be viewed as enforcing the orthogonality of n unit vectors simultaneously . So , more flexibly and generally , one may choose to compute any k , for 1 ≤ k ≤ n , leading orthonormal bases of Do by solving the program : max W 1 4 ‖W ∗Y ‖44 subject to W ∈ St ( n , k ; R ) ⊂ R n×k , ( 6 ) where St ( n , k ; R ) is the Stiefel manifold.5 The orthogonal group O ( n ; R ) and the unit sphere Sn−1 can be viewed as two special cases of the Stiefel manifold St ( n , k ; R ) , with k = n and k = 1 , respectively . In some specific tasks such as dictionary learning and blind deconvolution , optimization over the unit sphere has been widely practiced , such as in Sun et al . ( 2015 ) ; Bai et al . ( 2018 ) ; Zhang et al . ( 2018 ) ; Kuo et al . ( 2019 ) . The more general setting of maximizing a convex function over any compact set also has been studied by Journée et al . ( 2010 ) in the context of sparse PCA , which has provided convergence guarantees for this class of programs . 1Complete dictionary learning requires the learned dictionary D in equation 1 to be square and invertible . 2 Each entry xi , j of X can be represented as the product of a Bernoulli variable and a normal Gaussian variable : xi , j = Ωi , jVi , j , where Ωi , j ∼iid Ber ( θ ) and Vi , j ∼iid N ( 0 , 1 ) , similar for vectors or scalars . This is the standard setting adopted in Spielman et al . ( 2012 ) ; Sun et al . ( 2015 ) ; Bai et al . ( 2018 ) . 3We abuse the notation a bit , by denoting ‖·‖44 as the sum of element-wise 4 th power of all entries of a vector and matrix , that is , ∀a ∈ Rn , ‖a‖44 = ∑n i=1 a 4 i and ∀A ∈ Rn×m , ‖A‖44 = ∑ i , j a 4 i , j . 4For any symmetric matrix S ∈ Rn×n and an orthogonal matrix A ∈ O ( n ; R ) , the projection of SA onto the orthogonal group is A : PO ( n ; R ) [ SA ] = A , one may see Absil & Malick ( 2012 ) for details . 5For any 1 ≤ k ≤ n , St ( n , k ; R ) .= { W ∈ Rn×k : W ∗W = Ik } . Our Contributions . Our contributions are twofold . First , by taking a suitable analytical angle , we reveal novel geometric and statistical connections between PCA , ICA and the ` 4-norm maximization based SDL . We then show that algorithm-wise , the fixed-point type MSP algorithm for ` 4-norm maximization has the same nature as the classic power-iteration method for PCA ( Jolliffe , 2011 ) and the FastICA algorithm for ICA ( Hyvärinen & Oja , 1997 ) . This interpretation gives a unified view for problems that pursue principal , independent , or sparse components from high-dimensional data and enriches our understanding of low-dimensional structure recovery frameworks , classical and new , at both formulation and algorithmic fronts . Second , and more importantly from a practical perspective , we examine how MSP performs under a variety of more realistic conditions , when the measurements Y could be contaminated with noise , outliers , or sparse corruptions . We show that , similar to PCA , ` 4-norm maximization and the MSP algorithm are inherently stable to small noise . Somewhat surprisingly though , unlike PCA , the MSP algorithm is further robust to outliers and resilient to sparse gross errors ! We provide characterizations of these desirable properties of MSP . The claims are further corroborated with extensive experiments on both synthetic data and real images . Taken as a whole , our results contribute to the broad landscape of dictionary learning by affirming that ` 4-maximization based SDL and the corresponding global algorithm MSP provide a valuable toolkit to the existing literature . 2 SDL VERSUS PCA AND ICA . 2.1 PURSUIT OF PRINCIPAL , INDEPENDENT , OR SPARSE COMPONENTS . Relation with the Geometric Interpretation of PCA . For a data matrix Y ∈ Rn×p , Principal Component Analysis ( PCA ) , aiming to find the top ( top k ) left singular vector ( vectors ) of Y , max W 1 2 ‖W ∗Y ‖2F subject to W ∈ St ( n , k ; R ) , ( 7 ) can be understood as finding a direction ( a k-dimensional subspace ) in row ( Y ) in which Y has the largest ` 2-norm ( Frobenius norm ) . For instance , finding the direction with the largest ` 2-norm over the unit sphere can be viewed as calculating the spectral norm ( or the largest singular value ) , of matrix Y . In comparison , we may view equation 6 : max W 1 4 ‖W ∗Y ‖44 subject to W ∈ St ( n , k ; R ) as finding a direction , or a k-dimensional subspace , in row ( Y ) where the projection of Y has the largest ` 4-norm . For instance , finding the direction with the largest ` 4-norm over the unit sphere ( equation 6 ) can be viewed as calculating the induced ‖·‖2,4 norm of matrix Y : ‖Y ‖2,4 . = maxa∈Sn−1 ‖a∗Y ‖4 . Relation with the Statistical Interpretation of PCA . View each column yj , j ∈ [ p ] of data matrix Y ∈ Rn×p as an n dimensional random vector that is i.i.d . drawn from a distribution of random variable y and let Yc denote the centered Y : Yc . = Y [ I − 1p11 ∗ ] , where 1 ∈ Rp is a vector of all 1 ’ s . Then , finding the top k principal components of Yc : maxW∈St ( n , k ; R ) 12 ‖W ∗Yc‖22 is to find k uncorrelated projections of y ∈ Rn that has the top k sample variance ( i.e . 2nd order moment ) ( Jolliffe , 2011 ; Helwig , 2017 ) . Similar to PCA , the ` 4-norm maximization of centered data matrix Yc : maxW∈St ( n , k ; R ) 14 ‖W ∗Yc‖44 can be viewed as finding k uncorrelated projections of y that have the top k sample 4th order moment , whose statistical meaning is better revealed below . Relation with ICA and Nonnormality . The ` 4-norm maximization over the Stiefel manifold is strongly related to finding the maximal or minimal kurtosis in Independent Component Analysis ( ICA ) ( Hyvärinen & Oja , 1997 ; 2000 ) : In order to identify one component of a given random vector y ∈ Rn , ICA aims to find a unit vector ( a direction ) w ∈ Sn−1 that maximizes or minimizes the kurtosis of w∗y , defined as : kurt ( w∗y ) = E ( w∗y ) 4 − 3 ‖w‖42 . ( 8 ) Kurtosis is widely used for evaluating the nonnormality of a random variable ; see DeCarlo ( 1997 ) ; Hyvärinen & Oja ( 1997 ; 2000 ) . According to Huber ( 1985 ) , the nonnormality of data carries “ abnormal ” hence interesting information in real data for many applications ( e.g . Lee et al . ( 2003 ) ; Cain et al . ( 2017 ) ) . Thus , extracting the 4th order moment helps understand such statistics of real datasets ( Hyvärinen et al. , 2009 ) and even their topology ( Carlsson , 2009 ) . One may also find that the ` 4- maximization based dictionary learning formulation is similar to maximizing kurtosis ( equation 8 ) with spherical constraint w ∈ Sn−1 , in fact , these two formulations are exactly the same if one only wants to find one column of the dictionary . Intuitively , such coincidence occurs due to the fact that maximizing ` 4-norm or kurtosis have the same effect— they both promote the “ spikiness ” ( Zhang et al . ( 2018 ) ; Li & Bresler ( 2018 ) ) ( or “ peak ” ( DeCarlo , 1997 ) ) of a distribution . More rigorous analysis regarding the similarity between ` 4-based dictionary learning and ICA is beyond the scope of this paper and we leave them for future research .

Summary: This paper presents results on Dictionary Learning through l4 maximization. The authors base this paper heavily off of the formulation and algorithm in Zhai et. al. (2019) "Complete dictionary learning via l4-norm maximization over the orthogonal group". The paper draws connections between complete dictionary learning, PCA, and ICA by pointing out similarities between the objectives functions that are optimized as well as the algorithms used. The paper further presents results on dictionary learning in the presence of different types of noise (AWGN, sparse corruptions, outliers) and show that the l4 objective is robust to different types of noise. Finally the authors apply different types of noise to synthetic and real images and show that the dictionaries that they learn are robust to the noise applied.