Subgradient Descent Learns Orthogonal Dictionaries

1 INTRODUCTION . Dictionary learning ( DL ) , i.e . , sparse coding , concerns the problem of learning compact representations , i.e. , given data Y , one tries to find a representation basis A and coefficients X , so that Y ≈ AX where X is most sparse . DL has numerous applications especially in image processing and computer vision ( Mairal et al. , 2014 ) . When posed in analytical form , DL seeks a transformation Q such that QY is sparse ; in this sense DL can be considered as an ( extremely ! ) primitive “ deep ” network ( Ravishankar & Bresler , 2013 ) . Many heuristic algorithms have been proposed to solve DL since the seminal work of Olshausen & Field ( 1996 ) , most of them surprisingly effective in practice ( Mairal et al. , 2014 ; Sun et al. , 2015 ) . However , understandings on when and how DL is solvable have only recently started to emerge . Under appropriate generating models on A and X , Spielman et al . ( 2012 ) showed that complete ( i.e. , square , invertible ) A can be recovered from Y , provided that X is ultra-sparse . Subsequent works ( Agarwal et al. , 2017 ; Arora et al. , 2014 ; 2015 ; Chatterji & Bartlett , 2017 ; Awasthi & Vijayaraghavan , 2018 ) provided similar guarantees for overcomplete ( i.e . fat ) A , again in the ultra-sparse regime . The latter methods are invariably based on nonconvex optimization with model-dependent initialization , rendering their practicality on real data questionable . The ensuing developments have focused on breaking the sparsity barrier and addressing the practicality issue . Convex relaxations based on the sum-of-squares ( SOS ) SDP hierarchy can recover overcomplete A when X has linear sparsity ( Barak et al. , 2015 ; Ma et al. , 2016 ; Schramm & Steurer , 2017 ) , while incurring expensive computation ( solving large-scale SDP ’ s or large-scale tensor decomposition ) . By contrast , Sun et al . ( 2015 ) showed that complete A can be recovered in the linear sparsity regime by solving a certain nonconvex problem with arbitrary initialization . However , the second-order optimization method proposed there is still expensive . This problem is partially addressed by ( Gilboa et al. , 2018 ) which proved that the first-order gradient descent with random initialization enjoys a similar performance guarantee . A standing barrier toward practicality is dealing with nonsmooth functions . To promote sparsity in the coefficients , the ` 1 norm is the function of choice in practical DL , as is common in modern signal processing and machine learning ( Candès , 2014 ) : despite its nonsmoothness , this choice often admits highly scalable numerical methods , such as proximal gradient method and alternating direction The reader is welcome to refer to our arXiv version for future updates . method ( Mairal et al. , 2014 ) . The analyses in Sun et al . ( 2015 ) ; Gilboa et al . ( 2018 ) , however , focused on characterizing the algorithm-independent function landscape of a certain nonconvex formulation of DL , which takes a smooth surrogate to ` 1 to get around the nonsmoothness . The tactic smoothing there introduced substantial analysis difficulty , and broke the practical advantage of computing with the simple ` 1 function . In this paper , we show that working directly with a natural ` 1 norm formulation results in neat analysis and a practical algorithm . We focus on the problem of learning orthogonal dictionaries : given data { yi } i∈ [ m ] generated as yi = Axi , where A ∈ Rn×n is a fixed unknown orthogonal matrix and each xi ∈ Rn is an iid Bernoulli-Gaussian random vector with parameter θ ∈ ( 0 , 1 ) , recover A . This statistical model is the same as in previous works ( Spielman et al. , 2012 ; Sun et al. , 2015 ) . Write Y .= [ y1 , . . . , ym ] and similarly X . = [ x1 , . . . , xm ] . We propose to recover A by solving the following nonconvex ( due to the constraint ) , nonsmooth ( due to the objective ) optimization problem : minimizeq∈Rn f ( q ) . = 1 m ∥∥q > Y ∥∥ 1 = 1 m m∑ i=1 |q > yi| subject to ‖q‖2 = 1 . ( 1.1 ) Based on the statistical model , q > Y = q > AX has the highest sparsity when q is a column of A ( up to sign ) so that q > A is 1-sparse . Spielman et al . ( 2012 ) formalized this intuition and optimized the same objective as Eq . ( 1.1 ) with a ‖q‖∞ = 1 constraint , which only works when θ ∼ O ( 1/ √ n ) . Sun et al . ( 2015 ) worked with the sphere constraint but replaced the ` 1 objective with a smooth surrogate , introducing substantial analytical and computational deficiencies as alluded above . In constrast , we show that with sufficiently many samples , the optimization landscape of formulation ( 1.1 ) is benign with high probability ( over the randomness of X ) , and a simple Riemannian subgradient descent algorithm can provably recover A in polynomial time . Theorem 1.1 ( Main result , informal version of Theorem 3.1 ) . Assume θ ∈ [ 1/n , 1/2 ] . For m ≥ Ω ( θ−2n4 log4 n ) , the following holds with high probability : there exists a poly ( m , −1 ) - time algorithm , which runs Riemannian subgradient descent on formulation ( 1.1 ) from at most O ( n log n ) independent , uniformly random initial points , and outputs a set of vectors { â1 , . . . , ân } such that up to permutation and sign change , ‖âi − ai‖2 ≤ for all i ∈ [ n ] . In words , our algorithm works also in the linear sparsity regime , the same as established in Sun et al . ( 2015 ) ; Gilboa et al . ( 2018 ) , at a lower sample complexity O ( n4 ) in contrast to the existing O ( n5.5 ) in Sun et al . ( 2015 ) . 1 As for the landscape , we show that ( Theorems 3.4 and 3.6 ) each of the desired solutions { ±ai } i∈ [ n ] is a local minimizer of formulation ( 1.1 ) with a sufficiently large basin of attraction so that a random initialization will land into one of the basins with at least constant probability . To obtain the result , we integrate and develop elements from nonsmooth analysis ( on Riemannian manifolds ) , set-valued analysis , and random set theory , which might be valuable to studying other nonconvex , nonsmooth optimization problems . 1.1 RELATED WORK . Dictionary learning Besides the many results sampled above , we highlight similarities of our result to Gilboa et al . ( 2018 ) . Both propose first-order optimization methods with random initialization , and several quantities we work with in the proofs are the same . A defining difference is we work with the nonsmooth ` 1 objective directly , while Gilboa et al . ( 2018 ) built on the smoothed objective from Sun et al . ( 2015 ) . We put considerable emphasis on practicality : the subgradient of the nonsmooth objective is considerably cheaper to evaluate than that of the smooth objective in Sun et al . ( 2015 ) , and in the algorithm we use Euclidean projection rather than exponential mapping to remain feasible—again , the former is much lighter for computation . General nonsmooth analysis While nonsmooth analytic tools such as subdifferential for convex functions are now well received in machine learning and relevant communities , that for general functions are much less so . The Clarke subdifferential and relevant calculus developed for the family of locally Lipschitz functions seem to be particularly relevant , and cover several families of functions of interest , such as convex functions , differentiable functions , and many forms of composition 1The sample complexity in Gilboa et al . ( 2018 ) is not explicitly stated . ( Clarke , 1990 ; Aubin , 1998 ; Bagirov et al. , 2014 ) . Remarkably , majority of the tools and results can be generalized to locally Lipschitz functions on Riemannnian manifolds ( Ledyaev & Zhu , 2007 ; Hosseini & Pouryayevali , 2011 ) . Our formulation ( 1.1 ) is exactly optimization of a locally Lipschitz function ( as it is convex ) on a Riemannian manifold ( the sphere ) . For simplicity , we try to avoid the full manifold language , nonetheless . Nonsmooth optimization on Riemannian manifolds or with constraints Equally remarkable is many of the smooth optimization techniques and convergence results can be naturally adapted to optimization of locally Lipschitz functions on Riemannian manifolds ( Grohs & Hosseini , 2015 ; Hosseini , 2015 ; Hosseini & Uschmajew , 2017 ; Grohs & Hosseini , 2016 ) . New optimization methods such as gradient sampling and variants have been invented to solve general nonsmooth problems ( Burke et al. , 2005 ; 2018 ; Bagirov et al. , 2014 ; Curtis & Que , 2015 ; Curtis et al. , 2017 ) . Almost all available convergence results pertain to only global convergence , which is too weak for our purpose . Our specific convergence analysis gives us a local convergence result ( Theorem 3.8 ) . Nonsmooth landscape characterization Nonsmoothness is not a big optimization barrier if the problem is convex ; here we review some recent work on analyzing nonconvex nonsmooth problems . Loh & Wainwright ( 2015 ) study the regularized empirical risk minimization problem with nonsmooth regularizers and show results of the type “ all stationary points are within statistical error of ground truth ” under certain restricted strong convexity of the smooth risk . Duchi & Ruan ( 2017 ) ; Davis et al . ( 2017 ) study the phase retrieval problem with ` 1 loss , characterizing its nonconvex nonsmooth landscape and providing efficient algorithms . There is a recent surge of work on analyzing one-hidden-layer ReLU networks , which are nonconvex and nonsmooth . Algorithm-independent characterizations of the landscape are mostly local and require strong initialization procedures ( Zhong et al. , 2017 ) , whereas stronger global results can be established via designing new loss functions ( Ge et al. , 2017 ) , relating to PDEs ( Mei et al. , 2018 ) , or problem-dependent analysis of the SGD ( Li & Yuan , 2017 ; Li & Liang , 2018 ) . Our result provides an algorithm-independent chacaterization of the landscape of non-smooth dictionary learning , and is “ almost global ” in the sense that the initialization condition is satisifed by random initialization with high probability . Other nonsmooth problems in application Prevalence of nonsmooth problems in optimal control and economics is evident from all monographs on nonsmooth analysis ( Clarke , 1990 ; Aubin , 1998 ; Bagirov et al. , 2014 ) . In modern machine learning and data analysis , nonsmooth functions are often taken to encode structural information ( e.g. , sparsity , low-rankness , quantization ) , or whenever robust estimation is desired . In deep learning , the optimization problem is nonsmooth when nonsmooth activations are in use , e.g. , the popular ReLU . The technical ideas around nonsmooth analysis , set-valued analysis , and random set theory that we gather and develop here are particularly relevant to these applications .

Summary: This paper is a direct follow-up on the Sun-Qu-Wright non-convex optimization view on the Spielman-Wang-Wright complete dictionary learning approach. In the latter paper the idea is to simply realize that with Y=AX, X being nxm sparse and A a nxn rotation, one has the property that for m large enough, the rows of X will be the sparsest element of the subspace in R^m generated by the rows of Y. This leads to a natural non-convex optimization problem, whose local optimum are hopefully the rows of X. This was proved in SWW for *very* sparse X, and then later improved in SQW to the linear sparsity scenario. The present paper refines this approach, and obtain slightly better sample complexity by studying the most natural non-convex problem (ell_1 regularization on the sphere).