Year: 2020

Abstract:
Learning overcomplete representations finds many applications in machine learning and data analytics. In the past decade, despite the empirical success of heuristic methods, theoretical understandings and explanations of these algorithms are still far from satisfactory. In this work, we provide new theoretical insights for several important representation learning problems: learning (i) sparsely used overcomplete dictionaries and (ii) convolutional dictionaries. We formulate these problems as `-norm optimization problems over the sphere, and study the geometric properties of their nonconvex optimization landscapes. For both problems, we show the nonconvex objectives have benign (global) geometric structures, which enable development of efficient optimization methods finding the target solutions. Finally, our theoretical results are justified by numerical simulations.

1 INTRODUCTION
High dimensional data often has low-complexity structures (e.g., sparsity or low rankness). The performance of modern machine learning and data analytical methods heavily depends on appropriate low-complexity data representations (or features) which capture hidden information underlying the data. While we used to manually craft representations in the past, it has been demonstrated that learned representations from the data show much superior performance (Elad, 2010). Therefore, (unsupervised) learning of latent representations of high-dimensional data becomes a fundamental problem in signal processing, machine learning, theoretical neuroscience and many other fields (Bengio et al., 2013). Moreover, overcomplete representations for which the number of latent features exceeds the data dimensionality, have shown better representation of the data in various applications compared to complete representations (Lewicki & Sejnowski, 2000; Chen et al., 2001; Rubinstein et al., 2010). In this paper, we study the following overcomplete representation learning problems.
• Overcomplete dictionary learning (ODL). One of the most important unsupervised representation learning problems is learning sparsely used dictionaries (Olshausen & Field, 1997), which finds many applications in image processing and computer vision (Wright et al., 2010; Mairal et al., 2014). The task is given data
Y lomon
data
“ A lomon
dictionary
¨ X lomon
sparse code
, (1.1)
we want to learn the compact representation (or dictionary)A P Rnˆm along with the sparse code X P Rmˆp. For better representation of the data, it is often more desired that the dictionaryA is overcomplete m ą n, where it provides greater flexibility in matching structures in the data.
• Convolutional dictionary learning (CDL). Inspired by deconvolutional networks (Zeiler et al., 2010), the convolutional form of sparse representations (Bristow et al., 2013; Garcia-Cardona & ˚The full version of this work can be found at https://arxiv.org/abs/1912.02427.
Wohlberg, 2018) replaces the unstructured dictionaryA with a set of convolution filters ta0kuKk“1. Namely, the problem is that given multiple circulant convolutional measurements
yi “ K ÿ
k“1 a0k lomon
filter
f xik lomon
sparse code
, 1 ď i ď p, (1.2)
one wants to learn the filters ta0kuKk“1 along with the sparse codes. The problem resembles a lot similarities to classical ODL. Indeed, one can show that Equation (1.2) reduces to Equation (1.1) in overcomplete settings by reformulation (Huang & Anandkumar, 2015). The interest of studying CDL was spurred by its better modeling ability of human visual and cognitive systems and the development of more efficient computational methods (Bristow et al., 2013), and has led to a number of applications in which the convolutional form provides state-of-art performance (Gu et al., 2015; Papyan et al., 2017b; Lau et al., 2019). Recently, the connections between CDL and convolutional neural network have also been extensively studied (Papyan et al., 2017a; 2018).
In addition, variants of finding overcomplete representations appear in many other problems beyond the dictionary learning problems we introduced here, such as overcomplete tensor decomposition (Anandkumar et al., 2017; Ge & Ma, 2017), overcomplete ICA (Lewicki & Sejnowski, 1998; Le et al., 2011), and short-and-sparse blind deconvolution (Zhang et al., 2017; 2018; Kuo et al., 2019).
Prior arts on dictionary learning (DL). In the past decades, numerous heuristic methods have been developed for solving DL (Lee et al., 2007; Aharon et al., 2006; Mairal et al., 2010). Despite their empirical success (Wright et al., 2010; Mairal et al., 2014), theoretical understandings of when and why these methods work are still limited.
When the dictionaryA is complete (Spielman et al., 2012) (i.e., square and invertible, m “ n), by the fact that the row space of Y equals to that of X (i.e., rowpY q “ rowpXq), Sun et al. (2016a) reduced the problem to finding the sparsest vector in a subspace (Demanet & Hand, 2014; Qu et al., 2016). By considering a (smooth) variant of the following `1-minimization problem over the sphere,
min q
1
p
› ›qJY › › 1 , s.t. q P Sn´1, (1.3)
Sun et al. (2016a) showed that the nonconvex problem has no spurious local minima when the sparsity level1 θ P Op1q, and every local minimizer q‹ is a global minimizer with qJ‹ Y corresponding to one row ofX . The new discovery has led to efficient, guaranteed optimization methods for complete DL from random initializations (Sun et al., 2016b; Bai et al., 2018; Gilboa et al., 2019).
However, all these methods critically rely on the fact that rowpY q “ rowpXq for completeA, there is no obvious way to generalize the approach to the overcomplete setting m ą n. On the other hand, for learning incoherent overcomplete dictionaries, with sparsity θ P Op1{ ? nq and stringent assumptions on X , most of the current theoretical analysis results are local (Geng et al., 2011; Arora et al., 2015; Agarwal et al., 2016; Chatterji & Bartlett, 2017), in the sense that they require complicated initializations that could be difficult to implement in practice. Therefore, the legitimate question remains: why do heuristic methods solve ODL with simple initializations?
Contributions. In this work we study the geometry of nonconvex landscapes for overcomplete/convolutional DL, where our result can be simply summarized by the following statement.
There exists nonconvex formulations for ODL/CDL with benign optimization landscapes, that descent methods can learn overcomplete/convolutional dictionaries with simple2 initializations.
Our approach follows the spirit of Sun et al. (2016a), while we overcome the aforementioned obstacles for overcomplete dictionaries by directly finding columns ofA instead of recovering sparse rows of X . We achieve this by reducing the problem to maximizing the `4-norm3 of Y Jq over the sphere,
1Here, the sparsity level θ denotes the proportion of nonzero entries in X . 2Here, for ODL simple means random initializations; for CDL, it means simple data-driven initializations. 3The use of `4-norm can also be justified from the perspective of sum of squares (SOS) (Barak et al., 2015; Ma et al., 2016; Schramm & Steurer, 2017). One can utilize properties of higher order SOS polynomials (such as 4-th order polynomials) to correctly recover columns of A. But the complexity of these methods are quasi-polynomial, and hence much more expensive than the direct optimization approach we consider here.
which is known to promote the spikiness of the solution (Zhang et al., 2018; Li & Bresler, 2018; Zhai et al., 2019). In particular, we show the following results for ODL and CDL, respectively.
1. For the ODL problem, whenA is unit norm tight frame and incoherent, our nonconvex objective is strict saddle (Ge et al., 2015; Sun et al., 2015b) in the sense that any saddle point can be escaped by negative curvature and all local minimizers are globally optimal. Furthermore, every local minimizer is close to a column ofA.
2. For the CDL problem, when the filters are self and mutual incoherent, a similar nonconvex objective is strict saddle over a sublevel set, within which every local minimizer is close to a target solution. Moreover, we develop a simple data-driven initialization that falls into this sublevel set.
Our analysis on ODL provides the first global characterization for nonconvex optimization landscape in the overcomplete regime. On the other hand, our result also gives the first provable guarantee for CDL. Indeed, under mild assumptions, our landscape analysis implies that with simple initializations, any descent method with the ability of escaping strict saddle points4 provably finds global minimizers that are close to our target solutions for both problems. Moreover, our result opens up several interesting directions on nonconvex optimization that are worth of further investigations.
2 OVERCOMPLETE DICTIONARY LEARNING
In this section, we start stating our result with ODL. In Section 3, we will show how our geometric analysis here can be extended to CDL in a nontrivial way.
2.1 BASIC ASSUMPTIONS
We study the DL problem in Equation (1.1) under the following assumptions for A P Rnˆm and X P Rmˆp. In particular, our assumption for the dictionaryA can be viewed as a generalization of orthogonality in the overcomplete setting (Mixon, 2016). Assumption 2.1 (Tight frame and incoherent dictionaryA) We assume that the dictionaryA is unit norm tight frame (UNTF) (Mixon, 2016), in the sense that
n m AAJ “ I, }ai} “ 1 p1 ď i ď mq, (2.1)
and its columns satisfy the µ-incoherence condition. Namely, letA “ ra1 a2 ¨ ¨ ¨ ams,
µpAq :“ max 1ďi “jďm
ˇ ˇ ˇ ˇ B ai }ai} , aj }aj} F ˇ ˇ ˇ ˇ P p0, 1q. (2.2)
We assume the coherence ofA is small, i.e., µpAq ! 1.
Assumption 2.2 (Random Bernoulli-GaussianX) We assume entries ofX „i.i.d. BGpθq5, that
X “ B dG, Bij „i.i.d. Berpθq, Gij „i.i.d. N p0, 1q,
where the Bernoulli parameter θ P p0, 1q controls the sparsity level ofX .
Remark 1. The coherence parameter µ plays an important role in shaping the optimization landscape. A smaller coherence µ implies that the columns of A are less correlated, and hence easier for optimization. For matrices with `2-normalized columns, classical Welch bound (Welch, 1974; Foucart & Rauhut, 2013a) suggests that the coherence µ is lower bounded by µpAq ě b
m´n pm´1qn ,
which is achieved whenA is equiangular tight frame (Sustik et al., 2007). For a generic random6 matrix A, w.h.p. it is approximately UNTF, with coherence µpAq « b
logm n roughly achieving
the order of Welch bound. For a typical dictionaryA under Assumption 2.1, this suggests that the coherence parameter µpAq often decreases w.r.t. the feature dimension n.
4Recent results show that methods such as trust-region (Absil et al., 2007; Boumal et al., 2018), cubicregularization (Nesterov & Polyak, 2006), curvilinear search (Goldfarb et al., 2017), and even gradient descent (Lee et al., 2016) can provably escape strict saddle points.
5Here, we use BGpθq for abbreviation of Bernoulli-Gaussian distribution, with sparsity level θ P p0, 1q. 6For instance, when A is random Gaussian matrix, with each entry aij „i.i.d. N p0, 1{nq.
2.2 PROBLEM FORMULATION
We solve DL in the overcomplete regime by considering the following problem
min q ϕDLpqq :“ ´ cDL p › ›qJY › › 4 4 “ ´cDL p › ›qJAX › › 4 4 , s.t. }q}2 “ 1, (2.3)
where cDL ą 0 is a normalizing constant. At the first glance, our objective looks similar to Equation (1.3) in complete DL, but we tackle the problem from a very different aspect – we directly find columns ofA instead of recovering sparse rows ofX . Given UNTFA and randomX „ BGpθq, our intuition of solving Equation (2.3) originates from the fact (Lemma D.1)
EX rϕDLpqqs “ ϕTpqq ´ θ
2p1´ θq
´m
n
¯2
, ϕTpqq :“ ´ 1
4
› ›AJq › ›
4 4 , (2.4)
where ϕTpqq can be viewed as the objective for 4th order tensor decomposition in Ge & Ma (2017). When p is large, this tells us that optimizing Equation (2.3) is approximately maximizing `4-norm of ζ “ AJq over the sphere (see Figure 1). If q equals to one of the target solutions (e.g., q “ a1),
ζpqq :“ AJq “ „
}a1}2 lomon
“1
aJ1 a2 lomon
|¨| ă µ
¨ ¨ ¨ aJ1 am lomon
|¨| ă µ
J , (2.5)
then ζ is spiky when µ is small (e.g., µ ! 1). Here, we introduce a notion of spikiness % for a vector ζ P Rm by
%pζq :“ ˇ ˇζp1q ˇ ˇ { ˇ ˇζp2q ˇ ˇ , ˇ ˇζp1q ˇ ˇ ě ˇ ˇζp2q ˇ ˇ ě ¨ ¨ ¨ ě ˇ ˇζpmq ˇ ˇ , (2.6)
where ζpiq denotes the ith ordered entry of ζ. Figure 2 shows that larger %pζq leads to larger }ζ} 4 4 with `2-norm fixed. This implies that maximizing `4-norm over the sphere promotes the spikiness of ζ (Zhang et al., 2018; Li & Bresler, 2018; Zhai et al., 2019). Thus, from Equation (2.5), we expect the global minimizer q‹ of Equation (2.3) is close to one column of A. Ge & Ma (2017) proved that for ϕTpqq there is no spurious local minimizer below a sublevel set whose measure over Sn´1 geometrically shrinks w.r.t. the dimension n, and without providing valid initialization into the set. Therefore, the challenge still remains: can simple descent methods solve the nonconvex objective Equation (2.3) to global optimality? In this work, we show that the answer is affirmative. Under proper assumptions, we show that our objective actually has benign global geometric structure, explaining why descent methods with random initialization solve the problem to the target solutions.
2.3 GEOMETRIC ANALYSIS OF NONCONVEX OPTIMIZATION LANDSCAPE To characterize the landscape of ϕDLpqq over the sphere Sn´1, let us first introduce some basic tools from Riemannian optimization (Absil et al., 2009a). For any function f : Sn´1 ÞÑ R, we have
grad fpqq :“ PqK∇fpqq, Hess fpqq :“ PqK ` ∇2fpqq ´ xq,∇fpqqy I ˘ PqK
to be the Riemannian gradient and Hessian7 of fpqq. In addition, we partition Sn´1 into two regions
RN :“ ! q P Sn´1 ˇ ˇ ϕTpqq ě ´ξDL µ2{3 }ζpqq}23 ) , (2.7)
RC :“ ! q P Sn´1 ˇ ˇ ϕTpqq ď ´ξDL µ2{3 }ζpqq}23 ) , (2.8)
for some fixed numerical constant ξDL ą 0. Unlike the approach in Sun et al. (2016a), our partition and landscape analysis are based on function value ϕTpqq instead of target solutions. This is because in overcomplete case the optimization landscape is more irregular compared to that of the complete/orthogonal case, which introduces extra difficulties for explicit partition of the sphere. In particular, for each region we show the following results. Theorem 2.3 (Global geometry of nonconvex landscape for ODL) Suppose we have
K :“ m{n, θ P ` m´1, 3´1 ˘ , ξDL ą 26, µ P ` 0, 40´1 ˘ , (2.9)
and assume Y “ AX such thatA andX satisfy Assumption 2.1 and Assumption 2.2, respectively. 1. (Negative curvature inRN) W.h.p. over the randomness ofX , whenever
p ě CθK4n6 logpθn{µq and K ď 3 ¨ ´ 1` 6µ` 6ξ3{5DLµ 2{5 ¯´1 ,
any point q P RN exhibits negative curvature in the sense that
D v P Sn´1, s.t. vJHessϕDLpqqv ď ´3 }ζ}44 }ζ} 2 8 .
2. (No bad critical points inRC) W.h.p. over the randomness ofX , whenever
p ě CθK3 max µ´2,Kn2 ( n3 logpθn{µq and K ď ξ3{2DL{8,
every critical point qc of ϕDLpqq in RC is either a strict saddle point that exhibits negative curvature for descent, or it is near one of the target solutions (e.g. a1) such that
xa1{ }a1} , qcy ě 1´ 5ξ´3{2DL .
Here C ą 0 is a universal constant.
Remark 2. A combination of our geometric analysis for both regions provides the first global geometric analysis for ODL with θ P Op1q, which implies that ϕDLpqq has no spurious local minimizers over Sn´1: any critical point is either a strict saddle point that can be efficiently escaped, or it is near one of the target solutions. Moreover, recent results show that nonconvex problems with this type of optimization landscapes can be solved to optimal solutions by using (noisy) gradient descent methods with random initializations (Lee et al., 2016; Jin et al., 2017; Lee et al.; Criscitiello & Boumal, 2019). In addition, we point out several limitations of our result for future work. • As we have only characterized properties of critical points, our result does not directly lead to
convergence rate for descent methods. To show polynomial-time convergence, as suggested by Sun et al. (2016a; 2018); Li & Bresler (2018); Kuo et al. (2019), we need finer partitions of the sphere and uniform controls of derivatives in each region8. We leave this for future work.
• Our analysis in RN says that when µ is sufficiently small9 the maximum overcompleteness K allowed is roughly K “ 3, which is smaller than that of RC (which could be a large constant). We believe this is mainly due to loose bounds for controlling norms ofA inRC. Moreover, our experiment result in Section 4 suggests that there is a substantial gap of K between our theory and practice: the phase transition in Figure 3a shows that gradient descent with random initialization works even in the regime m ď n2. We leave improvement of our result as an open question.
7The Riemannian derivatives are similar to ordinary derivatives in Euclidean space, but they are defined in the tangent space of the manifoldM “ Sn´1. We refer readers to Absil et al. (2009a) for more details.
8Our preliminary investigation indicates that our premature analysis is not tight enough to achieve this. 9From Remark 1, for a typical A, we expect µ P rOppnKq´1{2q to be diminishing w.r.t. n.
Brief sketch of analysis. From Equation (2.4), we know that ϕDLpqq reduces to ϕTpqq in large sample limit as pÑ8. This suggests an expectation and concentration type of analysis: (i) we first characterize critical points and negative curvature for the deterministic function ϕTpqq inRC and RN (see Appendix B); (ii) for any small δ ą 0, we show the measure concentrates in the sense that for a finitely large p ě rΩpδ´2polypnqq,
sup qPSn´1 }gradϕDLpqq ´ gradϕTpqq} ď δ, sup qPSn´1 }HessϕDLpqq ´ HessϕTpqq} ď δ
holds w.h.p. over the randomness ofX . Thus we can turn our analysis of ϕTpqq to that of ϕDLpqq by a perturbation analysis (see Appendix C & D). Here, it should be noticed that gradϕDLpqq and HessϕDLpqq are 4th-order polynomials ofX , which are heavy-tailed empirical processes over q P Sn´1. To control suprema of heavy-tailed processes, we developed a general truncation and concentration type of analysis similar to Zhang et al. (2018); Zhai et al. (2019), so that we can utilize classical bounds for sub-exponential random variables (Boucheron et al., 2013) (see Appendix F).
3 CONVOLUTIONAL DICTIONARY LEARNING
3.1 PROBLEM FORMULATION
Recall from Section 1, the basic task of CDL is that given convolutional measurements in the form of Equation (1.2), we want to recover kernels ta0kuKk“1. Here, by reformulating10 CDL in the form of ODL, we generalize our analysis from Section 2.3 to CDL with a few new ingredients.
Reduction from CDL to ODL. For any z P Rn, let Cz P Rnˆn be the circulant matrix generated from z. From Equation (1.2), the properties of circulant matrix imply that
Cyi “ CřKk“1 a0kfxik “ K ÿ
k“1 Ca0kCxik “ A0 ¨Xi, 1 ď i ď p,
with A0 “ rCa01 Ca02 ¨ ¨ ¨ Ca0K s and Xi “ “ CJxi1 C J xi2 ¨ ¨ ¨ C J xiK
‰J , so that
A0 P RnˆnK is overcomplete and structured. Thus, contencating all Cyi , we have “
Cy1 Cy2 ¨ ¨ ¨ Cyp ‰ loooooooooooooomoooooooooooooon
Y PRnˆnp
“ A0 ¨ rX1 X2 ¨ ¨ ¨ Xps loooooooooooomoooooooooooon
XPRnKˆnp
ùñ Y “ A0 ¨X.
This suggests that we can view the CDL problem as ODL: if we can recover a column of the overcomplete dictionaryA0, we find one of the filters a0k p1 ď k ď Kq up to a circulant shift11.
Nonconvex problem formulation and preconditioning. To solve CDL, one may consider the same objective Equation (2.3) as ODL. However, for many applications our structured dictionaryA0 could be badly conditioned and not tight frame, which results in bad optimization landscape and even spurious local minimizers. To deal with this issue, we whiten our data Y by preconditioning12
PY “ PA0X, P “ ” ` θK2np ˘´1 Y Y J ı´1{2 . (3.1)
For large p, we approximately have P « ` K´1A0A J 0
˘´1{2 (see Appendix E.5), so that
PY « ` K´1A0A J 0 ˘´1{2 A0 ¨X “ A ¨X, A :“ ` K´1A0A J 0 ˘´1{2 A0,
whereA is automatically tight frame with K´1AAJ “ I . This suggests to consider
min q ϕCDLpqq :“ ´ cCDL np › ›qJ pPY q › › 4 4 , s.t. }q}2 “ 1, (3.2)
10Similar formulation ideas also appeared in (Huang & Anandkumar, 2015) with no theoretical guarantees. 11The CDL problem exhibits shift symmetry in the sense that a0k fxik “ s` ra0ks f s´` rxiks, where s` r¨s is a circulant shift operator by length `. This implies we can only hope to solve CDL up to a shift ambiguity. 12Again, the θ here is only for normalization purpose, which does not affect optimization landscape. Similar P is also considered in Sun et al. (2016a); Zhang et al. (2018); Qu et al. (2019).
Algorithm 1 Finding one filter with data-driven initialization
Input: data Y P Rnˆp Output: an esimated filter a‹
1: preconditioning. Cook up the preconditioning matrix P in Equation (3.1). 2: initialization. Initialize qinit “ PSn´1 pPy`q with a random sample y`, 1 ď ` ď p. 3: optimization with escaping saddle points. Optimize Equation (3.2) to a local minimizer q‹, by
using a descent method such as Goldfarb et al. (2017) that escapes strict saddle points. 4: return an estimated filter a‹ “ PSn´1 ` P´1q‹ ˘ .
for some normalizing constant cCDL ą 0, so that is close to optimizing
pϕCDLpqq :“ ´ cCDL np › ›qJAX › › 4 4 « ϕCDLpqq,
for a tight frame dictionaryA (we make this rigorous in Appendix E.4). To study the problem, we make assumptions on the sparse signals xik „i.i.d. BGpθq similar to Assumption 2.2. Furthermore, we assumeA0 andA satisfy the following properties which serve as counterparts to Assumption 2.1.
Assumption 3.1 (Properties ofA0 andA) We assume the filter matrixA0 has minimum singular value σminpA0q ą 0 with bounded condition number κpA0q :“ σmaxpA0q{σminpA0q. In addition, we assume the columns ofA are mutually incoherent: maxi “j ˇ ˇ ˇ A
ai }ai} , aj }aj}
E ˇ
ˇ ˇ ď µ.
3.2 GEOMETRIC ANALYSIS AND NONCONVEX OPTIMIZATION
Optimization landscape for CDL. We characterize the geometric structure of ϕCDLpqq over
RCDL :“ ! q P Sn´1 ˇ ˇ ϕTpqq ď ´ξCDL µ2{3κ4{3pA0q }ζpqq}23 ) , (3.3)
for some fixed numerical constant ξCDL ą 0, where ζpqq “ AJq and ϕTpqq “ ´4´1 }ζpqq}44 as introduced in Equation (2.4). We show ϕCDLpqq satisfies the following properties.
Theorem 3.2 (Local geometry of nonconvex landscape for CDL) Let us denote m :“ Kn, and let C0 ą 5 and η ă 2´6 be some positive constants. Suppose we have
θ P ` m´1, 3´1 ˘ , ξCDL “ C0 ¨ η´2{3, µ P ` 0, 40´1 ˘ ,
and assume that Assumption 3.1 and xik „i.i.d. BGpθq hold. There exists some constant C ą 0, w.h.p. over the randomness of xiks, whenever
p ě CθK2µ´2n4 max " K6κ6pA0q σ2minpA0q , n * log6pn{µq and K ă C0,
every critical point qc in RCDL is either a strict saddle point that exhibits negative curvature for descent, or it is near one of the target solutions (e.g. a1) such that xa1{ }a1} , qcy ě 1´ 5κ´2η. Remark 3. The analysis is similar to that of ODL inRC (see Appendix D). In contrast, our sample complexity p andRCDL have extra dependence on κpA0q due to preconditioning in Equation (3.1). On the other hand, because our preconditioned dictionaryA is tight frame but not necessarily UNTF, in the worst case we cannot exclude existence of spurious local minima inRcCDL Ş
Sn´1 for CDL. From geometry to optimization. Nonetheless, in Algorithm 1 we propose a simple data-driven initialization qinit such that qinit P RCDL. Since RCDL does not have bad local minimizers, by proving that all iterates stay withinRCDL, it suffices to show global convergence of Algorithm 1. We initialize q by randomly picking a preconditioned data sample Py` with ` P rps, and set
qinit “ PSn´1 pPy`q , s.t. ζinit “ AJqinit « ? KPSnK´1 ` AJAx` ˘ . (3.4)
For genericA, small µpAq implies thatAJA is close to a diagonal matrix13, so that ζinit is spiky for sparse x`. Therefore, we expect large }ζinit}44 and qinit P RCDL by leveraging sparsity of x`.
13This is because the off diagonal entries are bounded roughly by ? Kµ, which are tiny when µ is small.
Proposition 3.3 (Convergence of Algorithm 1 to target solutions) With m “ Kn, suppose
c1 logm
m ď θ ď c2
µ´2{3
κ4{3m logm ¨min
" κ4{3 µ4{3 , Kµ´4 m2 logm * . (3.5)
W.h.p. over the randomness of xiks, whenever
p ě CθK2µ´2 max K6κ6pA0q{σ2minpA0q, n ( n4 log6 pm{µq ,
we have qinit P RCDL, and all future iterates of Algorithm 1 stay withinRCDL and converge to an approximate solution (e.g., some circulant shift s` ra01s of a0k with 1 ď ` ď n) in the sense that
› ›PSn´1 ` P´1q‹ ˘ ´ s` ra01s › › ď ,
where is a small numerical constant. Here, c1, , c2, C ą 0 are some numerical constants.
Remark 4. Our result (Equation (3.5)) suggests that there is a tradeoff between µ and θ for optimization. For generic filters (e.g. drawn uniformly from the sphere), we approximately have14 µ P rOpm´1{2q and κ P Op1q, so that our theory suggests the maximum sparsity allowed is θ P rOpm´2{3q. For other smoother filters which may have larger µ and κ, the sparsity θ allowed tends to be smaller. Improving Equation (3.5) is the subject of future work. On the other hand, our result guarantees convergence to an approximate solution of constant error. We left exact recovery for future work. Finally, although we write CDL in the matrix-vector form, the optimization could be implemented very efficiently using fast Fourier transform (FFT) (see Appendix G).
4 EXPERIMENTS
In this section, we experimentally demonstrate our proposed formulation and approach for ODL and CDL. We solve our nonconvex problems in Equation (2.3) and Equation (3.2) using optimization methods15 with random initializations introduced in Appendix G.
14See Figure 3 of Zhang et al. (2018) for an illustration of these estimations. 15For simplicity, we use power method (see Algorithm 3) for optimizing without tuning step sizes. In practice,
we find both power method and Riemannian gradient descent have similar performance.
Experiments on ODL. We generate data Y “ AX , with dictionary A P Rnˆm being UNTF16, and sparse codeX P Rmˆp „i.i.d. BGpθq. To judge the success recovery of one column ofA, let
ρe “ min 1ďiďm p1´ |xq‹,ai{ }ai}y|q .
We have ρe “ 0 when q‹ “ PSn´1paiq, thus we assume a recovery is successful if ρe ă 5ˆ 10´2. • Overcompleteness. First, we fix θ “ 0.1, and test the limit of the overcompleteness K “ m{n
we can achieve by plotting the phase transition on pm,nq in log scale. To get rid of the influence of sample complexity p, we run our algorithm on ϕTpqq which is the sample limit of ϕDLpqq. For each pair of pm,nq, we repeat the experiment for 12 times. As shown in Figure 3a, it suggests that the limit of overcompleteness is roughly m « n2, which is much larger than our theory predicts. • Recovering full matrixA. Second, although our theory only guarantees recovery of one column ofA, Figure 3b suggests that we can recover the full dictionaryA by repetitive independent trials. As the result shows, Opm logmq independent runs suffice to recover the fullA. • Recovery with varying θ and p. Our simulation in Figure 3c implies that we need more samples p when the overcompleteness K increases. Meanwhile, Figure 3d shows successful recovery even when sparsity θ « 0.3. The maximum θ seems to remain as a constant when n increases.
Experiments on CDL. Finally, for CDL, we generate measurement according to Equation (1.2) with K “ 3, where the filters ta0kuKk“1 are drawn uniformly from the sphere Sn´1, and xik „i.i.d. BGpθq. Figure 4 shows that our method can approximately recover all the filters by running a few number of repetitive independent trials.
5 CONCLUSION AND DISCUSSION
In this work, we showed that nonconvex landscapes of overcomplete representation learning also possess benign geometric structures. In particular, by reducing the problem to an `4 optimization problem over the sphere, we proved that ODL has no spurious local minimizers globally: every critical point is either an approximate solution or a saddle point can be efficiently escaped. Moreover, we showed that this type of analysis can be carried over to CDL with a few new ingredients, leading to the first provable method for solving CDL globally. Our results have opened several interesting questions that are worth of further investigations, that we discuss as follows.
Tighter bound on overcompleteness for ODL. As shown in Theorem 2.3, our bound on the overcompleteness K “ m{n is an absolute constant, which we believe is far from tight (see experiments in Figure 3a). In the high overcompleteness regime (e.g., n ! m ď n2), one conjecture is that spurious local minimizer does exist but descent methods with random initializations implicitly regularizes itself such that bad regions are automatically avoided Ma et al. (2017); another conjecture is that there is actually no spurious local minimizers. We tend to believe the latter conjecture is true. Indeed, the looseness of our analysis appears in the regionRN (see Appendix B.2), for controlling the norms ofA.
One idea might be to consider i.i.d. Gaussian dictionary instead of the deterministic incoherent dictionaryA, and use probabilistic analysis instead of the worst-case deterministic analysis. However, our preliminary analysis suggests that elementary concentration tools for Gaussian empirical processes are not sufficient to achieve this goal. More advanced probabilistic tools might be needed here.
Another idea that might be promising is to leverage more advanced tools such as the sum of squares (SoS) techniques Lasserre (2001); Blekherman et al. (2012). Previous results Barak et al. (2015); Ma et al. (2016); Hopkins et al. (2015) used SoS as a computational tool for solving this type of problems, while the computational complexity is often quasi-polynomial and hence cannot handle problems of large-scale. In contrast, our idea here is to use SoS to verify the geometric structure of the optimizing landscape instead of computation, to have a better uniform control of the negative curvature inRN. If we succeeded, this might lead to a tighter bound on the overcompleteness. Moreover, analogous to building dual certificates for convex relaxations such as compressive sensing Candès & Wakin (2008); Candes & Plan (2011) and matrix completion Candès & Recht (2009); Candès et al. (2011), it could potentially lead to a more general approach for verifying benign geometry structures for nonconvex optimization.
16The UNTF dictionary is generated by Tropp et al. (2005): (i) generate a standard Gaussian matrix A0, (ii) from A0 alternate between preconditioning the matrix and normalize the columns until convergence.
Composition rules for nonconvex optimization? Another interesting phenomenon we found through understanding ODL is that under certain scenarios the benign nonconvex geometry can be preserved under nonnegative addition. Indeed, if we separate our dictionary A into several subdictionaries as A “ rA1 ¨ ¨ ¨ AN s, then the asymptotic version of nonconvex objective for ODL (Equation (2.3)) can be rewritten as
ϕTpqq “ ´ 1
4
› ›AJq › ›
4 4 “
N ÿ k“1 ϕkTpqq, ϕkTpqq :“ ´ 1 4 › ›AJk q › › 4 4 p1 ď k ď Nq. (5.1)
Presumably, every function ϕkTpqq also possess benign geometry for each submatrix Ak. This discovery might suggest more general properties in nonconvex optimization – benign geometry structures can be preserved under certain composition rules. Analogous to the study of convex functions Boyd & Vandenberghe (2004), discovering composition rules can potentially lead to simpler analytical tools for studying nonconvex optimization problems and hence have broad impacts.
Finding all components over Stiefel or Oblique manifolds. The nonconvex formulations considered in this work is only guaranteed to recover one column/filter at a time for ODL/CDL. Although our experimental results in Section 4 implies that the full dictionary or all the filters can be recovered by using repetitive independent trials, it is more desirable to have a formulation that can recover the whole dictionary/filters in one shot. This requires us to consider optimization problems constraint over more complicated manifolds rather than the sphere, such as Stiefel and Oblique manifolds Absil et al. (2009a). Despite of recent empirical evidences Lau et al. (2019); Li et al. (2019) and study of local geometry Zhai et al. (2019); Zhu et al. (2019), more technical tools need to be developed towards better understandings for nonconvex problems constraint over these more complicated manifolds.
Miscellaneous. Finally, we summarize several small questions that might be also worth of pursuing.
• Exact recovery. Our results only lead to approximate recovery of the target solutions. To obtain exact solutions, one might need to consider similar rounding steps as introduced in Qu et al. (2016); Sun et al. (2016b); Qu et al. (2019).
• Designing better loss functions. The `4 objective we considered here for ODL and CDL is heavy-tailed for sub-Gaussian random variables, resulting in bad sample complexity and large approximation error. It would be nice to design better loss functions that also promotes spikiness of the solutions.
• Non-asymptotic convergence for descent methods. Unlike the results in Sun et al. (2016a;c); Kuo et al. (2019), our geometric analysis here does not directly lead to non-asymptotic convergence guarantees of any descent methods to global minimizers. This is because we only characterized the geometric properties of critical points on the function landscape. To show non-asymptotic convergence of methods introduced in Appendix G, we need to uniformly characterize the geometric properties over the sphere.
• Finer models for CDL. Finally, for CDL, it is worth noting that in many cases the length of the filters ta0kuKk“1 is often much shorter than the observations tyiu p i“1 Zhang et al. (2017); Kuo et al.
(2019); Zhang et al. (2018); Lau et al. (2019), which has not been considered in this work. The extra structure leads to the so-called short-and-sparse CDL Lau et al. (2019), where the lower dimensional model can lead to fewer samples for recovery. Based on our results, we believe the short structure can be dealt with by developing finer analysis such as that in Kuo et al. (2019).
ACKNOWLEDGEMENT
Part of this work was done when QQ and YXZ were at Columbia University. QQ thanks the generous support of the Microsoft graduate research fellowship and Moore-Sloan fellowship. XL would like to acknowledge the support by Grant CUHK14210617 from the Hong Kong Research Grants Council. YQZ is grateful to be supported by NSF award 1740822. ZZ was partly supported by NSF Grant 1704458. The authors would like to thank Joan Bruna (NYU Courant), Yuxin Chen (Princeton University), Lijun Ding (Cornell University), Han-wen Kuo (Columbia University), Shuyang Ling (NYU Shanghai), Yi Ma (UC Berkeley), Ju Sun (University of Minnesota, Twin Cities), René Vidal (Johns Hopkins University), and John Wright (Columbia University) for helpful discussions and inputs regarding this work.
REFERENCES P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton
University Press, 2009a.
Pierre-Antoine. Absil, Christopher G. Baker, and Kyle A. Gallivan. Trust-region methods on Riemannian manifolds. Foundations of Computational Mathematics, 7(3):303–330, 2007.
Pierre-Antoine. Absil, Robert Mahoney, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009b.
Alekh Agarwal, Animashree Anandkumar, Prateek Jain, and Praneeth Netrapalli. Learning sparsely used overcomplete dictionaries via alternating minimization. SIAM Journal on Optimization, 26(4):2775–2799, 2016.
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11):4311–4322, 2006.
Animashree Anandkumar, Rong Ge, and Majid Janzamin. Analyzing tensor power method dynamics in overcomplete regime. The Journal of Machine Learning Research, 18(1):752–791, 2017.
Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. Journal of Machine Learning Research, 40(2015), 2015.
Yu Bai, Qijia Jiang, and Ju Sun. Subgradient descent learns orthogonal dictionaries. arXiv preprint arXiv:1810.10702, 2018.
Boaz Barak, Jonathan A Kelner, and David Steurer. Dictionary learning and tensor decomposition via the sum-of-squares method. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pp. 143–151. ACM, 2015.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
Grigoriy Blekherman, Pablo A Parrilo, and Rekha R Thomas. Semidefinite optimization and convex algebraic geometry. SIAM, 2012.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
Nicolas Boumal, Pierre-Antoine Absil, and Coralia Cartis. Global rates of convergence for nonconvex optimization on manifolds. IMA Journal of Numerical Analysis, 39(1):1–33, 2018.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Hilton Bristow, Anders Eriksson, and Simon Lucey. Fast convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 391–398, 2013.
Emmanuel J Candes and Yaniv Plan. A probabilistic and ripless theory of compressed sensing. IEEE transactions on information theory, 57(11):7235–7254, 2011.
Emmanuel J Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foundations of Computational mathematics, 9(6):717, 2009.
Emmanuel J Candès and Michael B Wakin. An introduction to compressive sampling [a sensing/sampling paradigm that goes against the common knowledge in data acquisition]. IEEE signal processing magazine, 25(2):21–30, 2008.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):11, 2011.
Niladri Chatterji and Peter L Bartlett. Alternating minimization for dictionary learning with random initialization. In Advances in Neural Information Processing Systems, pp. 1997–2006, 2017.
Scott Shaobing Chen, David L Donoho, and Michael A Saunders. Atomic decomposition by basis pursuit. SIAM review, 43(1):129–159, 2001.
C. Criscitiello and N. Boumal. Efficiently escaping saddle points on manifolds. In To appear in the proceedings of NeurIPS, 2019.
Laurent Demanet and Paul Hand. Scaling law for recovering the sparsest element in a subspace. Information and Inference: A Journal of the IMA, 3(4):295–309, 2014.
Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010.
Simon Foucart and Holger Rauhut. An invitation to compressive sensing. In A mathematical introduction to compressive sensing, pp. 1–39. Springer, 2013a.
Simon Foucart and Holger Rauhut. A Mathematical Introduction to Compressive Sensing. Springer, 2013b.
Cristina Garcia-Cardona and Brendt Wohlberg. Convolutional dictionary learning: A comparative review and new algorithms. IEEE Transactions on Computational Imaging, 4(3):366–381, 2018.
Rong Ge and Tengyu Ma. On the optimization landscape of tensor decompositions. In Advances in Neural Information Processing Systems, pp. 3653–3663, 2017.
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points—online stochastic gradient for tensor decomposition. In Proceedings of The 28th Conference on Learning Theory, pp. 797–842, 2015.
Quan Geng, Huan Wang, and John Wright. On the local correctness of lˆ 1 minimization for dictionary learning. arXiv preprint arXiv:1101.5672, 2011.
Dar Gilboa, Sam Buchanan, and John Wright. Efficient dictionary learning with gradient descent. In International Conference on Machine Learning, pp. 2252–2259, 2019.
Donald Goldfarb, Cun Mu, John Wright, and Chaoxu Zhou. Using negative curvature in solving nonlinear programs. Computational Optimization and Applications, 68(3):479–502, 2017.
Shuhang Gu, Wangmeng Zuo, Qi Xie, Deyu Meng, Xiangchu Feng, and Lei Zhang. Convolutional sparse coding for image super-resolution. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1823–1831, 2015.
Samuel B Hopkins, Jonathan Shi, and David Steurer. Tensor principal component analysis via sum-of-square proofs. In Conference on Learning Theory, pp. 956–1006, 2015.
Furong Huang and Animashree Anandkumar. Convolutional dictionary learning through tensor factorization. In Feature Extraction: Modern Questions and Challenges, pp. 116–129, 2015.
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1724–1732. JMLR. org, 2017.
Michel Journée, Yurii Nesterov, Peter Richtárik, and Rodolphe Sepulchre. Generalized power method for sparse principal component analysis. Journal of Machine Learning Research, 11(Feb):517–553, 2010.
Han-Wen Kuo, Yuqian Zhang, Yenson Lau, and John Wright. Geometry and symmetry in short-and-sparse deconvolution. In International Conference on Machine Learning, pp. 3570–3580, 2019.
Jean B Lasserre. Global optimization with polynomials and the problem of moments. SIAM Journal on optimization, 11(3):796–817, 2001.
Yenson Lau, Qing Qu, Han-Wen Kuo, Pengcheng Zhou, Yuqian Zhang, and John Wright. Short-and-sparse deconvolution–a geometric approach. arXiv preprint arXiv:1908.10959, 2019.
Quoc V Le, Alexandre Karpenko, Jiquan Ngiam, and Andrew Y Ng. Ica with reconstruction cost for efficient overcomplete feature learning. In Advances in neural information processing systems, pp. 1017–1025, 2011.
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Y Ng. Efficient sparse coding algorithms. In Advances in neural information processing systems, pp. 801–808, 2007.
Jason D Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael I Jordan, and Benjamin Recht. First-order methods almost always avoid strict saddle points. Mathematical Programming, pp. 1–27.
Jason D Lee, Max Simchowitz, Michael I Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Conference on Learning Theory, pp. 1246–1257, 2016.
Michael S Lewicki and Terrence J Sejnowski. Learning nonlinear overcomplete representations for efficient coding. In Advances in neural information processing systems, pp. 556–562, 1998.
Michael S Lewicki and Terrence J Sejnowski. Learning overcomplete representations. Neural computation, 12 (2):337–365, 2000.
Xiao Li, Shixiang Chen, Zengde Deng, Qing Qu, Zhihui Zhu, and Anthony Man Cho So. Nonsmooth optimization over stiefel manifold: Riemannian subgradient methods. arXiv preprint arXiv:1911.05047, 2019.
Yanjun Li and Yoram Bresler. Global geometry of multichannel sparse blind deconvolution on the sphere. In Advances in Neural Information Processing Systems, pp. 1132–1143, 2018.
Cong Ma, Kaizheng Wang, Yuejie Chi, and Yuxin Chen. Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval, matrix completion and blind deconvolution. arXiv preprint arXiv:1711.10467, 2017.
Tengyu Ma, Jonathan Shi, and David Steurer. Polynomial-time tensor decompositions with sum-of-squares. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 438–446. IEEE, 2016.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research, 11(Jan):19–60, 2010.
Julien Mairal, Francis Bach, Jean Ponce, et al. Sparse modeling for image and vision processing. Foundations and Trends R© in Computer Graphics and Vision, 8(2-3):85–283, 2014.
Dustin G Mixon. Unit norm tight frames in finite-dimensional spaces. Finite Frame Theory: A Complete Introduction to Overcompleteness, 93:53, 2016.
Yurii Nesterov and Boris T. Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming, 108(1):177–205, 2006.
Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research, 18(1):2887–2938, 2017a.
Vardan Papyan, Yaniv Romano, Jeremias Sulam, and Michael Elad. Convolutional dictionary learning via local processing. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5296–5304, 2017b.
Vardan Papyan, Yaniv Romano, Jeremias Sulam, and Michael Elad. Theoretical foundations of deep learning via sparse representations: A multilayer sparse model and its connection to convolutional neural networks. IEEE Signal Processing Magazine, 35(4):72–89, 2018.
Qing Qu, Ju Sun, and John Wright. Finding a sparse vector in a subspace: Linear sparsity using alternating directions. IEEE Transactions on Information Theory, 62(10):5855–5880, 2016.
Qing Qu, Xiao Li, and Zhihui Zhu. A nonconvex approach for exact and efficient multichannel sparse blind deconvolution. arXiv preprint arXiv:1908.10776, 2019.
Ron Rubinstein, Alfred M Bruckstein, and Michael Elad. Dictionaries for sparse representation modeling. Proceedings of the IEEE, 98(6):1045–1057, 2010.
Tselil Schramm and David Steurer. Fast and robust tensor decomposition with applications to dictionary learning. Proceedings of Machine Learning Research vol, 65:1–34, 2017.
Daniel A. Spielman, Huan Wang, and John Wright. Exact recovery of sparsely-used dictionaries. In Conference on Learning Theory, 2012.
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere. arXiv preprint arXiv:1504.06785, 2015a.
Ju Sun, Qing Qu, and John Wright. When are nonconvex problems not scary? arXiv preprint arXiv:1510.06096, 2015b.
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2):853–884, 2016a.
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere ii: Recovery by riemannian trust-region method. IEEE Transactions on Information Theory, 63(2):885–914, 2016b.
Ju Sun, Qing Qu, and John Wright. A geometric analysis of phase retreival. arXiv preprint arXiv:1602.06664, 2016c.
Ju Sun, Qing Qu, and John Wright. A geometric analysis of phase retrieval. Foundations of Computational Mathematics, 18(5):1131–1198, 2018.
Mátyás A Sustik, Joel A Tropp, Inderjit S Dhillon, and Robert W Heath Jr. On the existence of equiangular tight frames. Linear Algebra and its applications, 426(2-3):619–635, 2007.
Joel A Tropp, Inderjit S Dhillon, Robert W Heath, and Thomas Strohmer. Designing structured tight frames via an alternating projection method. IEEE Transactions on information theory, 51(1):188–209, 2005.
Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends R© in Machine Learning, 8(1-2):1–230, 2015.
Lloyd Welch. Lower bounds on the maximum cross correlation of signals (corresp.). IEEE Transactions on Information theory, 20(3):397–399, 1974.
John Wright, Yi Ma, Julien Mairal, Guillermo Sapiro, Thomas S Huang, and Shuicheng Yan. Sparse representation for computer vision and pattern recognition. Proceedings of the IEEE, 98(6):1031–1044, 2010.
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition, pp. 2528–2535. IEEE, 2010.
Yuexiang Zhai, Zitong Yang, Zhenyu Liao, John Wright, and Yi Ma. Complete dictionary learning via `4-norm maximization over the orthogonal group. arXiv preprint arXiv:1906.02435, 2019.
Yuqian Zhang, Yenson Lau, Han-wen Kuo, Sky Cheung, Abhay Pasupathy, and John Wright. On the global geometry of sphere-constrained sparse blind deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4894–4902, 2017.
Yuqian Zhang, Han-Wen Kuo, and John Wright. Structured local optima in sparse blind deconvolution. arXiv preprint arXiv:1806.00338, 2018.
Zhihui Zhu, Tianyu Ding, Daniel Robinson, Manolis Tsakiris, and René Vidal. A linearly convergent method for non-smooth non-convex optimization on the grassmannian with applications to robust subspace and dictionary learning. In Advances in Neural Information Processing Systems, pp. 9437–9447, 2019.
APPENDIX
The Appendix is organized as follows. In Appendix A, we introduce the basic notations and technical tools for analysis. Appendix B provides a determinsitic characterization of the optimization landscape in population. In Appendix C, we turn our analysis of Appendix B into finite sample version. Appendix D and Appendix E provide the detailed proof for ODL and CDL, respectively. The detailed concentration bounds are postponed to Appendix F. Finally, Appendix G provides some ideas of optimization methods.
A NOTATIONS AND BASIC TOOLS
A.1 BASIC NOTATIONS
Throughout this paper, all vectors/matrices are written in bold font a/A; indexed values are written as ai, Aij . We use Sn´1 to denote an n-dimensional unit sphere in the Euclidean space Rn. We let rms “ t1, 2, ¨ ¨ ¨ ,mu. We use d to denote Hadamard product between two vectors/matrices. For v P Rn, we use vdr to denote entry-wise power of order m, i.e., vdr “ rvr1, ¨ ¨ ¨ , vrns
J. Let Fn P Cnˆn denote a unnormalized nˆ n DFT matrix, with }Fn} “ ? n, and F´1n “ n´1F ˚n . In many cases, we just use F to denote the DFT matrix.
Some basic operators. We use Pv and PvK to denote projections onto v and its orthogonal complement, respectively. We let PSn´1 to be the `2-normalization operator. To sum up, we have
PvKu “ u´ vvJ
}v}2 v, Pvu “
vvJ }v}2 u, PSn´1v “ v }v} .
Circular convolution and circulant matrices. The convolution operator f is circular with modulo-m: paf xqi “ řm´1 j“0 ajxi´j . For v P Rm, let s`rvs denote the cyclic shift of v with length `. Thus, we can introduce the circulant matrix Cv P Rmˆm generated through v P Rm,
Cv “
»
— — — — — –
v1 vm ¨ ¨ ¨ v3 v2 v2 v1 vm v3 ... v2 v1 . . . ...
vm´1 . . . . . . vm vm vm´1 ¨ ¨ ¨ v2 v1
fi
ffi ffi ffi ffi ffi fl “ rs0 rvs s1 rvs ¨ ¨ ¨ sm´1 rvss . (A.1)
Now the circulant convolution can also be written in a simpler matrix-vector product form. For instance, for any u P Rm and v P Rm,
uf v “ Cu ¨ v “ Cv ¨ u, Cufv “ CuCv. In addition, the correlation between u and v can be also written in a similar form of convolution operator which reverses one vector before convolution.
Basics of Riemannian derivatives. Here, we give a brief introduction to manifold optimization over the sphere, and the forms of Riemannian gradient and Hessian. We refer the readers to the book (Absil et al., 2009b) for more backgrounds. Given a point q P Sn´1, the tangent space TqSn´1 is defined as TqSn´1 .“ v | vJq “ 0 (
. Therefore, we have the projection onto TqSn´1 equal to PqK . For a function fpqq defined over Sn´1, we use grad f and Hess f to denote the Riemannian gradient and the Hessian of f , then we have
grad fpqq .“ PqK∇fpqq, Hess fpqq .“ PqK ` ∇2fpqq ´ xq,∇fpqqy I ˘ PqK ,
where ∇fpqq and ∇2fpqq are the normal first and second derivatives in Euclidean space. For example, for the function ϕTpqq defined in Equation (2.4), direct calculations give that
gradϕTpqq “ ´PqKA ` AJq ˘d3 “ ´PqK
m ÿ
k“1
` aJk q ˘3 ak,
HessϕTpqq “ ´PqK ” 3A diag ´ ` AJq ˘d2¯ AJ ´ › ›AJq › › 4 4 I ı PqK .
A.2 BASIC TOOLS
Lemma A.1 (Norm Inequality) If p ą r ą 0, then for x P Rn, we have }x}p ď }x}r ď n 1{r´1{p }x}p .
Lemma A.2 Let z, r P R. We have p1` zqr ď 1` p2r ´ 1qz, @ z P r0, 1s, r P Rzp0, 1q, p1` zqr ď 1` rz, @ z P r´1,`8q, r P r0, 1s,
where the second inequality reverse when r P Rzp0, 1q.
Lemma A.3 (Moments of the Gaussian Random Variable) If X „ N ` 0, σ2X ˘
, then it holds for all integer m ě 1 that
E r|X|ms ď σmX pm´ 1q!!, k “ tm{2u.
Lemma A.4 (Noncentral moments of the χ Random Variable) If Z „ χ pmq, then it holds for all integer p ě 1 that
E rZps “ 2p{2 Γ pp{2`m{2q Γ pm{2q ď p!! m p{2.
Lemma A.5 (Bernstein’s Inequality for R.V.s (Foucart & Rauhut, 2013b)) Let X1, . . . , Xp be i.i.d. real-valued random variables. Suppose that there exist some positive numbers R and σ2X such that
E r|Xk|ms ď m!
2 σ2XR m´2, for all integers m ě 2.
Let S .“ 1p řp k“1Xk, then for all t ą 0, it holds that
P r|S ´ E rSs| ě ts ď 2 exp ˆ ´ pt 2
2σ2X ` 2Rt
˙
.
Lemma A.6 (Bernstein’s Inequality for Random Vectors (Sun et al., 2015a)) Let x1, . . . ,xp P Rd be i.i.d. random vectors. Suppose there exist some positive number R and σ2X such that
E r}xk}ms ď m!
2 σ2XR m´2, for all integers m ě 2.
Let s “ 1p řp k“1 xk, then for any t ą 0, it holds that
P r}s´ E rss} ě ts ď 2pd` 1q exp ˆ ´ pt 2
2σ2X ` 2Rt
˙
.
Lemma A.7 (Bernstein’s Inequality for Bounded R.M.s, Theorem 1.6.2 of Tropp et al. (2015)) LetX1,X2, ¨ ¨ ¨ ,Xp P Rd1ˆd2 be i.i.d. random matrices. Suppose we have
}Xi} ď R almost surely, max › ›E “ XiX J i ‰ › › , › ›E “ XJi Xi ‰ › › ( ď σ2X , 1 ď i ď p.
Let S “ 1p řp i“1Xi, then we have
P p}S ´ E rSs} ě tq ď pd1 ` d2q exp ˆ ´ pt 2
2σ2X ` 4Rt{3
˙
.
Lemma A.8 (Bernstein’s Inequality for Bounded Random Vectors) Let x1,x2, ¨ ¨ ¨ ,xp P Rd be i.i.d. random vectors. Suppose we have
}xi} ď R almost surely, E ” }xi}2 ı ď σ2X , 1 ď i ď p.
Let s “ 1p řp i“1 xi, then we have
P p}s´ E rss} ě tq ď d exp ˆ ´ pt 2
2σ2X ` 4Rt{3
˙
.
Lemma A.9 (Lemma A.4 of (Zhang et al., 2018)) Let v P Rd with each entry following i.i.d. Berpθq distribution, then
P p|}v}0 ´ θd| ě tθdq ď 2 exp ˆ ´ 3t 2 2t` 6θd ˙ .
Lemma A.10 (Matrix Perturbation Bound, Lemma B.12 of (Qu et al., 2019)) Suppose B ą 0 is a positive definite matrix. For any symmetric perturbation matrix ∆ with }∆} ď 12σminpBq, it holds that
› ›
› pB `∆q´1{2 ´B´1{2
› › › ď 4 }∆} σ2min pBq ,
› ›
› pB `∆q1{2B´1{2 ´ I
› › › ď 4 }∆} σ 3{2 min pBq ,
where σminpBq denotes the minimum singular value ofB.
Lemma A.11 For any q, q1, q2 P Sn´1, we have ›
›PqK › › ď 1, }Pq1 ´ Pq2} ď 2 }q1 ´ q2} .
Proof The first is obvious, and for the second inequality we have ›
› › PqK1 ´ PqK2
› › › “ › ›q1q J 1 ´ q2qJ2 › › ď › ›q1q J 1 ´ q1qJ2 › › ` › ›q1q J 2 ´ q2qJ2 › › ď 2 }q1 ´ q2} ,
as desired.
Lemma A.12 For any nonzero vectors u and v, we have ›
› › ›
u }u} ´ v }v}
› › › › ď 2}v} }u´ v} .
Proof We have ›
› › ›
u }u} ´ v }v}
› › › › “ 1}u} }v} › › }v} u´ }u} v › ›
“ 1}u} }v} › › }v} u´ }v} v ` }v} v ´ }u} v › ›
ď 1}u} }v} p}v} }u´ v} ` }v} |}u} ´ }v}|q ď 2 }u} }u´ v} ,
as desired.
B ANALYSIS OF ASYMPTOTIC OPTIMIZATION LANDSCAPE
In this part of the appendix, we present the detailed analysis of the optimization landscape of the asymptotic objective
min q ϕTpqq “ ´
1
4
› ›AJq › ›
4 4 , s.t. q P Sn´1
over the sphere. We denote the overcompleteness of the dictionaryA P Rnˆm and the correlation of columns ofA with q by
K :“ m n , ζpqq :“ AJq “ rζ1 ¨ ¨ ¨ ζmsJ .
Without loss of generality, for a given q P Sn´1, we assume that
|ζ1| ě |ζ2| ě ¨ ¨ ¨ ě |ζm| .
Assumption. We assume that the dictionaryA is tight frame with `2-norm bounded columns 1
K AAJ “ I, }ai} ď M p1 ď i ď mq. (B.1)
We also assume that the columns ofA satisfy the µ-incoherence condition. Namely, we have
µpAq :“ max 1ďi “jďm
ˇ ˇ ˇ ˇ B ai }ai} , aj }aj} F ˇ ˇ ˇ ˇ P p0, 1q, (B.2)
such that µ is sufficiently small. Based on the function value of the objective ϕTpqq, we partition the sphere into two regions
RCpq; ξq “ ! q P Sn´1 | }ζ}44 ě ξµ 2{3 }ζ}23 ) , (B.3)
RNpq; ξq “ ! q P Sn´1 | }ζ}44 ď ξµ 2{3 }ζ}23 ) , (B.4)
where ξ ą 0 is some scalar. In the following, for appropriate choices of K, µ, and ξ, we first show thatRC does not have any spurious local minimizers by characterizing all the critical points within the region. Second, under more stringent condition thatA is `2 column normalized, for the region RN we show that there exhibits large negative curvature throughout the region, such that there is no local/global minimizer withinRN.
B.1 GEOMETRIC ANALYSIS OF CRITICAL POINTS IN RC
In this subsection, we show that all the critical points of ϕTpqq inRC are either ridable saddle points, or satisfy second-order optimality condition and are close to the target solutions.
Proposition B.1 Suppose we have
KM ă 4´1 ¨ ξ3{2, M3 ă η ¨ ξ3{2, µ ă 1 20
(B.5)
for some constant η ă 2´6. Then any critical point q P RC, with gradϕTpqq “ 0, either is a ridable (strict) saddle point, or it satisfies second-order optimality condition and is near one of the components e.g., a1 in the sense that
B
a1 }a1} , q
F
ě 1´ 5ξ´3{2M3 ě 1´ 5η.
First, in Appendix B.1.1 we characterize some basic properties of critical points of ϕTpqq. Based on this, we prove Proposition B.1 in Appendix B.1.2.
B.1.1 BASIC PROPERTIES OF CRITICAL POINTS
Lemma B.2 (Properties of critical points) For any point q P Sn´1, if q is a critical point of ϕTpqq over the sphere, then it satisfies
fpζiq “ ζ3i ´ αiζi ` βi “ 0 (B.6)
for all i P rms with ζpqq “ AJq, where
αi :“ }ζ}44 }ai}2 , βi :“ ř j‰i xai,ajy ζ3j }ai}2 . (B.7)
Proof For any point q P Sn´1, if q is a critical point of ϕTpqq over the sphere, then its Riemannian gradient satisfies
gradϕTpqq “ PqKAζd3 “ 0 ùñ Aζd3 ´ }ζ} 4 4 q “ 0.
Multiple aJi (1 ď i ď m) on both sides of the equality, we obtain
}ai}2 ζ3i ´ }ζ} 4 4 ζi `
ÿ j‰i xai,ajy ζ3j “ 0.
By replacing αi and βi defined in Equation (B.7) into the equation above, we obtain the necessary condition in Equation (B.6) as desired.
Since the roots of fpzq correspond to the critical points of ϕTpqq, we characterize the properties of the roots as follows.
Lemma B.3 Consider the following cubic polynomial fpzq “ z3 ´ αz ` β (B.8)
with
0 ă |β| ď 1 4 α3{2, α ą 0. (B.9)
Then the roots of the function fpzq is contained in one of the nonoverlapping intervals:
I1 :“ " z P R ˇ ˇ ˇ
ˇ |z| ď 2 |β| α
* , I2 :“ " z P R ˇ ˇ ˇ
ˇ
ˇ ˇz ´ ? α ˇ ˇ ď 2 |β| α * ,
I3 :“ " z P R ˇ ˇ ˇ
ˇ
ˇ ˇz ` ? α ˇ ˇ ď 2 |β| α * .
Proof By our construction |β| ď 14α 3{2 and α ą 0 in Equation (B.9), it is obvious that the intervals I1, I2, and I3 are nonoverlapping. Without loss of generality, let us assume that β is positive. We have
fp ? αq “ fp´ ? αq “ fp0q “ β ą 0. (B.10)
Thus, as illustrated in Figure 5, if we can show that
f
ˆ
2β
α
˙ ă 0, f ˆ ´ ? α´ 2β
α
˙ ă 0, f ˆ? α´ 2β α ˙ ă 0, (B.11)
then this together with Equation (B.10) suffices to show that there exists at least one root in each of the three intervals I1, I2, and I3. Next, we show Equation (B.11) by direct calculations. First, notice that
f
ˆ
2β
α
˙ “ ˆ 2β
α
˙3
´ β “ β α3 ` 8β2 ´ α3 ˘ “ β α3
ˆ
1 2 α3 ´ α3
˙
ď ´1 2 β ă 0,
Second, we have
f
ˆ ´ ? α´ 2β
α
˙ “ ˆ ´ ? α´ 2β
α
˙3
´ α ˆ ´ ? α´ 2β
α
˙
` β
“ ´8β 3 α3 ´ α3{2 ´ 6β ´ 12β 2 α3{2 ` α3{2 ` 3β “ ´8β 3 α3 ´ 12β 2 α3{2 ´ 3β ă 0.
Similarly, we have
f ˆ? α´ 2β
α
˙
“ ´8β 3 α3 ` 12β 2 α3{2 ´ 3β “ β
ˆ
´8β 2 α3 ` 12β α3{2 ´ 3 ˙ ă ´8β 3 α3 ă 0.
This proves Equation (B.11). Similar argument also holds for β ă 0. Thus, we obtain the desired results.
B.1.2 GEOMETRIC CHARACTERIZATIONS OF CRITICAL POINTS IN RC
Based on the results in Appendix B.1.1, we prove Proposition B.1, showing that there is no spurious local minimizers inRC. Proof [Proof of Proposition B.1] First recall from Lemma B.2, we defined
αi “ }ζ}44 }ai}2 ą 0, βi “ ř j‰i xai,ajy ζ3j }ai}2 .
Then for any q P RC, we have
|βi| α 3{2 i “
ˇ ˇ ˇ ř j‰i xai,ajy ζ3j ˇ ˇ ˇ }ai}
}ζ}64 ď µM3 }ζ}33 }ζ}64 ď M3ξ´3{2, (B.12)
where for the first inequality we used the fact that for any i P rms, }ai} ďM and ˇ
ˇ ˇ ˇ ˇ ÿ
j‰i xai,ajy ζ3j
ˇ ˇ ˇ ˇ ˇ ď ˇ ˇ ˇ ˇ ˇ ÿ
j‰i
B
ai }ai} , aj }aj}
F
ζ3j
ˇ ˇ ˇ ˇ ˇ }ai} max 1ďjďm }aj} ď µM2 m ÿ
i“1 |ζi|3 “ µM2 }ζ}33 ,
and the last inequality derives from the fact that q P RC. Thus, by Equation (B.5) and Equation (B.12), we obtain
M3ξ´3{2 ď 1 4 ùñ |βi| α 3{2 i ď 1 4 .
This implies that the condition in Equation (B.9) holds, so that we can apply Lemma B.3 to characterize the critical points. Based on Lemma B.3, we classify critical points q P RC into three categories
1. All |ζi| (1 ď i ď m) are smaller than 2|βi|αi ; 2. Only |ζ1| is larger than 2|β1|α1 ; 3. At least |ζ1| and |ζ2| are larger than 2|β1|α1 and 2|β2| α2 , respectively.
For Case 1, Lemma B.4 shows that this type of critical point does not exist under the assumption in Equation (B.5). For Case 2, under the same assumption, Lemma B.5 implies that such a critical point q P RC satisfies the second-order optimality condition, and it is near one of the target solution with
B
a1 }a1} , q
F
ě 1´ 5ξ´3{2M3 ě 1´ 5η.
for some η ă 2´6. Finally, for Case 3, Lemma B.6 proves that this type of critical points q P RC is ridable saddle, for which the Riemannian Hessian exhibits negative eigenvalue. Therefore, the critical points in RC are either ridable saddle or near target solutions, so that there is no spurious local minimizer inRC.
In the following, we provided more detailed analysis for each case.
CASE 1: NO CRITICAL POINTS WITH SMALL ENTRIES.
First, we show by contradiction that if q P RC and is a critical points, then there is at least one coordinate, e.g., |ζ1| ě 2|β1|α1 . This implies that Case 1 (i.e., all |ζi| (1 ď i ď m) are smaller than 2|βi| αi
) is impossible to happen. In other words, this means that any critical point q P RC should be close to superpositions of columns ofA.
Lemma B.4 Suppose we have M4{3K1{3 ă 4´1{3ξ.
If q P RC is a critical point, then there exists at least one i P rms such that the entry ζi of ζpqq satisfies
|ζi| ě 2 |βi| αi .
Proof Suppose there exists a q P RC such that all entries ζi satisfying |ζi| ă 2|βi|αi . Then we have
max 1ďiďm
|ζi| “ }ζ}8 ď 2 ˇ ˇ řm k“2 xa1,aky ζ3k ˇ ˇ
}ζ}44 ď 2M2µ }ζ}33 }ζ}44 .
This implies that
}ζ}44 ď }ζ} 2 8 }ζ}
2 ď 4M4µ2 }ζ}63
}ζ}84 }ζ}2 ùñ }ζ}124 ď 4M 4µ2 }ζ}63 }ζ} 2
ùñ }ζ}44 ď 4 1{3M4{3K1{3µ2{3 }ζ}23 ,
where we used the fact that }ζ}2 “ K according to Equation (B.1). Thus, by our assumption, we have
M4{3K1{3 ă ξ{41{3 ùñ }ζ}44 ă ξµ 2{3 }ζ}23 .
This contradicts with the fact that q P RC.
CASE 2: CRITICAL POINTS NEAR GLOBAL MINIMIZERS
Second, we consider the case that there exists only one big ζ1, for which the critical point satisfies second-order optimality and is near a true component.
Lemma B.5 Suppose ξ is sufficiently large such that
M3 ă η ¨ ξ3{2, KM ă 4´1 ¨ ξ3{2, (B.13)
for some constant η ă 2´6. For any critical point q P RC, if there is only one entry in ζ such that ζ1 ě 2|β1|α1 ,
B
a1 }a1} , q
F
ě 1´ 5ξ´3{2M3 ě 1´ 5η.
Moreover, such a critical point q P RC satisfies the second-order optimality condition: for any v P Sn´1 with v K q,
vJHessϕTpqqv ě 1
20 }ζ}44 .
Proof We first show that under our assumptions the critical point q P RC is near a target solution. Following this, we prove that q also satisfies second-order optimality condition.
Closeness to target solutions. First, if q is a critical point such that there is only one ζ1 ě 2|β1|α1 , we show that such q is very close to a true component. By Lemma B.2 and Lemma B.3, we know that ζ1 needs to be upper bounded by
ζ21 ď ˆ ? α1 ` 2 |β1| α1
˙2 “ ˜ }ζ}24 }a1} ` 2 ˇ ˇ řm k“2 xa1,aky ζ3k ˇ ˇ
}ζ}44
¸2
ď }ζ}44 }a1}2
˜
1` 2µ }ζ}33 }a1} 2 max1ďjďm }aj} }ζ}64
¸2
.
By using the fact that q P RC and }aj} ďM p1 ď j ď mq, we have
}a1}2 ζ21 ď ˜ 1` 2µ }ζ}33 }a1} 2 max1ďjďm }aj} }ζ}64
¸2
}ζ}44 ď ´ 1` 2ξ´3{2M3 ¯2 }ζ}44 . (B.14)
On the other hand, by using the fact that |ζk| ď 2|βk|αk for all k ě 2, we have
ζ41 ě }ζ} 4 4 ´ ζ 2 2
m ÿ k“2 ζ2k ě }ζ} 4 4 ´ 4 |β2|2 α22 K ě }ζ}44
˜
1´ 4µ2 }ζ}63 }ζ}124 KM4
¸
ě }ζ}44 ` 1´ 4ξ´3KM4 ˘ . (B.15)
Combining the lower and upper bounds in Equation (B.14) and Equation (B.15), we obtain B
a1 }a1} , q
F2
“ ζ 2 1 }a1}2 ě 1´ 4ξ ´3KM4 ` 1` 2ξ´3{2M3 ˘2 ě
` 1´ 4ξ´3KM4 ˘
1` 6ξ´3{2M3
“ 1´ 2ξ´3M3 ´ 3ξ3{2 ` 2KM ¯
ě 1´ 8ξ´3{2M3 ě 1´ 8η,
where the second inequality follows by Lemma A.2, and the last inequality follows from Equation (B.13). This further gives B
a1 }a1} , q
F ě 1´ 8ξ ´3{2M3
` 1´ 8ξ´3{2M3 ˘1{2 ě
1´ 8ξ´3{2M3 1´ 4ξ´3{2M3 “ 1´ 5ξ´3{2M3 ě 1´ 5η. (B.16)
Second-order optimality condition. Second, we check the second order optimality condition for the critical point. Let v P Sn´1 be any vector such that v K q, then
vJHessϕTpqqv “ ´3vJAdiag ` ζd2 ˘ AJv ` }ζ}44
“ ´3 xa1,vy2 ζ21 ´ 3 m ÿ
k“2 xak,vy2 ζ2k ` }ζ} 4 4
ě ´3 xa1,vy2 ζ21 ´ 3ζ22 › ›AJv › › 2 ` }ζ}44 “ ´3 xa1,vy2 ζ21 ´ 3Kζ22 ` }ζ} 4 4 (B.17)
Next, we control xa1,vy2 ζ21 and Kζ22 in terms of }ζ} 4 4, respectively. By Equation (B.14) and xq,vy “ 0,
xa1,vy2 ¨ ζ21 “ B a1 }a1} ´ q,v F2 ´ }a1}2 ζ21 ¯
ď › › ›
› a1 }a1}
´ q › › ›
›
2 ´
1` 2ξ´3{2M3 ¯2
}ζ}44
“ 2 ˆ 1´ B a1 }a1} , q F˙ ´ 1` 2ξ´3{2M3 ¯2 }ζ}44
ď 10ξ´3{2M3 ´ 1` 2ξ´3{2M3 ¯2 }ζ}44 ď 1
4 }ζ}44 . (B.18)
On the other hand, for q P RC, using Equation (B.13) we have
Kζ22 ď K 4 |β2|2
α22 ď 4KM4 µ2 }ζ}63 }ζ}124 ¨ }ζ}44 ď 4KM 4ξ´3 }ζ}44 ď 1 15 }ζ}44 . (B.19)
Thus, combining the results in Equation (B.17), Equation (B.18), and Equation (B.19), we obtain
vJHessϕTpqqv ě ˆ 1´ 3 4 ´ 1 5 ˙ }ζ}44 ě 1 20 }ζ}44 .
This completes our proof.
CASE 3: CRITICAL POINTS ARE RIDABLE SADDLES.
Finally, we consider the critical points q P RC that at least two entries |ζ1| and |ζ2| are larger than 2|β1| α1
and 2|β2|α2 , respectively. For this type of critical points in RC, we show that they are ridable saddle points: the Hessian is nondegenerate and exhibits negative eigenvalues.
Lemma B.6 Suppose we have
M3 ă η ¨ ξ3{2, µ ă 1 20 , (B.20)
for some constant η ă 2´6 For any critical point q P RC, if there are at least two entries in ζpqq such that |ζi| ą 2|βi|αi pi P rmsq, then q is a strict saddle point: there exists some v P S
n´1 with v K q, such that
vJHessϕTpqqv ď ´}ζ}44 .
Proof Without loss of generality, for any critical point q P RC, we assume that ζ1 “ aJ1 q and ζ2 “ aJ2 q are the two largest entries in ζpqq. We pick a vector v P span ! a1 }a1} , a2 }a2} ) such that v K q with v P Sn´1. Thus,
vJHessϕTpqqv “ ´3vJAdiag ` ζd2 ˘ AJv ` }ζ}44
ď ´3 }a1}2 ζ21 B a1 }a1} ,v
F2
´ 3 }a2}2 ζ22 B a2 }a2} ,v
F2
` }ζ}44 .
Since |ζ1| ě 2|β1|α1 and |ζ2| ě 2|β2| α2 , by Lemma B.2, Lemma B.3, and the fact that q P RC, we have
}a1}2 ζ21 ě }a1} 2
ˆ
? α1 ´ 2 |β1| α1
˙2 ě ˜ 1´ 2µM2 }ζ}33 }a1}
}ζ}64
¸2
}ζ}44
ě ´ 1´ 2ξ´3{2M3 ¯2
}ζ}44 .
In the same vein, we can also show that
}a2}2 ζ22 ě ´ 1´ 2ξ´3{2M3 ¯2 }ζ}44 .
Therefore, combining the results above, we obtain
vJHessϕTpqqv ď }ζ}44
«
1´ 3 ´ 1´ 2ξ´3{2M3 ¯2
˜
B
a1 }a1} ,v
F2
` B a2 }a2} ,v
F2 ¸ff
.
As v P span !
a1 }a1} , a2 }a2}
)
, we can write
v “ c1 a1 }a1} ` c2 a2 }a2}
for some coefficients c1, c2 P R. As v P Sn´1, we observe
}v}2 “ c21 ` c22 ` 2c1c2 B a1 }a1} , a2 }a2} F “ 1 ùñ c21 ` c22 ě 1´ 2 |c1c2|µ ě 1´ 4µ,
where the last inequality follows from Lemma B.7. Thus, we observe B
a1 }a1} ,v
F2
` B a2 }a2} ,v
F2
“ ˆ c1 ` c2 B a1 }a1} , a2 }a2}
F˙2
` ˆ c2 ` c1 B a1 }a1} , a2 }a2}
F˙2
“ ` c21 ` c22 ˘ ` ` c21 ` c22 ˘
B
a1 }a1} , a2 }a2}
F2
` 4c1c2 B a1 }a1} , a2 }a2} F
ě 1´ 4µ´ p1´ 4µqµ2 ´ 4 1` µ 1´ µ2µ
ě 1´ 10µ
By the fact in Equation (B.20) and combining all the bounds above we obtain
vJHessϕTpqqv ď „ 1´ 3 ´ 1´ 2ξ´3{2M3 ¯2 p1´ 10µq  }ζ}44 ď ´ 1
4 }ζ}44 .
This completes the proof.
Lemma B.7 Suppose ˇ ˇ
ˇ
A
a1 }a1} , a1 }a1}
E ˇ
ˇ ˇ ď µ with µ ă 1{2. Let v P span
!
a1 }a1} , a2 }a2}
)
such that
}v} “ 1 and v “ c1 a1}a1} ` c2 a2 }a2} , then we have
|c1c2| ď 1` µ 1´ µ2 ,
Proof By the fact that ˇ ˇ
ˇ
A
v, a1}a1}
EA
v, a2}a2}
E ˇ
ˇ ˇ ď 1, we have
ˇ ˇ ˇ ˇ ˆ c1 ` c2 B a1 }a1} , a2 }a2} F˙ˆ c2 ` c1 B a1 }a1} , a2 }a2} F˙ ˇ ˇ ˇ ˇ ď 1,
which further implies that ˇ
ˇ ˇ ˇ ˇ c1c2 ` ` c21 ` c22 ˘
B
a1 }a1} , a2 }a2}
F ` c1c2 B a1 }a1} , a2 }a2}
F2 ˇ
ˇ ˇ ˇ ˇ ď 1.
Since }v} “ 1, we also have
c21 ` c22 “ 1´ 2c1c2 B a1 }a1} , a2 }a2} F .
Combining the two (in)equalities above, we obtain
1 ě
ˇ ˇ ˇ ˇ ˇ c1c2 ` B a1 }a1} , a2 }a2} F ´ c1c2 B a1 }a1} , a2 }a2} F2 ˇ ˇ ˇ ˇ ˇ
ě |c1c2| ˜ 1´ B
a1 }a1} , a2 }a2}
F2 ¸
´ ˇ ˇ ˇ
ˇ
B
a1 }a1} , a2 }a2}
F ˇ
ˇ ˇ ˇ ě |c1c2| ` 1´ µ2 ˘ ´ µ.
Thus, we obtain the desired result.
B.2 NEGATIVE CURVATURE IN RN
Finally, we make more stringent assumption onA that each column ofA is `2 normalized, i.e.,
}ai} “ 1, 1 ď i ď m.
We show that the function ϕTpqq exhibits negative curvature in the regionRN. Namely, the Riemannian Hessian for any points q P RN has a negative eigenvalue, such that the Hessian is negative in a certain direction.
Lemma B.8 Suppose each column ofA is `2 normalized and
K ď 3 ´ 1` 6µ` 6ξ3{5µ2{5 ¯´1 .
For any point q P RN, there exists some direction d P Sn´1, such that
dJHessϕTpqqd ă ´4 }ζ}44 }ζ} 2 8 .
Proof By definition, we have
aJ1 HessϕTpqqa1 “ ´ 3aJ1 PqKAdiag ` ζd2 ˘ A˚PqKa1 ` }ζ} 4 4 › ›PqKa1 › › 2
“ ´ 3aJ1Adiag ` ζd2 ˘ AJa1 ` 6 }ζ}8 ζ J diag ` ζd2 ˘ AJa1 ´ 3 }ζ}28 }ζ} 4 4 ` }ζ} 4 4
´ }a1}2 ´ }ζ}28 ¯
ď ´ 3 }ζ}28 }a1} 4 ` 6 }ζ}48 }a1} 2 ` 6µ }ζ}8 }ζ} 3 3 ´ 3 }ζ} 2 8 }ζ} 4 4 ` }a1} 2 }ζ}44 ´ }ζ} 2 8 }ζ} 4 4 “ ´ 3 }ζ}28 ` 6 }ζ} 4 8 ` 6µ }ζ}8 }ζ} 3 3 ´ 4 }ζ} 2 8 }ζ} 4 4 ` }ζ} 4 4 ď }ζ}28 ´ ´3` 6 }ζ}28 ` 6µ }ζ} 2 ´ 4 }ζ}44 ` }ζ} 2 ¯
“ }ζ}28 ´ ´3` 6 }ζ}28 ` 6µK ´ 4 }ζ} 4 4 `K ¯
where for the second inequality we used the fact that }ζ}44 ď }ζ} 2 8 }ζ} 2, and for the last equality we applied that }ζ}2 “ qJAAJq “ K. Moreover, as q P RN, we have
}ζ}28 ď }ζ} 2 4 ď ξ 1{2µ1{3 }ζ}3
}ζ}3 “ ˜ m ÿ
k“1 |ζk|3
¸1{3
ď }ζ}1{38 K 1{3.
Thus, we obtain
}ζ}28 ď ξ 1{2µ1{3 }ζ}1{38 K 1{3 ùñ }ζ}28 ď ξ 3{5 pµKq2{5 .
Hence, we have
aJ1 HessϕTpqqa1 ď }ζ} 2 8
´ ´3` 6ξ3{5 pµKq2{5 ` 6µK ´ 4 }ζ}44 `K ¯ ď ´4 }ζ}44 }ζ} 2 8 ,
whenever
K ď 3 ´ 1` 6µ` 6ξ3{5µ2{5 ¯´1 .
Thus, we obtain the desired result.
C OPTIMIZATION LANDSCAPE IN FINITE SAMPLE
In this section, we will show that the finite sample objective functions in the overcomplete dictionary learning and convolutional dictionary learning have similar geometric properties as ϕTpqq “ ´14 › ›AJq › › 4 4 analyzed in Appendix B. Specifically, we will analyze the geometric properties of objective function ϕpqq (which could be ϕDLpqq and ϕCDLpqq) whose gradient and Hessian are close to ϕTpqq. We denote by
δgpqq :“ gradϕpqq ´ gradϕTpqq, ∆Hpqq :“Hessϕpqq ´HessϕTpqq,
(C.1)
both of which will be proved to be small for overcomplete dictionary learning and convolutional dictionary learning in Appendix F.
C.1 GEOMETRIC ANALYSIS OF CRITICAL POINTS IN RC
Proposition C.1 Assume
}δgpqq} ď µM }ζ}33 and }∆Hpqq} ă 1
20 }ζ}44 .
Also suppose we have
KM ă 8´1 ¨ ξ3{2, M3 ă 2η ¨ ξ3{2, µ ă 1 20
(C.2)
for some constant η ă 2´6. Then any critical point q P RC, with gradϕpqq “ 0, either is a ridable (strict) saddle point, or it satisfies second-order optimality condition and is near one of the components e.g., a1 in the sense that
B
a1 }a1} , q
F
ě 1´ 5ξ´3{2M3 ě 1´ 5η. (C.3)
Proof [Proof of Proposition C.1] With the same argument in Lemma B.2, we have that any critical point q P Sn´1 satisfies fpζiq “ ζ3i ´ αiζi ` β1i “ 0, for all i P rms with ζ “ AJq, where
αi “ }ζ}44 }ai}2 , β1i “ xδgpqq,aiy `
ř
j‰i xai,ajy ζ3j }ai}2 “ βi ` xδgpqq,aiy }ai}2 , (C.4)
with βi “ ř j‰ixai,ajyζ 3 j
}ai}2 which is defined in equation B.7.
Recall that a widely used upper bound for βi in Appendix B.1 is:
|βi| “
ˇ ˇ ˇ ř j‰i xai,ajy ζ3j ˇ ˇ ˇ
}ai}2 ď µM }ζ}33 }ai} ,
which together with }δgpqq} ď µM }ζ}33 gives
β1 “ βi ` xδgpqq,aiy }ai}2 ď 2 µM }ζ}33 }ai} . (C.5)
To easily utilize the proofs in Appendix B.1, we define ξ1 “ 2´2{3ξ such that ξ1´3{2 “ 2ξ´3{2. Plugging the assumption M3ξ1´3{2 ď 14 into equation C.5, we have
|β1i| α 3{2 i ď 2 µM }ζ}33 }ai} 2 }ζ}64 ď 2 µM3 }ζ}33 }ζ}64 ď 2M3ξ´3{2 ď 2M3ξ1´3{2 ď 1 4 .
This implies that the condition in equation B.9 holds, so that we can apply Lemma B.3 based on which we classify critical points q P RC into three categories
1. All |ζi| (1 ď i ď m) are smaller than 2|β1i| αi ; 2. Only |ζ1| is larger than 2|β11| α1 ; 3. At least |ζ1| and |ζ2| are larger than 2|β11| α1 and 2|β12| α2 , respectively.
For Case 1, using the same argument as in Lemma B.4 we can easily show that this type of critical point does not exist. For Case 2, with the same argument as in Lemma B.5, we obtain that such a critical point is near one of the target solution with
B
a1 }a1} , q
F
ě 1´ 5ξ1´3{2M3 ě 1´ 5η,
and satisfies the second-order optimality condition, i.e., for any v P Sn´1 with v K q, we have
vJHessϕpqqv ě vJHessϕTpqqv ´ }∆Hpqq} ě 1
20 }ζ}44 ´ }∆Hpqq} .
Finally, for Case 3, with the same v constructed in Lemma B.6 and using the assumption }∆Hpqq} ă 1 20 }ζ} 4 4, we have vJHessϕpqqv ď vJHessϕTpqqv ` }∆Hpqq} ď ´ }ζ}44 ` }∆Hpqq} ă 0, indicating that this type of critical points q P RC is ridable saddle, for which the Riemannian Hessian exhibits negative eigenvalue. Therefore, the critical points in RC are either ridable saddle or near target solutions, so that there is no spurious local minimizer inRC.
C.2 NEGATIVE CURVATURE IN RN
By directly using Lemma B.8, we obtain the negative curvature of ϕpqq inRN.
Lemma C.2 Assume }∆Hpqq} ă }ζ}44 }ζ} 2 8 .
Also suppose each column ofA is `2 normalized and
K ď 3 ´ 1` 6µ` 6ξ3{5µ2{5 ¯´1 .
For any point q P RN, there exists some direction d P Sn´1, such that dJHessϕpqqd ă ´3 }ζ}44 }ζ} 2 8 .
Proof First, it follows Lemma B.8 that for any point q P RN, there exists some direction d P Sn´1, such that
dJHessϕTpqqd ă ´4 }ζ}44 }ζ} 2 8 ,
which together with the assumption }∆Hpqq} ă }ζ}44 }ζ} 2 8 and the fact d JHessϕpqqd “ dJHessϕTpqqd` dJ∆Hpqqd ď dJHessϕTpqqd` }∆Hpqq} completes the proof.
D OVERCOMPLETE DICTIONARY LEARNING
In this section, we consider the nonconvex problem of
min q
ϕDLpqq “ ´ 1
12θp1´ θqp › ›qJY › ›
4 4 “ ´ 1 12θp1´ θqp › ›qJAX › › 4 4 , s.t. }q} “ 1.
We characterize its expectation and optimization landscape as follows.
D.1 EXPECTATION CASE: OVERCOMPLETE TENSOR DECOMPOSITION
First, we show that ϕDLpqq reduces to ϕTpqq in expectation w.r.t. X .
Lemma D.1 When X is i.i.d. drawn from Bernoulli Gaussian distribution as in Assumption 2.2, then we have
EX rϕDLpqqs “ ϕTpqq ´ θ
2p1´ θq
´m
n
¯2
.
Proof Let ζ “ AJq P Rm with }ζ}2 “ mn . By using the fact that X “ rx1 x2 ¨ ¨ ¨ xps , xk “ bk d gk, bk „ Berpθq, gk „ N p0, Iq,
we observe
EX rϕDLpqqs “ ´ 1 12p1´ θqθpEX ” › ›ζJX › › 4 4 ı “ ´ 1 12p1´ θqθp
p ÿ k“1 Exk ” ` ζJxk ˘4 ı
“ ´ 1 12p1´ θqθEb,g ” xζ d b, gy4 ı
“ ´ 1 4p1´ θqθEb ” }ζ d b}4 ı .
Write }z d b}2 “ řm k“1 pzkbkq 2, we obtain
EX rϕDLpqqs “ ´ 1
4p1´ θqθEb
»
–
˜
m ÿ k“1 pzkbkq2
¸2 fi
fl “ ´ 1 4p1´ θq
m ÿ k“1 z4k ´
θ
2p1´ θq ÿ i “j ζ2i z 2 j
“ ´1 4 }z}44 ´
θ
2p1´ θq }z} 4
“ ϕTpqq ´ θ
2p1´ θq
´m
n
¯2
,
as desired.
D.2 MAIN GEOMETRIC RESULT
Combining Proposition C.1 and Lemma C.2 together with the concentration results of the gradient and Hessian in Proposition F.3 and Proposition F.6, we obtain the following geometry results of overcomplete dictionary learning.
Theorem D.2 SupposeA satisfies Equation (2.2) andX P Rmˆp follows BGpθq with θ P ` 1 m , 1 2 ˘
. Also suppose we have
K ă max " 8´1 ¨ ξ3{2, 3 ´ 1` 6µ` 6ξ3{5µ2{5 ¯´1 * , 1 ă 2η ¨ ξ3{2, µ ă 1 20
for some constant η ă 2´6.
• If p ě CθK3n3 max ! logpθn7{2{µq µ2 ,Kn 2 logpθn2q )
, then with probability at least 1´ cp´2, any critical point q P RC of ϕDLpqq either is a ridable (strict) saddle point, or it satisfies second-order optimality condition and is near one of the components e.g., a1 in the sense that
B
a1 }a1} , q
F
ě 1´ 5ξ´3{2M3 ě 1´ 5η.
• If p ě CθK4n6 logpθn5q, then with probability at least 1 ´ cp´2, any critical point q P RN of ϕDLpqq is a ridable (strict) saddle point.
Here, c, C ą 0 are some numerical constants.
Proof First note that for overcomplete dictionaryA in Equation (2.2), it satisfies Equation (F.9) with M “ 1. Now it follows from Proposition F.3 and Proposition F.6 that when
p ě CθK5n2 max # logpθKn{µ }ζ}33q µ2 }ζ}63 , Kn logpθKn{ }ζ}44q }ζ}84 + , (D.1)
then with probability at least 1´ cp´2, sup
qPSn´1 }gradϕDLpqq ´ gradϕTpqq} ď µM }ζ}33 ,
sup qPSn´1
}HessϕDLpqq ´HessϕTpqq} ă 1
20 }ζ}44 ,
which together with Proposition C.1 implies that any critical point q P RC of ϕDLpqq either is a ridable (strict) saddle point, or it satisfies second-order optimality condition and is near one of the components e.g., a1 in the sense that
B
a1 }a1} , q
F
ě 1´ 5η.
We complete the proof for q P RC by plugging inequalities }ζ}3 ě m´1{6 }ζ}2 “ K1{3n´1{6 and }ζ}4 ě m´1{4 }ζ}2 “ K1{4n´1{4 into Equation (D.1). Similarly, by Proposition F.6, when
p ě CθK6n3 logpθKn{ }ζ}44 }ζ} 2 8q
}ζ}84 }ζ} 4 8
, (D.2)
then with probability at least 1´ cp´2, sup
qPSn´1 }HessϕDLpqq ´HessϕTpqq} ă max }ζ}28 }ζ} 4 4 ,
which together with Lemma C.2 implies that any critical point q P RN of ϕDLpqq either is a ridable (strict) saddle point. The proof is completed by plugging }ζ}8 ě n´1{2into Equation (D.2).
E CONVOLUTIONAL DICTIONARY LEARNING
In this part of appendix, we provide the detailed analysis for CDL. Recall from Section 3, we denote Y “ “ Cy1 Cy2 ¨ ¨ ¨ Cyp ‰ P Rnˆp, A0 “ rCa1 Ca2 ¨ ¨ ¨ CaK s P Rnˆm,
xi “
»
— — – xi1 xi2 ... xiK
fi
ffi ffi fl P Rm, Xi “
»
— — – Cxi1 Cxi2 ...
CxiK
fi
ffi ffi fl P Rmˆn, X “ rX1 X2 ¨ ¨ ¨ Xps P Rnˆnp,
For simplicity we let
A “ ` K´1A0A J 0 ˘´1{2 A0, m “ nK.
Recall from Section 3, for CDL we make the following assumptions onA0,A andX .
Assumption E.1 (Properties ofA0 andA) We assume the matrixA0 has full row rank with
minimum singular value: σminpA0q ą 0, condition number: κpA0q :“ σmaxpA0q σminpA0q .
In addition, we assume the columns ofA are mutually incoherent in the sense that
max i“j
ˇ ˇ ˇ ˇ B ai }ai} , aj }aj} F ˇ ˇ ˇ ˇ ď µ.
Assumption E.2 (Bernoulli-Gaussian xik) We assume entries of xik „i.i.d. BGpθq that xik “ bik d gik, bik „i.i.d. Berpθq, gik „i.i.d. N p0, Iq, 1 ď i ď p, 1 ď k ď K.
In comparison with Assumption 2.1, it should be noted that the preconditioning does not necessarily result in `2-normalized columns ofA. But their norms are still bounded in the sense that
}ak}2 ď › ›AJak › › ď ? K }ak} ùñ }ak} ď ? K, 1 ď k ď nK. (E.1)
Because of the unbalanced columns ofA, unlike the ODL problem, the CDL problem
min qPSn´1
ϕCDLpqq “ ´ 1
12θp1´ θqnp › ›qJPY › ›
4 4 “ ´ 1 12θp1´ θqp › ›qJPA0X › › 4 4
does not have global geometric structures in the worst case. But still we can show that the problem is benign in local regions in the following. Moreover, we also show that we can cook up data driven initialization which falls into the local region.
E.1 MAIN RESULT OF OPTIMIZATION LANDSCAPE
In this part, we show our main result for optimization landscape for CDL. Namely, consider the region introduced in Equation (3.3) as
RCDL :“ ! q P Sn´1 ˇ ˇ ϕTpqq ď ´ξCDL κ4{3µ2{3 }ζpqq}23 ) ,
where ξCDL ą 0 is a fixed numerical constant. We show the following result.
Theorem E.3 (Local geometry of nonconvex landscape for CDL) Let C0 ą 5 be some constant and η ă 2´6. Suppose we have
θ P ˆ 1
nK ,
1
3
˙
, ξCDL “ C0 ¨ η´2{3K, µ ă 1
40 , K ă C0,
and we assume Assumption E.1 and Assumption E.2 hold. There exists some constant C ą 0, with probability at least 1´ c1pnKq´c2 over the randomness of xiks, whenever
p ě CθK2µ´2n4 max " K6κ6pA0q σ2minpA0q , n * log6pm{µq,
every critical point qc of ϕCDLpqq in RCDL is either a strict saddle point that exhibits negative curvature for descent, or it is near one of the target solutions (e.g. a1) such that
B
a1 }a1} , qc
F
ě 1´ 5κ´2η.
Proof Noting Equation (E.1), we set M “ ? K in Proposition C.1. It follows from Proposition E.11 that when
p ě CθK4n2 log5pmKqmax " K6κ6pA0q σ2minpA0q , n * ¨max # logpθKn{µK1{2 }ζ}33q µ2K }ζ}63 , logpθKn{ }ζ}44q }ζ}84 + ,
(E.2)
then with probability at least 1´ c1pnKq´c2 ,
sup qPSn´1
}gradϕCDLpqq ´ gradϕTpqq} ď µ ? K }ζ}33 ,
sup qPSn´1
}HessϕCDLpqq ´HessϕTpqq} ă 1
20 }ζ}44 .
Thus, by using Proposition C.1, we have that any critical point qc P RCDL of ϕCDLpqq either is a ridable (strict) saddle point, or it satisfies second-order optimality condition and is near one of the components, e.g., a1 in the sense that
B
a1 }a1} , qc
F
ě 1´ 5ξ´3{2CDLK 3{2κ´2 ě 1´ 5ηκ´2,
where we have plugged M “ ? K and ξ “ ξCDLκ4{3 in Equation (C.3). Finally, we complete the proof by using inequalities }ζ}3 ě m´1{6 }ζ}2 “ K1{3n´1{6 and }ζ}4 ě m´1{4 }ζ}2 “ K1{4n´1{4 in Equation (E.2).
E.2 PROOF OF OPTIMIZATION
In the following, we show that with high probability Algorithm 1 with initialization returns an approximate solution of one of the kernels up to a shift.
Proposition E.4 (Global convergence of Algorithm 1) With m “ nK, suppose
c1 logm
m ď θ ď c2
µ´2{3
κ4{3m logm ¨min
" κ4{3 µ4{3 , Kµ´4 m2 logm * . (E.3)
Whenever
p ě CθK2µ´2 max " K6κ6pA0q σ2minpA0q , n * n4 log6 pm{µq ,
our initialization in Algorithm 1 satisfies
qinit P RCDL :“ ! q P Sn´1 | ϕTpqq ď ´ξCDL µ2{3κ4{3K ) Ă RCDL, (E.4)
such that all future iterates of Algorithm 1 stays within RCDL and converge to an approximate solution (e.g., a circulant shift s` ra01s of a01) in the sense that
› ›PSn´1 ` P´1q‹ ˘ ´ s` ra01s › › ď , where is a small numerical constant.
Proof Note thatRCDL Ď RCDL is due to the fact that ›
›AJq › ›
2 3 ď › ›AJq › › 2 “ K.
We show that the iterates of Algorithm 1 converge to one of the target solutions by the following.
Initialization falls into RCDL. From Proposition E.5, taking ξ “ ξCDLκ4{3, with θ satisfies Equation (E.3), whenever
p ě C1 K2
µ4{3θ κ10{3pA0q σ2minpA0q logpmq,
w.h.p. our initialization qinit satisfies ϕTpqinitq ď ´2ξCDL µ2{3κ4{3K.
Iterate stays within the region. Let tqpkqu be the sequence generated by Algorithm 1 with qp0q “ qinit. From Proposition E.12, we know that whenever
p ě C2 θK2
µ4{3κ8{3 max
"
K6κ6pA0q σ2minpA0q , n
*
n2 log ´ θnµ´2{3κ´4{3 ¯ log5pmKq,
we have
sup qPSn´1
ˇ ˇ ˇ ˇ ϕCDLpqq ´ ˆ ϕTpqq ´ θ
2p1´ θqK 2
˙ ˇ
ˇ ˇ ˇ ď 1 2 ξCDL µ 2{3κ4{3K,
which together with the fact that the sequence tqpkqu satisfies ϕCDLpqpkqq ď ϕCDLpqp0qq implies
ϕTpqpkqq ď ϕCDLpqpkqq ` θ 2p1´ θqK 2 ` 1 2 ξCDL µ 2{3κ4{3K
ď ϕCDLpqp0qq ` θ 2p1´ θqK 2 ` 1 2 ξCDL µ 2{3κ4{3K
ď ϕTpqp0qq ` ξCDL µ2{3κ4{3K ď ´ξCDL µ2{3κ4{3K.
Closeness to the target solution. From Theorem E.3, we know that whenever
p ě CθK2µ´2n4 max " K6κ6pA0q σ2minpA0q , n * log6pm{µq,
the function ϕCDLpqq has benign optimization landscape, that whenever our method can efficient escape strict saddle points, Algorithm 1 produces a solution q‹ that is close to one of the target solutions (e.g. a1, the first column ofA) in the sense that
B
a1 }a1} , q‹
F
ě 1´ ε,
with ε “ κ´2η. In the following, we show that our final output a‹ “ PSn´1 ` P´1q‹ ˘
should be correspondingly close to a circulant shift of one of the kernels ta0kuKk“1. Without loss of generality, suppose q‹ “ a1, then the corresponding solution should be a01 with zero shift (or in other words, the first column a01 ofA0). In the following, we make this rigorous. Notice that ›
›PSn´1 ` P´1q‹ ˘ ´ a01 › › “ › › ›
›
PSn´1 ` P´1q‹ ˘ ´ PSn´1 ˆ a01 }a1}
˙›
› › › ď 2 }a1} › › › › P´1q‹ ´ a01 }a1} › › › › ,
where for the last inequality we used Lemma A.12. Next, by triangle inequality, we have ›
›PSn´1 ` P´1q‹ ˘ ´ a01 › ›
ď 2 }a1} › › ›
› P´1 a1 }a1} ´ a01}a1}
› › › › ` 2 }a1} › › › › P´1 ˆ a1 }a1} ´ q‹ ˙ › › › ›
“ 2 › ›
›
´
P´1 ` K´1A0A J 0
˘´1{2 ´ I ¯
a01
› › › ` 2 }a1}
› › › › P´1 ˆ a1 }a1} ´ q‹ ˙ › › › ›
ď 2
› › › › › ˆ 1 θmp Y Y J ˙1{2 ` A0A J 0 ˘´1{2 ´ I › › › › › ` 2 ? 2 }a1} › ›P´1 › ›
d
1´ B a1 }a1} , q‹ F .
Let δ P p0, 1q be a small constant. From Lemma E.18 and Corollary E.19, we know that whenever
p ě Cθ´1K3 κ 6pA0q
σ2minpA0q δ´2 logpmq,
we have ›
› › › ›
ˆ
1
θmp Y Y J
˙1{2 `
A0A J 0
˘´1{2 ´ I
› › › › › ď δ, › ›P´1 › › ď 2K´1{2 }A0} .
Therefore, we obtain ›
›PSn´1 ` P´1q‹ ˘ ´ a01 › › ď 2δ ` 4 ? 2 }A0} ? ε
ď 2δ ` 4 ? 2 ? ησmaxpA0qκ´1 ď 2δ ` 4 ? 2 ? η ď
when η is sufficiently small. Here, is a small numerical constant.
E.3 PROOF OF INITIALIZATION
In this subsection, we show that we can cook up a good data-driven initialization. We initialize the problem by using a random sample (1 ď ` ď p)
qinit “ PSn´1 pPy`q , 1 ď ` ď p,
which roughly equals to
qinit « PSn´1 pAx`q , AJqinit « ? KPSm´1 ` AJAx` ˘ .
For generic kernels,AJA is a close to a diagonal matrix, as the magnitudes of off-diagonal entries are bounded by column mutual incoherence. Hence, the sparse property of x` should be approximately preserved, so thatAJqinit is spiky with large › ›AJqinit › › 4
4 . We define
ζinit “ AJqinit, pζinit “ ? KPSm´1 ` AJAx` ˘ .
By leveraging the sparsity level θ, one can make sure that such an initialization qinit suffices.
Proposition E.5 Let m “ nK. Suppose the sparsity level θ satisfies
c1 logm
m ď θ ď c2
Kµ´2{3
ξm logm ¨min
"
ξ
Kµ4{3 ,
µ´4
m2 logm
*
.
Whenever
p ě C K 2
µ4{3ξ2θ κ6pA0q σ2minpA0q logpmq,
for some ξ ą 0 we have
}ζinit}44 ě ξKµ 2{3
holds with probability at least 1´ cm´c1 . Here, c1, c2, c, c1, C ą 0 are some numerical constants.
Proof By using the convexity of `4-loss, we can show that the values of }ζinit}44 and › › › pζinit › › ›
4 4 are
close,
}ζinit}44 ě › › › pζinit › › ›
4 4 ` 4 A pζd3init, ζinit ´ pζinit E ě › › › pζinit › › › 4 4 ´ 4 › › › pζd3init › › › › › › ζinit ´ pζinit › › ›
ě › ›
› pζinit
› › ›
4 4 ´ 4K3{2 › › › ζinit ´ pζinit › › ›
loooooomoooooon
small
. (E.5)
Thus, it is enough to lower bound › ›
› pζinit
› › ›
4 4 . Let I “ supppx`q, and let PI : Rm ÞÑ Rm that maps all
off support entries to zero and all on support entries to themselves. Thus, we have ›
› › pζinit
› › ›
4 4 “ K2 › ›AJAx` › › ´4 › ›AJAx` › › 4 4
ě K2 ´ › ›PI ` AJAx` ˘ › › 2 ` › ›PIc ` AJAx` ˘ › ›
2 ¯´2 ›
›PI ` AJAx` ˘ › ›
4
4
“ K 2
p1` ρq2 › ›PSn´1 ` PI ` AJAx` ˘˘ › ›
4 4 ,
with ρ :“ ˆ }PIcpAJAx`q} }PIpAJAx`q}
˙2
. By Lemma E.7 and Lemma E.9, whenever
c1 logm
m ď θ ď c2
µ´2
m logm ,
we have ›
›PIc ` AJAx` ˘› › ď C1Kµm a θ logm, › ›PI ` AJAx` ˘› › ě 1? 2 K ? θm
holding with probability at least 1´ c3m´c4 , so that
ρ “ ˜ › ›PIc ` AJAx` ˘ › ›
}PI pAJAx`q}
¸2
ď C2µ2m logm.
Thus, we have ›
› › pζinit
› › ›
4 4 ě K2p1` ρq´2 › ›PSm´1 ` PIAJAx` ˘› › 4 4 ě C3K
2
µ4m2 log2m
› ›PSm´1 ` PIAJAx` ˘› ›
4 4 .
By Lemma E.10, we have
› ›PSm´1 ` PIAJAx` ˘› ›
4 4 ě 1
2θm
with probability at least 1´ c5m´c6 . Thus, with high probability, we have ›
› › pζinit
› › ›
4 4 ě C3K
2
µ4m2 log2m ¨ 1 2θm ě 2ξKµ2{3, (E.6)
whenever
θ ď C4 Kµ´2{3
ξm ¨ 1 µ4m2 log2m .
Finally, Lemma E.6 implies that for any δ P p0, 1q, whenever
p ě C5θ´1K3 κ6pA0q σ2minpA0q δ´2 logpmq,
it holds that ›
› › ζinit ´ pζinit
› › › ď δ,
with probability at least 1´ c7pmq´c8 . Choose δ such that
4K3{2 › › › ζinit ´ pζinit › › › ď 4K3{2δ ď ξKµ2{3 ùñ δ ď C6ξK´1{2µ2{3, (E.7)
then by Equations (E.5) to (E.7) we have
}ζinit}44 ě › › › pζinit › › ›
4 4 ´ 4K3{2 › › › ζinit ´ pζinit › › › ě ξKµ2{3.
Summarizing all the result above, we obtain the desired result.
Lemma E.6 Let δ P p0, 1q. Whenever
p ě Cθ´1K3 κ 6pA0q
σ2minpA0q δ´2 logpmq,
we have ›
› › ζinit ´ pζinit
› › › ď δ
with probability at least 1´ c1pKnq´c2 . Here, c1, c2, C ą 0 are some numerical constants.
Proof By definition, we observe ›
› › ζinit ´ pζinit
› › › “ › › › AJPSn´1 pPy`q ´ ? KPSn´1 ` AJAx` ˘ › › ›
“
› › › › › AJPSn´1 ˜ ˆ 1 θKmp Y Y J ˙´1{2 A0x` ¸ ´ ? KPSn´1 ` AJAx` ˘ › › › › ›
“
› › › › › › › › AJ ´ 1 θmpY Y J ¯´1{2 A0x` › › ›
›
´
1 θmpY Y
J ¯´1{2
A0x`
› › › ›
´ A JAx` }Ax`}
› › › › › › › ›
ď 2 }A} }Ax`}
› › › › › ˆ 1 θmp Y Y J ˙´1{2 A0x` ´ ` A0A J 0 ˘´1{2 A0x` › › › › ›
ď 2 ? K }x`} }Ax`} }A0}
› › › › › ˆ 1 θmp Y Y J ˙´1{2 ´ ` A0A J 0 ˘´1{2 › › › › ›
“ 2 ? K }A0}
› › › › › ˆ 1 θmp Y Y J ˙´1{2 ´ ` A0A J 0 ˘´1{2 › › › › › ,
where for the first inequality we invoked Lemma A.12, and the last equality follows the fact that minimum singular value ofA is unity. Next, by Lemma E.18, for some P p0, 1q, whenever
p ě Cθ´1K2 κ 4pA0q
σ4minpA0q ´2 logpmq,
we have ›
› › ζinit ´ pζinit
› › › ď 8 ? K }A0}
holding with probability at least 1´ c1pmq´c2 . Here, c1, , c2, C ą 0 are some numerical constants. Replace δ “ 8 ? K }A0} , we obtain the desired result.
Lemma E.7 Suppose the columns ofA are µ-incoherent and satisfies Assumption 3.1, and suppose x` satisfies Assumption E.2. Let I “ supp px`q. For any t ě 0, we have
› ›PIc ` AJAx` ˘ › › ď › ›offdiag ` AJA ˘ x` › › ď t
holds with probability at least 1´ 4m exp ´ ´min !
t2
4K2µ2θm2 , t 4Kµm ? m
)¯
.
Proof Since we have ›
›PIc ` AJAx` ˘ › › ď › ›offdiag ` AJA ˘ x` › › , (E.8)
we could bound › ›PIcAJAx` › › via controlling › ›offdiag ` AJA ˘ x` › ›. Let
M “ offdiag ` AJA ˘ “ rm1 ¨ ¨ ¨ mms P Rmˆm, and s “ Mx` “ m ÿ
k“1 mkx`k loomoon
sk
.
Thus, we can apply vector version Bernstein inequality. By Lemma A.3 and the fact that }mk} ď Kµ ? m,
E rsks “ 0, E r}sk}ps “ θ }mk}p Eg„N p0,1q r|g| ps ď m!
2 θ `
Kµ ? m ˘p .
Therefore, by applying Lemma A.6, we obtain
P ` › ›offdiag ` AJA ˘ x` › › ě t ˘ “ P ˜ › › › ›
›
m ÿ k“1 sk ´ E rss
› › › › › ě t ¸
ď 2pm` 1q exp ˆ ´ t 2
2µ2K2θm2 ` 2Kµm ? mt
˙
.
Finally, Equation (E.8) gives the desired result.
Lemma E.8 We have ›
›diag ` AJA ˘ x` › › 2 ď K2θm ` t (E.9)
with probability at least 1´ exp ´ ´ 18 min ! t2 K4θm , t K2m )¯ .
Proof First, let
d “ diag ` AJA ˘ , s “ › ›diag ` AJA ˘ x` › › 2 “ m ÿ
k“1 d2kx 2 `k lomon
sk
,
where by Lemma A.4, we have
E r|sk|ps ď θK2p p!2p
2 , E rss “ θ
› ›diag ` AJA ˘ › ›
2 F ă K2θm.
Thus, by Bernstein inequality in Lemma A.5, we obtain
P ´ › ›diag ` AJA ˘ x` › › 2 ´K2θm ě t ¯ ď exp ˆ ´ t 2
4K4θm` 4K2mt
˙
,
as desired.
Lemma E.9 Suppose x` satisfies Assumption E.2. Suppose x` satisfies Assumption E.2. Let I “ supp px`q. Whenever θ satisfies
c1 logm
m ď θ ď c2
µ´2
m logm , (E.10)
we have ›
›PI ` AJAx` ˘ › › 2 ě 1 2 K2θm (E.11)
with probability at least 1´m´c. Here, c, c1, c2 ą 0 are some numerical constants.
Proof Notice that ›
›PI ` AJAx` ˘ › ›
2
“ › ›diag ` AJA ˘ x` ` PI ` offdiag ` AJA ˘ x` ˘ › ›
2
“ › ›diag ` AJA ˘ x` › › 2 ` › ›PI ` offdiag ` AJA ˘ x` ˘ › › 2 ` 2 @ diag ` AJA ˘ x`,PI ` offdiag ` AJA ˘ x` ˘D
ě › ›diag ` AJA ˘ x` › › 2 ´ 2 › ›diag ` AJA ˘ x` › › › ›PI ` offdiag ` AJA ˘ x` ˘ › › .
By Lemma A.9, Lemma E.7, and Lemma E.8, we have ›
›diag ` AJA ˘ x` › › 2 ď K2θm ` C1K2 a
θm logm › ›PI ` offdiag ` AJA ˘ x` ˘ › › ď C2θKµm a logm
holds with probability at least 1´m´c0 . Thus, we obtain
› ›PI ` AJAx` ˘ › › 2 ě K2θm ˜ 1´ C1 c logm
θm ´ C3µ
a
θm logm
¸
.
Finally, by using Equation (E.10), we have
› ›PI ` AJAx` ˘ › › 2 ě 1 2 K2θm
as desired.
Lemma E.10 Suppose x` satisfies Assumption E.2. Let I “ supp px`q. Whenever θ P ´ logm m , 1 2 ¯
, then we have
› ›PSm´1 ` PI ` AJAx` ˘˘ › ›
4 4 ě 1
2θm
with probability at least 1´m´c.
Proof By Lemma A.1, we know that for any z,
}z}44 ě }z} ´1 0 }z} 4 ,
and the fact that › ›PSm´1 ` PI ` AJAx` ˘˘ › › 0 “ }x`}0, we have
› ›PSm´1 ` PI ` AJAx` ˘˘ › ›
4 4 ě }x`}´10 .
By Lemma A.9, we have
}x`}0 ď 2θm ùñ › ›PSm´1 ` PI ` AJAx` ˘˘ › ›
4 4 ě 1
2θm
holds with probability at least 1´m´c.
E.4 CONCENTRATION AND PERTURBATION
We prove the following concentration results for Riemannian gradient and Hessian, and its function value.
Proposition E.11 For some small δ P p0, 1q, whenever the sample complexity satisfies
p ě Cδ´2θK4 max " K6κ6pA0q σ2minpA0q , n * n2 log ˆ θKn δ ˙ log5pmKq,
we have
sup qPSn´1
}gradϕCDLpqq ´ gradϕTpqq} ď δ
sup qPSn´1
}HessϕCDLpqq ´HessϕTpqq} ď δ
hold with probability at least 1´ c1pmKq´c2 . Here, c1, c2, C ą 0 are some numerical constants.
Proof Let pϕCDLpqq be introduced as Equation (E.12)
pϕCDLpqq “ ´ 1
12θp1´ θqnp › ›qJAX › ›
4 4 ,
so that we bound the Riemannian gradient and Hessian separately using triangle inequalities via pϕCDLpqq.
Riemannian gradient. Notice that
sup qPSn´1
}gradϕCDLpqq ´ gradϕTpqq}
ď sup qPSn´1 }gradϕCDLpqq ´ grad pϕCDLpqq} ` sup qPSn´1 }grad pϕCDLpqq ´ gradϕTpqq} .
From Proposition E.13, we know that whenever
p ě C1θK10 κ6pA0q σ2minpA0q δ´2n2 log5pmKq,
we have
sup qPSn´1
}gradϕCDLpqq ´ grad pϕCDLpqq} ď δ
2
with probability at least 1´ c1pmKq´c2 . On the other hand, Corollary F.9 implies that whenever
p ě C2δ´2θK5n2 log ˆ θKn
δ
˙
,
we have
sup qPSn´1
}grad pϕCDLpqq ´ gradϕTpqq} ď δ
2
holds with probability at least 1´ c3np´2. Combining the bounds above gives the desired result on the gradient.
Riemannian Hessian. Similarly, we have sup
qPSn´1 }HessϕCDLpqq ´HessϕTpqq}
ď sup qPSn´1 }HessϕCDLpqq ´Hess pϕCDLpqq} ` sup qPSn´1 }Hess pϕCDLpqq ´HessϕTpqq} .
From Proposition E.15, we know that whenever
p ě C3θK10 κ6pA0q σ2minpA0q δ´2n2 log5pmKq,
we have
sup qPSn´1
}HessϕCDLpqq ´Hess pϕCDLpqq} ď δ
2
with probability at least 1´ c4pmKq´c5 . On the other hand, Corollary F.10 implies that whenever p ě C4θK6δ´2n3 log pθKn{δq ,
we have
sup qPSn´1
}HessϕDLpqq ´HessϕTpqq} ă δ
2
holds with probability at least 1´ c4np´2. Combining the bounds above gives the desired result on the Hessian.
Similar to Lemma D.1, for convolutional dictionary learning, asymptotically we have
EX rϕCDLpqqs « EX rpϕCDLpqqs “ ϕTpqq ´ θ
2p1´ θqK 2, ϕTpqq “ ´
1
4
› ›qJA › ›
4 4 .
Next, we turn this asymptotical results into finite sample for the function value via concentration and preconditioning.
Proposition E.12 For some small δ P p0, 1q, whenever the sample complexity satisfies
p ě Cδ´2θK4 max " K6κ6pA0q σ2minpA0q , n * n2 log ˆ θKn δ ˙ log5pmKq,
we have
sup qPSn´1
› › › › ϕCDLpqq ´ ˆ ϕTpqq ´ θ
2p1´ θqK 2
˙ ›
› › › ď δ
hold with probability at least 1´ c1pmKq´c2 . Here, c1, c2, C ą 0 are some numerical constants.
Proof By triangle inequality, we have
sup qPSn´1
ˇ ˇ ˇ ˇ ϕCDLpqq ´ ˆ ϕTpqq ´ θ
2p1´ θqK 2
˙ ˇ
ˇ ˇ ˇ
ď sup qPSn´1 |ϕCDLpqq ´ pϕCDLpqq| loooooooooooooooooomoooooooooooooooooon
T1
` sup qPSn´1 |pϕCDLpqq ´ EX rpϕCDLpqqs| looooooooooooooooooooomooooooooooooooooooooon
T2
.
Thus, by using Corollary E.14 we can control T1. For T2, we can control in a similar way as Corollary F.9 or Corollary F.10. For simplicity, we omitted here.
E.5 PRECONDITIONING
In this part of appendix, let us introduce
ϕCDLpqq “ ´ 1
12θp1´ θqnp › ›qJpPA0qX ›
› , pϕCDLpqq :“ ´ 1
12θp1´ θqnp › ›qJAX › › .
(E.12)
In the following, we show that the differences of function value, Riemannian gradient, and Hessian of those two functions are small by preconditioning analysis. For simplicity, let us also introduce
v0pqq “ XJpPA0qJq, vpqq “ XJAJq. (E.13)
E.5.1 CONCENTRATION AND PRECONDITIONING FOR RIEMANNIAN GRADIENT AND FUNCTION VALUE
First, the gradients of ϕCDLpqq and pϕCDLpqq and their Riemannian variants can be written as
∇ϕCDLpqq “ ´ 1
3θp1´ θqnpPA0Xv d3 0 , ∇pϕCDLpqq “ ´
1
3θp1´ θqnpAXv d3,
gradϕCDLpqq “ PqK∇ϕCDLpqq, grad pϕCDLpqq “ PqK∇pϕCDLpqq,
where recall from Section 3 that we introduced the following preconditioning matrix
P “ ˆ 1
θKmp Y Y J
˙´1{2 “
«
A0
˜
1
θKmp
p ÿ i“1 XiX J i
¸
AJ0
ff´1{2
.
In the following, we show that the difference between gradϕCDLpqq and grad pϕCDLpqq is small.
Proposition E.13 Suppose θ P ` 1 m , 1 2 ˘ . For any δ P p0, 1q, whenever
p ě CθK10 κ 6pA0q
σ2minpA0q δ´2n2 log5pmKq,
we have
sup qPSn´1
}gradϕCDLpqq ´ grad pϕCDLpqq} ď δ
sup qPSn´1
}∇ϕCDLpqq ´∇pϕCDLpqq} ď δ
with probability at least 1´ c1pmKq´c2 . Here, c1, c2, C ą 0 are some numerical constants.
Proof Notice that we have
sup qPSn´1
}gradϕCDLpqq ´ grad pϕCDLpqq}
ď sup qPSn´1 }∇ϕCDLpqq ´∇pϕCDLpqq}
ď 1 3θp1´ θqnp supqPSn´1 › ›PA0Xv d3 0 ´AXvd3 › ›
ď 1 3θp1´ θqnp
ˆ
sup qPSn´1
› ›PA0X “ vd30 ´ vd3 ‰ › ›
loooooooooooooooooomoooooooooooooooooon
T1
` sup qPSn´1
› ›pPA0 ´AqXvd3 › ›
loooooooooooooooomoooooooooooooooon
T2
˙
.
Controlling T1. For the first term, we observe
T1 ď 1
3θp1´ θqnp }PA0} }X} supqPSn´1 › ›vd30 ´ vd3 › › ,
where for all q P Sn´1 we have ›
›vd30 ´ vd3 › › ď › ›vd2 ´ vd20 › › 8 }v} ` }v ´ v0} }v0} 2 8
ď ? K ´? K ` }PA0} ¯ }PA0 ´A} ˆ
max 1ďkďnp
}Xek} ˙2 }X}
` }PA0 ´A} }X} }PA0}2 ˆ
max 1ďkďnp
}Xek} ˙2
ď ´? K ` }PA0} ¯2 }X} ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A}
where for the last two inequalities we used Lemma E.16. Thus, we have
T1 ď ´? K ` }PA0} ¯2 }PA0} }X}2 ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} .
Controlling T2. For the second term, by Lemma E.16, we have
T2 ď }PA0 ´A} }X} }v}36 ď K 3{2 }X}2
ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} .
Summary. Putting all the bounds together, we have sup
qPSn´1 }gradϕCDLpqq ´ grad pϕCDLpqq}
ď 1 3θp1´ θqnp
„ ´? K ` }PA0} ¯2 }PA0} `K3{2  }X}2 ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} .
By Lemma E.17 and Lemma E.20, we have
}X} ď 2 a
θmp, max 1ďkďnp
}Xek} ď 4 ? θm logpKpq
with probably at least 1 ´ 2p´2. On the other hand, by Lemma E.19, there exists some constant C ą 0, for any P p0, 1q whenever
p ě Cθ´1K3 κ 6pA0q
σ2minpA0q ´2 logpmKq,
we have }PA0 ´A} ď , }PA0} ď 2 ? K hold with probability at least 1 ´ c1pmKq´c2 for some numerical constants c1, c2 ą 0. These together give T1 ď CK5{2θm log2pKmq . Replacing δ “ CK5{2θm log2 pKmq gives the desired result.
Here, the perturbation analysis for gradient also leads to the following result
Corollary E.14 For some small δ P p0, 1q, under the same setting of Proposition E.13, we have sup
qPSn´1 |ϕCDLpqq ´ pϕCDLpqq| ď δ
hold with probability at least 1´ c1pmKq´c2 . Here, c1, c2 ą 0 are some numerical constants.
Proof Under the same setting of Proposition E.13, we have
sup qPSn´1 |ϕCDLpqq ´ pϕCDLpqq| “ sup qPSn´1
1
4
ˇ ˇ ˇ ˇ
1
3θp1´ θqnp }v0} 4 4 ´
1
3θp1´ θqnp }v} 4 4
ˇ ˇ ˇ ˇ
“ sup qPSn´1
1
4
ˇ ˇ ˇ ˇ
1
3θp1´ θqnp @ q,PA0Xv d3 0 ´AXvd3 D
ˇ ˇ ˇ ˇ
ď 1 4 sup qPSn´1 }∇ϕCDLpqq ´ pϕCDLpqq} ď δ 4 ,
as desired.
E.5.2 CONCENTRATION AND PRECONDITIONING FOR RIEMANNIAN HESSIAN
For simplicity, let v0 and v be as introduced in Equation (E.13). Similarly, the Riemannian Hessian of ϕCDLpqq and pϕCDLpqq can be written as
HessϕCDLpqq “ ´ 1
3θp1´ θqnpPq K
”
3 pPA0qX diag ` vd20 ˘ XJ pPA0qJ ´ }v0}44 I ı PqK ,
Hess pϕCDLpqq “ ´ 1
3θp1´ θqnpPq K
”
3AX diag ` vd2 ˘ XJAJ ´ }v}44 I ı PqK ,
respectively. In the following, we show that the difference between gradϕCDLpqq and grad pϕCDLpqq is small.
Proposition E.15 Suppose θ P ` 1 m , 1 2 ˘ . For any δ P p0, 1q, whenever
p ě CθK10 κ 6pA0q
σ2minpA0q δ´2n2 log5pmKq,
we have
sup qPSn´1
}HessϕCDLpqq ´Hess pϕCDLpqq} ď δ
with probability at least 1´ c1pmKq´c2 . Here, c1, c2, C ą 0 are some numerical constants.
Proof Notice that
sup qPSn´1
}HessϕCDLpqq ´Hess pϕCDLpqq}
ď 1 θp1´ θqnp supqPSn´1 › › › pPA0 ´AqX diag ` vd20 ˘ XJ pPA0qJ › › ›
looooooooooooooooooooooooooooooomooooooooooooooooooooooooooooooon
T1
` 1 θp1´ θqnp supqPSn´1 › › › AX diag ` vd2 ˘ X pPA0 ´AqJ › › ›
loooooooooooooooooooooooooomoooooooooooooooooooooooooon
T2
` 1 θp1´ θqnp supqPSn´1 › › › AX diag ` vd20 ´ vd2 ˘ XJ pPA0qJ › › ›
loooooooooooooooooooooooooooomoooooooooooooooooooooooooooon
T3
` 1 3θp1´ θqnp supqPSn´1 ˇ ˇ ˇ }v}44 ´ }v0} 4 4 ˇ ˇ ˇ looooooomooooooon
T4
.
By using Lemma E.16, we have
T1 ď }PA0} }X}2 }PA0 ´A} sup qPSn´1 }v0}28 ď }PA0} 3 }X}2
ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} ,
T2 ď }A} }X}2 sup qPSn´1 }v}28 ď K 3{2 }X}2
ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} .
Similarly, Lemma E.16 implies that
T3 ď }PA0} }A} }X}2 sup qPSn´1
› ›vd20 ´ vd2 › ›
8
ď ? K ´? K ` }PA0} ¯ }PA0} }X}2 ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} ,
and
T4 ď sup qPSn´1
ˇ ˇ ˇ }v}44 ´ }v0} 4 4 ˇ ˇ ˇ ď 2 sup
qPSn´1
ˇ ˇ @ v ´ v0, 4vd3 D ˇ ˇ
ď 8 sup qPSn´1 }v ´ v0} }v}36
ď 8K3{2 }X}2 ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} .
Thus, combining all the results above, we obtain
sup qPSn´1
}HessϕCDLpqq ´Hess pϕCDLpqq}
ď 1 θp1´ θqnp
”´? K ` }PA0} ¯ }PA0}2 `K }PA0} ` 4K3{2 ı }X}2 ˆ
max 1ďkďnp
}Xek} ˙2 }PA0 ´A} .
By Lemma E.17 and Lemma E.20, we have
}X} ď 2 a
θmp, max 1ďkďnp
}Xek} ď 4 ? θm logpKpq
with probably at least 1 ´ 2p´2. On the other hand, by Lemma E.19, there exists some constant C ą 0, for any P p0, 1q whenever
p ě Cθ´1K3 κ 6pA0q
σ2minpA0q ´2 logpmKq,
we have
}PA0 ´A} ď , }PA0} ď 2 ? K
hold with probability at least 1 ´ c1pmKq´c2 for some numerical constants c1, c2 ą 0. These together gives
sup qPSn´1
}HessϕCDLpqq ´Hess pϕCDLpqq} ď C 1K5{2θm log2 pKpq .
Replacing δ “ C 1K5{2θm log2 pKpq gives the desired result.
E.5.3 AUXILIARY RESULTS
Lemma E.16 Let v0 and v be defined as in Equation (E.13), with
v0pqq “ XJ pPA0qJ q, vpqq “ XJAJq,
For all q P Sn´1, we have
}v}8 ď ? K max
1ďkďnp }Xek} , }v0}8 ď }PA0} max 1ďkďnp }Xek} ,
}v} ď ? K }X} , }v}66 ď K 3 }X}2 ˆ
max 1ďkďnp
}Xek} ˙4 ,
› ›vd2 ´ vd20 › › 8 ď ´? K ` }PA0} ¯ }PA0 ´A} ˆ
max 1ďkďnp
}Xek} ˙2 ,
}v ´ v0} ď }PA0 ´A} }X} .
Proof In the following, we bound each term, respectively.
Bounding norms of v and v0. For the `2-norm, notice that
}v} ď }X} }A} ď ? K }X}
On the other hand, for the `8-norm, we have
}v}8 “ max 1ďkďnp
› ›eJkX JAJq ›
› ď ? K max
1ďkďnp }Xek}
}v0}8 “ max 1ďkďnp
› › › eJkX J pPA0qJ q › › › ď }PA0} max
1ďkďnp }Xek} .
Thus, the results above give
}v}66 ď }v} 4 8 }v}
2 ď K3 }X}2 ˆ
max 1ďkďnp
}Xek} ˙4 .
Bounding the difference between v and v0. First, we bound the difference in `2-norm,
}v ´ v0} “ › › › XJ pPA0 ´AqJ q › › › ď }PA0 ´A} }X} .
On the other hand, we have ›
›vd2 ´ vd20 › › 8 ď }v ´ v0}8 }v ` v0}8 ď p}v}8 ` }v0}8q }v ´ v0}8 , where
}v ´ v0}8 “ max 1ďkďnp
› › › eJkX J pPA0 ´AqJ q › › › ď }PA0 ´A} max
1ďkďnp }Xek} ,
Thus, we obtain
› ›vd2 ´ vd20 › › 8 ď ´? K ` }PA0} ¯ }PA0 ´A} ˆ
max 1ďkďnp
}Xek} ˙2 ,
as desired.
Lemma E.17 SupposeX satisfies Assumption E.2, we have
max 1ďkďnp
}Xek} ď 4 ? θm logpKpq
with probability at least 1´ p´2θm.
Proof Let us write
Xi “ rrxi1 rxi2 ¨ ¨ ¨ rxins , with rxij “
»
— – sj´1 rxi1s ...
sj´1 rxiKs
fi
ffi fl 1 ď i ď p, 1 ď j ď n,
where s` r¨s denotes circulant shift of length `. GivenX “ rX1 ¨ ¨ ¨ Xps, we have
max 1ďkďnp }Xek} “ max 1ďiďp,1ďjďn }rxij} “ max 1ďiďp,1ďjďn
g f f e K ÿ
`“1 }sj´1 rxi`s}2
ď ? K max
1ďiďp,1ď`ďK }xi`} .
Next, we bound max1ďiďp,1ď`ďK }xi`}. By using Bernstein inequality in Lemma A.5, we obtain
P ´ ˇ ˇ ˇ }xi`}2 ´ nθ ˇ ˇ ˇ ě t ¯ ď 2 exp ˆ ´ t 2
4nθ ` 4t
˙
Thus, by using a union bound, we obtain
max 1ďiďp,1ď`ďK
}xi`} ď 4 ? θn logpKpq,
with probability at least 1´ p´2θm. Summarizing the bounds above, we obtain the desired result.
E.6 INTERMEDIATE RESULTS FOR PRECONDITIONING
Lemma E.18 SupposeX satisfies Assumption E.2. For any δ P p0, 1q, whenever
p ě Cθ´1K2 κ 4pA0q
σ4minpA0q δ´2 logpmq,
we have ›
› › › ›
ˆ
1
θmp Y Y J
˙´1{2 ´ `
A0A J 0
˘´1{2 › ›
› › › ď δ, › › › › › ˆ 1 θmp Y Y J ˙1{2 ` A0A J 0 ˘´1{2 ´ I › › › › › ď σminpA0q ¨ δ,
hold with probability at least 1´ c1pmKq´c2 . Here, c1, c2, C ą 0 are some numerical constants.
Proof Notice that 1
θmp Y Y J “ 1 θmp A0XX JAJ0 “ A0AJ0 loomoon
B
`A0 ˆ 1
θmp XXJ ´ I
˙
AJ0 loooooooooooooooomoooooooooooooooon
∆
.
By Lemma E.20, for any P p0, 1{Kq, whenever p ě Cθ´1K2 ´2 logpmKq,
we have ›
› › ›
1
θmp XXJ ´ I
› › › › ď ,
with probability at least 1´ c1pmKq´c2 . Thus, by the first inequality in Lemma A.10 we observe ›
› › › ›
ˆ
1
θmp Y Y J
˙´1{2 ´ pA0A0q´1{2
› › › › › “ › › › pB `∆q´1{2 ´B´1{2 › › ›
ď 4σ´2min pBq }∆} ď 4κ 2pA0q
σ2minpA0q
› › › › 1
θmp XXJ ´ I
› › › › ď 4κ 2pA0q
σ2minpA0q ¨ .
On the other hand, by using the second inequality in Lemma A.10, we have ›
› › › ›
ˆ
1
θmp Y Y J
˙1{2 `
A0A J 0
˘´1{2 ´ I
› › › › › “ › › › pB `∆q1{2B´1{2 ´ I › › ›
ď 4σ´3{2min pBq }∆} ď 4κ 2pA0q
σminpA0q
› › › › 1
θmp XXJ ´ I
› › › › ď 4κ 2pA0q
σminpA0q ¨ .
Choose “ ´ 4κ2pA0q σ2minpA0q ¯´1 δ, we obtain the desired results.
Given the definition of preconditioning matrix P , the result above leads to the following corollary.
Corollary E.19 Under the same settings of Lemma E.18, for any δ P p0, 1q, whenever
p ě Cθ´1K3 κ 6pA0q
σ2minpA0q δ´2 logpmKq,
we have
}PA0 ´A} ď δ, › ›P´1 › › ď 2K´1{2 }A0} ,
}PA0} ď }A} ` δ ď ? K ` δ
hold with probability at least 1´ c1pmKq´c2 . Here, c1, c2, C ą 0 are some numerical constants.
Proof For the first inequality, we have
}PA0 ´A} ď ? K
› › › › › ˆ 1 θmp Y Y J ˙´1{2 ´ ` A0A J 0 ˘´1{2 › › › › › }A0} .
Thus, for any δ P p0, 1q, Lemma E.18 implies that whenever
p ě Cθ´1K3 κ 6pA0q
σ2minpA0q δ´2 logpmKq,
we have
}PA0 ´A} ď δ, }PA0} ď }A} ` }PA0 ´A} ď ? K ` δ
with probability at least 1´ c1pmKq´c2 . On the other hand, by Lemma E.18 we have ›
›P´1 › › ď › › › P´1 ´ ` K´1A0A0 ˘1{2 › › › ` › › › ` K´1A0A0 ˘1{2 › › ›
ď › ›
›
` K´1A0A0 ˘1{2
› › › ´ 1 ` › › › P´1 ` K´1A0A0 ˘´1{2 ´ I › › › ¯
ď K´1{2 }A0} ˜ 1`
› › › › › ˆ 1 θmp Y Y J ˙1{2 ` A0A J 0 ˘´1{2 ´ I › › › › › ¸ ď 2K´1{2 }A0} ,
as desired.
Lemma E.20 SupposeX satisfies Assumption E.2. For any δ P p0, 1q, we have ›
› › ›
1
θmp XXJ ´ I
› › › › ď δ, }X} ď a θmp p1` δq
with probability at least 1 ´ c1mK exp ´ ´c2θpmin ! ` δ K ˘2 , δK )¯
. Here, c1, c2 ą 0 are some numerical constants.
Proof By using the fact thatX “ rX1 X2 ¨ ¨ ¨ Xps, we observe
XXJ “ p ÿ
k“1 XkX
J k , Xk “
»
— – Cxk1 ...
CxkK
fi
ffi fl
For any z P Sn´1, write z “
»
— – z1 ... zK
fi
ffi fl . We have
› › › › 1
θmp XXJ ´ I
› › › ›
“ sup zPSn´1
ˇ ˇ ˇ ˇ zJ ˆ 1
θmp XXJ ´ I
˙
z
ˇ ˇ ˇ ˇ
“ sup zPSn´1
ˇ ˇ ˇ ˇ ˇ 1 θmp zJ ˜ p ÿ
i“1 XiX
J i
¸ z ´ }z}2 ˇ ˇ ˇ
ˇ ˇ
“ sup zPSn´1
ˇ ˇ ˇ ˇ ˇ ˇ 1 θmp p ÿ
i“1
˜
K ÿ k“1 Cxikzk
¸J˜ K ÿ
i“1 Cxikzk
¸ ´ }z}2 ˇ ˇ ˇ ˇ
ˇ ˇ
“ sup zPSn´1
ˇ ˇ ˇ ˇ ˇ 1 θmp p ÿ
i“1
˜
K ÿ k“1 zJk C J xik Cxikzk ` 2 ÿ k “` zJk C J xik Cxi`z`
¸ ´ }z}2 ˇ ˇ ˇ
ˇ ˇ
ď sup zPSn´1
K ÿ
k“1
ˇ ˇ ˇ ˇ ˇ zJk ˜ 1 θmp p ÿ
i“1 CJxikCxik ´ I
¸
zk
ˇ ˇ ˇ ˇ ˇ ` 2 ÿ
k “`
ˇ ˇ ˇ ˇ ˇ zJk ˜ 1 θmp p ÿ
i“1 CJxikCxi`
¸
z`
ˇ ˇ ˇ ˇ ˇ
ď K´1 K ÿ
k“1
› › › › › 1 θnp p ÿ
i“1 CJxikCxik ´ I
› › › › › ` 2K´1 ÿ
k “`
› › › › › 1 θnp p ÿ
i“1 CJxikCxi`
› › › › › .
By Lemma E.21, we obtain ›
› › ›
1
θmp XXJ ´ I
› › › › ď t1 ` 2Kt2 ď δ
with probability at least
1´ 2m exp ` ´c1θpmin δ2, δ (˘ ´ 2mK exp ´ ´c2θpmin ! ` K´1δ ˘2 ,K´1δ )¯ .
Finally, the second inequality directly follows from the fact that ›
› › ›
1
θmp XXJ ´ I
› › › › ď δ ùñ }X}2 ď pθmpq p1` δq ,
as desired.
Lemma E.21 Suppose xij satisfies Assumption E.2. For any j P rKs, we have ›
› › › › 1 θnp
p ÿ i“1 CJxijCxij ´ I
› › › › › ď t1
holding with probability at least 1 ´ 2m exp ´ ´ θp8 min ! t21 2 , t1 )¯
. Moreover, for any k, ` P rKs with k “ `, we have
› › › › › 1 θnp p ÿ
i“1 CJxikCxi`
› › › › › ď t2
holding with probability at least 1´ 2m 2 n exp ´ ´ θp2 min t22, t2 ( ¯ .
Proof Notice that
CJxijCxij “ F ˚ diag
´ |Fxij |d2 ¯ F , CJxikCxi` “ F ˚ diag ` Fxik d Fxi` ˘ F . (E.14)
Bounding › ›
› 1 θnp řp i“1C J xijCxij ´ I
› › › . From Equation (E.14), we have
› › › › › 1 θnp p ÿ
i“1 CJxijCxij ´ I
› › › › › “ › › › › › F ˚ diag ˜ 1 θnp p ÿ
i“1 |Fxij |d2 ´ 1
¸
F
› › › › ›
ď
› › › › › 1 θnp p ÿ
i“1 |Fxij |d2 ´ 1
› › › › ›
8
.
Let f˚k be a row of F , by Lemma A.3 we have for any ` ě 1,
E ” |f˚k xij | 2` ı ď 2 ``!
2 Ebk„Berpθq
” }bk d fk}2` ı ď `! 2 θp2nq`.
Thus, by Bernstein inequality in Lemma A.5, we have
P
˜ ˇ
ˇ ˇ ˇ ˇ 1 θnp
p ÿ i“1 |f˚k xij | d2 ´ 1
ˇ ˇ ˇ ˇ ˇ ě t1 ¸ ď 2 exp ˆ ´ pθt 2 1
8` 4t1
˙
.
Thus, by using union bounds, we obtain ›
› › › › 1 θnp
p ÿ i“1 CJxijCxij ´ I
› › › › › ď › › › › › 1 θnp p ÿ
i“1 |Fxij |d2 ´ 1
› › › › ›
8
ď t1
for all 1 ď j ď K with probability at least 1´ 2nK exp ´ ´ θp8 min ! t21 2 , t1 )¯ .
Bounding › ›
› 1 θnp řp i“1C J xik Cxi`
› › › . On the other hand, by Equation (E.14), we know that
› › › › › 1 θnp p ÿ
i“1 CJxikCxi`
› › › › › ď › › › › › 1 θnp p ÿ
i“1 Fxik d Fxi`
› › › › ›
8
.
Let zk`id “ f˚d xikf˚d xi` “ xJikfdf˚d xi` (1 ď d ď n), we have its moments for s ě 1
E ” ˇ
ˇzk`id ˇ ˇ
s ı ď E ” ˇ
ˇxJikfd ˇ ˇ
s ı
E “ |f˚d xi`| s‰ ď s!
2 Ebd„Berpθq
” }bd d fd}2s ı ď s! 2 θns.
Thus, by Bernstein inequality in Lemma A.5, we obtain
P
˜
1
θnp
ˇ ˇ ˇ ˇ ˇ p ÿ
i“1 zk`id
ˇ ˇ ˇ ˇ ˇ ě t2 ¸ ď 2 exp ˆ ´ θpt 2 2
2` 2t2
˙
.
Thus, by applying union bounds, we have ›
› › › › 1 θnp
p ÿ i“1 CJxikCxi`
› › › › › ď t2
for all 1 ď k, ` ď K and k “ ` with probability at least 1´ 2mK exp ´
´ θp2 min t22, t2 (
¯
.
F MEASURE CONCENTRATION
In this part of the appendix, we show measure concentration of Riemannian gradient and Hessian for both ϕDLpqq and ϕCDLpqq over the sphere. Before that, we first show the following preliminary results that are key for our proof. For simplicity, we also use K “ m{n throughout the section.
F.1 PRELIMINARY RESULTS
Here, as the gradient and Hessian of `4-loss is heavy-tailed, traditional concentration tools do not directly apply to our cases. Therefore, we first develop some general tools for concentrations of superema of heavy-tailed empirical process over the sphere. In later part of this appendix, we will apply these results for concentration of Riemannian gradient and Hessian for both overcomplete dictionary learning and convolutional dictionary learning.
Theorem F.1 (Concentration of heavy-tailed random matrices over the sphere) Let Z1,Z2, ¨ ¨ ¨ ,Zp P Rn1ˆn2 be i.i.d. centered subgaussian random matrices, with Zi ”d Z p1 ď i ď pq and
E rZijs “ 0, P p|Zij | ą tq ď 2 exp ˆ ´ t 2
2σ2
˙
.
For a fixed q P Sn´1, let us define a function fqp¨q : Rn1ˆn2 ÞÑ Rd1ˆd2 , such that
1. fqpZq is a heavy tailed process of Z, in the sense of P ` }fqpZq} ě t ˘ ď 2 exp ` ´C ? t ˘ .
2. The expectation E rfqpZqs is bounded and Lf -Lipschitz, i.e., }E rfqpZqs} ď Bf , and }E rfq1pZqs ´ E rfq2pZqs} ď Lf }q1 ´ q2} , @ q1, q2 P Sn´1.
(F.1)
3. Let Z be a truncated random matrix of Z, such that
Z “ Z ` pZ, Zij “ " Zij if |Zij | ă B, 0 otherwise.
(F.2)
with B “ 2σ a
log pn1n2pq. For the truncated matrix Z, we further assume that › ›fqpZq › › ď R1pσq, max › ›E “ fqpZqJfqpZq ‰ › › , › ›fqpZqfqpZqJ › › (
ď R2pσq, (F.3) › ›fq1pZq ´ fq2pZq › › ď Lf pσq }q1 ´ q2} , @ q1, q2 P Sn´1. (F.4)
Then for any δ P ´
0, 6R2R1
¯
, whenever
p ě C max #
min td1, d2uBf n1n2δ , δ´2R2
«
n log
˜
6 ` Lf ` Lf ˘
δ
¸ ` logpd1 ` d2q ff+
we have
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpZiq ´ E rfqpZqs
› › › › › ď δ,
holding with probability at least 1´ pn1n2pq´2 ´ n´c logppLf`Lf q{δq for some constant c, C ą 0.
Proof As aforementioned, traditional concentration tools does not directly apply due to the heavytailed behavior of fqpZq. To circumvent the difficulties, we first truncate Z and introduce bounded random variable Z as in Equation (F.2), with truncation level B “ 2σ a log pn1n2pq. Thus, we have
P
˜
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpZiq ´ E rfqpZqs
› › › › › ě t ¸
ď P ˜
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpZiq ´ E rfqpZqs
› › › › › ě t ¸
looooooooooooooooooooooooooooomooooooooooooooooooooooooooooon
P1ptq
` P ˆ
max 1ďiďp
}Zi}8 ě B ˙
loooooooooooooomoooooooooooooon
P2
.
As fqpZq is also bounded, then we can apply classical concentration tools to P1ptq, and bound P2 by using subgaussian tails of Z. In the following, we make this argument rigorous with more technical details.
Tail bound for P2. Since Zijk is centered subgaussian, by an union bound, we have
P2 “ P ˆ
max 1ďiďp
}Zi}8 ě B ˙ ď n1n2pP `ˇ ˇZijk ˇ ˇ ě B ˘ ď exp ˆ ´ B 2
2σ2 ` log pn1n2pq
˙
.
Choose B “ 2σ a log pn1n2pq, we obtain
P2 “ P ˆ
max 1ďiďp
}Zi}8 ě B ˙ ď pn1n2pq´2 .
Tail Bound for › ›
› 1 p řp i“1 fqpZiq ´ E rfqpZqs
› › › with a fixed q P Sn´1. First, we control the quan-
tity for a given q P Sn´1. Later, we will turn the tail bound result to a uniform bound over the sphere for all q P Sn´1. We first apply triangle inequality, where we have ›
› › › › 1 p
p ÿ i“1 fqpZiq ´ E rfqpZqs
› › › › › ď › › › › › 1 p p ÿ
i“1 fqpZiq ´ E
“ fqpZq ‰
› › › › › ` › ›E rfqpZqs ´ E “ fqpZq ‰ › › ,
such that
P
˜ ›
› › › › 1 p
p ÿ i“1 fqpZiq ´ E rfqpZqs
› › › › › ě t ¸
ď P ˜ › › ›
› ›
1
p
p ÿ i“1 fqpZiq ´ E “ fqpZq ‰
› › › › › ě t´ › ›E rfqpZqs ´ E “ fqpZq ‰ › › ¸ .
Notice that › ›E “ fqpZq ‰ ´ E rfqpZqs › › ď › ›E “ fqpZq d 1Z‰Z ‰ › › F ď }E rfqpZqs}F › ›E “ 1Z‰Z ‰ › › F
ď min td1, d2uBf d ÿ
ij
P ` Zij “ Zij ˘
ď min td1, d2uBf
d
n1n2 exp
ˆ
´ B 2
2σ2
˙
,
where for the second inequality we used Cauchy-Schwarz inequality, the third one follows from
and the last one follows from the fact in Z is subgaussian. With B “ 2σ a
log pn1n2pq, we obtain › ›E “ fqpZq ‰ ´ E rfqpZqs ›
› ď min td1, d2uBf n1n2p ,
so that
P
˜ ›
› › › › 1 p
p ÿ i“1 fqpZiq ´ E rfqpZqs
› › › › › ě t ¸ ď P ˜› › › › › 1 p p ÿ
i“1 fqpZiq ´ E
“ fqpZq ‰
› › › › › ě t´ Bf n1n2p ¸ .
Next, we need to show concentration of › ›
› 1 p řp i“1 fqpZiq ´ E “ fqpZq ‰
› › › to finish this part of proof.
By our assumption in Equation (F.3), we apply bounded Bernstein’s inequality in Lemma A.7, such that
P
˜ ›
› › › › 1 p
p ÿ i“1 fqpZiq ´ E “ fqpZq ‰
› › › › › ě t1 ¸ ď pd1 ` d2q exp ˆ ´ pt 2 1
2R2 ` 4R1t2{3
˙
.
Choose p large enough such that
p ě 2 min td1, d2uBf n1n2t
ùñ min td1, d2uBf n1n2p ď t 2 .
Thus, for a fixed q P Sn´1, we have
P
˜ ›
› › › › 1 p
p ÿ i“1 fqpZiq ´ E rfqpZqs
› › › › › ě t ¸ ď P ˜ › › › › › 1 p p ÿ
i“1 fqpZiq ´ E
“ fqpZq ‰
› › › › › ě t{2 ¸
ď pd1 ` d2q exp ˆ ´ pt 2
8R2 ` 8R1t{3
˙
.
Bounding P1ptq via covering over the sphere Sn´1. Finally, we finish by . Let N p q be an epsilon net of the sphere, where we know that
@ q P Sn´1, D q1 P N p q, s.t. › ›q ´ q1 › › ď , and #N p q ď ˆ 3 ˙n´1 .
Thus, we have
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpZiq ´ E rfqpZqs
› › › › ›
“ sup q1PN p q,}e}ď
› › › › › 1 p p ÿ
i“1 fq1`epZiq ´ E rfq1`epZqs
› › › › ›
ď sup q1PN p q
› › › › › 1 p p ÿ
i“1 fq1pZiq ´ E rfq1pZqs
› › › › › ` sup q1PN p q,}e}ď › › › › › 1 p p ÿ i“1 fq1`epZiq ´ 1 p p ÿ i“1 fq1pZiq › › › › ›
` sup q1PN p q,}e}ď }E rfq1`epZqs ´ E rfq1pZqs} .
By our Lipschitz continuity assumption in Equation (F.1) and Equation (F.4), for any q P Sn´1, we obtain
}E rfq1`epZqs ´ E rfq1pZqs} ď Lf }e} , › › › › › 1 p p ÿ
i“1 fq1`epZiq ´
1
p
p ÿ i“1 fq1pZiq
› › › › › ď › ›fq1`epZq ´ fq1pZq › › ď Lf }e} ,
which implies that
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpZiq ´ E rfqpZqs
› › › › › ď sup q1PN p q › › › › › 1 p p ÿ i“1 fq1pZiq ´ E rfq1pZqs › › › › › ` ` Lf ` Lf ˘ .
Therefore, for any t ą 0, choose
ď t 2pLf ` Lf q ,
so that we obtain
P
˜
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpZiq ´ E rfqpZqs
› › › › › ě t ¸
ď P ˜
sup q1PN p q
› › › › › 1 p p ÿ
i“1 fq1pZiq ´ E rfq1pZqs
› › › › › ě t´ ` Lf ` Lf ˘ ¸
ď P ˜
sup q1PN p q
› › › › › 1 p p ÿ
i“1 fq1pZiq ´ E rfq1pZqs
› › › › › ě t{2 ¸
ď #N p q ¨ P ˜ › › ›
› ›
1
p
p ÿ i“1 fqpZiq ´ E rfqpZqs
› › › › › ě t{2 ¸
ď ˆ 3 ˙n´1 pd1 ` d2q exp ˆ ´ pt 2
32R2 ` 16R1t{3
˙
ď exp ˜ ´min " pt2
64R2 ,
3pt
32R1
* ` n log ˜ 6 ` Lf ` Lf ˘
t
¸ ` logpd1 ` d2q ¸ .
Summary of the results. Therefore, combining all the results above, for any δ P ´
0, 6R2R1
¯
, whenever
p ě C max #
min td1, d2uBf n1n2δ , δ´2R2
«
n log
˜
6 ` Lf ` Lf ˘
δ
¸ ` logpd1 ` d2q ff+ ,
we have
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpZiq ´ E rfqpZqs
› › › › › ď δ,
holding with probability at least 1´ pn1n2pq´2 ´ n´c logppLf`Lf q{δq for some constant c, C ą 0.
Corollary F.2 (Concentration of heavy-tailed random vectors over the sphere) Let z1, z2, ¨ ¨ ¨ , zp P Rn1 be i.i.d. centered subgaussian random matrices, with zi ”d z p1 ď i ď pq and
E rzis “ 0, P p|zi| ą tq ď 2 exp ˆ ´ t 2
2σ2
˙
.
For a fixed q P Sn´1, let us define a function fqp¨q : Rn1 ÞÑ Rd1 , such that
1. fqpzq is a heavy tailed process of z, in the sense of P ` }fqpzq} ě t ˘ ď 2 exp ` ´C ? t ˘ .
2. The expectation E rfqpzqs is bounded and Lf -Lipschitz, i.e., }E rfqpzqs} ď Bf , and }E rfq1pzqs ´ E rfq2pzqs} ď Lf }q1 ´ q2} , @ q1, q2 P Sn´1.
(F.5)
3. Let z be a truncated random matrix of z, such that
z “ z ` pz, zi “ " zi if |zi| ă B, 0 otherwise.
(F.6)
with B “ 2σ a log pn1pq. For the truncated matrix z, we further assume that
}fqpzq} ď R1pσq, E ” }fqpzq}2 ı ď R2pσq, (F.7)
}fq1pzq ´ fq2pzq} ď Lf pσq }q1 ´ q2} , @ q1, q2 P Sn´1. (F.8)
Then for any δ P ´
0, 6R2R1
¯
, whenever
p ě C max #
Bf n1δ , δ´2R2
«
n log
˜
6 ` Lf ` Lf ˘
δ
¸ ` logpd1q ff+ ,
we have
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpziq ´ E rfqpzqs
› › › › › ď δ,
holding with probability at least 1´ pn1pq´2 ´ n´c logppLf`Lf q{δq for some constant c, C ą 0.
Proof The proof is analogous to that of Theorem F.1. The slight difference is that we need to apply vector version Bernstein’s inequality in Lemma A.8 instead of matrix version in Lemma A.7, by utilizing our assumption in Equation (F.7). We omit the detailed proof here.
F.2 CONCENTRATION FOR OVERCOMPLETE DICTIONARY LEARNING
In this part of appendix, we assume that the dictionary A is tight frame with `2-norm bounded columns
1
K AAJ “ I, }ai} ď M p1 ď i ď mq. (F.9)
for some M with 1 ďM ď ? K.
F.2.1 CONCENTRATION OF gradϕDLp¨q
First, we show concentration of gradϕDLpqq to its expectation E rgradϕDLpqqs “ gradϕTpqq,
gradϕDLpqq “ ´ 1
3θp1´ θqpPq K
p ÿ
k“1
` qJAxk ˘3 pAxkq ÝÑ gradϕTpqq “ ´PqKA ` AJq ˘d3 ,
where xk follows i.i.d. BGpθq distribution in Assumption 2.2. Concretely, we have the following result.
Proposition F.3 (Concentration of gradϕDLp¨q) Suppose A satisfies Equation (F.9) and X P Rmˆp follows BGpθq with θ P `
1 m , 1 2
˘ . For any given δ P ` 0, cK2{pm log2 p log2 npq ˘
, whenever
p ě Cδ´2θK5n2 log ˆ θKn
δ
˙
,
we have
sup qPSn´1
}gradϕDLpqq ´ gradϕTpqq} ă δ
holds with probability at least 1´ c1p´2. Here, c, c1, C ą 0 are some numerical constants.
Proof Since we have
gradϕDLpqq “ ´ 1
3θp1´ θqpPq K
p ÿ
k“1
` qJAxk ˘3 pAxkq ,
we invoke Corollary F.2 to show this result by letting
fqpxq “ ´ 1
3θp1´ θq ` qJAx ˘3 PqKAx P Rn, (F.10)
where x „ BGpθq and we need to check the conditions in Equation (F.5), Equation (F.7), and Equation (F.8).
Calculating subgaussian parameter σ2 for x and truncation. Since each entry of x follows xi „i.i.d. BGpθq, its tail behavior is very similar and can be upper bounded by the tail of Gaussian, i.e.,
P p|xi| ě tq ď exp ` ´t2{2 ˘ ,
so that we choose the truncation level B “ 2 a log pnpq.
Calculating R1 and R2 in Equation (F.7). First, for each i p1 ď i ď pq, we have
}fqpxiq} “ 1
3θp1´ θq
› › › ` qJAxi ˘3 PqKAxi › › › ď
}Axi}4 3θp1´ θq ď }A}4 }xi}4 3θp1´ θq ď K2 }xi}4 3θp1´ θq .
By Lemma A.9 and a union bound, we know that for any 1 ď i ď p,
}xi}0 ď 4θm log p, }xi}0 ď 4θm log p ùñ }xi} 2 ď B2 }xi}0 “ 4B 2θm log p (F.11)
with probability at least 1´ p´2θm. Thus, by our truncation level, we have w.h.p.
}fqpxiq} ď 6θ
p1´ θqK 2B4m2 log2 p “ R1.
On the other hand, by Lemma F.5, for the second moment we have
E ” }fqpxiq}2 ı ď E ” }fqpxiq}2 ı ď cθK4m
for some constant c ą 0. Thus, we obtain
R1 “ 6θ
p1´ θqK 2B4m2 log2 p, R2 “ cθK4m. (F.12)
Calculating Lf in Equation (F.8). Notice that for any q1, q2 P Sn´1, let ζi “ AJqi pi “ 1, 2q, by Lemma F.4 we have
}fq1pxq ´ fq2pxq} “ 1
3θp1´ θq
› › › ` ζJ1 x ˘3 PqK1 Ax´ ` ζJ2 x ˘3 PqK2 Ax › › ›
ď }A} }x} 3θp1´ θq › › › ` ζJ1 x ˘3 PqK1 ´ ` ζJ2 x ˘3 PqK2 › › ›
ď }A} }x} 3θp1´ θq ” ˇ ˇζJ1 x ˇ ˇ 3 › › › PqK1 ´ PqK2 › › › ` ˇ ˇ ˇ ` ζJ1 x ˘3 ´ ` ζJ2 x ˘3 ˇ ˇ ˇ ı
ď }A} }x} 3θp1´ θq ” 2 }A}3 }x}3 }q1 ´ q2} ` 3 }A}3 }x}3 }q1 ´ q2} ı
ď 2 }A}4 }x}4
θp1´ θq }q1 ´ q2} .
where for the last two inequalities we used Lemma A.11 and ˇ
ˇ ˇ
` ζJ1 x ˘3 ´ ` ζJ2 x ˘3 ˇ ˇ ˇ “ ˇ ˇ ˇ pζ1 ´ ζ2qJ x ˇ ˇ ˇ ˇ ˇ ˇ ` ζJ1 x ˘2 ` ` ζJ1 x ˘ ` ζJ2 x ˘ ` ` ζJ2 x ˘2 ˇ ˇ ˇ
ď }A} }x} }q1 ´ q2} ” ` ζJ1 x ˘2 ` ` ζJ2 x ˘2 ` ˇ ˇζJ1 x ˇ ˇ ˇ ˇζJ2 x ˇ ˇ ı
ď 3 }A}3 }x}3 }q1 ´ q2} . Furthermore, by Equation (F.11) we obtain
}fq1pxq ´ fq2pxq} ď 2 }A}4 }x}4
θp1´ θq }q1 ´ q2} ď 32θ 1´ θK 2B4m2 log2 p }q1 ´ q2} .
This gives
Lf “ 32θ
1´ θK 2B4m2 log2 p. (F.13)
Calculating Bf and Lf in Equation (F.5). From Lemma F.4 we know that E rfqpxqs “ PqKAζ d3, so that
}E rfqpxqs} “ › › › PqKA ` AJq ˘d3 › › › ď › ›PqK › › }A} › ›AJq › › 3
6
ď }A} › ›AJq › › 3 ď }A}4 “ K2 “ Bf , (F.14)
where we used Lemma A.1 for the second inequality. Moreover, we have
}E rfq1pxqs ´ E rfq2pxqs}
ď › ›
› PqK1 Aζ d3 1 ´ PqK1 Aζ d3 2
› › › ` › ›
› PqK1 Aζ d3 2 ´ PqK2 Aζ d3 2
› › ›
ď }A} › ›ζd31 ´ ζd32 ›
› ` › ›
› PqK1 ´ PqK2
› › › }A} › ›ζd32 › ›
ď }A} › ›pζ1 ´ ζ2q d ` ζd21 ` ζ1 d ζ2 ` ζd21 ˘ › › ` 2 }A} }ζ2}3 }q1 ´ q2}
ď 5 }A}4 }q1 ´ q2} “ 5K2 }q1 ´ q2} “ Lf }q1 ´ q2} . (F.15)
where for the last inequality, we used the fact that ›
›pζ1 ´ ζ2q d ` ζd21 ` ζ1 d ζ2 ` ζd21 ˘ › › ď }ζ1 ´ ζ2}4 › ›ζd21 ` ζ1 d ζ2 ` ζd21 › ›
4
ď › ›AJ pq1 ´ q2q › › ` › ›ζd21 › › ` }ζ1 d ζ2} ` › ›ζd21 › › ˘
ď 3 }A}3 }q1 ´ q2} .
Thus, from Equation (F.14) and Equation (F.15), we obtain
Bf “ K2, Lf “ 5K2. (F.16)
Final calculation. Finally, we are now ready to put all the estimations in Equations (F.12), (F.13) and (F.16) together and apply Corollary F.2 to obtain our result. For any δ P ´
0, 6R2R1
¯
, whenever
p ě Cδ´2θK5n2 log pθKn{δq ,
we have
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpziq ´ E rfqpzqs
› › › › › ď δ,
holding with probability at least 1´pnpq´2´n´c1 logpθKn{δq´ p´2θm for some constant c1, C ą 0.
Lemma F.4 (Expectation of gradϕDLp¨q) @q P Sn´1, the expectation of gradϕDLp¨q satisfies
gradϕDLpqq “ gradϕTpqq “ ´PqKA ` AJq ˘d3
Proof Direct calculation.
Lemma F.5 Suppose x „ BGpθq and let fqpxq be defined as Equation (F.10), then we have
E ” }fqpxq}2 ı ď CθK4m pK “ m{nq.
Proof Since x „ BGpθq, we write x “ b d g with „ Berpθq and g „ N p0, Iq. Let I be the nonzero support of x with I “ suppx. And let PIp¨q be an operator that restricts a vector to the support I, so that we can write x “ PIpgq. Notice that
E }fqpxq}2 “ E « m ÿ
k“1
“ fd2q pxq ‰
k
ff
ď m max kPrms E “ fd2q pxq ‰ k .
LetW “ PqKA with wk being the kth row ofW . For @k P rns,
“ Efd2q pxq ‰ k “ 1 9θ2p1´ θq2E
»
–
` qJAx ˘6
˜
m ÿ i“1 wk,ixi
¸2 fi
fl
ď 1 9θ2p1´ θq2 ´ E @ AJq,x D12
¯ 1 2 ´
E xwk,xy4 ¯
1 2
“ 1 9θ2p1´ θq2 ´ E @ PI ` AJq ˘ , g D12
¯ 1 2 ´
E xPI pwkq , gy4 ¯
1 2
.
Notice that @
PI ` AJq ˘ ,v D „ N p0, › ›PI ` AJq ˘ › › 2q and xPI pwkq ,vy „ N p0, }PI pwkq}2q,
hence ´
E @ PI ` AJq ˘ ,v D12 ¯ 1 2 “ ? 11!! ´ EI › ›PI ` AJq ˘ › › 12 ¯ 1 2 .
LetAJq “ ζ, then we have
EI › ›PI ` AJq ˘ › › 12 “ ÿ
k1,k2,...,k6
m2k11k1PIζ 2 k21k2PIζ 2 k31k3PIζ 2 k41k4PIζ 2 k51k5PIζ 2 k61k6PI , (F.17)
for bounding equation F.17, we discuss the following cases:
• When only one index among k1, k2, . . . , k6 is in I:
EI › ›PI ` AJq ˘ › › 12 “ θ ÿ
k1
ζ12k1 ď θK 6
• When only two indices among k1, k2, . . . , k6 are in I:
EI › ›PI ` AJq ˘ › › 12 “ θ2 ÿ
k1,k2
`
ζ2k1ζ 10 k2 ` ζ 4 k1ζ 8 k2 ` ζ 6 k1ζ 6 k2
˘
ď 3θ2K6
• When only three indices among k1, k2, . . . , k6 are in I:
EI › ›PI ` AJq ˘ › › 12 “ θ3 ÿ
k1,k2,k3
`
ζ2k1ζ 2 k2ζ 8 k3 ` ζ 2 k1ζ 4 k2ζ 6 k3 ` ζ 4 k1ζ 4 k2ζ 4 k3
˘
ď 3θ3K6
• When only four indices among k1, k2, . . . , k6 are in I:
EI › ›PI ` AJq ˘ › › 12 “ θ4 ÿ
k1,k2,k3,k4
`
ζ2k1ζ 2 k2ζ 2 k3ζ 6 k4 ` ζ 2 k1ζ 2 k2ζ 4 k3ζ 4 k4
˘
ď 2θ4K6
• When only five indices among k1, k2, . . . , k6 are in I:
EI › ›PI ` AJq ˘ › › 12 “ θ5 ÿ
k1,k2,k3,k4,k5
`
ζ2k1ζ 2 k2ζ 2 k3ζ 2 k4ζ 4 k5
˘
ď θ5K6
• When all six indices of k1, k2, . . . , k6 are in I:
EI › ›PI ` AJq ˘› › 12 “ θ6 ÿ
k1,k2,k3,k4,k5,k6
`
ζ2k1ζ 2 k2ζ 2 k3ζ 2 k4ζ 2 k5ζ 2 k6
˘
ď θ6K6.
Hence, we have
EI › ›PI ` AJq ˘ › › 12 “ θK6 ` 3θ2K6 ` 3θ3K6 ` 2θ4K6 ` θ5K6 ` θ6K6 ď C1θK6
for a constant C1 ą 11. Similarly, we have ´
E xPI pwkq ,vy4 ¯ 1 2 “ ? 3 ´ EI }PI pwkq}4 ¯ 1 2 ,
and
EI }PI pwkq}4 “ ÿ
k1,k2
w2k,k11k1PIw 2 k,k21k2PI ď C2θ
m2 n2 ,
for a constant C2 ą 2. Hence, we have ´
E @ PI ` AJq ˘ , g D12
¯ 1 2 ´
E xPIwk, gy4 ¯ 1 2 ď C3θ m4
n4 ,
for a constant C3 ą 829. Hence, we know that @k P rns,
“ Efd2q pxq ‰ k ď C4 θp1´ θq2
m4 n4 “ CθK4,
for a constant C4 ą 93. Therefore
E }fqpxq}2 ď CθK4m,
for a constant C ą 93θ2p1´θq2 .
F.2.2 CONCENTRATION OF HessϕDLp¨q
Proposition F.6 (Concentration of HessϕDLp¨q) Suppose A satisfies Equation (F.9) and X P Rmˆp follows BGpθq with θ P `
1 m , 1 2
˘ . For any given δ P ` 0, cK2{plog2 p log2 npq ˘ , whenever
p ě Cδ´2θK6n3 log pθKn{δq ,
we have
sup qPSn´1
}HessϕDLpqq ´HessϕTpqq} ă δ
holds with probability at least 1´ c1p´2. Here, c, c1, C ą 0 are some numerical constants.
Proof Since we have
HessϕDLpqq “ ´ 1
3θp1´ θqp
p ÿ k“1 PqK ” 3 ` qJAxk ˘2 Axk pAxkqJ ´ ` qJAxk ˘4 I ı PqK ,
we invoke Theorem F.1 to show our result by letting
fqpxq “ ´ 1
3θp1´ θqPq K
”
3 ` qJAx ˘2 Ax pAxqJ ´ ` qJAx ˘4 I ı PqK P Rnˆn, (F.18)
where x „ BGpθq and we need to check the conditions in Equation (F.1), Equation (F.3), and Equation (F.4).
Calculating subgaussian parameter σ2 for x and truncation. Since each entry of x follows xi „i.i.d. BGpθq, its tail behavior is very similar and can be upper bounded by the tail of Gaussian, i.e.,
P p|xi| ě tq ď exp ` ´t2{2 ˘ ,
so that we choose the truncation level B “ 2 a
log pnpq. By Lemma A.9 and a union bound, we know that for any 1 ď i ď p,
}xi}0 ď 4θm log p, }xi}0 ď 4θm log p ùñ }xi} 2 ď B2 }xi}0 “ 4B 2θm log p (F.19)
with probability at least 1´ p´2θm.
Calculating R1 and R2 in Equation (F.3). For simplicity, let ξ “ Ax. First of all, we have
}fqpxq} “ 1
3θp1´ θq
› › › PqK ” 3 ` qJξ ˘2 ξξ J ´ ` qJξ ˘4 I ı PqK › › ›
ď 1 3θp1´ θq ` qJξ ˘2 › › › 3ξξ J ´ ` qJξ ˘2 I › › ›
ď 4 3θp1´ θq › ›ξ › › 4 ď 4 3θp1´ θq }A} 4 }x}4 ď 64B 4 3p1´ θqθK 2m2 log2 p.
On the other hand, by Lemma F.7, we have ›
›E “ fqpxqfqpxqJ ‰ › › “ › ›E “ fqpxqJfqpxq ‰ › › ď › ›E “ fqpxqJfqpxq ‰ › › ď c1θK4m2,
for some numerical constant c1 ą 0. In summary, we obtain
R1 “ 64B4
3p1´ θqθK 2m2 log2 p, R2 “ c1K4θm2. (F.20)
Calculating Lf in Equation (F.4). For any q1, q2 P Sn´1, we have
}fq1pxq ´ fq2pxq}
“ 1 3θp1´ θq › › › PqK1 ” 3 ` qJ1 ξ ˘2 ξξ J ´ ` qJ1 ξ ˘4 I ı PqK1 ´ PqK2 ” 3 ` qJ2 ξ ˘2 ξξ J ´ ` qJ2 ξ ˘4 I ı PqK2 › › ›
ď 1 θp1´ θq › › › PqK1 ` qJ1 ξ ˘2 ξξ J PqK1 ´ PqK2 ` qJ2 ξ ˘2 ξξ J PqK2 › › ›
looooooooooooooooooooooooooooomooooooooooooooooooooooooooooon
T1
` 1 3θp1´ θq › › › ` qJ1 ξ ˘4 PqK1 ´ ` qJ2 ξ ˘4 PqK2 › › ›
looooooooooooooooomooooooooooooooooon
T2
,
where by Lemma A.11, we have
T1 ď › ›
› PqK1
` qJ1 ξ ˘2 ξξ J PqK1 ´ PqK1 ` qJ1 ξ ˘2 ξξ J PqK2
› › › ` › ›
› PqK1
` qJ1 ξ ˘2 ξξ J PqK2 ´ PqK2 ` qJ2 ξ ˘2 ξξ J PqK2
› › ›
ď › ›ξ › ›
4 ›
› › PqK1 ´ PqK2
› › › ` › ›
› PqK1
` qJ1 ξ ˘2 ξξ J ´ PqK1 ` qJ2 ξ ˘2 ξξ J › › › ` › › › PqK1 ` qJ2 ξ ˘2 ξξ J ´ PqK2 ` qJ2 ξ ˘2 ξξ J › › ›
ď › ›ξ › ›
4 ›
› › PqK1 ´ PqK2
› › › ` › ›ξ › › 2 ` qJ1 ξ ` qJ2 ξ ˘ ` qJ1 ξ ´ qJ2 ξ ˘
ď 4 › ›ξ › › 4 }q1 ´ q2} ď 4 }A}4 }x}4 }q1 ´ q2} ď 64K2B4θ2m2 log2 p }q1 ´ q2} ,
and
T2 ď › › › ` qJ1 ξ ˘4 PqK1 ´ ` qJ2 ξ ˘4 PqK1 › › › ` › › › ` qJ2 ξ ˘4 PqK1 ´ ` qJ2 ξ ˘4 PqK2 › › ›
ď ´ ` qJ1 ξ ˘2 ` ` qJ2 ξ ˘2 ¯ pq1 ` q2qJ ξξ J pq1 ´ q2q ` 2 › ›ξ › › 4 }q1 ´ q2}
ď 6 › ›ξ › › 4 }q1 ´ q2} ď 6 }A}4 }x}4 }q1 ´ q2} ď 96K2B4θ2m2 log2 p }q1 ´ q2} ,
where for the last inequality we used Equation (F.19). Therefore, we have
}fq1pxq ´ fq2pxq} ď 96θ
1´ θK 2B4m2 log2 p }q1 ´ q2} ,
so that
Lf “ 96θ
1´ θK 2B4m2 log2 p. (F.21)
Calculating Bf and Lf in Equation (F.1). We have
}E rfqpxqs} “ › › › PqK ” 3A diag ` ζd2 ˘ AJ ´ }ζ}44 I ı PqK › › ›
ď › › › 3A diag ` ζd2 ˘ AJ ´ }ζ}44 I › › ›
ď 3 }A}2 }A}2`1Ñ`2 ` }A} 4 ď K ` 3M2 `K ˘ ,
where }A}`1Ñ`2 “ max1ďkďm }ak} ďM . On the other hand, for any q1, q2 P Sn´1, we have }E rfq1pxqs ´ E rfq2pxqs}
“ › ›
› PqK1
”
3Adiag ` ζd21 ˘ AJ ´ }ζ1}44 I ı PqK1 ´ PqK2 ” 3Adiag ` ζd22 ˘ AJ ´ }ζ2}44 I ı PqK2
› › ›
ď 3 › ›
› PqK1 Adiag
` ζd21 ˘ AJPqK1 ´ PqK2 Adiag ` ζd22 ˘ AJPqK2
› › ›
looooooooooooooooooooooooooooooooooooomooooooooooooooooooooooooooooooooooooon
L1
` › ›
› }ζ1}44PqK1 ´ }ζ2} 4 4PqK2
› › ›
loooooooooooooomoooooooooooooon
L2
.
By direct calculation, we have
L1 ď › ›
› PqK1 Adiag
` ζd21 ˘ AJPqK1 ´ PqK2 Adiag ` ζd22 ˘ AJPqK2
› › ›
ď › ›
› PqK1 Adiag
` ζd21 ˘
AJ ´ PqK1 ´ PqK2 ¯ › › › ` › › › ” PqK1 Adiag ` ζd21 ˘ ´ PqK2 Adiag ` ζd22 ˘ ı AJPqK2 › › ›
ď }A}2 }ζ1}28 › › › PqK1 ´ PqK2 › › › ` }A} ´ › › › ´ PqK1 ´ PqK2 ¯ A diag ` ζd21 ˘ › › › ` › › › PqK2 A diag ` ζd21 ´ ζd22 ˘ › › › ¯
ď 2 }A}2 }ζ1}28 }q1 ´ q2} ` 2 }A} 2 }ζ1}28 }q1 ´ q2} ` }A} 2 }ζ1 ` ζ2}8 }ζ1 ´ ζ2}8 ď 6 }A}2 }A}2`1Ñ`2 }q1 ´ q2} ď 6KM
2 }q1 ´ q2} , and
L2 ď }ζ1}44 › › › PqK1 ´ PqK2 › › › ` ˇ ˇ ˇ }ζ1}44 ´ }ζ2} 4 4 ˇ ˇ ˇ › › › PqK2 › › ›
ď 2 }A}4 }q1 ´ q2} ` |}ζ1}4 ´ }ζ2}4| p}ζ1}4 ` }ζ2}4q ´ }ζ1}24 ` }ζ2} 2 4 ¯
ď 2 }A}4 }q1 ´ q2} ` }ζ1 ´ ζ2} p}ζ1} ` }ζ2}q ´ }ζ1}2 ` }ζ2}2 ¯
ď 6 }A}4 }q1 ´ q2} “ 6K2 }q1 ´ q2} . These together give us
}E rfq1pxqs ´ E rfq2pxqs} ď 6K ` K `M2 ˘ }q1 ´ q2} . Summarizing everything together, we have
Bf “ K ` 3M2 `K ˘ , Lf “ 6K ` K `M2 ˘ . (F.22)
Final calculation. Finally, we are now ready to put all the estimations in Equations (F.20) to (F.22) together and apply Theorem F.1 to obtain our result. For any δ P ´
0, 6R2R1
¯
, whenever
p ě Cδ´2θK6n3 log pθKn{δq , we have
sup qPSn´1
› › › › › 1 p p ÿ
i“1 fqpziq ´ E rfqpzqs
› › › › › ď δ,
holding with probability at least 1´pnpq´2´n´c1 logpθKn{δq´ p´2θm for some constant c1, C ą 0.
Lemma F.7 Suppose θ P ` 1 m , 1 2 ˘ . Let fqpxq be defined as in Equation (F.18). We have ›
›E “ fqpxqJfqpxq ‰ › › ď CK4θm2
for some numerical constant C ą 0.
Proof Let x “ bd g with b „ Berpθq and g „ N p0, Iq. First, let ξ “ Ax, we have › ›E “ fqpxqJfqpxq ‰ › › “ › › › E ” 9 ` qJξ ˘4 PqKξξ JPqKξξ JPqK ´ 6 ` qJξ ˘6 PqKξξPqK ` ` qJξ ˘8 PqK ı › › ›
ď 9 › › › E ” ` qJξ ˘4 PqKξξ JPqKξξ JPqK ı › › ›
looooooooooooooooooooomooooooooooooooooooooon
T1
`6 › ›
› PqKE
”
` qJξ ˘6 ξξJ
ı PqK › › ›
loooooooooooooooomoooooooooooooooon
T2
`E ” ` qJξ ˘8 ı
looooomooooon
T3
.
Bound
T1 “ › › › E ” ` qJξ ˘4 PqKξξ JPqKξξ JPqK ı › › › ď › › › E ” ` qJξ ˘4 ξξJPqKξξ J ı › › ›
“ › › › E ” ` qJξ ˘4 › ›PqKξ › › 2 ξξJ ı › › › ď E ” ` qJξ ˘4 }ξ}4 ı ď EpqJξq8 (1{2 !E }ξ}8
)1{2
“ ! E ” @ PIpAJqq, g D8 ı)1{2
"
´m
n
¯4
E “ pxJxq4 ‰ *1{2 ,
where !
E ” @ PIAJq, g D8 ı) 1 2 “ ? 7!! ´ EI › ›PIAJq › › 8 ¯ 1 2 ď C1θ ´m
n
¯2
(F.23)
the proof of the last inequality is omitted, more details can be found in Lemma F.5, and
E ” ` xJx ˘4 ı “ E ” xPIx,PIxy4 ı “ E ” @ PIp1mq, gd2 D4 ı
ď c1mθ ` c2m2θ2 ` c3m3θ3 ` c4m4θ4. (F.24) combine, equation F.23 and equation F.24, yield
T1 ď C1θ3m2 ´m
n
¯4
.
T2 “ › › › PqKE ” ` qJξ ˘6 ξξJ ı PqK › › › ď › › › E ” ` qJξ ˘6 ξξJ ı › › › “ E ” ` qJξ ˘6 }ξ}2 ı ď EpqJξq12 (1{2 !E }ξ}4
)1{2
ď ! E @ AJq,x D12 )1{2 ! E }Ax}4 )1{2 “ ! E @ PIpAJqq, g D12 )1{2 " ´m
n
¯2 EpxJxq2 *1{2
ď C2EI ” › ›PIpAJqq › ›
12 ı1{2
„
´m
n
¯2 `
3mθ `mpm´ 1qθ2 ˘ 1{2 ď C2θ2m ´m
n
¯4
.
the proof of the first inequality in the last line is omitted, more details can be found in Lemma F.5.
T3 “ E ” @ PI ` AJq ˘ , g D8 ı ď C3EI ” › ›PI ` AJq ˘ › › 8 ı ď C3θ }A}8 ď C3θ ´m
n
¯4
.
Hence, summarizing all the results above, we obtain ›
›E “ fqpxqJfqpxq ‰ › › ď Cθm2 ´m
n
¯4
as desired.
Lemma F.8 (Expectation of HessϕDLp¨q) @q P Sn´1, the expectation of HessϕDLp¨q satisfies
HessϕDLpqq “ HessϕTpqq “ ´PqK ” 3Adiag ` pAqJqd2 ˘ AJ ´ › ›qJA › › 4 4 I ı PqK
Proof Direct calculation.
F.3 CONCENTRATION FOR CONVOLUTIONAL DICTIONARY LEARNING
In this section, we show concentration for the Riemannian gradient and Hessian of the following objective for convolutional dictionary learning,
pϕCDLpqq “ ´ 1
12θ p1´ θqnp › ›qJAX › ›
4 4 “ ´ 1
12θ p1´ θqnp
p ÿ
i“1
› ›qJAXi › ›
4
4
with
X “ rX1 X2 ¨ ¨ ¨ Xps , Xi “
»
— – Cxi1 ...
CxiK
fi
ffi fl , (F.25)
as we introduced in Section 3, where xij follows i.i.d. BGpθq distribution as in Assumption E.2. Since Cxij is a circulant matrix generated from xij , it should be noted that each row and column of X is not statistically independent, so that our concentration result of dictionary learning in the previous subsection does not directly apply here. However, from Lemma D.1, asymptotically we still have
EX rpϕCDLpqqs “ ϕTpqq ´ θ
2p1´ θqK 2, ϕTpqq “ ´
1
4
› ›qJA › ›
4 4 ,
in the following we prove finite sample concentration of pϕCDLpqq to its expectation ϕTpqq by leveraging our previous results for overcomplete dictionary learning in Proposition F.3 and Proposition F.6.
F.3.1 CONCENTRATION FOR grad pϕCDLp¨q
Corollary F.9 (Concentration of grad pϕCDLp¨q) Suppose A satisfies Equation (F.9) and X P Rmˆnp is generated as in Equation (F.25) with xij „i.i.d. BGpθq p1 ď i ď p, 1 ď j ď Kq and θ P `
1 m , 1 2
˘ . For any given δ P ` 0, cK2{pm log2 p log2 npq ˘ , whenever
p ě Cδ´2θK5n2 log ˆ θKn
δ
˙
,
we have
sup qPSn´1
}grad pϕCDLpqq ´ gradϕTpqq} ă δ
holds with probability at least 1´ c1np´2. Here, c, c1, C ą 0 are some numerical constants.
Remark. Note that our prove have not utilized the convolutional structure of the problem, so that our sample complexity could be loose of a factor of order n.
Proof Let us write
Xi “ rrxi1 rxi2 ¨ ¨ ¨ rxins , with rxij “
»
— – sj´1 rxi1s ...
sj´1 rxiKs
fi
ffi fl 1 ď i ď p, 1 ď j ď n,
(F.26)
where s` r¨s denotes circulant shift of length `. Thus, the Riemannian gradient of pϕCDLpqq can be written as
grad pϕCDLpqq “ ´ 1
3θp1´ θqnpPq K
p ÿ
i“1
n ÿ
j“1
` qJArxij ˘3 pArxijq
“ 1 n
n ÿ
j“1
„
´ 1 3θp1´ θqpPq K
p ÿ
i“1
` qJArxij ˘3 pArxijq
looooooooooooooooooooooooomooooooooooooooooooooooooon
gradj pϕCDLpqq

,
so that for each j with 1 ď j ď n,
gradj pϕCDLpqq “ ´ 1
3θp1´ θqpPq K
p ÿ
i“1
` qJArxij ˘3 pArxijq
is a summation of independent random vectors across p. Hence, we have
sup qPSn´1
}grad pϕCDLpqq ´ gradϕTpqq} ă 1
n
n ÿ
j“1
˜
sup qPSn´1
› ›gradj pϕCDLpqq ´ gradϕTpqq › ›
¸
,
where for each j we can apply concentration results in Proposition F.3 for controlling each individual quantity ›
›gradj pϕCDLpqq ´ gradϕTpqq ›
›. Therefore, by using a union bound we can obtain the desired result.
Corollary F.10 (Concentration of Hess pϕCDLp¨q) Suppose A satisfies Equation (F.9) and X P Rmˆnp is generated as in Equation (F.25) with xij „i.i.d. BGpθq p1 ď i ď p, 1 ď j ď Kq and θ P `
1 m , 1 2
˘ . For any given δ P ` 0, cK2{pm log2 p log2 npq ˘ , whenever
p ě Cδ´2θK6n3 log pθKn{δq , we have
sup qPSn´1
}HessϕDLpqq ´HessϕTpqq} ă δ
holds with probability at least 1´ c1np´2. Here, c, c1, C ą 0 are some numerical constants.
Proof Similar to the proof of Corollary F.9, the Riemannian Hessian of pϕCDLpqq can be written as Hess pϕCDLpqq
“ ´ 1 3θp1´ θqnp
p ÿ
i“1
n ÿ j“1 PqK ” 3 ` qJArxij ˘2 Axk pArxijqJ ´ ` qJArxij ˘4 I ı PqK
“ 1 n
n ÿ
j“1
"
´ 1 3θp1´ θqp
p ÿ i“1 PqK ” 3 ` qJArxij ˘2 Axk pArxijqJ ´ ` qJArxij ˘4 I ı PqK
looooooooooooooooooooooooooooooooooooooooooooooomooooooooooooooooooooooooooooooooooooooooooooooon
Hessj pϕCDLpqq
*
,
so that for each j with 1 ď j ď n,
Hessj pϕCDLpqq “ ´ 1
3θp1´ θqp
p ÿ i“1 PqK ” 3 ` qJArxij ˘2 Axk pArxijqJ ´ ` qJArxij ˘4 I ı PqK
is a summation of independent random vectors across p. Hence, we have
sup qPSn´1
}Hess pϕCDLpqq ´HessϕTpqq} ă 1
n
n ÿ
j“1
˜
sup qPSn´1
}Hessj pϕCDLpqq ´HessϕTpqq} ¸ ,
where for each j we can apply concentration results in Proposition F.6 for controlling each individual quantity }Hessj pϕCDLpqq ´HessϕTpqq}. Therefore, by using a union bound we can obtain the desired result.
G OPTIMIZATION ALGORITHMS
G.1 OPTIMIZATION
In this part of the appendix, we introduce algorithmic details for optimizing the following problem
min q
ϕpqq, q P Sn´1,
where the loss function ϕpqq and its gradient ∇ϕpqq for different problems are listed in Table 1.
Algorithm 2 Projected Riemannian Gradient Descent Algorithm
Input: Data Y P Rnˆp Output: the vector q‹
1: Initialize the iterate qp0q randomly, and set a stepsize τ p0q. 2: while not converged do 3: Compute Riemannian gradient gradϕpqpkqq “ PpqpkqqK∇ϕpq
pkqq. 4: Update the iterate by
qpk`1q “ PSn´1 ´ qpkq ´ τ pkq gradϕpqpkqq ¯ .
5: Choose a new stepsize τ pk`1q, and set k Ð k ` 1. 6: end while
Algorithm 3 Power Method
Input: Data Y P Rnˆp Output: the vector q‹
1: Randomly initialize the iterate qp0q. 2: while not converged do 3: Compute the gradient∇ϕpqpkqq. 4: Update the iterate by
qpk`1q “ PSn´1 ´ ´∇ϕpqpkqq ¯ .
5: Set k Ð k ` 1. 6: end while
Riemannian gradient descent. To optimize the problem, the most natural idea is starting from a random initialization, and taking projected Riemannian gradient descent steps
q Ð PSn´1 pq ´ τ ¨ gradϕpqqq , gradϕpqq “ PqK∇ϕpqq, (G.1)
where τ is the stepsize that can be chosen via linesearch or set as a small constant. We summarize this simple method in Algorithm 2.
Power method. In Algorithm 3 we also introduce a simple power method17 Journée et al. (2010) by noting that the loss function ϕpqq is concave so that the problem is equivalent to maximizing a convex function. For each iteration, we simply update q by
q Ð PSn´1 p´∇ϕpqqq
which is parameter-free and enjoys much faster convergence speed. We summarized the method in Algorithm 3. Notice that the power iteration can be interpreted as the Riemannian gradient descent with varied step sizes in the sense that
PSn´1 pq ´ τ ¨ gradϕpqqq “ PSn´1 ˆ ´ τ∇ϕpqq ` ` 1´ τ ¨ qJ∇ϕpqq ˘
loooooooooomoooooooooon
“0
q
˙
“ PSn´1 p´∇ϕpqqq
by setting τ “ 1qJ∇ϕpqq .
G.2 FAST IMPLEMENTATION OF CDL VIA FFT
Given the problem setup of CDL in Section 3, in the following we describe more efficient implementation of solving CDL using convolution and FFTs. Namely, we show how to rewrite the gradient of ϕCDLpqq in the convolutional form. Notice that the preconditioning matrix can be rewrite as a
17Similar approach also appears in (Zhai et al., 2019).
circulant matrix by
P “ ˜ 1
θnp
p ÿ i“1 CyiC J yi
¸´1{2
“ F ˚ diag ppqF “ Cp, p “ F´1 ˜ 1
θnp
p ÿ i“1 |pyi|d2
¸´1{2
,
where pyi “ Fyi. Thus, we have
PCyi “ CpCyi “ Cpfyi “ Cypi , y p i “ pf yi,
so that
min q
ϕCDLpqq “ ´ 1
4np
p ÿ
i“1
› ›CJpfyiq › ›
4 4 “ ´ 1
4np
p ÿ
i“1
› › › |ypi f q › › ›
4 4 , s.t. q P Sn´1,
Thus, we have the gradient
∇ϕCDLpqq “ ´ 1
np
p ÿ i“1 ypi f ´ |ypi f q ¯d3 ,
where qv denote a cyclic reversal of any v P Rn, i.e., qv “ rv1, vn, vn´1, ¨ ¨ ¨ , v2sJ.

Review summaries:
This paper studies the problem of non-convex optimization for Dictionary Learning (DL) in the situation when the underlying dictionary is over-complete (more basis vectors m than the dimension n). The paper proves that the L4 maximization formulation has a nice global landscape and can be efficiently minimized by (Riemannian) gradient descent, when the over-complete ratio m/n is less than an absolute constant. A similar result is proved for convolutional dictionary learning.