Optimization and Adaptive Generalization of Three layer Neural Networks

1 INTRODUCTION The ability of overparameterized neural networks trained by ( stochastic ) gradient descent to generalize well on test data ( Krizhevsky et al. , 2012 ; Silver et al. , 2016 ; Hinton et al. , 2012 ) , even if they perfectly fit the the training data , has intrigued theoretical researchers and led to many approaches for generalization bounds ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2018 ; Dziugaite & Roy , 2017 ; Wei et al. , 2019 ; Golowich et al. , 2018 ; Arora et al. , 2018b ; Zhou et al. , 2018 ; Konstantinos et al. , 2017 ) . This generalization ability is closely tied to the optimization procedure – the trajectory of the training algorithm in a non-convex loss landscape – and the structure of the data . Hence , several recent works study the training of neural networks . For instance , Safran & Shamir ( 2018 ) address the role of overparametrization in avoiding bad local minima , and Zhang et al . ( 2016 ) empirically show that overparametrized networks trained by SGD can even perfectly fit to random labels . Within the popular framework of the Neural Tangent Kernel ( NTK ) ( Jacot et al. , 2018 ) , which uses a linear approximation of the network at initialization , several works analyze the optimization trajectory and show global convergence of ( S ) GD to a global optimum of the empirical loss ( AllenZhu et al. , 2019 ; Li & Liang , 2018 ; Zou et al. , 2018 ; Du et al. , 2018 ) . Extending the viewpoint to generalization , Arora et al . ( 2019a ; b ) exploit the kernel-like behaviour of two-layer networks close to their initialization to prove generalization for the final network , showing that two-layer neural networks generalize as well as Kernel Ridgeless Regression ( KRLR ) with the NTK . Cao & Gu ( 2019 ) show a tighter bound with a Neural Tangent Random Feature Model . The kernel approach , however , has two main limitations : First , while KRLR can generalize well in specific high dimensional regimes ( Liang et al. , 2020 ) , there is theoretical and empirical evidence that it can be inconsistent with noise ( Rakhlin & Zhai , 2019 ) . Is there an approach for analyzing neural networks that shows they perform at least as well as KRLR , but is also robust to noise ? Second , importantly , neural networks are known to outperform traditional statistical methods in many regimes as they are able to automate the process of feature extraction from data , as opposed to kernel methods that work with a fixed feature representation . This poses the question of other , adaptive , regimes beyond the linear network approximation . In this realm , Wu et al . ( 2018 ) show generalization bounds that , instead of the NTK norm , scale with respect to another functional norm . This norm corresponds to the minimum RKHS norm of the function among a family of kernels , i.e. , their method in a sense picks the best kernel in this family . However , this result ignores the computational aspect of the problem . Are there particular nonlinear regimes beyond NTK for which a gradient-type polynomial-time algorithm , in a way , adaptively chooses a suitable kernel ? Going beyond the NTK view , a line of work convexifies the optimization problem via an approximation of SGD dynamics with a continuous time gradient flow in the space of probability measures on the hidden units of the network , equipped with the Wasserstein metric ( Mei et al. , 2018 ; Chizat & Bach , 2018 ; Mei et al. , 2019 ; Wei et al. , 2018 ; Sirignano & Spiliopoulos , 2020 ; Javanmard et al. , 2019 ; Lu et al. , 2020 ) . Taking another perspective , Allen-Zhu et al . ( 2018 ) consider a three-layer network model that is not captured by the NTK approximation , and learn an underlying concept class by exploiting saddle-point escape theory for nonconvex SGD ( Ge et al. , 2015a ) . However , evaluating the complexity measure of Allen-Zhu et al . ( 2018 ) is rather involved , and only aligns well with functions that are described by a particular network . Whether one can recover the NTK bound ( e.g . the NTK norm ) from these results is not clear , while computing these generalization bounds from a given dataset is another question . For the NTK setting , in contrast , Arora et al . ( 2019a ) develop a purely data dependent generalization bound . Going to deeper than two layers , is it possible to prove a data-dependent complexity measure beyond the NTK regime that recovers the NTK result ( Arora et al. , 2019a ) as a special case ? In this work , we address the above questions : • We consider a regime for 3-layer neural networks that is not captured by the NTK approximation and show that , despite nonconvexity , a variant of projected SGD finds a good solution , as measured by the regularized empirical loss , importantly , after polynomially many iterations . • We introduce a new function norm ∥.∥ζ as the minimum RKHS norm with respect to a family of kernels K , which is upper bounded by the NTK norm up to constants . We show that for an arbitrary function f , the generalization gap of the trained network scales by ∥f∥ζ . This makes our generalization bound adaptive , in the sense that it scales with the best kernel inK . As a byproduct , our bounds are comparable with kernel regression bounds simultaneously with all kernels in K. We hope that our techniques motivate researchers to prove such adaptive generalization bounds for deeper networks , which can potentially result in stronger depth separation . • We show generalization bounds with a new data-dependent complexity measure that generalizes the NTK-based complexity measure in ( Arora et al. , 2019a ) . Up to logarithmic factors , our bounds are upper bounded by the NTK-based bounds in ( Arora et al. , 2019a ) and hence strictly improve over them ( if one substitutes their Lipschitz loss with a smooth one ) – see Appendix A.1 for a simple explicit example . Importantly , our bound can also handle noisy distributions as opposed to ( Arora et al. , 2019a ) . Further Related work . While the idea of a learning algorithm that combines multiple kernels has been employed for a while in the community ( Sonnenburg et al. , 2006 ; Rakotomamonjy et al. , 2007 ; Duan et al. , 2012 ) , our understanding of the connections between deep learning and multiple kernel learning is yet in its infancy . Recently , Dou & Liang ( 2020 ) define a time-varying kernel based on the network weights and show that the limit of the gradient flow converge to a suitable dynamic kernel , in the sense that the residual of the link function onto its RKHS could be in a smaller ranked space compared to the orthogonal complement of the RKHS . Ghorbani et al . ( 2019 ) analyze the difference between training a two layer ReLu network compared to its NTK or random feature simplifications , for a mixture of Gaussians input distribution and quadratic target functions . Ignoring the computational hardness imposed by nonconvexity , Bach ( 2017 ) prove a dimension dependent generalization bound beyond NTK . In another line of work , Chizat & Bach ( 2020 ) study gradient flow on losses with exponential tail and its relation to the max margin solution . Wei et al . ( 2019 ) show an interesting separation between the learning power of two layer ReLu nets and their NTK approximation , by showing a sample complexity gap for an artificially constructed distribution . With a different approach , Allen-Zhu & Li ( 2020 ) analyze multi-layer networks with quadratic activations , and prove generalization bounds polynomial in the dimension and precision by assuming an underlying teacher network , which shows a remarkable algorithmic depth separation . The problem of depth separation for neural networks and more generally their expressive power has been investigated by several researchers before ( Raghu et al. , 2017 ; Daniely , 2017 ; Barron , 1994 ; Funahashi , 1989 ; Safran & Shamir , 2016 ; Safran et al. , 2019 ) . The assumption of an underlying teacher network that one seeks to recover is common , too ( Li & Yuan , 2017 ; Zhong et al. , 2017 ; Brutzkus & Globerson , 2017 ) . Other works focus mainly on the algorithm and use other techniques , such as tensor factorization , to find a global optimum ( Tian , 2016 ; Bakshi et al. , 2019 ; Janzamin et al. , 2015 ; Zhong et al. , 2017 ) . Finally , many authors study the loss landscape under various assumptions ( Freeman & Bruna , 2016 ; Nguyen & Hein , 2017 ; Soudry & Carmon , 2016 ; Soltanolkotabi et al. , 2018 ; Ge et al. , 2017 ) , some of them consider the simplified case of deep linear networks ( Arora et al. , 2018a ; Saxe et al. , 2013 ; Bartlett et al. , 2018 ; Kawaguchi , 2016 ) . 2 SETUP AND APPROXIMATION BY KERNELS We analyze a 3-layer ReLu neural network from inputs x ∈ Rd to outputs y ∈ R , of the form fV ′ , W ′ ( x ) = 1√ m2 aTσ ( ( V ( 0 ) + V ′ ) W s 1√ m1 σ ( ( W ( 0 ) +W ′ ) x ) ) , ( 1 ) where a ∈ Rm2 is a vector of random signs , V ( 0 ) ∈ Rm2×m3 and W ( 0 ) ∈ Rm1×d are random weight initializations with i.i.d Gaussian entries V ( 0 ) j , k ∼ N ( 0 , κ22 ) , W ( 0 ) j , k ∼ N ( 0 , κ21 ) , and W s ∈ Rm3×m1 is a random sign matrix , which is roughly a random projection and change of coordinates into a lower dimensional space . We refer to W s 1√ m1 σ ( ( W ( 0 ) +W ′ ) x ) as the first layer and 1√ m2 aTσ ( ( V ( 0 ) + V ′ ) ( . ) ) as the second layer . The algorithm trains weight matrices V ′ , W ′ , and W s , a are fixed . We assume that the outputs are a.s. bounded by a constant , |yi| ≤ B , and ∥xi∥ = 1 . We consider the square loss denoted by ℓ ( . , . ) . We denote the training ( empirical ) loss of a function f on our dataset { xi } ni=1 , { yi } ni=1 , and the expected loss with respect to the data distribution ( population loss ) by Rn ( f ) = 1 n ∑n i=1 ℓ ( f ( xi ) , yi ) , and R ( f ) = Eℓ ( f ( x ) , y ) , respectively . Sometimes , we refer to the vector of labels ( yi ) ni=1 by y . We denote the space of functions with bounded RKHS-norm of kernel K byHK and write Õ to hide the log factors . 2.1 KERNEL APPROXIMATIONS , DECOMPOSITION AND ADAPTIVITY Kernel approximations of neural networks play an important role in our analysis . First , a common approximation is the NTK . The Neural Tangent kernel for a 2-layer ReLu network is H∞ ( x1 , x2 ) = ⟨x1 , x2⟩ · F2 ( ⟨x1 , x2⟩/ ( ∥x1∥∥x2∥ ) ) , for F2 ( x ) = 14 + arcsin ( x ) / ( 2π ) . ( 2 ) To introduce adaptivity , a key part of our analysis is to approximate the second layer in the 3-layer network by a product kernel K∞ ⊙ G that decomposes into a “ fixed ” part K∞ and an “ adaptive ” part G. To define these kernels , for every i ∈ [ n ] , let ϕ ( 0 ) ( xi ) be the output of the first layer of the network at initialization , ϕ ( 0 ) ( xi ) = 1√m1W sσ ( W ( 0 ) xi ) , and ϕ ( 0 ) ( xi ) + ϕ′ ( xi ) be that output for weights W ( 0 ) +W ′ . The adaptive kernel G captures the dot product between the learned weights : G ( xi , xj ) = ⟨ϕ′ ( xi ) , ϕ′ ( xj ) ⟩ . ( 3 ) This form of G motivates the complexity measure we define in the next section , if one thinks of the entries of ϕ′ as bounded NTK-norm functions of the input . Next , we consider the second layer , where the partK∞ arises from roughly stable activations . To formalize this stability , let Sgn ( V x ) be the diagonal matrix whose diagonal contains the coordinate-wise signs of vector V x . If we assume that Sgn ( ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ′ ( xi ) ) ) ≈ Sgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) – we prove a rigorous statement in Appendix A.12 – then fW ′ , V ′ ( xi ) = 1√ m2 aT ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ′ ( xi ) ) ( 4 ) ≈ 〈 V ( 0 ) + V ′ , 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) ( ϕ ( 0 ) ( xi ) + ϕ ′ ( xi ) ) T〉 . ( 5 ) Focusing on the adaptive part ϕ′ ( x ) of the first layer , we write〈 V ( 0 ) + V ′ , 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) ϕ′ ( xi ) T 〉 : = 〈 V ( 0 ) + V ′ , Υ ( xi ) 〉 ( 6 ) and can then view the second layer as a function in the RKHS of the product kernel ⟨Υ ( xi ) , Υ ( xj ) ⟩ = : K̃∞ ( xi , xj ) G ( xi , xj ) . This defines the new kernel K̃∞ , which we simplify into the kernel K∞ that is independent of initialization . To do so , we first observe that K̃∞ concentrates around Ew∼N ( 0 , κ22I ) 1 { w Tϕ ( 0 ) ( xi ) } 1 { wTϕ ( 0 ) ( xj ) } = F2 ( ⟨ϕ ( 0 ) ( xi ) , ϕ ( 0 ) ( xj ) ⟩ ∥ϕ ( 0 ) ( xi ) ∥∥ϕ ( 0 ) ( xj ) ∥ ) . Moreover , the Gaussian initialization and assumption ∥xi∥ = 1 imply that ⟨ϕ ( 0 ) ( xi ) , ϕ ( 0 ) ( xj ) ⟩ concentrates around m3F3 ( ⟨xi , xj⟩ ) , for F3 : [ −1 , 1 ] → [ 0 , 12 ] defined as F3 ( x ) = 1 2π √ 1− x2 + 14x+ 1 2πx arcsin ( x ) , ( 7 ) so K̃∞ ( xi , xj ) ≈ F2 ( 2F3 ( ⟨xi , xj⟩ ) ) = : K∞ ( xi , xj ) . ( 8 ) For general x1 , x2 not necessarily unit norm , we define K∞ ( x1 , x2 ) = K∞ ( x1/∥x1∥ , x2/∥x2∥ ) . It is easy to check that the coefficients in the Taylor series of F2 and F3 are nonnegative . Combining this with Schur ’ s Product Theorem implies K∞ is PSD ( Appendix A.4 ) . We also denote the data kernel matrix on ( xi ) ni=1 by K ∞ and H∞ . As Arora et al . ( 2019a ) , we assume the data distribution is ( λ0 , δ , n ) -non-degenerate with respect to H∞ and K∞ , i.e. , with probability at least 1 − δ , the smallest eigenvalues of H∞ and K∞ are at least λ0 > 0 . 3 DATA DEPENDENT COMPLEXITY MEASURE AND GENERALIZATION The emergence of G ⊙ K∞ above gives rise to an adaptive kernel-like complexity measure that will determine generalization bounds . Intuitively , this complexity measure reflects the two layers . Here , we viewG as the Gram matrix of some “ ideal ” first-layer feature functions gk . We measure the complexity of the prediction function via the RKHS ofA : = G⊙K∞ , and allow a flexible choice of the features gk . Here , gk ’ s substitute the entries of ϕ′ in Equation ( 3 ) : G ( xi , xj ) : = ∑ gk ( xi ) gk ( xj ) . To regularize this choice , we penalize the complexity of the features g via the NTK norm , so we seek gk ’ s with the minimum possible norms . Alternatively , the features gk are flexible but have bounded NTK norm . For a labeling f∗ ∈ Rn of the n data points and fixed G , this leads to the complexity ζ ( f∗ , G ) = f∗TA−1f∗ · ⟨H∞−1 , G⟩ = f∗TA−1f∗ ∑m3 k=1 ∥gk∥2H∞ ; A = G⊙K∞ , ( 9 ) where m3 is the number of intermediate features ( The choice of m3 is discussed in Appendix A.13 ) . Our data-dependent complexity measure implicitly selects the G ( or equivalently the feature vectors gk ) that leads to the tightest bound , trading off data fit and function complexity : ℑ = ℑ ( ( xi ) ni=1 , ( yi ) ni=1 ) : = min f∗∈Rn { 2nRn ( f ∗ ) +ϖmin G≻0 ζ ( f∗ , G ) } , ( 10 ) where we use a log factor ϖ = O ( log ( n ) 3 + log ( 1/λ0 ) ) . To make the relation to adaptive kernel spaces even more explicit , assume that the gk are bounded as ∑ k ∥gk∥2H∞ ≤ 1 . Then we define a family K of corresponding kernels of the form K { g } ( x1 , x2 ) = K ∞ ( x1 , x2 ) ( ∑ g∈ { g } g ( x1 ) g ( x2 ) ) , ( 11 ) i.e. , K : = { K { g } | { g } finite , ∑ g∈ { g } ∥g∥2H∞ ≤ 1 } . With this notation , the complexity measure is ℑ ( ( xi ) , ( yi ) ) = min f∗∈Rn { 2nRn ( f ∗ ) +ϖ min K∈K f∗TK−1f∗ } . ( 12 ) Hence , this measure may be understood as searching for the most efficient and effective feature representation within a family of RKHSs . We may also relate this complexity measure to the NTK-based complexity measure yTH∞−1y ( Arora et al. , 2019a ) . For any labeling f∗ , let f̃∗ ∈ HH∞ be the function with minimum NTK norm that maps xi ’ s to f∗i ’ s , so ∥f̃∗∥H∞ = f∗ TH∞−1f∗ . If we set { g } = { f̃∗/∥f̃∗∥ } , then one can show ( Appendix A.5 ) f∗TK−1 { f̃∗/∥f̃∗∥ } f ∗ ≤ 4f∗T ( f∗f∗T ) −1f∗ × ∥f̃∗∥2 = 4∥f̃∗∥2 = 4f∗TH∞−1f∗ , ( 13 ) which implies ℑ ≤ min f∗∈Rn { 2nRn ( f ∗ ) + ( 4ϖ ) f∗TH∞−1f∗ } . ( 14 ) One can further set f∗ = y above and obtain ℑ ≤ ( 4ϖ1 ) yTH∞−1y . ( 15 ) 3.1 GENERALIZATION With the complexity ℑ in hand , we can now state our generalization result . It assumes optimization by a Projected Stochastic Gradient Descent ( PSGD ) , described in detail in Section 4 . Theorem 1 Suppose we run Projected Stochastic Gradient Descent ( PSGD ) on the regularized empirical risk with parameters as in Section 4 . Then , with high probability ( e.g . 0.99 ) over the randomness of data , initialization and noise of the gradient steps , PSGD converges in poly ( B ∨ 1/B , 1/λ0 , n ) iterations to a solution ( WPSGD , VPSGD ) with population risk bounded as ( recall |yi| ≤ B a.s. ) R ( fWPSGD , VPSGD ) ≤ ℑ ( ( xi ) ni=1 , ( yi ) ni=1 ) n + B2ϖ n . ( 16 ) As a side remark , the factor 2 in front of Rn ( f∗ ) in the definition of ℑ in ( 12 ) is not special and a similar generalization bound can be obtained for any γ > 1 . Substituting the upper bound on the complexity in Equation ( 15 ) , one recovers an NTK-based generalization bound that scales with yTH∞−1y/n up to log factors , which is roughly the square of the generalization bound presented in ( Arora et al. , 2019a ) . The reason for the faster squared rate here is that we are considering smooth losses , while Arora et al . ( 2019a ) work with a bounded Lipschitz loss . Indeed , it is not hard to apply a more rigorous uniform convergence analysis from ( Srebro et al. , 2010 ) to also obtain a faster squared rate for the approach used in ( Arora et al. , 2019a ) . Since Equation ( 15 ) is an upper bound on our complexity , our result generalizes and tightens the NTK bound ( Arora et al. , 2019a ) . To illustrate the flexibility of our complexity measure , we show in Appendix A.1 a simple explicit example of functions represented as polynomial series where our bound improves upon the NTK bound ( Arora et al. , 2019a ) . Notably , we only substitute low-rank matrices G in our complexity measure for this construction . We leave further investigation of our complexity measure for arbitrary G ’ s to future work . 3.2 UNDERLYING CONCEPT CLASS Instead of data dependent generalization bounds , one may study the generalization gap with respect to some concept class . The complexity measure ℑ implicitly uses the following adaptive norm on the space of functions from Rd , the infimum of the RKHS norms for the family of kernels K : ∥f∥ζ = inf K { g } ∈K ∥f∥K { g } . ( 17 ) It is not hard to check that ∥.∥ζ is in fact a norm , and that the inf is achieved by a particular set { g } . Similar to the derivation of the upper bound on the complexity measure in Equation ( 15 ) , by setting { g } = { f∗/∥f∗∥H∞ } , we obtain the following NTK upper bound : ∥f∥ζ ≤ 4∥f∥H∞ . ( 18 ) This leads to a function-dependent generalization bound which bounds the risk of the learned network against an arbitrary function f with ∥f∥ζ < ∞ . Theorem 2 For any measurable function f : Rd → R , in the same setting as Theorem 1 , the population risk of the trained network can be bounded as R ( fWPSGD , VPSGD ) ≤ 2R ( f ) +O ( ϖ ∥f∥2ζ+B 2 n ) . ( 19 ) As in the data-dependent case , the factor 2 of the risk of f can be reduced to any constant γ > 1 . 3.3 INTERACTION OF LAYERS BEYOND THE LINEAR APPROXIMATION Here , we give a high level intuition on how the adaptivity is achieved in our regime compared to NTK . In the NTK approach , for every input x , the neural net fW , V ( x ) is approximated by its linear approximation at ( W ( 0 ) , V ( 0 ) ) ( the initialized network ) , fW , V ( x ) = ⟨∇W , V fW ( 0 ) , V ( 0 ) ( x ) , ( W − W ( 0 ) , V − V ( 0 ) ) ⟩ . The NTK approximation works as long as ( W , V ) are close enough to their initialization that the linear approximation remains accurate and the interaction of weights between layers is negligible . Specifically , the features ϕ′ ( xi ) behave almost linearly with respect to W − W ( 0 ) as ∥W − W ( 0 ) ∥ is taken to be small and the sign pattern Sgn ( ( W ( 0 ) + W ′ ) xi ) is proven not to change much compared to Sgn ( W ( 0 ) xi ) . Additionally , the NTK-type analysis needs the following two conditions to be satisfied : ( 1 ) the sign pattern of ϕ ( 0 ) ( xi ) + ϕ′ ( xi ) with respect to V ( 0 ) + V ′ remains almost the same as the sign pattern of ϕ ( 0 ) ( x ) with respect to V ( 0 ) , and ( 2 ) the weight changes W ′ and V ′ should not interact , which means the “ interaction ” term , 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) V ′ϕ′ ( xi ) ≈ 0 , should be negligible . Therefore , the non-negligible terms for the NTK are : ( 1 ) 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) V ( 0 ) ϕ′ ( xi ) , which is almost linear in W ′ ( recall that ϕ′ ( xi ) depends on W ′ ) , and ( 2 ) 1√m2 a TSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) V ′ϕ ( 0 ) ( xi ) which is linear in V ′ . This approach has two important implications : ( 1 ) it convexifies the optimization ( for convex loss ) , as the approximation is now linear in W ; and ( 2 ) it simplifies proving generalization , as it works with the class of functions in the RKHS space of some fixed kernel . However , this simplification leaves no room for the ability of the neural network to learn intermediate feature representations which could result in better function classes . In our regime , however , we enforce the condition ∀j , i : V ′j ⊥ ϕ ( 0 ) ( xi ) ( ⋆ ) , which implies the second ( 2 ) above is zero , while the interaction term is not negligible any more and the network behaves similar to a quadratic function with respect to ( W ′ , V ′ ) ( for fixed xi ) . Condition ( ⋆ ) is critical both in proving the convergence of the algorithm as well as bounding the Rademacher complexity of the class of networks with bounded weights . Rather than working with a fixed kernel , the interaction term enables us to use the first layer for representing the input in a suitable feature space , which can be interpreted as picking a suitable kernel , then use the second layer to describe the output based on those features . This is also indirectly encoded in our complexity measure.In addition to enforcing the orthogonality condition ( ⋆ ) ( in the SGD variant ) , conditions for entering our regime are that the overparameterization m1 , m2 , m3 and κ1 , κ2 are within a specific range with respect to each other . We listed these relations in Appendix A.3 . To illustrate the benefit of going to this more involved regime , denote the class of neural nets with Frobenius bounded norms ∥W ′∥ ≤ γ1 , ∥V ′∥ ≤ γ2 by Gγ1 , γ2 ( and a bit more structure which we elaborate upon in the proofs ) ; it turns out that Gγ1 , γ2 roughly includes HK ( O ( γ1γ2 ) ) for every kernel K ∈ K , in the sense that each f ∈ HK ( O ( γ1γ2 ) ) is well-approximated within Gγ1 , γ2 to arbitrarily small error on fixed input ( the error goes down with the size of the network ) . On the other hand , we show that the Rademacher Complexity ( RC ) of Gγ1 , γ2 behaves similar to the RC of the NTK class HH∞ ( O ( γ1γ2 ) ) ! As our algorithm guarantees finding a network with sufficiently small empirical risk within Gγ1 , γ2 , this phenomenon underlies our adaptive generalization bounds . Compared to previous work that provides an adaptive kernel analysis still for a two layer model ( Dou & Liang , 2020 ) ( although their analysis is for the gradient flow and non-algorithmic ) , our model requires an additional layer so it can , in a sense , “ simulate ” the process of feature extraction in one layer to be used in the next layer . 3.4 COMPARISON WITH KERNEL FITTING We compare our generalization bounds with some kernel fitting rates . Given a kernel K with K ( x , x ) ≤ 1 for every x : ∥x∥ ≤ 1 , suppose we want to fit a function fromHK ( B′ ) , i.e . having KRKHS norm bounded by B′ . In the realizable setting , when there is an underlying f∗∗ ∈ HK ( B′ ) with zero risk , Empirical Risk Minimization ( ERM ) enjoys a fast rate using the smoothness of the loss ( Srebro et al. , 2010 ) . The Rademacher Complexity bound R ( HK ( B′ ) ) ≤ O ( B ′ √ n ) then implies : R ( fERM ) ≤ Õ ( B′2/n ) ( 20 ) for the square loss , which is minimax optimal up to log factors . To compare the neural network , we substitute f∗∗ into Theorem 2 . To relate the B in our bound to B′ , assume for simplicity of exposition that ∥f∥ζ = ∥f∥K∗ for some K∗ ∈ K ( otherwise we can use a convergent sequence ) . Observing that K { g } ( x , x ) ≤ 1 for every kernel K { g } ∈ K , we obtain that |f∗∗ ( x ) | ≤ ∥f∗∗∥K∗ = ∥f∗∗∥ζ ( Appendix A.6 ) . Combining this fact with the realizability assumption , we can then upper bound the parameter B in Theorem 2 by ∥f∗∗∥ζ , and obtain R ( fWPSGD , VPSGD ) = Õ ( ∥f∗∗∥ζ 2 /n ) . ( 21 ) If we further takeK to be inK , then Equation ( 21 ) combined with ∥f∗∗∥ζ ≤ ∥f∗∗∥K ≤ B′ implies : R ( fWPSGD , VPSGD ) = Õ ( B ′2/n ) , ( 22 ) that is , for every kernel K ∈ K , our deep learning approach almost achieves the conventional kernel bound in Equation ( 20 ) . Repeating the uniform risk bound stated in Theorem 1 in ( Srebro et al. , 2010 ) for HK ( B′ ) where B′ is set to all powers of two , followed by a union bound , one can easily obtain a fast rate of R ( fKRLR ) ≤ Õ ( y TK−1y n + B2 n ) , ( 23 ) for the solution of KRLR in the general case ( not realizable ) for the square loss . On the other hand , for a B-bounded Lipschitz loss , we instead get a slow rate for KRLR : R ( fKRLR ) ≤ Õ ( √ yTK−1y n + B√ n ) , whereB is an a.s. bound on |y| as before . This bound is similar to Arora et al . ( 2019a ) . Note that our data dependent generalization bound in Theorem 1 already achieves the fast rate for KRLR in ( 23 ) for any K ∈ K. Finally , in the non-realizable case , we still have the following fast rate for ERM regarding the hypothesis classHK ( B′ ) ( Srebro et al. , 2010 ) : R ( fERM ) ≤ Õ ( R ( f∗∗ ) + B ′2+B2 n ) , where now f∗∗ : = argminf∈HK ( B′ ) R ( f ) , while Theorem 2 also implies ( again for every K ∈ K ) : R ( fWPSGD , VPSGD ) ≤ Õ ( R ( f∗∗ ) + ∥f∗∗∥2ζ+B 2 n ) = Õ ( R ( f ∗∗ ) + B ′2+B2 n ) . 4 ALGORITHM : PROJECTED STOCHASTIC GRADIENT DESCENT Next , we discuss the algorithm that achieves the generalization bounds above . Two specific challenges are to address the interaction of layers , and to establish that the algorithm remains in a regime in which it converges to a good solution that generalizes . In this section , we describe our algorithm PSGD , presented as pseudocode in Figure 1 , which is roughly Stochastic Gradient Descent modified to project out a low-dimensional random subspace from the second-layer weights.PSGD roughly runs SGD on a smoothed version of the following loss function ( ψ1 , ψ2 are defined in Appendix A.2 ) L1 ( W ′ , V ′ ) = Rn ( fW ′ , V ′ ) + ψ1∥W ′∥2 + ψ2∥V ′∥2 . Compared to standard SGD , our algorithm makes two modifications : ( 1 ) it uses randomized smoothing to alleviate the non-smoothness of the ReLus , ( 2 ) it ensures that the weights in the second layer are orthogonal to the data features ϕ ( 0 ) ( x ) computed by the first layer at initialization . This helps to control layer interactions as pointed out in Section 3.3 . For smoothing , we add Gaussian smoothing matrices W ρ and V ρ to the weights with i.i.d . entries from N ( 0 , β21/m1 ) and N ( 0 , β22/m2 ) respectively , for β2 = Op ( ( κ1 √ m1 ) −1 ( κ2 √ m2 ) −2/3 ) , β1 = Op ( m23κ2 √ m2 ( κ1 √ m1 ) −1 ) . To simplify the exposition , Op ( . ) is hiding the dependencies on the basic parameters B , n , 1/λ0 and log factors . Our convergence proof uses the loss with respect to this smoothed network . For the projection , let Φ⊥ ⊂ Rm2×m3 be the subspace of weights of the second layer whose rows are orthogonal to the first-layer data representations ϕ ( 0 ) ( xi ) ’ s ∀i ∈ [ n ] at initialization : V ′ ∈ Φ⊥ ↔ ∀j ∈ [ m2 ] , ∀i ∈ [ n ] : V ′j , ϕ ( 0 ) ( xi ) = 0 . ( 24 ) In summary , at point ( W ′ , V ′ ) , the algorithm samples a random ( xi , yi ) from the data , as well as smoothing matrices W ρ,1 , V ρ,1 , W ρ,2 , V ρ,2 . It then computes an unbiased estimate for the gradient ( ∇̂W , ∇̂V ) , adds additional normalized Gaussian noise matrices Ξ1 , Ξ2 and moves in this direction with step size η = 1/poly ( n , B ∨ 1/B , 1/λ0 ) : ( W ′ , V ′ ) ← ( W ′ , V ′ ) + η ( ∇̂W + Ξ1/ ( √ m1∥Ξ1∥ ) , ProjΦ⊥ ( ∇̂V + Ξ2/∥Ξ2∥ ) ) . ( 25 ) Parameters . Our results apply to the overparametrization regime , when the size of the network , i.e . parameters m1 , m2 , m3 is polynomially large in n , B ∨ 1/B , 1/λ0 . This ensures that we can guarantee that the network has good enough function representation capacity , and PSGD is able to find a good local direction in every iteration . The regularization coefficients ψ1 , ψ2 can be set with respect to any candidate ( f∗ , G ) for our complexity measure ( 9 ) . In Appendix A.2 , we introduce a simple doubling trick which handles when we do not have access to an optimal candidate solution . With such an f∗ , define ν : = max { Rn ( f∗ ) /2 , B2/n } , and set ψ1 = ν/4 , ψ2 = ν/ ( 4ζ ( f∗ , G ) ) . We list the suitable regime for our overparameterization in Appendix A.3 . Algorithm 1 PSGD ( Projected Stochastic Gradient Descent ) Input : network architecture m1 , m2 , m3 , initialization parameters κ1 , κ2 , smoothing parameters β1 , β2 , training set ( xi , yi ) ni=1 , label parameter B , ( f ∗ , G ) from the complexity measure 1 : Gaussian initialization W ( 0 ) j , k ← N ( 0 , κ1 ) , V ( 0 ) j , k ← N ( 0 , κ2 ) 2 : Define parameters ψ1 , ψ2 , ν , η , subspace Φ⊥ , and objective L1 as described in Section 4 3 : while L1 ( W ′ , V ′ ) > Rn ( f∗ ) + 2ν do 4 : Gaussian matrices W ρ,1j , k , W ρ,2 j , k ← N ( 0 , β21 m1 ) , V ρ,1j , k , V ρ,2 j , k ← N ( 0 , β22 m2 ) 5 : Sample data ( xi , yi ) uniformly at random 6 : Compute gradient estimates 7 : { ∇̂W = ℓ̇ ( fW ′+Wρ,1 , V ′+V ρ,1 ( xi ) , yi ) ∇W fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) + 2ψ1W ′ , ∇̂V = ℓ̇ ( fW ′+Wρ,1 , V ′+V ρ,1 ( xi ) , yi ) ∇V fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) + 2ψ2V ′ 8 : Move as ( W ′ , V ′ ) ← ( W ′ , V ′ ) + η ( ∇̂W +Ξ1/ ( √ m1∥Ξ1∥ ) , ProjΦ⊥ ( ∇̂V +Ξ2/∥Ξ2∥ ) ) 9 : Return ( W ′ , V ′ ) 5 HIGH LEVEL IDEA OF THE PSGD ANALYSIS Aside from the network ’ s representation ability , a major challenge is the poly-time computation of some good weights that generalize . On the one hand , we want the weights to remain close to their initialization so the final network is in the class Gγ1 , γ2 for suitably chosen γ1 , γ2 , where recall that Gγ1 , γ2 is roughly the class of nets with bounded norms ∥W ′∥ ≤ γ1 , ∥V ′∥ ≤ γ2 . On the other hand , we aim to reduce the nonconvex empirical loss Rn ( fW ′ , V ′ ) . Aligned with these goals , we roughly run SGD on our smoothed objective consisting of the empirical risk regularized by the Frobenius norm of the weight matrices . We prove convergence by building on ideas from Allen-Zhu et al . ( 2018 ) , with a framework based on the classic result that SGD can escape saddle points in face of nonconvexity . Compared to ( AllenZhu et al. , 2018 ) , we take a different approach driven by our purely data-dependent complexity measure in Theorem 1 . We augment this by a careful Rademacher complexity analysis in Appendix A.11 for Frobenius norm bounded networks in our regime , to precisely get the same behavior as the RC bound for the class of NTK-norm bounded functions ( Arora et al. , 2019a ) . We face two challenges in the proof : first , to have an SGD-type algorithm , we can not directly guide the optimization trajectory based on our complexity measure , so we have little control on the random trajectory of PSGD . Therefore , we mainly rely on uniform convergence techniques in our convergence proof . In particular , we prove it is unlikely that PSGD reaches a “ bad '' saddle point if the objective is above some threshold . Second , to prove the last fact , we require a weak regularity condition that the value of the objective does not exceed a certain threshold during the algorithm . This puts us in a different setting compared to the traditional result on escaping saddle points ( Ge et al. , 2015a ) , so we need to refine their analysis for our setting . We give a high-level sketch of our proof in Section 5 . Below , we outline the steps of the convergence proof at a high level . Construction of a good Network To study the loss landscape , similar to ( Allen-Zhu et al. , 2018 ) , we show the existence of a good local update for PSGD at reasonable points ( W ′ , V ′ ) , using the ideal weight matrices ( W ∗ , V ∗ ) that we carefully construct from our complexity measure . Here , we sketch our proof for constructing ( W ∗ , V ∗ ) . Let ( W ′ , V ′ ) be the current weights of the algorithm . Fix a sample i ∈ [ n ] . In Appendix A.12 , we use G to construct W ∗ for the first layer weights with decomposition W ∗ = ∑m3 k=1W ∗ k and O ( 1 ) bounded norm , such that ϕ∗ ( xi ) k : = 1√ m1 W sSgn ( ( W ( 0 ) + W ′ ) xi ) W ∗xi . This decomposition ensures that for every k , k′ ∈ [ m3 ] , negating the matrix W ∗k only negates ϕ∗ ( xi ) k′ when k′ = k and has no effect on ϕ∗ ( xi ) k′ for k′ ̸= k. This way , we can easily generate any arbitrary sign flip of the entries of ϕ∗ ( xi ) . We use this property to generate a suitable random descent direction . Next , we want to construct a suitable weight matrix V ∗ for the second layer which maps the features ϕ∗ ( xi ) into f∗i ( recall the definition of the complexity measure ) . The key here is that we consider a regime where the norm of ϕ ( 0 ) ( xi ) is typically larger than that of ϕ′ ( xi ) and ϕ∗ ( xi ) , so it is very likely that the sign pattern in the second layer is determined by ϕ ( 0 ) ( xi ) in most of the rows . In such a senario , having the condition V ′j ⊥ ϕ ( 0 ) ( xi ) becomes vital as the interaction of V ′ with ϕ ( 0 ) ( xi ) becomes problematic both for the generalization and optimization . From the standpoint of generalization , without excluding this interaction , one can exploit the large size of ϕ ( 0 ) ( xi ) and build a network within the class Gγ1 , γ2 ( i.e. , with ∥W ′∥ , ∥V ′∥ suitably bounded ) corresponding to a complex function which overfits the data . Indeed , we utilize the large magnitude of ϕ ( 0 ) and its orthogonality to the rows of V ′ in our RC bound . On the other hand , mainly because changing the weights of the first layer can not affect ϕ ( 0 ) ( xi ) , the interaction of V ′ with ϕ ( 0 ) ( xi ) is problematic in our convergence analysis for optimization part , particularly in proving the existence of a local descent direction . This is the rationale behind our orthogonality constraint ( 24 ) . Finally , the ϕ∗ ( xi ) ’ s , the above control on the signs , and the fact that ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ concentrates around m3Ew∼N ( 0 , κ1I ) [ σ ( wTxi1 ) σ ( wTxi2 ) ] which recovers the structure of the kernel K∞ ( Section 2 ) , give rise to the kernelG⊙K∞ in the second layer . Using this structure , we construct V ∗ that maps ϕ∗ ( xi ) ’ s to f∗i ’ s , which has additional good properties , includingO ( f ∗T ( G⊙K∞ ) −1f∗ ) bounded norm , and rows that are orthogonal to ϕ ( 0 ) ( xi ) ’ s . For more details , see Appendix A.12 . Nonexistence of Bad Saddle Points Next , we want to exploit ( W ∗ , V ∗ ) to prove the existence of a good direction along which reduces the objective locally . Moving along ( W ∗ , V ∗ ) is the first idea which fails as the cross terms created between W ′ , V ∗ and V ′ , W ∗ can not be bounded effectively . Instead , we randomly perturb W ∗ and V ∗ in a coupled way and prove a reduction in expectation . We elaborate more on this suitable random direction . Multiplying random signs Σk ontoW ∗k , we define the sumW ∗Σ = ∑m3 k=1 ΣkW ∗ k . We also multiply the same signs to the columns of V ∗ and project it back onto Φ⊥ to obtain V ∗Σ . Then , we move in the random direction ( √ ηW ∗Σ − ηW/2 , √ ηV ∗Σ − ηV/2 ) ; this update creates additional cross terms in the objective that we must bound to prove a local reduction argument . A key point here is that we prove that with high probability , the norm of the weights remains bounded along the algorithm , i.e . ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 . This norm restriction enables us to substitute terms that we do not have control over by their worst-case supremum counterparts . We refer to Appendix A.13 for similar techniques . Convergence of PSGD Finally , we use the fact that SGD escapes good saddle points ( Ge et al. , 2015b ) . For proving the existence of a good random direction to escape saddle points above , we use that the norm of weights is uniformly bounded along all iterations ; this bound , in fact , is looser than the bound that we show for the final weights of the network . Yet , this additional restriction can not be addressed by the classical nonconvex theory of SGD . Consequently , we refine and adapt the proof of ( Ge et al. , 2015b ) to incorporate this additional constraint . At a high level , Ge et al . ( 2015b ) work with a supermartingale based on the loss value . To guarantee the additional norm restriction , it is initially tempting to apply Azuma-Hoeffding concentration to bound the upward deviations of this process . However , this fails as the process has a two-fold behavior , depending on how large the gradient is . At the core of our refinement proof here , we instead directly bound the MGF of the martingale using Doobs maximal inequality . We refer to Section A.16 for more details . REFERENCES Zeyuan Allen-Zhu and Yuanzhi Li . Backward feature correction : How deep learning performs deep learning . arXiv preprint arXiv:2001.04413 , 2020 . Zeyuan Allen-Zhu , Yuanzhi Li , and Yingyu Liang . Learning and generalization in overparameterized neural networks , going beyond two layers . arXiv preprint arXiv:1811.04918 , 2018 . Zeyuan Allen-Zhu , Yuanzhi Li , and Zhao Song . A convergence theory for deep learning via overparameterization . In International Conference on Machine Learning , pp . 242–252 . PMLR , 2019 . Sanjeev Arora , Nadav Cohen , Noah Golowich , and Wei Hu . A convergence analysis of gradient descent for deep linear neural networks . arXiv preprint arXiv:1810.02281 , 2018a . Sanjeev Arora , Rong Ge , Behnam Neyshabur , and Yi Zhang . Stronger generalization bounds for deep nets via a compression approach . In International Conference on Machine Learning , pp . 254–263 . PMLR , 2018b . Sanjeev Arora , Simon Du , Wei Hu , Zhiyuan Li , and Ruosong Wang . Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks . In International Conference on Machine Learning , pp . 322–332 . PMLR , 2019a . Sanjeev Arora , Simon S Du , Wei Hu , Zhiyuan Li , Ruslan Salakhutdinov , and Ruosong Wang . On exact computation with an infinitely wide neural net . arXiv preprint arXiv:1904.11955 , 2019b . Francis Bach . Breaking the curse of dimensionality with convex neural networks . The Journal of Machine Learning Research , 18 ( 1 ) :629–681 , 2017 . Ainesh Bakshi , Rajesh Jayaram , and David P Woodruff . Learning two layer rectified neural networks in polynomial time . In Conference on Learning Theory , pp . 195–268 . PMLR , 2019 . Andrew R Barron . Approximation and estimation bounds for artificial neural networks . Machine learning , 14 ( 1 ) :115–133 , 1994 . Peter Bartlett , Dylan J Foster , and Matus Telgarsky . Spectrally-normalized margin bounds for neural networks . arXiv preprint arXiv:1706.08498 , 2017 . Peter Bartlett , Dave Helmbold , and Philip Long . Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks . In International conference on machine learning , pp . 521–530 . PMLR , 2018 . Alon Brutzkus and Amir Globerson . Globally optimal gradient descent for a convnet with gaussian inputs . In International conference on machine learning , pp . 605–614 . PMLR , 2017 . Yuan Cao and Quanquan Gu . Generalization bounds of stochastic gradient descent for wide and deep neural networks . In Advances in Neural Information Processing Systems ( NeurIPS ) , 2019 . Lenaic Chizat and Francis Bach . On the global convergence of gradient descent for overparameterized models using optimal transport . arXiv preprint arXiv:1805.09545 , 2018 . Lenaic Chizat and Francis Bach . Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss . In Conference on Learning Theory , pp . 1305–1338 . PMLR , 2020 . Amit Daniely . Depth separation for neural networks . In Conference on Learning Theory , pp . 690– 696 . PMLR , 2017 . Xialiang Dou and Tengyuan Liang . Training neural networks as learning data-adaptive kernels : Provable representation and approximation benefits . Journal of the American Statistical Association , pp . 1–14 , 2020 . Simon S Du , Xiyu Zhai , Barnabas Poczos , and Aarti Singh . Gradient descent provably optimizes over-parameterized neural networks . arXiv preprint arXiv:1810.02054 , 2018 . Lixin Duan , Ivor W Tsang , and Dong Xu . Domain transfer multiple kernel learning . IEEE Transactions on Pattern Analysis and Machine Intelligence , 34 ( 3 ) :465–479 , 2012 . Gintare Karolina Dziugaite and Daniel M Roy . Computing nonvacuous generalization bounds for deep ( stochastic ) neural networks with many more parameters than training data . arXiv preprint arXiv:1703.11008 , 2017 . C Daniel Freeman and Joan Bruna . Topology and geometry of half-rectified network optimization . arXiv preprint arXiv:1611.01540 , 2016 . Ken-Ichi Funahashi . On the approximate realization of continuous mappings by neural networks . Neural networks , 2 ( 3 ) :183–192 , 1989 . Rong Ge , Furong Huang , Chi Jin , and Yang Yuan . Escaping from saddle pointsonline stochastic gradient for tensor decomposition . In Conference on learning theory , pp . 797–842 . PMLR , 2015a . Rong Ge , Furong Huang , Chi Jin , and Yang Yuan . Escaping from saddle points — online stochastic gradient for tensor decomposition . In Peter Grünwald , Elad Hazan , and Satyen Kale ( eds . ) , Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pp . 797–842 , Paris , France , 03–06 Jul 2015b . PMLR . URL http : //proceedings.mlr.press/v40/Ge15.html . Rong Ge , Jason D Lee , and Tengyu Ma . Learning one-hidden-layer neural networks with landscape design . arXiv preprint arXiv:1711.00501 , 2017 . Behrooz Ghorbani , Song Mei , Theodor Misiakiewicz , and Andrea Montanari . Limitations of lazy training of two-layers neural networks . arXiv preprint arXiv:1906.08899 , 2019 . Noah Golowich , Alexander Rakhlin , and Ohad Shamir . Size-independent sample complexity of neural networks . In Conference On Learning Theory , pp . 297–299 . PMLR , 2018 . Geoffrey Hinton , Li Deng , Dong Yu , George E Dahl , Abdel-rahman Mohamed , Navdeep Jaitly , Andrew Senior , Vincent Vanhoucke , Patrick Nguyen , Tara N Sainath , et al . Deep neural networks for acoustic modeling in speech recognition : The shared views of four research groups . IEEE Signal processing magazine , 29 ( 6 ) :82–97 , 2012 . Arthur Jacot , Franck Gabriel , and Clément Hongler . Neural tangent kernel : Convergence and generalization in neural networks . arXiv preprint arXiv:1806.07572 , 2018 . Majid Janzamin , Hanie Sedghi , and Anima Anandkumar . Beating the perils of non-convexity : Guaranteed training of neural networks using tensor methods . arXiv preprint arXiv:1506.08473 , 2015 . A. Javanmard , M. Mondelli , and A. Montanari . Analysis of a two-layer neural network via displacement convexity . ArXiv , abs/1901.01375 , 2019 . Kenji Kawaguchi . Deep learning without poor local minima . In nips , 2016 . Pitas Konstantinos , Mike Davies , and Pierre Vandergheynst . Pac-bayesian margin bounds for convolutional neural networks-technical report . arXiv preprint arXiv:1801.00171 , 2017 . Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . Imagenet classification with deep convolutional neural networks . Advances in neural information processing systems , 25:1097–1105 , 2012 . Yuanzhi Li and Yingyu Liang . Learning overparameterized neural networks via stochastic gradient descent on structured data . arXiv preprint arXiv:1808.01204 , 2018 . Yuanzhi Li and Yang Yuan . Convergence analysis of two-layer neural networks with relu activation . arXiv preprint arXiv:1705.09886 , 2017 . Tengyuan Liang , Alexander Rakhlin , et al . Just interpolate : Kernel ridgeless regression can generalize . Annals of Statistics , 48 ( 3 ) :1329–1347 , 2020 . Yiping Lu , Chao Ma , Yulong Lu , Jianfeng Lu , , and Lexing Ying . A mean-field analysis of deep resnet and beyond : Towards provable optimization via overparameterization from depth . In icml , 2020 . Song Mei , Andrea Montanari , and Phan-Minh Nguyen . A mean field view of the landscape of twolayer neural networks . Proceedings of the National Academy of Sciences , 115 ( 33 ) : E7665–E7671 , 2018 . Song Mei , Theodor Misiakiewicz , and Andrea Montanari . Mean-field theory of two-layers neural networks : dimension-free bounds and kernel limit . In Conference on Learning Theory , pp . 2388– 2464 . PMLR , 2019 . Behnam Neyshabur , Ryota Tomioka , and Nathan Srebro . Norm-based capacity control in neural networks . In Conference on Learning Theory , pp . 1376–1401 . PMLR , 2015 . Behnam Neyshabur , Srinadh Bhojanapalli , and Nathan Srebro . A pac-bayesian approach to spectrally-normalized margin bounds for neural networks . In International Conference on Learning Representations , 2018 . Quynh Nguyen and Matthias Hein . The loss surface of deep and wide neural networks . In International conference on machine learning , pp . 2603–2612 . PMLR , 2017 . Maithra Raghu , Ben Poole , Jon Kleinberg , Surya Ganguli , and Jascha Sohl-Dickstein . On the expressive power of deep neural networks . In Doina Precup and Yee Whye Teh ( eds . ) , Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp . 2847–2854 . PMLR , 06–11 Aug 2017 . URL http : //proceedings.mlr.press/v70/raghu17a.html . Alexander Rakhlin and Xiyu Zhai . Consistency of interpolation with laplace kernels is a highdimensional phenomenon . In Conference on Learning Theory , pp . 2595–2623 . PMLR , 2019 . Alain Rakotomamonjy , Francis Bach , Stéphane Canu , and Yves Grandvalet . More efficiency in multiple kernel learning . In Proceedings of the 24th international conference on Machine learning , pp . 775–782 , 2007 . Itay Safran and Ohad Shamir . Depth separation in relu networks for approximating smooth nonlinear functions . arXiv preprint arXiv:1610.09887 , 14 , 2016 . Itay Safran and Ohad Shamir . Spurious local minima are common in two-layer relu neural networks . In International Conference on Machine Learning , pp . 4433–4441 . PMLR , 2018 . Itay Safran , Ronen Eldan , and Ohad Shamir . Depth separations in neural networks : what is actually being separated ? In Conference on Learning Theory , pp . 2664–2666 . PMLR , 2019 . Andrew M Saxe , James L McClelland , and Surya Ganguli . Exact solutions to the nonlinear dynamics of learning in deep linear neural networks . arXiv preprint arXiv:1312.6120 , 2013 . David Silver , Aja Huang , Chris J Maddison , Arthur Guez , Laurent Sifre , George Van Den Driessche , Julian Schrittwieser , Ioannis Antonoglou , Veda Panneershelvam , Marc Lanctot , et al . Mastering the game of go with deep neural networks and tree search . nature , 529 ( 7587 ) :484–489 , 2016 . Justin Sirignano and Konstantinos Spiliopoulos . Mean field analysis of neural networks : A central limit theorem . Stochastic Processes and their Applications , 130 ( 3 ) :1820–1852 , 2020 . Mahdi Soltanolkotabi , Adel Javanmard , and Jason D Lee . Theoretical insights into the optimization landscape of over-parameterized shallow neural networks . IEEE Transactions on Information Theory , 65 ( 2 ) :742–769 , 2018 . Sören Sonnenburg , Gunnar Rätsch , Christin Schäfer , and Bernhard Schölkopf . Large scale multiple kernel learning . The Journal of Machine Learning Research , 7:1531–1565 , 2006 . Daniel Soudry and Yair Carmon . No bad local minima : Data independent training error guarantees for multilayer neural networks . arXiv preprint arXiv:1605.08361 , 2016 . Nathan Srebro , Karthik Sridharan , and Ambuj Tewari . Smoothness , low noise and fast rates . In J. Lafferty , C. Williams , J. Shawe-Taylor , R. Zemel , and A. Culotta ( eds . ) , Advances in Neural Information Processing Systems , volume 23 . Curran Associates , Inc. , 2010 . URL https : //proceedings.neurips.cc/paper/2010/file/ 76cf99d3614e23eabab16fb27e944bf9-Paper.pdf . Yuandong Tian . Symmetry-breaking convergence analysis of certain two-layered neural networks with relu nonlinearity . Network , 100 ( 1 ) :1–1 , 2016 . Colin Wei , Jason Lee , Qiang Liu , and Tengyu Ma . On the margin theory of feedforward neural networks . 2018 . Colin Wei , Jason Lee , Qiang Liu , and Tengyu Ma . Regularization matters : Generalization and optimization of neural nets vs their induced kernel . 2019 . Lei Wu , Chao Ma , and Weinan E. A priori estimates of the generalization error for two-layer neural networks . 2018 . Chiyuan Zhang , Samy Bengio , Moritz Hardt , Benjamin Recht , and Oriol Vinyals . Understanding deep learning requires rethinking generalization . arXiv preprint arXiv:1611.03530 , 2016 . Kai Zhong , Zhao Song , Prateek Jain , Peter L Bartlett , and Inderjit S Dhillon . Recovery guarantees for one-hidden-layer neural networks . In International conference on machine learning , pp . 4140– 4149 . PMLR , 2017 . Wenda Zhou , Victor Veitch , Morgane Austern , Ryan P Adams , and Peter Orbanz . Non-vacuous generalization bounds at the imagenet scale : a pac-bayesian compression approach . arXiv preprint arXiv:1804.05862 , 2018 . Difan Zou , Yuan Cao , Dongruo Zhou , and Quanquan Gu . Stochastic gradient descent optimizes overparameterized deep relu networks . arxiv e-prints , art . arXiv preprint arXiv:1811.08888 , 2018 . A APPENDIX Below is the list of different sections of the proof . We have a separate Appendix list for lower level Lemmas in our proof in B . CONTENTS A.1 Stronger Generalization bounds for polynomials . . . . . . . . . . . . . . . . . . . 15 A.2 The Doubling Trick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.3 Amount of Overparameterization . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 A.4 PSD property of K∞ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 A.5 Complexity upper bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.6 Complexity measure and the ζ-norm . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.7 Core Generalization Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.8 Structure of the proof , setting m3 , and further definitions . . . . . . . . . . . . . . 22 A.9 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 A.10 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 A.11 Rademacher Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 A.12 Constructing W ∗ , V ∗ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 A.12.1 First Layer , Construction of W ∗ . . . . . . . . . . . . . . . . . . . . . . . 34 A.12.2 Second Layer , Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . . 39 A.12.3 Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 A.13 Existence of a good direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 A.14 Existence of a good direction Helper Lemmas . . . . . . . . . . . . . . . . . . . . 53 A.15 Bounding the worst-case Senario . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 A.16 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 A.17 Process from a higher view : definition of the ( X ) sequence . . . . . . . . . . . . . 77 A.18 Bounding the MGF of Xi ’ s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 A.19 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 A.20 Gaussian Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 A.20.1 Setting β1 and β2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 A.21 Basic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 A.21.1 Defining the rare events Ej . . . . . . . . . . . . . . . . . . . . . . . . . . 91 A.21.2 Bounding the value of f ′ . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 A.21.3 Bounding the difference between Original and Smoothed Functions . . . . 94 A.1 STRONGER GENERALIZATION BOUNDS FOR POLYNOMIALS In this section , we prove an explicit generalization bound for functions represented as a polynomial sum . Note that the bounds in Arora et al . ( 2019a ) for polynomials assume the monomials with degree larger than one to have even powers , while here we do not impose this restriction . In addition , note that despite Arora et al . ( 2019a ) , our bounds remain meaningful in the noisy case ( recall our Theorem 2 ) . More specifically , we bound the ζ norm of such functions . Consider the target function s with the following power series formula : y = s ( x ) = ∞∑ p=1 ap ( w T p x ) p , ( 26 ) where ap ∈ R and wp ∈ Rd . We can write s ( x ) = g1 ( x ) + d∑ k=1 xkg k 2 ( x ) , ( 27 ) where xk denotes the kth entry of vector x here and g1 ( x ) = ∑ p∈A1 : = { p=1 or p even } ap ( w T p x ) p , and for all k ∈ [ d ] : gk2 ( x ) = ∑ p∈A2 : = { p > 2 , p odd } wpkap ( w T p x ) p−1 . Then , using the tailor series of x ( 14 + arcsin ( x ) 2π ) = ∑∞ p=1 γpx p for |x| ≤ 1 , the RKHS spaceH ( H∞ ) of NTK can be identified by square-summable sequences of reals ( ap′ ) ∞p′=1 with dot product ⟨ ( ap′ ) ∞p′=1 , ( bp′ ) ∞p′=1⟩ = ∞∑ p′=1 γλ ( p′ ) ap′bp′ , where λ ( p′ ) : Z≥0 → Z≥0 such that it maps zero to zero , the first d positive integers are mapped to one , the next d2 ones are mapped to 2 , etc . Moreover , the RKHS mapping Ψ : Rd → H ( H∞ ) from the Euclidean space is : Ψ ( x ) = ∥x∥ ( x′1 , ... , x ′ d , ( x ′ k1x ′ k2 ) k1 , k2∈ [ d ] , . . . , ( x ′ k1x ′ k2 . . . x ′ kp ) k1 , ... , kp∈ [ d ] , . . . ) , where x′ = x/∥x∥ and in the notation above we are presenting a sequence of sequences , by which we mean the inner sequences simply unfold . Using this identification , one can see using the linear representations of g1 , gk2 inH : ∥g1∥2H∞ = ∑ p∈A1 γpa 2 p∥wp∥ 2p 2 , ( 28 ) ∥gk2∥2H∞ = ∑ p∈A2 γpwp 2 ka 2 p∥wp∥ 2 ( p−1 ) 2 , ( 29 ) Summing above and noting the linear representation of g : ∥g1∥2H∞ + d∑ k=1 ∥gk2∥2H∞ = ∑ p∈A1 γpa 2 p∥wp∥ 2p 2 + ∑ p∈A2 γpa 2 p∥wp∥ 2p 2 = ∞∑ p=1 γpa 2 p∥wp∥ 2p 2 = ∥g∥2H∞ . ( 30 ) Now for { g } : = { g′k } d+1 k=1 : = { g1 } ∪ { gk2 } dk=1 , we consider the kernel K { g } . Expanding the tailor series of F2 ( 2F3 ) ( x ) j = ∑∞ p=0 µpx p , we find the identification ( hkp′ ) k∈ [ d+1 ] , p′=0 , ... , ∞ with dot product = ∞∑ p′=0 µλ ( p′ ) d+1∑ k=1 hkp′q k p′ , with RKHS map Ψ2 ( x ) = ( g′1 ( x ) , . . . , g ′ d+1 ( x ) , x ′ 1g ′ 1 ( x ) , . . . , x ′ 1g ′ d+1 ( x ) , . . . , x ′ dg ′ 1 ( x ) , . . . , x′dg ′ d+1 ( x ) , ( x ′ k1x ′ k2g ′ k ( x ) ) k1 , k2∈ [ d ] , k∈ [ d+1 ] , . . . , ( x ′ k1x ′ k2 . . . x ′ kpg ′ k ( x ) ) k1 , ... , kp∈ [ d ] , k∈ [ d+1 ] , . . . ) . Now , we compute the norm of function s with respect to K { g } , combining the above representation and dot product with Equation ( 27 ) and the fact that we work with unit norm x , so x′ = x : ∥s∥2K { g } = µ0 + ( d+ 1 ) µ1 . ( 31 ) Plugging the above and Equation ( 30 ) into the definition of ∥.∥ζ in ( 17 ) , we conclude ∥s∥2ζ ≤ ∥s∥2K { g } ( ∥g1∥ 2 H∞ + d∑ k=1 ∥gk2∥2H∞ ) ≤ ( µ0 + ( d+ 1 ) µ1 ) ∞∑ p=1 γpa 2 p∥wp∥ 2p 2 . ( 32 ) Note that if the odd exponents ( except possibly one ) in the definition of s in ( 26 ) are zero , then we could consider only the function g1 and kernel Kg1 , which would have implied a bound of µ0 ∑∞ p=1 γpa 2 p∥wp∥ 2p 2 . A.2 THE DOUBLING TRICK For the SGD optimization , we set the regularization coefficients in the loss L1 as ψ1 = ν/4 , ψ2 = ν/ ( 4ζ ( f ∗ , G ) ) , ( 33 ) with ν : = max { Rn ( f∗ ) /2 , B2/n } . This assumes we know the f∗ and G that minimize the adaptation within the complexity measure ( 12 ) . Here , we explain how to use a simple doubling trick to get over the fact that we might not know these optimal solutions f∗ and G. Theorem 1 Without explicitly knowing the exact value of the complexity measure , i.e. , the optimal solution of Equation ( 12 ) , one can still achieve the generalization bound in Theorem 1 . Proof of Theorem 1 Our core generalization result is in Theorem 3 . The proof of Theorem 1 is simply adding a doubling trick on top of the argument of Theorem 3 . We also prove Theorem 2 as a consequence of Theorem 1 in Appendix 2 . In the rest of the proofs , for simplicity , we refer to ( WPSGD , VPSGD ) by ( W ′ , V ′ ) . Let f∗∗ , G∗ be the optimal solution to ( 10 ) . With a simple rescaling of G∗ , we can assume ⟨H∞−1 , G∗⟩ = 1 . ( Note that the complexity does not change by such rescaling ) . Now one can exploit the condition ∥yi∥∞ ≤ B , and consider the setting f∗ = 0 to get the following trivial upper bound on the complexity measure : ℑ ( ( xi ) , ( yi ) ) ≤ 2nB2 . Therefore , 2nRn ( f ∗∗ ) + f∗∗A−1f∗∗ ( c′ϖ ) ≤ 2nB2 . ( 34 ) Using Equation ( 34 ) and the optimality of ( f∗∗ , G∗ ) : Rn ( f ∗∗ ) ≤ B2 , ζ = ζ ( f∗∗ , G∗ ) ≤ 2nB2 . ( 35 ) Combining the first equation above with the definition of ν in Equation ( 43 ) , we get B2/n ≤ ν ≤ B2 . ( 36 ) To initialize ψ1 and ψ2 , we use Equations ( 33 ) for any f∗ and G , and as a result we get a generalization bound as in Equation ( 47 ) . However , to achieve the best possible rate characterized by our complexity measure in Theorem 1 without explicitly computing the answer of ( 12 ) , we use a simple doubling trick ; for every pair ( ζ ′ , ν′ ) such that ζ ′ is a power of 2 between B2 and 2nB2 , and ν′ is a power of two between B2/n and B2 , we initialize ψ1 , ψ2 as in Equation ( 33 ) and run the algorithm , then return the network that minimizes the upper bound of the risk given by Equation ( 46 ) . Note that we do not need to know Rn ( f∗∗ ) to compute this upper bound . Now let ν′ be the power of 2 within ν ( G∗ , f∗∗ ) /2 ≤ ν′ < ν ( G∗ , f∗∗ ) . If we are in the case f∗∗TA−1f∗∗ < B2 , ( 37 ) then for ζ ′ equal to the smallest power of two larger than B2 , when we run PSGD with pair ( ν′ , ζ ′ ) , by Theorem 3 : R ( fW ′ , V ′ ) ≤ 2Rn ( f∗ ) + c′′ϖ 2B2 +B2 n ≤ ℑ ( ( xi ) n i=1 , ( yi ) n i=1 ) n + c′ B2ϖ n . ( 38 ) Because we return the minimum upper bound on the risk ( the tighter lower bound of Equation ( 46 ) ) among all such powers of two , we certainly achieve the above rate in ( 38 ) . Otherwise , if f∗∗TA−1f∗∗ ≥ B2 , let ζ ′ be the power of two within f∗TA−1f∗ ≤ ζ ′ ≤ 2f∗TA−1f∗ , then again it is easy to check that conditions of Theorem 3 are satisfied , hence we get the following generalization bound : R ( fW ′ , V ′ ) ≤ 2Rn ( f∗∗ ) + c′′ϖ ( ζ ′ +B2 ) n ≤ 2Rn ( f∗∗ ) + c′′ϖ 2ζ ( f∗∗ , G∗ ) +B2 n ( 39 ) ≤ ℑ ( ( xi ) n i=1 , ( yi ) n i=1 ) n + c′ B2ϖ n , ( 40 ) which proves the bound of Theorem 1 . A.3 AMOUNT OF OVERPARAMETERIZATION In this section , to provide high-level insight , we indicate the right order of magnitude that our overparameterization should be in , with respect to one another . Note that the exact coefficients in these inequalities would depend on the basic parameters B , 1/λ0 , n , which we have avoided here for sake of simplicity . We refer the reader to our main proof ( mostly Appendix parts A.12 , A.13 ) for more details . κ1κ2m3 < < 1 , κ2 √ m2 > > 1 , κ1 √ m3 > > κ2 √ m2 , m1 > > m 4 3 , κ1 √ m1 > > m 3/2 3 , κ2 < < 1/ √ m3√ m3κ2 < < 1/ √ m3 √ m2 > > m 3/2 3 κ1κ2 m33 ( κ2m2 ) < < κ1m1 m1 , m2 , m3 , 1/κ1 , 1/κ2 = poly ( n , B ∨ 1/B , 1/λ0 ) . In addition , we set the smoothing parameters as β2 : = Θp ( ( κ1 √ m3 ) −1 ( √ m2κ2 ) − 23 ) , β1 : = Θp ( m3 √ m3/ ( κ1 √ m1 ) ) , where Θp only shows polynomial dependencies on the overparameterization . A.4 PSD PROPERTY OF K∞ The Schur product theorem states that for PSD matrices A and B , A ⊙ B is also PSD . Now given an analytic function F whose tailor series coefficients are all nonnegative , Suppose we apply F on some PSD matrix A entrywise , denoted by F ( A ) , under the condition that the entries of A are in the radius of convergence of F , then using Schur product theorem , it is straightforward that F ( A ) is also PSD . Using the above property , one can then check that the tailor series of the defined functions F2 and F3 are nonnegative , hence , the application of the function F2 ( 2F3 ( x ) ) on the gram matrix of ( xi ) ni=1 is a PSD matrix , ( note that ∣∣∣⟨xi , xj⟩∣∣∣ ≤ 1 is in the convergence radius of F2 ( 2F3 ( x ) ) . ) thus K∞ is indeed a kernel . A.5 COMPLEXITY UPPER BOUND First we mention a simple fact that hadamard product respects matrix orderings . Given PSD matrices A , B , C such thatA ⪯ B , the fact thatA⊙C ⪯ B⊙C is an easy consequence of the Schur Product Theorem ; indeed , B −A is PSD by definition , so ( B −A ) ⊙ C = B ⊙ C −A⊙ C is also PSD . Next , it is easy to check that the tailor series of arcsin ( x ) has all nonnegative coefficients . Therefore , for a PSD matrix X , as we discussed in Appendix A.4 , applying arcsin entrywise on X , namely arcsinX , is also PSD . Setting X equal to the entrywise application of 2F3 to the gram matrix of datapoints ( xi ) ni=1 , we realize the matrix arcsin ( 2F3 ( ( ⟨xi , xj⟩ ) 1≤i , j≤n ) ) is also PSD . Noting the definition of K∞ in Equation ( 8 ) , we conclude that for the data kernel matrix K∞ we have K∞ ≥ 1 4 11 T , where 1 is the all ones n-dimensional vector . Combining the two mentioned facts , we can lower bound the matrix K = K∞ ⊙G for any matrix G as K = K∞ ⊙G ≥ 1 4 11 T ⊙G = 1 4 G. Substituting the rank one matrix f∗f∗T for the n-dimensional vector f∗ in Equation ( 13 ) : K−1 { f̃∗/∥f̃∗∥ } = K ∞ ⊙ f∗f∗T /∥f̃∗∥2 ≥ 1 4 f∗f∗T /∥f̃∗∥2 . ( 41 ) The inequality used in ( 13 ) then follows from Equation ( 41 ) . A.6 COMPLEXITY MEASURE AND THE ζ-NORM This is a brief section regarding some basic properties of ℑ and ∥.∥ζ . First , note that the two versions of the complexity measure in Equations ( 10 ) and ( 11 ) are equivalent , as for any finite set of functions { g } , we can define the gram matrix with respect to the feature vectors of these functions on data , and for an arbitrary nonzero PSD G we can consider a Cholesky factorization for G as G = X̄T X̄ , then define the functions { gk } as the minimum-NTK norm functions which map the input to the features corresponding to X̄ . This observation further implies we can suppose the factor matrix X̄ is in Rn×n , and there is a set of at most n functions { gk } nk=1 which corresponds to this G. Next , we show that for an arbitrary function f , its sup norm over the unit ball is bounded by its ζ norm : sup ∥x∥=1 |f ( x ) | ≤ ∥f∥ζ . ( 42 ) Note that for a kernel K which satisfies K ( x , x ) ≤ 1 , using Cauchy Schwarz we simply obtain f ( x ) ≤ ∥f∥K √ K ( x , x ) ≤ ∥f∥K , where recall that ∥.∥K is the norm corresponding to the RKHS space of K. Hence , to show ( 42 ) , it suffice to show that for all kernels K ∈ K and unit norm x we have K ( x , x ) ≤ 1 . To see this fact , note that the norm of each x ∈ Rd in the NTK-space isH∞ ( x , x ) = 12 . Therefore , for each function g with bounded-NTK norm , again using Cauchy Schwarz : |g ( x ) | ≤ 1 2 ∥g∥H∞ . As a result , for a family of functions { g } with ∑ g∈ { g } ∥g∥2H∞ ≤ 1 , we have on every unit norm x : ∑ g∈ { g } g ( x ) 2 ≤ 1 . On the other hand , it is easy to check that for every unit norm x , we have K∞ ( x , x ) ≤ 12 , so for every such { g } , we have by definition K { g } ( x ) ≤ 1 , which completes the proof of Equation ( 42 ) . A.7 CORE GENERALIZATION RESULT In this section , we prove our core generalization result for the trained network , Theorem 3 , which underlies our generalization bounds in Theorems 1 and 2 . Recall that in the rest of the proofs , we refer to the solution ( WPSGD , VPSGD ) returned by PSGD simply by ( W ′ , V ′ ) . Theorem 3 Suppose we have a candidate pair ( f∗ , G ) regarding our complexity measure in ( 10 ) that satisfies ⟨H∞−1 , G⟩ ≤ 1 , f∗TA−1f∗ ≤ ζ ( recall A = G⊙H∞ ) . Then , for ν = max { Rn ( f∗ ) /4 , B2/n } , ( 43 ) if we are given ν/2 ≤ ν′ ≤ ν , and we set ψ1 = ν′ 4 , ( 44 ) ψ2 = ν′ 4ζ , ( 45 ) then for the solution ( W ′ , V ′ ) returned by PSGD we have the following generalization bound : R ( fW ′ , V ′ ) ≤ 5 4 Rn ( fW ′ , V ′ ) + c ′′′ϖ ( ζ +B2 ) n ( 46 ) ≤ 5 3 Rn ( f ∗ ) + c′′ϖ ( ζ +B2 ) n , ( 47 ) for constants c′′ , c′′′ and log factor ϖ = log ( n ) 3 + log ( 1/λ0 ) . Proof of Theorem 3 Almost all of our proofs in the rest are in the aim of proving Theorem 3 . Crucially , to prove this Theorem , we need to establish two big results : 1 . We need to show that the final network has small training loss , and is within the class Gγ1 , γ2 for some suitable γ1 , γ2 . This is handled by Theorem 4 in Appendix A.10 . We define the class Gγ1 , γ2 roughly as the class of networks with norm bounds ∥W −W ( 0 ) ∥ ≤ γ1 , ∥V −V ( 0 ) ∥ ≤ γ2 where the rows of V −V ( 0 ) are orthogonal to the subspace Φ , plus an additional structure defined in Appendix A.11 . This task , on its own , has three main steps in our proof : ( a ) we construct a “ good ” underlying network , Appendix A.12 ( b ) we find a “ good ” random direction and study the landscape of the objective , Ap- pendix A.13 ( c ) we prove the convergence , Appendix A.16 2 . The Rademacher Complexity of the class Gγ1 , γ2 needs to be suitably bounded . This is handled by Theorem 5 in Appendix A.11 . With access to these results , here we show how Theorem 3 follows by a simple application of the generalization bound in ( Srebro et al. , 2010 ) . Specifically , for fixed constants z1 , z3 and every integer i ≥ 0 , we use Theorem 1 of ( Srebro et al. , 2010 ) for the class Gz1 , γi , γi = 2i×B/z3 with confidence probability 1 − 2−iδ3 , which , with a union bound , implies that with probability at least 1 − δ3 , for every i and fW ′ , V ′ ∈ Gz1 , γi : R ( fW ′ , V ′ ) ≤ Rn ( fW ′ , V ′ ) +K ( √ Rn ( fW ′ , V ′ ) ( √ log ( n ) 1.5R ( Gz1 , γi ) + √ b log ( 1/ ( 2−iδ3 ) ) n ) ( 48 ) + log ( n ) 3R ( Gz1 , γi ) 2 + b log ( 1/ ( 2−iδ3 ) ) n ) , ( 49 ) where ℓ ( fW ′ , V ′ ( x ) , y ) is a.s. bounded by b for function within the class Gz1 , γi , and K is a universal constant . In the following , we aim to further bound the Rademacher complexityR and parameter b . Applying the AM-GM inequality with respect to ratio z4 > 0 for the second term : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) +K2/z4 ( log ( n ) 1.5R ( Gz1 , γi ) + √ b log ( 1/ ( 2−iδ3 ) ) n ) 2 + log ( n ) 3R ( Gz1 , γi ) 2 + b log ( 1/ ( 2−iδ3 ) ) n ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) +K2/z4 ( log ( n ) 1.5R ( Gz1 , γi ) + √ b log ( 1/ ( 2−iδ3 ) ) n ) 2 + log ( n ) 3R ( Gz1 , γi ) 2 + b log ( 1/ ( 2−iδ3 ) ) n ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) log ( n ) 3R ( Gz1 , γi ) 2 + ( 2K2/z4 + 1 ) b log ( 1/ ( 2−iδ3 ) ) n . ( 50 ) Now let γ∗ be the smallest number of the form 2iB/z3 ( for some i ) which is not smaller than z2 √ ζ . This definition implies γ∗ ≤ max { 2z2 √ ζ , B/z3 } . ( 51 ) Now Theorem 5 in Appendix A.11 bounds the Rademacher complexity : R ( Gz1 , γ∗ ) ≤ 2z1γ ∗ √ n . ( 52 ) On the other hand , from Theorem 4 by setting z1 , z2 = √ 40 , we get fW ′ , V ′ ∈ Gz1 , γ∗ . Moreover , from Lemma 34 , for fW ′ , V ′ ∈ Gz1 , γ∗ , we have for every ∥x∥ ≤ 1 : |fW ′ , V ′ ( x ) | ≤ 2z1γ∗ , ( 53 ) so the loss ℓ ( fW ′ , V ′ ( x ) , y ) can be bounded by ( B + 2z1γ∗ ) 2 using the 1 smoothness property . Therefore , for the class Gz1 , γ∗ we can set b = ( B + 2z1γ∗ ) 2 . Combining this with Equation ( 52 ) and plugging into Equation ( 50 ) : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) log ( n ) 3 4z21γ ∗2 n + ( 2K2/z4 + 1 ) ( B + 2z1γ ∗ ) 2 log ( 1/ ( 2−i ∗ δ3 ) ) n . Furthermore , by definition of γ∗ , we have 2i ≤ 2z2z3 √ ζ/B : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) 4z21 ( log ( n ) 3 + 2 log ( 2z2z3 √ ζ/B ) ) 4z21γ ∗2 n ( 54 ) + ( 2K2/z4 + 1 ) 2B2 log ( 2z2z3 √ ζ/B ) n . ( 55 ) Now applying the upper bound on γ∗ : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) 4z21 ( log ( n ) 3 + 2 log ( 2z2z3 √ ζ/B ) ) 4z21 ( 2z2 √ ζ + 2B/z3 ) 2 n ( 56 ) + ( 2K2/z4 + 1 ) 2B2 log ( 2z2z3 √ ζ/B ) n . ( 57 ) If ζ > B2 , in the third term above we substitute B by √ ζ . Finally , similar to the bound we stated in Equation ( 15 ) , note that we have the following trivial bound for ζ : ζ ≤ yTH∞−1y ≤ 4nB2/λ0 , ( 58 ) i.e . there is no point in considering larger ζ ’ s , which implies log ( 2z2z3 √ ζ/B ) = O ( log ( n ) + log ( 1/λ0 ) ) . Plugging this above and picking z4 = 1/3 show the proof of Equation ( 46 ) . Furthermore , applying Equation ( 70 ) in Theorem 4 to the Rn ( fW ′ , V ′ ) term in Equation ( 46 ) further gives the second Equation ( 47 ) . Remark 1 In the same setting of Theorem 3 , if we have ν/2 ≤ ν′ but not generally upper bounded by ν , then PSGD leads to the following generalization bound : R ( fW ′ , V ′ ) ≤ Rn ( f∗ ) + ν′ + c′′′′ϖ ( ζ +B2 ) n , using a similar argument as we did for Theorem 3 . A.8 STRUCTURE OF THE PROOF , SETTING m3 , AND FURTHER DEFINITIONS Throughout the proof , ( W ′ , V ′ ) represents the pair of matrices of the current iteration of PSGD , ( W ∗ , V ∗ ) are the “ ideal ” matrices that we construct in Appendix A.12 , ( W ρ , V ρ ) and refers to the gaussian smoothing matrices . Importantly , note that our squared loss ℓ ( f , y ) is zero at f = y . We have tried to make the lower level proofs into sub-lemmas and create a manageable hierarchy as much as we could , to make the document more clear and readable . Similar to the conditions in Theorem 3 , through out most of the proofs we assume that we are given a pair ( f∗ , G ) with a slightly more general setting of Theorem 3 : f∗TA−1f∗ ≤ ζ2 , for A = G⊙K∞ , ⟨G , H∞⟩ ≤ ζ1 . Particularly , ζ1 , ζ2 appear in Appendix A.13 . Because we are allowed to rescale G , we do not really gain much by assuming this more general setting , though we pick to work with the general setting as the abstraction makes the proof more straightforward to understand . We refer to the parameters B , 1/λ0 , n as the “ basic parameters ” , m1 , m2 , m3 , 1/κ1 , κ2 as the “ overparameterization ” , and β1 , β2 as the “ smoothing parameters. ” By the phrase “ having enough overparameterization ” we mean it suffices to pick the overparameterization m1 , m2 , m3 , 1/κ1 , 1/κ2 only polynomially large in the basic parameters . Throughout the proof , we denote the change in the output of the first layer at W ( 0 ) +W ′ +W ρ compared to the initialization value by ϕ ( 2 ) ( xi ) , i.e . ϕ ( 2 ) ( xi ) = 1√ m1 W sσ ( ( W ( 0 ) +W ′ +W ρ ) xi ) − ϕ ( 0 ) ( xi ) , while recall that ϕ′ ( xi ) has a similar definition except without the smoothing matrix W ρ . Although our model is a three layer network , throughout the proof , we refer to the parts W s 1√ m1 σ ( ( W ( 0 ) + W ′ ) x ) and 1√ m2 aTσ ( ( V ( 0 ) + V ′ ) ( . ) ) as the “ first layer ” and “ second layer , ” respectively . Also , we sometimes refer to the binary sign pattern of vector x multiplied to matrix W by DW , x ( DW , x : = Sgn ( Wx ) ) , i.e . the jth diagonal entry of DW , x is one if WTj , x ≥ 0 , and is zero otherwise . To refer to the jth row of W as a vector , we sometimes drop the comma in Wj , and write it as Wj . For brevity , we denote the Frobenius norm ∥W∥F of matrix by ∥W∥ , and the Euclidean norm of a vector x by ∥x∥ . For matrices W1 , W2 we denote their natural dot product by ⟨W1 , W2⟩ : = tr ( WT1 W2 ) . We refer to the smallest eigenvalue of a matrix by λmin ( . ) . We write R ( . ) for the Rademacher complexity of a function class . We refer to the smoothed version of the network by f ′W ′ , V ′ ( x ) , defined by f ′W ′ , V ′ ( x ) = EWρ , V ρfW ′+Wρ , V ′+V ρ ( x ) . In the proof , we mainly work with the loss over the smoothed network f ′ , defined as L ( W ′ , V ′ ) = Rn ( f ′ W ′ , V ′ ) + ψ1∥W ′∥2 + ψ2∥V ′∥2 . ( 59 ) Our algorithm , PSGD can be regarded roughly as an SGD over L. Similar to what we discussed in section 3 , let the functions { gk } m3k=1 be some feature representation whose gram matrix is equal to G and ⟨H∞ , G⟩ = ∑m3 k=1 ∥gk∥2H∞ . In such setting , it is not hard to check that we can assume each gk is the minimum norm NTK function which maps ( xi ) ni=1 to ( gk ( xi ) ) n i=1 ’ s . Indeed , if this is not the case for some gk , we can project the RKHS representation of gk onto the span of the representations of ( xi ) ni=1 , which can only decrease the complexity measure . Hence we can represent gk ∈ HH∞ as a linear combination of basic functions H∞ ( xi , . ) on data points : ∀k ∈ [ m3 ] , gk ( x ) : = n∑ i=1 Vk , iH∞ ( xi , x ) . ( 60 ) Here , the sum of squared-H∞ norms of Vk is bounded as∑ k ∥Vk∥2H∞ = ∑ k ∥gk∥2H∞ = ⟨H∞ , G⟩ ≤ ζ1 . ( 61 ) For each i ∈ [ n ] , we refer to the feature representation vector ( gk ( xi ) ) m3k=1 on xi as x̄i . Note that we have the relationship x̄i = ( H ∞ i , Vk ) m3 k=1 , ( 62 ) whereH∞i , is the ith row ofH ∞ . In the analysis , we work with a bound ξ on the quantity maxk ∥Vk∥ which should be bounded polynomially by other basic parameters ; in particular , it is defined in Lemma 10 and is used to bound a cross term in Lemma 14 . However , maxk ∥Vk∥ might not be effectively bounded for an arbitrary feature representation . Fortunately , we can remedy this by a simple trick ; for every natural number s , one can substitute every gk by s copies of gk/ √ s , without changing the gram matrix G. Therefore , for any δ , one can increase the multiset of functions ( gk ) to a bigger set ( g̃k ) , by adding at most O ( ζ1/δ ) functions , making sure of the following for the new functions : ∀k : VTk H∞Vk = ∥g̃k∥2H∞ ≤ δ . ( 63 ) ( This is because ∑ k ∥g̃k∥2H∞ ≤ 1 ) . Furthermore , observe that for each gram matrix G , we have an n-dimensional feature representation ( gk ) nk=1 for G according to the Cholesky factorization . Combining these facts , we conclude , to guarantee Equation ( 63 ) , in the worst case , we need m3 to be as large as n+O ( ζ1/δ ) . Finally , observing the following inequality ∥Vk∥2 ≤ ∥Vk∥2H∞/λ0 ≤ ∥g̃k∥2H∞/λ0 . ( 64 ) in order to guarantee maxk ∥Vk∥ ≤ ξ we need to take m3 as large as n + O ( ζ1/ ( ξ2λ0 ) ) , which is indeed bounded polynomially by the basic parameters because of the same condition for 1/ξ . This computation also brings into sight an important point : “ Although each gram matrix G is representable by n features , in order for the algorithm to be able to find a suitable network , m3 might need to be larger than n. ” Moreover , for every 1 ≤ k , i ≤ n , we define the matrices Zki ∈ Rmd as Zik = 1/ √ m1 ( W sk , j1 { W ( 0 ) j Txi } xi ) m1 j=1 , ( 65 ) where in the above notation , j is enumerating the columns of the matrix . We also define the following matrices which we use in our construction later : W+k T =W k+ T = n∑ i=1 Vk , iZik , ( 66 ) and W+ as W+ = m3∑ k=1 W k+ . Finally , to avoid unnecessary complication , we often argue high probability bounds without an explicit representation of their dependency on the chance of failure ( which is a negligible logarithmic factor ) . We also ignore all constants and log factors , and mainly work with the notation ≲ which ignores constants ; we write a ≲ b ± c as a short form for b − O ( c ) ≤ a ≤ b + O ( c ) . As there are several hierarchies of new parameters that are defined based on lower-level ones , we rename the new parameters and continue viewing them as black-box . This makes the proofs more readable , since we also do not care about the exact dependency of the underlying parameters most of the time , rather we are interested in their orders of magnitude , for example that a given parameter goes to zero polynomially fast with respect to the overparameterization , etc . Due to the large number of symbols that we have to work with , we might use a symbol more than once , of course when it is clear from the context which one we are refering to . A.9 PROOF OF THEOREM 2 In this section , we prove Theorem 2 , stated below . Theorem 2 For any function f : Rd → R , in the same setting as Theorem 1 , the population risk of the trained network ( W ′ , V ′ ) can be bounded as R ( fW ′ , V ′ ) ≤ 2R ( f ) +O ( αϖ ∥f∥2ζ +B2 n ) . ( 67 ) Proof of Theorem 2 Theorem 2 is a simple consequence of Theorem 1 ; for the given function f , we apply Theorem 2 with the smaller coefficient γ = 43 for Rn ( f ∗ ) , regarding the complexity upper bound , by setting f∗ : = ( f ( xi ) ) ni=1 : R ( fW ′ , V ′ ) ≤ 4 3 Rn ( f ∗ ) + ( αϖ ) min K∈K f∗TK−1f∗ n + B2αϖ n = 4 3 Rn ( f ) + ( αϖ ) min K∈K f∗TK−1f∗ n + B2αϖ n . On the other hand , because f∗TK−1f∗ is the minimum-RKHS norm of a function with respect to kernel K which maps xi ’ s to f∗i and f is one such function , we have f ∗TK−1f∗ ≤ ∥f∥K . This inequality implies min K∈K f∗TK−1f∗ ≤ ∥f∥ζ , so we obtain R ( fW ′ , V ′ ) ≤ 4 3 Rn ( f ) + ( αϖ ) ∥f∥2ζ +B2 n . ( 68 ) Therefore , it remains to bound Rn ( f ) by R ( f ) . As we showed in Appendix A.6 , for every input x we have f ( x ) ≤ ∥f∥ζ , so for every data ( x , y ) , by the fact that |y| ≤ B a.s. and α smoothness of the loss , we have ℓ ( f ( x ) , y ) ≤ α ( ∥f∥ζ + B ) 2 . Moreover , note that the random variable ℓ ( f ( x ) , y ) has mean R ( f ) . It is easy to check that in this setting , the variance of ℓ ( f ( x ) , y ) is at most R ( f ) α ( ∥f∥ζ + B ) 2 . Therefore , an application of the Bernstein inequality , we have with high probability over the dataset Rn ( f ) ≤ R ( f ) +O ( √ R ( f ) α ( ∥f∥ζ +B ) 2 n + α ( ∥f∥ζ +B ) 2 n ) ≤ 3 2 R ( f ) +O ( α ( ∥f∥ζ +B ) 2 n ) . Plugging this back to Equation ( 68 ) completes the proof . As a result , the learned network can compete with any function that has reasonably small ∥f∥ζ : R ( fW ′ , V ′ ) ≤ min f { 2R ( f ) +O ( αϖ ∥f∥2ζ +B2 n ) } . A.10 OPTIMIZATION In this section , we glue together • the existence of a good random direction that we prove in Appendix A.13 • the convergence analysis of PSGD that we do based on the work Ge et al . ( 2015b ) in Appendix A.16 . Theorem 4 In the same setting of Theorem 3 , assume the network ( W ′ , V ′ ) returned by PSGD , has sufficient polynomially large “ overparameterization ” . Then , for the solution ( W ′ , V ′ ) returned by PSGD we have L ( W ′ , V ′ ) ≤ Rn ( f∗ ) + ν , ( 69 ) which further implies Rn ( fW ′ , V ′ ) ≤ Rn ( f∗ ) + 2ν , ( 70 ) ∥W ′∥2 ≤ 40 , ∥V ′∥2 ≤ 40ζ . ( 71 ) Moreover , for every i ∈ [ n ] , j ∈ [ m1 ] , j /∈ P for P defined in 1 , we have that sign ( ( W ( 0 ) j +W ′j ) Txi ) and sign ( W ( 0 ) j T xi ) are the same . Proof of Theorem 4 Let Υ ∈ Rm2 ( m3−n ) ×m2m3 be a matrix whose rows are an orthonormal basis for the space of matrices whose rows are orthogonal to span ( { ϕ ( 0 ) ( xi ) } ni=1 ) , i.e . Φ⊥ , as defined in ( 24 ) . Then , we consider a linear change of coordinates for the subspace Φ⊥ , regarding the second layer weights , as v′ = Υvec ( V ′ ) where vec ( . ) splits out the vectorized version of a matrix . For consistent notation , we also denote w′ = W ′ , so we now have a new coordinate system ( w′ , v′ ) ∈ Rm2 ( m3−n ) ×m1d for pairs of weights ( W ′ , V ′ ) such that V ′ ∈ Φ⊥ . We also define the loss function LΠ ( w : = ( w′ , v′ ) ) = L ( W ′ , V ′ ) , with respect to the change of coordinate . Now it is easy to see that running PSGD on L in the normal coordinates is equivalent to running stochastic gradient descent on LΠ with respect to ( w′ , v′ ) . Moreover , because multiplying to matrix Υ is an orthonormal change of coordinates for Φ⊥ and because V ′ is already in ϕ⊥ at each step of PSGD , then ∥v′∥ = ∥V ′∥ , so the conditions ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 are equivalent to ∥w′∥ ≤ C1 , ∥v′∥ ≤ C2 . Furthermore , by our construction , the random matrix V ∗Σ is in the subspace Φ⊥ , so the norm bounds ∥W ∗∥ ≤ ζ1 , ∥V ∗∥ ≤ ζ2 are equivalent to ∥w∗∥ ≤ ζ1 , ∥v∗∥ ≤ ζ2 for w∗ : = W ∗Σ and v∗ : = Υvec ( V ∗ ) . Now we apply the result of Theorem 6 on LΠ with parameter ν set as ν′ , ζ2 : = ζ and ζ1 : = 1 , and ∆ : = Rn ( f ∗ ) , as defined in Theorem 3 . More specifically , based on our arguments above regarding the natural isometry in the change of coordinate , any pair ( w′ , v′ ) in the domain ∥w′∥ ≤ C1 , ∥v′∥ ≤ C2 , LΠ ( w′ , v′ ) ≥ Rn ( f∗ ) + ν′ translates into a pair ( W ′ , V ′ ) in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , L ( W ′ , V ′ ) ≥ Rn ( f∗ ) + ν′ , for which by Theorem 6 there exists ( W ∗Σ , V ∗Σ ) such that EΣL ( W ′ − η/2W ′ + √ ηW ∗Σ , V ′ − η/2V ′ +√ηV ∗Σ ) ≤ L ( W ′ , V ′ ) − ην′/4 . ( 72 ) Translating back to the change of coordinates : EΣLΠ ( w′ − η/2w′ + √ ηw∗Σ , v ′ − η/2v′ +√ηv∗Σ ) ≤ L ( w′ , v′ ) − ην′/4 . ( 73 ) Now we apply Lemma 44 to translate this into an argument about the landscape of LΠ . As a result , applying the bounds in Equations ( 107 ) and ( 127 ) , we obtain that for ( w′ , v′ ) such that LΠ ( w′ , v′ ) ≥ Rn ( f∗ ) + ν′ , we should either have ∥∇LΠ ( w′ , v′ ) ∥ ≥ ν/4 4 √ ∥w′∥2 + ∥v′∥2 = ν/4 4 √ ∥W ′∥2 + ∥V ′∥2 = ν 16 √ C21 + C 2 2 , or λmin ( ∇2LΠ ( w′ , v′ ) ) ≤ − ν/4 2minΣ ( ∥w∗∥2 + ∥v∗∥2 ) = − ν 2minΣ ( ∥W ∗Σ∥2 + ∥V ∗Σ∥2 ) ≤ − ν 16 ( ζ1 + ζ2 ) = − ν 16 ( 1 + ζ ) . Next , we want to apply Theorem 7 by setting γ = ν 16 ( 1 + ζ ) , ℵℓ = Rn ( f∗ ) + ν′ , and lipschitz parameters ρ1 , ρ2 , ρ3 = poly ( B , C1 , C2 , m1 , m2 , m3 ) set as described in Appendix B.1 , Theorem 9 . Also , note that as prescribed by Theorem 7 , we set C1 : = ℵ+ 4l ψ1 , C2 : = ℵ+ 4l ψ2 , ( 74 ) where l = O ( 1 ) depends on our desired chance of success for the algorithm , specified in Theorem 7 . Finally , note that Theorem 7 needs to work with a bounded noise on the gradient whose covariance matrix is bounded between two multipliers of identity . The point of injecting extra noise to SGD in PSGD is in fact because of this covariance condition that we need in Theorem 7 . On the other hand , note that in general , because of the gaussian smoothing that we use , the noise vector is not supported on a bounded domain , which makes it a bit harder to apply Hoeffding type concentration . To remedy this , we introduce a coupling between our unbounded noise vector for L ( W ′ , V ′ ) and another noise random variable whose support is bounded , which with high probability is equal to the real noise , along all iterations . In Corollary , we further translate this coupling for the objective LΠ after change of cooridnates , and write down the exact dependencies of the parameters Q , σ1 and σ2 , which are all polynomial in the basic parameters and the overparameterization . Hence , the conditions of Theorem 7 are satisfied , so we conclude that after at most poly ( ρ1 , ρ2 , ρ3 , Q , ℵ , C1 , C2 , 1/γ , log ( σ1/σ2 ) ) = poly ( B , m1 , m2 , m3 , C1 , C2 , ζ1 , ζ2 ) = poly ( n , B ∨ 1/B , 1/γ0 ) number of iterations , PSGD reach a point wt in some iteration t with LΠ ( wt ) ≤ ℵℓ . Translating back this wt = ( w′t , v ′ t ) by multiplying the v ′ t part to Υ T , we get a pair ( W ′t , V ′ t ) with objective value bounded as L ( W ′t , V ′ t ) ≤ Rn ( f∗ ) + ν′ . But note that we obviously have the condition ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 through the whole iterations , for the choice of C1 , C2 in Equation ( 74 ) . Therefore , using Lemma 34 , for every i ∈ [ n ] : |f ′W ′ , V ′ ( xi ) | = O ( C1 , C2 ) , ( 75 ) |fW ′ , V ′ ( xi ) | = O ( C1 , C2 ) ( 76 ) From Equations ( 76 ) , as also stated in Theorem 9 , we know that for all i ∈ [ n ] , ℓ ( . , yi ) is O ( C1C2 ) + B 2-Lipschitz at points fW ′ , V ′ ( xi ) and f ′W ′ , V ′ ( xi ) , so we can bound the difference |ℓ ( f ′W ′ , V ′ ( xi ) , yi ) − ℓ ( fW ′ , V ′ ( xi ) , yi ) | by ( O ( C1C2 ) +B2 ) |f ′W ′ , V ′ ( xi ) , yi ) − fW ′ , V ′ ( xi ) | , which in turn can become arbitrarily small having enough overparameterization using Lemma 35 , in particular , we force it to be smaller than O ( ν′/ ( B2 + C1C2 ) ) ( recall ν′ ≥ ν/2 ≥ B2/ ( 2n ) ) . As a result , we get |ℓ ( f ′W ′ , V ′ ( xi ) , yi ) − ℓ ( fW ′ , V ′ ( xi ) , yi ) | = O ( ν ) for every i ∈ [ n ] , which in turn implies |L ( W ′ , V ′ ) −L1 ( W ′ , V ′ ) | ≤ ν by picking small constants , where recall that the objective L1 is the same as L but without the smoothing . Now applying this bound to Equation ( 75 ) , we get L1 ( W ′ t , V ′ t ) ≤ Rn ( f∗ ) + 2ν′ . Therefore , as PSGD check the values of L1 in the loop , it terminates at such pair ( Wt , Vt ) . From this point onward , we refer to the returned ( W ′t , V ′ t ) as just ( W ′ , V ′ ) . Opening the definition of L1 ( W ′ , V ′ ) , we clearly get Rn ( fW ′ , V ′ ) ≤ L1 ( W ′ , V ′ ) ≤ Rn ( f∗ ) + 2ν′ ≤ Rn ( f∗ ) + 2ν . ( 77 ) Furthermore , noting the setting of ψ1 , ψ2 in Theorem 3 and the fact that ν′ ≥ ν/2 ≥ Rn ( f∗ ) /8 , we get ∥W ′∥2 ≤ 4 ( Rn ( f ∗ ) + 2ν′ ) ν′ ≤ 40 , ( 78 ) ∥V ′∥2 ≤ 4ζ ( Rn ( f ∗ ) + 2ν′ ) ν′ ≤ 40ζ , ( 79 ) which completes the proof . The fact that for every i ∈ [ n ] , j ∈ [ m1 ] , j /∈ P we have that sign ( ( W ( 0 ) j +W ′ j ) Txi ) and sign ( W ( 0 ) j T xi ) are the same follows from Lemma 1 . A.11 RADEMACHER COMPLEXITY In this section we show the proof for our Rademacher Complexity bound , which is used in Theorem 3 . Theorem 5 Let Gγ1 , γ2 be the class of neural nets with weights ( W , V ) in our three layer setting , such that ∥W − W ( 0 ) ∥ ≤ γ1 , ∥V − V ( 0 ) ∥ ≤ γ2 , where for every j ∈ [ m2 ] , i ∈ [ n ] : V ′j ⊥ ϕ ( 0 ) ( xi ) , and for every i ∈ [ n ] , j ∈ [ m1 ] , j /∈ P for P ⊆ [ m1 ] defined in Lemma 1 , it satisfies sign ( ( W ( 0 ) j +W ′ j ) xi ) = sign ( W ( 0 ) xi ) . Then , for large enough overparameterization , we have the following bound on the Rademacher complexity : R ( Gγ1 , γ2 ) ≤ 2γ1γ2√ n . Proof of Theorem 5 Here , we do not have the smoothing matrices W ρ , V ρ anymore . In this section , unlike the optimization section that we used { x′i } ni=1 to denote the output of the first layer by incorporating also the smoothing matrices , here we define it without them : x′i = 1√ m1 W sσ ( ( W ( 0 ) +W ′ ) xi ) . Now define the matrices Z ′i = 1/ √ m2 ( aj1 { V ( 0 ) j , x ′ i ≥ 0 } x′i ) m2 j=1 , Z ′i + = 1/ √ m2 ( aj ( 1 { Vj , x′i ≥ 0 } − 1 { V ( 0 ) j , x ′ i ≥ 0 } ) x′i ) m2 j=1 . To bound the Z ′i + part , note that substituting C1 by γ1 in lemma 6 and assuming conditions m1 = Ω̃ ( m 4 3 ) , 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 ≤ γ1 , ( we can use this result because we do not have the smoothing matrix W ρ here ) , we get with high probability over the initialization for every i ∈ [ n ] : ∥ϕ′ ( xi ) ∥ = ∥x′i − ϕ ( 0 ) ( xi ) ∥ ≲ γ1 . ( 80 ) Therefore , we can write |trace ( V Z ′i + ) | = 1√ m2 | ∑ j aj1 { sign ( Vj , x′i ) ̸= sign ( V ( 0 ) j , x ′ i ) } Vj , x′i| ≤ 1√ m2 ∑ j 1 { sign ( Vj , x′i ) ̸= sign ( V ( 0 ) j , x ′ i ) } |Vj , x′i| ≤ 1√ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ | ( Vj , − V ( 0 ) j , ) x ′ i| } ( | ( Vj , − V ( 0 ) j , ) x ′ i|+ |V ( 0 ) j , x ′ i| ) ≤ 1√ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ | ( Vj , − V ( 0 ) j , ) x ′ i| } ( 2| ( Vj , − V ( 0 ) j , ) x ′ i| ) . ≤ 1√ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| , | ( Vj , − V ( 0 ) j , ) x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3 min ( γ1 , ∥x′i∥ ) } ( 2| ( Vj , − V ( 0 ) j , ) x ′ i| ) + 1√ m2 ∑ j 1 { | ( Vj , − V ( 0 ) j , ) x ′ i| ≥ γ 2/3 2 ( κ2 m2 ) 1/3 min ( γ1 , ∥x′i∥ ) } ( 2| ( Vj , − V ( 0 ) j , ) x ′ i| ) . Now using the fact that Vj − V ( 0 ) j is orthogonal to ϕ ( 0 ) ( xi ) ’ s : LHS ≤ 2γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } + 2 √ m2 ∑ j 1 { ∥Vj , − V ( 0 ) j , ∥∥x ′ i − ϕ ( 0 ) ( xi ) ∥ ≥ γ 2/3 2 ( κ2 m2 ) 1/3γ1 } ∥Vj , − V ( 0 ) j , ∥∥x ′ i − ϕ ( 0 ) ( xi ) ∥ . Next , using the upper bound on ∥x′i − ϕ ( 0 ) ( xi ) ∥ : LHS ≲ 2γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ∑ j ( 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } . + γ1√ m2 ∑ j 1 { ∥Vj , − V ( 0 ) j , ∥ ≳ γ 2/3 2 ( κ2 m2 ) 1/3 } ∥Vj , − V ( 0 ) j , ∥ ≲ γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ∑ j ( 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } + γ1√ m2 √∑ j 1 { ∥Vj , − V ( 0 ) j , ∥2 ≥ γ 4/3 2 ( κ2 m2 ) 2/3 } √∑ j ∥ ( Vj , − V ( 0 ) j , ) ∥2 ≤ γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ( ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } ) + γ22γ1√ m2 × ( m2 κ2 ) 1/3 1 γ 2/3 2 . Then , applying the first argument of Lemma 29 , we have with high probability over the randomness of V ( 0 ) : ≤ γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ( m2 κ2 ( κ2 m2 ) 1/3γ 2/3 2 ) + γ22γ1√ m2 × ( m2 κ2 ) 1/3 1 γ 2/3 2 ≤ γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 ≲ γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 . Therefore , we can write : R ( Gγ1 , γ2 ) | ( xi ) , ( yi ) = 1 n Eϵ sup V , W∈S n∑ t=1 ϵifV , W ( xi ) = 1 n Eϵ sup V , W∈S n∑ i=1 ϵia Tσ ( 1/ √ m2VW sσ ( 1/ √ m1Wxi ) ) = 1 n Eϵ sup V , W∈S n∑ i=1 ϵia Tσ ( 1/ √ m2V x ′ i ) = 1 n Eϵ sup W∈S sup V ∈S n∑ i=1 ϵitrace ( V ( Z ′i + Z ′ i + ) ) ≲ 1 n Eϵ sup W , V ∈S n∑ i=1 ϵitrace ( V Z ′i ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 ≤ 1 n Eϵ sup W∈S sup V ∈S trace ( V ( n∑ i=1 ϵiZ ′ i ) ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 = 1 n Eϵ sup W∈S sup V ∈S trace ( ( V − V ( 0 ) ) ( n∑ i=1 ϵiZ ′ i ) ) + 1 n Eϵ sup W∈S trace ( V ( 0 ) ( n∑ i=1 ϵiZ ′ i ) ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 . ( 81 ) For the first term , for every j ∈ [ m2 ] , define Hj to be the set of i ’ s in [ n ] where the jth column of Z ′i is non-zero , i.e . Hj = { i ∈ [ n ] : V ( 0 ) j x ′ i ≥ 0 } . Here , we use the crucial assumption that ( V − V ( 0 ) ) j T ϕ ( 0 ) ( xi ) = 0 , so we can drop the ϕ ( 0 ) ( xi ) term when x′i is multiplied to V − V ( 0 ) . Using this trick and applying Cauchy Schwarz , we bound the first term as : 1 n Eϵ sup W , V ∈S trace ( ( V − V ( 0 ) ) ( n∑ i=1 ϵiZ ′ i ) ) ≤ 1 n Eϵ∥V − V ( 0 ) ∥ sup W∈S √√√√ 1 m2 m2∑ j=1 ∥ ∑ i∈Hj ϵiϕ ( 2 ) ( xi ) ∥2 . Further using Jensen ’ s inequality : 1 n Eϵ sup W , V ∈S trace ( ( V − V ( 0 ) ) ( n∑ i=1 ϵiZ ′ i ) ) ≤ ∥V − V ( 0 ) ∥ n √√√√Eϵ 1 m2 m2∑ j=1 sup W∈S ∥ ∑ i∈Hj ϵiϕ ( 2 ) ( xi ) ∥2 . ( 82 ) Using Equation ( 111 ) of Lemma 6 ( note that we do not have the smoothing matrix W ρ here , so we are allowed to use this result ) , we obtain ∥⟨W −W ( 0 ) , Zki ⟩ − ϕ′ ( xi ) ∥ ≲ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 , where Zki ’ s are defined in Equation ( 318 ) . Plugging this back in ( 82 ) : Eϵ sup W∈S ∥ ∑ i∈Hj ϵiϕ ′ ( xi ) ∥2 ≤ Eϵ sup W∈S ( ∥ ∑ i∈Hj ϵi⟨W −W ( 0 ) , Zki ⟩∥+ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 ) 2 ≲ Eϵ sup W∈S ∥ ∑ i∈Hj ϵi⟨W −W ( 0 ) , Zki ⟩∥2 + C31 κ1 ( n3m33 m1λ0 ) 1/2 = Eϵ sup W∈S m3∑ k=1 ( trace ( ( W −W ( 0 ) ) ( ∑ i∈Hj ϵiZ k i ) ) ) 2 + C31 κ1 ( n3m33 m1λ0 ) 1/2 . ( 83 ) Now for every fixed dataset , with high probability over the randomness of W s , for every k1 ̸= k2 : ∣∣∣⟨∑ i∈Hj ϵiZ k1 i , ∑ i∈Hj ϵiZ k2 i ⟩ ∣∣∣ ≤ ∑ i1 , i2∈Hj ∣∣∣⟨Zk1i1 , Zk2i2 ⟩∣∣∣ = 1 m1 ∑ i1 , i2∈Hj ∣∣∣ m1∑ j=1 W sk1 , jW s k2 , j⟨xi1 , xi2⟩1 { W ( 0 ) j , xi1 ≥ 0 } 1 { W ( 0 ) j , xi2 ≥ 0 } ∣∣∣ But note that because ⟨xi1 , xi2⟩ ≤ 1 , the variables W sk1 , jW s k2 , j ⟨xi1 , xi2⟩1 { W ( 0 ) j , xi1 ≥ 0 } 1 { W ( 0 ) j , xi2 ≥ 0 } are subgaussian with parameter one with respect to the randomness of W s. Hence , with high probability over the randomness of W s , we get∣∣∣⟨∑ i∈Hj ϵiZ k1 i , ∑ i∈Hj ϵiZ k2 i ⟩ ∣∣∣ ≲ 1 m1 ∑ i1 , i2∈Hj √ m1 ≤ n2√ m1 . ( 84 ) Therefore , with high probability over the randomness of W ( 0 ) and W ′ and the dataset , we get Equation ( 84 ) . In order to get rid of the high probability argument on the dataset , we use the stronger Equation ( 319 ) in Lemma 46 which uniformly bounds ⟨Zk1 ( x ) , Zk2 ( x′ ) ⟩ by log ( m1 ) d/m1 for any x , x′ , which in turn gives∣∣∣⟨∑ i∈Hj ϵiZ k1 i , ∑ i∈Hj ϵiZ k2 i ⟩ ∣∣∣ ≤ ∑ i1 , i2∈Hj ∣∣∣⟨Zk1i1 , Zk2i2 ⟩∣∣∣ ≲ n2d log ( m1 ) √m1 , with high probability , independent of the choice of dataset . This bounds is slightly worse comapred to ( 84 ) , but still efficient for our purpose . Furthermore , a similar bound to Equation ( 84 ) can be obtained in a more adversarial situation when we also take maximum against the choice of the dataset . Note that the entries of ∑ i∈Hj ϵiZ k i for 1 ≤ k ≤ m3 can differ only in a sign . Therefore , their norms are all equal . Now suppose that Cj is the random variable of the norm of these variables : Cj : = ∥ ∑ i∈Hj ϵiZ k i ∥ . Then , by substituting rk = 1Cj ∑ i∈Hj ϵiZ k i in Lemma 40 , we get m3∑ k=1 ( trace ( ( W −W ( 0 ) ) ( ∑ i∈Hj ϵiZ k i ) ) ) 2 ≤ C2j ( 1 +m23O ( n2d log ( m1 ) √ m1Cj 2 ) ) ∥W −W ( 0 ) ∥2F ( 85 ) = ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F . ( 86 ) Now recall from Equation ( 80 ) , we have ∥ϕ′ ( xi ) ∥ ≤ γ1 . ( 87 ) Hence , we can apply Corollary 5.1 with ϕ ( 2 ) ( xi ) and C1 substituted by ϕ′ ( xi ) and γ1 respectively , to argue with high probability over the initialization , there exists a set P̃i such that for every i ∈ [ n ] and j /∈ P̃i , sign of VjTx′i is the same as V ( 0 ) j T ϕ ( 0 ) ( xi ) , and moreover , |P̃i| ≲ ( C21 ( m3κ21 ) ) 1/3 m2 . Now let H̄j = { i ∈ [ n ] : V ( 0 ) j ϕ ( 0 ) ( xi ) ≥ 0 } . Note that for j /∈ P̃ = ⋃ i P̃i , we have Hj = H̄j . Now note that the norm of each ∑ i∈Hj ϵiZ k i is at most one . for each index 1 ≤ ℓ ≤ m1d , as the random variables ∑ i∈H̄j ϵi ( Z k i ) ℓ are ∑ i∈H̄j ( Z k i ) 2 ℓ ≤∑ i∈ [ n ] ( Z k i ) 2 ℓ subgaussian , we have with probability at least 1 − 1n over the randomness of ϵi ’ s , for every 1 ≤ ℓ ≤ m1d and every 1 ≤ j ≤ m2 : ∣∣∣ ∑ i∈H̄j ϵi ( Z k i ) ℓ ∣∣∣ ≤√∑ i∈ [ n ] ( Zki ) 2 ℓ log ( m1dm2n ) , which implies for every j ∈ [ m2 ] : ∥ ∑ i∈H̄j ϵiZ k i ∥2 ≤ ∑ ℓ ∑ i∈ [ n ] ( Zki ) 2 ℓ log ( m1d ) ≤ n log ( m1dm2n ) . Name this event B , so P ( B ) ≤ 1 n . Note that although Hj might depend on the randomness of ϵi ’ s , H̄j does not , and if j /∈ P̃ , we obtain Cj = ∥ ∑ i∈H̄j ϵiZ k i ∥ ≤ √ n log ( m1dm2n ) . Moreover , note that we also have the following worse case bound : Cj = ∥ ∑ i∈Hj ϵiZ k i ∥ ≤ ∑ i∈H̄j ∥Zki ∥ ≤ n. Applying the last two inequalities into Equations ( 298 ) and ( 86 ) : Eϵ 1 m2 m2∑ j=1 sup W∈S ∥ ∑ i∈Hj ϵiϕ ′ ( xi ) ∥2 ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + 1 m2 m2∑ j=1 Eϵ sup W∈S m3∑ k=1 ( trace ( ( W −W ( 0 ) ) ( ∑ i∈Hj ϵiZ k i ) ) ) 2 ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + 1 m2 Eϵ1 { B } ∑ j∈P̃ ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F + 1 m2 Eϵ1 { B } ∑ j /∈P̃ ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F + 1 m2 Eϵ1 { Bc } m2∑ j=1 ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + ∥W −W ( 0 ) ∥2F [ |P̃ | m2 ( n2 + n2m23d log ( m1 ) √ m1 ) + 2 ( n+ n2m23d log ( m1 ) √ m1 ) ] ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + γ21 [ n3 ( C21 ( m3κ21 ) ) 1/3 + ( C21 ( m3κ21 ) ) 1/3n3m23d log ( m1 ) √ m1 + 2 n2m23d log ( m1 ) √ m1 + 2n ] . ( 88 ) Next , we analyze the term 1nEϵ supW∈S trace ( V ( 0 ) ( ∑n i=1 ϵiZ ′ i ) ) . Noting that ∥ϕ ( 0 ) ( xi ) ∥ ≲ κ1 √ m3 with high probability and using Equation ( 87 ) : ∥ n∑ i=1 ϵiZ ′ i∥F ≤ n∑ i=1 ∥Z ′i∥F ≤ n∑ i=1 ∥x′i∥ ≤ ∑ i ( ∥ϕ′ ( xi ) ∥+ ∥ϕ ( 0 ) ( xi ) ∥ ) ≲ n ( √ m3κ1 + γ1 ) . ( 89 ) Hence 1 n Eϵ sup W∈S trace ( V ( 0 ) ( n∑ i=1 ϵiZ ′ i ) ) = 1 n Eϵ sup W∈S n∑ i=1 ϵitrace ( V ( 0 ) Z ′i ) = 1 n Eϵ sup W∈S n∑ i=1 ϵi ( 1√ m2 m2∑ j=1 ajV ( 0 ) j , x ′ i1 { V ( 0 ) j , x ′ i ≥ 0 } ) . = 1 n Eϵ sup W∈S n∑ i=1 ϵi 1√ m2 aTσ ( V ( 0 ) x′i ) ≤ sup W∈S 1√ m2 aTσ ( V ( 0 ) x′i ) . But using Lemma 30 : LHS ≲ κ2 √ m3∥x′i∥ . Applying a similar bound as we did in Equation ( 89 ) on ∥x′i∥ : ∥x′i∥ ≤ ∥ϕ ( 0 ) ( xi ) ∥+ ∥ϕ′ ( xi ) ∥ ≲ κ1 √ m3 + γ1 . Substituting above , we get 1 n Eϵ sup W∈S trace ( V ( 0 ) ( n∑ i=1 ϵiZ ′ i ) ) ≤ κ2 √ m3 ( κ1 √ m3 + γ1 ) . ( 90 ) Finally , Substituting Equations ( 88 ) into ( 82 ) , then combining it with ( 90 ) into ( 81 ) , we obtain a bound on Rademacher complexity which holds w.h.p over both the randomness of the initialization and the dataset : R ( Gγ1 , γ2 ) |x , y ≲ √ C31 κ1 ( n3m33 m1λ0 ) 1/2 ( 91 ) + γ1γ2 n √ n3 ( C21 ( m3κ21 ) ) 1/3 + ( C21 ( m3κ21 ) ) 1/3n3m23d log ( m1 ) √ m1 + 2 n2m23d log ( m1 ) √ m1 + 2n ( 92 ) + κ2 √ m3 ( κ1 √ m3 + γ1 ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 . Having enough overparameterization , we have for every dataset ( x , y ) ( i.e . worst-case Rademacher complexity ) : R ( Gγ1 , γ2 ) |x , y ≤ 2γ1γ2/ √ n. ( 93 ) Note that for the bound ( 93 ) to hold , the overparameterization should be picked poly large in γ1 , γ2 , as well as in other basic parameters . However , noting Equations ( 51 ) and ( 58 ) in the proof of Theorem 3 , we set γ1 = 1 , γ2 ≥ Ω ( B , n , 1/γ0 ) in Theorem 3 , so γ1γ2 is at most poly in the basic parameters . Therefore , again the overparameterization can be picked polynomially large in the basic only parameters ( i.e . independent of γ1 , γ2 or ζ ) . A.12 CONSTRUCTING W ∗ , V ∗ This section consists of two subsections ; First , we prove a structural result for the first layer weights ( W ′ , V ′ ) that the algorithm visits , then construct a weight matrix W ∗ for the first layer with some good properties . Second , we do the same thing for the second layer ( however , the structure of the first and second layers are completely different ) . Through out this section , we assume we have the norm bounds ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 . Notably , we rely on a number of basic Lemmas more related to the representation power of the network , which we defer their proof into a later Appendix B.2 and refer to them here as needed . A.12.1 FIRST LAYER , CONSTRUCTION OF W ∗ Lemma 1 Suppose m1 ≥ 16n2m23/λ20 . Let Pi = { j ∈ [ m1 ] | |W ( 0 ) j xi| ≤ c2/ √ m1 } and P = ∪Pi . During SGD iterations , suppose we have ∥W ′∥F ≤ C1 . Then , for a value c2 satisfying 2C1 √ nm3/ √ λ0 ≤ c2 ≤ κ1λ0 √ m1/ ( 2n 2 ) , with high probability ∀i : |Pi| ≲ c2 √ m1/κ1 , and for j /∈ P , during the whole algorithm we have ∥W ′j∥ ≤ √ nm3C1√ m1λ0 + c2/ ( 4 √ m1 ) ≤ c2/ ( 2 √ m1 ) , c2/ √ m2 ≤ |W ( 0 ) j xi| . So the signs of neurons outside P never changes . In particular , we can set c2 as small as c2 = C1 √ nm3/ √ λ0 . In the rest of the proof ( i.e . other sections ) , we set c2 to this value . Proof of Lemma 1 Define the matrix Z̃ik = 1√ m1 ( W sk , jxi1 { ∀i : W ( 0 ) j Txi ≥ c2/ √ m1 } ) nj=1 . Let Pi be the set of indices j such that 1 { W ( 0 ) j Txi ≥ c2/ √ m1 } is zero . First of all , note that by Bernstein inequality : |Pi| ≤ c2 √ m1/κ1 +O ( √ c2 √ m1/κ1 + 1 ) ≲ c2 √ m1/κ1 . Now suppose that until the current iteration of the algorithm the assumption has been true , i.e . the signs of the neurons outside of P have never changed . As a result , due to the specific update of the SGD for both of the terms EZℓ ( fV ′ , W ′ ( x ) , y ) and ∥W ′∥2F , if we define W ′|P to be the restriction of W ′ to indices that are not in P ( i.e . the columns in P are equal to zero ) , then we can write W ′|P T = m3∑ k=1 n∑ i=1 αk , iZ̃ i k. ( 94 ) An issue here is that we also have some injected noise by PSGD intoW ′ which violates Equation ( 94 ) . To handle the injected noise as well , we define the subspace Φ′ of Rm1×d matrices to be the set of vectors with arbitrary rows for j ∈ [ m1 ] with j ∈ P , while restricted to the other rows j /∈ P in should be in the span of ( Z̃ik ) i , k . Then , we decompose W ′ into subspaces Φ′ and Φ′⊥ respectively as W ′ = W ′ ( 1 ) +W ′ ( 2 ) , where W ′ ( 1 ) ∈ Φ′ , W ′ ( 2 ) ∈ Φ′⊥ . Here , we want to prove ∥W ′ ( 1 ) j ∥ ≤ c2/ ( 4 √ m1 ) . We handle the ∥W ′ ( 2 ) j ∥ part in Appendix 45 . So instead of W ′|P in Equation ( 94 ) we consider W ′ ( 1 ) |P : W ′ ( 1 ) |P T = m3∑ k=1 n∑ i=1 αk , iZ̃ i k. ( 95 ) We handle the other part W ′ ( 2 ) in Appendix 45 . Now exactly similar to the drivation in Lemma 38 , we can state with high probability C21 ≥ ∥W ′∥2 ≥ ∥W ′ ( 1 ) ∥2 ≥ ∥W ′ ( 1 ) |P ∥2 ≥ m3∑ k=1 ∥ n∑ i=1 αk , iZ̃ i k∥2 −O ( nm3/ √ m1 ) ∑ k ∥αk∥2 . ( 96 ) Note that we are exploiting the fact that the norm ∥W ′∥ remains bounded by C1 . Now using a Hoeffding bound for matrix H∞′ defined below , we write : H∞′i1 , i2 : = Ew : N ( 0 , Rd ) 1 { ∀i : |w Txi| ≥ c2/ √ m1 } xTi1xi2 ( 1 { w Txi1 ≥ 0 } 1 { wTxi2 ≥ 0 } ) = Ew : N ( 0 , Rd ) ( 1 { wTxi1 ≥ 0 } 1 { wTxi2 ≥ 0 } ) xTi1xi2 ±O ( E1 { ∃i : |wTxi| ≤ c2/ √ m1 } ( 1 { wTxi1 ≥ 0 } 1 { wTxi2 ≥ 0 } ) ) xTi1xi2 = H∞i1 , i2 ±O ( nc2/ ( √ m1κ1 ) ∥xi1∥∥xi2∥ ) = H∞i1 , i2 ±O ( nc2/ ( √ m1κ1 ) ) . ( 97 ) Now opening Equation ( 96 ) and using the property c2 ≤ k1λ0 √ m1/ ( 2n 2 ) , we get LHS = ∑ k ∑ i1 , i2 αk , i1αk , i2⟨Z̃ i1 k , Z̃ i2 k ⟩ −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) = ∑ k ∑ i1 , i2 αk , i1αk , i2 ( H ∞′ i1 , i2 ±O ( 1/ √ m1 ) ) −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) ≥ ∑ k ∑ i1 , i2 αk , i1αk , i2H ∞ i1 , i2 ± ∥αk∥ 2 1O ( nc2/ √ m1κ1 ) −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) ≥ ∑ k αTkH ∞αk −O ( nc2/ √ m1κ1 ) ∑ k ∥αk∥21 −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) ≥ ∑ k αTkH ∞αk −O ( c2n2/ √ m1κ1 ) ∑ k ∥αk∥22 −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) = ( λ0 −O ( nm3/ √ m1 ) −O ( c2n2/ √ m1κ1 ) ) ∑ k ∥αk∥2 ≥ λ0/2 ∑ k ∥αk∥2 . For the last line to hold , we need enough overparameterization . This implies∑ k ∥αk∥2 ≲ C21/λ0 . Now again , exactly similar to the derivation in Lemma 38 , for j /∈ P we have ∥W ′ ( 1 ) j ∥ ≤ √ nm3/ √ m1 √∑ k ∥αk∥2 ≲ √ m3nC1/ √ m1λ0 , which completes most of the proof . For the rest , we are left to show that for the other part W ′ ( 2 ) , we have ∥W ′ ( 2 ) j ∥ ≤ c2/ ( 4 √ m1 ) , which we do in Appendix 45 . Lemma 2 Under condition m3n/ √ m1 ≤ λ0/4 , there exist matrices { W ∗k } m3 k=1 ∈ Rm1×d s.t . for every k ̸= k′ ∈ [ m3 ] and i ∈ [ n ] : ⟨W ∗k , Zik′⟩ = 0 , ∥W ∗k −W+k ∥ ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ , |⟨W ∗k , Zik⟩ − ⟨W+k , Z i k⟩| ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ . Furthermore , for k1 ̸= k2 : |⟨W ∗k1 , W ∗ k2⟩| ≤ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ∥Vk1∥H∞∥Vk2∥H∞ . ( 98 ) Proof of Lemma 2 Let W+k = ∑ i Vk , iZik . we want to compute the norm of the projection P ( W+k ) of W + k onto the subspace spanned by all Zik′ for k ′ ̸= k and i ∈ [ n ] : ∥P ( W+k ) ∥ 2 = ( ⟨W+k , Z i k′⟩ ) Tk′ ̸=k , i∈ [ n ] ( ⟨Zi1k1 , Z i2 k2 ⟩ ) −1 ( k1 , i1 ) , ( k2 , i2 ) ∈ [ m3 ] − { k } × [ n ] ( ⟨W+k , Z i k′⟩ ) k′ ̸=k , i∈ [ n ] , ( 99 ) where the first and third terms are vectors and the middle term is a matrix . Now note that for each k′ , k1 , k2 ̸= k , by Hoeffding inequality : ( ⟨Zi1k′ , Z i2 k′ ⟩ ) i1 , i2∈ [ n ] = H∞ + ( ±1/ √ m1 ) i1 , i2∈ [ n ] , ( 100 ) ⟨W+k , Z i k′⟩ = ⟨ ∑ i Vk , iZik , Zik′⟩ ≲ 1√ m1 ∑ i |Vk , i| ≤ √ n√ m1 ∥Vk∥ ≤ √ n√ m1λ0 ∥Vk∥H∞ . ( 101 ) Therefore , ∥ ( ⟨W+k , Z i k′⟩ ) Tk′ ̸=k , i∈ [ n ] ∥ ≤ n √ m3 m1λ0 ∥Vk∥H∞ . ( 102 ) Now Equation ( 100 ) implies for small enough m1 λmin ( ( ⟨Zi1k′ , Z i2 k′ ⟩ ) i1 , i2∈ [ n ] ) ≥ λ0/2 , ( 103 ) as long as λ0 ≥ 2n/m1 . Moreover , define Ã to be the block version of A′ = ( ⟨Zi1k1 , Z i2 k2 ⟩ ) −1 ( k1 , i1 ) , ( k2 , i2 ) ∈ [ m3 ] − { k } × [ n ] , i.e . for k1 = k2 they are the same but for k1 ̸= k2 Ã is zero . Then λmin ( Ã ) ≥ λ0/2 , because the eigenvalues of each block is at least λ0/2 using Equation ( 103 ) . But note that ∥A′ − Ã∥2 ≤ ∥A′ − Ã∥F ≤ m3n/ √ m1 . So as long as m3n/ √ m1 ≤ λ0/4 , we have λmin ( A ) ≥ λ0/4 . Combining this fact with Equation ( 102 ) and plugging it into Equation ( 99 ) , we obtain ∥P ( W+k ) ∥ 2 ≲ n 2m3 m1λ20 ∥Vk∥2H∞ . Now define W ∗k =W + k − P ( W + k ) . Then ∥W ∗k −W+k ∥ = ∥P ( W + k ) ∥ ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ , |⟨W ∗k −W+k , Z i k⟩| ≤ ∥P ( W+k ) ∥∥Z i k∥ ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ . Furthermore , note that W ∗k2 is orthogonal to W + k1 for k1 ̸= k2 , so |⟨W ∗k1 , W ∗ k2⟩| = |⟨W + k1 − P ( W+k1 ) , W ∗ k2⟩| = |⟨P ( W+k1 ) , W + k2 − P ( W+k2 ) ⟩| ≤ ∥P ( W+k1 ) ∥∥W + k2 − P ( W+k2 ) ∥ ≤ ∥P ( W+k1 ) ∥ ( ∥W + k2 ∥+ ∥P ( W+k2 ) ∥ ) . ( 104 ) But note that ∥W+k ∥ = ∥ ∑ i Vk , iZik∥ ≤ ∑ i |Vk , i|∥Zik∥ ≤ ∑ i |Vk , i| ≤ √ n∥Vk∥2 . ≤ √ n λ0 ∥Vk∥H∞ . Therefore , we can bound Equation ( 104 ) as : |⟨W ∗k1 , W ∗ k2⟩| ≤ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ∥Vk1∥H∞∥Vk2∥H∞ . Lemma 3 There exists a matrix W+2k such that for every j ∈ P , W +2 k j = 0 , and |trace ( W+2k Z k i ) − x̄i , k| ≤ C1 √ m3n 2/ ( λ0κ1 √ m1 ) ∥Vk∥H∞ . Proof of Lemma 3 Define W+2k to be equal to W + k for j /∈ P and equal to zero vector otherwise . Then , by Lemma 38 : ( note that |Pi| ≤ C1 √ nm3 √ m1/ ( √ λ0κ1 ) ) |trace ( W+k Z k i ) − trace ( W+2k Z k i ) | ≤ 1/ √ m1 ∑ j∈P |W+k jxi| ≤ |P |√ m1 ∥W+k ∥ ≤ √ nm3/ ( m1 √ λ0 ) |P |∥Vk∥H∞ ≤ C1m3n2/ ( ( λ0κ1 √ m1 ) ∥Vk∥H∞ . Combining this with Lemma 37 , the desired result follows . Lemma 4 Under conditionm3n/ √ m1 ≤ λ0/4 , there exist matrixW ∗k ’ s exactly satisfying the same conditions in Lemma 2 but with respect to W+2k instead of W + k , and moreover , for j ∈ P we have W ∗k j = 0 . Proof of Lemma 4 We can repeat the exact same procedure of Lemma 2 for W+2k . Using the bound in Equation ( 97 ) , we have ( ⟨Z̃i1k′ , Z̃ i2 k′ ⟩ ) i1 , i2∈ [ n ] = H ′∞ +O ( ±1/ √ m1 ) i1 , i2∈ [ n ] = H∞ + ( ±nc2/ √ m1κ1 ) i1 , i2∈ [ n ] +O ( ±1/ √ m1 ) i1 , i2∈ [ n ] = H∞ + ( ±nc2/ √ m1κ1 ) i1 , i2∈ [ n ] , so as long as n2c2/ √ m1κ1 = n 2C1 √ nm3/ ( κ1 √ m1λ0 ) ≤ λ0/2 , with similar argument as in Lemma 2 , we get λmin ( ( ⟨Z̃i1k′ , Z̃ i2 k′ ⟩ ) i1 , i2∈ [ n ] ) ≥ λ0/2 . Moreover , ⟨W+2k , Z̃ i k′⟩ = ⟨ ∑ i Vk , iZ̃ik , Z̃ik′⟩ ≲ 1√ m1 ∑ i |Vk , i| ≤ √ n√ m1 ∥Vk∥ ≤ √ n√ m1λ0 ∥Vk∥H∞ . Thus , using the same argument as before the proof is complete . Lemma 5 Suppose m1 ≥ n7m3/λ0 . During SGD , suppose we are currently at ( V ′ , W ′ ) with W ′ ≤ C1 . For any matrix W1 , we denote the signs of the first layer imposed by W1 by DW1 , xi . Then with high probability , there exists W ∗ = ∑ k∈ [ m3 ] W ∗ k such that W ∗ k ’ s is orthogonal to all other Z i k′ ’ s for k ′ ̸= k , and for every i ∈ [ n ] , we have : ∥ 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗xi − x̄i∥∞ ≲ nm3√ m1λ0 [ 1 + nC1 κ1 ] ∥Vk∥H∞ : = ℜ∥Vk∥H∞ . Moreover , we have ∥W ∗j ∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ + n √ m3 λ0 √ m1 ( ∑ k ∥Vk∥H∞ ) : = ϱ √ m3 m1 , ( 105 ) Particularly , for any diagonal sign matrix Σ ∈ Rm3×m3 , we have ∥W ∗Σ∥2F ≤ ( n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) + ( 1 +O ( n/ ( λ0 √ m1 ) + n2m3 λ20m1 ) ) ) ∑ k ∥Vk∥2H∞ . ( 106 ) which , by having enough overparameterization , implies ∥W ∗Σ∥F ≤ √ 2 ∑ k ∥Vk∥2H∞ = √ 2ζ1 , ( 107 ) where W ∗Σ : = m3∑ k=1 ΣkW ∗ k . ( 108 ) Moreover , we have 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗ Σxi = Σ 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗xi . ( 109 ) Proof of Lemma 5 From Lemma 3 , we have |x̄i , k − trace ( W+2k Z i k ) | ≤ C1m3n2/ ( λ0κ1 √ m1 ) ∥Vk∥H∞ . Combining this with the result of Lemma 4 , we get : |x̄i , k − trace ( W ∗kZik ) | ≲ [ n√m3 λ0 √ m1 + C1m3n 2 λ0κ1 √ m1 ] ∥Vk∥H∞ = nm3√ m1λ0 [ 1 + nC1 κ1 ] ∥Vk∥H∞ . ( 110 ) On the other hand , based on the property that W ∗k j = 0 for j ∈ P and its orthogonal property from Lemma 4 , for j ∈ P we get 1√ m1 W skDW ( 0 ) +W ′ , xiW ∗xi = 1√ m1 W skDW ( 0 ) , xiW ∗xi = trace ( W ∗Zik ) = trace ( W ∗ kZ i k ) = 1√ m1 W skDW ( 0 ) , xiW ∗ k xi , which combined with Equation ( 110 ) completes the proof . From the above , Equation ( 109 ) is also clear . Finally , note that by Lemma 38 we have ∥W+2j∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ . which Combined with Lemma 4 implies ∥W ∗j ∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ + n √ m3 λ0 √ m1 ( ∑ k ∥Vk∥H∞ ) : = ϱ √ m3 m1 , while the other claims follows from Lemma 39 and Lemma 4 , combined with Equation ( 98 ) : ∥W ∗Σ∥2F ≤ ∑ k ∥W ∗k ∥2 + ∑ k1 ̸=k2 |⟨W ∗k1 , W ∗ k2⟩| ≤ ∑ k ∥W ∗k ∥2 + n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ( ∑ k ∥Vk∥H∞ ) 2 ≤ ∑ k ∥W ∗k ∥2 + n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) √ n ( ∑ k ∥Vk∥2H∞ ) ≤ ∑ k ∥W ∗k ∥2 + n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 ≤ n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 + ∑ k ∥W+2k ∥ 2 + n2m3 λ20m1 ∑ k ∥Vk∥2H∞ ≤ n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 + ∑ k ∥W+k ∥ 2 + n2m3 λ20m1 ∑ k ∥Vk∥2H∞ ≲ n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 + ( 1 +O ( n/ ( λ0 √ m1 ) + n2m3 λ20m1 ) ) ζ1 . Next , we move on to construct V ∗ for the second layer . A.12.2 SECOND LAYER , CONSTRUCTION OF V ∗ In this section , we present a couple of lemmas that step by step lead to the construction of V ∗ . we remind the reader that ϕ ( 0 ) ( xi ) is the output of the first layer at initialization weights , ϕ′ ( xi ) and ϕ ( 2 ) ( xi ) are the changes in the output of the first layer when W ′ and W ′ + W ρ are added , respectively , and finally ϕ∗ ( xi ) is the optimal features that are generated by the matrix W ∗ but with the sign pattern of W ( 0 ) +W ′ , i.e . ϕ∗ ( xi ) = 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗xi . We also define x′i as x′i = ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) = 1√ m1 W sσ ( ( W ( 0 ) +W ′ +W ρ ) xi ) . To begin , we state a lemma to bound the magnitude of ∥ϕ′ ( xi ) ∥ , given that the norm of W ′ is bounded by C1 and the sign pattern Sgn ( ( W ( 0 ) +W ′ ) xi ) satisfies condition stated for the set of indices P in Lemma 1 . Later on , we exploit this Lemma in Lemma 33 to state bounds for ∥ϕ ( 2 ) ( xi ) ∥ . Lemma 6 Let the matrix W ′ with norm bound ∥W ′∥ ≤ C1 , such that the signs of ( W ( 0 ) j +W ′j ) xi and W ( 0 ) j xi can be different only for j ∈ P , for P defined in Lemma 1 . ( Note that for W ′ at every step of the algorithm , this is automatically satisfied by Lemma 1 ) Then ∥ϕ′ ( xi ) ∥ ≤ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 + ( 1 +O ( m23/ √ m1 ) ) C1 . Particularly for large enough m1 compared to n , m3 , λ0 , κ1 , C1 , we have ∥ϕ′ ( xi ) ∥ ≲ C1 . Proof of Lemma 6 We write |ϕ′k ( xi ) − ⟨W ′ , Zik⟩| ≤ 2/ √ m1 ∑ j∈P |W ′jxi| ≤ 2/ √ m1 ∑ j∈P ∥W ′j∥ ≤ 2 √ |P |/ √ m1∥W ′∥F ≤ 2C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 , ( 111 ) where the last line follows from the bound on |P | from Lemma 1 . On the other hand , because by Hoeffding we know that ⟨Zik , Zik′⟩ ≲ 1/ √ m1 by Lemma 40 , we get m3∑ k=1 ⟨W ′ , Zik⟩2 ≤ ( 1 +O ( m23/ √ m1 ) ) ∥W ′∥2F ≤ ( 1 +O ( m23/ √ m1 ) ) C 2 1 . Combining this with Equation ( 111 ) , we get ∥ϕ′ ( xi ) ∥ ≤ √∑ k |ϕ ( 2 ) k ( xi ) − ⟨W ′ , Zik⟩|2 + √∑ k ⟨W ′ , Zik⟩2 ≤ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 + ( 1 +O ( m23/ √ m1 ) ) C1 . ( 112 ) Next , we prove a structural lemma regarding the sign pattern in the second layer when we feed in x′i to it , with the important message that the dominance of sign patterns are specified by ϕ ( 0 ) ( xi ) . Lemma 7 Suppose we have m3κ21 ≳ C21 , κ2 √ m2 ≥ C2 , and m1 satisfies the condition on Lemma 6 . If we have the condition ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 , which happens under the high probability event Ec defined in Lemma 33 , then for every i ∈ [ n ] , there exist a subset P̃i which might depend on W ( 0 ) , V ( 0 ) , W ′ , V ′ , such that |P̃i| ≲ ( ( C21 ( m3κ21 ) ) 1/3 + ( c3 + C21 c23m3κ 2 1 ) ( C22 κ22m2 ) 1/3 ) m2 . Moreover , for every i ∈ [ n ] , for j /∈ P̃i , : 2 3 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≥ |V ( 0 ) j ϕ ( 2 ) ( xi ) |+ |V ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | , |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ ( κ2 m2 ) 1/3C 2/3 2 c3∥ϕ ( 0 ) ( xi ) ∥ , |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ ( κ2 m2 ) 1/3C 2/3 2 c3∥x′i∥ . Proof of Lemma 7 By assumption , we know that during the algorithm , we have ∥V ′∥ ≤ C2 . Also , we know by Lemma 33 that under Ec : ∥ϕ ( 2 ) ( xi ) ∥ ≤ 2C1 . Define the set P ′i = { j| |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ c3 ( κ2 m2 ) 1/3C 2/3 2 ∥ϕ ( 0 ) ( xi ) ∥ } ( 113 ) and P ′ = ∪P ′i . We have P ( |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ c3 ( κ2 m2 ) 1/3C 2/3 2 ∥ϕ ( 0 ) ( xi ) ∥ ) ≤ c3 ( κ2 m2 ) 1/3C 2/3 2 /κ2 , so by Bernstein , with high probability : |P ′i | ≤ m2C 2/3 2 c3 ( κ2 m2 ) 1/3/κ2 + √ m2C 2/3 2 c3 ( κ2 m2 ) 1/3/κ2 + 1 ≲ c3m2C2/32 ( κ2 m2 ) 1/3/κ2 , so with high prob . |P ′i | ≲ c3C 2/3 2 ( m2 κ2 ) 2/3 . ( 114 ) On the other hand , Note that ϕ ( 0 ) k ( xi ) = m1∑ j=1 1/ √ m1W s k , jσ ( W ( 0 ) j xi ) ( 115 ) is subGaussian with parameter σ2 = O ( 1/m1 ∑ j σ ( W ( 0 ) j xi ) 2 ) . Furthermore , note that if we compute the variance of ϕ ( 0 ) k ( xi ) with respect to the randomness of W s : Eϕ ( 0 ) k ( xi ) 2 = 1/m1 m1∑ j=1 σ ( W ( 0 ) j xi ) 2 : = ℵ which itself concentrates around 1/2κ21∥xi∥2 = 1/2κ21 by another Bernstein , i.e . ℵ = 1/2κ21 ( 1 ± O ( 1/ √ m1 ) ) . Therefore , by concentration of subexponential variables ( Bernstein ) , it is not hard to see that the squared norm of the vector ϕ ( 0 ) ( xi ) is ( m3κ41 , κ2 ) -subexponential and concentrates around m3ℵ , i.e . ∥ϕ ( 0 ) ( xi ) ∥2 = m3ℵ ±O ( κ21 √ m3 ) = m3κ 2 1/2±O ( m3κ21/ √ m1 ) ±O ( κ21 √ m3 ) , ( 116 ) with high probability . Combining this with the fact that ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 implies with high probability : ∥ϕ ( 0 ) ( xi ) ∥ ∥ϕ ( 2 ) ( xi ) ∥ ≳ √ m3κ1 C1 . ( 117 ) Now define P ′′i = { j| |V ′j x′i| ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 } . If j ∈ P ′′i − P ′i , then by Equation ( 117 ) , with high probability ∥V ′j ∥∥ϕ ( 2 ) ( xi ) ∥ ≥ |V ′jϕ ( 2 ) ( xi ) | = |V ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | = |V ′j x′i| ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 ≳ c3 ( κ2 m2 ) 1/3C 2/3 2 ∥ϕ ( 0 ) ( xi ) ∥ , or ∥V ′j ∥2 ≳ c23 ( κ2 m2 ) 2/3C 4/3 2 m3κ 2 1 C21 . But note that ∥V ′∥2F ≤ C22 by our assumption , which implies |P ′′i − P ′i | ≲ C22/ [ c23 ( κ2 m2 ) 2/3C 4/3 2 m3κ 2 1 C21 ] = C 2/3 2 C 2 1 c23m3κ 2 1 ( m2 κ2 ) 2/3 . ( 118 ) Now combining Equations ( 114 ) and ( 118 ) , we finally obtain |P ′′i | = |P ′′i − P ′i |+ |P ′i | ≲ ( c3 + C21 c23m3κ 2 1 ) C 2/3 2 ( m2 κ2 ) 2/3 . Now define the set P ′′′i = { j| |V ( 0 ) j ϕ ( 2 ) ( xi ) | ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 } . ( 119 ) Note that for every j ∈ [ m2 ] , V ( 0 ) j ϕ ( 0 ) ( xi ) is gaussian with variance ∥ϕ ( 0 ) ( xi ) ∥ over the randomness of V ( 0 ) j , so P ( V ( 0 ) j ϕ ( 0 ) ( xi ) ≤ ακ2∥ϕ ( 0 ) ( xi ) ∥ ) ≲ α . Therefore , if we define the set Qi = { j ∈ [ m2 ] | |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ ακ2∥ϕ ( 0 ) ( xi ) ∥ } , then for large enough m2 , by Bernstein with high prob . : |Qi| ≲ αm2 . ( 120 ) Now note that ϕ ( 0 ) ( xi ) is fixed during the algorithm . On the other hand , by random matrix theory , we know that with high probability , the eigenvalues of the matrix V ( 0 ) are in ( κ2 ( √ m2 −√ m3 ) , κ2 ( √ m2 + √ m3 ) ) . Therefore , even if the vector ϕ ( 2 ) ( xi ) is picked adversarialy ( because it keeps changing during the algorithm ) , we get that with high probability over the randomness of V ( 0 ) : |V ( 0 ) ϕ ( 2 ) ( xi ) ∥2 ≤ κ22 ( √ m2 + √ m3 ) 2∥ϕ ( 2 ) ( xi ) ∥2 ≲ κ22m2∥ϕ ( 2 ) ( xi ) ∥2 . ( 121 ) Moreover , because ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 and from Equation ( 116 ) , with high probability over the randomness of W ( 0 ) : ∥ϕ ( 0 ) ( xi ) ∥ ∥ϕ ( 2 ) ( xi ) ∥ ≳ √ m3κ1 C1 . This means that for j ∈ P ′′′i −Qi , combining these inequalities we conclude with high probability |V ( 0 ) j ϕ ( 2 ) ( xi ) | ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 ≳ ακ2∥ϕ ( 0 ) ( xi ) ∥ ≥ ακ2 √ m3κ1 C1 ∥ϕ ( 2 ) ( xi ) ∥ , which combined with ( 121 ) implies ∥P ′′′i ∥ ≲ m2C 2 1 m3κ21α 2 . Balancing this term with the one in Equation ( 120 ) , we set α : = C 2/3 1 m 1/3 3 κ 2/3 1 , which implies |P ′′′i | ≲ |P ′′′i −Qi|+ |Qi| ≤ ( C21 ( m3κ21 ) ) 1/3 m2 . Defining P̃i = P ′′i ∪ P ′′′i , we finally get |P̃i| ≲ ( ( C21 ( m3κ21 ) ) 1/3 + ( c3 + C21 c23m3κ 2 1 ) C 2/3 2 ( 1 κ22m2 ) 1/3 ) m2 . Clearly by the definition of P ′′i and P ′′′ i the proof is complete . Corollary 5.1 Under the condition ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 ( which happens under the event Ec defined in Lemma 33 ) , setting c3 : = C 2/3 1 m 1/3 3 κ 2/3 1 ( κ22m2 C22 ) 1/3 in the previous Lemma , we obtain ∀i ∈ [ n ] : |P̃i| ≲ ( C21 ( m3κ21 ) ) 1/3 m2 . Also for j /∈ P̃i , the conditions in ( 113 ) and ( 119 ) becomes the same as |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ κ2 C 2/3 1 m 1/3 3 κ 2/3 1 ∥ϕ ( 0 ) ( xi ) ∥ . Hence , for every i ∈ [ n ] and for j /∈ P̃i , with high probability : 2 3 |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≥ |W ( 0 ) j ϕ ( 2 ) ( xi ) |+ |W ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | , ( 122 ) |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ κ2 C 2/3 1 m 1/3 3 κ 2/3 1 ∥ϕ ( 0 ) ( xi ) ∥ ≳ κ2 ( √ m3κ1C 2 1 ) 1/3 , ( 123 ) |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ C 2/3 1 m 1/3 3 κ 2/3 2 ∥x′i∥ . ( 124 ) Next , we state concentration result for the gram matrix of ϕ ( 0 ) ( xi ) ’ s . Lemma 8 For every i1 , i2 ∈ [ n ] , with high probability over the randomness of W ( 0 ) and V ( 0 ) we have ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 + √ m3κ 2 1 ) . Proof of Lemma 8 First , we compute the expectation : E⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = 1/m1 ∑ j1 , j2∈ [ m1 ] ∑ k∈ [ m3 ] EW sk , j1W s k , j2σ ( W ( 0 ) j1 xi1 ) σ ( W ( 0 ) j2 xi2 ) = 1/m1 ∑ j1 ̸=j2 E ∑ k∈ [ m3 ] W sk , j1W s k , j2σ ( W ( 0 ) j1 xi1 ) σ ( W ( 0 ) j2 xi2 ) +m3/m1 ∑ j∈ [ m1 ] σ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) . But σ ( W ( 0 ) j1 xi1 ) σ ( W ( 0 ) j2 xi2 ) is ( m1κ 4 1 , κ 2 1 ) -sub-exponential , so∑ j∈ [ m1 ] σ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) = m1Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( √ m1κ 2 1 ) , which means with high probability : E⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 ) . On the other side , we know that ϕ ( 0 ) k ( xi1 ) is subgaussian with parameters σ 2 = 1/m1 ∑ j ( W ( 0 ) j xi1 ) 2 : = ℵ1 and σ2 = 1/m1 ∑ j ( W ( 0 ) j xi2 ) 2 : = ℵ2 respectively . On the other hand , we know that by Bernstein w.h.p ℵ1 = 1/2κ21 ( 1±O ( 1/ √ m1 ) ) , ℵ2 = 1/2κ21 ( 1±O ( 1/ √ m1 ) ) . Hence , ϕ ( 0 ) k ( xi1 ) ϕ ( 2 ) k ( xi2 ) is ( ℵ1ℵ2 , √ ℵ1ℵ2 ) -subexponential , and so ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ is ( m3ℵ1ℵ2 , √ ℵ1ℵ2 ) -subexponential . Therefore , applying another Bernstein on the top , we get ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = E⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ ±O ( √ m3 √ ℵ1ℵ2 ) = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 ) ± √ m3κ 2 1 2 ( 1±O ( 1/ √ m1 ) ) = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 + √ m3κ 2 1 ) . Now we define the matrix Li ∈ Rm3×m2 , with its jth column Li , j equal to aj√ m2 1 { V ( 0 ) j ϕ ( 0 ) ( xi ) ≥ 0 } ϕ∗ ( xi ) . First , we state the following lemma which characterize a concentration result for the gram matrix of ( Li ) n i=1 . Lemma 9 With high probability , we have the following approximation : ⟨Li1 , Li2⟩ = ⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ [ F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ±O ( m−1/41 +m −1/4 2 +m −1/4 3 ) ] . Proof of Lemma 9 By Hoeffding : ⟨Li1 , Li2⟩ = 1/m2 ∑ j∈m2 ϕ∗ ( xi1 ) Tϕ∗ ( xi2 ) 1 { V ( 0 ) j ϕ ( 1 ) ( xi1 ) ≥ 0 } 1 { V ( 0 ) j ϕ ( 1 ) ( xi2 ) ≥ 0 } = ϕ∗ ( xi1 ) Tϕ∗ ( xi2 ) ( E1 { V ( 0 ) j ϕ ( 1 ) ( xi1 ) ≥ 0 } 1 { V ( 0 ) j ϕ ( 1 ) ( xi2 ) ≥ 0 } ±O ( 1/ √ m2 ) ) = ϕ∗ ( xi1 ) Tϕ∗ ( xi2 ) ( F2 ( ⟨ϕ ( 1 ) ( xi1 ) , ϕ ( 1 ) ( xi2 ) ⟩/ ( ∥ϕ ( 1 ) ( xi1 ) ∥∥ϕ ( 1 ) ( xi2 ) ∥ ) ) ±O ( 1/ √ m2 ) ) , where recall F2 ( x ) = 1/4 + arcsin ( x ) /2π , measures the angle between two unit vectors based on their dot product . Now notice that according to Lemma 8 , with high probability : ⟨Li1 , Li2⟩/⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ = F2 ( m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( ( m3/√m1 +√m3 ) κ21 ) √ ( m3Eσ ( W ( 0 ) j xi1 ) 2 ±O ( ( m3/ √ m1 + √ m3 ) κ 2 1 ) ) ( m3Eσ ( W ( 0 ) j xi2 ) 2 ±O ( ... ) ±O ( 1/ √ m2 ) ) = F2 ( F3 ( ⟨xi1 , xi2⟩ ) ±O ( 1/√m1 + 1/√m3 ) 1/2±O ( 1/ √ m1 + 1/ √ m3 ) ±O ( 1/ √ m2 ) ) , where recall F3 : [ −1 , +1 ] → [ −1/2 , 1/2 ] is defined as : F3 ( x ) : = √ 1− x2 2π + x 4 + x arcsinx 2π . It is easy to see F3 has the property that for unit vectors x1 , x2 and w sampled as standard normal : F3 ( ⟨x1 , x2⟩ ) = Eσ ( wTx1 ) σ ( wTx2 ) . But because |F3 ( . ) | = O ( 1 ) , we have ⟨Li1 , Li2⟩/⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ = F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ±O ( 1/ √ m1 + 1/ √ m2 + 1/ √ m3 ) ) . Now notice that the derivative of F2 , i.e . 1/2π √ 1− x2 is increasing in the interval ( 0 , 1 ) , so for a fixed δ , the maximum of |F2 ( x ) − F2 ( x− δ ) | happens at x = 1 . On the other hand , by writing the first order approximation of arcsin ( 1 − t2 ) around t = 0 and upper bounding its derivative in the interval [ 0 , 1 ] , we get that for 0 ≤ δ ≤ 1 : arcsin ( 1− δ ) ≥ arcsin ( 1 ) − 2 √ δ . Therefore , F2 ( x± δ ) = F2 ( x ) ±O ( √ δ ) . Hence : ⟨Li1 , Li2⟩/⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ = F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ±O ( √ 1/ √ m1 + 1/ √ m2 + 1/ √ m3 ) = F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ±O ( m−1/41 +m −1/4 2 +m −1/4 3 ) , which completes the proof . Finally , we are ready to construct the weights V ∗ for the second layer . A.12.3 CONSTRUCTION OF V ∗ Lemma 10 Let ℜ = n √ m3√ m1λ0 [ 1 + nC1 κ1 ] . Suppose we have the condition that for every k ∈ [ m3 ] : max k ∥Vk∥ ≤ ξ , ( 125 ) where recall the definition of Vk in Equation ( 60 ) . We assume enough overparameterization to make sure ℜ < 1 . Recall for the matrix A defined by A = ( ⟨x̄i1 , x̄i2⟩F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ) 1≤i1 , i2≤n , ( 126 ) we have ( f∗ ( xi ) ) n i=1 TA−1 ( f∗ ( xi ) ) n i=1 ≤ ζ2 . Then , there exists weight matrix V ∗ which only depends on the random initializations W ( 0 ) , V ( 0 ) ( e.g . not on V ′ and W ′ ) for the second layer , such that having enough overparameterization ∥V ∗∥2F ≤ 2ζ2 , ( 127 ) and for every j ∈ [ m2 ] : ∥V ∗j ∥∞ ≤ ( 1 + ℜ ) n √ nζ2√ m2 ξ : = ϱ3ξ/ √ m2 , ( 128 ) ∥V ∗j ∥2 ≤ 1√ m2 n ( 1 + ℜ ) √ ζ2 ∑ k ∥Vk∥2H∞ λ0 : = ϱ2/ √ m2 , ( 129 ) and further under the high probability event Ec defined in Lemma 33 : | 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) − f∗ ( xi ) | ≲ ( C1√ m3κ1 ) 1/3 ( 1 + ℜ ) √ ζ2 ∑ k ∥Vk∥2H∞ : = ℜ3 . ( 130 ) Proof of Lemma 10 Let V ∗ = n∑ i=1 V∗i Li , be the minimum norm vector which maps Li ’ s to f∗ ( xi ) ’ s . As a result , for the matrix L = ( ⟨Li1 , Li2⟩ ) i1 , i2 it is easy to see ∥V ∗∥2F = ( f∗ ( xi ) ) ni=1TL−1 ( f∗ ( xi ) ) ni=1 . Now combining Lemmas 5 and 41 , we get ∥ϕ∗ ( xi ) ∥∞ ≤ ( 1 + ℜ ) ξ , ( 131 ) ∥ϕ∗ ( xi ) ∥ ≤ ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ , ( 132 ) and ⟨|ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ − ⟨x̄i1 , x̄i2⟩| ≤ ( 2ℜ+ ℜ2 ) ∑ k ∥Vk∥2H∞ . Now by Lemma 9 : |⟨Li1 , Li2⟩ −Ai1 , i2 | ≲ ( 2ℜ+ ℜ2 ) ( ∑ k ∥Vk∥2H∞ ) ∣∣∣F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ∣∣∣+ ⟨x̄i1 , x̄i2⟩ ( m−1/41 +m−1/42 +m−1/43 ) + other cross term . By applying ⟨x̄i1 , x̄i2⟩ ≤ ∥x̄i1∥∥x̄i2∥ we get LHS ≤ ( ∑ k ∥Vk∥2H∞ ) ( ( 2ℜ+ ℜ2 ) ∣∣∣F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ∣∣∣+ ( m−1/41 +m−1/42 +m−1/43 ) ) ≲ ( ∑ k ∥Vk∥2H∞ ) ( ℜ+m−1/41 +m −1/4 2 +m −1/4 3 ) . Therefore , ∥A− ( ⟨Li1 , Li2⟩ ) i1 , i2 ∥2 ≤ ∥A− ( ⟨Li1 , Li2⟩ ) i1 , i2 ∥F ≤ n ( ∑ k ∥Vk∥2H∞ ) ( ℜ+m−1/41 +m −1/4 2 +m −1/4 3 ) : = ℜ2 . Note that ℜ2 naturally goes to zero ( with poly dependence ) as ℜ → 0 and m1 , m2 , m3 are large enough . Now if all of the eigenvalues of the matrix A are Ω ( 1/n2 ) , then if we overparameterize enough such that ℜ2 = O ( 1/n2 ) with small enough constant so that ℜ2 is less than half of the smallest eigenvalue of A , then for the ith eigenvalue λi of A and L we can write λi ( L ) ≥ λi ( A ) −ℜ2 ≥ λi ( A ) /2 , so λi ( L −1 ) ≤ 2λi ( A−1 ) , which implies the property ∥V ∗∥2F ≤ 2ζ2 . ( 133 ) However , A might have very small eigenvalues . To remedie this , we use Lemma 42 ; we can substitute f∗ with some f̄∗ such that Rn ( f̄∗ ) ≤ 2Rn ( f∗ ) + B2 n , ( 134 ) f̄∗ T A−1f̄∗ ≤ f∗TA−1f∗ , ( 135 ) where f̄∗ is on the subspace of eigenvectors of A whose eigenvalues are larger than Ω ( 1/n2 ) . But it is easy to check that in the context of Theorem 3 , such substitution results in a f̄∗TA−1f̄∗ ≤ f∗TA−1f∗ ≤ ζ and v̄ ( f̄∗ ) parameter ( as defined in ( 43 ) ) with respect to f̄∗ which satisfies ν̄/2 ≤ ν . This enables us to use remark ? ? with respect to f̄∗ . Note that the algorithm is with respect to the setting ν , however we want to exploit generalization bound with respect to f̄∗ whose parameter is ν̄ as it enables us to use our analysis in this Lemma . Furthermore , note that using Equation ( 134 ) we can further upper bound the empirical risk of f̄∗ with that of f∗ , which makes it straightforward to derive a similar generalization bound as in ( 47 ) with respect to f∗ , of course with a change of constants . Therefore , without loss of generality we can use substitute f∗ by f̄∗ and still obtain Equation ( 133 ) . On the other hand , the definition of V ∗ implies 1√ m2 aTDV ( 0 ) , xiV ∗ϕ∗ ( xi ) = f ∗ ( xi ) . But note that by Corollary 5.1 , under the high probability event Ec defined in Lemma 33 , DV ( 0 ) , xi and DV ( 0 ) +V ′ , xi can only be different in the index set P̃i and |P̃i| ≲ ( C21 ( m3κ21 ) ) 1/3 m2 , Therefore , for all i ∈ [ n ] : | 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) − 1√ m2 aTDV ( 0 ) , xiV ∗ϕ∗ ( xi ) | ≤1/ √ m2 ∑ j∈P̃ |V ∗j ϕ∗ ( xi ) | ≤1/ √ m2 ∑ j∈P̃ ∥V ∗j ∥∥ϕ∗ ( xi ) ∥ ≤ √ |P̃ | √ m2 ∥V ∗∥ ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ ≲ ( C1√ m3κ1 ) 1/3√ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ , which proves the first claim . On the other hand , we get : 2ζ2 ≥ ∥V ∗∥2F ≥ VTLV . But because λmin ( L ) ≳ 1/n2 , we get VTLV ≳ ∥V∥22/n2 , which implies ∥V∥2 ≲ n √ ζ2 . But now using Equation ( 131 ) , we can write |V ∗j , k| ≤ n∑ i=1 |Vi||Lij , k| ≤ 1√ m2 ∥ϕ∗ ( xi ) ∥∞ ∑ i |Vi| ≲ ( 1 + ℜ ) ξ√ m2 ∑ i |Vi| ≤ ( 1 + ℜ ) ξ √ n∥V∥/ √ m2 ≲ ( 1 + ℜ ) n √ nζ2√ m2 ξ , which proves the other part . Moreover , ∥V ∗j ∥2 ≤ ∥V∥2 1√ m2 ( ∑ i ∥ϕ∗ ( xi ) ∥22 ) ≤ 1√ m2 n ( 1 + ℜ ) √ ζ2 ∑ k ∥Vk∥2H∞/λ0 . ( 136 ) A.13 EXISTENCE OF A GOOD DIRECTION Our aim in this section is to show that if the objective value is above certain threshold , there exists a good random direction which reduces the objective in expectation . Particularly our aim is to prove the following theorem ( informal ) : Theorem 6 For a given pair ( f∗ , G ) with ⟨H∞ , G⟩ ≤ ζ1 , f∗T ( K∞ ⊙G ) −1f∗ ≤ ζ2 , Rn ( f ∗ ) ≤ ∆ , recall the ideal random matrices ( W ∗Σ , V ∗ Σ ) constructed in Appendix A.12 , where Σ is a random diagonal sign matrix . Specifically , W ∗Σ is defined in Equation ( 108 ) , and V ∗ Σ is the projection of the rows of matrix V ∗ onto the orthogonal subspace spanned by ( ϕ ( 0 ) ( xi ) ) ni=1 . Using the parameter setting for i = 1 , 2 ψi = ν 4ζi , ( 137 ) with respect to an arbitrary parameter ν > 0 , then for every pair ( W ′ , V ′ ) such that ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 and L ( W ′ , V ′ ) ≥ ∆+ ν , ( 138 ) for parametersm1 , m2 , m3 , 1/κ1 , 1/κ2 polynomially large enough in B , 1/λ0 , n , C1 , C2 and small enough step size η , we have EΣL ( W ′ − η/2W ′ + √ ηW ∗Σ , V ′ − η/2V ′ +√ηV ∗Σ ) ≤ L ( W ′ , V ′ ) − ην/4 . ( 139 ) In order to prove the above theorem , we first state and prove the following lemma which is the core of Theorem 6 . Lemma 11 For matrices ( W ∗ , V ∗ ) constructed in Appendix A.12 , specifically for their random coupling ( W ∗Σ , V ∗ Σ ) as denoted above , we have : EΣℓ ( f ′ ( 1−η/2 ) W ′+√ηW∗Σ , ( 1−η/2 ) V ′+ √ ηV ∗Σ ( xi ) , yi ) ≤ ( 1− η ) ℓ ( f ′W ′ , V ′ ( xi ) , yi ) + ηℓ ( f∗ ( xi ) , yi ) ± η℘ , where ℘ goes to zero with polynomially large overparameterization ( the exact dependence is revealed via the proof ) . Proof of Lemma 11 For brevity , we use the notation D′ , ρ here to refer to the diagonal binary sign matrix when the input is multiplied by the sum of weight and smoothing matrices . It will be clear in the context of the equation that what the “ input ” and the “ weight ” matrices are . This notation is also defined and used in Lemma 28 ) . Here , we bound multiple cross terms that are created as a result of moving in the random direction . To simplify the presentation and avoid confusing recursions in the proof , we have made a sublemma for each of these cross terms and has deferred its proof to Appendix A.14 . We use difference sub-indices of the symbol ℜ to illustrate terms that go to zero by growing the overparameterization in our architecture . We start by using Lemma 28 , EΣℓ ( f ′ ( 1−η/2 ) W ′+√ηW∗Σ , ( 1−η/2 ) V ′+ √ ηV ∗Σ ( xi ) , yi ) = EΣℓ ( EWρ , V ρf ( 1−η/2 ) W ′+√ηW∗Σ+Wρ , ( 1−η/2 ) V ′+ √ ηV ∗Σ+V ρ ( xi ) , yi ) = EΣℓ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ + √ ηV ∗Σ ) W sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ +√ηW ∗Σ ) xi + ℜ8η , yi ) . = EΣℓ ( EWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi + ηaTD′ , ρV ∗ ΣW sD′ , ρW ∗ Σxi ] + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] + ℜ8η , yi ) . Now using the notation introduced in Lemma 18 , we have W sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi = ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) + η 2 ϕ ( 2 ) ′ ( xi ) . By Lemma 18 , we have the following bound for ϕ ( 2 ) ′ ( xi ) : EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5 . Therefore , Combining this with Lemma 20 , we get = EΣℓ ( EWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W s ( ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) ) + ηaTD′ , ρV ∗ ΣW sD′ , ρW ∗ Σxi ] + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) = EΣℓ ( EWρ , V ρ [ ( 1− η ) aTD′ , ρ ( V ( 0 ) + V ′ + V ρ ) W s ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) + ηaTD′ , ρV ∗ ΣW sD′ , ρW ∗ Σxi ] + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) . = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗ΣW sD′ , ρW ∗Σxi + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) . Moreover , using the notation ϕ∗′ ( xi ) introduced in Lemma 15 and the bound in Lemma 17 , we can rewrite the second term as : LHS = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σ ( ϕ∗ ( xi ) + ϕ∗ ′ ( xi ) ) + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( ηℜ10 ) ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ± O ( ℜ8η ) , yi ) . ( 140 ) Now we write the gradient-lipshitz inequality for ℓ at point p ( 1 ) Σ : = ( 1− η ) f ′ W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) ± η℘1 , and regarding the following vector , where ℘1 is the sum of all the noise terms above and goes to zero by over parameterization : p ( 2 ) Σ : = √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] . Hence , using the 1 smoothness of ℓ ( . , yi ) : LHS ≤ EΣℓ ( p ( 1 ) Σ ) + EΣℓ̇ ( p ) √ ηp ( 2 ) Σ + 1 2 ηEΣ ( p ( 2 ) Σ ) 2 . ( 141 ) But note that EΣℓ̇ ( p ) √ ηp ( 2 ) Σ = ℓ̇ ( p ) √ ηEΣp ( 2 ) Σ = 0 . ( 142 ) On the other hand , using the notation of Lemma 15 and the result of Lemma 16 : EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi ) 2 ( 143 ) = EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) ( Σϕ∗ ( xi ) + ϕ∗′Σ ( xi ) ) ) 2 ( 144 ) ≤ 4EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) Σϕ∗ ( xi ) ) 2 ( 145 ) + 4EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) ϕ∗′Σ ( xi ) ) 2 ( 146 ) ≲ ℜ212 + ℜ211 . ( 147 ) Moreover , using again the result on ϕ ( 2 ) ′ ( xi ) from Lemma 18 and the fact that ϕ ( 0 ) ( xi ) is orthogonal to the rows of V ∗Σ : EWρ , V ρaTD′ , ρV ∗ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi = aTD′ , ρV ∗ Σ ( ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) ) + η 2 aTD′ , ρV ∗ Σϕ ( 2 ) ′ ( xi ) = ( 1− η 2 ) aTD′ , ρV ∗ Σϕ ( 2 ) ( xi ) + η 2 aTD′ , ρV ∗ ΣW sD′ , ρϕ ( 2 ) ′ ( xi ) ≲ ( 1− η 2 ) aTD′ , ρV ∗ Σϕ ( 2 ) ( xi ) ±ℜ5 . Combining the last Equation with Lemma 14 : EΣ ( EWρ , V ρaTD′ , ρV ∗ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ) 2 ≲ ( 1− η 2 ) 2aTD′ , ρV ∗ ΣW sD′ , ρϕ ( 2 ) ( xi ) + ℜ25 ≲ ℜ20 + ℜ25 . ( 148 ) Combining Equations ( 147 ) and ( 148 ) : EΣ ( p ( 2 ) Σ ) 2 ≲ ℜ211 + ℜ212 + ℜ20 + ℜ25 : = ℘2 . ( 149 ) Combining Equations ( 142 ) and ( 149 ) , plugging into ( 141 ) , and reopening the definition of p ( 1 ) Σ : LHS ≲ EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) ± η℘1 , yi ) + ηµ2℘2 . ( 150 ) Now note that we can easily bound the magnitude of the term ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) as : |EWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) | ≤ EWρ , V ρ |aTD′ , ρV ∗Σϕ∗ ( xi ) | ≤ ∥V ∗Σ∥F ∥ϕ∗ ( xi ) ∥ ≤ ∥V ∗∥ϕ∗ ( xi ) ∥ ≤ √ 2ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ , while using Lemma 34 : |f ′W ′ , V ′ ( xi ) | ≤ ( κ2 √ m3 + β2 ) ( √ m3κ1 + C1 + √ m3β1 ) + C2 ( C1 + √ m3β1 ) , which is O ( C1C2 ) for enough overparameterization and smoothing parameters β1 , β2 as defined in A.20.1 . Furthermore , from Equations ( 132 ) and ( 127 ) , we easily see that EWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) ≤ √ 2ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . Now taking η small enough so that the bound η √ 2ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ and η℘1 both also be bounded of order O ( C1C2 ) , we observe that the term inside the argument of ℓ ( . , yi ) Equation in ( 150 ) is O ( C1C2 ) . Hence , we can use the lipschitz parameter of ℓ in the interval [ −O ( C1C2 ) , O ( C1C2 ) ] , given by Lemma 9 to take out the noise term : LHS ≲ EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) , yi ) ± η℘1 + ηO ( C1C2 +B ) ℘2 . ( 151 ) Now by applying Lemma 19 and writing the Lipchitz property of ℓ at point ( 1 − η ) f ′W ′ , V ′ ( xi ) = O ( C1C2 ) : LHS ≲ EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηf∗ ( xi ) ± ηℜ9 , yi ) ± η℘1 + ηO ( C1C2 +B ) ℘2 : = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηf∗ ( xi ) , yi ) ± ηℜ9 ± η℘1 + ηO ( C1C2 +B ) ℘2 = ℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηf∗ ( xi ) , yi ) ± η℘ , where the last line is just definition . Now Convexity of ℓ finishes the proof . Next , using Lemma 11 we prove Theorem 6 . Restating Theorem 6 In the same setting as Theorem 6 and having enough overparameterization such that ℘ ≤ ν8 ( ℘ defined in Lemma 11 ) and polynomially small enough step size η , we have EΣL ( W ′ − ηW ′ + √ ηW ∗Σ , V ′ − ηV ′ +√ηV ∗Σ ) ≤ L ( W ′ , V ′ ) − ην/4 . Proof of Theorem 6 First , note that taking expectation w.r.t Σ : EΣ∥ ( 1− η/2 ) W ′ + √ ηW ∗Σ∥2 = EΣ ( 1− η/2 ) 2∥W ′∥2 + 2 ( 1− η/2 ) √ η⟨W ′ , m3∑ k=1 ΣkW ∗ k ⟩+ η∥ m3∑ k=1 ΣkW ∗ k ∥2 = ( 1− η/2 ) 2∥W ′∥2 + η ∑ k ∥W ∗k ∥2 , which by orthogonality of W ∗k ’ s : LHS = ( 1− η/2 ) 2∥W ′∥2 + η∥W ∗∥2 = ( 1− η ) ∥W ′∥2 + η∥W ∗∥2 + η2∥W ′∥2 . Similarly for V ′ : EΣ∥ ( 1−η/2 ) V ′+ √ ηV ∗Σ∥ = ( 1−η/2 ) 2∥V ′∥2+ηEΣ∥V ∗Σ∥2 = ( 1−η ) ∥V ′∥2+η∥V ∗∥2+η2∥V ′∥2 . Now using Lemma 11 : EΣL ( W ′ − η/2W ′ + √ ηW ∗Σ , V ′ − η/2V ′ +√ηV ∗Σ ) ≤ ( 1− η ) EZℓ ( f ′W ′ , V ′ ( x ) , y ) + ηEZℓ ( f∗ ( x ) , y ) + ( 1− η ) ( ψ1∥W ′∥2 + ψ2∥V ′∥2 ) + η ( ψ1∥W ∗∥2 + ψ2∥V ∗∥2 ) + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) ≤ L ( W ′ , V ′ ) − η ( L ( W ′ , V ′ ) −∆− ψ1ζ1 − ψ2ζ2 ) + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) , which by the choice of ψi ’ s is equal to LHS ≤ L ( W ′ , V ′ ) − η ( L ( W ′ , V ′ ) −∆− ν/2 ) + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) LHS ≤ L ( W ′ , V ′ ) − ην/2 + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) . Moreover , using the condition ℘ ≤ ν/8 , and picking η as small as η ( ∥W ′∥2 + ∥V ′∥2 ) ≤ η ( C21 + C22 ) ≤ ν/8 , we finally get LHS ≤ L ( W ′ , V ′ ) − ην/4 . A.14 EXISTENCE OF A GOOD DIRECTION HELPER LEMMAS In this section , we state and prove the core lemmas that are used in the proof of Lemma 11 . Notably , through all of this section , we assume the norm bounds ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 and that as our usual assumption , the rows of V ′ are orthogonal to ϕ ( 0 ) ( xi ) ’ s for all i ∈ [ n ] . A notation that we use throughout the proofs is V ∗Σ which refers to the projectiono of V ∗Σ onto the orthogonal subspace to ( ϕ ( 0 ) ( xi ) ) n i=1 . Lemma 12 Let P ( . ) be the projection operator onto the subspace spanned by ( ϕ ( 0 ) ( xi ) ) ni=1 . Also , we denote the projection of rows of V ∗Σ onto the orthogonal subspace to ( ϕ ( 0 ) ( xi ) ) ni=1 by V ∗ Σ j . Then EΣ∥V ∗Σ j − V ∗ j Σ ) ∥2 ≤ ϱ23ξ2n/m2 , with high probability ∥V ∗Σ j − V ∗ j Σ ) ∥ ≲ ϱ3ξ √ n√ m2 , Proof of Lemma 12 By Equation ( 129 ) , we have ∥V ∗j ∥∞ ≤ ϱ3ξ/ √ m2 . Now suppose that u1 , ... , un are an orthonormal basis for the subspace span ( ϕ ( 0 ) ( xi ) ) ni=1 . Then EΣ∥V ∗Σ j − V ∗ j Σ ) ∥2 = EΣ∥P ( V ∗j Σ ) ∥2 = ∑ i ∑ k V ∗j 2 k ui 2 k ≤ ∥V ∗j ∥2∞n ≤ ϱ23ξ2n/m2 . Also , by Hoeffding , with high probability : ∥P ( V ∗j Σ ) ∥2 = ∑ i ( m3∑ k=1 V ∗j kuikΣk ) 2 ≲ n∥V ∗j ∥2∞ , which implies the second part . Lemma 13 The first cross term goes away because of the definition of V ∗Σ . ( inside the expectations is zero almost surely ) EΣ ( EV ρ , Wρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 0 ) ( xi ) ] ) 2 = 0 . Lemma 14 Second cross term : EΣ ( EV ρ , Wρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ( xi ) ] ) 2 ( 152 ) ≲ ξ2 ( ( 1 + ℜ ) 2nζ2 + ϱ23n ) ( C21 +m3β21 ) = ℜ20 . ( 153 ) Proof of Lemma 14 This time we use Equation ( 129 ) in Lemma 10 and Lemma 12 : EΣ ( EV ρ , Wρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ( xi ) ] ) 2 ≤ EΣ ( EV ρ , Wρ1/ √ m2 ∑ j |V ∗Σ jϕ ( 2 ) ( xi ) | ) 2 ≤ 1 m2 EΣ , V ρ , Wρ ( ∑ j |V ∗Σ jϕ ( 2 ) ( xi ) | ) 2 ≤ EV ρ , WρEΣ ∑ j |V ∗Σ jϕ ( 2 ) ( xi ) |2 ≲ EV ρ , WρEΣ ∑ j | ( V ∗Σ j − V ∗ j Σ ) ϕ ( 2 ) ( xi ) |2 + ∑ j |V ∗j Σϕ ( 2 ) ( xi ) |2 ≲ EV ρ , WρEΣ ∑ j ∥V ∗Σ j − V ∗ j Σ ) ∥2∥ϕ ( 2 ) ( xi ) ∥2 + ∑ j ∥V ∗j ∥2∞∥ϕ ( 2 ) ( xi ) ∥22 ≲ ( ( 1 + ℜ ) 2nζ2ξ2 + ϱ23ξ2n ) EV ρ , Wρ∥ϕ ( 2 ) ( xi ) ∥22 . Now according to Lemma 33 , we have EV ρ , Wρ∥ϕ ( 2 ) ( xi ) ∥2 ≲ C12 +m3β12 , which completes the proof . Lemma 15 We get an additional term ϕ∗′ ( xi ) as a result of smoothing which we define as ϕ∗′ ( xi ) = 1√ m1 W sDW ( 0 ) +W ′+Wρ , xiW ∗ Σxi − ϕ∗Σ ( xi ) . ( 154 ) Then P ( ϕ∗′ ( xi ) ̸= 0 ) ≤ m1 exp { −c22/ ( 8β21 ) } . Moreover , we have the following inequality almost surely ( over the randomness of W ρ ) : ∥ϕ∗′ ( xi ) ∥∞ ≲ √∑ k ∥Vk∥2H∞ . Proof of Lemma 15 According to Lemma 1 , for j /∈ P , for every i ∈ [ n ] we have | ( W ( 0 ) j +W ′ j ) xi| ≥ c2/2 √ m1 . Now note that as long as the sign patterns for j /∈ P does not change , ϕ∗′ ( xi ) will be zero . Therefore by union bound P ( ϕ∗′ ( xi ) ̸= 0 ) ≤ m1∑ j=1 P ( sign change in j ) ≤ m1P ( | ( W ( 0 ) j +W ′ j ) xi| ≤ |W ρ j xi| ) ≤ m1P ( |W ρj xi| ≥ c2/ ( 2 √ m1 ) ) . But ( W ρj ) xi is Gaussian with variance β 2 1/m1 . Hence LHS ≲ m1 exp { −c22/ ( 8β21 ) } , which proves the first part . For the second part , according to Equation ( 106 ) in Lemma 5 , for every k ∈ [ m3 ] : |ϕ∗′k ( xi ) | ≤ | 1√ m1 W skDW ( 0 ) +W ′+Wρ , xiW ∗xi|+ |ϕ∗k ( xi ) | ( 155 ) ≤ 2/ √ m1 ∑ j ∥W ∗j ∥ ≤ 2∥W ∗∥F ≲ √∑ k ∥Vk∥2H∞ , ( 156 ) which implies the second part . Lemma 16 Fourth Extra term : EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ∗′ ( xi ) ] ≲ ( κ2 √ m2m3 + C2 + √ m3β2 ) m1 exp { −c22/ ( 8β21 ) } √∑ k ∥Vk∥2H∞ : = ℜ11 . Proof of Lemma 16 Note that with high probability over the randomness of V ( 0 ) , we have ∥V ( 0 ) ∥F ≲ √ m2m3κ2 . Now according to Lemma 15 and using the fact that ∥V ′∥F ≤ C2 : ≤ EWρ , V ρ 1√ m2 ∥a∥∥V ( 0 ) + V ρ + ( 1− η ) V ′∥2∥ϕ∗′ ( xi ) ∥ = EWρ , V ρ∥V ( 0 ) + V ρ + ( 1− η/2 ) V ′∥F ∥ϕ∗′ ( xi ) ∥ ≤ √ EV ρ ( ∥V ( 0 ) + ( 1− η/2 ) V ′∥2F + ∥V ρ∥2F ) m1 exp { −c 2 2/ ( 8β 2 1 ) } √∑ k ∥Vk∥2H∞ ≲ √ ∥V ( 0 ) ∥2F + ∥V ′∥2F +m3β22 m1 exp { −c 2 2/ ( 8β 2 1 ) } √∑ k ∥Vk∥2H∞ ≲ ( κ2 √ m2m3 + C2 + √ m3β2 ) m1 exp { −c22/ ( 8β21 ) } √∑ k ∥Vk∥2H∞ . Lemma 17 Fifth extra term : ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ∗′ ( xi ) ] ∣∣∣ ≲√ζ2m1 exp { −c22/ ( 8β21 ) } √∑ k ∥Vk∥2H∞ : = ℜ10 . Proof of Lemma 17 Similar to the previous Lemma , the inner expectation can be bounded as : ≤ E 1√ m2 ∥a∥∥V ∗Σ∥F ∥ϕ∗ ′ ( xi ) ∥ ≤ E∥V ∗∥F ∥ϕ∗′ ( xi ) ∥ ≲ √ ζ2m1 exp { −c 2 2/ ( 8β 2 1 ) } √∑ k ∥Vk∥2H∞ . Lemma 18 We have another extra term as a product of the movement −η/2W ′ in the first layer : ϕ ( 2 ) ′ ( xi ) = 2 η √ m1 [ W sDW ( 0 ) +Wρ+W ′ ( W ( 0 ) +W ρ+ ( 1−η/2 ) W ′ ) xi−ϕ ( 0 ) ( xi ) − ( 1−η/2 ) ϕ ( 2 ) ( xi ) ] . Then ∣∣∣EWρ , V ρ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ √ ζ2m3 ( β21 m1 + c2C21√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 ) : = ℜ5 . ( 157 ) EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5 . ( 158 ) Proof of Lemma 18 First we prove the following approximation argument ( for all k ∈ [ m3 ] ) : EWρ |ϕ ( 2 ) ′ ( xi ) k|2 ≤ β21 m1 + c2C 2 1√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 . ( 159 ) We have LHS = EWρ ∣∣∣ 1√ m1 W skDW ( 0 ) +Wρ+W ′ , xi ( W ( 0 ) +W ρ ) xi − 1√ m1 W skDW ( 0 ) , xiW ( 0 ) xi ∣∣∣2 ≲ EWρ ∣∣∣ 1√ m1 W skDW ( 0 ) +Wρ , xi ( W ( 0 ) +W ρ ) xi − 1√ m1 W skDW ( 0 ) , xiW ( 0 ) xi ∣∣∣2 +EWρ ∣∣∣ 1√ m1 W skDW ( 0 ) +Wρ , xi ( W ( 0 ) +W ρ ) xi − 1√ m1 W skDW ( 0 ) +Wρ+W ′ , xi ( W ( 0 ) +W ρ ) xi ∣∣∣2 By the independence of W ρj ’ s , the first term can be upper bounded as = EWρ 1 m1 m1∑ j=1 ( ( W ( 0 ) j +W ρ j ) xi1 { ( W ( 0 ) j +W ρ j ) xi ≥ 0 } −W ( 0 ) j xi1 { W ( 0 ) j xi ≥ 0 } ) 2 = ≤ 1 m1 m1∑ j=1 EWρ |W ρj xi| 2 = 1 m1 ∑ β21 m1 = β21 m1 . For the second term , note that for every j /∈ P , the jth entries of DW ( 0 ) +Wρ , xi and DW ( 0 ) +Wρ+W ′ , xi are different only if W ρ j can make a sign change in the jth row , i.e . | ( W ( 0 ) j + W ′j ) xi| ≤ |W ρ j xi| should happen . We denote this event for every j /∈ P by Ẽj . Furthermore , if this happens for some j , then the value of ( W ( 0 ) +W ρ ) jxi is upper bounded by |W ′jxi| . Now similar to our discussion in Lemma 15 and using the result of Lemma 1 : P ( ∪j /∈P Ẽj ) = P ( sign change in some j /∈ P ) ≤ m1∑ j /∈P P ( sign change in j ) ≤ m1P ( | ( W ( 0 ) j +W ′ j ) xi| ≤ |W ρ j xi| ) ≤ m1P ( |W ρ j xi| ≥ c2/ ( 2 √ m1 ) ) . But note that ( W ρj ) xi is Gaussian with variance β 2 1/m1 . Hence LHS ≲ m1 exp { −c22/ ( 8β21 ) } , So finally we can write ≲ β 2 1 m1 + EWρ 1 m1 ( ∑ j∈P |W ′jxi| ) 2 + EWρ 1 m1 ( 1 { ∪j /∈P Ẽj } ∑ j /∈P |W ′jxi| ) 2 ≲ β 2 1 m1 + |P | m1 ∥W ′∥2 + P ( ∪j /∈P Ẽj ) ∥W ′∥2 ≤ β 2 1 m1 + c2C 2 1√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 . which completes the proof for Equation ( 159 ) . This immediately implies EWρ∥ϕ ( 2 ) ′ ( xi ) ∥ ≤ √ EWρ∥ϕ ( 2 ) ′ ( xi ) ∥2 ≤ √ m3 ( β21 m1 + c2C21√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 ) . Now we first prove Equation ( 157 ) : ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ′ ( xi ) ] ∣∣∣ ≤ EWρ 1√ m2 ∥a∥∥DV ( 0 ) +V ρ+V ′ , xiV ∗ Σ∥F ∥ϕ ( 2 ) ′ ( xi ) ] |∥ ≤ ∥V ∗Σ∥FEWρ∥ϕ ( 2 ) ′ ( xi ) ] |∥ ≤ ∥V ∗∥FEWρ∥ϕ ( 2 ) ′ ( xi ) ] |∥ ≲ √ ζ2m3 ( β21 m1 + c2C21√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 ) . To prove Equation ( 158 ) : EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ EWρ , V ρ 1√ m2 ∥a∥∥DV ( 0 ) +V ρ+ ( 1−η/2 ) V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ∥F ∥ϕ ( 2 ) ′ ( xi ) ∥ ≲ EWρ , V ρ 1√ m2 ∥a∥∥V ( 0 ) + V ρ + ( 1− η/2 ) V ′∥F ∥ϕ ( 2 ) ′ ( xi ) ∥ ≲ 1√ m2 ∥a∥ √ EV ρ ( ∥V ( 0 ) ∥2 + ∥V ρ∥2F + ∥ ( 1− η/2 ) V ′∥2F ) EWρ∥ϕ ( 2 ) ′ ( xi ) ∥ ≲ √ ( κ22m2m3 +m3β 2 2 + C 2 2 ) ℜ5 ≲ ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5 . Lemma 19 Closeness condition : EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ∗ Σ ( xi ) ] − f∗ ( xi ) ∣∣∣ ≲ ℜ9 , where ℜ9 : = ϱ3ξ √ n ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ + ℜ3 ( 160 ) +m2 ( exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } +m1 exp { −C21/ ( 8m3β21 ) } ) √ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . ( 161 ) Proof of Lemma 19 Note that by Corollary 5.1 and according to the proof of Equation 130 in Lemma 10 , if for every j /∈ P̃ we don ’ t have a sign change in DV ( 0 ) +V ρ+V ′ , xiV ∗ϕ∗ ( xi ) , then get | 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ϕ∗ ( xi ) − f∗ ( xi ) | ≤ ℜ3 . Also , note that we need the event Ec ( defined in Lemma 33 ) to happen in order to be able to use Corrolary 5.1 . Hence , given a W ρ for which Ec happens , we upper bound the probability of sign change with respect to the randomness of V ρ . We define the following event with respect to the randomness of V ρ when conditioned on a W ρ for which Ec happens ( Pi ’ s are defined in Lemma 7 ) : SC : = { ∃j /∈ Pi s.t.|V ρj x ′ i| ≳ ( κ2 m2 ) 1/3C 2/3 2 ∥x′i∥ } . Now from the result in Corollary 5.1 we have ≤ 1 { sign change in j /∈ Pi } ≤ 1 { SC } . Therefore , E1 { sign change } ≤ 1 { SC } ≤ ∑ j /∈Pi P ( |V ρj x ′ i| ≳ ( κ2 m2 ) 1/3C 2/3 2 ∥x′i∥ ) ≤ m2P ( |V ρj x ′ i| ≳ ( κ2 m2 ) 1/3C 2/3 2 ∥x′i∥ ) . But note that ( V ρj ) x ′ i is Gaussian with variance β 2 2∥x′i∥2/m2 . Hence LHS ≲ m2 exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } . ( 162 ) Now let D be a sign matrix random variable such that if Ec and SCc both happens , then it is equal to the valid sign matrixDV ( 0 ) +V ρ+V ′ , xi , and otherwise it is equal to an arbitrary valid sign matrix in the case when both Ec and SCc happen . Now using Equation ( 117 ) we have with high probability over the initialization : EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ∗ Σ ( xi ) ] − f∗ ( xi ) ∣∣∣ ≤ EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ∗ Σ − V ∗Σ ) ϕ∗Σ ( xi ) ] + EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗Σϕ∗Σ ( xi ) ] − f∗ ( xi ) ≤ EWρ , V ρEΣ ∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ∗ Σ − V ∗Σ ) ϕ∗Σ ( xi ) ∣∣ + EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ϕ∗ ( xi ) ] − f∗ ( xi ) ∣∣∣ ≤ EWρ , V ρEΣ 1 m2 ∑ j ∥V ∗Σ j − V ∗ j Σ∥∥ϕ∗Σ ( xi ) ∥ + EΣ ∣∣∣EWρ , V ρ [ ( 1√ m2 aTDV ∗ϕ∗ ( xi ) − f∗ ( xi ) ) + 1 { SC ∪ E } ( 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) −D ) ] ∣∣∣ ≤ EWρ , V ρEΣ 1 m2 ∑ j ∥V ∗Σ j − V ∗ j Σ∥∥ϕ∗Σ ( xi ) ∥ + EΣ ∣∣∣EWρ , V ρ [ ( 1√ m2 aTDV ∗ϕ∗ ( xi ) − f∗ ( xi ) ) ∣∣∣ + EΣEWρ , V ρ ∣∣∣1 { SC ∪ E } ( 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) − 1√ m2 aTDV ∗ϕ∗ ( xi ) ) ] ∣∣∣ ≤ EWρ , V ρ 1√ m2 ∑ j √ EΣ∥V ∗Σ j − V ∗ j Σ∥2∥ϕ ∗ ( xi ) ∥ + ℜ3 + 2P ( SC ∪ E ) max D′ ∣∣∣ 1√ m2 aTD′V ∗ϕ∗ ( xi ) ∣∣∣ . Now note that for any sign matrix D′ , we have the following bound : ∣∣∣ 1√ m2 aTD′V ∗ϕ∗ ( xi ) ∣∣∣ ≤ 1√ m2 ∥a∥∥V ∗∥F ∥ϕ∗ ( xi ) ∥ ≲ √ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . Also , applying a union bound and using Lemmas 33 P ( SC ∪ E ) ≤ P ( SC ) + P ( E ) ≲ exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } +m1 exp { −C21/ ( 8m3β21 ) } . Hence , also applying Lemma 40 , we further write LHS ≲ ϱ3ξ √ n ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ + ℜ3 +m2 ( exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } +m1 exp { −C21/ ( 8m3β21 ) } ) √ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . Lemma 20 Suppose we have m3κ21 ≥ C21 . Then , for the following basic term we have : EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ( ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) ) ≲ ( 1− η ) EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ±η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) , where ℜ4 : = C2 ( C1 + √ m3β1 ) m2 exp { −C 4/3 2 ( √ m2κ2 ) 2/3/8β22 } + C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1 + √ m3β1 ) , ℜ′6 : = m1 exp { −C21/ ( 8m3β21 ) } √ m3κ1 ( √ m2 + β2 ) + ℜ6 , and ℜ6 is defined in Lemma 22 . Proof of Lemma 20 First , note that by orthogonality of ϕ ( 0 ) ( xi ) to the rows of V ′ : LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) = LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 0 ) ( xi ) = ( 1− η/2 ) EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] = ( 1− η/2 ) 2EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] + ( 1− η/2 ) ( η/2 ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ( 163 ) But note that for the second term : EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ≲ EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ± 1√ m2 ∑ j : sign change |V ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | = EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ± 1√ m2 ∑ j : sign change |V ′jϕ ( 2 ) ( xi ) | . ( 164 ) Now conditioned on x′i , by the result of Lemma 31 we know there exists a set of indices O ⊆ [ m2 ] , s.t . |O| ≤ C 2/3 2 ( √ m2κ2 ) 2/3m2 and for j /∈ O we have |V ( 0 ) j x ′ i| ≥ C 2/3 2 ( √ m2κ2 ) 1/3 √ m2 ∥x′i∥ and |V ′j x′i| ≤ C 2/3 2 ( √ m2κ2 ) 1/3 2 √ m2 ∥x′i∥ . Now for j ∈ [ m2 ] , define the event Rj = { |W ρj x ′ i| ≥ C 2/3 2 ( √ m2κ2 ) 1/3 2 √ m2 ∥x′i∥ } , and R = ∪jRj . First , note that using Gaussian tail bound , R is a rare event : P ( R ) ≤ ∑ j P ( Rj ) ≤ m2 exp { −C4/32 ( √ m2κ2 ) 2/3/8β22 } . Now for j /∈ O and underRc , clearly we have that the signs of ( V ( 0 ) j +V ρ j ) x ′ i and ( V ( 0 ) j +V ρ j +V ′ j ) x ′ i are the same . Therefore , applying Lemma 33 , we can argue under Rc : EWρ , V ρ 1√ m2 ∑ j : sign change |V ′jϕ ( 2 ) ( xi ) | ≤ EWρ 1√ m2 ∑ j∈O |V ′jϕ ( 2 ) ( xi ) | ≤ √ |O| m2 ∥V ′∥EWρ∥ϕ ( 2 ) ( xi ) ∥ ≤ C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1 + √ m3β1 ) . Hence , overall , using Cauchy-Shwartz EWρ , V ρ 1√ m2 ∑ j : sign change |V ′jϕ ( 2 ) ( xi ) | ≤ ∥V ′∥EWρ∥ϕ ( 2 ) ( xi ) ∥P ( R ) + C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1+ √ m3β1 ) ≤ C2 ( C1 + √ m3β1 ) m2 exp { −C 4/3 2 ( √ m2κ2 ) 2/3/8β22 } + C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1 + √ m3β1 ) : = ℜ4 . ( 165 ) On the other hand , using Lemma 30 , we have with high probability over the randomness of initialization 1√ m2 aTDV ( 0 ) , xiV ( 0 ) ϕ ( 2 ) ( xi ) ≤ √ m3κ2∥ϕ ( 2 ) ( xi ) ∥ . Hence : EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ , xi ( V ( 0 ) + V ρ ) ϕ ( 2 ) ( xi ) ] ≤ EWρ , V ρ [ 1√ m2 aTDV ( 0 ) , xiV ( 0 ) ϕ ( 2 ) ( xi ) + 1√ m2 ∑ j |V ρj ϕ ( 2 ) ( xi ) | ] ≤ EWρ 1√ m2 aTDV ( 0 ) , xiV ( 0 ) ϕ ( 2 ) ( xi ) + β2EWρ∥ϕ ( 2 ) ( xi ) ∥ ≲ ( √ m3κ2 + β2 ) EWρ∥ϕ ( 2 ) ( xi ) ∥ ≲ ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) . ( 166 ) Combining Equations ( 165 ) and ( 166 ) into Equation ( 164 ) : ∣∣∣EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 2 ) ( xi ) ] ∣∣∣ ≲ ℜ4 + ( √m3κ2 + β2 ) ( C1 +√m3β1 ) . ( 167 ) Moreover , for the first term in ( 163 ) , using Equation ( 166 ) and Lemmas 33 and Lemma 30 we have |EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] | ≲ |EWρ , V ρ 1√ m2 aTDV ( 0 ) V ( 0 ) ϕ ( 2 ) ( xi ) |+ EWρ , V ρ 1√ m2 ∑ j |V ρj ϕ ( 2 ) ( xi ) |+ 1√ m2 ∑ j |V ′jϕ ( 2 ) ( xi ) | ≲ κ2 √ m3 ( C1 + β1 √ m3 ) + ( C2 + β2 ) EWρ∥ϕ ( 2 ) ( xi ) ∥ ≲ κ2 √ m3 ( C1 + β1 √ m3 ) + ( C2 + β2 ) ( C1 + √ m3β1 ) . ( 168 ) Substituting Equations ( 167 ) and ( 168 ) into Equation ( 163 ) , we finally get LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ≲ ( 1− η/2 ) 2EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ± η 2 ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ≲ ( 1− η ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ± η 2 4 ∣∣∣EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ∣∣∣ ± η 2 ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ≲ ( 1− η ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ± η2 ( κ2 √ m3 ( C1 + β1 √ m3 ) + ( C2 + β2 ) ( C1 + √ m3β1 ) ) ± η ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) . ( 169 ) Now by picking η small enough so that the second term is dominated by the third term we get : LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ( 170 ) ≲ ( 1− η ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ( 171 ) ± η ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) . ( 172 ) Now we aim to bound the term EWρ , V ρ [ 1√m2 a TDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) +V ρ ) ϕ ( 0 ) ( xi ) . First assume that we are in the event Ec defined in Lemma 33 , i.e . we have ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 . Conditioned on such W ρ , we now work with the randomness of the initialization and V ρ . Note that the random matrix V ( 0 ) + V ρ jointly over the randomness of V ρ and the initialization is also Gaussian , and its variance is κ22 ≤ κ22 + β22 m2 ≤ 2κ22 , ( 173 ) where the inequality follows from the fact that κ2 ≥ 1√m2 and β2 ≤ 1 . Therefore , applying Lemma 22 for the random matrix V ( 0 ) in the Lemma as V ( 0 ) +V ρ here , the bound does not change up to constants because of the inequality ( 173 ) . Hence , with high probability , lets say with prob . 1− δ1 this time over both the randomness of initialization and V ρ : L = ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ≤ ℜ6 ( 174 ) This means that with probability at least 1− √ δ1 over the random initialization , then we have ( 174 ) with prob . at least 1− √ δ1 over the randomenss of V ρ . We name the latter high probability statement as ( ⋆ ) . Moreover , note that by Lemma 32 and assuming m3 log ( m2 ) < m2 , we have the following almost surely bound ( also note that V ρj ϕ ( 0 ) ( xi ) is Gaussian with std β2√m2 ∥ϕ ( 0 ) ( xi ) ∥ ) : EV ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ( 175 ) = EV ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ( 176 ) ≤ 1√ m2 m2∑ j=1 |V ( 0 ) j ϕ ( 0 ) ( xi ) |+ EV ρ 1√ m2 m2∑ j=1 |V ρj ϕ ( 0 ) ( xi ) | ( 177 ) ≲ ∥ϕ ( 0 ) ( xi ) ∥ sup ∥x′∥=1 1√ m2 m2∑ j=1 |V ( 0 ) j x ′|+ 1√ m2 m2∑ j=1 β2√ m2 ∥ϕ ( 0 ) ( xi ) ∥ ( 178 ) ≲ ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) . ( 179 ) Furthermore , note because each variable |V ρj ϕ ( 0 ) ( xi ) | is β2√ m2 ∥ϕ ( 0 ) ( xi ) ∥ subGaussian . Therefore , L is subGaussian with parameter ∥ϕ ( 0 ) ( xi ) ∥β2 with respect to the randomness of V ρ . Now the point is that the high probability argument in ( ⋆ ) is much stronger than what one can get from the subGaussian ineqaulity with parameter ∥ϕ ( 0 ) ( xi ) ∥β2 ( with the corresponding expectation term ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) ) . However , the disadvantage of ( ⋆ ) is that it only works for a fixed δ1 . In other words , at least it is not obvious from this argument that why for a fixed W ( 0 ) in a high probaiblity region of the random initialization , whether we can send δ1 to zero by growing the constant behind ℜ6 with logarithmic rate log ( 1/δ ) . This makes our job hard for bounding the expectation with respect to V ρ if we only wish to rely on ( ⋆ ) . Therefore , we combine it with the inequality that we get from the subGaussian parameter that we introudced above . More rigorously , we define the thresholding parameter ℧ : = ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) + ∥ϕ ( 0 ) ( xi ) ∥β2 log ( ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) /ℜ6 ) = Θ ( ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 log ( ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) /ℜ6 ) ) ) , for which we have E [ L| ℧ ≤ L ] P ( ℧ ≤ L ) ≲ ℜ6 . we divide the range of values for L into three parts : E [ L ] = E [ L| L ≤ ℜ ] P ( L ≤ ℜ6 ) + E [ L| ℜ6 ≤ L ≤ ℧ ] P ( ℜ6 ≤ L ≤ ℧ ) + E [ L| ℧ ≤ L ] P ( ℧ ≤ L ) ≤ E [ L| L ≤ ℜ6 ] + P ( ℜ6 ≤ L ≤ ℧ ) + ℜ6 ≲ ℜ6 + √ δ1℧ . Now by choosing δ1 ≲ 1/℧ , we conclude with high probability over initialization and conditioned on W ρ ’ s such that Ec happens we have EV ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ = E [ L ] ≲ ℜ6 . Finally , we integrate also with respect to W ρ . To control the random variable when E happens , we use the bound in ( 179 ) and the fact that E is a rare event due to Lemma 33 : EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) +V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ≲ P ( E ) ∥ϕ ( 0 ) ( xi ) ∥ ( √m2+β2 ) +P ( Ec ) ℜ6 ≤ m1 exp { −C21/ ( 8m3β21 ) } √ m3κ1 ( √ m2 + β2 ) + ℜ6 : = ℜ′6 . Substituting this into ( 172 ) the proof is finally complete . Lemma 21 Third cross term : with high probability over initialization , we have EΣ ( EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) Σϕ∗ ( xi ) ] ) 2 ≲ ξ2 ( m1 exp { −C21/ ( 8m3β21 ) } ( κ22m2m3 + β21m3 ) + ℜ27C22 ) : = ℜ212 . Proof of Lemma 21 Note that the way we defined the matrix W ∗ and as a result ϕ∗ ( xi ) only depends on the randomness of W ( 0 ) , not on W ′ or the randomness of V ( 0 ) . Now using Equation ( 131 ) and Jensen inequality we can write ( for vector v , the notation v2⊙ is another vector with each entry as the second power of the corresponding entry in v ) : = EΣ ( EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) Σϕ∗ ( xi ) ] ) 2 ≤ EΣEWρ , V ρ ( 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) Σϕ∗ ( xi ) ) 2 = EWρ , V ρEΣ ( ⟨ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) , Σϕ∗ ( xi ) ⟩ ) 2 = EWρ , V ρ 〈 ( 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) ) 2⊙ , ϕ∗ ( xi ) 2⊙ 〉 ≤ EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) ∥∥∥2 2 ∥∥∥ϕ∗ ( xi ) ∥∥∥2 ∞ ≤ ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) ∥∥∥2 2 ≤ 2ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ∥∥∥2 2 + 2ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( 1− η ) V ′ ∥∥∥2 2 ≤ ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ∥∥∥2 2 + ξ2 ( 1− η ) 2∥V ′∥2F ≤ ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ∥∥∥2 2 + ξ2 ( 1− η ) 2C22 . Now under the event Ec defined in Lemma 33 we get that ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 , so we can bound the above as ≤ ξ2EV ρ sup ∥V ′∥≤C2 , V ′⊥ϕ ( 0 ) ( xi ) , ∥x′∥≤C1 1 m2 ∥∥∥∑ j aj1 { ( V ( 0 ) j + V ρ j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ( V ( 0 ) j + V ρ j ) ∥∥∥2 ( 180 ) + ξ2 ( 1− η ) 2C22 . ( 181 ) Now defining L2 : = 1 m2 ∥∥∥∑ j aj1 { ( V ( 0 ) j + V ρ j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ( V ( 0 ) j + V ρ j ) ∥∥∥2 , to bound the first term , we want to apply Lemma 23 using the same trick that we did in the proof of Lemma 20 . Note that L2 is the same term as Γ2x′ , V ′ in Lemma 23 except that it is defined with respect to V ( 0 ) + Vρ instead of V ( 0 ) . On the other hand , note that V ( 0 ) + V ρ has Gaussian entries with variance κ22+ β22 m2 and we know κ22 ≤ κ22+ β22 m2 ≤ 2κ22 , which means the argument of Lemma 23 holds true here up to constants : sup ∥V ′∥≤C2 , V ′⊥ϕ ( 0 ) ( xi ) , ∥x′∥≤C1 L2 ≲ ℜ27 . This holds with probability say 1 − δ2 over the randomness of both V ( 0 ) and V ρ . Therefore , with probability 1− √ δ2 over the initialization , then with probability at least 1− √ δ2 over the randomness of V ρ we have the above . Moreover , with a simple Cauchy-Swuartz we get the following almost surely bound : L2 ≲ ∥V ( 0 ) ∥2F + ∥V ρ∥2F . ( 182 ) Now the variable ∥V ρ∥2 is subexponential with parameter ( β41m23 , β21m3 ) . Furthermore , with high probability we have ∥V ( 0 ) ∥2F ≲ m2m3κ22 . Therefore , taking ℧2 : = Θ ( κ22m2m3 + β 2 1m3 log ( ( κ22m2m3 + β 2 1m3 ) /ℜ7 ) ) , then one can easily see by the subexponential tail : E [ L2| L2 ≥ ℧2 ] = Θ ( ℧2 ) , P ( L2 ≥ ℧2 ) ≤ ℜ27/℧2 . Hence , we can apply the same trick as Lemma 20 as E [ L2 ] = E [ L2| L2 ≤ ℜ27 ] P ( L2 ≤ ℜ27 ) + E [ L2| ℜ27 ≤ L2 ≤ ℧2 ] P ( ℜ27 ≤ L2 ≤ ℧2 ) + E [ L2| ℧2 ≤ L2 ] P ( ℧2 ≤ L2 ) ≲ E [ L2| L2 ≤ ℜ27 ] + P ( ℜ27 ≤ L2 ≤ ℧2 ) + ℜ27 ≲ ℜ27 + √ δ2℧2 . Now taking δ2 ≲ ℜ47/℧22 , we finally get that conditioned on W ρ ’ s where E happens , then EV ρL2 ≤ ℜ27 . On the other hand , to handle the case when E happens , we can use the bound in ( 182 ) as it does not depend on the occurrence of E as well : EWρ , V ρL2 ≤ P ( E ) EV ρ ( ∥V ( 0 ) ∥2 + ∥V ρ∥2 ) + P ( Ec ) ℜ27 ≲ m1 exp { −C21/ ( 8m3β21 ) } ( κ22m2m3 + β21m3 ) + ℜ27 . Plugging this back into ( 181 ) we finally get LHS ≲ ξ2 ( m1 exp { −C21/ ( 8m3β21 ) } ( κ22m2m3 + β21m3 ) + ℜ27 ) + ξ2C22 . A.15 BOUNDING THE WORST-CASE SENARIO Lemma 22 Suppose m3 ≥ log ( m2 ) and √ m3κ1 ≳ C1 . We define the sign matrices Dx ′ V ( 0 ) +V ′ , xi and Dx ′ V ( 0 ) , xi with respect to the multiplications ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + x ′ ) , and V ( 0 ) ( ϕ ( 0 ) ( xi ) + x ′ ) . Then , with high probability : sup ∥x′∥≲C1 , ∥V ′∥F≤C2 , V ′⊥ϕ ( 0 ) 1√ m2 aTDx ′ V ( 0 ) +V ′ , xi V ( 0 ) ϕ ( 0 ) ( xi ) ≲ ( ( C1C2 ) 4/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 + ( C1C2 ) 2/3m 2/3 3 ( κ1κ2 ) 1/3 √ log ( m2 ) m 1/3 2 ) ( 1 + log ( m2 ) C 1/3 1 ( κ2 √ m2 ) 2/3 C 2/3 2 ( κ1 √ m3 ) 1/3 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) + κ1κ2 √ m3 log ( m2 ) : = ℜ6 . Proof of Lemma 22 Consider a cover for the euclidean ball of radius C1 in Rm3 with precision ϵ , i.e . BC1 ( ϵ ) . So for every x′ ∈ Rm3 , there exists an x ∈ BC1 ( ϵ ) such that ∥x− x′∥ ≤ ϵ , and |BC1 ( ϵ ) | ≲ ( 1ϵ ) m3 . Now fix x′ and x . We have Γx′ , V ′ : = 1√ m2 aTDx ′ V ( 0 ) +V ′ , xi V ( 0 ) ϕ ( 0 ) ( xi ) = 1√ m2 m2∑ j=1 aj1 { ( V ( 0 ) j +V ′ j ) ( ϕ ( 0 ) ( xi ) +x ′ ) ≥ 0 } V ( 0 ) j ϕ ( 0 ) ( xi ) . Now by a union bound , because each variable V ( 0 ) j ϕ ( 0 ) ( xi ) is Gaussian with parameter κ2∥ϕ ( 0 ) ( xi ) ∥ and using Equation ( 116 ) , with high probability we have for every j ∈ [ m2 ] : V ( 0 ) j ϕ ( 0 ) ( xi ) ≲ κ2∥ϕ ( 0 ) ( xi ) ∥ √ log ( m2 ) ≲ κ1κ2 √ m3 log ( m2 ) . ( 183 ) Therefore , by Hoeffding over the randomness of the Bernoulli variables aj , for a fixed x′ with high probability : Γx′ : = 1√ m2 m2∑ j=1 aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ϕ ( 0 ) ( xi ) ≲ κ1κ2 √ m3 log ( m2 ) . On the other hand , We know that the VC-dimension of the class of binary functions with respect to halfspaces in Rm3 is m3 + 1 . Therefore , the set of different sign patterns in matrices Dx ′ V ( 0 ) , xi is bounded by mm3+12 , i.e . for D = { Dx ′ V ( 0 ) , xi | x′ ∈ Rm3 } , we have |D| ≲ mm3+12 . Therefore , by taking a union bound over all sign matrices in D , we get with high probability sup x′ Γx′ ≲ κ1κ2 √ m3 log ( m2 ) √ log ( m2m3+1 ) = κ1κ2m3 log ( m2 ) . ( 184 ) Now for a threshold r which satisfies r ≥ 2 √ m3κ2ϵ , ( 185 ) we define Jx , r = { j ∈ [ m2 ] | |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | ≤ r } . Now by Equation ( 116 ) and the assumption of the Lemma √ m3κ1 ≳ C1 , we have ∥ϕ ( 0 ) ( xi ) + x∥ ≤ ∥ϕ ( 0 ) ( xi ) ∥+ ∥x∥ ≲ √ m3κ1 + C1 . ( 186 ) ∥ϕ ( 0 ) ( xi ) + x∥ ≥ ∥ϕ ( 0 ) ( xi ) ∥ − ∥x∥ ≳ √ m3κ1 − C1 ≳ √ m3κ1 . ( 187 ) Hence , V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) is Gaussian with standard deviation at least Ω ( κ2 √ m3κ1 ) . Therefore , P ( |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | ≤ r ) ≲ r√ m3κ1κ2 . This implies E [ |Jx , r| ] ≲ r√ m3κ1κ2 m2 . ( 188 ) On the other hand , note that |Jx , r| is the sum ofm2 Bernoulli random variables , so it is subGaussian with parameter m2 . Therefore , with high probability |Jx , r| ≲ r√ m3κ1κ2 m2 + √ m2 . Now taking maximum over all x ∈ BC1 ( ϵ ) and exploiting the subGaussian tail of the random variables , we get with high probability max x∈BC1 ( ϵ ) |Jx , r| ≲ r√ m3κ1κ2 m2 + √ m2 log ( |BC1 ( ϵ ) | ) ≲ r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) . ( 189 ) Moreover , consider a threshold 1 < θ , such that e−θ 2/8 ≤ m2/m3 , and define the following set of indices J ( 2 ) x′ , θ : = { j ∈ [ m2 ] | |V ( 0 ) j x ′| ≥ θκ2C1 } . Then , using Lemma 29 and noting the fact that the standard deviation of Gaussians in V ( 0 ) is κ2 and that ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 , with high probability : sup x′ : ∥x′∥=1 |J ( 2 ) x′ , θ| ≤ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . ( 190 ) Now note that for each j ∈ [ m2 ] , ∥V ( 0 ) j ∥2 is subexponential with parameters ( m3κ42 , κ22 ) , which means that with high probability : max j ∥V ( 0 ) j ∥ 2 ≲ m3κ22 + √ m3κ 2 2 √ log ( m2 ) + κ 2 2 log ( m2 ) . But with condition m3 ≥ log ( m2 ) , we can further upper bound it as max j ∥V ( 0 ) j ∥ 2 ≲ m3κ22 . Now for fixed x , x′ , for j ∈ Jx , r we have |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≤ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) |+ |V ( 0 ) j ( x ′ − x ) | ≤ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) |+ ∥V ( 0 ) j ∥∥x ′ − x∥ ≲ r + √ m3κ2ϵ . On the other hand , for j /∈ J ( 2 ) x′ , θ : |V ( 0 ) j x ′| ≤ θκ2C1 . ( 191 ) Therefore , for j ∈ Jx , r − J ( 2 ) x′ , θ : |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) |+ |V ( 0 ) j x ′| ≲ r + √ m3κ2ϵ+ θκ2C1 . ( 192 ) In a similar fashion , if j /∈ Jx , r , then using assumption ( 185 ) : |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≥ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | − |V ( 0 ) j ( x− x ′ ) | ≳ r − √ m3κ2ϵ ≥ r/2 . ( 193 ) Hence , using the fact that ϕ ( 0 ) ( xi ) is orthogonal to V ′j : ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣ ≤ 1 { |V ′j ( ϕ ( 0 ) ( xi ) + x′ ) | ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { |V ′j x′| ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { ∥V ′j ∥∥x′∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { ∥V ′j ∥C1 ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { ∥V ′j ∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) | 2C1 + r 4C1 } . ( 194 ) Now by triangle inequality and Equations ( 194 ) , ( 192 ) , ( 191 ) and the fact that ∥V ′∥F ≤ C2 , we can write : |Γx′ − Γx′ , V ′ | ≤ 1√ m2 ∑ j∈Jx , r−J ( 2 ) x′ , θ ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣|V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣|V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j∈J ( 2 ) x′ , θ ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣|V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| max j∈Jx , r−J ( 2 ) x′ , θ |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) 1 { ∥V ′j ∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) | 2C1 + r 4C1 } |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| max j∈Jx , r−J ( 2 ) x′ , θ |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) 1 { ∥V ′j ∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) | 2C1 + r 4C1 } ( |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) |+ θκ2C1 ) + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≲ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| max j∈Jx , r−J ( 2 ) x′ , θ |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) 1 { ∥V ′j ∥ ≳ r C1 } ( C1∥V ′j ∥+ θκ2C1 ) + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≲ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| ( r + √ m3κ2ϵ+ θκ2C1 ) + C1√ m2 √ # ( j : ∥V ′j ∥ ≳ r C1 ) ∥V ′∥2F + 1√ m2 # ( j : ∥V ′j ∥ ≳ r C1 ) θκ2C1 + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≲ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| ( r + √ m3κ2ϵ+ θκ2C1 ) + C21C 2 2√ m2r + C31C 2 2√ m2r 2 θκ2 + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | . Now using Equations ( 208 ) , ( 190 ) , ( 183 ) , and ( 185 ) , and the bound on |J ( 2 ) x′ , θ| from Lemma 29 , we write ≲ 1√ m2 |Jx , r| ( r + θκ2C1 ) + 1√ m2 |J ( 2 ) x′ , θ|κ1κ2 √ m3 log ( m2 ) + C21C 2 2√ m2r + C31C 2 2√ m2r 2 θκ2 ≤ 1√ m2 ( r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) ) ( r + θκ2C1 ) + 1√ m2 ( m3 ( log ( m3 ) + log ( log ( m2 ) ) ) ) κ1κ2 √ m3 log ( m2 ) + C21C 2 2 r √ m2 + C31C 2 2√ m2r 2 θκ2 . ≤ 1√ m2 ( r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) ) ( r + θκ2C1 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) + C21C 2 2 r √ m2 + C31C 2 2√ m2r 2 θκ2 . ( 195 ) Now setting r∗ : = ( C1C2 ) 2/3m 1/6 3 ( κ1κ2 ) 1/3 m 1/3 2 . By this choice , from ( 195 ) we obtain |Γx′−Γx′ , V ′ | ≤ ( ( C1C2 ) 4/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 + ( C1C2 ) 2/3m 2/3 3 ( κ1κ2 ) 1/3 √ log ( 1/ϵ ) m 1/3 2 ) ( 1+θ C 1/3 1 ( κ2 √ m2 ) 2/3 C 2/3 2 ( κ1 √ m3 ) 1/3 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) . Now we set θ∗ : = 3 log ( m2 ) , which also satisfies the condition of Lemma 29 and combining with Equation ( 184 ) , we get that with high probability |Γx′ , V ′ | ≤ |Γx′ , V ′ − Γx′ |+ |Γx′ | ≲ ( ( C1C2 ) 4/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 + ( C1C2 ) 2/3m 2/3 3 ( κ1κ2 ) 1/3 √ log ( 1/ϵ ) m 1/3 2 ) ( 1 + log ( m2 ) C 1/3 1 ( κ2 √ m2 ) 2/3 C 2/3 2 ( κ1 √ m3 ) 1/3 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) + κ1κ2 √ m3 log ( m2 ) , where ∥x′∥ ≲ C2 and ∥V ′∥F ≤ C2 , ∀j : V ′jϕ ( 0 ) ( xi ) = 0 . We also need to satisfy condition ( 185 ) , which regarding this choice for θ = θ∗ becomes r∗ = ( C1C2 ) 2/3m 1/6 3 ( κ1κ2 ) 1/3 m 1/3 2 ≥ 2 √ m3κ2ϵ , ( 196 ) for which it suffices to set ϵ∗ : = ( C1C2 ) 2/3 κ 1/3 1 2 ( m2m3 ) 1/3κ 2/3 2 , ( 197 ) Substituting this choice of ϵ above and picking the overparameterization large enough to dominate the magnitude of C1 , C2 so that log ( 1/ϵ∗ ) ≲ log ( m2 ) , the proof is complete . Lemma 23 Under the following condition ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ≥ log−7/6 ( m2 ) ( C1C2 ) 1/3 , with high probability we have sup ∥x′∥≲C1 , ∥V ′∥F≤C2 , V ′⊥ϕ ( 0 ) 1√ m2 ∥ ∑ j aj1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ ≲ √ m3κ2 log ( m2 ) + ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ( C1C2 ) 2/3 log1/6 ( m2 ) : = ℜ7 . Proof of Lemma 23 Similar to Lemma 22 , define the helper functions Γx′ and Γx′ , V ′ as Γx′ , V ′ = 1√ m2 ∥ ∑ j aj1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ , ( 198 ) Γx′ = 1√ m2 ∥ ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ . ( 199 ) First we bound supx′ Γx′ . To this end , note that because V ( 0 ) j ∈ Rm3 and the VC-dimension of half-planes is m3 + 1 , then by Sauer ’ s Lemma , the set D = { Dx ′ V ( 0 ) , xi | x′ ∈ Rm3 , ∥x′∥ ≲ C1 } of all sign pattern matrices has cardinality at most |D| ≤ mm3+12 . Now note that with high probability , the entries of the matrix V ( 0 ) are all less than O ( κ2 √ log ( m2m3 ) ) . On the other hand , for each fixed sign pattern Dx ′ V ( 0 ) , xi , we have for the sum with respect to this sign pattern : ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ( 200 ) is ( m3κ42 log 2 ( m2m3 ) , κ 2 2 log ( m2m3 ) ) sub-exponential with respect to the randomness of a , because each entry of the vector 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j is ( κ2 √ log ( m2m3 ) ) subGaussian . Therefore , with high probability we have ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ( 201 ) ≤ Ea [ ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ] + deviation ( 202 ) ≲ m3κ22 log ( m2m3 ) + √ m3κ 2 2 log ( m2m3 ) + κ 2 2 log ( m2m3 ) . ( 203 ) Similarly , if we take a union bound over all sign matrices in D and using the fact that m2 > m3 : sup x′ Γ2x′ = sup x′∈Rm3 ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ( 204 ) ≲ m3κ22 log ( m2m3 ) + √ m3κ 2 2 log ( m2m3 ) √ log ( mm3+12 ) + κ 2 2 log ( m2m3 ) log ( m m3+1 2 ) ( 205 ) ≲ m3κ22 log2 ( m2 ) , ( 206 ) which implies sup x′ Γx′ ≲ √ m3κ2 log ( m2 ) . ( 207 ) Moreover , defining Jv , x similar to Lemma 22 and using the similar approach we get with high probability max x∈BC1 ( ϵ ) |Jx , r| ≲ r√ m3κ1κ2 m2 + √ m2 log ( |BC1 ( ϵ ) | ) ≲ r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) . ( 208 ) Now for simplifying the analysis , we assume that for indices j ∈ Jx , r we can change the sign pattern with no cost on V ′ , i.e . we can pick any subset of them . Therefore , we first compute a high probability upper bound on the following quantity : 1√ m2 sup S⊂Jx , r , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ . ( 209 ) If we form the matrix V ( 0 ) ( Jx , r ) be the matrix which only keeps the rows with indices in Jx , r , then the above quantity can be computed as 1√ m2 sup S⊂Jx , r , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ = 1√ m2 sup v∈ { 0,1 , −1 } |Jx , r| ∥vTV ( 0 ) ( Jx , r ) ∥ ( 210 ) ≤ 1√ m2 λmax ( V ( 0 ) ( Jx , r ) ) sup ∥v∥ ≤ 1√ m2 λmax ( V ( 0 ) ( Jx , r ) ) |Jx , r| , ( 211 ) where λmax is the maximum singular value of the matrix . Now by random matrix theory , we know for a fixed x and arbitrary t ≥ 0 , the following argument holds : P ( λmax ( V ( 0 ) ( Jx , r ) ) /κ2 ≳ √ m3 + √ |Jx , r|+ t ) ≤ 2e−ct 2 . ( 212 ) Therefore , as |D| ≤ mm3+12 , we get with high probability max x∈BC1 ( ϵ ) λmax ( V ( 0 ) ( Jx , r ) ) ≲ κ2 ( √ m3 + √ |Jx , r|+ √ log ( mm3+12 ) ) ( 213 ) ≤ κ2 √ log ( m2 ) m3 + κ2 √ |Jx , r| . ( 214 ) Therefore , with high probability sup x∈BC1 ( ϵ ) 1√ m2 sup S⊂Jx , r ∥ ∑ j∈S V ( 0 ) j ∥ ≤ κ2√ m2 ( √ log ( m2 ) m3|Jx , r|+ |Jx , r| ) . ( 215 ) On the other hand , as in Equation ( 193 ) in the proof of Lemma 22 , for j /∈ Jx , r we have : |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≥ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | − |V ( 0 ) j ( x− x ′ ) | ≳ r − √ m3κ2ϵ . ( 216 ) Picking ϵ∗ : = r 2 √ m3κ2 , we get for j /∈ Jx , r |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≳ r. Now similar to the derivation in ( 194 ) we have∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣ ( 217 ) ≤ 1 { ∥V ′j ∥C1 ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ( 218 ) ≤ 1 { ∥V ′j ∥ ≳ r C1 } . ( 219 ) Hence , because ∥V ′∥F ≤ C2 , the number of indices for which 1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ̸= 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } is at most l = ( C1C2 ) 2 r2 . Therefore , we bound the following quantity to use in the analysis : sup S⊂ [ m2 ] & |S|≤l , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ . ( 220 ) But if we define for m2 < j ≤ 2m2 , V ( 0 ) j = −V ( 0 ) j−m2 , then sup S⊂ [ m2 ] & |S|≤l , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ ≤ sup S⊂ [ 2m2 ] & |S|≤l ∥ ∑ j∈S V ( 0 ) j ∥ . Now note that each entry of ∑ j∈S V ( 0 ) j is √ lκ2 subGaussian . Hence , the quantity ∥ ∑ j∈S V ( 0 ) j ∥2 is ( m3l2κ42 , lκ 2 2 ) subexponential . Therefore , we have with high probability sup |S|≤l ∥ ∑ j∈S V ( 0 ) j ∥ 2 ≲ E∥ ∑ j∈S V ( 0 ) j ∥ 2 + √ m3lκ 2 2 √ log ( 2m2 l ) + lκ22 log ( 2m2 l ) ≲ m3lκ22 + √ m3lκ 2 2 √ log ( 2m2 l ) + lκ22 log ( 2m2 l ) ≲ m3lκ22 + √ m3l 3/2κ22 √ log ( m2 ) + l 2κ22 log ( m2 ) . Hence sup |S|≤l ∥ ∑ j∈S V ( 0 ) j ∥ ≲ √ m3 √ lκ2 + lκ2 √ log ( m2 ) . Now using Equation , we can write |Γx′ − Γx′ , V ′ | ≤ 1√ m2 ∥ ∑ j∈Jx , r ( 1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ) V ( 0 ) j ∥ + 1√ m2 ∥ ∑ j /∈Jx , r ( 1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ) V ( 0 ) j ∥ ≤ 1√ m2 sup S⊂Jx , r , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ + 1√ m2 sup S⊂ [ m2 ] & |S|≤ ( C1C2r ) 2 , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ ≤ κ2√ m2 ( √ log ( m2 ) m3|Jx , r|+ |Jx , r| ) + 1√ m2 ( √ m3 √ lκ2 + lκ2 √ log ( m2 ) ) ≤ κ2√ m2 ( rm2√ m3κ1κ2 + √ m2m3 log ( √ m3κ2 r ) ) 1/2 ( √ log ( m2 ) m3 + √ |Jx , r| ) + 1√ m2 ( √ m3 ( C1C2/r ) κ2 + ( C1C2/r ) 2κ2 √ log ( m2 ) ) ≲ κ2√ m2 ( rm2√ m3κ1κ2 + √ m2m3 log ( √ m3κ2 r ) ) + 1√ m2 ( √ m3 ( C1C2/r ) κ2 + ( C1C2/r ) 2κ2 √ log ( m2 ) ) . Combining this with ( 207 ) : |Γx′ , V ′ | ≲ √ m3κ2 log ( m2 ) ( 221 ) + r √ m2√ m3κ1 + κ2 √ m3 log ( √ m3κ2 r ) + 1√ m2 ( √ m3 ( C1C2/r ) κ2 + ( C1C2/r ) 2κ2 √ log ( m2 ) ) . ( 222 ) Now setting r∗ : = m 1/6 3 ( κ1κ2 ) 1/3 m 1/3 2 ( C1C2 ) 2/3 log1/6 ( m2 ) , we get LHS ≲ √ m3κ2 log ( m2 ) + ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ( C1C2 ) 2/3 log1/6 ( m2 ) ( 223 ) + κ2m 1/2 3 log 1/2 ( m2 ) + m 1/3 3 κ 2/3 2 ( C1C2 ) 1/3 m 1/6 2 κ 1/3 1 log 1/6 ( m2 ) ( 224 ) ≲ √ m3κ2 log ( m2 ) + ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ( C1C2 ) 2/3 log1/6 ( m2 ) ( 225 ) + m 1/2 3 κ2 ( C1C2 ) 1/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 log1/6 ( m2 ) . ( 226 ) Now under the condition ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ≥ log−7/6 ( m2 ) ( C1C2 ) 1/3 , The final term is dominated by the first term , which finally completes the proof . A.16 CONVERGENCE The goal of this section is to prove Theorem 7 . Theorem 7 Letting ℵ = 4B2 , by Corollary 8.1 , we have LΠ ( 0 ) ≤ ℵ . We define the domain Dl : = { ∥w′∥ ≤ C1 : = ℵ+4lψ1 , ∥v ′∥ ≤ C2 : = ℵ+4lψ2 } . For a large enough constant l = O ( 1 ) and function LΠ ( w : = ( w′ , v′ ) ) : RN → R. Moreover , suppose LΠ is ρ1 lipschitz , ρ2 gradient lipschitz , and ρ3 hessian lipschitz in the domain Dl , in the sense that their first , second , and third directional derivatives in an arbitrary unit direction is bounded by the corresponding parameters . Suppose we have access to the gradient of LΠ at each point in Dl plus a zero mean noise vector £ such that σ21I ≤ E££T ≤ σ22I and ∥£∥ ≤ Q almost surely . Also , suppose for a threshold ℵℓ ≤ ℵ , if LΠ ( w ) ≥ ℵℓ for any w ∈ Dl , then we either have ( 1 ) ∥∇LΠ ( w ) ∥ ≥ ν 16 √ C21 + C 2 2 , ( 227 ) ( 2 ) λmin ( ∇2LΠ ( w ) ) ≤ −γ . ( 228 ) Then starting from w0 = 0 , with probability at least 0.999 after at most poly ( ρ1 , ρ2 , ρ3 , Q , ℵ , C1 , C2 , 1/γ , log ( σ1/σ2 ) ) number of iterations , we reach a point wt such that LΠ ( wt ) ≤ ℵℓ . Proof Our proof here is a refined version of that in Ge et al . ( 2015a ) . As we mentioned in section 5 , the key fact that we are using in the other parts of our proof is a uniform upper bound ∥w′∥ ≤ C1 , ∥v′∥ ≤ C2 which is unjustified by only naively using Ge et al . ( 2015a ) . Here , first we restate a refined version of Lemmas 14 and 16 in Ge et al . ( 2015a ) in Lemmas 24 and 26 respectively , and then use them to also bound the upward deviations of LΠ . Moreover , to avoid writing repeated proofs and overwhelm the reader , we mostly treat the arguments in Lemma 16 of Ge et al . ( 2015a ) as blackbox and use them for our purpose here . A point to mention before we start , unlike Lemma 14 of Ge et al . ( 2015a ) where the dependency on other parameters than the step size η is more explicit , Lemma 16 hides the dependencies on all the other parameters ( which is polynomial ) . Here , we follow the same style . We refer to the trajectory of the steps of algorithm by ( wt ) t≥0 . In Lemmas of this section , To avoid introducing new notation and complicating things , we refer to the current point of the algorithm by w0 , while for the next point of the algorithm we use w1 ( in Lemma 24 ) , and wT ( in Lemma 26 ) respectively . Also , similar to Ge et al . ( 2015a ) , Õ and Ω̃ below means we are looking at the dependency on η. Lemma 24 SupposeLΠ ( w0 ) ≤ ℵ+2l , and consider a parameter χ > 1 which can be set arbitrarily . For every pointw0 such thatLΠ ( w0 ) ≤ ℵ+2 , ∥∇LΠ ( w0 ) ∥ ≥ 2 √ η ( Q2 + σ22N ) ρ2ρ 2 1 ( 2χ+ 1 2 ) , then for w1 : = w0 − η ( ∇LΠ ( w0 ) +£ ) and random variable R1 ( depending on w0 ) defined as ELΠ ( w1 ) − LΠ ( w0 ) = −η2R21 , ( 229 ) we have R1 = Ω̃ ( 1 ) a.s. , and almost surely : ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣ ≤ ηR1/√ρ2χ . ( the expectation is over the randomness of £ ) . Proof This lemma is a tuned version of Lemma 14 in Ge et al . ( 2015a ) . First , note that the condition LΠ ( w0 ) ≤ ℵ + 2l assures the smoothness coefficients ρ1 , ρ2 and ρ3 for LΠ by Corollary ? ? . We follow similar to Ge et al . ( 2015a ) ( picking η < 1/ ( 2/ρ2 ) ) : ELΠ ( w1 ) − LΠ ( w0 ) ≤ − η 2 ∥∇LΠ ( w0 ) ∥2 + η2σ22ρ2N 2 ≤ −η 4 ∥∇LΠ ( w0 ) ∥2 − η2 ( σ22N +Q2 ) ρ2ρ21 ( 2χ+ 1 2 ) + η2σ22ρ2N 2 ≤ −η 4 ∥∇LΠ ( w0 ) ∥2 − 2η2Q2ρ2ρ21χ . ( 230 ) On ther other hand , LΠ is ρ1 Lipcshitz , so we have almost surely |LΠ ( w1 ) − LΠ ( w0 ) | ≤ ρ1η ( ∥∇LΠ ( w0 ) +£∥ ) ≤ ρ1η ( ∥∇LΠ ( w0 ) ∥+ ∥£∥ ) ≤ ρ1η ( ∥∇LΠ ( w0 ) ∥+Q ) . ( 231 ) ( To be completely precise , we should justify that we can write the Lipschitz inequality at point w0 , we also need to make sure that w1 remains in the domain that we have the Lipschitz parameter in , i.e . Dl . To see why this is true , see the next Corollary ) . Therefore ( ρ2χ ) ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣2 ≤ 2η2ρ21ρ2χ∥∇LΠ ( w0 ) ∥2 + 2ρ2χη2ρ21Q2 . ( 232 ) Taking η ≤ ( 8ρ21ρ2χ ) −1 , ( 233 ) we get from Equation ( 230 ) : ELΠ ( w1 ) − LΠ ( w0 ) ≤ −2η2ρ21ρ2χ∥∇LΠ ( w0 ) ∥2 − 2ρ2χη2.ρ21Q2 . ( 234 ) Combining Equations ( 232 ) and ( 234 ) , we see that ( ρ2χ ) ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣2 ≤ − ( ELΠ ( w1 ) − LΠ ( w0 ) ) . Hence , if we define ELΠ ( w1 ) − LΠ ( w0 ) : = −η2R21 , we get ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣ ≤ ηR1/√ρ2χ , ( 235 ) and from Equation ( 234 ) , that R21 ≥ 2ρ21ρ2χ∥∇LΠ ( w0 ) ∥2 + 2ρ2χρ21Q2 ≥ 2ρ2χρ21Q2 = Ω̃ ( 1 ) . Moreover , because the function is ρ1-Lipshitz at the domain point w0 , we get from Equation ( 231 ) : −η2RRR21 = ELΠ ( w1 ) − LΠ ( w0 ) ≥ −ηρ1 ( ∥∇LΠ ( w0 ) ∥+Q ) ≥ −ηρ1 ( ρ1 +Q ) ≥ − 1 ρ2χ , by taking η ≤ ( ρ1 ( ρ1 +Q ) ρ2χ ) −1 , which implies ηR1 ≤ 1 √ ρ2χ . This combined with Equation ( 235 ) and a triangle inequality implies : ∣∣∣LΠ ( w1 ) − ELΠ ( w1 ) ∣∣∣ ≤ ηR1/√ρ2χ+ η2R21 ≤ 2ηR1/√ρ2χ . ( 236 ) Lemma 25 As long as the value of the function at some w is bounded by ℵ+ 2l ( LΠ ( w ) ≤ ℵ+ 2l ) , then η can be picked small enough ( polynomially in other parameters ) so that the change of the function by a step is a.s. bounded by l. Proof Let ψ = min { ψ1 , ψ2 } . First , note that as the function is bounded by ℵ + 2l , we have the lipschitz parameter ρ1 , hence ∥∇LΠ ( w ) ∥ ≤ ρ1 . Therefore , the change in w in a step is bounded as ∥∇LΠ ( w ) +£∥ ≤ Q+ ρ1 . So by picking η ≤ ( √ℵ+ 3l ψ − √ ℵ+ 2l ψ ) / ( Q+ ρ1 ) , we guarantee that the value of w after a step remains in the ball of radius √ ℵ+3l ψ , hence we still have the smoothing parameters even after one step . Therefore , now we can use the Lipcshitz parameter ρ1 to bound the value of the function after one step as it is written in Equation Equation ( 231 ) . Using this Equation , it is enough to pick η as small as : η ≤ l/ ( ∥∇LΠ ( w0 ) ∥+Q ) , ( 237 ) so that the change in the function would be most l as desired . Lemma 26 For a fixed point w0 s.t . LΠ ( w0 ) ≤ ℵ+ 2l , suppose we pick η small enough such that § ( η ) : = 2 √ η ( Q2 + σ22N ) ρ2ρ 2 1 ( 2χ+ 1 2 ) < ν 16 √ C21 + C 2 2 . Then , note that for ∥∇LΠ ( w0 ) ∥ ≤ § ( η ) , condition 228 implies : λmin ( ∇2LΠ ( w0 ) ) ≤ γ . Then , using the notation ET for the high probability event corresponding to Equations ( 36 ) and ( 44 ) in Ge et al . ( 2015a ) , for small enough η ( polynomially small w.r.t other parameters ) , for R2 defined as E [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } = −R22η2 , ( 238 ) we have almost surely ∣∣∣ [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } ∣∣∣ ≤ R2η/√ρ2χ . ( 239 ) Note that in the expectations above w0 is assumed fixed . Furthermore , we can assume P ( ET ) ≥ 1− Õ ( η5 ) . Proof This Lemma is a tuned version of Lemma 16 in Ge et al . ( 2015a ) . We change a couple of things here . First , we consider an implicit coupling that if wt exits Dl we do not move it anymore , i.e . wt′ = wt , ∀t′ ≥ t , which means the noise vectors also becomes zero , i.e . £t′ = 0 , ∀t′ ≥ t. This way , the sequence of noise vectors remain bounded by Q , because if wt is inside Dl , then by assumption ∥£t∥ ≤ Q , while otherwise £t = 0 . We call this event ET for T defined in Lemma 16 of Ge et al . ( 2015a ) . Note that we also have the smoothing parameters ρ1 , ρ2 , ρ3 for all ( wt ) because of this coupling . In fact , we will use a more strict coupling ; we consider the event ET to be the high probability event corresponding to the bounds in Equations ( 44 ) and ( 36 ) of Ge et al . ( 2015a ) holding for all t ≤ T ; We will see that ET ⊆ ET at the end of this proof , but for now we assume it is true . An important point to note here is that in Ge et al . ( 2015a ) , P ( ET ) is bounded by O ( η2 ) . However , the exponent dependency of η in this bound comes from Azuma-Hoeffding type inequalities , particularly used in Equations ( 60 ) and ( 42 ) in Ge et al . ( 2015a ) , in which by considering larger constants one can easily get higher exponents . Unlike their analysis though , a bit stronger dependence of η5 is sufficient for our later use . Also , because the distribution of our noise depends on the point w , our sequence of noise vectors ( £t ) is a martingale instead of being i.i.d , so we apply Azuma-Hoeffding inequality instead of the simple Hoeffdings in Lemma 16 of Ge et al . ( 2015a ) . ( because we are also sampling also a random ( xi , yi ) to compute the estimate of gradient , this could be simplified to the case where we compute the actual gradient and the injecting an i.i.d noise vector in each step , but it is an overhead to compute the actual gradient , so here we choose to analyze the more complicated case . ) Next , notice the definition of Λ and Λ̃ right after Equation ( 66 ) in Ge et al . ( 2015a ) , which in our notation translates to Λ̃ : = ∇LΠ ( w0 ) T δ̃ + 1 2 δ̃THδ̃ , Λ = ∇LΠ ( w0 ) T δ + 1 2 δTHδ . ( 240 ) where δ̃ = w̃T − w0 , δ = wT − w̃T , for ( w̃t ) which is a coupled sequence with ( wt ) as defined in Ge et al . ( 2015a ) . Note that we apply the coupling for the sequence w̃t as well , i.e . if wt+1 = wt , we also set w̃t+1 = w̃t . To show Equation ( 238 ) , we want to use Equation ( 67 ) in Ge et al . ( 2015a ) , though we only use the expansion for the first term which is under 1 { ET } , i.e . E [ LΠ ( wT ) − LΠ ( w0 ) ] 1ET = EΛ̃1ET + EΛ1ET . ( 241 ) First of all , as it is mentioned in Lemma 16 of ( Ge et al. , 2015a ) , in the case where the noise vector σ1 2I ≤ E££T ≤ σ22I instead of having E££T = σ2I for a fixed σ , in order to still get a negative term of order η in Equation ( 68 ) of ( Ge et al. , 2015a ) , we just need the size of Tmax to be as large as O ( 1γη ( log d + log σ2 σ1 ) ) , and it does not change the order of η in any other part of Lemma 16 . Now similar to Equation ( 68 ) of ( Ge et al. , 2015a ) , if w.l.o.g we assume the smallest eigenvalue γ0 corresponds to i = 1 : EΛ̃1ET ≤ 1 2 N∑ i=1 λi T−1∑ τ=0 1 { λi < 0 } ( 1− ηλi ) 2τη2σ21P ( ET ) + ( 242 ) 1 2 N∑ i=1 λi T−1∑ τ=0 1 { λi≥0 } ( 1− ηλi ) 2τη2σ22 ( 243 ) ≤ η 2 2 [ σ22 N − 1 η − γ0σ21P ( ET ) T−1∑ τ=0 ( 1 + ηγ0 ) 2τ ] ≤ −ησ 2 1 2 . ( 244 ) where in the last line we use the fact that P ( ET ) ≤ 1/2 plus the additional log ( σ2/σ1 ) factor . Second , note that our threshold § ( η ) for the size of gradient in Lemmas 24 and 26 has the same order of η compared to that of Lemmas 14 and 16 in Ge et al . ( 2015a ) . Therefore , the arguments in Lemma 16 that considers the order of η and treat the other parameters as constants is true here as well . Hence , we still have Equation ( 69 ) of Ge et al . ( 2015a ) which is under the event ET . Applying it to Equation ( 241 ) , Hence , finally by a similar derivation of Equation ( 67 ) in Ge et al . ( 2015a ) : E [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } ≤ −Ω̃ ( η ) . ( 245 ) Furthermore , combining Equations ( 36 ) and ( 44 ) we get with high probability ( we use the final high probability parameter of Lemma 16 which is the result of a union bound over all the high probability arguments which is equivalent to the occurrence of ET ) , i.e . when ET happens , ∥wT − w0∥ ≤ Õ ( η 1 2 log 1 η ) . ( 246 ) Picking η small enough such that for the bound above we have O ( η 1 2 log 1 η ) ≤ √ ℵ+ 3l ψ − √ ℵ+ 2l ψ , we get for every w̄ in the line connecting w0 to wT : ∥w̄∥ ≤ √ ℵ+ 3l ψ , which implies that LΠ has the smoothing parameters ρ1 , ρ2 , ρ3 along w0 to wT . Therefore , by the ρ2-gradient smoothness property of LΠ : ∥∇LΠ ( w̄ ) −∇LΠ ( w0 ) ∥ ≤ ρ2∥w0 − w̄∥ ≤ O ( ρ2η 1 2 log 1 η ) . Combining the assumption of the Lemma ∥∇LΠ ( w0 ) ∥ ≤ Õ ( η 1 2 ) , we get ∥∇LΠ ( w̄ ) ∥ = Õ ( η1/2 log ( 1/η ) ) . ( The last Õ also hides the dependency on ρ2 ) . Now integrating over the derivative along the direction from w0 to wT : LΠ ( wT ) = L Π ( w0 ) + ∫ 1 t=0 ∇LΠ ( tw0 + ( 1− t ) wT ) T ( wT − w0 ) dt . Therefore , using ( 246 ) one more time , under the event ET : |LΠ ( wT ) − LΠ ( w0 ) | ≤ ∫ ∣∣∣∇LΠ ( tw0 + ( 1− t ) wT ) T ( wT − w0 ) ∣∣∣dt ≤ ∫ 1 0 ∥∇LΠ ( tw0 + ( 1− t ) ∥∥wT − w0∥dt ≤ Õ ( η1/2 log 1/η ) ∥wT − w0∥ = Õ ( η log2 1/η ) . Hence ∣∣∣ [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } ∣∣∣ ≤ Õ ( η log2 1/η ) , ( 247 ) which Now comparing Equations ( 245 ) and ( 247 ) , it is obvious that one can pick η small enough ( again polynomially small in the other parameters ) such that for some random variable R2 , which also depends on η Equations ( 238 ) and ( 239 ) hold . Furthermore , note that the bound in ( 247 ) is an a.s. upper bound on the change of the function value under the event ET for every 1 ≤ t ≤ T . Therefore , by picking η small enough ( polynomially ) s.t . the quantity O ( η log ( 1/η ) 2 ) in Equation ( 247 ) is bounded by l , we again make sure that the value of function during these steps changes by at most compared to w0 , i.e . for every 1 ≤ t ≤ T : ∣∣∣ [ LΠ ( wt ) − LΠ ( w0 ) ] 1 { Et } ∣∣∣ ≤ l ( 248 ) , therefore remains bounded by ℵ+ 3l . This also implies that ET ⊆ ET as promised . A.17 PROCESS FROM A HIGHER VIEW : DEFINITION OF THE ( X ) SEQUENCE The goal is to find a w∗ with LΠ ( w∗ ) ≤ ℵℓ using Lemmas 24 and 26 . For this purpose we define a coupled sequence this way : First , as done in Ge et al . ( 2015a ) , define a sequence of times τi inductively in the following way : To define τi+1 based on τi , if the condition ℵℓ ≤ LΠ ( wτi+1 ) ≤ ℵ+ 2l ( 249 ) does not hold , then just set τi+1 = τi ⋆ ( 1 ) . Otherwise , using the conditions ( 228 ) , we are either in the situation of Lemma 24 or Lemma 26 , setting w0 = wτi . In the first case , define τi+1 = τi + 1 ⋆ ( 2 ) . In the latter case , Let ET be the same high probability event that we consider in Lemma ( 26 ) , which happens when the aggregate behavior of the noise vectors does not behave oddly , so that we remain close to the starting point w0 . Note that from Lemma 26 , we know P ( ET ) ≥ 1 − O ( η2 ) . Now if the event ET happens , define τi+1 : = τi + T ⋆ ( 3 ) , for T also from Lemma 26 and defined originally in Lemma 16 of Ge et al . ( 2015a ) , while otherwise define τi+1 = τi ⋆ ( 4 ) and , moreover , define all the rest of τi′ ’ s equal to τi : τi′ = τi for every i′ ≥ i . At the same time , we define the event Gi , where Gi happens in the case ⋆ ( 4 ) , and Gi+1 happens in case ⋆ ( 4 ) . Also , Gi happens if any of the previous Gi′ ’ s happen for i′ < i ; in other words , Gi is included in Gi+1 . We use these events to bound the probability that the process remains above ℵℓ . Moreover , define the sequence of random variables ( Xi ) as Xi : = LΠ ( wτi ) . Note that by Lemma 25 and Equation ( 248 ) in Lemma 26 , we have ℵℓ − l ≤ Xi ≤ ℵ + 3l . The key idea behind defining Xi ’ s is that we want to bound the MGF of LΠ ( wt ) , without worrying about falling out of the assumptions of Lemmas 24 and 26 . With the definition of ( Xi ) and Gi , we are ready to state the theorem which roughly says the sequence τi will most likely stop after a number of steps . Lemma 27 Let QR : = ⋃∞ i=1 ( { τi ≤ R } ∩ Ḡi ) . Then , for some R = O ( log ( 1/δ1 ) ) ( ℵ+ 3l ) θη2 , ( 250 ) we have P ( QR ) ≤ δ1 . In other words , after R iterations of PSGD , the defined sequence ( Xi ) above has either been in situation ⋆ ( 1 ) or ⋆ ( 4 ) . Proof By Equations ( 229 ) and ( 245 ) in Lemmas 24 and 26 , there exist a constant θ depending polynomially on all parameters except η such that E [ Xi+1 −Xi| Ḡi ] ≤ − ( τi+1 − τi ) η2 . ( 251 ) Now for some constant C that we specify later , define the random time ı as the largest i where τi ≤ C/η2 . Using the fact that Gi−1 ⊆ Gi , for every i we have a.s. : Xi+11 { Ḡi } −Xi1 { Ḡi−1 } = 1 { Gi − Gi−1 } ( −Xi ) + ( Xi+1 −Xi ) 1 { Ḡi } . Now summing this for i = 1 to ı , taking expectation from both sides and using ( 251 ) : EXı+11 { Ḡı } −X0 = ∞∑ i=1 E1 { Gi − Gi−1 } ( −Xi ) 1 { ı ≥ i } + ∞∑ i=1 ( Xi+1 −Xi ) 1 { Ḡi ∩ { ı ≥ i } } ≤ ∞∑ i=1 E ( 1 { Gi } − 1 { Gi−1 } ) ( −Xi ) + ∞∑ i=1 E ( Xi+1 −Xi ∣∣∣ Ḡi ∩ { ı ≥ i } ) P ( Ḡi ∩ { ı ≥ i } ) ≤ sup i sup |Xi| + ∞∑ i=1 E ( − ( τi+1 − τi ) η2 ∣∣∣ Ḡi ∩ { ı ≥ i } ) P ( Ḡi ∩ { ı ≥ i } ) = sup i sup |Xi| − η2 ∞∑ i=1 E ( τi+1 − τi ) 1 { Ḡi ∩ { ı ≥ i } } . Now using Lemma 25 , we know that in except when ET happens ( in which we stop the time sequence τi ) , the increments of Xi are at most l. Therefore , the value of Xi ’ s always remain bounded by ℵ+ 3l , hence : LHS ≤ ℵ+ 3l − θη2 ∞∑ i=1 E ( τi+1 − τi ) 1 { Ḡi ∩ { ı ≥ i } } . Also , by restricting the integration of the second term to the part ⋃∞ i=1 ( Ḡi ∩ { τi ≥ 2C/η2 } ) of the sample space , we know that under the event { ı ≥ i } , Ḡi automatically happens ( it is easy to check ) . Therefore : LHS ≤ ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ∞∑ i=1 ( τi+1 − τi ) 1 { Ḡi ∩ { ı ≥ i } } = ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ∞∑ i=1 ( τi+1 − τi ) 1 { ı ≥ i } = ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ı∑ i=1 ( τi+1 − τi ) = ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } τı+1 . Now by the definition of ı , τı+1 ≥ C/η2 : LHS ≤ ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ( C/η2 ) ℵ+ 3l − CθP ( ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) ) . But note that Xi ’ s are a.s. bounded between 0 and ℵ + 3l , which implies the LHS above is at least − ( ℵ+ 3l ) . Therefore , we finally get : P ( ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) ) ≤ 2ℵ+ 6l Cθ . Therefore , by picking C∗ = 2 ( 2ℵ+ 6l ) /θ : P ( ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C∗/η2 } ) ) ≤ 1 2 . ( 252 ) Note that the differences between τi ’ s is at most Tmax = Õ ( 1/η ) Ge et al . ( 2015a ) . Hence , again for η polynomially small in other parameters , ( 252 ) implies that for R = 2C ∗ η2 , there exist R̃ = poly ( . ) such that after R̃ iterations on the main sequence ( wt ) , the corresponding sequence ( τi ) has either been in ⋆ ( 1 ) or ⋆ ( 4 ) with chance at least 1/2 . Repeating this argument log ( 1/δ1 ) times ( using the markov property of the process ) we conclude the proof . A.18 BOUNDING THE MGF OF Xi ’ S Next , we want to exploit Xi ’ s to bound the upward deviation of LΠ ( wt ) . For a fix θ the goal here is to bound E [ exp { θXi } ] ( this is a different θ ! ) . More precisely , let Ft be the sub-sigma field generated by variables wt from time zero to t , and Fi : = Fτi be the sigma field of the stop time τi . Then , obviously , Xi is measurable w.r.t Fi . We prove the following theorem : Theorem 8 For any θ > 0 , the sequence ( Eeθ ( Xi−X0 ) ) ∞i=1 is a supermartingale with respect to the filteration ( Fi ) , Proof We proceed inductively by jointly conditioning on the previous Xi and whether Gi has happened or not , and whether we are in situation ⋆ ( 2 ) or ⋆ ( 3 ) . We have E [ exp { θ ( Xi+1 −X0 ) } | Fi ] = E [ exp { θ ( Xi+1 −Xi +Xi −X0 ) } 1 { Gi } |Fi ] + E [ exp { θ ( Xi+1 −Xi +Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } |Fi ] + E [ exp { θ ( Xi+1 −Xi +Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } |Fi ] . Now by the a.s. bounds of Lemmas 24 and 26 : E [ Xi+1 −Xi| Ḡi , wτis.t . ⋆ ( 2 ) ] = −R21η2 , E [ Xi+1 −Xi| Ḡi , wτis.t . ⋆ ( 3 ) ] = −R22η2 , ( Xi+1 −Xi ) 1 { Gi } = 0 . ( a.s. ) Where R1 and R2 are r.v . defined in Lemmas 24 and 26 and are clearly Fi measurable . This implies E [ ( Xi+1 −Xi ) 1 { Ḡi , wτis.t . ⋆ ( 2 ) } | Fi ] = −R21η21 { Ḡi , wτis.t . ⋆ ( 2 ) } , E [ ( Xi+1 −Xi ) 1 { Ḡi , wτis.t . ⋆ ( 3 ) } | Fi ] = −R22η21 { Ḡi , wτis.t . ⋆ ( 3 ) } . Now we mention the following fact : Fact For a σ subGaussian random variable X we have E [ exp { θX } ] ≤ exp { θ2σ2 } . Using the a.s. bounds of Lemmas 24 and 26 , we get that conditioned on { Ḡi , wτi s.t . ⋆ ( 2 ) } , Xi+1 − EXi+1 is a.s. bounded by 2ηθR1/ ( ρ2χ ) , and conditioned on { Ḡi , wτi s.t . ⋆ ( 3 ) } , Xi+1 − EXi+1 is bounded by 2ηθR2/ ( ρ2χ ) . Therefore , using the above fact E [ exp { θ ( Xi+1 − E ( Xi+1| Ḡi , wτi s.t . ⋆ ( 2 ) ) ) } ∣∣∣ Ḡi , wτi s.t . ⋆ ( 2 ) ] ≤ exp { 4η2θ2R21/ ( ρ2χ ) } , ( 253 ) E [ exp { θ ( Xi+1 − E ( Xi+1| Ḡi , wτi s.t . ⋆ ( 3 ) ) ) ∣∣∣ Ḡi , wτi s.t . ⋆ ( 3 ) } ] ≤ exp { 4η2θ2R22/ ( ρ2χ ) } , ( 254 ) which implies in the notation of conditional expectation on sigma field : E [ exp { θ ( Xi+1 − E ( Xi+1|Fi ) ) } 1 { Ḡi , ⋆ ( 2 ) } ∣∣∣Fi ] ≤ exp { 4η2θ2R21/ ( ρ2χ ) } 1 { Ḡi , ⋆ ( 2 ) } , ( 255 ) E [ exp { θ ( Xi+1 − E ( Xi+1|Fi ) ) } 1 { Ḡi , ⋆ ( 3 ) } ∣∣∣Fi ] ≤ exp { 4η2θ2R22/ ( ρ2χ ) } 1 { Ḡi , ⋆ ( 3 ) } . ( 256 ) Now we write : LHS ≤ E [ exp { θ ( Xi −X0 ) } 1 { Gi−1 } | Fi ] +E [ exp { θ ( Xi+1 − E [ Xi+1| Fi ] ) } exp { θ ( E [ Xi+1| Fi ] −Xi ) } exp { θ ( Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } | Fi ] +E [ exp { θ ( Xi+1 − E [ Xi+1| Fi ] ) } exp { θ ( E [ Xi+1| Fi ] −Xi ) } exp { θ ( Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } |Fi ] ≤ exp { θ ( Xi −X0 ) } 1 { Gi } + exp { θ ( Xi −X0 ) } E [ exp { θ2R21η2/ ( ρ2χ ) } exp { −θ ( R21η2 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } | Fi ] + exp { θ ( Xi −X0 ) } E [ exp { θ2R22η2/ ( ρ2χ ) } exp { −θ ( R22η2 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } | Fi ] ≤ exp { θ ( Xi −X0 ) } E [ ( 1 { Gi } + exp { θ2R21η2/ ( ρ2χ ) − θ ( R21η2 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } + exp { θ2R22η2/ ( ρ2χ ) − θ ( R22η2 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } ) | Fi ] . Now setting θ : = 1 and picking χ ≥ 1/ρ2 : LHS ≤ exp { ( Xi −X0 ) } E [ 1 { Gi−1 } + 1 { Ḡi−1 ∩ ⋆ ( 2 ) } + 1 { Ḡi−1 ∩ ⋆ ( 3 ) } | Fi ] . = exp { Xi −X0 } . Now by hypothesis of Induction we have E [ exp { Xi+1 −X0 } ] = E [ E [ exp { Xi+1 −X0 } | Fi ] ] ≤ E [ exp { Xi −X0 } ] ≤ 1 , which finishes the proof of step of induction . Now using Doob ’ s Maximal inequality for positive supermartingales and R defined in ( 250 ) : P ( sup 1≤i≤R ( Xi −X0 ) ≥ z ) = P ( sup 1≤i≤R exp { Xi −X0 } ≥ exp { z } ) ≤ E [ exp { XR −X0 } ] / exp { z } ≤ e−z . ( 257 ) A.19 PROOF OF THEOREM 7 Finally with the developed tools , we are ready to prove Theorem 7 . Proof of Theorem 7 Starting from w0 = 0 with LΠ ( w0 ) ≤ ℵ , we use Equation ( 257 ) to get P ( sup1≤i≤R exp { Xi − X0 } ≥ Ω ( log ( 1/δ1 ) ) ) ≤ δ1 . Therefore , setting l = Θ ( log ( 1/δ1 ) ) and a union bound implies with probability at least 1 − 2δ1 we should have gotten into situation ⋆ ( 1 ) or ⋆ ( 4 ) without the value of Xi exceeding ℵ + 2l . On the other hand , using Lemma 26 we know that ET happens with probability at least 1− Õ ( η5 ) for every 1 ≤ t ≤ R which is equal to τi for some i and when we are in the situation of Lemma 26 . As a result , the chance that even one of ET ’ s happen along R iterations is at most RÕ ( η5 ) = η3 O ( log ( 1/δ1 ) ) ( ℵ+ 3l ) θ . But picking η small enough with respect to log ( δ1 ) and other parameters , we conclude that with probability at least 1 − 3δ , after R rounds , we should have got into situation ⋆ ( 1 ) and not ⋆ ( 4 ) and not exceeding ℵ+2l , which means that Xi = LΠ ( wτi ) has gotten under the threshold ℵℓ . Note that as soon as that happens , we terminate the algorithm . We elaborate on this more in Appendix A.10 . A.20 GAUSSIAN SMOOTHING In this section , we describe our smoothing scheme and the approximation that it provides which enables us to keep the signs from the case η = 0 . Recall that we use Gaussian smoothing matrices V ρj , k ∼ N ( 0 , β21/m1 ) and W ρ j , k ∼ N ( 0 , β22/m2 ) . Here , we will particularly specify lower bounds for β1 and β2 in order for our sign approximation to be precise . On the other hand , we normally prefer the smoothing noise to be as low as possible so the primary and smoothed functions are close , so we set β1 , β2 equal to their lower bounds , and use this setting in the other parts . To begin fix one of the inputs xi . In order to reduce and simplify the amount of notations , we refer to the sign pattern matrix ( diagonal sign matrix ) of both the first and second layers by D with the appropriate indices . More specifically , for the first layer , we refer to Sgn ( W ( 0 ) +W ′ +W ρ ) xi by D′ , ρ and Sgn ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ + √ ηW ∗ ) xi by D′ , ρ . Similarly , for the second layer , of course depending on the input vector , we refer to the sign matrix with respect to the matrices V ( 0 ) + V ′ + V ρ and V ( 0 ) + ( 1 − η/2 ) V ′ + V ρ +√ηV ∗ by by D′.ρ and D′ , ρ , η , respectively . We introduce two new notations as well for the output of the first layer with respect to different matrix and sign patterns : x′ ( 1 ) : =W sD′ , ρ ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi , ( 258 ) x′ ( 2 ) : =W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi . ( 259 ) For further brevity , we sometimes refer to x′ ( 2 ) by x′ . Now we are ready to mention our approximation theorem regarding the smoothing and the sign changes . Lemma 28 Under the conditions κ1 √ m3 ≳ C1 + β1 √ m3 and m2 ≥ m3 log ( m2 ) , then for every i ∈ [ n ] : ∣∣∣EWρ , V ρ aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sD′ , ρ ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi ∣∣∣ ≤ ηϱ22β−12 [ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } ] × [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] + η ( κ2 √ m2 + C1 ) ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) ϱ2m3√m3 β1 : = ηℜ8 . ( 260 ) Proof of Lemma 28 We can bound the Left hand side above as LHS ≤∣∣∣EWρ , V ρaTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi ∣∣∣ + ∣∣∣EWρ , V ρ aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sD′ , ρ ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi ∣∣∣ . : = A1 +A2 . ( 261 ) We bound A1 and A2 separately . First , we start with A1 . Let P̂i be the set of indices j for which 1 { | ( V ( 0 ) j + V ′j ) x′| ≤ R∗κ2∥x′∥ } happens . Then , from Lemma 31 , we have |P̂i| ≲ R∗m2 . Now for j ∈ [ m2 ] , we write 1 { sign change in thejth neuron } × |amount of change| ( 262 ) ≤ 1 { V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ + ηV ′j x′ − √ ηV ∗j x ′ , −V ( 0 ) j x ′ − V ′j x′ ) } × 1√ m2 ( |ηV ′j x′|+ | √ ηV ∗j x ′| ) ( 263 ) Moreover , note that |V ′j x′| = |V ′j ( x′ − ϕ ( 0 ) ( xi ) ) | , |V ∗j x′| = |V ∗j ( x′ − ϕ ( 0 ) ( xi ) ) | . Also , because ∥V ′j ∥ ≤ ∥V ′∥ ≤ 2C2 plus using Equation ( 129 ) , we can further upper bound the above indicator as : ≤ 1 { V ρj x ′ ∈ ( −V ( 0 ) j x ′−V ′j x′− ( η∥V ′j ∥+ √ η∥V ∗j ∥ ) min { ∥x′∥ , ∥x′−ϕ ( 0 ) ( xi ) ∥ } , −V ( 0 ) j x ′−V ′j x′ ) } × 1√ m2 ( η∥V ′j ∥+ √ η∥V ∗j ∥ ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≤ 1 { V ρj x ′ ∈ ( −V ( 0 ) j x ′−V ′j x′− ( 2ηC2+ √ ηϱ2/ √ m2 ) min { ∥x′∥ , ∥x′−ϕ ( 0 ) ( xi ) ∥ } , −V ( 0 ) j x ′−V ′j x′ ) } × 1√ m2 ( 2ηC2 + √ ηϱ2/ √ m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ . Taking √ η ≤ ϱ/ ( 2C2 √ m2 ) , we can further upper bound as ≲ 1 { V ρj x′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) } × ( √ηϱ2/m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ . Therefore , conditioned on x′ : EV ρ [ 1 { sign change in thejth neuron } × |amount of change| | x′ ] ≤ P ( V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) ) × ( √ηϱ2/m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ . Now notice that for j /∈ P̂i , we have | − V ( 0 ) j x ′ − V ′j x′| ≥ R∗κ2∥x′∥ . Also , note that the variable V ρj x ′ is gaussian with variance ∥x′∥β2/ √ m2 Therefore , conditioned on x′ , for j /∈ P̂i , we have ( note that x′ does not depend on the randomness of V ρ ) : P ( V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) ) ≲ exp { min { | − V ( 0 ) j x′ − V ′j x′ − 2 √ ηϱ2∥x′∥/ √ m2| , | − V ( 0 ) j x ′ − V ′j x′| } / ( √ 2∥x′∥β2/ √ m2 ) } 2 × ( ∥x′∥β2/ √ m2 ) −1 × ( √ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 ) . This equation follows from the fact that P ( a ≤ N ≤ b ) ≲ |a− b| σ emin 2 { a , b } /σ2 . ( 264 ) On the other hand , note that for √ η ≲ C 2/3 2 ( √ m2κ2 ) 1/3 ϱ2 we have : R∗κ2∥x′∥/2 = C 2/3 2 ( √ m2κ2 ) 1/3 2 √ m2 ∥x′∥ ≥ 2√ηϱ2∥x′∥/ √ m2 which implies ≲ exp { R∗κ2∥x′∥/ ( 2 √ 2∥x′∥β2/ √ m2 ) } 2 ( √ ηϱ2/β2 ) ≤ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } ( √ ηϱ2/β2 ) . On the other side , for j ∈ P̂i , we can write P ( V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) ) ≲ ( ∥x′∥β2/ √ m2 ) −1 ( √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 ) = min { ∥x′ − ϕ ( 0 ) ( xi ) ∥/∥x′∥ , 1 } √ ηϱ2/β2 . Therefore , overall using the fact that ∥V ∗j ∥ ≤ ϱ2/ √ m2 , we can write A1 ≲ ∑ j /∈P̂ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } ( √ ηϱ2/β2 ) min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } + ∑ j∈P̂ ( √ ηϱ2/β2 ) min { ∥x′ − ϕ ( 0 ) ( xi ) ∥/∥x′∥ , 1 } × ( √ ηϱ2/m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≲ [ m2 × exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } ( √ ηϱ2/β2 ) × ( √ ηϱ2/m2 ) + ηϱ22β −1 2 |P̂i| m2 ] min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } ≤ ηϱ22β−12 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] . ( 265 ) Next , we boundA2 . First we bound EWρ∥x′ ( 1 ) −x′ ( 2 ) ∥ . Recalling the setting c2 = 2 √ nm3C1/ √ λ0 and the definition of in P from Lemma 1 , we obtain that for j /∈ P , we have for all i ∈ [ n ] : |W ( 0 ) j xi| ≥ c2/ √ m1 , |W ′jxi| ≤ c2/ ( 2 √ m1 ) , which means for j /∈ P : | ( W ( 0 ) j +W ′ j ) xi| ≥ c2/ ( 2 √ m1 ) . ( 266 ) Also , we have |P | ≤ c2 √ m1/κ1 . ( 267 ) Now using Equation ( 105 ) in Lemma 5 , we can write for every i ∈ [ n ] : valj : = 1 { sign change in thejth neuron } × ∣∣∣amount of change∣∣∣ ≤ 1 { W ρj xi ∈ ( −W ( 0 ) j x−W ′ jxi + ηW ′ jxi − √ ηW ∗j xi , −W ( 0 ) j xi −W ′ jxi ) } × 1√ m1 ( |√ηW ∗j xi + ηW ′jxi| ) . Using the fact that ∥W ′j∥ ≤ ∥W ′∥F ≤ C1 , and Equation ( 105 ) ( ∥W ∗j ∥ ≤ ϱ √ m3 m1 ) and picking √ η ≤ ϱ √ m3 C1 √ m1 , we obtain ≤ 1 { W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − η∥W ′j∥ − √ η∥W ∗j ∥ , −W ( 0 ) j xi −W ′ jxi ) } × 1√ m1 ( √ η∥W ∗j ∥+ η∥W ′j∥ ) ≤ 1 { W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − 2 √ ηϱ √ m3√ m1 , −W ( 0 ) j xi −W ′ jxi ) } × 2√ m1 ( √ ηϱ √ m3 m1 ) . Now for j /∈ P , because W ρj xi is Gaussian with std β1√ m1 : EWρ [ valj ] ≤ P ( W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − 2 √ ηϱ √ m3√ m1 , −W ( 0 ) j xi −W ′ jxi ) ) × 1√ m1 √ ηϱ √ m3√ m1 ≲ exp− { min { | −W ( 0 ) j xi −W ′jxi − 2 √ ηϱ √ m3√ m1 | , | −W ( 0 ) j x ′ −W ′jx′| } / ( √ 2β1/ √ m1 ) } 2 × ( β1/ √ m1 ) −1 × ( √ηϱ √ m3√ m1 ) × 1√ m1 √ ηϱ √ m3√ m1 . Now from Equation ( 266 ) and by picking √ η ≲ c2 ϱ √ m3 so that 2 √ ηϱ √ m3√ m1 ≤ c2/ ( 4 √ m1 ) , then LHS ≲ exp { −c22/ ( 32β21 ) } η ϱ2m3 β1m1 . ( 268 ) On the other hand , for j ∈ P we have Evalj ≤ P ( W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − 2 √ ηϱ √ m3√ m1 , −W ( 0 ) j xi −W ′ jxi ) ) × 1√ m1 √ ηϱ √ m3√ m1 ≲ ( β1/ √ m1 ) −1√ηϱ √ m3√ m1 ×√ηϱ √ m3 m1 = η ϱ2m3 β1m1 . ( 269 ) Now define the following random variable with respect to the randomness of W ρ : V al : = m1∑ j=1 1 { sign change in thejth neuron } × |amount of change| then for every k ∈ [ m3 ] , we have |x′ ( 1 ) k − x ′ ( 2 ) k | ≤ V al , which implies ∥x′ ( 1 ) − x′ ( 2 ) ∥ ≤ √ m3V al . But Combining Equations ( 268 ) and ( 269 ) : EV al ≤ ( exp { −c22/ ( 32β21 ) } + |P | m1 ) η ϱ2m3 β1 ≤ ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) η ϱ2m3 β1 , which implies EWρ∥x′ ( 1 ) − x′ ( 2 ) ∥ ≤ ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) η ϱ2m3 √ m3 β1 . ( 270 ) Now we can write∣∣∣aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 2 ) − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 1 ) ∣∣∣ ≤ 1√ m2 m2∑ j=1 | ( V ( 0 ) j + ( 1− η ) V ′ j + V ρ j + √ ηV ∗j ) ( x ′ ( 2 ) − x′ ( 1 ) ) | ≤ 1√ m2 m2∑ j=1 |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ ( 1− η ) |V ′j ( x′ ( 2 ) − x′ ( 1 ) ) |+ |V ρ j ( x ′ ( 2 ) − x′ ( 1 ) ) |+√η|V ∗j ( x′ ( 2 ) − x′ ( 1 ) ) | = 1√ m2 m2∑ j=1 |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ |V ρj ( x ′ ( 2 ) − x′ ( 1 ) ) |+ 1√ m2 m2∑ j=1 ( ( 1− η ) ∥V ′j ∥+ √ η∥V ∗j ∥ ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ ≤ 1√ m2 m2∑ j=1 |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ |V ρj ( x ′ ( 2 ) − x′ ( 1 ) ) |+ ( ( 1− η ) ∥V ′∥F + √ η∥V ∗∥F ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ . Now by Equation ( 127 ) in Lemma 10 ( i.e . ∥V ∗∥F ≲ √ ζ2 ) and the fact that ∥V ′∥F ≤ C2 , and by taking √ η ≤ C2√ ζ2 , we have ( ( 1− η ) ∥V ′∥F + √ η∥V ∗∥F ) ≲ C1 , ( 271 ) so we can bound the above as LHS ≲ 1√ m2 m2∑ j=1 ( |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ |V ρj ( x ′ ( 2 ) − x′ ( 1 ) ) | ) + C1∥x′ ( 2 ) − x′ ( 1 ) ∥ . Furthermore , using Lemma 32 and noting the fact that the entries of V ( 0 ) are normal with standard deviation κ2 , we get with high probability over the randomness of V ( 0 ) : ≲ 1√ m2 m2∑ j=1 |V ρj ( x ′ ( 2 ) −x′ ( 1 ) ) |+ ( κ2 √ m2+κ2 √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) +C1 ) ∥x′ ( 2 ) −x′ ( 1 ) ∥ . Now note that V ρj ( x ′ ( 2 ) −x′ ( 1 ) ) is normal with standard deviation β2√ m2 ∥x′ ( 2 ) −x′ ( 1 ) ∥ . Hence , taking expectation with respect to V ρ : EV ρ ∣∣∣aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 2 ) − aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 1 ) ∣∣∣ ≲ ( κ2 √ m2 + κ2 √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) + C1 ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ . ( 272 ) Finally , Combining Equation ( 265 ) and ( 272 ) and applying it to Equation ( 261 ) implies with high probability over the random initialization : ∣∣∣EWρ , V ρ aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sx′ ( 2 ) − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sx′ ( 1 ) ∣∣∣ ≤ A1 +A2 ≲ EWρηϱ22β−12 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] + EWρ ( κ2 √ m2 + κ2 √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) + C1 ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ Now notice that under Ec , using the assumption κ1 √ m3 ≳ C1 + √ m3β1 and Lemma 33 we have ∥x′∥ ≥ ∥ϕ ( 0 ) ( xi ) ∥ − ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≥ κ1 √ m3 − ( C1 + √ m3β1 ) ≳ κ1 √ m3 , and ∥x′ − ϕ ( 0 ) ( xi ) ∥2 ≤ ( C1 + √ m3β1 ) 2 , ( 273 ) which implies : EWρ min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } ≤ EWρ1 { Ec } ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥+ 1 { E } ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≲ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + EWρ1 { E } ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≲ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } . Substituting this above and further applying the result of Lemma 33 and Equation ( 270 ) and the assumption that m2 ≥ m3 log ( m2 ) : A1 +A2 ≲ ηϱ22β−12 [ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } ] × [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] + η ( κ2 √ m2 + C1 ) ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) ϱ2m3√m3 β1 , which completes the proof . A.20.1 SETTING β1 AND β2 As we mentioned , to minimize the amount of deviation of the smoothed function compared to the original one , we prefer to choose β1 , β2 as small as possible . ( The benefit of such choice , indeed , can be observed more explicitly in other parts of the proof , e.g . Appendix A.14 . ) Observing the bound in Equation ( 260 ) and noting that we can easily make the exponential terms orders of magnitude smaller than the poly terms , it is easy to find the following optimal setting for the smoothing parameters : β2 : = Θp ( ( κ1 √ m3 ) −1 ( √ m2κ2 ) − 23 ) , β1 : = Θp ( m3 √ m3/ ( κ1 √ m1 ) ) . Using this setting , we still can make ℜ8 arbitrarily small . Here , we remind the reader that Op only cares about the non-logarithmic dependencies on the overparameterization , i.e . m1 , m2 , m3 , κ1 , κ2 . A.21 BASIC TOOLS In this section , we introduce and prove some lemmas that we use in our analysis as basic tools . Lemma 29 Suppose V ( 0 ) ∈ Rm2×m3 has standard normal entries and a is a random sign vector . Suppose theta > 1 , R < 1 are given thresholds , such that m2R ≳ m3 ( log ( 1/R ) + log ( m3 ) + log ( log ( m2 ) ) , e−θ 2/8 ≲ m3/m2 . Then , for the following quantities : N1R ( x ) = # ( j ∈ [ m ] : |V ( 0 ) j x| ≤ R ) N2θ ( x ) = # ( j ∈ [ m ] : |V ( 0 ) j x| ≥ θ ) , with high probability we have sup ∥x′∥=1 N1R ( x ′ ) ≲ m2R , sup ∥x′∥=1 N2θ ( x ′ ) ≲ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Proof of Lemma 29 Suppose B1 ( ϵ ) is a cover for the Euclidean ball in Rm3 with precision ϵ . We know |B1 ( ϵ ) | ≲ ( 1/ϵ ) m3 . Now for a fixed ∥x∥ = 1 , we have P ( W ( 0 ) j x ≤ 2R ) ≲ R. Therefore , using Bernstein , with high probability we have # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≤ 2R ) ≲ m2R+ √ m2R+ 1 . Hence , using union bound , we have with high probability sup x∈B1 ( ϵ ) # ( j ∈ [ m ] : |V ( 0 ) j x| ≤ 2R ) ≲ m2R+ √ log |B1 ( ϵ ) | √ m2R+ log |B1 ( ϵ ) | = m2R+ √ m2Rm3 log ( 1/ϵ ) +m3 log ( 1/ϵ ) . By picking ϵ ≲ R/ ( √ m3 log ( m2m3 ) ) , The assumption implies m2R ≥ m3 log ( 1/ϵ ) , which implies LHS ≲ m2R . On the other hand , note that with high probability we have sup j∈ [ m2 ] , k∈ [ m3 ] |V ( 0 ) j , k | ≤ √ log ( m2m3 ) . ( 274 ) Now for ∥x′∥ = 1 which is not in the cover , if x is the closest point to it in the cover , i.e . x ∈ B1 ( ϵ ) and ∥x− x′∥ ≤ ϵ , then for every j ∈ [ m2 ] we have ||V ( 0 ) j x| − |V ( 0 ) j x ′|| ≤ ∥V ( 0 ) j ∥∥x− x ′∥ ≤ √ m3 log ( m2m3 ) ϵ ≤ R , by picking a small enough constant . Therefore , for a j that |V ( 0 ) j x| ≥ 2R , then |V ( 0 ) j x ′| ≥ 2R−R = R. Therefore , we get that with high probability , for every ∥x′∥ = 1 : sup ∥x′∥=1 # ( j ∈ [ m2 ] : |V ( 0 ) j x ′| ≤ R ) ≲ m2R . For the second part , note that for ∥x∥ = 1 , by the tail bound for normal vars : P ( W ( 0 ) j x ≥ θ/2 ) ≲ e−θ 2/8 . Hence , again using Bernstein , we have with high probability sup x∈B1 ( ϵ ) # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≥ θ/2 ) ≲ m2e−θ 2/8 + √ log |B1 ( ϵ ) | √ m2e−θ 2/8 + log |B1 ( ϵ ) | ≲ m2e−θ 2/8 + √ m3 log ( 1/ϵ ) √ m2e−θ 2/8 +m3 log ( 1/ϵ ) . By picking ϵ ≲ 1/ ( √ m3 log ( m2m3 ) ) , and using the assumption m2e−θ 2/8 ≲ m3 , all terms are dominated by the third term so we can bound the above as sup x∈B1 ( ϵ ) # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≥ θ/2 ) ≲ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Now for ∥x′∥ = 1 not in the cover , for the new ϵ we can write ||V ( 0 ) j x| − |V ( 0 ) j x ′|| ≤ ∥V ( 0 ) j ∥∥x− x ′∥ ≤ √ m3 log ( m2m3 ) ϵ ≤ 1/2 ≤ θ/2 . Hence , with high prob . sup ∥x′∥=1 # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≥ θ ) ≲ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Lemma 30 For x ∈ Rd and W ( 0 ) ∈ Rm×d which has standard normal entries ( and a is a random sign vector ) , we have with high probability : sup ∥x∥=1 f ( x ) : = 1√ m aTσ ( W ( 0 ) x ) ≤ √ d. Proof of Lemma 30 For the first part , we first compute an upper bound on E sup ∥x∥=1 1√ m aTσ ( W ( 0 ) x ) . To do so , we use Dudley ’ s chaining . Note that the for x1 , x2 ∈ Rd , the variable σ ( W ( 0 ) j x1 ) − σ ( W ( 0 ) j x2 ) is subGaussian with parameter ∥x1 − x2∥ , so the variable f ( x1 ) − f ( x2 ) is also sub- Gaussian with parameter ∥x1 − x2∥ . Hence , by Dudley ’ s integral : E sup ∥x∥=1 1√ m aTσ ( W ( 0 ) x ) ≤ ∫ 1 0 √ log ( N ( Bd1 , ϵ ) ) ≲ √ d. Now for a fixed x , note that 1√ m aTσ ( W1x ) − 1√ m aTσ ( W2x ) ≤ 1√ m m∑ j=1 ∥W1j −W2j∥ ≤ ∥W1 −W2∥F . Hence , the function f ( x ) is 1-lipchitz with respect to W and l2 norm , so is the function sup f ( x ) . Hence , by Gaussian concentration , sup f ( x ) is 1-subGaussian around its mean , so we finally get with high probability sup f ( x ) ≲ √ d+ 1 ≲ √ d. Lemma 31 For R∗ : = C 2/3 2 ( √ m2κ2 ) 2/3 , we have with high probability over the randomness of V ( 0 ) : sup x′ , V ′ : ∥V ′∥≤C2 # ( j ∈ [ m2 ] : | ( V ( 0 ) j + V ′ j ) x ′| ≤ R∗κ2∥x′∥ ) ≲ R∗m2 . Proof of Lemma 31 Note that obviously the condition of Lemma 29 is satisfied with this choice of R = R∗ . Therefore , with high probability we have for an arbitrary x′ : # ( |V ( 0 ) j x ′| ≤ 2R∗κ2∥x′∥ ) ≤ m2R∗ . On the other hand , note that for j ∈ [ m2 ] such that |V ′j x′| ≥ Rκ2∥x′∥ , we have ∥V ′j ∥∥x′∥ ≥ |V ′j x′| ≥ Rκ2∥x′∥ , which implies ∥V ′j ∥ ≥ Rκ2 . Therefore , there are at most C 2 2 R2κ22 . Therefore , setting aside m2R+ C22 R2κ22 of j ’ s , for the rest we have | ( V ( 0 ) j + V ′ j ) x ′| ≥ |V ( 0 ) j x ′| − |V ′j x′| ≥ 2Rκ2∥x′∥ −Rκ2∥x′∥ = Rκ2∥x′∥ . Setting R∗ as defined above balances the terms m2R and C22 R2κ22 , which completes the proof . Lemma 32 If V ( 0 ) ∈ Rm2×m3 is a matrix with standard normal entries , then with high probability sup ∥x′∥=1 1√ m2 m2∑ j=1 |V ( 0 ) j x ′| ≲ √m2 + √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Proof of Lemma 32 LetB1 ( ϵ ) be a cover for the unit Euclidean ball with precision ϵ , for which we have |B1 ( ϵ ) | ≲ ( 1ϵ ) m3 . Now for a fixed x ∈ B1 ( ϵ ) , note that because V ( 0 ) j x is a standard normal variable , the random variable |V ( 0 ) j x′|−E|V ( 0 ) j x ′| isO ( 1 ) -subGaussian , which means 1√ m2 ∑m2 j=1 ( |V ( 0 ) j x ′|−E|V ( 0 ) j x′| ) is also O ( 1 ) −subGaussian . Now from the tail of maximum of subGaussian variables : sup x∈B1 ( ϵ ) 1√ m2 m2∑ j=1 ( |V ( 0 ) j x| − E|V ( 0 ) j x| ) ≲ √ log ( |B1 ( ϵ ) | ) = √ m3 log ( 1/ϵ ) . On the other hand , note that E|V ( 0 ) j x′| ) = O ( 1 ) , which implies w.h.p : sup x∈B1 ( ϵ ) 1√ m2 m2∑ j=1 |V ( 0 ) j x| ≲ √ m2 + √ m3 log ( 1/ϵ ) . Moreover , note that again by the tail of subGaussian variables , we have w.h.p : max j∈ [ m2 ] , k∈ [ m3 ] |V ( 0 ) j , k | ≲ √ log ( m2m3 ) , which implies with high prob for every j ∈ [ m2 ] : ∥V ( 0 ) j ∥ ≲ √ m3 log ( m2m3 ) . Now by picking ϵ : = ( √ m3 log ( m2m3 ) ) −1 , we get with high probability sup x∈B1 ( ϵ ) 1√ m2 m2∑ j=1 |V ( 0 ) j x| ≲ √ m2 + √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . ( 275 ) On the other hand , for an arbitrary x′ with ∥x′∥ = 1 , if x ∈ B1 ( ϵ ) is the representative of x′ , we have by definition ∥x′ − x∥ ≤ ϵ , which combined with ( A.21 ) implies∣∣∣|V ( 0 ) j x′| − |V ( 0 ) j x|∣∣∣ ≤ |V ( 0 ) j ( x′ − x ) | ≤ ∥V ( 0 ) j ∥∥x′ − x∥ , ≲ √ m3 log ( m2m3 ) ( √ m3 log ( m2m3 ) ) −1 ≤ 1 . Therefore ∣∣∣ 1√ m2 m2∑ j=1 |V ( 0 ) j x ′| − 1√ m2 m2∑ j=1 |V ( 0 ) j x| ∣∣∣ ≤ √m2 . ( 276 ) Combining Equations ( 275 ) and ( 276 ) , we conclude the result . A.21.1 DEFINING THE RARE EVENTS Ej Lemma 33 For x′ ( 2 ) defined in Equation ( 259 ) we have EWρ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ , EWρ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 + √ m3β1 , EWρ∥ϕ ( 2 ) ( xi ) ∥2 ≲ C21 +m3β21 . Moreover , for the events Ej = { |W ρj xi| ≥ C1/ √ m3m1 } , E = ∪jEj , we have under Ec : ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ , ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 . Furthermore , E happens rarely : P ( E ) ≲ m1 exp { −C21/ ( 8m3β21 ) } , EWρ1 { E } ∥ϕ ( 2 ) ( xi ) ∥ ≤ [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } . EWρ1 { E } ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ ≤ [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } . Finally , we have the following almost surely bound : ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 + √ m3 m1∑ j=1 1√ m1 |W ρj xi| . Proof of Lemma 33 We start by writing |x′ ( 2 ) k − 1√ m1 W skσ ( W ( 0 ) + ( 1− η ) W ′ +√ηW ∗ ) xi| ≤ m1∑ j=1 1√ m1 |W ρj xi| . ( 277 ) Now notice that by Lemma 1 , we know for every j /∈ P̃ : |W ( 0 ) j xi| ≥ c2/ √ m2 , ( 278 ) | ( 1− η ) W ′jxi| ≤ c2/ ( 2 √ m1 ) . ( 279 ) In addition , by Equations in ( 105 ) from Lemma 5 , for every j ∈ [ m1 ] : ∥W ∗j ∥ ≤ ϱ1 √ m3 m1 , so by picking η ≤ c2/ ( 4ϱ √ m3 ) we obtain η|W ∗j xi| ≤ c2 4 √ m1 . ( 280 ) Combining this with Equations in ( 279 ) , we see that the signs of ( W ( 0 ) j + ( 1− η ) W ′j + √ ηW ∗j ) xi and W ( 0 ) j xi are the same for j /∈ P̃ . Moreover , the matrix ( 1− η ) W ′ +√ηW ∗ satisfies ∥ ( 1− η ) W ′ +√ηW ∗∥ ≤ ( 1− η ) C1 + √ η √ 2ζ2 ≤ C1 , by picking √ η ≤ C1/ √ ζ2 . Hence , the conditions of Lemma 6 are satisfied and we get : ∥ 1√ m1 W sσ ( W ( 0 ) + ( 1− η ) W ′ +√ηW ∗ ) xi − ϕ ( 0 ) ( xi ) ∥ ≤ C1 . ( 281 ) Combining Equations ( 277 ) and ( 281 ) : ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ ≤ C1 + √ m3 m1∑ j=1 1√ m1 |W ρj xi| . ( 282 ) In exactly similar fashion , one can derive ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 + √ m3 m1∑ j=1 1√ m1 |W ρj xi| . ( 283 ) Now first of all , note EWρ m1∑ j=1 1√ m1 |W ρj xi| ≤ β1 , which proves the first part of the claims . For the second part , note that by the Gaussian tail bound P ( |W ρj xi| ≥ C1/ √ m3m1 ) ≲ exp { −C21/ ( 8m3β21 ) } . Therefore , P ( E ) ≤ ∑ j P ( Ej ) ≤ m1 exp { −C21/ ( 8m3β21 ) } . Moreover EWρ1 { E } m1∑ j=1 1√ m1 |W ρj xi| ≤ EWρ 1√ m1 ∑ j2 ∑ j ̸=j2 1 { Ej2 } |W ρ j xi|+ 1√ m1 ∑ j EWρ1 { Ej } |W ρj xi| , = [ 1√ m1 ∑ j2 ∑ j ̸=j2 EWρ |W ρj xi|+ 1√ m1 ∑ j EWρ [ |W ρj xi| ∣∣∣ Ej ] ] P ( Ej ) ≲ [ m1β1 + C1/ √ m3 ] exp { −C21/ ( 8m3β21 ) } . Plugging this into ( 282 ) finishes the proof . Also , under Ec by Equation ( 282 ) we have ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ , ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 . Finally , exploiting Equation ( 283 ) : EWρ∥ϕ ( 2 ) ( xi ) ∥2 ≲ C21 +m3 1 m1 EWρ ( ∑ j |W ρj xi| ) 2 ≤ C21 +m3EWρ ∑ j |W ρj xi| 2 ≤ C21 +m3β21 . A.21.2 BOUNDING THE VALUE OF f ′ The following Lemma provides a reasonable bound on the value of the smoothed function . Lemma 34 We have the following general bound on the values of the smoothed function : With high probability over the initialization , for ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 and ∀i ∈ [ n ] ( having small enough choices of β1 , β2 described in Appendix A.20 ) : |f ′W ′ , V ′ ( xi ) | ≲ ( κ2 √ m3 + β2 ) ( √ m3κ1 + C1 + √ m3β1 ) + C2 ( C1 + √ m3β1 ) , which is O ( C1C2 ) for large enough overparameterization as described in AppendixA.3 . Moreover , we have the following almost surely bound ( with respect to the randomness of W ρ and V ρ ) : |fW ′+Wρ , V ′+V ρ ( xi ) | ≲ ( κ2 √ m3 + ∥V ρ∥F ) ( √ m3κ1 + C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) + C2 ( C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) . Notably , with slightly higher overparameterization , the high probability bound in ( 34 ) holds even if we take supremum over x . Proof of Lemma 34 Using Lemmas 30 and 33 and using the fact that ϕ ( 0 ) ( xi ) is orthogonal to the rows of V ′ ( recall x′i = ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) : |fW ′+Wρ , V ′+V ρ ( xi ) | ≤ 1√ m2 aTσ ( V ( 0 ) x′i ) + 1√ m2 ∑ j | ( V ρj + V ′ j ) x ′ i| ≤ ( κ2 √ m3 ) ∥x′i∥+ 1√ m2 ∑ j |V ρj x ′ i|+ C2∥ϕ ( 2 ) ( xi ) ∥ ≤ ( κ2 √ m3 + ∥V ρ∥F ) ∥x′i∥+ C2∥ϕ ( 2 ) ( xi ) ∥ ≲ ( κ2 √ m3 + ∥V ρ∥F ) ( √ m3κ1 + C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) + C2 ( C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) . ( 284 ) Note that above , if we apply the stronger worst-case norm bound of the first layer ’ s output presented in Lemma 46 , we would get supx , ∥x∥=1 |fW ′+Wρ , V ′+V ρ ( x ) | is bounded by the RHS , which in turn proves a stronger uniform bound on f ′ . Similarly , this time by taking expectation with respect to W ρ and V ρ : |f ′W ′ , V ′ ( xi ) | = |EWρ , V ρfW ′ , V ′ ( xi ) | ≤ EWρ , V ρ |fW ′ , V ′ ( xi ) | = EWρ ( κ2 √ m3 + β2 ) ∥x′i∥+ C2∥x′i − ϕ ( 0 ) ( xi ) ∥ ≲ ( κ2 √ m3 + β2 ) ( √ m3κ1 + C1 + √ m3β1 ) + C2 ( C1 + √ m3β1 ) . Corollary 8.1 If we set C1 = C2 = 0 above , we get |f ′0,0 ( xi ) | ≤ ( κ2 √ m3 + β2 ) ( √ m3κ1 + √ m3β1 ) , the point being these terms go to zero by an order of O ( ( √ m2κ2 ) − 23 ) . Therefore , taking ( √ m2κ2 ) − 23 < < B , we make sure that |f ′0,0| < B , so by the 1 smoothness of ℓ and B boundedness of the labels we get ℓ ( f ′0,0 ( xi ) , yi ) < 4B 2 . A.21.3 BOUNDING THE DIFFERENCE BETWEEN ORIGINAL AND SMOOTHED FUNCTIONS The following Lemma bounds the difference between the smoothed function and original function of the network . Lemma 35 Bound on the smoothing change under the assumption m2 ≥ m3 log ( m2 ) : with high probability over the initialization , for any ( W ′ , V ′ ) with ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 : |fW ′ , V ′ ( xi ) − f ′W ′ , V ′ ( xi ) | ≤ β2 ( κ1 √ m3 + C1 + √ m3β1 ) + ( C2 + κ2 √ m2 ) √ m3β1 . Proof of Lemma 35 We write |fW ′ , V ′ ( xi ) − f ′W ′ , V ′ ( xi ) | = |fW ′ , V ′ ( xi ) − EWρ , V ρfW ′+Wρ , V ′+V ρ ( xi ) | = ∣∣∣EWρ , V ρ ( fW ′ , V ′ ( xi ) − fW ′+Wρ , V ′+V ρ ( xi ) ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣fW ′ , V ′ ( xi ) − fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣ . In the following , σ means we apply Relu activation to the vector in front of it ( entrywise ) : LHS ≤ EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ + V ρ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi − 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi ∣∣∣ +EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi − 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ ) xi ∣∣∣ . Now for the first term above , using the previous notation of x′i representing the output of the first layer and using Lemma 33 : EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ + V ρ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi − 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi ∣∣∣ ≤ EWρ , V ρ 1√ m2 ∑ j |V ρj x ′ i| ≲ β2EWρ∥x′i∥ ≤ β2EWρ , V ρ ( ∥ϕ ( 0 ) ( xi ) ∥+ ∥ϕ ( 2 ) ( xi ) ∥ ) ≲ β2 ( κ1 √ m3 + C1 + √ m3β1 ) . ( 285 ) For the second term , by starting off with a simple triangle inequality : EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ ) x′i − 1√ m2 aTσ ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣ ≤ EWρ , V ρ 1√ m2 m2∑ j=1 ∣∣∣ ( V ( 0 ) j + V ′j ) ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣ ≤ EWρ , V ρ 1√ m2 m2∑ j=1 ∣∣∣V ( 0 ) j ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣+ ∣∣∣V ′j ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) |∣∣∣ ≤ C2EWρ∥x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ∥+ EWρ 1√ m2 m2∑ j=1 ∣∣∣V ( 0 ) j ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣ . Now using Lemma 32 : ≲ ( C2 + κ2 √ m2 ) EWρ∥x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ∥ ≲ ( C2 + κ2 √ m2 ) √ m3EWρ ∑ j 1√ m1 |W ρj xi| ≲ ( C2 + κ2 √ m2 ) √ m3β1 . ( 286 ) Combining Equations ( 285 ) and ( 286 ) we conclude the proof . B APPENDIX CONTENTS B.1 Smoothness coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 B.1.1 Computing the Lipschitz Coefficients of f ′W ′ , V ′ . . . . . . . . . . . . . . . 98 B.2 Representation Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 B.2.1 Representation Toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 B.2.2 Some Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 B.2.3 Bound on the norm of x̄i ’ s . . . . . . . . . . . . . . . . . . . . . . . . . . 106 B.3 Coupling for ∇̂W , ∇̂V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 B.4 Handling the Injected Noise by PSGD . . . . . . . . . . . . . . . . . . . . . . . . 113 B.4.1 Bounding the Norm of the first Layer ’ s Output in the Worst Case . . . . . . 114 B.1 SMOOTHNESS COEFFICIENTS Recall that for a function f ∈ C3 on Rd , we say it is µ1 lipschitz , µ2 gradient lipschitz , and µ3 hessian lipschitz at point x if for every unit direction v , | ddλf ( x+ λv ) | ≤ µ1 , | d2 dλ2 f ( x+ λv ) | ≤ µ2 , and | d 2 dλ2 f ( x+ λv ) | ≤ µ3 . The aim of this section is to bound the lipschitz coefficients of the loss ℓ ( , y ) and objectiveL ( W ′ , V ′ ) in a bounded domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 . The following is our main Theorem in this regard : Theorem 9 For given values C1 , C2 > 0 , in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , for any label |y| ≤ B , the loss function ℓ ( . , y ) isO ( ( C1C2+B2 ) ) -lipschitz ( having enough overparameterization ) and 1 gradient-lipschitz x = f ′W ′ , V ′ . Moreover , the loss function L ( W ′ , V ′ ) is ( O ( C1C2 ) +B ) Ψ1+ 2 ( C1+C2 ) lipschitz , Ψ21+ ( O ( C1C2 ) +B ) Ψ2+4 gradient lipschitz , and 3Ψ2Ψ1+ ( O ( C1C2 ) +B ) Ψ3 hessian lipschitz , where Ψ1 , Ψ2 , Ψ3 are defined in Lemma 36 . Proof of Lemma 9 As in the proof of Lemma 36 , let ( W̃ , Ṽ ) be a unit direction , i.e . ∥W̃∥2 + ∥Ṽ ∥2 = 1 . Then , using Lemma 34 , we know that for every i ∈ [ n ] : |f ′W ′ , V ′ ( xi ) | = O ( C1C2 ) , so by 1-smoothness of the loss and B-boundedness of the labels , we get that ℓ ( . , y ) is ( O ( C1C2 ) + B ) lipshcitz at point f ′W ′ , V ′ . The gradient smoothness parameter of the square loss ℓ is bounded by 1 and its third derivative is zero . Now using these coefficients , we can easily compute the coefficients for L as well by simple differentiation : | d dλ ℓ ( f ′ W ′+λW̃ , V ′+λṼ ( xi ) , yi ) | = |ℓ̇ ( f ′ , yi ) d dλ f ′| ≤ ( O ( C1C2 ) +B ) Ψ1 . | d dλ2 ℓ ( f ′ W ′+λW̃ , V ′+λṼ ( xi ) , yi ) | = |ℓ̈ ( f ′ , yi ) ( d dλ f ′ ) 2 + ℓ̇ ( f ′ , yi ) d2 dλ2 f ′| ≤ Ψ21 + ( O ( C1C2 ) +B ) Ψ2 . | d dλ3 ℓ ( f ′ W ′+λW̃ , V ′+λṼ ( xi ) , yi ) | = | ... ℓ ( f ′ , yi ) ( d dλ f ′ ) 3 + 3ℓ̈ ( f ′ , yi ) d2 dλ2 f ′ d dλ f ′ + ℓ̇ ( f ′ , yi ) d3 dλ3 f ′| ≤ Ψ31 + 3Ψ2Ψ1 + ( O ( C1C2 ) +B ) Ψ3 . Moreover , note that d dλ ∥W ′ + λW̃∥2 = 2⟨W ′ + λW̃ , W̃ ⟩ ∣∣∣ λ=0 = 2⟨W ′ , W̃ ⟩ ≤ 2∥W ′∥ = 2C1 , d2 dλ2 ∥W ′ + λW̃∥2 = ⟨W̃ , W̃ ⟩ = 2 , d3 dλ3 ∥W ′ + λW̃∥2 = 0 , and similarly for ∥V ′ + λṼ ∥2 . Combining these results finishes the proof . Above , we used parameters Ψ1 , Ψ2 , Ψ3 , the lipschitz coefficients of f ′ in domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , which we bound in Lemma 36 below . B.1.1 COMPUTING THE LIPSCHITZ COEFFICIENTS OF f ′W ′ , V ′ In this section , we bound the lipschitz coefficients of f ′W ′ , V ′ in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 by poly ( m1 , m2 , m3 , β1 , β2 ) functions . Lemma 36 For every point ( W ′ , V ′ ) in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , we have the following bounds on the lipschitz coefficients of f ′W ′ , V ′ ( ( W̃ , Ṽ ) is a unit direction with ∥W̃∥2 + ∥Ṽ ∥2 = 1 ) : ∣∣∣ d dλ f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ ≲ m2 β22 ( β2√ m2 √ m3 ( κ1 + C1 + β1 ) ( κ2 √ m3 + C2 ) + β22√ m2 √ m3 ( κ1 + C1 + β1 ) ) + m1 β21 ( √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ( κ2 √ m3 + C2 ) + β2 √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ) : = Ψ1 , d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) √ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] : = Ψ2∣∣∣ d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 3/2√ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] : = Ψ3 ( 287 ) Proof of Lemma 36 Let ρ ( W ρ , V ρ ) : = 1 ( √ 2π ) m2m3+m1d ( β1/ √ m1 ) m1d ( β2/ √ m2 ) m2m3 exp { − ∥W ρ∥2 2β21/m1 − ∥V ρ∥2 2β22/m2 } , be the density function of the law of W ρ and V ρ which is a joint Gaussian . Then to compute the derivative and second derivative of the function in the unit direction ( W̃ , Ṽ ) , s.t . ∥W̃∥2F + ∥Ṽ ∥2F , we can write the value of the smoothed function as an integration with density ρ , change variable , and then take derivatives : d dλ f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 = d dλ EWρ , V ρfW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) = d dλ ∫ fW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) ρ ( W ρ , V ρ ) d ( W ρ , V ρ ) = d dλ ∫ fW+ , V + ( xi ) ρ ( W + − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) . But one can easily see that for fixed V ′ and Ṽ , the set of functions fW+ , V + ( xi ) ρ ( W+ − ( W ′ + λW̃ ) , V +− ( V ′ + λṼ ) ) for a small neighborhood of λ can simultaneously be upper bounded by an integrable function . Hence , the Leibnitz rule holds here because of dominated convergence theorem , and we can change the order of integration and derivation : = ∫ fW+ , V + ( xi ) d dλ ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = − ∫ fW+ , V + ( xi ) 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉 ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) 〈 ( W̃ , Ṽ ) , ( m1 β21 W ρ , m2 β22 V ρ ) 〉 ρ ( W ρ , V ρ ) d ( W ρ , V ρ ) = EWρ , V ρ ( m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉 ) fW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) ( xi ) ∣∣∣ λ=0 = EWρ , V ρ ( m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉 ) fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) . ( 288 ) Similarly we can compute the second derivative : d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 = d dλ ∫ fW+ , V + ( xi ) 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉 ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉2 −〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1W̃ , ( β22 m2 ) −1Ṽ ) 〉 ] ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( m1 β21 W ρ , m2 β22 V ρ ) 〉2 + 〈 ( W̃ , Ṽ ) , ( m1 β21 W̃ , m2 β22 Ṽ ) 〉 ] ρ ( W ρ , V ρ ) d ( W ρ , V ρ ) = EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 − ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] fW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) ( xi ) ∣∣∣ λ=0 = EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 − ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) . ( 289 ) Similarly for the third derivative : d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 d dλ ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉2 −〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1W̃ , ( β22 m2 ) −1Ṽ ) 〉 ] ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉3 − 3 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1W̃ , ( β22 m2 ) −1Ṽ ) 〉 ] ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ] . Now for first derivative , exactly similar to the derivation in ( 284 ) , we can write∣∣∣EWρ , V ρ ( m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉 ) fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉∣∣∣ ( κ2√m3∥x′ ( 2 ) i ∥+ ∥V ′∥F ∥x′ ( 2 ) i ∥+ EV ρ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) ≤ EWρ , V ρ ( ∣∣∣m1 β21 〈 W̃ , W ρ 〉∣∣∣+ ∣∣∣m2 β22 〈 Ṽ , V ρ 〉∣∣∣ ) ( κ2√m3∥x′ ( 2 ) i ∥+ ∥V ′∥F ∥x′ ( 2 ) i ∥+ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) = m2 β22 ( EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣EWρ∥x′ ( 2 ) i ∥ ( κ2√m3 + ∥V ′∥F ) + EWρEV ρ 1√m2 ∣∣∣〈Ṽ , V ρ〉∣∣∣∑ j |V ρj x ′ ( 2 ) i | ) + m1 β21 ( ( EWρ∥x′ ( 2 ) i ∥ ∣∣∣〈W̃ , W ρ〉∣∣∣ ) ( κ2√m3 + ∥V ′∥F ) + EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣EV ρ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) ≲ m2 β22 ( EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣EWρ∥x′ ( 2 ) i ∥ ( κ2√m3 + C2 ) + EWρEV ρ 1√m2 ∣∣∣〈Ṽ , V ρ〉∣∣∣∑ j |V ρj x ′ ( 2 ) i | ) + m1 β21 ( ( EWρ∥x′ ( 2 ) i ∥ ∣∣∣〈W̃ , W ρ〉∣∣∣ ) ( κ2√m3 + C2 ) + EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣EV ρ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) , where the last line follows because ∥V ′∥ ≲ C2 . But notice that because ∥Ṽ ∥F ≤ 1 , ∥W̃∥F ≤ 1 , then 〈 Ṽ , V ρ 〉 and 〈 W̃ , W ρ 〉 are Gaussian variables with variances at most β22/m2 and β 2 1/m1 . Hence EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣ ≲ β2/√m2 , ( 290 ) EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣ ≲ β1/√m1 . ( 291 ) Similarly , using the same derivation as in ( 33 ) , one can also get the following a.s. bound ( over the randomness of W ρ ) : ∥x′ ( 2 ) i ∥ ≲ √ m3 ( κ1 + C1 + 1√ m1 ∑ j |W ρj xi| ) , ( 292 ) therefore EWρ∥x′ ( 2 ) i ∥ ≲ √ m3 ( κ1 + C1 + EWρ 1√ m1 ∑ j |W ρj xi| ) ( 293 ) ≲ √ m3 ( κ1 + C1 + β1 ) . ( 294 ) Moreover , for every j ∈ [ m2 ] : EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣|V ρj x′ ( 2 ) i | ≤ √ EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣2√EV ρ |V ρj x′ ( 2 ) i |2 = β2√ m2 β2√ m2 ∥x′ ( 2 ) i ∥ = β22 m2 ∥x′ ( 2 ) i ∥ , EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣|W ρj xi| ≤ β21m1 . ( 295 ) Similarly , using Equation ( 292 ) we bound EWρ∥x′ ( 2 ) i ∥ ∣∣∣〈W̃ , W ρ〉∣∣∣ ≲ √m3 ( EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣ ( κ1 + C1 ) + EWρ 1√m1 ∑ j ∣∣∣〈W̃ , W ρ〉∣∣∣|W ρj xi| ) ≤ √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) . ( 296 ) Now applying these bounds ( 291 ) , ( 294 ) , ( 295 ) , and ( 296 ) to ( 290 ) and using the fact that EV ρ 1√ m2 ∑ j |V ρj x ′ ( 2 ) i | ≤ β2∥x ′ ( 2 ) i ∥ , we get LHS ≲m2 β22 ( β2√ m2 √ m3 ( κ1 + C1 + β1 ) ( κ2 √ m3 + C2 ) + EWρ β22√ m2 ∥x′ ( 2 ) i ∥ ) + m1 β21 ( √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ( κ2 √ m3 + C2 ) + EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣β2∥x′ ( 2 ) i ∥ ) ≲m2 β22 ( β2√ m2 √ m3 ( κ1 + C1 + β1 ) ( κ2 √ m3 + C2 ) + β22√ m2 √ m3 ( κ1 + C1 + β1 ) ) + m1 β21 ( √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ( κ2 √ m3 + C2 ) + β2 √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ) . To make it easier for handling the second and third derivatives , we first bound the expectations of f2W ′+Wρ , V ′+V ρ ( xi ) ( xi ) which enables us to use Cauchy-schwartz . Again using similar derivation as in ( 284 ) and Equation ( 292 ) : EWρ , V ρf2W ′+Wρ , V ′+V ρ ( xi ) ( xi ) ≤ EWρ , V ρ ( κ2 √ m3∥x ′ ( 2 ) i ∥+ C2∥x ′ ( 2 ) i ∥+ 1√ m2 ∑ j |V ρj x ′ ( 2 ) i | ) 2 ≲ ( κ2 √ m3 + C2 ) 2EWρ∥x′ ( 2 ) i ∥ 2 + EWρEV ρ 1 m2 ( ∑ j |V ρj x ′ ( 2 ) i | ) 2 ≲ ( κ2 √ m3 + C2 ) 2EWρ∥x′ ( 2 ) i ∥ 2 + EWρ 1 m2 ∑ j EV ρj |V ρ j x ′ ( 2 ) i | 2 + EWρ 1 m2 ∑ j1 ̸=j2 EV ρj1 |V ρj1x ′ ( 2 ) i |EV ρj2 |V ρ j2 x ′ ( 2 ) i | ≲ ( κ2 √ m3 + C2 ) 2EWρ∥x′ ( 2 ) i ∥ 2 + EWρ β22 m2 ∥x′ ( 2 ) i ∥ 2 + EWρβ 2 2∥x ′ ( 2 ) i ∥ 2m ( m− 1 ) m2 ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) EWρ∥x′ ( 2 ) i ∥ 2 ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) EWρm3 ( κ1 + C1 + 1√ m1 ∑ j |W ρj xi| ) 2 ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) m3 [ ( κ1 + C1 ) 2 + 1 m1 ∑ j EWρ |W ρj xi| 2 + 1 m1 ∑ j1 ̸=j2 EWρj1 |W ρ j1 xi|EWρj1 |W ρ j2 xi| ] ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) m3 [ ( κ1 + C1 ) 2 + β21 m1 + β21 m ( m− 1 ) m2 ] = m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] . ( 297 ) Now for the second derivative , we can proceed by applying Cauchy-Swartz : d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ≤ EWρ , V ρ ∣∣∣ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 − ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣m21 β41 ⟨W̃ , W ρ⟩2 − m1 β21 ∥W̃∥2 + m 2 2 β42 ⟨Ṽ , V ρ⟩2 − m2 β22 ∥Ṽ ∥2 + 2m1m2 β21β 2 2 ⟨W̃ , W ρ⟩⟨Ṽ , V ρ⟩ ∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ √ EWρ , V ρ ∣∣∣m21 β41 ⟨W̃ , W ρ⟩2 − m1 β21 ∥W̃∥2 + m 2 2 β42 ⟨Ṽ , V ρ⟩2 − m2 β22 ∥Ṽ ∥2 + 2m1m2 β21β 2 2 ⟨W̃ , W ρ⟩⟨Ṽ , V ρ⟩ ∣∣∣2√ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 . Now note that the cross terms have expectation zero , so we get = √ EWρ , V ρ ( m21 β41 ⟨W̃ , W ρ⟩2 − m1 β21 ∥W̃∥2 ) 2 + ( m22 β42 ⟨Ṽ , V ρ⟩2 − m2 β22 ∥Ṽ ∥2 ) 2 + 4m21m 2 2 β41β 4 2 EWρ , V ρ⟨W̃ , W ρ⟩2⟨Ṽ , V ρ⟩2√ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 ≲ √ m21 β41 ∥W̃∥4 + m 2 2 β42 ∥Ṽ ∥4 + 4m1m2 β21β 2 2 ∥W̃∥2∥Ṽ ∥2 √ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) √ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 . Now applying Cauchy-shwartz and Equation ( 297 ) to above and combining it with Equations ( 289 ) : d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) √ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] . Similarly for the third derivative : ∣∣∣ d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ = EWρ , V ρ ∣∣∣ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣ ≤ √ EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] 2 √ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣2 . ( 298 ) But note that EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] 2 ≤ 2EWρ , V ρ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 6 + 18 ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 2 EWρ , V ρ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 Now note that m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ is a normal variable with variance m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 . Therefore , by the bound on the moments of normal random variables : LHS ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 3 . ( 299 ) Plugging this into Equation ( 298 ) and also using Equation ( 297 ) : ∣∣∣ d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 3/2√ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] . B.2 REPRESENTATION LEMMAS In this section , we prove lemmas mostly related to the representation power of the network , which we mainly use in Appendix A.12 . B.2.1 REPRESENTATION TOOLBOX Lemma 37 Recall the definitions of W+k , and x̄i , k from Equations ( 66 ) , and ( 62 ) . For all k , i ∈ [ n ] : |x̄i , k − trace ( W+k Z i k ) | ≤ √ n/ ( m1λ0 ) ∥Vk∥H∞ . Proof of Lemma 37 trace ( W+k Z i k ) = trace ( Z i k n∑ j=1 Vk , jZjk ) = n∑ i′=1 Vk , i′⟨Zik , Zi ′ k ⟩ . ( 300 ) But note ⟨Zik , Zi ′ k ⟩ = 1/m1 m1∑ j=1 W ′k , j 2 1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } ⟨xi′ , xi⟩ = ⟨xi′ , xi⟩ m1 m1∑ j=1 1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } . Now note that ⟨xi′ , xi⟩ ≤ 1 , and 1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } is a Bernoulli with E1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } = 1/4 + arcsin ( ⟨x , y⟩ ) /2π . Therefore , by Hoeffding inequality we get |⟨Zik , Zi ′ k ⟩ −H∞i , i′ | = O ( 1/ √ m1 ) . Hence , because obviously ∥H∞∥2 ≤ 1 , we get trace ( W+k Z i k ) = n∑ i′=1 Vk , i′H∞i , i′ +O ( 1/ √ m1 ) n∑ i′=1 Vk , i′ = x̄i , k +O ( 1/ √ m1 ) n∑ i′=1 Vk , i′ , which implies |trace ( W+k Z i k ) − x̄i , k| ≤ O ( 1/ √ m1 ) √ n∥Vk∥2 ≲ O ( √ n/ ( √ m1 √ λ0 ) ) ∥Vk∥H∞ ≤ √ n/ ( m1λ0 ) ∥Vk∥H∞ . Lemma 38 ( Bounding the rows norm ) For every 1 ≤ j ≤ m1 , we have ∥W+j ∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ . Furthermore , for every k ∈ [ m3 ] , we have ∥W k+j ∥ ≤ √ n/ √ λ0m1∥Vk∥H∞ . ( 301 ) For the ease of notation , because here we want to work with row sub indices of the matrix W+k , we refer to it by W k+ . Proof of Lemma 38 For a fixed 1 ≤ j ≤ m1 we have with high probability over the randomness of the sign matrix W s : ∥W+j ∥ = ∥ m3∑ k=1 W k+j ∥ = ∥ m3∑ k=1 W sk , j1/ √ m1 n∑ i=1 Vk , ixi1 { W ( 0 ) j xi ≥ 0 } ∥ ≤ √ m3/ √ m1 √√√√m3∑ k=1 ∥ n∑ i=1 Vk , ixi1 { W ( 0 ) j xi ≥ 0 } ∥2 ≤ √ m3/ √ m1 √∑ k ( ∑ i |Vk , i| ) 2 ≤ √ m3/ √ m1 √ n ∑ k ∥Vk∥2 ≤ √ m3n/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ . Furthermore , for every k ∈ [ m3 ] , we have ∥W k+j ∥ ≤ 1/ √ m1 ∑ i |Vk , i| ≤ ( √ n/ √ m1 ) ∥Vk∥2 ≤ ( √ n/ √ λ0m1 ) ∥Vk∥H∞ . Lemma 39 With high probability over the initialization , we have ∥W k+∥2F ≤ ( 1±O ( n/ ( λ0 √ m1 ) ) ) ∥Vk∥2H∞ . Proof of Lemma 39 Recall from the definition of W k+ in Equation ( 66 ) : ∥W k+∥2F = ∥ n∑ i=1 Vk , iZik∥2F = n∑ i=1 n∑ i′=1 Vk , iVk , i′⟨Zik , Zi ′ k ⟩ = n∑ i=1 n∑ i′=1 Vk , iVk , i′ ( H∞i , i′ ±O ( 1/ √ m1 ) ) ≲ VTk H∞Vk ±O ( ( ∥Vk∥1 ) 2/ √ m1 ) = ∥Vk∥2H∞ ± ∥Vk∥2H∞O ( n/ ( λ0 √ m1 ) ) = ( 1±O ( n/ ( λ0 √ m1 ) ) ) ∥Vk∥2H∞ B.2.2 SOME LINEAR ALGEBRA Lemma 40 For n ≤ s , let r1 , . . . , rn be s-dimensional vectors that are approximately normalized and orthogonal to one another , i.e . given some δ > 0 , for every 1 ≤ i ̸= j ≤ n : −δ ≤ ⟨ri , rj⟩ ≤ δ , ∥ri∥2 ≤ 1 + δ . Then , for any vector v we have n∑ i=1 ⟨v , ri⟩2 ≤ ( 1 + δ + n ( n− 1 ) δ ( 1 + δ ) 2 ) ∥v∥2 . Proof of Lemma 40 Define v1 = n∑ i=1 ⟨v , ri⟩ri , v2 = v − v1 . First , note that n∑ i=1 ⟨v , ri⟩2 ≤ ( 1 + δ ) n∑ i=1 ⟨v , ri⟩2∥ri∥2 = ( 1 + δ ) ∥ ∑ i ⟨v , ri⟩ri∥2 − 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ = ( 1 + δ ) ∥v1∥2 − 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ . Next , we write ⟨v1 , v − v1⟩ = ⟨v , n∑ i=1 ⟨v , ri⟩ri⟩ − ⟨ n∑ i=1 ⟨v , ri⟩ri , n∑ i=1 ⟨v , ri⟩ri⟩ = ∑ i ⟨v , ri⟩2 − ∑ i ⟨v , ri⟩2 − 2 ∑ i ̸=j ⟨v , ri⟩⟨v , rj⟩⟨ri , rj⟩ = −2 ∑ i ̸=j ⟨v , ri⟩⟨v , rj⟩⟨ri , rj⟩ . Therefore n∑ i=1 ⟨v , ri⟩2 ≤ ( 1 + δ ) ( ∥v1∥2 + ∥v − v1∥2 ) − 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ . ≤ ( 1 + δ ) ( ∥v1∥2 + ∥v − v1∥2 + 2⟨v1 , v − v1⟩ ) + 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ ≤ ( 1 + δ ) ∥v∥2 + 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ∥v∥2∥ri∥∥rj∥δ ≤ ( 1 + δ ) ∥v∥2 + 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ∥v∥2 ( 1 + δ ) δ = ( 1 + δ ) ∥v∥2 + n ( n− 1 ) δ ( 1 + δ ) 2∥v∥2 = ( 1 + δ + n ( n− 1 ) δ ( 1 + δ ) 2 ) ∥v∥2 , which completes the proof . In the following lemma , we state a trivial bound on the norm of x̄i based on ζ1 . B.2.3 BOUND ON THE NORM OF x̄i ’ S Lemma 41 For every i ∈ [ n ] , we have ∥x̄i∥ ≤ √∑ k ∥Vk∥2H∞ = √ ζ1 . Proof of Lemma 41 By definition : x̄Ti = ( H∞i , Vk ) m3 k=1 . Now consider the Cholskey factorization H∞ = KKT . Because of the assumption ∥xi∥ = 1 , we know that the diagonal of H∞ is all 1/2 . Hence , for the ith row of K we have ∥Ki∥ = 1/2 . Now by Cauchy-Swartz , we have x2i1 = ⟨ ∑ i Vk , iKi , Ki1⟩2 ≤ ∥ ∑ i Vk , iKi∥2∥Ki1∥2 = 1/2∥Vk∥2H∞ . Summing over i and noting Equation ( 61 ) completes the proof . Lemma 42 In the context of Lemma 10 , for ζ2 ≤ 2nB2 , one can substitute f∗ by f∗1 such that Rn ( f̄ ∗ ) ≤ 2Rn ( f∗ ) + B2 n , f̄∗TA−1f̄∗ ≤ f∗TA−1f̄∗ , and furthermore , f̄∗ is in the subspace of eigenvectors of A with eigenvaue larger than Ω ( 1n2 ) . Proof of Lemma 42 For an arbitrary i ∈ [ n ] , define δ = |f∗i − f̄∗i | , and suppose the slope of ℓ ( . , yi ) at point f∗i is equal to c. Then , using the convexity , the fact that ℓ ( yi , yi ) = 0 , and the 1-smoothness of ℓ ( . , yi ) , it is not hard to see the following poincare inequality between the value and derivative of ℓ ( . , yi ) at point f∗i : c ≤ √ 2ℓ ( f∗i , yi ) : = 2ℓ . ( 302 ) where from now on , for brevity , we refer to ℓ ( f∗i , yi ) by ℓ . Also , from the definition of δ and again using 1 smoothness property , it is easy to see that ℓ ( f̄∗i , yi ) ≤ ( c+ δ ) δ + ℓ ( f∗i , yi ) = ( c+ δ ) δ + ℓ , ( 303 ) Plugging Equation ( 302 ) into ( 303 ) and using AM-GM inequality : ℓ ( f̄∗i , yi ) ≤ δ2 + cδ + ℓ ≤ δ2 + √ 2ℓδ + ℓ ≤ δ2 + ℓ+ δ2/2 + ℓ ≤ 2ℓ+ 3δ2/2 . Summing above for i ∈ [ n ] , we obtain Rn ( f̄ ∗ ) ≤ 2Rn ( f∗ ) + 3∥f∗ − f̄∗∥22/2 . ( 304 ) Now we write an eigendecomposition for A as A = ∑n i=1 λiuiu T i for orthonormal basis { ui } , and let f∗ = ∑ i γiui be the representation of f ∗ in this basis . Then , from our assumption , for arbitrary ω > 0 ∑ i γ2i λ −1 i = f ∗TA−1f∗ ≤ 4nB2 , which implies ω−1 ∑ i : λi≤ω γ2i ≤ 4nB2 , or equivalently ∑ i : λi≤ω γ2i ≤ 4nB2ω , ( 305 ) where notice that ∑ i : λi≤ω γ 2 i is the squared norm of the projection of f ∗ onto the directions whose eigenvalue is at most ω . Now taking ω = 112n2 and defining f̄ ∗ by keeping only the directions for which λi > ω completes the proof . B.3 COUPLING FOR ∇̂W , ∇̂V In general , because the gaussian smoothing matrices ( W ρ , V ρ ) can become unbounded , the gradient estimates ( ∇̂W , ∇̂V ) = ∇W , V ℓ ( fW ′+Wρ , V ′+V ρ ( xi ) , yi ) also become unbounded . However , in analyzing the stochastic behavior of SGD and showing that it can escape saddle points , it is convenient to assume the gradient ’ s noise vector is almost surely bounded . The goal of this section is to introduce a coupling between ( W ρ , V ρ ) and another random variable that is a.s. bounded polynomially in other parameters . As that the coupled random variables take different values is exponentially small while the number of iterations in our algorithm is only polynomially large , without any concern we instead work with this new random varaible , and with an overload of notation we also denote it by ( W ρ , V ρ ) . Lemma 43 For an arbitrary parameter χ > > 1 , On any pair for ( W ′ , V ′ ) with ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , there exist a mean zero random vector Λ̄ with respect to the randomness of the uniformly picked data point ( xi , yi ) and the smoothing matrices W ρ,1 , W ρ,2 , V ρ,1 , and V ρ,2 which define ∇̂W ′ , V ′ ( meaning it is a function of those variables ) , such that with probability at least 1− 2 exp { − ( χ2 − 1 ) dm1/4 } − 2 exp { − ( χ2 − 1 ) m3m2/4 } : = 1− δ1 , we have ∇̂W ′ , V ′ = ∇W ′ , V ′L ( W ′ , V ′ ) + Λ̄ , ( 306 ) and finally Λ̄ is a.s. polynomially bounded , i.e . almost surely we have ∥Λ̄∥ ≤ poly ( m1 , m2 , m3 , C1 , C2 , B , χ ) . Proof of Lemma 43 Remember that x′i was the output of the first layer ( by considering the smoothing matrix W ρ ) . Now with high probability over the initialization , ∥∇W ′fW ′+Wρ , V ′+V ρ ( xi ) ∥F = ∥∇x′ifW ′+Wρ , V ′+V ρ ( xi ) T D ( x ′ i ) dW ′ ∥ ∥∇x′ifW ′+Wρ , V ′+V ρ ( xi ) ∥∥ D ( x′i ) dW ′ ∥ ∥ 1√ m2 aTDV ′+V ρ ( V ( 0 ) + V ′ + V ρ ) ∥∥ 1√ m1 m3∑ k=1 diag ( W sk ) DW ′+Wρ , xix T i ∥F ≤ ( ∥V ( 0 ) ∥F + ∥V ′∥F + ∥V ρ∥F ) ( 1√ m1 ∑ k ∥diag ( W sk ) DW ′+Wρ , xixTi ∥F ) ≤ ( κ2 √ m2m3 + C2 + ∥V ρ∥F ) ( 1√ m1 ∑ k ∥diag ( W sk ) DW ′+Wρ , xixTi ∥F ) ≤ ( κ2 √ m2m3 + C2 + ∥V ρ∥F ) . ( 307 ) On the other hand , using the final bound in Lemma 33 : ∥∇V ′fW ′+Wρ , V ′+V ρ ( xi ) ∥F = ∥ 1√ m2 diag ( a ) DV ′+V ρx′i T ∥F ≤ ∥x′i∥ ≤ κ1 √ m3 + C1 + √ m3 ∑ j 1√ m1 |W ρj xi| ≤ κ1 √ m3 + C1 + √ m3∥W ρ∥F . ( 308 ) Denoting ℓ̇ ( fW ′+Wρ,1 , V ′+V ρ,1 , yi ) ∇W ′ , V ′ℓ ( fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) , yi ) by ∇̃W ′ , V ′ , then combining Equations ( 307 ) and ( 308 ) and using the 1 gradient lipschitzness property of the square loss , : ∥∇̃W ′ , V ′∥F = |d ( ℓ ( f , yi ) ) df | √ ∥∇W ′fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) ∥2F + ∥∇V ′fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) ∥2F ≤ ( |fW ′+Wρ1 , V ′+V ρ1 ( xi ) |+ |B| ) ( κ1 √ m3 + C1 + √ m3∥W ρ,2∥F + κ2 √ m2m3 + C2 + ∥V ρ,2∥F ) . ( 309 ) Finally , applying Cauchy-Swartz to the second a.s. bound in Lemma 34 we have : ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣ ≤ ( κ2 √ m3 + ∥V ρ∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ∥ ) + C2 ( C1 + √ m3∥W ρ∥ ) . Combining this with ( 309 ) : ∥∇̃W ′ , V ′∥F ≤ [ B + ( κ2 √ m3 + ∥V ρ,1∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ,1∥ ) + C2 ( C1 + √ m3∥W ρ,1∥ ) ] × ( κ1 √ m3 + C1 + √ m3∥W ρ,2∥F + κ2 √ m2m3 + C2 + ∥V ρ,2∥F ) . Therefore , using the lipschitz bound in Theorem 9 : ∥∇̃W ′ , V ′ −∇W ′ , V ′E ( xi , yi ) ∼Zℓ ( f ′ W ′ , V ′ ( xi ) , yi ) ∥F ≤ ∥∇̂W ′ , V ′∥F + ∥E ( xi , yi ) ∼Z∇W ′ , V ′ℓ ( fW ′+Wρ , V ′+V ρ ( xi ) , yi ) ∥F ≤ [ B + ( κ2 √ m3 + ∥V ρ,1∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ,1∥ ) + C2 ( C1 + √ m3∥W ρ,1∥ ) ] × ( κ1 √ m3 + C1 + √ m3∥W ρ,2∥F + κ2 √ m2m3 + C2 + ∥V ρ,2∥F ) + ( O ( C1C2 ) +B ) Ψ1 . ( 310 ) Now we define the following events Ξ1 : = { ∥W ρ1∥F ≥ χ √ dβ1 ∨ ∥W ρ2∥F ≥ χ √ dβ1 } , Ξ2 : = { ∥V ρ1∥F ≥ χ √ m3β2 ∨ ∥V ρ2∥F ≥ χ √ m3β2 } , where recall we assume χ > > 1 . Then , as we know the variable ∥W ρ∥2F has mean dβ21 and is subexponential with parameters ( dβ41/m1 , β 2 1/m1 ) . Hence , by a union bound and Bernstein ( Note that W ρ,1 , W ρ,2 are independent ) : P ( Ξ1 ) ≤ 2P ( ∥W ρ∥F ≥ χβ1 √ d ) = 2P ( ∥W ρ∥2F ≥ χ2β21d ) ≤ 4max ( exp { − ( χ2 − 1 ) 2β41d2/ ( 4dβ41/m1 ) } , exp { − ( χ2 − 1 ) β21d/ ( 4β21/m1 ) } ) = 4max ( exp { − ( χ2 − 1 ) 2dm1/4 } , 2 exp { − ( χ2 − 1 ) dm1/4 } ) ≤ 2 exp { − ( χ2 − 1 ) dm1/4 } . Similarly for ∥V ρ∥F : P ( Ξ2 ) = P ( ∥V ρ∥F ≥ χβ2 √ m3 ) ≤ 4 exp { − ( χ2 − 1 ) m3m2/4 } . Moreover , because of the subexponential tails of ∥W ρ∥2F and ∥V ρ∥2F , for each of W ρ,1 or W ρ2 , V ρ1 , or V ρ2 : E ( ∥W ρ∥F | Ξ1 ) ≲ χ √ dβ1 , E ( ∥W ρ∥2F | Ξ1 ) ≲ χ2dβ21 . E ( ∥V ρ∥F | Ξ2 ) ≲ χ √ m3β2 , E ( ∥V ρ∥2F | Ξ2 ) ≲ χ2m3β22 . Now Defining Ξ = Ξ1 ∪ Ξ2 and combining the above equations : E ( ∥W ρ∥1 { Ξ } ) ≤ E∥W ρ∥ ( 1 { Ξ1 } + 1 { Ξ2 } ) = E ( ∥W ρ∥| Ξ1 ) P ( Ξ1 ) + E ( ∥W ρ∥ ) P ( Ξ2 ) ≲ χ √ dβ12 exp { − ( χ2 − 1 ) dm1/4 } + √ dβ12 exp { − ( χ2 − 1 ) m3m2/4 } = 2 √ dβ1 ( χ exp { − ( χ2 − 1 ) dm1/4 } + exp { − ( χ2 − 1 ) m3m2/4 } ) , and E∥W ρ∥21 { Ξ1 } = E ( ∥W ρ∥2| Ξ1 ) P ( Ξ1 ) ≤ 2dβ21χ2 exp { − ( χ2 − 1 ) dm1/4 } . Similarly E ( ∥V ρ∥1 { Ξ } ) ≤ 2 √ m3β2 ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) , and E ( ∥V ρ∥21 { Ξ2 } ) ≤ 2m3β22χ2 exp { − ( χ2 − 1 ) m3m2/4 } . Applying these equations to ( 310 ) with Cauchy-Schwartz to get the upper bounds E1 { Ξ1 } ∥W ρ,1∥∥W ρ,2∥ ≤ E1 { Ξ1 } ∥W ρ∥2 and ( EWρ∥W ρ∥ ) 2 ≤ EWρ∥W ρ∥2 ( for terms with only one W ρ , i or V ρ , i , we simply write them as W ρ and V ρ ) : EWρ , V ρ∥∇̃W ′ , V ′ −∇W ′ , V ′E ( xi , yi ) ∼Zℓ ( f ′ W ′ , V ′ ( xi ) , yi ) ∥F1 { Ξ } ≤ EWρ , V ρ , ( xi , yi ) [ B + ( κ2 √ m3 + ∥V ρ,1∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ∥ ) + C2 ( C1 + √ m3∥W ρ∥ ) ] × ( κ1 √ m3 + C1 + √ m3∥W ρ∥F + κ2 √ m2m3 + C2 + ∥V ρ∥F ) + α ( O ( C1C2 ) +B ) Ψ1 = [ B + ( κ2 √ m3 ) ( √ m3κ1 + C1 ) + C1C2 ] × ( κ1 √ m3 + C1 + κ2 √ m2m3 + C2 + √ m3EWρ1 { Ξ } ∥W ρ∥F + EV ρ1 { Ξ } ∥V ρ∥F ) + ( B + ( √ m3κ1 + C1 ) √ m3 + ( κ2m3 + C2 √ m3 ) + √ m3 ( κ1 √ m3 + C1 + κ2 √ m2m3 + C2 ) ) × ( EWρ1 { Ξ1 } ∥W ρ∥FEV ρ∥V ρ∥F + EWρ∥W ρ∥FEV ρ1 { Ξ2 } ∥V ρ∥F ) + ( B + √ m3κ1 + C1 ) ( EV ρ1 { Ξ2 } ∥V ρ∥2 + P ( Ξ1 ) E∥V ρ∥2 ) + C2m3 ( EWρ1 { Ξ1 } ∥W ρ∥2 + P ( Ξ2 ) E∥W ρ∥2 ) +m3 ( EWρ1 { Ξ1 } ∥W ρ∥2EV ρ∥V ρ∥+ EWρ∥W ρ∥2EV ρ1 { Ξ2 } ∥V ρ∥ ) + √ m3 ( EV ρ1 { Ξ2 } ∥V ρ∥2EWρ∥W ρ∥+ EV ρ∥V ρ∥2EWρ1 { Ξ1 } ∥W ρ∥ ) + ( O ( C1C2 ) +B ) Ψ1P ( Ξ ) ≤ ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) poly ( m1 , m2 , m3 ) = negligible . ( 311 ) But note that ∇̂W ′ , V ′ = ∇̃W ′ , V ′ +∇W ′ , V ′ ( ψ1∥W ′∥2 + ψ2∥V ′∥2 ) , ∇W ′ , V ′L ( W ′ , V ′ ) = ∇W ′ , V ′E ( xi , yi ) ∼Zℓ ( f ′ W ′ , V ′ ( xi ) , yi ) +∇W ′ , V ′ ( ψ1∥W ′∥2 + ψ2∥V ′∥2 ) . Applying this to Equation ( 311 ) , we get that if we define Λ : = ∇̂W ′ , V ′ −∇W ′ , V ′L ( W ′ , V ′ ) , then E∥Λ1 { Ξ } ∥ ≤ ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) poly ( m1 , m2 , m3 ) . On the other hand , note that using again Equation ( 310 ) , we have the following a.s. bound : ∥Λ1 { Ξc } ∥ = poly ( m1 , m2 , m3 , C1 , C2 , χ ) . Defining Λ1 = Λ1 { Ξc } , Λ2 = 1 { Ξ } E ( Λ|Ξ ) , Λ̄ = Λ1 + Λ2 , we get that with probability at least 1− P ( Ξ ) : ∇̂W ′ , V ′ = ∇W ′ , V ′L ( W ′ , V ′ ) + Λ̄ and also note that EΛ̄ = EΛ = 0 . Finally by the a.s. bound for Λ1 , we have a.s. : ∥Λ̄∥ ≤ EWρ , V ρ , ( xi , yi ) ∥Λ1 { Ξ } ∥+ ∥Λ1 { Ξ c } ∥ ≤ ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) poly ( m1 , m2 , m3 ) + poly ( m1 , m2 , m3 ) = poly ( m1 , m2 , m3 , C1 , C2 , B , χ ) , which completes the proof . Corollary 9.1 It is easy to check that running PSGD with unbiased gradient estimate ∇̂W ′ , V ′ is equivalent to running SGD after our change of coordinates , with unbiased gradient estimate ∇̂w′ , v′ : = Ῡ∇W ′ , V ′ , where Ῡ is the matrix for our change of coordinate , which is equal to Υ defined in Appendix A.10 for the coordinates in V ′ and simply identity for the coordinates in W ′ . Therefore , projecting both sides in Equation ( 312 ) of Lemma 43 onto Φ⊥ by multiplying Υ implies that with high probability for all iterations of the algorithm ∇̂w′ , v′ = Ῡ∇W ′ , V ′LΠ ( W ′ , V ′ ) + ῩΛ̄ = ∇w′ , v′LΠ ( w′ , v′ ) + ῩΛ̄ , ( 312 ) where £̄ : = ῩΛ̄ ( using the properties of Λ̄ in Lemma 43 ) is a mean zero noise vector with almost surely bounded norm , i.e . ∥£̄∥ ≤ Q′ for some Q′ = poly ( m1 , m2 , m3 , C1 , C2 ) . ( we dropped the χ parameters by considering constant high probability argument ) . Finally , note that injecting noise ( Ξ1/∥Ξ1∥ , Ξ2/ ( √ m1∥Ξ2∥ ) ) by PSGD results in adding an extra zero mean noise ( Ξ̃1 , Ξ̃2 ) : = ( ῩΞ1/∥Ξ1∥ , ῩΞ2/ ( √ m1∥Ξ2∥ ) ) to the gradient ∇w′ , v′LΠ ( w′ , v′ ) . Therefore , overall running SGD on LΠ ( which is equivalent to PSGD on L ) observe an unbiased noise vector defined as £ : = £̄ + ( Ξ̃1 , Ξ̃2 ) . Now it is easy to check that the moment matrix of Ξ̃1 and Ξ̃2 are σ′1 2I and σ′2 2I for σ′1 2 : = 1 m21d , ( 313 ) σ′2 2 : = m2 ( m3 − n ) mm2 3 , ( 314 ) which implies the moment matrix of £ is upper bounded by σ22I : = ( Q ′/ ( m2m3 +m1d ) + max { σ′12 , σ′22 } ) I , and lower bounded by σ22I : = min { σ′12 , σ′22 } I , i.e . σ21IE££T ≤ σ22I . ( Note that we look at the new coordinates ( w′ , v′ ) as a vectors , so the term E££T makes sense . ) Moreover , ∥Ξ̃1∥ = 1/ √ m1 , ∥Ξ̃2∥ ≤ 1 almost surely , which implies the following almost surely bound for £ : ∥£∥ ≤ Q : = Q′ + 1 + 1/ √ m1 . Lemma 44 Let g ( x ) be a second order differentiable function over RN such that at point x , there exist a random direction y and deterministic direction z and fixed positive real r with : Ey = 0 , Eyg ( x+ ηz + √ ηy ) ≤ g ( x ) − ηr . Then , for the gradient and Hessian at point x , we have either ∥∇g ( x ) ∥ ≥ r 4∥z∥ , or λmin ( ∇2g ( x ) ) ≤ − r 2∥y∥2 . Proof of Lemma 44 We write the second order tailor approximation of g around x : g ( x+ w ) = g ( x ) +∇g ( x ) Tw + 1 2 wT∇2g ( x ) w + o ( ∥w∥2 ) . Now substituting w with ηz + √ ηy and taking expectation with respect to y , as we send η → 0 and using the fact that Ey = 0 : Eyg ( x+ ηz + √ ηy ) = g ( x ) + Ey∇g ( x ) T ( ηz + √ ηy ) + 1 2 ( ηz + √ ηy ) T∇2g ( x ) ( ηz +√ηy ) + o ( ∥ηz +√ηy∥2 ) = Eyg ( x ) + η∇g ( x ) T z + 1 2 η2zT∇2g ( x ) z + η 1 2 yT∇2g ( x ) y + o ( η∥y∥2 ) = Eyg ( x ) + η∇g ( x ) T z + η 1 2 yT∇2g ( x ) y + o ( η ) . Combining the assumption with the above Equation , we get that for small enough η , we have η∇g ( x ) T z + η 1 2 EyyT∇2g ( x ) y ≤ −ηr/2 , i.e . ∇g ( x ) T z + 1 2 EyyT∇2g ( x ) y ≤ −r/2 , which means we should either have ∇g ( x ) T z ≤ −r/4 , which implies ∥∇g ( x ) ∥ ≥ r 4∥z∥ , or EyyT∇2g ( x ) y ≤ −r/2 , which implies λmin ( ∇2g ( x ) ) ≤ − r 2maxỹ∈support ( y ) ∥ỹ∥2 . B.4 HANDLING THE INJECTED NOISE BY PSGD In this section , we prove that having SGD injecting noise into our gradient estimates mostly does not change the sign pattern of the first layer , namely among the set of rows in P defined in Lemma 1 . Lemma 45 Having enough overparameterization , with high probability , at every iteration of the PSGD for W ′ ( 2 ) defined in Lemma 1 , we have for every j ∈ [ m1 ] : ∥W ′ ( 2 ) j ∥ ≤ c2/ ( 4 √ m1 ) . Proof of Lemma 45 Let Φ′ be the subspace of the first layer weight matrices which is zero in rows j ∈ P ( P is defined in Lemma 1 ) , while in other rows it is the span of Zki ’ s , i.e . using our notation Z̃ i k introduced in the proof of Lemma 1 , we can write Φ′ is span ( Zki ) i , k . Recall from Lemma 1 that we decompose the first layer weight W ′ as W ′ ( 1 ) + W ′ ( 2 ) , namely the parts in the subspace Φ′ and subspace Φ′⊥ respectively . Moreover , let Ξ1/ ( √ m1∥Ξ1∥ ) = Ξ ( 1 ) +Ξ ( 2 ) be the decomposition of the injected noise at some iteration of PSGD into subspaces Φ′ and Φ′⊥ respectively . Now recall that the current W ′ is the value of the previous iteration moved by the gradient plus the injected noise : W ′ =W ′ − η ( ∇̂W ′ + Ξ ( 1 ) + Ξ ( 2 ) ) =W ′ − η ( ∇̃W ′ + 2ψ1W ′ − , ( 1 ) + 2ψ1W ′− , ( 2 ) + Ξ ( 1 ) + Ξ ( 2 ) ) , whereW ′ is the weight of the previous iteration andW ′− , ( 1 ) , W ′− , ( 2 ) are again its decomposition to Φ′ and Φ′⊥ , where ∇̃W ′ , V ′ is defined in Lemma 43 . Applying Lemma 24 for the previous iteration of the algorithm , we get ∇̃W ′ ∈ Φ′ since the bad events E defined in Lemma 33 occurs only with probability exponentially small ( hence union bound across all the iterations rules it out ) . Hence , the decomposition for the current iteration becomes W ′ ( 1 ) =W ′ − , ( 1 ) − η ( ∇̂W ′ + 2ψ1W ′ − , ( 1 ) + Ξ ( 1 ) ) , ( 315 ) W ′ ( 2 ) = ( 1− 2ηψ1 ) W ′ − , ( 2 ) + ηΞ ( 2 ) . ( 316 ) We handle the W ′ ( 1 ) part in Lemma 1 and prove that as long as ∥W ′ ( 1 ) ∥2 ≤ ∥W ′∥2 remains bounded by C21 , then the sign pattern of the first layer , when only considering the W ′ ( 1 ) part , is specified by the initialization except within set P ; here we handle the W ′ ( 2 ) part as well . Note that for every row j ∈ [ m1 ] , the variable ∥ ( Ξ1/ ( √ m1∥Ξ1∥ ) ) j ∥2 is ( O ( 1/ ( m41d ) ) , O ( 1/ ( m 2 1d ) ) ) -subexponential with mean 1/m1 . Therefore , with probability that is exponentially small in m1 , ∥ ( Ξ1/ ( √ m1∥Ξ1∥ ) ) j ∥ is bounded by O ( 1/m1 ) . It is not hard to see the same argument holds for the projection of Ξ1/ ( √ m1∥Ξ1∥ ) onto Φ⊥ , i.e . Ξ ( 2 ) . Applying a union bound for all iterations , again using the fact that we run PSGD for poly iterations while the chance of error is exponentially small in m1 , we can then argue that with high probability over the noise of gradients , at every iteration and for every j ∈ [ m1 ] : ∥Ξ ( 2 ) j ∥ = Õ ( 1/m1 ) . ( 317 ) But applying trinagle inequality to Equation ( 316 ) and writing it in a telescope form , particularly for the jth row , and further using the assumption in 317 , we get that ∥W ′ ( 2 ) j ∥ grows at most to O ( 1/ ( m1ψ1 ) ) ; as we set 1/ψ1 = O ( poly ( n ) ) , assuming polynomially large enough m1 concludes the claim . B.4.1 BOUNDING THE NORM OF THE FIRST LAYER ’ S OUTPUT IN THE WORST CASE Lemma 46 Suppose W ′ satisfies the assumption of Lemma 1 , i.e . ∥W ′∥ ≤ C1 , and ∥W ′j∥ ≤ c2/ ( 2 √ m1 ) except possible for indices in P , also defined in 1 . Then , with high probability over initialization sup x , ∥x∥=1 ∥ϕ′ ( x ) ∥ ≲ ( 1 +O ( m23d2 log ( m1 ) 2/m1 ) ) C1 + √ m3 C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 , which is O ( C1 ) for large enough overparameterization . Proof of Lemma 46 Note the because the VC-dimension of the class of binary functions with respect to halfspaces in Rd is d + 1 , the number of different sign patterns DW ( 0 ) , x for different x can be at most md+11 . Now similar to Equation ( 318 ) , for k ∈ [ m3 ] define Zk ( x ) = 1/ √ m1 ( W sk , j1 { W ( 0 ) j Tx } x ) m1 j=1 . ( 318 ) Then , for k1 ̸= k2 , as ∥x∥ = 1 : ⟨Zk1 ( x ) , Zk2 ( x ) ⟩ = 1 m1 m1∑ j=1 W sk1 , jW s k2 , j1 { W ( 0 ) j Tx } ≤ 1 m1 sup x m1∑ j=1 DW ( 0 ) , xj , jW s k1 , jW s k2 , j . But for each fixed DW ( 0 ) , x , using Hoeffding bound , we have with probability 1− δ : 1√ m1 m1∑ j=1 DW ( 0 ) , xj , jW s k1 , jW s k2 , j ≲ √ log ( 1/δ ) m1 . Applying the above for all possible sign patterns with δ < O ( 1/m1d+1 ) and a union bound , we have with high probability sup x , ∥x∥=1 ⟨Zk1 ( x ) , Zk2 ( x ) ⟩ ≤ 1 m1 sup x m1∑ j=1 DW ( 0 ) , xj , jW s k1 , jW s k2 , j ≲ d log ( m1 ) / √ m1 . We can even state the following stronger bound with respect to two adversarially picked vectors x , x′ : sup ∥x∥=1 , ∥x′∥=1 ⟨Zk1 ( x ) , Zk2 ( x′ ) ⟩ ≤ 1 m1 sup x m1∑ j=1 DW ( 0 ) , xj , jDW ( 0 ) , x′ j , jW s k1 , jW s k2 , j ≲ d log ( m1 ) / √ m1 , ( 319 ) because eachDW ( 0 ) , x′ j , j has at mostm d+1 1 cases as we discussed above , thenDW ( 0 ) , xj , jDW ( 0 ) , x′ j , j has at most md+11 possible cases , and applying a similar Hoeffding bound for each of them and a union bound as we did will imply ( 319 ) . We will use this generalized version in another section . Now combining Equation ( 319 ) with the fact that ∥W ′∥ ≤ C1 and applying Lemma 40 : sup x , ∥x∥=1 m3∑ k=1 ⟨W ′ , Zk ( x ) ⟩2 ≤ ( 1 +O ( m23d2 log ( m1 ) 2/m1 ) ) C21 . ( 320 ) On the other hand , setting m2 = m1 , m3 = d , and R = c2/ ( 2 √ m1κ1 ) in Lemma 29 , we get with high probability # ( j ∈ [ m ] : |V ( 0 ) j x| ≤ c2/ ( 2 √ m1 ) ) ≤ m1c2/ ( 2 √ m1 ) = √ m1c2/ ( 2κ1 ) . Noting that ∥W ′j∥ ≤ c2/ ( 2 √ m1 ) for j /∈ P , we conclude that with high probability , for any x , 1 { ( W ( 0 ) +W ′ ) Tj x ≥ 0 } and 1 { W ( 0 ) j Tx ≥ 0 } can be different in at most √m1c2/ ( 2κ1 ) of the j ’ s outside of [ m1 ] \ P . Therefore , as we have also |P | ≲ nc2 √ m1/κ1 from Lemma 1 , we conclude that with high probability , for any x , there are at most O ( nc2 √ m1/κ1 ) sign changes by adding W ′ to W ( 0 ) . This further implies : |ϕ′k ( x ) − ⟨W ′ , Zk ( x ) ⟩| ≤ 2/ √ m1 ∑ j : Sgn ( W ( 0 ) j T x ) ̸=Sgn ( ( W ( 0 ) +W ′ ) Tj x ) |W ′jx| ≤ ∥W ′∥2 √∣∣∣ { j| Sgn ( W ( 0 ) j Tx ) ̸= Sgn ( ( W ( 0 ) +W ′ ) Tj x ) } ∣∣∣/√m1 ≲ C1 √ nc2 √ m1/κ1/ √ m1 = C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 . Combining this with ( 320 ) , we conclude with high probability : sup x , ∥x∥=1 ∥ϕ′ ( x ) ∥ ≲ ( 1 +O ( m23d2 log ( m1 ) 2/m1 ) ) C1 + √ m3 C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 , which completes the proof . You may include other additional sections here .

Summary: This paper studies the behavior of a three-layer neural network trained by a projected SGD algorithm. The authors theoretically show that, in this setting the neural network has more complicated behavior than in the NTK regime. Instead of sticking at one kernel, now the neural network explores different kernels decided by the features learned by the first layer. Therefore, the generalization error is bounded by a minimum of many RKHS norms, and is proven to be smaller than the single NTK norm. The proposed generalization error bound outperforms the NTK bound. For target functions in the NTK, the proposed bound is no bigger than the NTK bound. The model and the algorithm are close to commonly used ones, while there are components specially designed for the theoretical analysis.