Optimization and Adaptive Generalization of Three layer Neural Networks

1 INTRODUCTION The ability of overparameterized neural networks trained by ( stochastic ) gradient descent to generalize well on test data ( Krizhevsky et al. , 2012 ; Silver et al. , 2016 ; Hinton et al. , 2012 ) , even if they perfectly fit the the training data , has intrigued theoretical researchers and led to many approaches for generalization bounds ( Neyshabur et al. , 2015 ; Bartlett et al. , 2017 ; Neyshabur et al. , 2018 ; Dziugaite & Roy , 2017 ; Wei et al. , 2019 ; Golowich et al. , 2018 ; Arora et al. , 2018b ; Zhou et al. , 2018 ; Konstantinos et al. , 2017 ) . This generalization ability is closely tied to the optimization procedure – the trajectory of the training algorithm in a non-convex loss landscape – and the structure of the data . Hence , several recent works study the training of neural networks . For instance , Safran & Shamir ( 2018 ) address the role of overparametrization in avoiding bad local minima , and Zhang et al . ( 2016 ) empirically show that overparametrized networks trained by SGD can even perfectly fit to random labels . Within the popular framework of the Neural Tangent Kernel ( NTK ) ( Jacot et al. , 2018 ) , which uses a linear approximation of the network at initialization , several works analyze the optimization trajectory and show global convergence of ( S ) GD to a global optimum of the empirical loss ( AllenZhu et al. , 2019 ; Li & Liang , 2018 ; Zou et al. , 2018 ; Du et al. , 2018 ) . Extending the viewpoint to generalization , Arora et al . ( 2019a ; b ) exploit the kernel-like behaviour of two-layer networks close to their initialization to prove generalization for the final network , showing that two-layer neural networks generalize as well as Kernel Ridgeless Regression ( KRLR ) with the NTK . Cao & Gu ( 2019 ) show a tighter bound with a Neural Tangent Random Feature Model . The kernel approach , however , has two main limitations : First , while KRLR can generalize well in specific high dimensional regimes ( Liang et al. , 2020 ) , there is theoretical and empirical evidence that it can be inconsistent with noise ( Rakhlin & Zhai , 2019 ) . Is there an approach for analyzing neural networks that shows they perform at least as well as KRLR , but is also robust to noise ? Second , importantly , neural networks are known to outperform traditional statistical methods in many regimes as they are able to automate the process of feature extraction from data , as opposed to kernel methods that work with a fixed feature representation . This poses the question of other , adaptive , regimes beyond the linear network approximation . In this realm , Wu et al . ( 2018 ) show generalization bounds that , instead of the NTK norm , scale with respect to another functional norm . This norm corresponds to the minimum RKHS norm of the function among a family of kernels , i.e. , their method in a sense picks the best kernel in this family . However , this result ignores the computational aspect of the problem . Are there particular nonlinear regimes beyond NTK for which a gradient-type polynomial-time algorithm , in a way , adaptively chooses a suitable kernel ? Going beyond the NTK view , a line of work convexifies the optimization problem via an approximation of SGD dynamics with a continuous time gradient flow in the space of probability measures on the hidden units of the network , equipped with the Wasserstein metric ( Mei et al. , 2018 ; Chizat & Bach , 2018 ; Mei et al. , 2019 ; Wei et al. , 2018 ; Sirignano & Spiliopoulos , 2020 ; Javanmard et al. , 2019 ; Lu et al. , 2020 ) . Taking another perspective , Allen-Zhu et al . ( 2018 ) consider a three-layer network model that is not captured by the NTK approximation , and learn an underlying concept class by exploiting saddle-point escape theory for nonconvex SGD ( Ge et al. , 2015a ) . However , evaluating the complexity measure of Allen-Zhu et al . ( 2018 ) is rather involved , and only aligns well with functions that are described by a particular network . Whether one can recover the NTK bound ( e.g . the NTK norm ) from these results is not clear , while computing these generalization bounds from a given dataset is another question . For the NTK setting , in contrast , Arora et al . ( 2019a ) develop a purely data dependent generalization bound . Going to deeper than two layers , is it possible to prove a data-dependent complexity measure beyond the NTK regime that recovers the NTK result ( Arora et al. , 2019a ) as a special case ? In this work , we address the above questions : • We consider a regime for 3-layer neural networks that is not captured by the NTK approximation and show that , despite nonconvexity , a variant of projected SGD finds a good solution , as measured by the regularized empirical loss , importantly , after polynomially many iterations . • We introduce a new function norm ∥.∥ζ as the minimum RKHS norm with respect to a family of kernels K , which is upper bounded by the NTK norm up to constants . We show that for an arbitrary function f , the generalization gap of the trained network scales by ∥f∥ζ . This makes our generalization bound adaptive , in the sense that it scales with the best kernel inK . As a byproduct , our bounds are comparable with kernel regression bounds simultaneously with all kernels in K. We hope that our techniques motivate researchers to prove such adaptive generalization bounds for deeper networks , which can potentially result in stronger depth separation . • We show generalization bounds with a new data-dependent complexity measure that generalizes the NTK-based complexity measure in ( Arora et al. , 2019a ) . Up to logarithmic factors , our bounds are upper bounded by the NTK-based bounds in ( Arora et al. , 2019a ) and hence strictly improve over them ( if one substitutes their Lipschitz loss with a smooth one ) – see Appendix A.1 for a simple explicit example . Importantly , our bound can also handle noisy distributions as opposed to ( Arora et al. , 2019a ) . Further Related work . While the idea of a learning algorithm that combines multiple kernels has been employed for a while in the community ( Sonnenburg et al. , 2006 ; Rakotomamonjy et al. , 2007 ; Duan et al. , 2012 ) , our understanding of the connections between deep learning and multiple kernel learning is yet in its infancy . Recently , Dou & Liang ( 2020 ) define a time-varying kernel based on the network weights and show that the limit of the gradient flow converge to a suitable dynamic kernel , in the sense that the residual of the link function onto its RKHS could be in a smaller ranked space compared to the orthogonal complement of the RKHS . Ghorbani et al . ( 2019 ) analyze the difference between training a two layer ReLu network compared to its NTK or random feature simplifications , for a mixture of Gaussians input distribution and quadratic target functions . Ignoring the computational hardness imposed by nonconvexity , Bach ( 2017 ) prove a dimension dependent generalization bound beyond NTK . In another line of work , Chizat & Bach ( 2020 ) study gradient flow on losses with exponential tail and its relation to the max margin solution . Wei et al . ( 2019 ) show an interesting separation between the learning power of two layer ReLu nets and their NTK approximation , by showing a sample complexity gap for an artificially constructed distribution . With a different approach , Allen-Zhu & Li ( 2020 ) analyze multi-layer networks with quadratic activations , and prove generalization bounds polynomial in the dimension and precision by assuming an underlying teacher network , which shows a remarkable algorithmic depth separation . The problem of depth separation for neural networks and more generally their expressive power has been investigated by several researchers before ( Raghu et al. , 2017 ; Daniely , 2017 ; Barron , 1994 ; Funahashi , 1989 ; Safran & Shamir , 2016 ; Safran et al. , 2019 ) . The assumption of an underlying teacher network that one seeks to recover is common , too ( Li & Yuan , 2017 ; Zhong et al. , 2017 ; Brutzkus & Globerson , 2017 ) . Other works focus mainly on the algorithm and use other techniques , such as tensor factorization , to find a global optimum ( Tian , 2016 ; Bakshi et al. , 2019 ; Janzamin et al. , 2015 ; Zhong et al. , 2017 ) . Finally , many authors study the loss landscape under various assumptions ( Freeman & Bruna , 2016 ; Nguyen & Hein , 2017 ; Soudry & Carmon , 2016 ; Soltanolkotabi et al. , 2018 ; Ge et al. , 2017 ) , some of them consider the simplified case of deep linear networks ( Arora et al. , 2018a ; Saxe et al. , 2013 ; Bartlett et al. , 2018 ; Kawaguchi , 2016 ) . 2 SETUP AND APPROXIMATION BY KERNELS We analyze a 3-layer ReLu neural network from inputs x ∈ Rd to outputs y ∈ R , of the form fV ′ , W ′ ( x ) = 1√ m2 aTσ ( ( V ( 0 ) + V ′ ) W s 1√ m1 σ ( ( W ( 0 ) +W ′ ) x ) ) , ( 1 ) where a ∈ Rm2 is a vector of random signs , V ( 0 ) ∈ Rm2×m3 and W ( 0 ) ∈ Rm1×d are random weight initializations with i.i.d Gaussian entries V ( 0 ) j , k ∼ N ( 0 , κ22 ) , W ( 0 ) j , k ∼ N ( 0 , κ21 ) , and W s ∈ Rm3×m1 is a random sign matrix , which is roughly a random projection and change of coordinates into a lower dimensional space . We refer to W s 1√ m1 σ ( ( W ( 0 ) +W ′ ) x ) as the first layer and 1√ m2 aTσ ( ( V ( 0 ) + V ′ ) ( . ) ) as the second layer . The algorithm trains weight matrices V ′ , W ′ , and W s , a are fixed . We assume that the outputs are a.s. bounded by a constant , |yi| ≤ B , and ∥xi∥ = 1 . We consider the square loss denoted by ℓ ( . , . ) . We denote the training ( empirical ) loss of a function f on our dataset { xi } ni=1 , { yi } ni=1 , and the expected loss with respect to the data distribution ( population loss ) by Rn ( f ) = 1 n ∑n i=1 ℓ ( f ( xi ) , yi ) , and R ( f ) = Eℓ ( f ( x ) , y ) , respectively . Sometimes , we refer to the vector of labels ( yi ) ni=1 by y . We denote the space of functions with bounded RKHS-norm of kernel K byHK and write Õ to hide the log factors . 2.1 KERNEL APPROXIMATIONS , DECOMPOSITION AND ADAPTIVITY Kernel approximations of neural networks play an important role in our analysis . First , a common approximation is the NTK . The Neural Tangent kernel for a 2-layer ReLu network is H∞ ( x1 , x2 ) = ⟨x1 , x2⟩ · F2 ( ⟨x1 , x2⟩/ ( ∥x1∥∥x2∥ ) ) , for F2 ( x ) = 14 + arcsin ( x ) / ( 2π ) . ( 2 ) To introduce adaptivity , a key part of our analysis is to approximate the second layer in the 3-layer network by a product kernel K∞ ⊙ G that decomposes into a “ fixed ” part K∞ and an “ adaptive ” part G. To define these kernels , for every i ∈ [ n ] , let ϕ ( 0 ) ( xi ) be the output of the first layer of the network at initialization , ϕ ( 0 ) ( xi ) = 1√m1W sσ ( W ( 0 ) xi ) , and ϕ ( 0 ) ( xi ) + ϕ′ ( xi ) be that output for weights W ( 0 ) +W ′ . The adaptive kernel G captures the dot product between the learned weights : G ( xi , xj ) = ⟨ϕ′ ( xi ) , ϕ′ ( xj ) ⟩ . ( 3 ) This form of G motivates the complexity measure we define in the next section , if one thinks of the entries of ϕ′ as bounded NTK-norm functions of the input . Next , we consider the second layer , where the partK∞ arises from roughly stable activations . To formalize this stability , let Sgn ( V x ) be the diagonal matrix whose diagonal contains the coordinate-wise signs of vector V x . If we assume that Sgn ( ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ′ ( xi ) ) ) ≈ Sgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) – we prove a rigorous statement in Appendix A.12 – then fW ′ , V ′ ( xi ) = 1√ m2 aT ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ′ ( xi ) ) ( 4 ) ≈ 〈 V ( 0 ) + V ′ , 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) ( ϕ ( 0 ) ( xi ) + ϕ ′ ( xi ) ) T〉 . ( 5 ) Focusing on the adaptive part ϕ′ ( x ) of the first layer , we write〈 V ( 0 ) + V ′ , 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) ϕ′ ( xi ) T 〉 : = 〈 V ( 0 ) + V ′ , Υ ( xi ) 〉 ( 6 ) and can then view the second layer as a function in the RKHS of the product kernel ⟨Υ ( xi ) , Υ ( xj ) ⟩ = : K̃∞ ( xi , xj ) G ( xi , xj ) . This defines the new kernel K̃∞ , which we simplify into the kernel K∞ that is independent of initialization . To do so , we first observe that K̃∞ concentrates around Ew∼N ( 0 , κ22I ) 1 { w Tϕ ( 0 ) ( xi ) } 1 { wTϕ ( 0 ) ( xj ) } = F2 ( ⟨ϕ ( 0 ) ( xi ) , ϕ ( 0 ) ( xj ) ⟩ ∥ϕ ( 0 ) ( xi ) ∥∥ϕ ( 0 ) ( xj ) ∥ ) . Moreover , the Gaussian initialization and assumption ∥xi∥ = 1 imply that ⟨ϕ ( 0 ) ( xi ) , ϕ ( 0 ) ( xj ) ⟩ concentrates around m3F3 ( ⟨xi , xj⟩ ) , for F3 : [ −1 , 1 ] → [ 0 , 12 ] defined as F3 ( x ) = 1 2π √ 1− x2 + 14x+ 1 2πx arcsin ( x ) , ( 7 ) so K̃∞ ( xi , xj ) ≈ F2 ( 2F3 ( ⟨xi , xj⟩ ) ) = : K∞ ( xi , xj ) . ( 8 ) For general x1 , x2 not necessarily unit norm , we define K∞ ( x1 , x2 ) = K∞ ( x1/∥x1∥ , x2/∥x2∥ ) . It is easy to check that the coefficients in the Taylor series of F2 and F3 are nonnegative . Combining this with Schur ’ s Product Theorem implies K∞ is PSD ( Appendix A.4 ) . We also denote the data kernel matrix on ( xi ) ni=1 by K ∞ and H∞ . As Arora et al . ( 2019a ) , we assume the data distribution is ( λ0 , δ , n ) -non-degenerate with respect to H∞ and K∞ , i.e. , with probability at least 1 − δ , the smallest eigenvalues of H∞ and K∞ are at least λ0 > 0 . 3 DATA DEPENDENT COMPLEXITY MEASURE AND GENERALIZATION The emergence of G ⊙ K∞ above gives rise to an adaptive kernel-like complexity measure that will determine generalization bounds . Intuitively , this complexity measure reflects the two layers . Here , we viewG as the Gram matrix of some “ ideal ” first-layer feature functions gk . We measure the complexity of the prediction function via the RKHS ofA : = G⊙K∞ , and allow a flexible choice of the features gk . Here , gk ’ s substitute the entries of ϕ′ in Equation ( 3 ) : G ( xi , xj ) : = ∑ gk ( xi ) gk ( xj ) . To regularize this choice , we penalize the complexity of the features g via the NTK norm , so we seek gk ’ s with the minimum possible norms . Alternatively , the features gk are flexible but have bounded NTK norm . For a labeling f∗ ∈ Rn of the n data points and fixed G , this leads to the complexity ζ ( f∗ , G ) = f∗TA−1f∗ · ⟨H∞−1 , G⟩ = f∗TA−1f∗ ∑m3 k=1 ∥gk∥2H∞ ; A = G⊙K∞ , ( 9 ) where m3 is the number of intermediate features ( The choice of m3 is discussed in Appendix A.13 ) . Our data-dependent complexity measure implicitly selects the G ( or equivalently the feature vectors gk ) that leads to the tightest bound , trading off data fit and function complexity : ℑ = ℑ ( ( xi ) ni=1 , ( yi ) ni=1 ) : = min f∗∈Rn { 2nRn ( f ∗ ) +ϖmin G≻0 ζ ( f∗ , G ) } , ( 10 ) where we use a log factor ϖ = O ( log ( n ) 3 + log ( 1/λ0 ) ) . To make the relation to adaptive kernel spaces even more explicit , assume that the gk are bounded as ∑ k ∥gk∥2H∞ ≤ 1 . Then we define a family K of corresponding kernels of the form K { g } ( x1 , x2 ) = K ∞ ( x1 , x2 ) ( ∑ g∈ { g } g ( x1 ) g ( x2 ) ) , ( 11 ) i.e. , K : = { K { g } | { g } finite , ∑ g∈ { g } ∥g∥2H∞ ≤ 1 } . With this notation , the complexity measure is ℑ ( ( xi ) , ( yi ) ) = min f∗∈Rn { 2nRn ( f ∗ ) +ϖ min K∈K f∗TK−1f∗ } . ( 12 ) Hence , this measure may be understood as searching for the most efficient and effective feature representation within a family of RKHSs . We may also relate this complexity measure to the NTK-based complexity measure yTH∞−1y ( Arora et al. , 2019a ) . For any labeling f∗ , let f̃∗ ∈ HH∞ be the function with minimum NTK norm that maps xi ’ s to f∗i ’ s , so ∥f̃∗∥H∞ = f∗ TH∞−1f∗ . If we set { g } = { f̃∗/∥f̃∗∥ } , then one can show ( Appendix A.5 ) f∗TK−1 { f̃∗/∥f̃∗∥ } f ∗ ≤ 4f∗T ( f∗f∗T ) −1f∗ × ∥f̃∗∥2 = 4∥f̃∗∥2 = 4f∗TH∞−1f∗ , ( 13 ) which implies ℑ ≤ min f∗∈Rn { 2nRn ( f ∗ ) + ( 4ϖ ) f∗TH∞−1f∗ } . ( 14 ) One can further set f∗ = y above and obtain ℑ ≤ ( 4ϖ1 ) yTH∞−1y . ( 15 ) 3.1 GENERALIZATION With the complexity ℑ in hand , we can now state our generalization result . It assumes optimization by a Projected Stochastic Gradient Descent ( PSGD ) , described in detail in Section 4 . Theorem 1 Suppose we run Projected Stochastic Gradient Descent ( PSGD ) on the regularized empirical risk with parameters as in Section 4 . Then , with high probability ( e.g . 0.99 ) over the randomness of data , initialization and noise of the gradient steps , PSGD converges in poly ( B ∨ 1/B , 1/λ0 , n ) iterations to a solution ( WPSGD , VPSGD ) with population risk bounded as ( recall |yi| ≤ B a.s. ) R ( fWPSGD , VPSGD ) ≤ ℑ ( ( xi ) ni=1 , ( yi ) ni=1 ) n + B2ϖ n . ( 16 ) As a side remark , the factor 2 in front of Rn ( f∗ ) in the definition of ℑ in ( 12 ) is not special and a similar generalization bound can be obtained for any γ > 1 . Substituting the upper bound on the complexity in Equation ( 15 ) , one recovers an NTK-based generalization bound that scales with yTH∞−1y/n up to log factors , which is roughly the square of the generalization bound presented in ( Arora et al. , 2019a ) . The reason for the faster squared rate here is that we are considering smooth losses , while Arora et al . ( 2019a ) work with a bounded Lipschitz loss . Indeed , it is not hard to apply a more rigorous uniform convergence analysis from ( Srebro et al. , 2010 ) to also obtain a faster squared rate for the approach used in ( Arora et al. , 2019a ) . Since Equation ( 15 ) is an upper bound on our complexity , our result generalizes and tightens the NTK bound ( Arora et al. , 2019a ) . To illustrate the flexibility of our complexity measure , we show in Appendix A.1 a simple explicit example of functions represented as polynomial series where our bound improves upon the NTK bound ( Arora et al. , 2019a ) . Notably , we only substitute low-rank matrices G in our complexity measure for this construction . We leave further investigation of our complexity measure for arbitrary G ’ s to future work . 3.2 UNDERLYING CONCEPT CLASS Instead of data dependent generalization bounds , one may study the generalization gap with respect to some concept class . The complexity measure ℑ implicitly uses the following adaptive norm on the space of functions from Rd , the infimum of the RKHS norms for the family of kernels K : ∥f∥ζ = inf K { g } ∈K ∥f∥K { g } . ( 17 ) It is not hard to check that ∥.∥ζ is in fact a norm , and that the inf is achieved by a particular set { g } . Similar to the derivation of the upper bound on the complexity measure in Equation ( 15 ) , by setting { g } = { f∗/∥f∗∥H∞ } , we obtain the following NTK upper bound : ∥f∥ζ ≤ 4∥f∥H∞ . ( 18 ) This leads to a function-dependent generalization bound which bounds the risk of the learned network against an arbitrary function f with ∥f∥ζ < ∞ . Theorem 2 For any measurable function f : Rd → R , in the same setting as Theorem 1 , the population risk of the trained network can be bounded as R ( fWPSGD , VPSGD ) ≤ 2R ( f ) +O ( ϖ ∥f∥2ζ+B 2 n ) . ( 19 ) As in the data-dependent case , the factor 2 of the risk of f can be reduced to any constant γ > 1 . 3.3 INTERACTION OF LAYERS BEYOND THE LINEAR APPROXIMATION Here , we give a high level intuition on how the adaptivity is achieved in our regime compared to NTK . In the NTK approach , for every input x , the neural net fW , V ( x ) is approximated by its linear approximation at ( W ( 0 ) , V ( 0 ) ) ( the initialized network ) , fW , V ( x ) = ⟨∇W , V fW ( 0 ) , V ( 0 ) ( x ) , ( W − W ( 0 ) , V − V ( 0 ) ) ⟩ . The NTK approximation works as long as ( W , V ) are close enough to their initialization that the linear approximation remains accurate and the interaction of weights between layers is negligible . Specifically , the features ϕ′ ( xi ) behave almost linearly with respect to W − W ( 0 ) as ∥W − W ( 0 ) ∥ is taken to be small and the sign pattern Sgn ( ( W ( 0 ) + W ′ ) xi ) is proven not to change much compared to Sgn ( W ( 0 ) xi ) . Additionally , the NTK-type analysis needs the following two conditions to be satisfied : ( 1 ) the sign pattern of ϕ ( 0 ) ( xi ) + ϕ′ ( xi ) with respect to V ( 0 ) + V ′ remains almost the same as the sign pattern of ϕ ( 0 ) ( x ) with respect to V ( 0 ) , and ( 2 ) the weight changes W ′ and V ′ should not interact , which means the “ interaction ” term , 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) V ′ϕ′ ( xi ) ≈ 0 , should be negligible . Therefore , the non-negligible terms for the NTK are : ( 1 ) 1√ m2 aTSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) V ( 0 ) ϕ′ ( xi ) , which is almost linear in W ′ ( recall that ϕ′ ( xi ) depends on W ′ ) , and ( 2 ) 1√m2 a TSgn ( V ( 0 ) ϕ ( 0 ) ( xi ) ) V ′ϕ ( 0 ) ( xi ) which is linear in V ′ . This approach has two important implications : ( 1 ) it convexifies the optimization ( for convex loss ) , as the approximation is now linear in W ; and ( 2 ) it simplifies proving generalization , as it works with the class of functions in the RKHS space of some fixed kernel . However , this simplification leaves no room for the ability of the neural network to learn intermediate feature representations which could result in better function classes . In our regime , however , we enforce the condition ∀j , i : V ′j ⊥ ϕ ( 0 ) ( xi ) ( ⋆ ) , which implies the second ( 2 ) above is zero , while the interaction term is not negligible any more and the network behaves similar to a quadratic function with respect to ( W ′ , V ′ ) ( for fixed xi ) . Condition ( ⋆ ) is critical both in proving the convergence of the algorithm as well as bounding the Rademacher complexity of the class of networks with bounded weights . Rather than working with a fixed kernel , the interaction term enables us to use the first layer for representing the input in a suitable feature space , which can be interpreted as picking a suitable kernel , then use the second layer to describe the output based on those features . This is also indirectly encoded in our complexity measure.In addition to enforcing the orthogonality condition ( ⋆ ) ( in the SGD variant ) , conditions for entering our regime are that the overparameterization m1 , m2 , m3 and κ1 , κ2 are within a specific range with respect to each other . We listed these relations in Appendix A.3 . To illustrate the benefit of going to this more involved regime , denote the class of neural nets with Frobenius bounded norms ∥W ′∥ ≤ γ1 , ∥V ′∥ ≤ γ2 by Gγ1 , γ2 ( and a bit more structure which we elaborate upon in the proofs ) ; it turns out that Gγ1 , γ2 roughly includes HK ( O ( γ1γ2 ) ) for every kernel K ∈ K , in the sense that each f ∈ HK ( O ( γ1γ2 ) ) is well-approximated within Gγ1 , γ2 to arbitrarily small error on fixed input ( the error goes down with the size of the network ) . On the other hand , we show that the Rademacher Complexity ( RC ) of Gγ1 , γ2 behaves similar to the RC of the NTK class HH∞ ( O ( γ1γ2 ) ) ! As our algorithm guarantees finding a network with sufficiently small empirical risk within Gγ1 , γ2 , this phenomenon underlies our adaptive generalization bounds . Compared to previous work that provides an adaptive kernel analysis still for a two layer model ( Dou & Liang , 2020 ) ( although their analysis is for the gradient flow and non-algorithmic ) , our model requires an additional layer so it can , in a sense , “ simulate ” the process of feature extraction in one layer to be used in the next layer . 3.4 COMPARISON WITH KERNEL FITTING We compare our generalization bounds with some kernel fitting rates . Given a kernel K with K ( x , x ) ≤ 1 for every x : ∥x∥ ≤ 1 , suppose we want to fit a function fromHK ( B′ ) , i.e . having KRKHS norm bounded by B′ . In the realizable setting , when there is an underlying f∗∗ ∈ HK ( B′ ) with zero risk , Empirical Risk Minimization ( ERM ) enjoys a fast rate using the smoothness of the loss ( Srebro et al. , 2010 ) . The Rademacher Complexity bound R ( HK ( B′ ) ) ≤ O ( B ′ √ n ) then implies : R ( fERM ) ≤ Õ ( B′2/n ) ( 20 ) for the square loss , which is minimax optimal up to log factors . To compare the neural network , we substitute f∗∗ into Theorem 2 . To relate the B in our bound to B′ , assume for simplicity of exposition that ∥f∥ζ = ∥f∥K∗ for some K∗ ∈ K ( otherwise we can use a convergent sequence ) . Observing that K { g } ( x , x ) ≤ 1 for every kernel K { g } ∈ K , we obtain that |f∗∗ ( x ) | ≤ ∥f∗∗∥K∗ = ∥f∗∗∥ζ ( Appendix A.6 ) . Combining this fact with the realizability assumption , we can then upper bound the parameter B in Theorem 2 by ∥f∗∗∥ζ , and obtain R ( fWPSGD , VPSGD ) = Õ ( ∥f∗∗∥ζ 2 /n ) . ( 21 ) If we further takeK to be inK , then Equation ( 21 ) combined with ∥f∗∗∥ζ ≤ ∥f∗∗∥K ≤ B′ implies : R ( fWPSGD , VPSGD ) = Õ ( B ′2/n ) , ( 22 ) that is , for every kernel K ∈ K , our deep learning approach almost achieves the conventional kernel bound in Equation ( 20 ) . Repeating the uniform risk bound stated in Theorem 1 in ( Srebro et al. , 2010 ) for HK ( B′ ) where B′ is set to all powers of two , followed by a union bound , one can easily obtain a fast rate of R ( fKRLR ) ≤ Õ ( y TK−1y n + B2 n ) , ( 23 ) for the solution of KRLR in the general case ( not realizable ) for the square loss . On the other hand , for a B-bounded Lipschitz loss , we instead get a slow rate for KRLR : R ( fKRLR ) ≤ Õ ( √ yTK−1y n + B√ n ) , whereB is an a.s. bound on |y| as before . This bound is similar to Arora et al . ( 2019a ) . Note that our data dependent generalization bound in Theorem 1 already achieves the fast rate for KRLR in ( 23 ) for any K ∈ K. Finally , in the non-realizable case , we still have the following fast rate for ERM regarding the hypothesis classHK ( B′ ) ( Srebro et al. , 2010 ) : R ( fERM ) ≤ Õ ( R ( f∗∗ ) + B ′2+B2 n ) , where now f∗∗ : = argminf∈HK ( B′ ) R ( f ) , while Theorem 2 also implies ( again for every K ∈ K ) : R ( fWPSGD , VPSGD ) ≤ Õ ( R ( f∗∗ ) + ∥f∗∗∥2ζ+B 2 n ) = Õ ( R ( f ∗∗ ) + B ′2+B2 n ) . 4 ALGORITHM : PROJECTED STOCHASTIC GRADIENT DESCENT Next , we discuss the algorithm that achieves the generalization bounds above . Two specific challenges are to address the interaction of layers , and to establish that the algorithm remains in a regime in which it converges to a good solution that generalizes . In this section , we describe our algorithm PSGD , presented as pseudocode in Figure 1 , which is roughly Stochastic Gradient Descent modified to project out a low-dimensional random subspace from the second-layer weights.PSGD roughly runs SGD on a smoothed version of the following loss function ( ψ1 , ψ2 are defined in Appendix A.2 ) L1 ( W ′ , V ′ ) = Rn ( fW ′ , V ′ ) + ψ1∥W ′∥2 + ψ2∥V ′∥2 . Compared to standard SGD , our algorithm makes two modifications : ( 1 ) it uses randomized smoothing to alleviate the non-smoothness of the ReLus , ( 2 ) it ensures that the weights in the second layer are orthogonal to the data features ϕ ( 0 ) ( x ) computed by the first layer at initialization . This helps to control layer interactions as pointed out in Section 3.3 . For smoothing , we add Gaussian smoothing matrices W ρ and V ρ to the weights with i.i.d . entries from N ( 0 , β21/m1 ) and N ( 0 , β22/m2 ) respectively , for β2 = Op ( ( κ1 √ m1 ) −1 ( κ2 √ m2 ) −2/3 ) , β1 = Op ( m23κ2 √ m2 ( κ1 √ m1 ) −1 ) . To simplify the exposition , Op ( . ) is hiding the dependencies on the basic parameters B , n , 1/λ0 and log factors . Our convergence proof uses the loss with respect to this smoothed network . For the projection , let Φ⊥ ⊂ Rm2×m3 be the subspace of weights of the second layer whose rows are orthogonal to the first-layer data representations ϕ ( 0 ) ( xi ) ’ s ∀i ∈ [ n ] at initialization : V ′ ∈ Φ⊥ ↔ ∀j ∈ [ m2 ] , ∀i ∈ [ n ] : V ′j , ϕ ( 0 ) ( xi ) = 0 . ( 24 ) In summary , at point ( W ′ , V ′ ) , the algorithm samples a random ( xi , yi ) from the data , as well as smoothing matrices W ρ,1 , V ρ,1 , W ρ,2 , V ρ,2 . It then computes an unbiased estimate for the gradient ( ∇̂W , ∇̂V ) , adds additional normalized Gaussian noise matrices Ξ1 , Ξ2 and moves in this direction with step size η = 1/poly ( n , B ∨ 1/B , 1/λ0 ) : ( W ′ , V ′ ) ← ( W ′ , V ′ ) + η ( ∇̂W + Ξ1/ ( √ m1∥Ξ1∥ ) , ProjΦ⊥ ( ∇̂V + Ξ2/∥Ξ2∥ ) ) . ( 25 ) Parameters . Our results apply to the overparametrization regime , when the size of the network , i.e . parameters m1 , m2 , m3 is polynomially large in n , B ∨ 1/B , 1/λ0 . This ensures that we can guarantee that the network has good enough function representation capacity , and PSGD is able to find a good local direction in every iteration . The regularization coefficients ψ1 , ψ2 can be set with respect to any candidate ( f∗ , G ) for our complexity measure ( 9 ) . In Appendix A.2 , we introduce a simple doubling trick which handles when we do not have access to an optimal candidate solution . With such an f∗ , define ν : = max { Rn ( f∗ ) /2 , B2/n } , and set ψ1 = ν/4 , ψ2 = ν/ ( 4ζ ( f∗ , G ) ) . We list the suitable regime for our overparameterization in Appendix A.3 . Algorithm 1 PSGD ( Projected Stochastic Gradient Descent ) Input : network architecture m1 , m2 , m3 , initialization parameters κ1 , κ2 , smoothing parameters β1 , β2 , training set ( xi , yi ) ni=1 , label parameter B , ( f ∗ , G ) from the complexity measure 1 : Gaussian initialization W ( 0 ) j , k ← N ( 0 , κ1 ) , V ( 0 ) j , k ← N ( 0 , κ2 ) 2 : Define parameters ψ1 , ψ2 , ν , η , subspace Φ⊥ , and objective L1 as described in Section 4 3 : while L1 ( W ′ , V ′ ) > Rn ( f∗ ) + 2ν do 4 : Gaussian matrices W ρ,1j , k , W ρ,2 j , k ← N ( 0 , β21 m1 ) , V ρ,1j , k , V ρ,2 j , k ← N ( 0 , β22 m2 ) 5 : Sample data ( xi , yi ) uniformly at random 6 : Compute gradient estimates 7 : { ∇̂W = ℓ̇ ( fW ′+Wρ,1 , V ′+V ρ,1 ( xi ) , yi ) ∇W fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) + 2ψ1W ′ , ∇̂V = ℓ̇ ( fW ′+Wρ,1 , V ′+V ρ,1 ( xi ) , yi ) ∇V fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) + 2ψ2V ′ 8 : Move as ( W ′ , V ′ ) ← ( W ′ , V ′ ) + η ( ∇̂W +Ξ1/ ( √ m1∥Ξ1∥ ) , ProjΦ⊥ ( ∇̂V +Ξ2/∥Ξ2∥ ) ) 9 : Return ( W ′ , V ′ ) 5 HIGH LEVEL IDEA OF THE PSGD ANALYSIS Aside from the network ’ s representation ability , a major challenge is the poly-time computation of some good weights that generalize . On the one hand , we want the weights to remain close to their initialization so the final network is in the class Gγ1 , γ2 for suitably chosen γ1 , γ2 , where recall that Gγ1 , γ2 is roughly the class of nets with bounded norms ∥W ′∥ ≤ γ1 , ∥V ′∥ ≤ γ2 . On the other hand , we aim to reduce the nonconvex empirical loss Rn ( fW ′ , V ′ ) . Aligned with these goals , we roughly run SGD on our smoothed objective consisting of the empirical risk regularized by the Frobenius norm of the weight matrices . We prove convergence by building on ideas from Allen-Zhu et al . ( 2018 ) , with a framework based on the classic result that SGD can escape saddle points in face of nonconvexity . Compared to ( AllenZhu et al. , 2018 ) , we take a different approach driven by our purely data-dependent complexity measure in Theorem 1 . We augment this by a careful Rademacher complexity analysis in Appendix A.11 for Frobenius norm bounded networks in our regime , to precisely get the same behavior as the RC bound for the class of NTK-norm bounded functions ( Arora et al. , 2019a ) . We face two challenges in the proof : first , to have an SGD-type algorithm , we can not directly guide the optimization trajectory based on our complexity measure , so we have little control on the random trajectory of PSGD . Therefore , we mainly rely on uniform convergence techniques in our convergence proof . In particular , we prove it is unlikely that PSGD reaches a “ bad '' saddle point if the objective is above some threshold . Second , to prove the last fact , we require a weak regularity condition that the value of the objective does not exceed a certain threshold during the algorithm . This puts us in a different setting compared to the traditional result on escaping saddle points ( Ge et al. , 2015a ) , so we need to refine their analysis for our setting . We give a high-level sketch of our proof in Section 5 . Below , we outline the steps of the convergence proof at a high level . Construction of a good Network To study the loss landscape , similar to ( Allen-Zhu et al. , 2018 ) , we show the existence of a good local update for PSGD at reasonable points ( W ′ , V ′ ) , using the ideal weight matrices ( W ∗ , V ∗ ) that we carefully construct from our complexity measure . Here , we sketch our proof for constructing ( W ∗ , V ∗ ) . Let ( W ′ , V ′ ) be the current weights of the algorithm . Fix a sample i ∈ [ n ] . In Appendix A.12 , we use G to construct W ∗ for the first layer weights with decomposition W ∗ = ∑m3 k=1W ∗ k and O ( 1 ) bounded norm , such that ϕ∗ ( xi ) k : = 1√ m1 W sSgn ( ( W ( 0 ) + W ′ ) xi ) W ∗xi . This decomposition ensures that for every k , k′ ∈ [ m3 ] , negating the matrix W ∗k only negates ϕ∗ ( xi ) k′ when k′ = k and has no effect on ϕ∗ ( xi ) k′ for k′ ̸= k. This way , we can easily generate any arbitrary sign flip of the entries of ϕ∗ ( xi ) . We use this property to generate a suitable random descent direction . Next , we want to construct a suitable weight matrix V ∗ for the second layer which maps the features ϕ∗ ( xi ) into f∗i ( recall the definition of the complexity measure ) . The key here is that we consider a regime where the norm of ϕ ( 0 ) ( xi ) is typically larger than that of ϕ′ ( xi ) and ϕ∗ ( xi ) , so it is very likely that the sign pattern in the second layer is determined by ϕ ( 0 ) ( xi ) in most of the rows . In such a senario , having the condition V ′j ⊥ ϕ ( 0 ) ( xi ) becomes vital as the interaction of V ′ with ϕ ( 0 ) ( xi ) becomes problematic both for the generalization and optimization . From the standpoint of generalization , without excluding this interaction , one can exploit the large size of ϕ ( 0 ) ( xi ) and build a network within the class Gγ1 , γ2 ( i.e. , with ∥W ′∥ , ∥V ′∥ suitably bounded ) corresponding to a complex function which overfits the data . Indeed , we utilize the large magnitude of ϕ ( 0 ) and its orthogonality to the rows of V ′ in our RC bound . On the other hand , mainly because changing the weights of the first layer can not affect ϕ ( 0 ) ( xi ) , the interaction of V ′ with ϕ ( 0 ) ( xi ) is problematic in our convergence analysis for optimization part , particularly in proving the existence of a local descent direction . This is the rationale behind our orthogonality constraint ( 24 ) . Finally , the ϕ∗ ( xi ) ’ s , the above control on the signs , and the fact that ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ concentrates around m3Ew∼N ( 0 , κ1I ) [ σ ( wTxi1 ) σ ( wTxi2 ) ] which recovers the structure of the kernel K∞ ( Section 2 ) , give rise to the kernelG⊙K∞ in the second layer . Using this structure , we construct V ∗ that maps ϕ∗ ( xi ) ’ s to f∗i ’ s , which has additional good properties , includingO ( f ∗T ( G⊙K∞ ) −1f∗ ) bounded norm , and rows that are orthogonal to ϕ ( 0 ) ( xi ) ’ s . For more details , see Appendix A.12 . Nonexistence of Bad Saddle Points Next , we want to exploit ( W ∗ , V ∗ ) to prove the existence of a good direction along which reduces the objective locally . Moving along ( W ∗ , V ∗ ) is the first idea which fails as the cross terms created between W ′ , V ∗ and V ′ , W ∗ can not be bounded effectively . Instead , we randomly perturb W ∗ and V ∗ in a coupled way and prove a reduction in expectation . We elaborate more on this suitable random direction . Multiplying random signs Σk ontoW ∗k , we define the sumW ∗Σ = ∑m3 k=1 ΣkW ∗ k . We also multiply the same signs to the columns of V ∗ and project it back onto Φ⊥ to obtain V ∗Σ . Then , we move in the random direction ( √ ηW ∗Σ − ηW/2 , √ ηV ∗Σ − ηV/2 ) ; this update creates additional cross terms in the objective that we must bound to prove a local reduction argument . A key point here is that we prove that with high probability , the norm of the weights remains bounded along the algorithm , i.e . ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 . This norm restriction enables us to substitute terms that we do not have control over by their worst-case supremum counterparts . We refer to Appendix A.13 for similar techniques . Convergence of PSGD Finally , we use the fact that SGD escapes good saddle points ( Ge et al. , 2015b ) . For proving the existence of a good random direction to escape saddle points above , we use that the norm of weights is uniformly bounded along all iterations ; this bound , in fact , is looser than the bound that we show for the final weights of the network . Yet , this additional restriction can not be addressed by the classical nonconvex theory of SGD . Consequently , we refine and adapt the proof of ( Ge et al. , 2015b ) to incorporate this additional constraint . At a high level , Ge et al . ( 2015b ) work with a supermartingale based on the loss value . To guarantee the additional norm restriction , it is initially tempting to apply Azuma-Hoeffding concentration to bound the upward deviations of this process . However , this fails as the process has a two-fold behavior , depending on how large the gradient is . At the core of our refinement proof here , we instead directly bound the MGF of the martingale using Doobs maximal inequality . We refer to Section A.16 for more details . REFERENCES Zeyuan Allen-Zhu and Yuanzhi Li . Backward feature correction : How deep learning performs deep learning . arXiv preprint arXiv:2001.04413 , 2020 . Zeyuan Allen-Zhu , Yuanzhi Li , and Yingyu Liang . Learning and generalization in overparameterized neural networks , going beyond two layers . arXiv preprint arXiv:1811.04918 , 2018 . Zeyuan Allen-Zhu , Yuanzhi Li , and Zhao Song . A convergence theory for deep learning via overparameterization . In International Conference on Machine Learning , pp . 242–252 . PMLR , 2019 . Sanjeev Arora , Nadav Cohen , Noah Golowich , and Wei Hu . A convergence analysis of gradient descent for deep linear neural networks . arXiv preprint arXiv:1810.02281 , 2018a . Sanjeev Arora , Rong Ge , Behnam Neyshabur , and Yi Zhang . Stronger generalization bounds for deep nets via a compression approach . In International Conference on Machine Learning , pp . 254–263 . PMLR , 2018b . Sanjeev Arora , Simon Du , Wei Hu , Zhiyuan Li , and Ruosong Wang . Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks . In International Conference on Machine Learning , pp . 322–332 . PMLR , 2019a . Sanjeev Arora , Simon S Du , Wei Hu , Zhiyuan Li , Ruslan Salakhutdinov , and Ruosong Wang . On exact computation with an infinitely wide neural net . arXiv preprint arXiv:1904.11955 , 2019b . Francis Bach . Breaking the curse of dimensionality with convex neural networks . The Journal of Machine Learning Research , 18 ( 1 ) :629–681 , 2017 . Ainesh Bakshi , Rajesh Jayaram , and David P Woodruff . Learning two layer rectified neural networks in polynomial time . In Conference on Learning Theory , pp . 195–268 . PMLR , 2019 . Andrew R Barron . Approximation and estimation bounds for artificial neural networks . Machine learning , 14 ( 1 ) :115–133 , 1994 . Peter Bartlett , Dylan J Foster , and Matus Telgarsky . Spectrally-normalized margin bounds for neural networks . arXiv preprint arXiv:1706.08498 , 2017 . Peter Bartlett , Dave Helmbold , and Philip Long . Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks . In International conference on machine learning , pp . 521–530 . PMLR , 2018 . Alon Brutzkus and Amir Globerson . Globally optimal gradient descent for a convnet with gaussian inputs . In International conference on machine learning , pp . 605–614 . PMLR , 2017 . Yuan Cao and Quanquan Gu . Generalization bounds of stochastic gradient descent for wide and deep neural networks . In Advances in Neural Information Processing Systems ( NeurIPS ) , 2019 . Lenaic Chizat and Francis Bach . On the global convergence of gradient descent for overparameterized models using optimal transport . arXiv preprint arXiv:1805.09545 , 2018 . Lenaic Chizat and Francis Bach . Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss . In Conference on Learning Theory , pp . 1305–1338 . PMLR , 2020 . Amit Daniely . Depth separation for neural networks . In Conference on Learning Theory , pp . 690– 696 . PMLR , 2017 . Xialiang Dou and Tengyuan Liang . Training neural networks as learning data-adaptive kernels : Provable representation and approximation benefits . Journal of the American Statistical Association , pp . 1–14 , 2020 . Simon S Du , Xiyu Zhai , Barnabas Poczos , and Aarti Singh . Gradient descent provably optimizes over-parameterized neural networks . arXiv preprint arXiv:1810.02054 , 2018 . Lixin Duan , Ivor W Tsang , and Dong Xu . Domain transfer multiple kernel learning . IEEE Transactions on Pattern Analysis and Machine Intelligence , 34 ( 3 ) :465–479 , 2012 . Gintare Karolina Dziugaite and Daniel M Roy . Computing nonvacuous generalization bounds for deep ( stochastic ) neural networks with many more parameters than training data . arXiv preprint arXiv:1703.11008 , 2017 . C Daniel Freeman and Joan Bruna . Topology and geometry of half-rectified network optimization . arXiv preprint arXiv:1611.01540 , 2016 . Ken-Ichi Funahashi . On the approximate realization of continuous mappings by neural networks . Neural networks , 2 ( 3 ) :183–192 , 1989 . Rong Ge , Furong Huang , Chi Jin , and Yang Yuan . Escaping from saddle pointsonline stochastic gradient for tensor decomposition . In Conference on learning theory , pp . 797–842 . PMLR , 2015a . Rong Ge , Furong Huang , Chi Jin , and Yang Yuan . Escaping from saddle points — online stochastic gradient for tensor decomposition . In Peter Grünwald , Elad Hazan , and Satyen Kale ( eds . ) , Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pp . 797–842 , Paris , France , 03–06 Jul 2015b . PMLR . URL http : //proceedings.mlr.press/v40/Ge15.html . Rong Ge , Jason D Lee , and Tengyu Ma . Learning one-hidden-layer neural networks with landscape design . arXiv preprint arXiv:1711.00501 , 2017 . Behrooz Ghorbani , Song Mei , Theodor Misiakiewicz , and Andrea Montanari . Limitations of lazy training of two-layers neural networks . arXiv preprint arXiv:1906.08899 , 2019 . Noah Golowich , Alexander Rakhlin , and Ohad Shamir . Size-independent sample complexity of neural networks . In Conference On Learning Theory , pp . 297–299 . PMLR , 2018 . Geoffrey Hinton , Li Deng , Dong Yu , George E Dahl , Abdel-rahman Mohamed , Navdeep Jaitly , Andrew Senior , Vincent Vanhoucke , Patrick Nguyen , Tara N Sainath , et al . Deep neural networks for acoustic modeling in speech recognition : The shared views of four research groups . IEEE Signal processing magazine , 29 ( 6 ) :82–97 , 2012 . Arthur Jacot , Franck Gabriel , and Clément Hongler . Neural tangent kernel : Convergence and generalization in neural networks . arXiv preprint arXiv:1806.07572 , 2018 . Majid Janzamin , Hanie Sedghi , and Anima Anandkumar . Beating the perils of non-convexity : Guaranteed training of neural networks using tensor methods . arXiv preprint arXiv:1506.08473 , 2015 . A. Javanmard , M. Mondelli , and A. Montanari . Analysis of a two-layer neural network via displacement convexity . ArXiv , abs/1901.01375 , 2019 . Kenji Kawaguchi . Deep learning without poor local minima . In nips , 2016 . Pitas Konstantinos , Mike Davies , and Pierre Vandergheynst . Pac-bayesian margin bounds for convolutional neural networks-technical report . arXiv preprint arXiv:1801.00171 , 2017 . Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . Imagenet classification with deep convolutional neural networks . Advances in neural information processing systems , 25:1097–1105 , 2012 . Yuanzhi Li and Yingyu Liang . Learning overparameterized neural networks via stochastic gradient descent on structured data . arXiv preprint arXiv:1808.01204 , 2018 . Yuanzhi Li and Yang Yuan . Convergence analysis of two-layer neural networks with relu activation . arXiv preprint arXiv:1705.09886 , 2017 . Tengyuan Liang , Alexander Rakhlin , et al . Just interpolate : Kernel ridgeless regression can generalize . Annals of Statistics , 48 ( 3 ) :1329–1347 , 2020 . Yiping Lu , Chao Ma , Yulong Lu , Jianfeng Lu , , and Lexing Ying . A mean-field analysis of deep resnet and beyond : Towards provable optimization via overparameterization from depth . In icml , 2020 . Song Mei , Andrea Montanari , and Phan-Minh Nguyen . A mean field view of the landscape of twolayer neural networks . Proceedings of the National Academy of Sciences , 115 ( 33 ) : E7665–E7671 , 2018 . Song Mei , Theodor Misiakiewicz , and Andrea Montanari . Mean-field theory of two-layers neural networks : dimension-free bounds and kernel limit . In Conference on Learning Theory , pp . 2388– 2464 . PMLR , 2019 . Behnam Neyshabur , Ryota Tomioka , and Nathan Srebro . Norm-based capacity control in neural networks . In Conference on Learning Theory , pp . 1376–1401 . PMLR , 2015 . Behnam Neyshabur , Srinadh Bhojanapalli , and Nathan Srebro . A pac-bayesian approach to spectrally-normalized margin bounds for neural networks . In International Conference on Learning Representations , 2018 . Quynh Nguyen and Matthias Hein . The loss surface of deep and wide neural networks . In International conference on machine learning , pp . 2603–2612 . PMLR , 2017 . Maithra Raghu , Ben Poole , Jon Kleinberg , Surya Ganguli , and Jascha Sohl-Dickstein . On the expressive power of deep neural networks . In Doina Precup and Yee Whye Teh ( eds . ) , Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pp . 2847–2854 . PMLR , 06–11 Aug 2017 . URL http : //proceedings.mlr.press/v70/raghu17a.html . Alexander Rakhlin and Xiyu Zhai . Consistency of interpolation with laplace kernels is a highdimensional phenomenon . In Conference on Learning Theory , pp . 2595–2623 . PMLR , 2019 . Alain Rakotomamonjy , Francis Bach , Stéphane Canu , and Yves Grandvalet . More efficiency in multiple kernel learning . In Proceedings of the 24th international conference on Machine learning , pp . 775–782 , 2007 . Itay Safran and Ohad Shamir . Depth separation in relu networks for approximating smooth nonlinear functions . arXiv preprint arXiv:1610.09887 , 14 , 2016 . Itay Safran and Ohad Shamir . Spurious local minima are common in two-layer relu neural networks . In International Conference on Machine Learning , pp . 4433–4441 . PMLR , 2018 . Itay Safran , Ronen Eldan , and Ohad Shamir . Depth separations in neural networks : what is actually being separated ? In Conference on Learning Theory , pp . 2664–2666 . PMLR , 2019 . Andrew M Saxe , James L McClelland , and Surya Ganguli . Exact solutions to the nonlinear dynamics of learning in deep linear neural networks . arXiv preprint arXiv:1312.6120 , 2013 . David Silver , Aja Huang , Chris J Maddison , Arthur Guez , Laurent Sifre , George Van Den Driessche , Julian Schrittwieser , Ioannis Antonoglou , Veda Panneershelvam , Marc Lanctot , et al . Mastering the game of go with deep neural networks and tree search . nature , 529 ( 7587 ) :484–489 , 2016 . Justin Sirignano and Konstantinos Spiliopoulos . Mean field analysis of neural networks : A central limit theorem . Stochastic Processes and their Applications , 130 ( 3 ) :1820–1852 , 2020 . Mahdi Soltanolkotabi , Adel Javanmard , and Jason D Lee . Theoretical insights into the optimization landscape of over-parameterized shallow neural networks . IEEE Transactions on Information Theory , 65 ( 2 ) :742–769 , 2018 . Sören Sonnenburg , Gunnar Rätsch , Christin Schäfer , and Bernhard Schölkopf . Large scale multiple kernel learning . The Journal of Machine Learning Research , 7:1531–1565 , 2006 . Daniel Soudry and Yair Carmon . No bad local minima : Data independent training error guarantees for multilayer neural networks . arXiv preprint arXiv:1605.08361 , 2016 . Nathan Srebro , Karthik Sridharan , and Ambuj Tewari . Smoothness , low noise and fast rates . In J. Lafferty , C. Williams , J. Shawe-Taylor , R. Zemel , and A. Culotta ( eds . ) , Advances in Neural Information Processing Systems , volume 23 . Curran Associates , Inc. , 2010 . URL https : //proceedings.neurips.cc/paper/2010/file/ 76cf99d3614e23eabab16fb27e944bf9-Paper.pdf . Yuandong Tian . Symmetry-breaking convergence analysis of certain two-layered neural networks with relu nonlinearity . Network , 100 ( 1 ) :1–1 , 2016 . Colin Wei , Jason Lee , Qiang Liu , and Tengyu Ma . On the margin theory of feedforward neural networks . 2018 . Colin Wei , Jason Lee , Qiang Liu , and Tengyu Ma . Regularization matters : Generalization and optimization of neural nets vs their induced kernel . 2019 . Lei Wu , Chao Ma , and Weinan E. A priori estimates of the generalization error for two-layer neural networks . 2018 . Chiyuan Zhang , Samy Bengio , Moritz Hardt , Benjamin Recht , and Oriol Vinyals . Understanding deep learning requires rethinking generalization . arXiv preprint arXiv:1611.03530 , 2016 . Kai Zhong , Zhao Song , Prateek Jain , Peter L Bartlett , and Inderjit S Dhillon . Recovery guarantees for one-hidden-layer neural networks . In International conference on machine learning , pp . 4140– 4149 . PMLR , 2017 . Wenda Zhou , Victor Veitch , Morgane Austern , Ryan P Adams , and Peter Orbanz . Non-vacuous generalization bounds at the imagenet scale : a pac-bayesian compression approach . arXiv preprint arXiv:1804.05862 , 2018 . Difan Zou , Yuan Cao , Dongruo Zhou , and Quanquan Gu . Stochastic gradient descent optimizes overparameterized deep relu networks . arxiv e-prints , art . arXiv preprint arXiv:1811.08888 , 2018 . A APPENDIX Below is the list of different sections of the proof . We have a separate Appendix list for lower level Lemmas in our proof in B . CONTENTS A.1 Stronger Generalization bounds for polynomials . . . . . . . . . . . . . . . . . . . 15 A.2 The Doubling Trick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.3 Amount of Overparameterization . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 A.4 PSD property of K∞ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 A.5 Complexity upper bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.6 Complexity measure and the ζ-norm . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.7 Core Generalization Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.8 Structure of the proof , setting m3 , and further definitions . . . . . . . . . . . . . . 22 A.9 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 A.10 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 A.11 Rademacher Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 A.12 Constructing W ∗ , V ∗ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 A.12.1 First Layer , Construction of W ∗ . . . . . . . . . . . . . . . . . . . . . . . 34 A.12.2 Second Layer , Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . . 39 A.12.3 Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 A.13 Existence of a good direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 A.14 Existence of a good direction Helper Lemmas . . . . . . . . . . . . . . . . . . . . 53 A.15 Bounding the worst-case Senario . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 A.16 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 A.17 Process from a higher view : definition of the ( X ) sequence . . . . . . . . . . . . . 77 A.18 Bounding the MGF of Xi ’ s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 A.19 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 A.20 Gaussian Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 A.20.1 Setting β1 and β2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 A.21 Basic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 A.21.1 Defining the rare events Ej . . . . . . . . . . . . . . . . . . . . . . . . . . 91 A.21.2 Bounding the value of f ′ . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 A.21.3 Bounding the difference between Original and Smoothed Functions . . . . 94 A.1 STRONGER GENERALIZATION BOUNDS FOR POLYNOMIALS In this section , we prove an explicit generalization bound for functions represented as a polynomial sum . Note that the bounds in Arora et al . ( 2019a ) for polynomials assume the monomials with degree larger than one to have even powers , while here we do not impose this restriction . In addition , note that despite Arora et al . ( 2019a ) , our bounds remain meaningful in the noisy case ( recall our Theorem 2 ) . More specifically , we bound the ζ norm of such functions . Consider the target function s with the following power series formula : y = s ( x ) = ∞∑ p=1 ap ( w T p x ) p , ( 26 ) where ap ∈ R and wp ∈ Rd . We can write s ( x ) = g1 ( x ) + d∑ k=1 xkg k 2 ( x ) , ( 27 ) where xk denotes the kth entry of vector x here and g1 ( x ) = ∑ p∈A1 : = { p=1 or p even } ap ( w T p x ) p , and for all k ∈ [ d ] : gk2 ( x ) = ∑ p∈A2 : = { p > 2 , p odd } wpkap ( w T p x ) p−1 . Then , using the tailor series of x ( 14 + arcsin ( x ) 2π ) = ∑∞ p=1 γpx p for |x| ≤ 1 , the RKHS spaceH ( H∞ ) of NTK can be identified by square-summable sequences of reals ( ap′ ) ∞p′=1 with dot product ⟨ ( ap′ ) ∞p′=1 , ( bp′ ) ∞p′=1⟩ = ∞∑ p′=1 γλ ( p′ ) ap′bp′ , where λ ( p′ ) : Z≥0 → Z≥0 such that it maps zero to zero , the first d positive integers are mapped to one , the next d2 ones are mapped to 2 , etc . Moreover , the RKHS mapping Ψ : Rd → H ( H∞ ) from the Euclidean space is : Ψ ( x ) = ∥x∥ ( x′1 , ... , x ′ d , ( x ′ k1x ′ k2 ) k1 , k2∈ [ d ] , . . . , ( x ′ k1x ′ k2 . . . x ′ kp ) k1 , ... , kp∈ [ d ] , . . . ) , where x′ = x/∥x∥ and in the notation above we are presenting a sequence of sequences , by which we mean the inner sequences simply unfold . Using this identification , one can see using the linear representations of g1 , gk2 inH : ∥g1∥2H∞ = ∑ p∈A1 γpa 2 p∥wp∥ 2p 2 , ( 28 ) ∥gk2∥2H∞ = ∑ p∈A2 γpwp 2 ka 2 p∥wp∥ 2 ( p−1 ) 2 , ( 29 ) Summing above and noting the linear representation of g : ∥g1∥2H∞ + d∑ k=1 ∥gk2∥2H∞ = ∑ p∈A1 γpa 2 p∥wp∥ 2p 2 + ∑ p∈A2 γpa 2 p∥wp∥ 2p 2 = ∞∑ p=1 γpa 2 p∥wp∥ 2p 2 = ∥g∥2H∞ . ( 30 ) Now for { g } : = { g′k } d+1 k=1 : = { g1 } ∪ { gk2 } dk=1 , we consider the kernel K { g } . Expanding the tailor series of F2 ( 2F3 ) ( x ) j = ∑∞ p=0 µpx p , we find the identification ( hkp′ ) k∈ [ d+1 ] , p′=0 , ... , ∞ with dot product = ∞∑ p′=0 µλ ( p′ ) d+1∑ k=1 hkp′q k p′ , with RKHS map Ψ2 ( x ) = ( g′1 ( x ) , . . . , g ′ d+1 ( x ) , x ′ 1g ′ 1 ( x ) , . . . , x ′ 1g ′ d+1 ( x ) , . . . , x ′ dg ′ 1 ( x ) , . . . , x′dg ′ d+1 ( x ) , ( x ′ k1x ′ k2g ′ k ( x ) ) k1 , k2∈ [ d ] , k∈ [ d+1 ] , . . . , ( x ′ k1x ′ k2 . . . x ′ kpg ′ k ( x ) ) k1 , ... , kp∈ [ d ] , k∈ [ d+1 ] , . . . ) . Now , we compute the norm of function s with respect to K { g } , combining the above representation and dot product with Equation ( 27 ) and the fact that we work with unit norm x , so x′ = x : ∥s∥2K { g } = µ0 + ( d+ 1 ) µ1 . ( 31 ) Plugging the above and Equation ( 30 ) into the definition of ∥.∥ζ in ( 17 ) , we conclude ∥s∥2ζ ≤ ∥s∥2K { g } ( ∥g1∥ 2 H∞ + d∑ k=1 ∥gk2∥2H∞ ) ≤ ( µ0 + ( d+ 1 ) µ1 ) ∞∑ p=1 γpa 2 p∥wp∥ 2p 2 . ( 32 ) Note that if the odd exponents ( except possibly one ) in the definition of s in ( 26 ) are zero , then we could consider only the function g1 and kernel Kg1 , which would have implied a bound of µ0 ∑∞ p=1 γpa 2 p∥wp∥ 2p 2 . A.2 THE DOUBLING TRICK For the SGD optimization , we set the regularization coefficients in the loss L1 as ψ1 = ν/4 , ψ2 = ν/ ( 4ζ ( f ∗ , G ) ) , ( 33 ) with ν : = max { Rn ( f∗ ) /2 , B2/n } . This assumes we know the f∗ and G that minimize the adaptation within the complexity measure ( 12 ) . Here , we explain how to use a simple doubling trick to get over the fact that we might not know these optimal solutions f∗ and G. Theorem 1 Without explicitly knowing the exact value of the complexity measure , i.e. , the optimal solution of Equation ( 12 ) , one can still achieve the generalization bound in Theorem 1 . Proof of Theorem 1 Our core generalization result is in Theorem 3 . The proof of Theorem 1 is simply adding a doubling trick on top of the argument of Theorem 3 . We also prove Theorem 2 as a consequence of Theorem 1 in Appendix 2 . In the rest of the proofs , for simplicity , we refer to ( WPSGD , VPSGD ) by ( W ′ , V ′ ) . Let f∗∗ , G∗ be the optimal solution to ( 10 ) . With a simple rescaling of G∗ , we can assume ⟨H∞−1 , G∗⟩ = 1 . ( Note that the complexity does not change by such rescaling ) . Now one can exploit the condition ∥yi∥∞ ≤ B , and consider the setting f∗ = 0 to get the following trivial upper bound on the complexity measure : ℑ ( ( xi ) , ( yi ) ) ≤ 2nB2 . Therefore , 2nRn ( f ∗∗ ) + f∗∗A−1f∗∗ ( c′ϖ ) ≤ 2nB2 . ( 34 ) Using Equation ( 34 ) and the optimality of ( f∗∗ , G∗ ) : Rn ( f ∗∗ ) ≤ B2 , ζ = ζ ( f∗∗ , G∗ ) ≤ 2nB2 . ( 35 ) Combining the first equation above with the definition of ν in Equation ( 43 ) , we get B2/n ≤ ν ≤ B2 . ( 36 ) To initialize ψ1 and ψ2 , we use Equations ( 33 ) for any f∗ and G , and as a result we get a generalization bound as in Equation ( 47 ) . However , to achieve the best possible rate characterized by our complexity measure in Theorem 1 without explicitly computing the answer of ( 12 ) , we use a simple doubling trick ; for every pair ( ζ ′ , ν′ ) such that ζ ′ is a power of 2 between B2 and 2nB2 , and ν′ is a power of two between B2/n and B2 , we initialize ψ1 , ψ2 as in Equation ( 33 ) and run the algorithm , then return the network that minimizes the upper bound of the risk given by Equation ( 46 ) . Note that we do not need to know Rn ( f∗∗ ) to compute this upper bound . Now let ν′ be the power of 2 within ν ( G∗ , f∗∗ ) /2 ≤ ν′ < ν ( G∗ , f∗∗ ) . If we are in the case f∗∗TA−1f∗∗ < B2 , ( 37 ) then for ζ ′ equal to the smallest power of two larger than B2 , when we run PSGD with pair ( ν′ , ζ ′ ) , by Theorem 3 : R ( fW ′ , V ′ ) ≤ 2Rn ( f∗ ) + c′′ϖ 2B2 +B2 n ≤ ℑ ( ( xi ) n i=1 , ( yi ) n i=1 ) n + c′ B2ϖ n . ( 38 ) Because we return the minimum upper bound on the risk ( the tighter lower bound of Equation ( 46 ) ) among all such powers of two , we certainly achieve the above rate in ( 38 ) . Otherwise , if f∗∗TA−1f∗∗ ≥ B2 , let ζ ′ be the power of two within f∗TA−1f∗ ≤ ζ ′ ≤ 2f∗TA−1f∗ , then again it is easy to check that conditions of Theorem 3 are satisfied , hence we get the following generalization bound : R ( fW ′ , V ′ ) ≤ 2Rn ( f∗∗ ) + c′′ϖ ( ζ ′ +B2 ) n ≤ 2Rn ( f∗∗ ) + c′′ϖ 2ζ ( f∗∗ , G∗ ) +B2 n ( 39 ) ≤ ℑ ( ( xi ) n i=1 , ( yi ) n i=1 ) n + c′ B2ϖ n , ( 40 ) which proves the bound of Theorem 1 . A.3 AMOUNT OF OVERPARAMETERIZATION In this section , to provide high-level insight , we indicate the right order of magnitude that our overparameterization should be in , with respect to one another . Note that the exact coefficients in these inequalities would depend on the basic parameters B , 1/λ0 , n , which we have avoided here for sake of simplicity . We refer the reader to our main proof ( mostly Appendix parts A.12 , A.13 ) for more details . κ1κ2m3 < < 1 , κ2 √ m2 > > 1 , κ1 √ m3 > > κ2 √ m2 , m1 > > m 4 3 , κ1 √ m1 > > m 3/2 3 , κ2 < < 1/ √ m3√ m3κ2 < < 1/ √ m3 √ m2 > > m 3/2 3 κ1κ2 m33 ( κ2m2 ) < < κ1m1 m1 , m2 , m3 , 1/κ1 , 1/κ2 = poly ( n , B ∨ 1/B , 1/λ0 ) . In addition , we set the smoothing parameters as β2 : = Θp ( ( κ1 √ m3 ) −1 ( √ m2κ2 ) − 23 ) , β1 : = Θp ( m3 √ m3/ ( κ1 √ m1 ) ) , where Θp only shows polynomial dependencies on the overparameterization . A.4 PSD PROPERTY OF K∞ The Schur product theorem states that for PSD matrices A and B , A ⊙ B is also PSD . Now given an analytic function F whose tailor series coefficients are all nonnegative , Suppose we apply F on some PSD matrix A entrywise , denoted by F ( A ) , under the condition that the entries of A are in the radius of convergence of F , then using Schur product theorem , it is straightforward that F ( A ) is also PSD . Using the above property , one can then check that the tailor series of the defined functions F2 and F3 are nonnegative , hence , the application of the function F2 ( 2F3 ( x ) ) on the gram matrix of ( xi ) ni=1 is a PSD matrix , ( note that ∣∣∣⟨xi , xj⟩∣∣∣ ≤ 1 is in the convergence radius of F2 ( 2F3 ( x ) ) . ) thus K∞ is indeed a kernel . A.5 COMPLEXITY UPPER BOUND First we mention a simple fact that hadamard product respects matrix orderings . Given PSD matrices A , B , C such thatA ⪯ B , the fact thatA⊙C ⪯ B⊙C is an easy consequence of the Schur Product Theorem ; indeed , B −A is PSD by definition , so ( B −A ) ⊙ C = B ⊙ C −A⊙ C is also PSD . Next , it is easy to check that the tailor series of arcsin ( x ) has all nonnegative coefficients . Therefore , for a PSD matrix X , as we discussed in Appendix A.4 , applying arcsin entrywise on X , namely arcsinX , is also PSD . Setting X equal to the entrywise application of 2F3 to the gram matrix of datapoints ( xi ) ni=1 , we realize the matrix arcsin ( 2F3 ( ( ⟨xi , xj⟩ ) 1≤i , j≤n ) ) is also PSD . Noting the definition of K∞ in Equation ( 8 ) , we conclude that for the data kernel matrix K∞ we have K∞ ≥ 1 4 11 T , where 1 is the all ones n-dimensional vector . Combining the two mentioned facts , we can lower bound the matrix K = K∞ ⊙G for any matrix G as K = K∞ ⊙G ≥ 1 4 11 T ⊙G = 1 4 G. Substituting the rank one matrix f∗f∗T for the n-dimensional vector f∗ in Equation ( 13 ) : K−1 { f̃∗/∥f̃∗∥ } = K ∞ ⊙ f∗f∗T /∥f̃∗∥2 ≥ 1 4 f∗f∗T /∥f̃∗∥2 . ( 41 ) The inequality used in ( 13 ) then follows from Equation ( 41 ) . A.6 COMPLEXITY MEASURE AND THE ζ-NORM This is a brief section regarding some basic properties of ℑ and ∥.∥ζ . First , note that the two versions of the complexity measure in Equations ( 10 ) and ( 11 ) are equivalent , as for any finite set of functions { g } , we can define the gram matrix with respect to the feature vectors of these functions on data , and for an arbitrary nonzero PSD G we can consider a Cholesky factorization for G as G = X̄T X̄ , then define the functions { gk } as the minimum-NTK norm functions which map the input to the features corresponding to X̄ . This observation further implies we can suppose the factor matrix X̄ is in Rn×n , and there is a set of at most n functions { gk } nk=1 which corresponds to this G. Next , we show that for an arbitrary function f , its sup norm over the unit ball is bounded by its ζ norm : sup ∥x∥=1 |f ( x ) | ≤ ∥f∥ζ . ( 42 ) Note that for a kernel K which satisfies K ( x , x ) ≤ 1 , using Cauchy Schwarz we simply obtain f ( x ) ≤ ∥f∥K √ K ( x , x ) ≤ ∥f∥K , where recall that ∥.∥K is the norm corresponding to the RKHS space of K. Hence , to show ( 42 ) , it suffice to show that for all kernels K ∈ K and unit norm x we have K ( x , x ) ≤ 1 . To see this fact , note that the norm of each x ∈ Rd in the NTK-space isH∞ ( x , x ) = 12 . Therefore , for each function g with bounded-NTK norm , again using Cauchy Schwarz : |g ( x ) | ≤ 1 2 ∥g∥H∞ . As a result , for a family of functions { g } with ∑ g∈ { g } ∥g∥2H∞ ≤ 1 , we have on every unit norm x : ∑ g∈ { g } g ( x ) 2 ≤ 1 . On the other hand , it is easy to check that for every unit norm x , we have K∞ ( x , x ) ≤ 12 , so for every such { g } , we have by definition K { g } ( x ) ≤ 1 , which completes the proof of Equation ( 42 ) . A.7 CORE GENERALIZATION RESULT In this section , we prove our core generalization result for the trained network , Theorem 3 , which underlies our generalization bounds in Theorems 1 and 2 . Recall that in the rest of the proofs , we refer to the solution ( WPSGD , VPSGD ) returned by PSGD simply by ( W ′ , V ′ ) . Theorem 3 Suppose we have a candidate pair ( f∗ , G ) regarding our complexity measure in ( 10 ) that satisfies ⟨H∞−1 , G⟩ ≤ 1 , f∗TA−1f∗ ≤ ζ ( recall A = G⊙H∞ ) . Then , for ν = max { Rn ( f∗ ) /4 , B2/n } , ( 43 ) if we are given ν/2 ≤ ν′ ≤ ν , and we set ψ1 = ν′ 4 , ( 44 ) ψ2 = ν′ 4ζ , ( 45 ) then for the solution ( W ′ , V ′ ) returned by PSGD we have the following generalization bound : R ( fW ′ , V ′ ) ≤ 5 4 Rn ( fW ′ , V ′ ) + c ′′′ϖ ( ζ +B2 ) n ( 46 ) ≤ 5 3 Rn ( f ∗ ) + c′′ϖ ( ζ +B2 ) n , ( 47 ) for constants c′′ , c′′′ and log factor ϖ = log ( n ) 3 + log ( 1/λ0 ) . Proof of Theorem 3 Almost all of our proofs in the rest are in the aim of proving Theorem 3 . Crucially , to prove this Theorem , we need to establish two big results : 1 . We need to show that the final network has small training loss , and is within the class Gγ1 , γ2 for some suitable γ1 , γ2 . This is handled by Theorem 4 in Appendix A.10 . We define the class Gγ1 , γ2 roughly as the class of networks with norm bounds ∥W −W ( 0 ) ∥ ≤ γ1 , ∥V −V ( 0 ) ∥ ≤ γ2 where the rows of V −V ( 0 ) are orthogonal to the subspace Φ , plus an additional structure defined in Appendix A.11 . This task , on its own , has three main steps in our proof : ( a ) we construct a “ good ” underlying network , Appendix A.12 ( b ) we find a “ good ” random direction and study the landscape of the objective , Ap- pendix A.13 ( c ) we prove the convergence , Appendix A.16 2 . The Rademacher Complexity of the class Gγ1 , γ2 needs to be suitably bounded . This is handled by Theorem 5 in Appendix A.11 . With access to these results , here we show how Theorem 3 follows by a simple application of the generalization bound in ( Srebro et al. , 2010 ) . Specifically , for fixed constants z1 , z3 and every integer i ≥ 0 , we use Theorem 1 of ( Srebro et al. , 2010 ) for the class Gz1 , γi , γi = 2i×B/z3 with confidence probability 1 − 2−iδ3 , which , with a union bound , implies that with probability at least 1 − δ3 , for every i and fW ′ , V ′ ∈ Gz1 , γi : R ( fW ′ , V ′ ) ≤ Rn ( fW ′ , V ′ ) +K ( √ Rn ( fW ′ , V ′ ) ( √ log ( n ) 1.5R ( Gz1 , γi ) + √ b log ( 1/ ( 2−iδ3 ) ) n ) ( 48 ) + log ( n ) 3R ( Gz1 , γi ) 2 + b log ( 1/ ( 2−iδ3 ) ) n ) , ( 49 ) where ℓ ( fW ′ , V ′ ( x ) , y ) is a.s. bounded by b for function within the class Gz1 , γi , and K is a universal constant . In the following , we aim to further bound the Rademacher complexityR and parameter b . Applying the AM-GM inequality with respect to ratio z4 > 0 for the second term : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) +K2/z4 ( log ( n ) 1.5R ( Gz1 , γi ) + √ b log ( 1/ ( 2−iδ3 ) ) n ) 2 + log ( n ) 3R ( Gz1 , γi ) 2 + b log ( 1/ ( 2−iδ3 ) ) n ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) +K2/z4 ( log ( n ) 1.5R ( Gz1 , γi ) + √ b log ( 1/ ( 2−iδ3 ) ) n ) 2 + log ( n ) 3R ( Gz1 , γi ) 2 + b log ( 1/ ( 2−iδ3 ) ) n ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) log ( n ) 3R ( Gz1 , γi ) 2 + ( 2K2/z4 + 1 ) b log ( 1/ ( 2−iδ3 ) ) n . ( 50 ) Now let γ∗ be the smallest number of the form 2iB/z3 ( for some i ) which is not smaller than z2 √ ζ . This definition implies γ∗ ≤ max { 2z2 √ ζ , B/z3 } . ( 51 ) Now Theorem 5 in Appendix A.11 bounds the Rademacher complexity : R ( Gz1 , γ∗ ) ≤ 2z1γ ∗ √ n . ( 52 ) On the other hand , from Theorem 4 by setting z1 , z2 = √ 40 , we get fW ′ , V ′ ∈ Gz1 , γ∗ . Moreover , from Lemma 34 , for fW ′ , V ′ ∈ Gz1 , γ∗ , we have for every ∥x∥ ≤ 1 : |fW ′ , V ′ ( x ) | ≤ 2z1γ∗ , ( 53 ) so the loss ℓ ( fW ′ , V ′ ( x ) , y ) can be bounded by ( B + 2z1γ∗ ) 2 using the 1 smoothness property . Therefore , for the class Gz1 , γ∗ we can set b = ( B + 2z1γ∗ ) 2 . Combining this with Equation ( 52 ) and plugging into Equation ( 50 ) : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) log ( n ) 3 4z21γ ∗2 n + ( 2K2/z4 + 1 ) ( B + 2z1γ ∗ ) 2 log ( 1/ ( 2−i ∗ δ3 ) ) n . Furthermore , by definition of γ∗ , we have 2i ≤ 2z2z3 √ ζ/B : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) 4z21 ( log ( n ) 3 + 2 log ( 2z2z3 √ ζ/B ) ) 4z21γ ∗2 n ( 54 ) + ( 2K2/z4 + 1 ) 2B2 log ( 2z2z3 √ ζ/B ) n . ( 55 ) Now applying the upper bound on γ∗ : R ( fW ′ , V ′ ) ≤ ( 1 + z4 ) Rn ( fW ′ , V ′ ) + ( 2K2/z4 + 1 ) 4z21 ( log ( n ) 3 + 2 log ( 2z2z3 √ ζ/B ) ) 4z21 ( 2z2 √ ζ + 2B/z3 ) 2 n ( 56 ) + ( 2K2/z4 + 1 ) 2B2 log ( 2z2z3 √ ζ/B ) n . ( 57 ) If ζ > B2 , in the third term above we substitute B by √ ζ . Finally , similar to the bound we stated in Equation ( 15 ) , note that we have the following trivial bound for ζ : ζ ≤ yTH∞−1y ≤ 4nB2/λ0 , ( 58 ) i.e . there is no point in considering larger ζ ’ s , which implies log ( 2z2z3 √ ζ/B ) = O ( log ( n ) + log ( 1/λ0 ) ) . Plugging this above and picking z4 = 1/3 show the proof of Equation ( 46 ) . Furthermore , applying Equation ( 70 ) in Theorem 4 to the Rn ( fW ′ , V ′ ) term in Equation ( 46 ) further gives the second Equation ( 47 ) . Remark 1 In the same setting of Theorem 3 , if we have ν/2 ≤ ν′ but not generally upper bounded by ν , then PSGD leads to the following generalization bound : R ( fW ′ , V ′ ) ≤ Rn ( f∗ ) + ν′ + c′′′′ϖ ( ζ +B2 ) n , using a similar argument as we did for Theorem 3 . A.8 STRUCTURE OF THE PROOF , SETTING m3 , AND FURTHER DEFINITIONS Throughout the proof , ( W ′ , V ′ ) represents the pair of matrices of the current iteration of PSGD , ( W ∗ , V ∗ ) are the “ ideal ” matrices that we construct in Appendix A.12 , ( W ρ , V ρ ) and refers to the gaussian smoothing matrices . Importantly , note that our squared loss ℓ ( f , y ) is zero at f = y . We have tried to make the lower level proofs into sub-lemmas and create a manageable hierarchy as much as we could , to make the document more clear and readable . Similar to the conditions in Theorem 3 , through out most of the proofs we assume that we are given a pair ( f∗ , G ) with a slightly more general setting of Theorem 3 : f∗TA−1f∗ ≤ ζ2 , for A = G⊙K∞ , ⟨G , H∞⟩ ≤ ζ1 . Particularly , ζ1 , ζ2 appear in Appendix A.13 . Because we are allowed to rescale G , we do not really gain much by assuming this more general setting , though we pick to work with the general setting as the abstraction makes the proof more straightforward to understand . We refer to the parameters B , 1/λ0 , n as the “ basic parameters ” , m1 , m2 , m3 , 1/κ1 , κ2 as the “ overparameterization ” , and β1 , β2 as the “ smoothing parameters. ” By the phrase “ having enough overparameterization ” we mean it suffices to pick the overparameterization m1 , m2 , m3 , 1/κ1 , 1/κ2 only polynomially large in the basic parameters . Throughout the proof , we denote the change in the output of the first layer at W ( 0 ) +W ′ +W ρ compared to the initialization value by ϕ ( 2 ) ( xi ) , i.e . ϕ ( 2 ) ( xi ) = 1√ m1 W sσ ( ( W ( 0 ) +W ′ +W ρ ) xi ) − ϕ ( 0 ) ( xi ) , while recall that ϕ′ ( xi ) has a similar definition except without the smoothing matrix W ρ . Although our model is a three layer network , throughout the proof , we refer to the parts W s 1√ m1 σ ( ( W ( 0 ) + W ′ ) x ) and 1√ m2 aTσ ( ( V ( 0 ) + V ′ ) ( . ) ) as the “ first layer ” and “ second layer , ” respectively . Also , we sometimes refer to the binary sign pattern of vector x multiplied to matrix W by DW , x ( DW , x : = Sgn ( Wx ) ) , i.e . the jth diagonal entry of DW , x is one if WTj , x ≥ 0 , and is zero otherwise . To refer to the jth row of W as a vector , we sometimes drop the comma in Wj , and write it as Wj . For brevity , we denote the Frobenius norm ∥W∥F of matrix by ∥W∥ , and the Euclidean norm of a vector x by ∥x∥ . For matrices W1 , W2 we denote their natural dot product by ⟨W1 , W2⟩ : = tr ( WT1 W2 ) . We refer to the smallest eigenvalue of a matrix by λmin ( . ) . We write R ( . ) for the Rademacher complexity of a function class . We refer to the smoothed version of the network by f ′W ′ , V ′ ( x ) , defined by f ′W ′ , V ′ ( x ) = EWρ , V ρfW ′+Wρ , V ′+V ρ ( x ) . In the proof , we mainly work with the loss over the smoothed network f ′ , defined as L ( W ′ , V ′ ) = Rn ( f ′ W ′ , V ′ ) + ψ1∥W ′∥2 + ψ2∥V ′∥2 . ( 59 ) Our algorithm , PSGD can be regarded roughly as an SGD over L. Similar to what we discussed in section 3 , let the functions { gk } m3k=1 be some feature representation whose gram matrix is equal to G and ⟨H∞ , G⟩ = ∑m3 k=1 ∥gk∥2H∞ . In such setting , it is not hard to check that we can assume each gk is the minimum norm NTK function which maps ( xi ) ni=1 to ( gk ( xi ) ) n i=1 ’ s . Indeed , if this is not the case for some gk , we can project the RKHS representation of gk onto the span of the representations of ( xi ) ni=1 , which can only decrease the complexity measure . Hence we can represent gk ∈ HH∞ as a linear combination of basic functions H∞ ( xi , . ) on data points : ∀k ∈ [ m3 ] , gk ( x ) : = n∑ i=1 Vk , iH∞ ( xi , x ) . ( 60 ) Here , the sum of squared-H∞ norms of Vk is bounded as∑ k ∥Vk∥2H∞ = ∑ k ∥gk∥2H∞ = ⟨H∞ , G⟩ ≤ ζ1 . ( 61 ) For each i ∈ [ n ] , we refer to the feature representation vector ( gk ( xi ) ) m3k=1 on xi as x̄i . Note that we have the relationship x̄i = ( H ∞ i , Vk ) m3 k=1 , ( 62 ) whereH∞i , is the ith row ofH ∞ . In the analysis , we work with a bound ξ on the quantity maxk ∥Vk∥ which should be bounded polynomially by other basic parameters ; in particular , it is defined in Lemma 10 and is used to bound a cross term in Lemma 14 . However , maxk ∥Vk∥ might not be effectively bounded for an arbitrary feature representation . Fortunately , we can remedy this by a simple trick ; for every natural number s , one can substitute every gk by s copies of gk/ √ s , without changing the gram matrix G. Therefore , for any δ , one can increase the multiset of functions ( gk ) to a bigger set ( g̃k ) , by adding at most O ( ζ1/δ ) functions , making sure of the following for the new functions : ∀k : VTk H∞Vk = ∥g̃k∥2H∞ ≤ δ . ( 63 ) ( This is because ∑ k ∥g̃k∥2H∞ ≤ 1 ) . Furthermore , observe that for each gram matrix G , we have an n-dimensional feature representation ( gk ) nk=1 for G according to the Cholesky factorization . Combining these facts , we conclude , to guarantee Equation ( 63 ) , in the worst case , we need m3 to be as large as n+O ( ζ1/δ ) . Finally , observing the following inequality ∥Vk∥2 ≤ ∥Vk∥2H∞/λ0 ≤ ∥g̃k∥2H∞/λ0 . ( 64 ) in order to guarantee maxk ∥Vk∥ ≤ ξ we need to take m3 as large as n + O ( ζ1/ ( ξ2λ0 ) ) , which is indeed bounded polynomially by the basic parameters because of the same condition for 1/ξ . This computation also brings into sight an important point : “ Although each gram matrix G is representable by n features , in order for the algorithm to be able to find a suitable network , m3 might need to be larger than n. ” Moreover , for every 1 ≤ k , i ≤ n , we define the matrices Zki ∈ Rmd as Zik = 1/ √ m1 ( W sk , j1 { W ( 0 ) j Txi } xi ) m1 j=1 , ( 65 ) where in the above notation , j is enumerating the columns of the matrix . We also define the following matrices which we use in our construction later : W+k T =W k+ T = n∑ i=1 Vk , iZik , ( 66 ) and W+ as W+ = m3∑ k=1 W k+ . Finally , to avoid unnecessary complication , we often argue high probability bounds without an explicit representation of their dependency on the chance of failure ( which is a negligible logarithmic factor ) . We also ignore all constants and log factors , and mainly work with the notation ≲ which ignores constants ; we write a ≲ b ± c as a short form for b − O ( c ) ≤ a ≤ b + O ( c ) . As there are several hierarchies of new parameters that are defined based on lower-level ones , we rename the new parameters and continue viewing them as black-box . This makes the proofs more readable , since we also do not care about the exact dependency of the underlying parameters most of the time , rather we are interested in their orders of magnitude , for example that a given parameter goes to zero polynomially fast with respect to the overparameterization , etc . Due to the large number of symbols that we have to work with , we might use a symbol more than once , of course when it is clear from the context which one we are refering to . A.9 PROOF OF THEOREM 2 In this section , we prove Theorem 2 , stated below . Theorem 2 For any function f : Rd → R , in the same setting as Theorem 1 , the population risk of the trained network ( W ′ , V ′ ) can be bounded as R ( fW ′ , V ′ ) ≤ 2R ( f ) +O ( αϖ ∥f∥2ζ +B2 n ) . ( 67 ) Proof of Theorem 2 Theorem 2 is a simple consequence of Theorem 1 ; for the given function f , we apply Theorem 2 with the smaller coefficient γ = 43 for Rn ( f ∗ ) , regarding the complexity upper bound , by setting f∗ : = ( f ( xi ) ) ni=1 : R ( fW ′ , V ′ ) ≤ 4 3 Rn ( f ∗ ) + ( αϖ ) min K∈K f∗TK−1f∗ n + B2αϖ n = 4 3 Rn ( f ) + ( αϖ ) min K∈K f∗TK−1f∗ n + B2αϖ n . On the other hand , because f∗TK−1f∗ is the minimum-RKHS norm of a function with respect to kernel K which maps xi ’ s to f∗i and f is one such function , we have f ∗TK−1f∗ ≤ ∥f∥K . This inequality implies min K∈K f∗TK−1f∗ ≤ ∥f∥ζ , so we obtain R ( fW ′ , V ′ ) ≤ 4 3 Rn ( f ) + ( αϖ ) ∥f∥2ζ +B2 n . ( 68 ) Therefore , it remains to bound Rn ( f ) by R ( f ) . As we showed in Appendix A.6 , for every input x we have f ( x ) ≤ ∥f∥ζ , so for every data ( x , y ) , by the fact that |y| ≤ B a.s. and α smoothness of the loss , we have ℓ ( f ( x ) , y ) ≤ α ( ∥f∥ζ + B ) 2 . Moreover , note that the random variable ℓ ( f ( x ) , y ) has mean R ( f ) . It is easy to check that in this setting , the variance of ℓ ( f ( x ) , y ) is at most R ( f ) α ( ∥f∥ζ + B ) 2 . Therefore , an application of the Bernstein inequality , we have with high probability over the dataset Rn ( f ) ≤ R ( f ) +O ( √ R ( f ) α ( ∥f∥ζ +B ) 2 n + α ( ∥f∥ζ +B ) 2 n ) ≤ 3 2 R ( f ) +O ( α ( ∥f∥ζ +B ) 2 n ) . Plugging this back to Equation ( 68 ) completes the proof . As a result , the learned network can compete with any function that has reasonably small ∥f∥ζ : R ( fW ′ , V ′ ) ≤ min f { 2R ( f ) +O ( αϖ ∥f∥2ζ +B2 n ) } . A.10 OPTIMIZATION In this section , we glue together • the existence of a good random direction that we prove in Appendix A.13 • the convergence analysis of PSGD that we do based on the work Ge et al . ( 2015b ) in Appendix A.16 . Theorem 4 In the same setting of Theorem 3 , assume the network ( W ′ , V ′ ) returned by PSGD , has sufficient polynomially large “ overparameterization ” . Then , for the solution ( W ′ , V ′ ) returned by PSGD we have L ( W ′ , V ′ ) ≤ Rn ( f∗ ) + ν , ( 69 ) which further implies Rn ( fW ′ , V ′ ) ≤ Rn ( f∗ ) + 2ν , ( 70 ) ∥W ′∥2 ≤ 40 , ∥V ′∥2 ≤ 40ζ . ( 71 ) Moreover , for every i ∈ [ n ] , j ∈ [ m1 ] , j /∈ P for P defined in 1 , we have that sign ( ( W ( 0 ) j +W ′j ) Txi ) and sign ( W ( 0 ) j T xi ) are the same . Proof of Theorem 4 Let Υ ∈ Rm2 ( m3−n ) ×m2m3 be a matrix whose rows are an orthonormal basis for the space of matrices whose rows are orthogonal to span ( { ϕ ( 0 ) ( xi ) } ni=1 ) , i.e . Φ⊥ , as defined in ( 24 ) . Then , we consider a linear change of coordinates for the subspace Φ⊥ , regarding the second layer weights , as v′ = Υvec ( V ′ ) where vec ( . ) splits out the vectorized version of a matrix . For consistent notation , we also denote w′ = W ′ , so we now have a new coordinate system ( w′ , v′ ) ∈ Rm2 ( m3−n ) ×m1d for pairs of weights ( W ′ , V ′ ) such that V ′ ∈ Φ⊥ . We also define the loss function LΠ ( w : = ( w′ , v′ ) ) = L ( W ′ , V ′ ) , with respect to the change of coordinate . Now it is easy to see that running PSGD on L in the normal coordinates is equivalent to running stochastic gradient descent on LΠ with respect to ( w′ , v′ ) . Moreover , because multiplying to matrix Υ is an orthonormal change of coordinates for Φ⊥ and because V ′ is already in ϕ⊥ at each step of PSGD , then ∥v′∥ = ∥V ′∥ , so the conditions ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 are equivalent to ∥w′∥ ≤ C1 , ∥v′∥ ≤ C2 . Furthermore , by our construction , the random matrix V ∗Σ is in the subspace Φ⊥ , so the norm bounds ∥W ∗∥ ≤ ζ1 , ∥V ∗∥ ≤ ζ2 are equivalent to ∥w∗∥ ≤ ζ1 , ∥v∗∥ ≤ ζ2 for w∗ : = W ∗Σ and v∗ : = Υvec ( V ∗ ) . Now we apply the result of Theorem 6 on LΠ with parameter ν set as ν′ , ζ2 : = ζ and ζ1 : = 1 , and ∆ : = Rn ( f ∗ ) , as defined in Theorem 3 . More specifically , based on our arguments above regarding the natural isometry in the change of coordinate , any pair ( w′ , v′ ) in the domain ∥w′∥ ≤ C1 , ∥v′∥ ≤ C2 , LΠ ( w′ , v′ ) ≥ Rn ( f∗ ) + ν′ translates into a pair ( W ′ , V ′ ) in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , L ( W ′ , V ′ ) ≥ Rn ( f∗ ) + ν′ , for which by Theorem 6 there exists ( W ∗Σ , V ∗Σ ) such that EΣL ( W ′ − η/2W ′ + √ ηW ∗Σ , V ′ − η/2V ′ +√ηV ∗Σ ) ≤ L ( W ′ , V ′ ) − ην′/4 . ( 72 ) Translating back to the change of coordinates : EΣLΠ ( w′ − η/2w′ + √ ηw∗Σ , v ′ − η/2v′ +√ηv∗Σ ) ≤ L ( w′ , v′ ) − ην′/4 . ( 73 ) Now we apply Lemma 44 to translate this into an argument about the landscape of LΠ . As a result , applying the bounds in Equations ( 107 ) and ( 127 ) , we obtain that for ( w′ , v′ ) such that LΠ ( w′ , v′ ) ≥ Rn ( f∗ ) + ν′ , we should either have ∥∇LΠ ( w′ , v′ ) ∥ ≥ ν/4 4 √ ∥w′∥2 + ∥v′∥2 = ν/4 4 √ ∥W ′∥2 + ∥V ′∥2 = ν 16 √ C21 + C 2 2 , or λmin ( ∇2LΠ ( w′ , v′ ) ) ≤ − ν/4 2minΣ ( ∥w∗∥2 + ∥v∗∥2 ) = − ν 2minΣ ( ∥W ∗Σ∥2 + ∥V ∗Σ∥2 ) ≤ − ν 16 ( ζ1 + ζ2 ) = − ν 16 ( 1 + ζ ) . Next , we want to apply Theorem 7 by setting γ = ν 16 ( 1 + ζ ) , ℵℓ = Rn ( f∗ ) + ν′ , and lipschitz parameters ρ1 , ρ2 , ρ3 = poly ( B , C1 , C2 , m1 , m2 , m3 ) set as described in Appendix B.1 , Theorem 9 . Also , note that as prescribed by Theorem 7 , we set C1 : = ℵ+ 4l ψ1 , C2 : = ℵ+ 4l ψ2 , ( 74 ) where l = O ( 1 ) depends on our desired chance of success for the algorithm , specified in Theorem 7 . Finally , note that Theorem 7 needs to work with a bounded noise on the gradient whose covariance matrix is bounded between two multipliers of identity . The point of injecting extra noise to SGD in PSGD is in fact because of this covariance condition that we need in Theorem 7 . On the other hand , note that in general , because of the gaussian smoothing that we use , the noise vector is not supported on a bounded domain , which makes it a bit harder to apply Hoeffding type concentration . To remedy this , we introduce a coupling between our unbounded noise vector for L ( W ′ , V ′ ) and another noise random variable whose support is bounded , which with high probability is equal to the real noise , along all iterations . In Corollary , we further translate this coupling for the objective LΠ after change of cooridnates , and write down the exact dependencies of the parameters Q , σ1 and σ2 , which are all polynomial in the basic parameters and the overparameterization . Hence , the conditions of Theorem 7 are satisfied , so we conclude that after at most poly ( ρ1 , ρ2 , ρ3 , Q , ℵ , C1 , C2 , 1/γ , log ( σ1/σ2 ) ) = poly ( B , m1 , m2 , m3 , C1 , C2 , ζ1 , ζ2 ) = poly ( n , B ∨ 1/B , 1/γ0 ) number of iterations , PSGD reach a point wt in some iteration t with LΠ ( wt ) ≤ ℵℓ . Translating back this wt = ( w′t , v ′ t ) by multiplying the v ′ t part to Υ T , we get a pair ( W ′t , V ′ t ) with objective value bounded as L ( W ′t , V ′ t ) ≤ Rn ( f∗ ) + ν′ . But note that we obviously have the condition ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 through the whole iterations , for the choice of C1 , C2 in Equation ( 74 ) . Therefore , using Lemma 34 , for every i ∈ [ n ] : |f ′W ′ , V ′ ( xi ) | = O ( C1 , C2 ) , ( 75 ) |fW ′ , V ′ ( xi ) | = O ( C1 , C2 ) ( 76 ) From Equations ( 76 ) , as also stated in Theorem 9 , we know that for all i ∈ [ n ] , ℓ ( . , yi ) is O ( C1C2 ) + B 2-Lipschitz at points fW ′ , V ′ ( xi ) and f ′W ′ , V ′ ( xi ) , so we can bound the difference |ℓ ( f ′W ′ , V ′ ( xi ) , yi ) − ℓ ( fW ′ , V ′ ( xi ) , yi ) | by ( O ( C1C2 ) +B2 ) |f ′W ′ , V ′ ( xi ) , yi ) − fW ′ , V ′ ( xi ) | , which in turn can become arbitrarily small having enough overparameterization using Lemma 35 , in particular , we force it to be smaller than O ( ν′/ ( B2 + C1C2 ) ) ( recall ν′ ≥ ν/2 ≥ B2/ ( 2n ) ) . As a result , we get |ℓ ( f ′W ′ , V ′ ( xi ) , yi ) − ℓ ( fW ′ , V ′ ( xi ) , yi ) | = O ( ν ) for every i ∈ [ n ] , which in turn implies |L ( W ′ , V ′ ) −L1 ( W ′ , V ′ ) | ≤ ν by picking small constants , where recall that the objective L1 is the same as L but without the smoothing . Now applying this bound to Equation ( 75 ) , we get L1 ( W ′ t , V ′ t ) ≤ Rn ( f∗ ) + 2ν′ . Therefore , as PSGD check the values of L1 in the loop , it terminates at such pair ( Wt , Vt ) . From this point onward , we refer to the returned ( W ′t , V ′ t ) as just ( W ′ , V ′ ) . Opening the definition of L1 ( W ′ , V ′ ) , we clearly get Rn ( fW ′ , V ′ ) ≤ L1 ( W ′ , V ′ ) ≤ Rn ( f∗ ) + 2ν′ ≤ Rn ( f∗ ) + 2ν . ( 77 ) Furthermore , noting the setting of ψ1 , ψ2 in Theorem 3 and the fact that ν′ ≥ ν/2 ≥ Rn ( f∗ ) /8 , we get ∥W ′∥2 ≤ 4 ( Rn ( f ∗ ) + 2ν′ ) ν′ ≤ 40 , ( 78 ) ∥V ′∥2 ≤ 4ζ ( Rn ( f ∗ ) + 2ν′ ) ν′ ≤ 40ζ , ( 79 ) which completes the proof . The fact that for every i ∈ [ n ] , j ∈ [ m1 ] , j /∈ P we have that sign ( ( W ( 0 ) j +W ′ j ) Txi ) and sign ( W ( 0 ) j T xi ) are the same follows from Lemma 1 . A.11 RADEMACHER COMPLEXITY In this section we show the proof for our Rademacher Complexity bound , which is used in Theorem 3 . Theorem 5 Let Gγ1 , γ2 be the class of neural nets with weights ( W , V ) in our three layer setting , such that ∥W − W ( 0 ) ∥ ≤ γ1 , ∥V − V ( 0 ) ∥ ≤ γ2 , where for every j ∈ [ m2 ] , i ∈ [ n ] : V ′j ⊥ ϕ ( 0 ) ( xi ) , and for every i ∈ [ n ] , j ∈ [ m1 ] , j /∈ P for P ⊆ [ m1 ] defined in Lemma 1 , it satisfies sign ( ( W ( 0 ) j +W ′ j ) xi ) = sign ( W ( 0 ) xi ) . Then , for large enough overparameterization , we have the following bound on the Rademacher complexity : R ( Gγ1 , γ2 ) ≤ 2γ1γ2√ n . Proof of Theorem 5 Here , we do not have the smoothing matrices W ρ , V ρ anymore . In this section , unlike the optimization section that we used { x′i } ni=1 to denote the output of the first layer by incorporating also the smoothing matrices , here we define it without them : x′i = 1√ m1 W sσ ( ( W ( 0 ) +W ′ ) xi ) . Now define the matrices Z ′i = 1/ √ m2 ( aj1 { V ( 0 ) j , x ′ i ≥ 0 } x′i ) m2 j=1 , Z ′i + = 1/ √ m2 ( aj ( 1 { Vj , x′i ≥ 0 } − 1 { V ( 0 ) j , x ′ i ≥ 0 } ) x′i ) m2 j=1 . To bound the Z ′i + part , note that substituting C1 by γ1 in lemma 6 and assuming conditions m1 = Ω̃ ( m 4 3 ) , 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 ≤ γ1 , ( we can use this result because we do not have the smoothing matrix W ρ here ) , we get with high probability over the initialization for every i ∈ [ n ] : ∥ϕ′ ( xi ) ∥ = ∥x′i − ϕ ( 0 ) ( xi ) ∥ ≲ γ1 . ( 80 ) Therefore , we can write |trace ( V Z ′i + ) | = 1√ m2 | ∑ j aj1 { sign ( Vj , x′i ) ̸= sign ( V ( 0 ) j , x ′ i ) } Vj , x′i| ≤ 1√ m2 ∑ j 1 { sign ( Vj , x′i ) ̸= sign ( V ( 0 ) j , x ′ i ) } |Vj , x′i| ≤ 1√ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ | ( Vj , − V ( 0 ) j , ) x ′ i| } ( | ( Vj , − V ( 0 ) j , ) x ′ i|+ |V ( 0 ) j , x ′ i| ) ≤ 1√ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ | ( Vj , − V ( 0 ) j , ) x ′ i| } ( 2| ( Vj , − V ( 0 ) j , ) x ′ i| ) . ≤ 1√ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| , | ( Vj , − V ( 0 ) j , ) x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3 min ( γ1 , ∥x′i∥ ) } ( 2| ( Vj , − V ( 0 ) j , ) x ′ i| ) + 1√ m2 ∑ j 1 { | ( Vj , − V ( 0 ) j , ) x ′ i| ≥ γ 2/3 2 ( κ2 m2 ) 1/3 min ( γ1 , ∥x′i∥ ) } ( 2| ( Vj , − V ( 0 ) j , ) x ′ i| ) . Now using the fact that Vj − V ( 0 ) j is orthogonal to ϕ ( 0 ) ( xi ) ’ s : LHS ≤ 2γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } + 2 √ m2 ∑ j 1 { ∥Vj , − V ( 0 ) j , ∥∥x ′ i − ϕ ( 0 ) ( xi ) ∥ ≥ γ 2/3 2 ( κ2 m2 ) 1/3γ1 } ∥Vj , − V ( 0 ) j , ∥∥x ′ i − ϕ ( 0 ) ( xi ) ∥ . Next , using the upper bound on ∥x′i − ϕ ( 0 ) ( xi ) ∥ : LHS ≲ 2γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ∑ j ( 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } . + γ1√ m2 ∑ j 1 { ∥Vj , − V ( 0 ) j , ∥ ≳ γ 2/3 2 ( κ2 m2 ) 1/3 } ∥Vj , − V ( 0 ) j , ∥ ≲ γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ∑ j ( 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } + γ1√ m2 √∑ j 1 { ∥Vj , − V ( 0 ) j , ∥2 ≥ γ 4/3 2 ( κ2 m2 ) 2/3 } √∑ j ∥ ( Vj , − V ( 0 ) j , ) ∥2 ≤ γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ( ∑ j 1 { |V ( 0 ) j , x ′ i| ≤ γ 2/3 2 ( κ2 m2 ) 1/3∥x′i∥ } ) + γ22γ1√ m2 × ( m2 κ2 ) 1/3 1 γ 2/3 2 . Then , applying the first argument of Lemma 29 , we have with high probability over the randomness of V ( 0 ) : ≤ γ 2/3 2 ( κ2 m2 ) 1/3γ1 √ m2 ( m2 κ2 ( κ2 m2 ) 1/3γ 2/3 2 ) + γ22γ1√ m2 × ( m2 κ2 ) 1/3 1 γ 2/3 2 ≤ γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 ≲ γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 . Therefore , we can write : R ( Gγ1 , γ2 ) | ( xi ) , ( yi ) = 1 n Eϵ sup V , W∈S n∑ t=1 ϵifV , W ( xi ) = 1 n Eϵ sup V , W∈S n∑ i=1 ϵia Tσ ( 1/ √ m2VW sσ ( 1/ √ m1Wxi ) ) = 1 n Eϵ sup V , W∈S n∑ i=1 ϵia Tσ ( 1/ √ m2V x ′ i ) = 1 n Eϵ sup W∈S sup V ∈S n∑ i=1 ϵitrace ( V ( Z ′i + Z ′ i + ) ) ≲ 1 n Eϵ sup W , V ∈S n∑ i=1 ϵitrace ( V Z ′i ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 ≤ 1 n Eϵ sup W∈S sup V ∈S trace ( V ( n∑ i=1 ϵiZ ′ i ) ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 = 1 n Eϵ sup W∈S sup V ∈S trace ( ( V − V ( 0 ) ) ( n∑ i=1 ϵiZ ′ i ) ) + 1 n Eϵ sup W∈S trace ( V ( 0 ) ( n∑ i=1 ϵiZ ′ i ) ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 . ( 81 ) For the first term , for every j ∈ [ m2 ] , define Hj to be the set of i ’ s in [ n ] where the jth column of Z ′i is non-zero , i.e . Hj = { i ∈ [ n ] : V ( 0 ) j x ′ i ≥ 0 } . Here , we use the crucial assumption that ( V − V ( 0 ) ) j T ϕ ( 0 ) ( xi ) = 0 , so we can drop the ϕ ( 0 ) ( xi ) term when x′i is multiplied to V − V ( 0 ) . Using this trick and applying Cauchy Schwarz , we bound the first term as : 1 n Eϵ sup W , V ∈S trace ( ( V − V ( 0 ) ) ( n∑ i=1 ϵiZ ′ i ) ) ≤ 1 n Eϵ∥V − V ( 0 ) ∥ sup W∈S √√√√ 1 m2 m2∑ j=1 ∥ ∑ i∈Hj ϵiϕ ( 2 ) ( xi ) ∥2 . Further using Jensen ’ s inequality : 1 n Eϵ sup W , V ∈S trace ( ( V − V ( 0 ) ) ( n∑ i=1 ϵiZ ′ i ) ) ≤ ∥V − V ( 0 ) ∥ n √√√√Eϵ 1 m2 m2∑ j=1 sup W∈S ∥ ∑ i∈Hj ϵiϕ ( 2 ) ( xi ) ∥2 . ( 82 ) Using Equation ( 111 ) of Lemma 6 ( note that we do not have the smoothing matrix W ρ here , so we are allowed to use this result ) , we obtain ∥⟨W −W ( 0 ) , Zki ⟩ − ϕ′ ( xi ) ∥ ≲ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 , where Zki ’ s are defined in Equation ( 318 ) . Plugging this back in ( 82 ) : Eϵ sup W∈S ∥ ∑ i∈Hj ϵiϕ ′ ( xi ) ∥2 ≤ Eϵ sup W∈S ( ∥ ∑ i∈Hj ϵi⟨W −W ( 0 ) , Zki ⟩∥+ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 ) 2 ≲ Eϵ sup W∈S ∥ ∑ i∈Hj ϵi⟨W −W ( 0 ) , Zki ⟩∥2 + C31 κ1 ( n3m33 m1λ0 ) 1/2 = Eϵ sup W∈S m3∑ k=1 ( trace ( ( W −W ( 0 ) ) ( ∑ i∈Hj ϵiZ k i ) ) ) 2 + C31 κ1 ( n3m33 m1λ0 ) 1/2 . ( 83 ) Now for every fixed dataset , with high probability over the randomness of W s , for every k1 ̸= k2 : ∣∣∣⟨∑ i∈Hj ϵiZ k1 i , ∑ i∈Hj ϵiZ k2 i ⟩ ∣∣∣ ≤ ∑ i1 , i2∈Hj ∣∣∣⟨Zk1i1 , Zk2i2 ⟩∣∣∣ = 1 m1 ∑ i1 , i2∈Hj ∣∣∣ m1∑ j=1 W sk1 , jW s k2 , j⟨xi1 , xi2⟩1 { W ( 0 ) j , xi1 ≥ 0 } 1 { W ( 0 ) j , xi2 ≥ 0 } ∣∣∣ But note that because ⟨xi1 , xi2⟩ ≤ 1 , the variables W sk1 , jW s k2 , j ⟨xi1 , xi2⟩1 { W ( 0 ) j , xi1 ≥ 0 } 1 { W ( 0 ) j , xi2 ≥ 0 } are subgaussian with parameter one with respect to the randomness of W s. Hence , with high probability over the randomness of W s , we get∣∣∣⟨∑ i∈Hj ϵiZ k1 i , ∑ i∈Hj ϵiZ k2 i ⟩ ∣∣∣ ≲ 1 m1 ∑ i1 , i2∈Hj √ m1 ≤ n2√ m1 . ( 84 ) Therefore , with high probability over the randomness of W ( 0 ) and W ′ and the dataset , we get Equation ( 84 ) . In order to get rid of the high probability argument on the dataset , we use the stronger Equation ( 319 ) in Lemma 46 which uniformly bounds ⟨Zk1 ( x ) , Zk2 ( x′ ) ⟩ by log ( m1 ) d/m1 for any x , x′ , which in turn gives∣∣∣⟨∑ i∈Hj ϵiZ k1 i , ∑ i∈Hj ϵiZ k2 i ⟩ ∣∣∣ ≤ ∑ i1 , i2∈Hj ∣∣∣⟨Zk1i1 , Zk2i2 ⟩∣∣∣ ≲ n2d log ( m1 ) √m1 , with high probability , independent of the choice of dataset . This bounds is slightly worse comapred to ( 84 ) , but still efficient for our purpose . Furthermore , a similar bound to Equation ( 84 ) can be obtained in a more adversarial situation when we also take maximum against the choice of the dataset . Note that the entries of ∑ i∈Hj ϵiZ k i for 1 ≤ k ≤ m3 can differ only in a sign . Therefore , their norms are all equal . Now suppose that Cj is the random variable of the norm of these variables : Cj : = ∥ ∑ i∈Hj ϵiZ k i ∥ . Then , by substituting rk = 1Cj ∑ i∈Hj ϵiZ k i in Lemma 40 , we get m3∑ k=1 ( trace ( ( W −W ( 0 ) ) ( ∑ i∈Hj ϵiZ k i ) ) ) 2 ≤ C2j ( 1 +m23O ( n2d log ( m1 ) √ m1Cj 2 ) ) ∥W −W ( 0 ) ∥2F ( 85 ) = ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F . ( 86 ) Now recall from Equation ( 80 ) , we have ∥ϕ′ ( xi ) ∥ ≤ γ1 . ( 87 ) Hence , we can apply Corollary 5.1 with ϕ ( 2 ) ( xi ) and C1 substituted by ϕ′ ( xi ) and γ1 respectively , to argue with high probability over the initialization , there exists a set P̃i such that for every i ∈ [ n ] and j /∈ P̃i , sign of VjTx′i is the same as V ( 0 ) j T ϕ ( 0 ) ( xi ) , and moreover , |P̃i| ≲ ( C21 ( m3κ21 ) ) 1/3 m2 . Now let H̄j = { i ∈ [ n ] : V ( 0 ) j ϕ ( 0 ) ( xi ) ≥ 0 } . Note that for j /∈ P̃ = ⋃ i P̃i , we have Hj = H̄j . Now note that the norm of each ∑ i∈Hj ϵiZ k i is at most one . for each index 1 ≤ ℓ ≤ m1d , as the random variables ∑ i∈H̄j ϵi ( Z k i ) ℓ are ∑ i∈H̄j ( Z k i ) 2 ℓ ≤∑ i∈ [ n ] ( Z k i ) 2 ℓ subgaussian , we have with probability at least 1 − 1n over the randomness of ϵi ’ s , for every 1 ≤ ℓ ≤ m1d and every 1 ≤ j ≤ m2 : ∣∣∣ ∑ i∈H̄j ϵi ( Z k i ) ℓ ∣∣∣ ≤√∑ i∈ [ n ] ( Zki ) 2 ℓ log ( m1dm2n ) , which implies for every j ∈ [ m2 ] : ∥ ∑ i∈H̄j ϵiZ k i ∥2 ≤ ∑ ℓ ∑ i∈ [ n ] ( Zki ) 2 ℓ log ( m1d ) ≤ n log ( m1dm2n ) . Name this event B , so P ( B ) ≤ 1 n . Note that although Hj might depend on the randomness of ϵi ’ s , H̄j does not , and if j /∈ P̃ , we obtain Cj = ∥ ∑ i∈H̄j ϵiZ k i ∥ ≤ √ n log ( m1dm2n ) . Moreover , note that we also have the following worse case bound : Cj = ∥ ∑ i∈Hj ϵiZ k i ∥ ≤ ∑ i∈H̄j ∥Zki ∥ ≤ n. Applying the last two inequalities into Equations ( 298 ) and ( 86 ) : Eϵ 1 m2 m2∑ j=1 sup W∈S ∥ ∑ i∈Hj ϵiϕ ′ ( xi ) ∥2 ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + 1 m2 m2∑ j=1 Eϵ sup W∈S m3∑ k=1 ( trace ( ( W −W ( 0 ) ) ( ∑ i∈Hj ϵiZ k i ) ) ) 2 ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + 1 m2 Eϵ1 { B } ∑ j∈P̃ ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F + 1 m2 Eϵ1 { B } ∑ j /∈P̃ ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F + 1 m2 Eϵ1 { Bc } m2∑ j=1 ( C2j + n2m23d log ( m1 ) √ m1 ) ∥W −W ( 0 ) ∥2F ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + ∥W −W ( 0 ) ∥2F [ |P̃ | m2 ( n2 + n2m23d log ( m1 ) √ m1 ) + 2 ( n+ n2m23d log ( m1 ) √ m1 ) ] ≤ C 3 1 κ1 ( n3m33 m1λ0 ) 1/2 + γ21 [ n3 ( C21 ( m3κ21 ) ) 1/3 + ( C21 ( m3κ21 ) ) 1/3n3m23d log ( m1 ) √ m1 + 2 n2m23d log ( m1 ) √ m1 + 2n ] . ( 88 ) Next , we analyze the term 1nEϵ supW∈S trace ( V ( 0 ) ( ∑n i=1 ϵiZ ′ i ) ) . Noting that ∥ϕ ( 0 ) ( xi ) ∥ ≲ κ1 √ m3 with high probability and using Equation ( 87 ) : ∥ n∑ i=1 ϵiZ ′ i∥F ≤ n∑ i=1 ∥Z ′i∥F ≤ n∑ i=1 ∥x′i∥ ≤ ∑ i ( ∥ϕ′ ( xi ) ∥+ ∥ϕ ( 0 ) ( xi ) ∥ ) ≲ n ( √ m3κ1 + γ1 ) . ( 89 ) Hence 1 n Eϵ sup W∈S trace ( V ( 0 ) ( n∑ i=1 ϵiZ ′ i ) ) = 1 n Eϵ sup W∈S n∑ i=1 ϵitrace ( V ( 0 ) Z ′i ) = 1 n Eϵ sup W∈S n∑ i=1 ϵi ( 1√ m2 m2∑ j=1 ajV ( 0 ) j , x ′ i1 { V ( 0 ) j , x ′ i ≥ 0 } ) . = 1 n Eϵ sup W∈S n∑ i=1 ϵi 1√ m2 aTσ ( V ( 0 ) x′i ) ≤ sup W∈S 1√ m2 aTσ ( V ( 0 ) x′i ) . But using Lemma 30 : LHS ≲ κ2 √ m3∥x′i∥ . Applying a similar bound as we did in Equation ( 89 ) on ∥x′i∥ : ∥x′i∥ ≤ ∥ϕ ( 0 ) ( xi ) ∥+ ∥ϕ′ ( xi ) ∥ ≲ κ1 √ m3 + γ1 . Substituting above , we get 1 n Eϵ sup W∈S trace ( V ( 0 ) ( n∑ i=1 ϵiZ ′ i ) ) ≤ κ2 √ m3 ( κ1 √ m3 + γ1 ) . ( 90 ) Finally , Substituting Equations ( 88 ) into ( 82 ) , then combining it with ( 90 ) into ( 81 ) , we obtain a bound on Rademacher complexity which holds w.h.p over both the randomness of the initialization and the dataset : R ( Gγ1 , γ2 ) |x , y ≲ √ C31 κ1 ( n3m33 m1λ0 ) 1/2 ( 91 ) + γ1γ2 n √ n3 ( C21 ( m3κ21 ) ) 1/3 + ( C21 ( m3κ21 ) ) 1/3n3m23d log ( m1 ) √ m1 + 2 n2m23d log ( m1 ) √ m1 + 2n ( 92 ) + κ2 √ m3 ( κ1 √ m3 + γ1 ) + γ 4/3 2 γ1 ( κ2 √ m2 ) 1/3 . Having enough overparameterization , we have for every dataset ( x , y ) ( i.e . worst-case Rademacher complexity ) : R ( Gγ1 , γ2 ) |x , y ≤ 2γ1γ2/ √ n. ( 93 ) Note that for the bound ( 93 ) to hold , the overparameterization should be picked poly large in γ1 , γ2 , as well as in other basic parameters . However , noting Equations ( 51 ) and ( 58 ) in the proof of Theorem 3 , we set γ1 = 1 , γ2 ≥ Ω ( B , n , 1/γ0 ) in Theorem 3 , so γ1γ2 is at most poly in the basic parameters . Therefore , again the overparameterization can be picked polynomially large in the basic only parameters ( i.e . independent of γ1 , γ2 or ζ ) . A.12 CONSTRUCTING W ∗ , V ∗ This section consists of two subsections ; First , we prove a structural result for the first layer weights ( W ′ , V ′ ) that the algorithm visits , then construct a weight matrix W ∗ for the first layer with some good properties . Second , we do the same thing for the second layer ( however , the structure of the first and second layers are completely different ) . Through out this section , we assume we have the norm bounds ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 . Notably , we rely on a number of basic Lemmas more related to the representation power of the network , which we defer their proof into a later Appendix B.2 and refer to them here as needed . A.12.1 FIRST LAYER , CONSTRUCTION OF W ∗ Lemma 1 Suppose m1 ≥ 16n2m23/λ20 . Let Pi = { j ∈ [ m1 ] | |W ( 0 ) j xi| ≤ c2/ √ m1 } and P = ∪Pi . During SGD iterations , suppose we have ∥W ′∥F ≤ C1 . Then , for a value c2 satisfying 2C1 √ nm3/ √ λ0 ≤ c2 ≤ κ1λ0 √ m1/ ( 2n 2 ) , with high probability ∀i : |Pi| ≲ c2 √ m1/κ1 , and for j /∈ P , during the whole algorithm we have ∥W ′j∥ ≤ √ nm3C1√ m1λ0 + c2/ ( 4 √ m1 ) ≤ c2/ ( 2 √ m1 ) , c2/ √ m2 ≤ |W ( 0 ) j xi| . So the signs of neurons outside P never changes . In particular , we can set c2 as small as c2 = C1 √ nm3/ √ λ0 . In the rest of the proof ( i.e . other sections ) , we set c2 to this value . Proof of Lemma 1 Define the matrix Z̃ik = 1√ m1 ( W sk , jxi1 { ∀i : W ( 0 ) j Txi ≥ c2/ √ m1 } ) nj=1 . Let Pi be the set of indices j such that 1 { W ( 0 ) j Txi ≥ c2/ √ m1 } is zero . First of all , note that by Bernstein inequality : |Pi| ≤ c2 √ m1/κ1 +O ( √ c2 √ m1/κ1 + 1 ) ≲ c2 √ m1/κ1 . Now suppose that until the current iteration of the algorithm the assumption has been true , i.e . the signs of the neurons outside of P have never changed . As a result , due to the specific update of the SGD for both of the terms EZℓ ( fV ′ , W ′ ( x ) , y ) and ∥W ′∥2F , if we define W ′|P to be the restriction of W ′ to indices that are not in P ( i.e . the columns in P are equal to zero ) , then we can write W ′|P T = m3∑ k=1 n∑ i=1 αk , iZ̃ i k. ( 94 ) An issue here is that we also have some injected noise by PSGD intoW ′ which violates Equation ( 94 ) . To handle the injected noise as well , we define the subspace Φ′ of Rm1×d matrices to be the set of vectors with arbitrary rows for j ∈ [ m1 ] with j ∈ P , while restricted to the other rows j /∈ P in should be in the span of ( Z̃ik ) i , k . Then , we decompose W ′ into subspaces Φ′ and Φ′⊥ respectively as W ′ = W ′ ( 1 ) +W ′ ( 2 ) , where W ′ ( 1 ) ∈ Φ′ , W ′ ( 2 ) ∈ Φ′⊥ . Here , we want to prove ∥W ′ ( 1 ) j ∥ ≤ c2/ ( 4 √ m1 ) . We handle the ∥W ′ ( 2 ) j ∥ part in Appendix 45 . So instead of W ′|P in Equation ( 94 ) we consider W ′ ( 1 ) |P : W ′ ( 1 ) |P T = m3∑ k=1 n∑ i=1 αk , iZ̃ i k. ( 95 ) We handle the other part W ′ ( 2 ) in Appendix 45 . Now exactly similar to the drivation in Lemma 38 , we can state with high probability C21 ≥ ∥W ′∥2 ≥ ∥W ′ ( 1 ) ∥2 ≥ ∥W ′ ( 1 ) |P ∥2 ≥ m3∑ k=1 ∥ n∑ i=1 αk , iZ̃ i k∥2 −O ( nm3/ √ m1 ) ∑ k ∥αk∥2 . ( 96 ) Note that we are exploiting the fact that the norm ∥W ′∥ remains bounded by C1 . Now using a Hoeffding bound for matrix H∞′ defined below , we write : H∞′i1 , i2 : = Ew : N ( 0 , Rd ) 1 { ∀i : |w Txi| ≥ c2/ √ m1 } xTi1xi2 ( 1 { w Txi1 ≥ 0 } 1 { wTxi2 ≥ 0 } ) = Ew : N ( 0 , Rd ) ( 1 { wTxi1 ≥ 0 } 1 { wTxi2 ≥ 0 } ) xTi1xi2 ±O ( E1 { ∃i : |wTxi| ≤ c2/ √ m1 } ( 1 { wTxi1 ≥ 0 } 1 { wTxi2 ≥ 0 } ) ) xTi1xi2 = H∞i1 , i2 ±O ( nc2/ ( √ m1κ1 ) ∥xi1∥∥xi2∥ ) = H∞i1 , i2 ±O ( nc2/ ( √ m1κ1 ) ) . ( 97 ) Now opening Equation ( 96 ) and using the property c2 ≤ k1λ0 √ m1/ ( 2n 2 ) , we get LHS = ∑ k ∑ i1 , i2 αk , i1αk , i2⟨Z̃ i1 k , Z̃ i2 k ⟩ −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) = ∑ k ∑ i1 , i2 αk , i1αk , i2 ( H ∞′ i1 , i2 ±O ( 1/ √ m1 ) ) −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) ≥ ∑ k ∑ i1 , i2 αk , i1αk , i2H ∞ i1 , i2 ± ∥αk∥ 2 1O ( nc2/ √ m1κ1 ) −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) ≥ ∑ k αTkH ∞αk −O ( nc2/ √ m1κ1 ) ∑ k ∥αk∥21 −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) ≥ ∑ k αTkH ∞αk −O ( c2n2/ √ m1κ1 ) ∑ k ∥αk∥22 −O ( nm3/ √ m1 ∑ k ∥αk∥2 ) = ( λ0 −O ( nm3/ √ m1 ) −O ( c2n2/ √ m1κ1 ) ) ∑ k ∥αk∥2 ≥ λ0/2 ∑ k ∥αk∥2 . For the last line to hold , we need enough overparameterization . This implies∑ k ∥αk∥2 ≲ C21/λ0 . Now again , exactly similar to the derivation in Lemma 38 , for j /∈ P we have ∥W ′ ( 1 ) j ∥ ≤ √ nm3/ √ m1 √∑ k ∥αk∥2 ≲ √ m3nC1/ √ m1λ0 , which completes most of the proof . For the rest , we are left to show that for the other part W ′ ( 2 ) , we have ∥W ′ ( 2 ) j ∥ ≤ c2/ ( 4 √ m1 ) , which we do in Appendix 45 . Lemma 2 Under condition m3n/ √ m1 ≤ λ0/4 , there exist matrices { W ∗k } m3 k=1 ∈ Rm1×d s.t . for every k ̸= k′ ∈ [ m3 ] and i ∈ [ n ] : ⟨W ∗k , Zik′⟩ = 0 , ∥W ∗k −W+k ∥ ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ , |⟨W ∗k , Zik⟩ − ⟨W+k , Z i k⟩| ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ . Furthermore , for k1 ̸= k2 : |⟨W ∗k1 , W ∗ k2⟩| ≤ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ∥Vk1∥H∞∥Vk2∥H∞ . ( 98 ) Proof of Lemma 2 Let W+k = ∑ i Vk , iZik . we want to compute the norm of the projection P ( W+k ) of W + k onto the subspace spanned by all Zik′ for k ′ ̸= k and i ∈ [ n ] : ∥P ( W+k ) ∥ 2 = ( ⟨W+k , Z i k′⟩ ) Tk′ ̸=k , i∈ [ n ] ( ⟨Zi1k1 , Z i2 k2 ⟩ ) −1 ( k1 , i1 ) , ( k2 , i2 ) ∈ [ m3 ] − { k } × [ n ] ( ⟨W+k , Z i k′⟩ ) k′ ̸=k , i∈ [ n ] , ( 99 ) where the first and third terms are vectors and the middle term is a matrix . Now note that for each k′ , k1 , k2 ̸= k , by Hoeffding inequality : ( ⟨Zi1k′ , Z i2 k′ ⟩ ) i1 , i2∈ [ n ] = H∞ + ( ±1/ √ m1 ) i1 , i2∈ [ n ] , ( 100 ) ⟨W+k , Z i k′⟩ = ⟨ ∑ i Vk , iZik , Zik′⟩ ≲ 1√ m1 ∑ i |Vk , i| ≤ √ n√ m1 ∥Vk∥ ≤ √ n√ m1λ0 ∥Vk∥H∞ . ( 101 ) Therefore , ∥ ( ⟨W+k , Z i k′⟩ ) Tk′ ̸=k , i∈ [ n ] ∥ ≤ n √ m3 m1λ0 ∥Vk∥H∞ . ( 102 ) Now Equation ( 100 ) implies for small enough m1 λmin ( ( ⟨Zi1k′ , Z i2 k′ ⟩ ) i1 , i2∈ [ n ] ) ≥ λ0/2 , ( 103 ) as long as λ0 ≥ 2n/m1 . Moreover , define Ã to be the block version of A′ = ( ⟨Zi1k1 , Z i2 k2 ⟩ ) −1 ( k1 , i1 ) , ( k2 , i2 ) ∈ [ m3 ] − { k } × [ n ] , i.e . for k1 = k2 they are the same but for k1 ̸= k2 Ã is zero . Then λmin ( Ã ) ≥ λ0/2 , because the eigenvalues of each block is at least λ0/2 using Equation ( 103 ) . But note that ∥A′ − Ã∥2 ≤ ∥A′ − Ã∥F ≤ m3n/ √ m1 . So as long as m3n/ √ m1 ≤ λ0/4 , we have λmin ( A ) ≥ λ0/4 . Combining this fact with Equation ( 102 ) and plugging it into Equation ( 99 ) , we obtain ∥P ( W+k ) ∥ 2 ≲ n 2m3 m1λ20 ∥Vk∥2H∞ . Now define W ∗k =W + k − P ( W + k ) . Then ∥W ∗k −W+k ∥ = ∥P ( W + k ) ∥ ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ , |⟨W ∗k −W+k , Z i k⟩| ≤ ∥P ( W+k ) ∥∥Z i k∥ ≲ n √ m3 λ0 √ m1 ∥Vk∥H∞ . Furthermore , note that W ∗k2 is orthogonal to W + k1 for k1 ̸= k2 , so |⟨W ∗k1 , W ∗ k2⟩| = |⟨W + k1 − P ( W+k1 ) , W ∗ k2⟩| = |⟨P ( W+k1 ) , W + k2 − P ( W+k2 ) ⟩| ≤ ∥P ( W+k1 ) ∥∥W + k2 − P ( W+k2 ) ∥ ≤ ∥P ( W+k1 ) ∥ ( ∥W + k2 ∥+ ∥P ( W+k2 ) ∥ ) . ( 104 ) But note that ∥W+k ∥ = ∥ ∑ i Vk , iZik∥ ≤ ∑ i |Vk , i|∥Zik∥ ≤ ∑ i |Vk , i| ≤ √ n∥Vk∥2 . ≤ √ n λ0 ∥Vk∥H∞ . Therefore , we can bound Equation ( 104 ) as : |⟨W ∗k1 , W ∗ k2⟩| ≤ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ∥Vk1∥H∞∥Vk2∥H∞ . Lemma 3 There exists a matrix W+2k such that for every j ∈ P , W +2 k j = 0 , and |trace ( W+2k Z k i ) − x̄i , k| ≤ C1 √ m3n 2/ ( λ0κ1 √ m1 ) ∥Vk∥H∞ . Proof of Lemma 3 Define W+2k to be equal to W + k for j /∈ P and equal to zero vector otherwise . Then , by Lemma 38 : ( note that |Pi| ≤ C1 √ nm3 √ m1/ ( √ λ0κ1 ) ) |trace ( W+k Z k i ) − trace ( W+2k Z k i ) | ≤ 1/ √ m1 ∑ j∈P |W+k jxi| ≤ |P |√ m1 ∥W+k ∥ ≤ √ nm3/ ( m1 √ λ0 ) |P |∥Vk∥H∞ ≤ C1m3n2/ ( ( λ0κ1 √ m1 ) ∥Vk∥H∞ . Combining this with Lemma 37 , the desired result follows . Lemma 4 Under conditionm3n/ √ m1 ≤ λ0/4 , there exist matrixW ∗k ’ s exactly satisfying the same conditions in Lemma 2 but with respect to W+2k instead of W + k , and moreover , for j ∈ P we have W ∗k j = 0 . Proof of Lemma 4 We can repeat the exact same procedure of Lemma 2 for W+2k . Using the bound in Equation ( 97 ) , we have ( ⟨Z̃i1k′ , Z̃ i2 k′ ⟩ ) i1 , i2∈ [ n ] = H ′∞ +O ( ±1/ √ m1 ) i1 , i2∈ [ n ] = H∞ + ( ±nc2/ √ m1κ1 ) i1 , i2∈ [ n ] +O ( ±1/ √ m1 ) i1 , i2∈ [ n ] = H∞ + ( ±nc2/ √ m1κ1 ) i1 , i2∈ [ n ] , so as long as n2c2/ √ m1κ1 = n 2C1 √ nm3/ ( κ1 √ m1λ0 ) ≤ λ0/2 , with similar argument as in Lemma 2 , we get λmin ( ( ⟨Z̃i1k′ , Z̃ i2 k′ ⟩ ) i1 , i2∈ [ n ] ) ≥ λ0/2 . Moreover , ⟨W+2k , Z̃ i k′⟩ = ⟨ ∑ i Vk , iZ̃ik , Z̃ik′⟩ ≲ 1√ m1 ∑ i |Vk , i| ≤ √ n√ m1 ∥Vk∥ ≤ √ n√ m1λ0 ∥Vk∥H∞ . Thus , using the same argument as before the proof is complete . Lemma 5 Suppose m1 ≥ n7m3/λ0 . During SGD , suppose we are currently at ( V ′ , W ′ ) with W ′ ≤ C1 . For any matrix W1 , we denote the signs of the first layer imposed by W1 by DW1 , xi . Then with high probability , there exists W ∗ = ∑ k∈ [ m3 ] W ∗ k such that W ∗ k ’ s is orthogonal to all other Z i k′ ’ s for k ′ ̸= k , and for every i ∈ [ n ] , we have : ∥ 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗xi − x̄i∥∞ ≲ nm3√ m1λ0 [ 1 + nC1 κ1 ] ∥Vk∥H∞ : = ℜ∥Vk∥H∞ . Moreover , we have ∥W ∗j ∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ + n √ m3 λ0 √ m1 ( ∑ k ∥Vk∥H∞ ) : = ϱ √ m3 m1 , ( 105 ) Particularly , for any diagonal sign matrix Σ ∈ Rm3×m3 , we have ∥W ∗Σ∥2F ≤ ( n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) + ( 1 +O ( n/ ( λ0 √ m1 ) + n2m3 λ20m1 ) ) ) ∑ k ∥Vk∥2H∞ . ( 106 ) which , by having enough overparameterization , implies ∥W ∗Σ∥F ≤ √ 2 ∑ k ∥Vk∥2H∞ = √ 2ζ1 , ( 107 ) where W ∗Σ : = m3∑ k=1 ΣkW ∗ k . ( 108 ) Moreover , we have 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗ Σxi = Σ 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗xi . ( 109 ) Proof of Lemma 5 From Lemma 3 , we have |x̄i , k − trace ( W+2k Z i k ) | ≤ C1m3n2/ ( λ0κ1 √ m1 ) ∥Vk∥H∞ . Combining this with the result of Lemma 4 , we get : |x̄i , k − trace ( W ∗kZik ) | ≲ [ n√m3 λ0 √ m1 + C1m3n 2 λ0κ1 √ m1 ] ∥Vk∥H∞ = nm3√ m1λ0 [ 1 + nC1 κ1 ] ∥Vk∥H∞ . ( 110 ) On the other hand , based on the property that W ∗k j = 0 for j ∈ P and its orthogonal property from Lemma 4 , for j ∈ P we get 1√ m1 W skDW ( 0 ) +W ′ , xiW ∗xi = 1√ m1 W skDW ( 0 ) , xiW ∗xi = trace ( W ∗Zik ) = trace ( W ∗ kZ i k ) = 1√ m1 W skDW ( 0 ) , xiW ∗ k xi , which combined with Equation ( 110 ) completes the proof . From the above , Equation ( 109 ) is also clear . Finally , note that by Lemma 38 we have ∥W+2j∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ . which Combined with Lemma 4 implies ∥W ∗j ∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ + n √ m3 λ0 √ m1 ( ∑ k ∥Vk∥H∞ ) : = ϱ √ m3 m1 , while the other claims follows from Lemma 39 and Lemma 4 , combined with Equation ( 98 ) : ∥W ∗Σ∥2F ≤ ∑ k ∥W ∗k ∥2 + ∑ k1 ̸=k2 |⟨W ∗k1 , W ∗ k2⟩| ≤ ∑ k ∥W ∗k ∥2 + n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ( ∑ k ∥Vk∥H∞ ) 2 ≤ ∑ k ∥W ∗k ∥2 + n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) √ n ( ∑ k ∥Vk∥2H∞ ) ≤ ∑ k ∥W ∗k ∥2 + n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 ≤ n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 + ∑ k ∥W+2k ∥ 2 + n2m3 λ20m1 ∑ k ∥Vk∥2H∞ ≤ n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 + ∑ k ∥W+k ∥ 2 + n2m3 λ20m1 ∑ k ∥Vk∥2H∞ ≲ n √ n λ20 √ m3√ m1 ( 1 + √ m3√ m1 ) ζ1 + ( 1 +O ( n/ ( λ0 √ m1 ) + n2m3 λ20m1 ) ) ζ1 . Next , we move on to construct V ∗ for the second layer . A.12.2 SECOND LAYER , CONSTRUCTION OF V ∗ In this section , we present a couple of lemmas that step by step lead to the construction of V ∗ . we remind the reader that ϕ ( 0 ) ( xi ) is the output of the first layer at initialization weights , ϕ′ ( xi ) and ϕ ( 2 ) ( xi ) are the changes in the output of the first layer when W ′ and W ′ + W ρ are added , respectively , and finally ϕ∗ ( xi ) is the optimal features that are generated by the matrix W ∗ but with the sign pattern of W ( 0 ) +W ′ , i.e . ϕ∗ ( xi ) = 1√ m1 W sDW ( 0 ) +W ′ , xiW ∗xi . We also define x′i as x′i = ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) = 1√ m1 W sσ ( ( W ( 0 ) +W ′ +W ρ ) xi ) . To begin , we state a lemma to bound the magnitude of ∥ϕ′ ( xi ) ∥ , given that the norm of W ′ is bounded by C1 and the sign pattern Sgn ( ( W ( 0 ) +W ′ ) xi ) satisfies condition stated for the set of indices P in Lemma 1 . Later on , we exploit this Lemma in Lemma 33 to state bounds for ∥ϕ ( 2 ) ( xi ) ∥ . Lemma 6 Let the matrix W ′ with norm bound ∥W ′∥ ≤ C1 , such that the signs of ( W ( 0 ) j +W ′j ) xi and W ( 0 ) j xi can be different only for j ∈ P , for P defined in Lemma 1 . ( Note that for W ′ at every step of the algorithm , this is automatically satisfied by Lemma 1 ) Then ∥ϕ′ ( xi ) ∥ ≤ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 + ( 1 +O ( m23/ √ m1 ) ) C1 . Particularly for large enough m1 compared to n , m3 , λ0 , κ1 , C1 , we have ∥ϕ′ ( xi ) ∥ ≲ C1 . Proof of Lemma 6 We write |ϕ′k ( xi ) − ⟨W ′ , Zik⟩| ≤ 2/ √ m1 ∑ j∈P |W ′jxi| ≤ 2/ √ m1 ∑ j∈P ∥W ′j∥ ≤ 2 √ |P |/ √ m1∥W ′∥F ≤ 2C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 , ( 111 ) where the last line follows from the bound on |P | from Lemma 1 . On the other hand , because by Hoeffding we know that ⟨Zik , Zik′⟩ ≲ 1/ √ m1 by Lemma 40 , we get m3∑ k=1 ⟨W ′ , Zik⟩2 ≤ ( 1 +O ( m23/ √ m1 ) ) ∥W ′∥2F ≤ ( 1 +O ( m23/ √ m1 ) ) C 2 1 . Combining this with Equation ( 111 ) , we get ∥ϕ′ ( xi ) ∥ ≤ √∑ k |ϕ ( 2 ) k ( xi ) − ⟨W ′ , Zik⟩|2 + √∑ k ⟨W ′ , Zik⟩2 ≤ 2C 3/2 1√ κ1 ( n3m33 m1λ0 ) 1/4 + ( 1 +O ( m23/ √ m1 ) ) C1 . ( 112 ) Next , we prove a structural lemma regarding the sign pattern in the second layer when we feed in x′i to it , with the important message that the dominance of sign patterns are specified by ϕ ( 0 ) ( xi ) . Lemma 7 Suppose we have m3κ21 ≳ C21 , κ2 √ m2 ≥ C2 , and m1 satisfies the condition on Lemma 6 . If we have the condition ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 , which happens under the high probability event Ec defined in Lemma 33 , then for every i ∈ [ n ] , there exist a subset P̃i which might depend on W ( 0 ) , V ( 0 ) , W ′ , V ′ , such that |P̃i| ≲ ( ( C21 ( m3κ21 ) ) 1/3 + ( c3 + C21 c23m3κ 2 1 ) ( C22 κ22m2 ) 1/3 ) m2 . Moreover , for every i ∈ [ n ] , for j /∈ P̃i , : 2 3 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≥ |V ( 0 ) j ϕ ( 2 ) ( xi ) |+ |V ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | , |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ ( κ2 m2 ) 1/3C 2/3 2 c3∥ϕ ( 0 ) ( xi ) ∥ , |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ ( κ2 m2 ) 1/3C 2/3 2 c3∥x′i∥ . Proof of Lemma 7 By assumption , we know that during the algorithm , we have ∥V ′∥ ≤ C2 . Also , we know by Lemma 33 that under Ec : ∥ϕ ( 2 ) ( xi ) ∥ ≤ 2C1 . Define the set P ′i = { j| |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ c3 ( κ2 m2 ) 1/3C 2/3 2 ∥ϕ ( 0 ) ( xi ) ∥ } ( 113 ) and P ′ = ∪P ′i . We have P ( |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ c3 ( κ2 m2 ) 1/3C 2/3 2 ∥ϕ ( 0 ) ( xi ) ∥ ) ≤ c3 ( κ2 m2 ) 1/3C 2/3 2 /κ2 , so by Bernstein , with high probability : |P ′i | ≤ m2C 2/3 2 c3 ( κ2 m2 ) 1/3/κ2 + √ m2C 2/3 2 c3 ( κ2 m2 ) 1/3/κ2 + 1 ≲ c3m2C2/32 ( κ2 m2 ) 1/3/κ2 , so with high prob . |P ′i | ≲ c3C 2/3 2 ( m2 κ2 ) 2/3 . ( 114 ) On the other hand , Note that ϕ ( 0 ) k ( xi ) = m1∑ j=1 1/ √ m1W s k , jσ ( W ( 0 ) j xi ) ( 115 ) is subGaussian with parameter σ2 = O ( 1/m1 ∑ j σ ( W ( 0 ) j xi ) 2 ) . Furthermore , note that if we compute the variance of ϕ ( 0 ) k ( xi ) with respect to the randomness of W s : Eϕ ( 0 ) k ( xi ) 2 = 1/m1 m1∑ j=1 σ ( W ( 0 ) j xi ) 2 : = ℵ which itself concentrates around 1/2κ21∥xi∥2 = 1/2κ21 by another Bernstein , i.e . ℵ = 1/2κ21 ( 1 ± O ( 1/ √ m1 ) ) . Therefore , by concentration of subexponential variables ( Bernstein ) , it is not hard to see that the squared norm of the vector ϕ ( 0 ) ( xi ) is ( m3κ41 , κ2 ) -subexponential and concentrates around m3ℵ , i.e . ∥ϕ ( 0 ) ( xi ) ∥2 = m3ℵ ±O ( κ21 √ m3 ) = m3κ 2 1/2±O ( m3κ21/ √ m1 ) ±O ( κ21 √ m3 ) , ( 116 ) with high probability . Combining this with the fact that ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 implies with high probability : ∥ϕ ( 0 ) ( xi ) ∥ ∥ϕ ( 2 ) ( xi ) ∥ ≳ √ m3κ1 C1 . ( 117 ) Now define P ′′i = { j| |V ′j x′i| ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 } . If j ∈ P ′′i − P ′i , then by Equation ( 117 ) , with high probability ∥V ′j ∥∥ϕ ( 2 ) ( xi ) ∥ ≥ |V ′jϕ ( 2 ) ( xi ) | = |V ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | = |V ′j x′i| ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 ≳ c3 ( κ2 m2 ) 1/3C 2/3 2 ∥ϕ ( 0 ) ( xi ) ∥ , or ∥V ′j ∥2 ≳ c23 ( κ2 m2 ) 2/3C 4/3 2 m3κ 2 1 C21 . But note that ∥V ′∥2F ≤ C22 by our assumption , which implies |P ′′i − P ′i | ≲ C22/ [ c23 ( κ2 m2 ) 2/3C 4/3 2 m3κ 2 1 C21 ] = C 2/3 2 C 2 1 c23m3κ 2 1 ( m2 κ2 ) 2/3 . ( 118 ) Now combining Equations ( 114 ) and ( 118 ) , we finally obtain |P ′′i | = |P ′′i − P ′i |+ |P ′i | ≲ ( c3 + C21 c23m3κ 2 1 ) C 2/3 2 ( m2 κ2 ) 2/3 . Now define the set P ′′′i = { j| |V ( 0 ) j ϕ ( 2 ) ( xi ) | ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 } . ( 119 ) Note that for every j ∈ [ m2 ] , V ( 0 ) j ϕ ( 0 ) ( xi ) is gaussian with variance ∥ϕ ( 0 ) ( xi ) ∥ over the randomness of V ( 0 ) j , so P ( V ( 0 ) j ϕ ( 0 ) ( xi ) ≤ ακ2∥ϕ ( 0 ) ( xi ) ∥ ) ≲ α . Therefore , if we define the set Qi = { j ∈ [ m2 ] | |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ ακ2∥ϕ ( 0 ) ( xi ) ∥ } , then for large enough m2 , by Bernstein with high prob . : |Qi| ≲ αm2 . ( 120 ) Now note that ϕ ( 0 ) ( xi ) is fixed during the algorithm . On the other hand , by random matrix theory , we know that with high probability , the eigenvalues of the matrix V ( 0 ) are in ( κ2 ( √ m2 −√ m3 ) , κ2 ( √ m2 + √ m3 ) ) . Therefore , even if the vector ϕ ( 2 ) ( xi ) is picked adversarialy ( because it keeps changing during the algorithm ) , we get that with high probability over the randomness of V ( 0 ) : |V ( 0 ) ϕ ( 2 ) ( xi ) ∥2 ≤ κ22 ( √ m2 + √ m3 ) 2∥ϕ ( 2 ) ( xi ) ∥2 ≲ κ22m2∥ϕ ( 2 ) ( xi ) ∥2 . ( 121 ) Moreover , because ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 and from Equation ( 116 ) , with high probability over the randomness of W ( 0 ) : ∥ϕ ( 0 ) ( xi ) ∥ ∥ϕ ( 2 ) ( xi ) ∥ ≳ √ m3κ1 C1 . This means that for j ∈ P ′′′i −Qi , combining these inequalities we conclude with high probability |V ( 0 ) j ϕ ( 2 ) ( xi ) | ≥ |V ( 0 ) j ϕ ( 0 ) ( xi ) |/3 ≳ ακ2∥ϕ ( 0 ) ( xi ) ∥ ≥ ακ2 √ m3κ1 C1 ∥ϕ ( 2 ) ( xi ) ∥ , which combined with ( 121 ) implies ∥P ′′′i ∥ ≲ m2C 2 1 m3κ21α 2 . Balancing this term with the one in Equation ( 120 ) , we set α : = C 2/3 1 m 1/3 3 κ 2/3 1 , which implies |P ′′′i | ≲ |P ′′′i −Qi|+ |Qi| ≤ ( C21 ( m3κ21 ) ) 1/3 m2 . Defining P̃i = P ′′i ∪ P ′′′i , we finally get |P̃i| ≲ ( ( C21 ( m3κ21 ) ) 1/3 + ( c3 + C21 c23m3κ 2 1 ) C 2/3 2 ( 1 κ22m2 ) 1/3 ) m2 . Clearly by the definition of P ′′i and P ′′′ i the proof is complete . Corollary 5.1 Under the condition ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 ( which happens under the event Ec defined in Lemma 33 ) , setting c3 : = C 2/3 1 m 1/3 3 κ 2/3 1 ( κ22m2 C22 ) 1/3 in the previous Lemma , we obtain ∀i ∈ [ n ] : |P̃i| ≲ ( C21 ( m3κ21 ) ) 1/3 m2 . Also for j /∈ P̃i , the conditions in ( 113 ) and ( 119 ) becomes the same as |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ κ2 C 2/3 1 m 1/3 3 κ 2/3 1 ∥ϕ ( 0 ) ( xi ) ∥ . Hence , for every i ∈ [ n ] and for j /∈ P̃i , with high probability : 2 3 |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≥ |W ( 0 ) j ϕ ( 2 ) ( xi ) |+ |W ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | , ( 122 ) |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ κ2 C 2/3 1 m 1/3 3 κ 2/3 1 ∥ϕ ( 0 ) ( xi ) ∥ ≳ κ2 ( √ m3κ1C 2 1 ) 1/3 , ( 123 ) |W ( 0 ) j ϕ ( 0 ) ( xi ) | ≳ C 2/3 1 m 1/3 3 κ 2/3 2 ∥x′i∥ . ( 124 ) Next , we state concentration result for the gram matrix of ϕ ( 0 ) ( xi ) ’ s . Lemma 8 For every i1 , i2 ∈ [ n ] , with high probability over the randomness of W ( 0 ) and V ( 0 ) we have ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 + √ m3κ 2 1 ) . Proof of Lemma 8 First , we compute the expectation : E⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = 1/m1 ∑ j1 , j2∈ [ m1 ] ∑ k∈ [ m3 ] EW sk , j1W s k , j2σ ( W ( 0 ) j1 xi1 ) σ ( W ( 0 ) j2 xi2 ) = 1/m1 ∑ j1 ̸=j2 E ∑ k∈ [ m3 ] W sk , j1W s k , j2σ ( W ( 0 ) j1 xi1 ) σ ( W ( 0 ) j2 xi2 ) +m3/m1 ∑ j∈ [ m1 ] σ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) . But σ ( W ( 0 ) j1 xi1 ) σ ( W ( 0 ) j2 xi2 ) is ( m1κ 4 1 , κ 2 1 ) -sub-exponential , so∑ j∈ [ m1 ] σ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) = m1Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( √ m1κ 2 1 ) , which means with high probability : E⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 ) . On the other side , we know that ϕ ( 0 ) k ( xi1 ) is subgaussian with parameters σ 2 = 1/m1 ∑ j ( W ( 0 ) j xi1 ) 2 : = ℵ1 and σ2 = 1/m1 ∑ j ( W ( 0 ) j xi2 ) 2 : = ℵ2 respectively . On the other hand , we know that by Bernstein w.h.p ℵ1 = 1/2κ21 ( 1±O ( 1/ √ m1 ) ) , ℵ2 = 1/2κ21 ( 1±O ( 1/ √ m1 ) ) . Hence , ϕ ( 0 ) k ( xi1 ) ϕ ( 2 ) k ( xi2 ) is ( ℵ1ℵ2 , √ ℵ1ℵ2 ) -subexponential , and so ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ is ( m3ℵ1ℵ2 , √ ℵ1ℵ2 ) -subexponential . Therefore , applying another Bernstein on the top , we get ⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ = E⟨ϕ ( 0 ) ( xi1 ) , ϕ ( 0 ) ( xi2 ) ⟩ ±O ( √ m3 √ ℵ1ℵ2 ) = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 ) ± √ m3κ 2 1 2 ( 1±O ( 1/ √ m1 ) ) = m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( m3κ 2 1/ √ m1 + √ m3κ 2 1 ) . Now we define the matrix Li ∈ Rm3×m2 , with its jth column Li , j equal to aj√ m2 1 { V ( 0 ) j ϕ ( 0 ) ( xi ) ≥ 0 } ϕ∗ ( xi ) . First , we state the following lemma which characterize a concentration result for the gram matrix of ( Li ) n i=1 . Lemma 9 With high probability , we have the following approximation : ⟨Li1 , Li2⟩ = ⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ [ F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ±O ( m−1/41 +m −1/4 2 +m −1/4 3 ) ] . Proof of Lemma 9 By Hoeffding : ⟨Li1 , Li2⟩ = 1/m2 ∑ j∈m2 ϕ∗ ( xi1 ) Tϕ∗ ( xi2 ) 1 { V ( 0 ) j ϕ ( 1 ) ( xi1 ) ≥ 0 } 1 { V ( 0 ) j ϕ ( 1 ) ( xi2 ) ≥ 0 } = ϕ∗ ( xi1 ) Tϕ∗ ( xi2 ) ( E1 { V ( 0 ) j ϕ ( 1 ) ( xi1 ) ≥ 0 } 1 { V ( 0 ) j ϕ ( 1 ) ( xi2 ) ≥ 0 } ±O ( 1/ √ m2 ) ) = ϕ∗ ( xi1 ) Tϕ∗ ( xi2 ) ( F2 ( ⟨ϕ ( 1 ) ( xi1 ) , ϕ ( 1 ) ( xi2 ) ⟩/ ( ∥ϕ ( 1 ) ( xi1 ) ∥∥ϕ ( 1 ) ( xi2 ) ∥ ) ) ±O ( 1/ √ m2 ) ) , where recall F2 ( x ) = 1/4 + arcsin ( x ) /2π , measures the angle between two unit vectors based on their dot product . Now notice that according to Lemma 8 , with high probability : ⟨Li1 , Li2⟩/⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ = F2 ( m3Eσ ( W ( 0 ) j xi1 ) σ ( W ( 0 ) j xi2 ) ±O ( ( m3/√m1 +√m3 ) κ21 ) √ ( m3Eσ ( W ( 0 ) j xi1 ) 2 ±O ( ( m3/ √ m1 + √ m3 ) κ 2 1 ) ) ( m3Eσ ( W ( 0 ) j xi2 ) 2 ±O ( ... ) ±O ( 1/ √ m2 ) ) = F2 ( F3 ( ⟨xi1 , xi2⟩ ) ±O ( 1/√m1 + 1/√m3 ) 1/2±O ( 1/ √ m1 + 1/ √ m3 ) ±O ( 1/ √ m2 ) ) , where recall F3 : [ −1 , +1 ] → [ −1/2 , 1/2 ] is defined as : F3 ( x ) : = √ 1− x2 2π + x 4 + x arcsinx 2π . It is easy to see F3 has the property that for unit vectors x1 , x2 and w sampled as standard normal : F3 ( ⟨x1 , x2⟩ ) = Eσ ( wTx1 ) σ ( wTx2 ) . But because |F3 ( . ) | = O ( 1 ) , we have ⟨Li1 , Li2⟩/⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ = F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ±O ( 1/ √ m1 + 1/ √ m2 + 1/ √ m3 ) ) . Now notice that the derivative of F2 , i.e . 1/2π √ 1− x2 is increasing in the interval ( 0 , 1 ) , so for a fixed δ , the maximum of |F2 ( x ) − F2 ( x− δ ) | happens at x = 1 . On the other hand , by writing the first order approximation of arcsin ( 1 − t2 ) around t = 0 and upper bounding its derivative in the interval [ 0 , 1 ] , we get that for 0 ≤ δ ≤ 1 : arcsin ( 1− δ ) ≥ arcsin ( 1 ) − 2 √ δ . Therefore , F2 ( x± δ ) = F2 ( x ) ±O ( √ δ ) . Hence : ⟨Li1 , Li2⟩/⟨ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ = F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ±O ( √ 1/ √ m1 + 1/ √ m2 + 1/ √ m3 ) = F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ±O ( m−1/41 +m −1/4 2 +m −1/4 3 ) , which completes the proof . Finally , we are ready to construct the weights V ∗ for the second layer . A.12.3 CONSTRUCTION OF V ∗ Lemma 10 Let ℜ = n √ m3√ m1λ0 [ 1 + nC1 κ1 ] . Suppose we have the condition that for every k ∈ [ m3 ] : max k ∥Vk∥ ≤ ξ , ( 125 ) where recall the definition of Vk in Equation ( 60 ) . We assume enough overparameterization to make sure ℜ < 1 . Recall for the matrix A defined by A = ( ⟨x̄i1 , x̄i2⟩F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ) 1≤i1 , i2≤n , ( 126 ) we have ( f∗ ( xi ) ) n i=1 TA−1 ( f∗ ( xi ) ) n i=1 ≤ ζ2 . Then , there exists weight matrix V ∗ which only depends on the random initializations W ( 0 ) , V ( 0 ) ( e.g . not on V ′ and W ′ ) for the second layer , such that having enough overparameterization ∥V ∗∥2F ≤ 2ζ2 , ( 127 ) and for every j ∈ [ m2 ] : ∥V ∗j ∥∞ ≤ ( 1 + ℜ ) n √ nζ2√ m2 ξ : = ϱ3ξ/ √ m2 , ( 128 ) ∥V ∗j ∥2 ≤ 1√ m2 n ( 1 + ℜ ) √ ζ2 ∑ k ∥Vk∥2H∞ λ0 : = ϱ2/ √ m2 , ( 129 ) and further under the high probability event Ec defined in Lemma 33 : | 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) − f∗ ( xi ) | ≲ ( C1√ m3κ1 ) 1/3 ( 1 + ℜ ) √ ζ2 ∑ k ∥Vk∥2H∞ : = ℜ3 . ( 130 ) Proof of Lemma 10 Let V ∗ = n∑ i=1 V∗i Li , be the minimum norm vector which maps Li ’ s to f∗ ( xi ) ’ s . As a result , for the matrix L = ( ⟨Li1 , Li2⟩ ) i1 , i2 it is easy to see ∥V ∗∥2F = ( f∗ ( xi ) ) ni=1TL−1 ( f∗ ( xi ) ) ni=1 . Now combining Lemmas 5 and 41 , we get ∥ϕ∗ ( xi ) ∥∞ ≤ ( 1 + ℜ ) ξ , ( 131 ) ∥ϕ∗ ( xi ) ∥ ≤ ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ , ( 132 ) and ⟨|ϕ∗ ( xi1 ) , ϕ∗ ( xi2 ) ⟩ − ⟨x̄i1 , x̄i2⟩| ≤ ( 2ℜ+ ℜ2 ) ∑ k ∥Vk∥2H∞ . Now by Lemma 9 : |⟨Li1 , Li2⟩ −Ai1 , i2 | ≲ ( 2ℜ+ ℜ2 ) ( ∑ k ∥Vk∥2H∞ ) ∣∣∣F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ∣∣∣+ ⟨x̄i1 , x̄i2⟩ ( m−1/41 +m−1/42 +m−1/43 ) + other cross term . By applying ⟨x̄i1 , x̄i2⟩ ≤ ∥x̄i1∥∥x̄i2∥ we get LHS ≤ ( ∑ k ∥Vk∥2H∞ ) ( ( 2ℜ+ ℜ2 ) ∣∣∣F2 ( 2F3 ( ⟨xi1 , xi2⟩ ) ) ∣∣∣+ ( m−1/41 +m−1/42 +m−1/43 ) ) ≲ ( ∑ k ∥Vk∥2H∞ ) ( ℜ+m−1/41 +m −1/4 2 +m −1/4 3 ) . Therefore , ∥A− ( ⟨Li1 , Li2⟩ ) i1 , i2 ∥2 ≤ ∥A− ( ⟨Li1 , Li2⟩ ) i1 , i2 ∥F ≤ n ( ∑ k ∥Vk∥2H∞ ) ( ℜ+m−1/41 +m −1/4 2 +m −1/4 3 ) : = ℜ2 . Note that ℜ2 naturally goes to zero ( with poly dependence ) as ℜ → 0 and m1 , m2 , m3 are large enough . Now if all of the eigenvalues of the matrix A are Ω ( 1/n2 ) , then if we overparameterize enough such that ℜ2 = O ( 1/n2 ) with small enough constant so that ℜ2 is less than half of the smallest eigenvalue of A , then for the ith eigenvalue λi of A and L we can write λi ( L ) ≥ λi ( A ) −ℜ2 ≥ λi ( A ) /2 , so λi ( L −1 ) ≤ 2λi ( A−1 ) , which implies the property ∥V ∗∥2F ≤ 2ζ2 . ( 133 ) However , A might have very small eigenvalues . To remedie this , we use Lemma 42 ; we can substitute f∗ with some f̄∗ such that Rn ( f̄∗ ) ≤ 2Rn ( f∗ ) + B2 n , ( 134 ) f̄∗ T A−1f̄∗ ≤ f∗TA−1f∗ , ( 135 ) where f̄∗ is on the subspace of eigenvectors of A whose eigenvalues are larger than Ω ( 1/n2 ) . But it is easy to check that in the context of Theorem 3 , such substitution results in a f̄∗TA−1f̄∗ ≤ f∗TA−1f∗ ≤ ζ and v̄ ( f̄∗ ) parameter ( as defined in ( 43 ) ) with respect to f̄∗ which satisfies ν̄/2 ≤ ν . This enables us to use remark ? ? with respect to f̄∗ . Note that the algorithm is with respect to the setting ν , however we want to exploit generalization bound with respect to f̄∗ whose parameter is ν̄ as it enables us to use our analysis in this Lemma . Furthermore , note that using Equation ( 134 ) we can further upper bound the empirical risk of f̄∗ with that of f∗ , which makes it straightforward to derive a similar generalization bound as in ( 47 ) with respect to f∗ , of course with a change of constants . Therefore , without loss of generality we can use substitute f∗ by f̄∗ and still obtain Equation ( 133 ) . On the other hand , the definition of V ∗ implies 1√ m2 aTDV ( 0 ) , xiV ∗ϕ∗ ( xi ) = f ∗ ( xi ) . But note that by Corollary 5.1 , under the high probability event Ec defined in Lemma 33 , DV ( 0 ) , xi and DV ( 0 ) +V ′ , xi can only be different in the index set P̃i and |P̃i| ≲ ( C21 ( m3κ21 ) ) 1/3 m2 , Therefore , for all i ∈ [ n ] : | 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) − 1√ m2 aTDV ( 0 ) , xiV ∗ϕ∗ ( xi ) | ≤1/ √ m2 ∑ j∈P̃ |V ∗j ϕ∗ ( xi ) | ≤1/ √ m2 ∑ j∈P̃ ∥V ∗j ∥∥ϕ∗ ( xi ) ∥ ≤ √ |P̃ | √ m2 ∥V ∗∥ ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ ≲ ( C1√ m3κ1 ) 1/3√ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ , which proves the first claim . On the other hand , we get : 2ζ2 ≥ ∥V ∗∥2F ≥ VTLV . But because λmin ( L ) ≳ 1/n2 , we get VTLV ≳ ∥V∥22/n2 , which implies ∥V∥2 ≲ n √ ζ2 . But now using Equation ( 131 ) , we can write |V ∗j , k| ≤ n∑ i=1 |Vi||Lij , k| ≤ 1√ m2 ∥ϕ∗ ( xi ) ∥∞ ∑ i |Vi| ≲ ( 1 + ℜ ) ξ√ m2 ∑ i |Vi| ≤ ( 1 + ℜ ) ξ √ n∥V∥/ √ m2 ≲ ( 1 + ℜ ) n √ nζ2√ m2 ξ , which proves the other part . Moreover , ∥V ∗j ∥2 ≤ ∥V∥2 1√ m2 ( ∑ i ∥ϕ∗ ( xi ) ∥22 ) ≤ 1√ m2 n ( 1 + ℜ ) √ ζ2 ∑ k ∥Vk∥2H∞/λ0 . ( 136 ) A.13 EXISTENCE OF A GOOD DIRECTION Our aim in this section is to show that if the objective value is above certain threshold , there exists a good random direction which reduces the objective in expectation . Particularly our aim is to prove the following theorem ( informal ) : Theorem 6 For a given pair ( f∗ , G ) with ⟨H∞ , G⟩ ≤ ζ1 , f∗T ( K∞ ⊙G ) −1f∗ ≤ ζ2 , Rn ( f ∗ ) ≤ ∆ , recall the ideal random matrices ( W ∗Σ , V ∗ Σ ) constructed in Appendix A.12 , where Σ is a random diagonal sign matrix . Specifically , W ∗Σ is defined in Equation ( 108 ) , and V ∗ Σ is the projection of the rows of matrix V ∗ onto the orthogonal subspace spanned by ( ϕ ( 0 ) ( xi ) ) ni=1 . Using the parameter setting for i = 1 , 2 ψi = ν 4ζi , ( 137 ) with respect to an arbitrary parameter ν > 0 , then for every pair ( W ′ , V ′ ) such that ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 and L ( W ′ , V ′ ) ≥ ∆+ ν , ( 138 ) for parametersm1 , m2 , m3 , 1/κ1 , 1/κ2 polynomially large enough in B , 1/λ0 , n , C1 , C2 and small enough step size η , we have EΣL ( W ′ − η/2W ′ + √ ηW ∗Σ , V ′ − η/2V ′ +√ηV ∗Σ ) ≤ L ( W ′ , V ′ ) − ην/4 . ( 139 ) In order to prove the above theorem , we first state and prove the following lemma which is the core of Theorem 6 . Lemma 11 For matrices ( W ∗ , V ∗ ) constructed in Appendix A.12 , specifically for their random coupling ( W ∗Σ , V ∗ Σ ) as denoted above , we have : EΣℓ ( f ′ ( 1−η/2 ) W ′+√ηW∗Σ , ( 1−η/2 ) V ′+ √ ηV ∗Σ ( xi ) , yi ) ≤ ( 1− η ) ℓ ( f ′W ′ , V ′ ( xi ) , yi ) + ηℓ ( f∗ ( xi ) , yi ) ± η℘ , where ℘ goes to zero with polynomially large overparameterization ( the exact dependence is revealed via the proof ) . Proof of Lemma 11 For brevity , we use the notation D′ , ρ here to refer to the diagonal binary sign matrix when the input is multiplied by the sum of weight and smoothing matrices . It will be clear in the context of the equation that what the “ input ” and the “ weight ” matrices are . This notation is also defined and used in Lemma 28 ) . Here , we bound multiple cross terms that are created as a result of moving in the random direction . To simplify the presentation and avoid confusing recursions in the proof , we have made a sublemma for each of these cross terms and has deferred its proof to Appendix A.14 . We use difference sub-indices of the symbol ℜ to illustrate terms that go to zero by growing the overparameterization in our architecture . We start by using Lemma 28 , EΣℓ ( f ′ ( 1−η/2 ) W ′+√ηW∗Σ , ( 1−η/2 ) V ′+ √ ηV ∗Σ ( xi ) , yi ) = EΣℓ ( EWρ , V ρf ( 1−η/2 ) W ′+√ηW∗Σ+Wρ , ( 1−η/2 ) V ′+ √ ηV ∗Σ+V ρ ( xi ) , yi ) = EΣℓ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ + √ ηV ∗Σ ) W sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ +√ηW ∗Σ ) xi + ℜ8η , yi ) . = EΣℓ ( EWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi + ηaTD′ , ρV ∗ ΣW sD′ , ρW ∗ Σxi ] + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] + ℜ8η , yi ) . Now using the notation introduced in Lemma 18 , we have W sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi = ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) + η 2 ϕ ( 2 ) ′ ( xi ) . By Lemma 18 , we have the following bound for ϕ ( 2 ) ′ ( xi ) : EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5 . Therefore , Combining this with Lemma 20 , we get = EΣℓ ( EWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W s ( ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) ) + ηaTD′ , ρV ∗ ΣW sD′ , ρW ∗ Σxi ] + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) = EΣℓ ( EWρ , V ρ [ ( 1− η ) aTD′ , ρ ( V ( 0 ) + V ′ + V ρ ) W s ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) + ηaTD′ , ρV ∗ ΣW sD′ , ρW ∗ Σxi ] + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) . = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗ΣW sD′ , ρW ∗Σxi + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) . Moreover , using the notation ϕ∗′ ( xi ) introduced in Lemma 15 and the bound in Lemma 17 , we can rewrite the second term as : LHS = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σ ( ϕ∗ ( xi ) + ϕ∗ ′ ( xi ) ) + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ±O ( ℜ8η ) , yi ) = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) + √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] ±O ( ηℜ10 ) ±O ( η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ) ±O ( ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5η ) ± O ( ℜ8η ) , yi ) . ( 140 ) Now we write the gradient-lipshitz inequality for ℓ at point p ( 1 ) Σ : = ( 1− η ) f ′ W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) ± η℘1 , and regarding the following vector , where ℘1 is the sum of all the noise terms above and goes to zero by over parameterization : p ( 2 ) Σ : = √ ηEWρ , V ρ [ aTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi + aTD′ , ρV ∗ ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ] . Hence , using the 1 smoothness of ℓ ( . , yi ) : LHS ≤ EΣℓ ( p ( 1 ) Σ ) + EΣℓ̇ ( p ) √ ηp ( 2 ) Σ + 1 2 ηEΣ ( p ( 2 ) Σ ) 2 . ( 141 ) But note that EΣℓ̇ ( p ) √ ηp ( 2 ) Σ = ℓ̇ ( p ) √ ηEΣp ( 2 ) Σ = 0 . ( 142 ) On the other hand , using the notation of Lemma 15 and the result of Lemma 16 : EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) W sD′ , ρW ∗Σxi ) 2 ( 143 ) = EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) ( Σϕ∗ ( xi ) + ϕ∗′Σ ( xi ) ) ) 2 ( 144 ) ≤ 4EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) Σϕ∗ ( xi ) ) 2 ( 145 ) + 4EΣ ( EWρ , V ρaTD′ , ρ ( V ( 0 ) + ( 1− η/2 ) V ′ + V ρ ) ϕ∗′Σ ( xi ) ) 2 ( 146 ) ≲ ℜ212 + ℜ211 . ( 147 ) Moreover , using again the result on ϕ ( 2 ) ′ ( xi ) from Lemma 18 and the fact that ϕ ( 0 ) ( xi ) is orthogonal to the rows of V ∗Σ : EWρ , V ρaTD′ , ρV ∗ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi = aTD′ , ρV ∗ Σ ( ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) ) + η 2 aTD′ , ρV ∗ Σϕ ( 2 ) ′ ( xi ) = ( 1− η 2 ) aTD′ , ρV ∗ Σϕ ( 2 ) ( xi ) + η 2 aTD′ , ρV ∗ ΣW sD′ , ρϕ ( 2 ) ′ ( xi ) ≲ ( 1− η 2 ) aTD′ , ρV ∗ Σϕ ( 2 ) ( xi ) ±ℜ5 . Combining the last Equation with Lemma 14 : EΣ ( EWρ , V ρaTD′ , ρV ∗ΣW sD′ , ρ ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ ) xi ) 2 ≲ ( 1− η 2 ) 2aTD′ , ρV ∗ ΣW sD′ , ρϕ ( 2 ) ( xi ) + ℜ25 ≲ ℜ20 + ℜ25 . ( 148 ) Combining Equations ( 147 ) and ( 148 ) : EΣ ( p ( 2 ) Σ ) 2 ≲ ℜ211 + ℜ212 + ℜ20 + ℜ25 : = ℘2 . ( 149 ) Combining Equations ( 142 ) and ( 149 ) , plugging into ( 141 ) , and reopening the definition of p ( 1 ) Σ : LHS ≲ EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) ± η℘1 , yi ) + ηµ2℘2 . ( 150 ) Now note that we can easily bound the magnitude of the term ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) as : |EWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) | ≤ EWρ , V ρ |aTD′ , ρV ∗Σϕ∗ ( xi ) | ≤ ∥V ∗Σ∥F ∥ϕ∗ ( xi ) ∥ ≤ ∥V ∗∥ϕ∗ ( xi ) ∥ ≤ √ 2ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ , while using Lemma 34 : |f ′W ′ , V ′ ( xi ) | ≤ ( κ2 √ m3 + β2 ) ( √ m3κ1 + C1 + √ m3β1 ) + C2 ( C1 + √ m3β1 ) , which is O ( C1C2 ) for enough overparameterization and smoothing parameters β1 , β2 as defined in A.20.1 . Furthermore , from Equations ( 132 ) and ( 127 ) , we easily see that EWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) ≤ √ 2ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . Now taking η small enough so that the bound η √ 2ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ and η℘1 both also be bounded of order O ( C1C2 ) , we observe that the term inside the argument of ℓ ( . , yi ) Equation in ( 150 ) is O ( C1C2 ) . Hence , we can use the lipschitz parameter of ℓ in the interval [ −O ( C1C2 ) , O ( C1C2 ) ] , given by Lemma 9 to take out the noise term : LHS ≲ EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηEWρ , V ρaTD′ , ρV ∗Σϕ∗ ( xi ) , yi ) ± η℘1 + ηO ( C1C2 +B ) ℘2 . ( 151 ) Now by applying Lemma 19 and writing the Lipchitz property of ℓ at point ( 1 − η ) f ′W ′ , V ′ ( xi ) = O ( C1C2 ) : LHS ≲ EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηf∗ ( xi ) ± ηℜ9 , yi ) ± η℘1 + ηO ( C1C2 +B ) ℘2 : = EΣℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηf∗ ( xi ) , yi ) ± ηℜ9 ± η℘1 + ηO ( C1C2 +B ) ℘2 = ℓ ( ( 1− η ) f ′W ′ , V ′ ( xi ) + ηf∗ ( xi ) , yi ) ± η℘ , where the last line is just definition . Now Convexity of ℓ finishes the proof . Next , using Lemma 11 we prove Theorem 6 . Restating Theorem 6 In the same setting as Theorem 6 and having enough overparameterization such that ℘ ≤ ν8 ( ℘ defined in Lemma 11 ) and polynomially small enough step size η , we have EΣL ( W ′ − ηW ′ + √ ηW ∗Σ , V ′ − ηV ′ +√ηV ∗Σ ) ≤ L ( W ′ , V ′ ) − ην/4 . Proof of Theorem 6 First , note that taking expectation w.r.t Σ : EΣ∥ ( 1− η/2 ) W ′ + √ ηW ∗Σ∥2 = EΣ ( 1− η/2 ) 2∥W ′∥2 + 2 ( 1− η/2 ) √ η⟨W ′ , m3∑ k=1 ΣkW ∗ k ⟩+ η∥ m3∑ k=1 ΣkW ∗ k ∥2 = ( 1− η/2 ) 2∥W ′∥2 + η ∑ k ∥W ∗k ∥2 , which by orthogonality of W ∗k ’ s : LHS = ( 1− η/2 ) 2∥W ′∥2 + η∥W ∗∥2 = ( 1− η ) ∥W ′∥2 + η∥W ∗∥2 + η2∥W ′∥2 . Similarly for V ′ : EΣ∥ ( 1−η/2 ) V ′+ √ ηV ∗Σ∥ = ( 1−η/2 ) 2∥V ′∥2+ηEΣ∥V ∗Σ∥2 = ( 1−η ) ∥V ′∥2+η∥V ∗∥2+η2∥V ′∥2 . Now using Lemma 11 : EΣL ( W ′ − η/2W ′ + √ ηW ∗Σ , V ′ − η/2V ′ +√ηV ∗Σ ) ≤ ( 1− η ) EZℓ ( f ′W ′ , V ′ ( x ) , y ) + ηEZℓ ( f∗ ( x ) , y ) + ( 1− η ) ( ψ1∥W ′∥2 + ψ2∥V ′∥2 ) + η ( ψ1∥W ∗∥2 + ψ2∥V ∗∥2 ) + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) ≤ L ( W ′ , V ′ ) − η ( L ( W ′ , V ′ ) −∆− ψ1ζ1 − ψ2ζ2 ) + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) , which by the choice of ψi ’ s is equal to LHS ≤ L ( W ′ , V ′ ) − η ( L ( W ′ , V ′ ) −∆− ν/2 ) + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) LHS ≤ L ( W ′ , V ′ ) − ην/2 + η ( ℘+ η ( ∥W ′∥2 + ∥V ′∥2 ) ) . Moreover , using the condition ℘ ≤ ν/8 , and picking η as small as η ( ∥W ′∥2 + ∥V ′∥2 ) ≤ η ( C21 + C22 ) ≤ ν/8 , we finally get LHS ≤ L ( W ′ , V ′ ) − ην/4 . A.14 EXISTENCE OF A GOOD DIRECTION HELPER LEMMAS In this section , we state and prove the core lemmas that are used in the proof of Lemma 11 . Notably , through all of this section , we assume the norm bounds ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 and that as our usual assumption , the rows of V ′ are orthogonal to ϕ ( 0 ) ( xi ) ’ s for all i ∈ [ n ] . A notation that we use throughout the proofs is V ∗Σ which refers to the projectiono of V ∗Σ onto the orthogonal subspace to ( ϕ ( 0 ) ( xi ) ) n i=1 . Lemma 12 Let P ( . ) be the projection operator onto the subspace spanned by ( ϕ ( 0 ) ( xi ) ) ni=1 . Also , we denote the projection of rows of V ∗Σ onto the orthogonal subspace to ( ϕ ( 0 ) ( xi ) ) ni=1 by V ∗ Σ j . Then EΣ∥V ∗Σ j − V ∗ j Σ ) ∥2 ≤ ϱ23ξ2n/m2 , with high probability ∥V ∗Σ j − V ∗ j Σ ) ∥ ≲ ϱ3ξ √ n√ m2 , Proof of Lemma 12 By Equation ( 129 ) , we have ∥V ∗j ∥∞ ≤ ϱ3ξ/ √ m2 . Now suppose that u1 , ... , un are an orthonormal basis for the subspace span ( ϕ ( 0 ) ( xi ) ) ni=1 . Then EΣ∥V ∗Σ j − V ∗ j Σ ) ∥2 = EΣ∥P ( V ∗j Σ ) ∥2 = ∑ i ∑ k V ∗j 2 k ui 2 k ≤ ∥V ∗j ∥2∞n ≤ ϱ23ξ2n/m2 . Also , by Hoeffding , with high probability : ∥P ( V ∗j Σ ) ∥2 = ∑ i ( m3∑ k=1 V ∗j kuikΣk ) 2 ≲ n∥V ∗j ∥2∞ , which implies the second part . Lemma 13 The first cross term goes away because of the definition of V ∗Σ . ( inside the expectations is zero almost surely ) EΣ ( EV ρ , Wρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 0 ) ( xi ) ] ) 2 = 0 . Lemma 14 Second cross term : EΣ ( EV ρ , Wρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ( xi ) ] ) 2 ( 152 ) ≲ ξ2 ( ( 1 + ℜ ) 2nζ2 + ϱ23n ) ( C21 +m3β21 ) = ℜ20 . ( 153 ) Proof of Lemma 14 This time we use Equation ( 129 ) in Lemma 10 and Lemma 12 : EΣ ( EV ρ , Wρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ( xi ) ] ) 2 ≤ EΣ ( EV ρ , Wρ1/ √ m2 ∑ j |V ∗Σ jϕ ( 2 ) ( xi ) | ) 2 ≤ 1 m2 EΣ , V ρ , Wρ ( ∑ j |V ∗Σ jϕ ( 2 ) ( xi ) | ) 2 ≤ EV ρ , WρEΣ ∑ j |V ∗Σ jϕ ( 2 ) ( xi ) |2 ≲ EV ρ , WρEΣ ∑ j | ( V ∗Σ j − V ∗ j Σ ) ϕ ( 2 ) ( xi ) |2 + ∑ j |V ∗j Σϕ ( 2 ) ( xi ) |2 ≲ EV ρ , WρEΣ ∑ j ∥V ∗Σ j − V ∗ j Σ ) ∥2∥ϕ ( 2 ) ( xi ) ∥2 + ∑ j ∥V ∗j ∥2∞∥ϕ ( 2 ) ( xi ) ∥22 ≲ ( ( 1 + ℜ ) 2nζ2ξ2 + ϱ23ξ2n ) EV ρ , Wρ∥ϕ ( 2 ) ( xi ) ∥22 . Now according to Lemma 33 , we have EV ρ , Wρ∥ϕ ( 2 ) ( xi ) ∥2 ≲ C12 +m3β12 , which completes the proof . Lemma 15 We get an additional term ϕ∗′ ( xi ) as a result of smoothing which we define as ϕ∗′ ( xi ) = 1√ m1 W sDW ( 0 ) +W ′+Wρ , xiW ∗ Σxi − ϕ∗Σ ( xi ) . ( 154 ) Then P ( ϕ∗′ ( xi ) ̸= 0 ) ≤ m1 exp { −c22/ ( 8β21 ) } . Moreover , we have the following inequality almost surely ( over the randomness of W ρ ) : ∥ϕ∗′ ( xi ) ∥∞ ≲ √∑ k ∥Vk∥2H∞ . Proof of Lemma 15 According to Lemma 1 , for j /∈ P , for every i ∈ [ n ] we have | ( W ( 0 ) j +W ′ j ) xi| ≥ c2/2 √ m1 . Now note that as long as the sign patterns for j /∈ P does not change , ϕ∗′ ( xi ) will be zero . Therefore by union bound P ( ϕ∗′ ( xi ) ̸= 0 ) ≤ m1∑ j=1 P ( sign change in j ) ≤ m1P ( | ( W ( 0 ) j +W ′ j ) xi| ≤ |W ρ j xi| ) ≤ m1P ( |W ρj xi| ≥ c2/ ( 2 √ m1 ) ) . But ( W ρj ) xi is Gaussian with variance β 2 1/m1 . Hence LHS ≲ m1 exp { −c22/ ( 8β21 ) } , which proves the first part . For the second part , according to Equation ( 106 ) in Lemma 5 , for every k ∈ [ m3 ] : |ϕ∗′k ( xi ) | ≤ | 1√ m1 W skDW ( 0 ) +W ′+Wρ , xiW ∗xi|+ |ϕ∗k ( xi ) | ( 155 ) ≤ 2/ √ m1 ∑ j ∥W ∗j ∥ ≤ 2∥W ∗∥F ≲ √∑ k ∥Vk∥2H∞ , ( 156 ) which implies the second part . Lemma 16 Fourth Extra term : EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ∗′ ( xi ) ] ≲ ( κ2 √ m2m3 + C2 + √ m3β2 ) m1 exp { −c22/ ( 8β21 ) } √∑ k ∥Vk∥2H∞ : = ℜ11 . Proof of Lemma 16 Note that with high probability over the randomness of V ( 0 ) , we have ∥V ( 0 ) ∥F ≲ √ m2m3κ2 . Now according to Lemma 15 and using the fact that ∥V ′∥F ≤ C2 : ≤ EWρ , V ρ 1√ m2 ∥a∥∥V ( 0 ) + V ρ + ( 1− η ) V ′∥2∥ϕ∗′ ( xi ) ∥ = EWρ , V ρ∥V ( 0 ) + V ρ + ( 1− η/2 ) V ′∥F ∥ϕ∗′ ( xi ) ∥ ≤ √ EV ρ ( ∥V ( 0 ) + ( 1− η/2 ) V ′∥2F + ∥V ρ∥2F ) m1 exp { −c 2 2/ ( 8β 2 1 ) } √∑ k ∥Vk∥2H∞ ≲ √ ∥V ( 0 ) ∥2F + ∥V ′∥2F +m3β22 m1 exp { −c 2 2/ ( 8β 2 1 ) } √∑ k ∥Vk∥2H∞ ≲ ( κ2 √ m2m3 + C2 + √ m3β2 ) m1 exp { −c22/ ( 8β21 ) } √∑ k ∥Vk∥2H∞ . Lemma 17 Fifth extra term : ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ∗′ ( xi ) ] ∣∣∣ ≲√ζ2m1 exp { −c22/ ( 8β21 ) } √∑ k ∥Vk∥2H∞ : = ℜ10 . Proof of Lemma 17 Similar to the previous Lemma , the inner expectation can be bounded as : ≤ E 1√ m2 ∥a∥∥V ∗Σ∥F ∥ϕ∗ ′ ( xi ) ∥ ≤ E∥V ∗∥F ∥ϕ∗′ ( xi ) ∥ ≲ √ ζ2m1 exp { −c 2 2/ ( 8β 2 1 ) } √∑ k ∥Vk∥2H∞ . Lemma 18 We have another extra term as a product of the movement −η/2W ′ in the first layer : ϕ ( 2 ) ′ ( xi ) = 2 η √ m1 [ W sDW ( 0 ) +Wρ+W ′ ( W ( 0 ) +W ρ+ ( 1−η/2 ) W ′ ) xi−ϕ ( 0 ) ( xi ) − ( 1−η/2 ) ϕ ( 2 ) ( xi ) ] . Then ∣∣∣EWρ , V ρ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ √ ζ2m3 ( β21 m1 + c2C21√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 ) : = ℜ5 . ( 157 ) EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5 . ( 158 ) Proof of Lemma 18 First we prove the following approximation argument ( for all k ∈ [ m3 ] ) : EWρ |ϕ ( 2 ) ′ ( xi ) k|2 ≤ β21 m1 + c2C 2 1√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 . ( 159 ) We have LHS = EWρ ∣∣∣ 1√ m1 W skDW ( 0 ) +Wρ+W ′ , xi ( W ( 0 ) +W ρ ) xi − 1√ m1 W skDW ( 0 ) , xiW ( 0 ) xi ∣∣∣2 ≲ EWρ ∣∣∣ 1√ m1 W skDW ( 0 ) +Wρ , xi ( W ( 0 ) +W ρ ) xi − 1√ m1 W skDW ( 0 ) , xiW ( 0 ) xi ∣∣∣2 +EWρ ∣∣∣ 1√ m1 W skDW ( 0 ) +Wρ , xi ( W ( 0 ) +W ρ ) xi − 1√ m1 W skDW ( 0 ) +Wρ+W ′ , xi ( W ( 0 ) +W ρ ) xi ∣∣∣2 By the independence of W ρj ’ s , the first term can be upper bounded as = EWρ 1 m1 m1∑ j=1 ( ( W ( 0 ) j +W ρ j ) xi1 { ( W ( 0 ) j +W ρ j ) xi ≥ 0 } −W ( 0 ) j xi1 { W ( 0 ) j xi ≥ 0 } ) 2 = ≤ 1 m1 m1∑ j=1 EWρ |W ρj xi| 2 = 1 m1 ∑ β21 m1 = β21 m1 . For the second term , note that for every j /∈ P , the jth entries of DW ( 0 ) +Wρ , xi and DW ( 0 ) +Wρ+W ′ , xi are different only if W ρ j can make a sign change in the jth row , i.e . | ( W ( 0 ) j + W ′j ) xi| ≤ |W ρ j xi| should happen . We denote this event for every j /∈ P by Ẽj . Furthermore , if this happens for some j , then the value of ( W ( 0 ) +W ρ ) jxi is upper bounded by |W ′jxi| . Now similar to our discussion in Lemma 15 and using the result of Lemma 1 : P ( ∪j /∈P Ẽj ) = P ( sign change in some j /∈ P ) ≤ m1∑ j /∈P P ( sign change in j ) ≤ m1P ( | ( W ( 0 ) j +W ′ j ) xi| ≤ |W ρ j xi| ) ≤ m1P ( |W ρ j xi| ≥ c2/ ( 2 √ m1 ) ) . But note that ( W ρj ) xi is Gaussian with variance β 2 1/m1 . Hence LHS ≲ m1 exp { −c22/ ( 8β21 ) } , So finally we can write ≲ β 2 1 m1 + EWρ 1 m1 ( ∑ j∈P |W ′jxi| ) 2 + EWρ 1 m1 ( 1 { ∪j /∈P Ẽj } ∑ j /∈P |W ′jxi| ) 2 ≲ β 2 1 m1 + |P | m1 ∥W ′∥2 + P ( ∪j /∈P Ẽj ) ∥W ′∥2 ≤ β 2 1 m1 + c2C 2 1√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 . which completes the proof for Equation ( 159 ) . This immediately implies EWρ∥ϕ ( 2 ) ′ ( xi ) ∥ ≤ √ EWρ∥ϕ ( 2 ) ′ ( xi ) ∥2 ≤ √ m3 ( β21 m1 + c2C21√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 ) . Now we first prove Equation ( 157 ) : ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ( 2 ) ′ ( xi ) ] ∣∣∣ ≤ EWρ 1√ m2 ∥a∥∥DV ( 0 ) +V ρ+V ′ , xiV ∗ Σ∥F ∥ϕ ( 2 ) ′ ( xi ) ] |∥ ≤ ∥V ∗Σ∥FEWρ∥ϕ ( 2 ) ′ ( xi ) ] |∥ ≤ ∥V ∗∥FEWρ∥ϕ ( 2 ) ′ ( xi ) ] |∥ ≲ √ ζ2m3 ( β21 m1 + c2C21√ m1κ1 +m1 exp { −c22/ ( 8β21 ) } C21 ) . To prove Equation ( 158 ) : EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 2 ) ′ ( xi ) ∣∣∣ ≲ EWρ , V ρ 1√ m2 ∥a∥∥DV ( 0 ) +V ρ+ ( 1−η/2 ) V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ∥F ∥ϕ ( 2 ) ′ ( xi ) ∥ ≲ EWρ , V ρ 1√ m2 ∥a∥∥V ( 0 ) + V ρ + ( 1− η/2 ) V ′∥F ∥ϕ ( 2 ) ′ ( xi ) ∥ ≲ 1√ m2 ∥a∥ √ EV ρ ( ∥V ( 0 ) ∥2 + ∥V ρ∥2F + ∥ ( 1− η/2 ) V ′∥2F ) EWρ∥ϕ ( 2 ) ′ ( xi ) ∥ ≲ √ ( κ22m2m3 +m3β 2 2 + C 2 2 ) ℜ5 ≲ ( κ2 √ m2m3 + √ m3β2 + C2 ) ℜ5 . Lemma 19 Closeness condition : EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ∗ Σ ( xi ) ] − f∗ ( xi ) ∣∣∣ ≲ ℜ9 , where ℜ9 : = ϱ3ξ √ n ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ + ℜ3 ( 160 ) +m2 ( exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } +m1 exp { −C21/ ( 8m3β21 ) } ) √ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . ( 161 ) Proof of Lemma 19 Note that by Corollary 5.1 and according to the proof of Equation 130 in Lemma 10 , if for every j /∈ P̃ we don ’ t have a sign change in DV ( 0 ) +V ρ+V ′ , xiV ∗ϕ∗ ( xi ) , then get | 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ϕ∗ ( xi ) − f∗ ( xi ) | ≤ ℜ3 . Also , note that we need the event Ec ( defined in Lemma 33 ) to happen in order to be able to use Corrolary 5.1 . Hence , given a W ρ for which Ec happens , we upper bound the probability of sign change with respect to the randomness of V ρ . We define the following event with respect to the randomness of V ρ when conditioned on a W ρ for which Ec happens ( Pi ’ s are defined in Lemma 7 ) : SC : = { ∃j /∈ Pi s.t.|V ρj x ′ i| ≳ ( κ2 m2 ) 1/3C 2/3 2 ∥x′i∥ } . Now from the result in Corollary 5.1 we have ≤ 1 { sign change in j /∈ Pi } ≤ 1 { SC } . Therefore , E1 { sign change } ≤ 1 { SC } ≤ ∑ j /∈Pi P ( |V ρj x ′ i| ≳ ( κ2 m2 ) 1/3C 2/3 2 ∥x′i∥ ) ≤ m2P ( |V ρj x ′ i| ≳ ( κ2 m2 ) 1/3C 2/3 2 ∥x′i∥ ) . But note that ( V ρj ) x ′ i is Gaussian with variance β 2 2∥x′i∥2/m2 . Hence LHS ≲ m2 exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } . ( 162 ) Now let D be a sign matrix random variable such that if Ec and SCc both happens , then it is equal to the valid sign matrixDV ( 0 ) +V ρ+V ′ , xi , and otherwise it is equal to an arbitrary valid sign matrix in the case when both Ec and SCc happen . Now using Equation ( 117 ) we have with high probability over the initialization : EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ Σϕ ∗ Σ ( xi ) ] − f∗ ( xi ) ∣∣∣ ≤ EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ∗ Σ − V ∗Σ ) ϕ∗Σ ( xi ) ] + EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗Σϕ∗Σ ( xi ) ] − f∗ ( xi ) ≤ EWρ , V ρEΣ ∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ∗ Σ − V ∗Σ ) ϕ∗Σ ( xi ) ∣∣ + EΣ ∣∣∣EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xiV ∗ϕ∗ ( xi ) ] − f∗ ( xi ) ∣∣∣ ≤ EWρ , V ρEΣ 1 m2 ∑ j ∥V ∗Σ j − V ∗ j Σ∥∥ϕ∗Σ ( xi ) ∥ + EΣ ∣∣∣EWρ , V ρ [ ( 1√ m2 aTDV ∗ϕ∗ ( xi ) − f∗ ( xi ) ) + 1 { SC ∪ E } ( 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) −D ) ] ∣∣∣ ≤ EWρ , V ρEΣ 1 m2 ∑ j ∥V ∗Σ j − V ∗ j Σ∥∥ϕ∗Σ ( xi ) ∥ + EΣ ∣∣∣EWρ , V ρ [ ( 1√ m2 aTDV ∗ϕ∗ ( xi ) − f∗ ( xi ) ) ∣∣∣ + EΣEWρ , V ρ ∣∣∣1 { SC ∪ E } ( 1√ m2 aTDV ( 0 ) +V ′ , xiV ∗ϕ∗ ( xi ) − 1√ m2 aTDV ∗ϕ∗ ( xi ) ) ] ∣∣∣ ≤ EWρ , V ρ 1√ m2 ∑ j √ EΣ∥V ∗Σ j − V ∗ j Σ∥2∥ϕ ∗ ( xi ) ∥ + ℜ3 + 2P ( SC ∪ E ) max D′ ∣∣∣ 1√ m2 aTD′V ∗ϕ∗ ( xi ) ∣∣∣ . Now note that for any sign matrix D′ , we have the following bound : ∣∣∣ 1√ m2 aTD′V ∗ϕ∗ ( xi ) ∣∣∣ ≤ 1√ m2 ∥a∥∥V ∗∥F ∥ϕ∗ ( xi ) ∥ ≲ √ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . Also , applying a union bound and using Lemmas 33 P ( SC ∪ E ) ≤ P ( SC ) + P ( E ) ≲ exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } +m1 exp { −C21/ ( 8m3β21 ) } . Hence , also applying Lemma 40 , we further write LHS ≲ ϱ3ξ √ n ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ + ℜ3 +m2 ( exp { − ( m2κ22C42 ) 1/3/ ( 2β22 ) } +m1 exp { −C21/ ( 8m3β21 ) } ) √ ζ2 ( 1 + ℜ ) √∑ k ∥Vk∥2H∞ . Lemma 20 Suppose we have m3κ21 ≥ C21 . Then , for the following basic term we have : EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ( ϕ ( 0 ) ( xi ) + ( 1− η/2 ) ϕ ( 2 ) ( xi ) ) ≲ ( 1− η ) EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ±η ( ℜ′6 + ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) , where ℜ4 : = C2 ( C1 + √ m3β1 ) m2 exp { −C 4/3 2 ( √ m2κ2 ) 2/3/8β22 } + C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1 + √ m3β1 ) , ℜ′6 : = m1 exp { −C21/ ( 8m3β21 ) } √ m3κ1 ( √ m2 + β2 ) + ℜ6 , and ℜ6 is defined in Lemma 22 . Proof of Lemma 20 First , note that by orthogonality of ϕ ( 0 ) ( xi ) to the rows of V ′ : LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) = LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ϕ ( 0 ) ( xi ) = ( 1− η/2 ) EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η/2 ) V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] = ( 1− η/2 ) 2EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] + ( 1− η/2 ) ( η/2 ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ( 163 ) But note that for the second term : EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ≲ EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ± 1√ m2 ∑ j : sign change |V ′j ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) | = EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ , xi ( V ( 0 ) + V ρ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) ] ± 1√ m2 ∑ j : sign change |V ′jϕ ( 2 ) ( xi ) | . ( 164 ) Now conditioned on x′i , by the result of Lemma 31 we know there exists a set of indices O ⊆ [ m2 ] , s.t . |O| ≤ C 2/3 2 ( √ m2κ2 ) 2/3m2 and for j /∈ O we have |V ( 0 ) j x ′ i| ≥ C 2/3 2 ( √ m2κ2 ) 1/3 √ m2 ∥x′i∥ and |V ′j x′i| ≤ C 2/3 2 ( √ m2κ2 ) 1/3 2 √ m2 ∥x′i∥ . Now for j ∈ [ m2 ] , define the event Rj = { |W ρj x ′ i| ≥ C 2/3 2 ( √ m2κ2 ) 1/3 2 √ m2 ∥x′i∥ } , and R = ∪jRj . First , note that using Gaussian tail bound , R is a rare event : P ( R ) ≤ ∑ j P ( Rj ) ≤ m2 exp { −C4/32 ( √ m2κ2 ) 2/3/8β22 } . Now for j /∈ O and underRc , clearly we have that the signs of ( V ( 0 ) j +V ρ j ) x ′ i and ( V ( 0 ) j +V ρ j +V ′ j ) x ′ i are the same . Therefore , applying Lemma 33 , we can argue under Rc : EWρ , V ρ 1√ m2 ∑ j : sign change |V ′jϕ ( 2 ) ( xi ) | ≤ EWρ 1√ m2 ∑ j∈O |V ′jϕ ( 2 ) ( xi ) | ≤ √ |O| m2 ∥V ′∥EWρ∥ϕ ( 2 ) ( xi ) ∥ ≤ C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1 + √ m3β1 ) . Hence , overall , using Cauchy-Shwartz EWρ , V ρ 1√ m2 ∑ j : sign change |V ′jϕ ( 2 ) ( xi ) | ≤ ∥V ′∥EWρ∥ϕ ( 2 ) ( xi ) ∥P ( R ) + C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1+ √ m3β1 ) ≤ C2 ( C1 + √ m3β1 ) m2 exp { −C 4/3 2 ( √ m2κ2 ) 2/3/8β22 } + C 1/3 2 ( √ m2κ2 ) 1/3 C2 ( C1 + √ m3β1 ) : = ℜ4 . ( 165 ) On the other hand , using Lemma 30 , we have with high probability over the randomness of initialization 1√ m2 aTDV ( 0 ) , xiV ( 0 ) ϕ ( 2 ) ( xi ) ≤ √ m3κ2∥ϕ ( 2 ) ( xi ) ∥ . Hence : EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ , xi ( V ( 0 ) + V ρ ) ϕ ( 2 ) ( xi ) ] ≤ EWρ , V ρ [ 1√ m2 aTDV ( 0 ) , xiV ( 0 ) ϕ ( 2 ) ( xi ) + 1√ m2 ∑ j |V ρj ϕ ( 2 ) ( xi ) | ] ≤ EWρ 1√ m2 aTDV ( 0 ) , xiV ( 0 ) ϕ ( 2 ) ( xi ) + β2EWρ∥ϕ ( 2 ) ( xi ) ∥ ≲ ( √ m3κ2 + β2 ) EWρ∥ϕ ( 2 ) ( xi ) ∥ ≲ ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) . ( 166 ) Combining Equations ( 165 ) and ( 166 ) into Equation ( 164 ) : ∣∣∣EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 2 ) ( xi ) ] ∣∣∣ ≲ ℜ4 + ( √m3κ2 + β2 ) ( C1 +√m3β1 ) . ( 167 ) Moreover , for the first term in ( 163 ) , using Equation ( 166 ) and Lemmas 33 and Lemma 30 we have |EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] | ≲ |EWρ , V ρ 1√ m2 aTDV ( 0 ) V ( 0 ) ϕ ( 2 ) ( xi ) |+ EWρ , V ρ 1√ m2 ∑ j |V ρj ϕ ( 2 ) ( xi ) |+ 1√ m2 ∑ j |V ′jϕ ( 2 ) ( xi ) | ≲ κ2 √ m3 ( C1 + β1 √ m3 ) + ( C2 + β2 ) EWρ∥ϕ ( 2 ) ( xi ) ∥ ≲ κ2 √ m3 ( C1 + β1 √ m3 ) + ( C2 + β2 ) ( C1 + √ m3β1 ) . ( 168 ) Substituting Equations ( 167 ) and ( 168 ) into Equation ( 163 ) , we finally get LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ≲ ( 1− η/2 ) 2EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ± η 2 ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ≲ ( 1− η ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ± η 2 4 ∣∣∣EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ∣∣∣ ± η 2 ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) ≲ ( 1− η ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ± η2 ( κ2 √ m3 ( C1 + β1 √ m3 ) + ( C2 + β2 ) ( C1 + √ m3β1 ) ) ± η ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) . ( 169 ) Now by picking η small enough so that the second term is dominated by the third term we get : LHS − η/2EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ( 170 ) ≲ ( 1− η ) EV ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + V ′ ) ϕ ( 2 ) ( xi ) ] ( 171 ) ± η ( ℜ4 + ( √ m3κ2 + β2 ) ( C1 + √ m3β1 ) ) . ( 172 ) Now we aim to bound the term EWρ , V ρ [ 1√m2 a TDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) +V ρ ) ϕ ( 0 ) ( xi ) . First assume that we are in the event Ec defined in Lemma 33 , i.e . we have ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 . Conditioned on such W ρ , we now work with the randomness of the initialization and V ρ . Note that the random matrix V ( 0 ) + V ρ jointly over the randomness of V ρ and the initialization is also Gaussian , and its variance is κ22 ≤ κ22 + β22 m2 ≤ 2κ22 , ( 173 ) where the inequality follows from the fact that κ2 ≥ 1√m2 and β2 ≤ 1 . Therefore , applying Lemma 22 for the random matrix V ( 0 ) in the Lemma as V ( 0 ) +V ρ here , the bound does not change up to constants because of the inequality ( 173 ) . Hence , with high probability , lets say with prob . 1− δ1 this time over both the randomness of initialization and V ρ : L = ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ≤ ℜ6 ( 174 ) This means that with probability at least 1− √ δ1 over the random initialization , then we have ( 174 ) with prob . at least 1− √ δ1 over the randomenss of V ρ . We name the latter high probability statement as ( ⋆ ) . Moreover , note that by Lemma 32 and assuming m3 log ( m2 ) < m2 , we have the following almost surely bound ( also note that V ρj ϕ ( 0 ) ( xi ) is Gaussian with std β2√m2 ∥ϕ ( 0 ) ( xi ) ∥ ) : EV ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ( 175 ) = EV ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ( 176 ) ≤ 1√ m2 m2∑ j=1 |V ( 0 ) j ϕ ( 0 ) ( xi ) |+ EV ρ 1√ m2 m2∑ j=1 |V ρj ϕ ( 0 ) ( xi ) | ( 177 ) ≲ ∥ϕ ( 0 ) ( xi ) ∥ sup ∥x′∥=1 1√ m2 m2∑ j=1 |V ( 0 ) j x ′|+ 1√ m2 m2∑ j=1 β2√ m2 ∥ϕ ( 0 ) ( xi ) ∥ ( 178 ) ≲ ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) . ( 179 ) Furthermore , note because each variable |V ρj ϕ ( 0 ) ( xi ) | is β2√ m2 ∥ϕ ( 0 ) ( xi ) ∥ subGaussian . Therefore , L is subGaussian with parameter ∥ϕ ( 0 ) ( xi ) ∥β2 with respect to the randomness of V ρ . Now the point is that the high probability argument in ( ⋆ ) is much stronger than what one can get from the subGaussian ineqaulity with parameter ∥ϕ ( 0 ) ( xi ) ∥β2 ( with the corresponding expectation term ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) ) . However , the disadvantage of ( ⋆ ) is that it only works for a fixed δ1 . In other words , at least it is not obvious from this argument that why for a fixed W ( 0 ) in a high probaiblity region of the random initialization , whether we can send δ1 to zero by growing the constant behind ℜ6 with logarithmic rate log ( 1/δ ) . This makes our job hard for bounding the expectation with respect to V ρ if we only wish to rely on ( ⋆ ) . Therefore , we combine it with the inequality that we get from the subGaussian parameter that we introudced above . More rigorously , we define the thresholding parameter ℧ : = ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) + ∥ϕ ( 0 ) ( xi ) ∥β2 log ( ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) /ℜ6 ) = Θ ( ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 log ( ∥ϕ ( 0 ) ( xi ) ∥ ( √ m2 + β2 ) /ℜ6 ) ) ) , for which we have E [ L| ℧ ≤ L ] P ( ℧ ≤ L ) ≲ ℜ6 . we divide the range of values for L into three parts : E [ L ] = E [ L| L ≤ ℜ ] P ( L ≤ ℜ6 ) + E [ L| ℜ6 ≤ L ≤ ℧ ] P ( ℜ6 ≤ L ≤ ℧ ) + E [ L| ℧ ≤ L ] P ( ℧ ≤ L ) ≤ E [ L| L ≤ ℜ6 ] + P ( ℜ6 ≤ L ≤ ℧ ) + ℜ6 ≲ ℜ6 + √ δ1℧ . Now by choosing δ1 ≲ 1/℧ , we conclude with high probability over initialization and conditioned on W ρ ’ s such that Ec happens we have EV ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ = E [ L ] ≲ ℜ6 . Finally , we integrate also with respect to W ρ . To control the random variable when E happens , we use the bound in ( 179 ) and the fact that E is a rare event due to Lemma 33 : EWρ , V ρ ∣∣∣ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) +V ρ ) ϕ ( 0 ) ( xi ) ∣∣∣ ≲ P ( E ) ∥ϕ ( 0 ) ( xi ) ∥ ( √m2+β2 ) +P ( Ec ) ℜ6 ≤ m1 exp { −C21/ ( 8m3β21 ) } √ m3κ1 ( √ m2 + β2 ) + ℜ6 : = ℜ′6 . Substituting this into ( 172 ) the proof is finally complete . Lemma 21 Third cross term : with high probability over initialization , we have EΣ ( EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) Σϕ∗ ( xi ) ] ) 2 ≲ ξ2 ( m1 exp { −C21/ ( 8m3β21 ) } ( κ22m2m3 + β21m3 ) + ℜ27C22 ) : = ℜ212 . Proof of Lemma 21 Note that the way we defined the matrix W ∗ and as a result ϕ∗ ( xi ) only depends on the randomness of W ( 0 ) , not on W ′ or the randomness of V ( 0 ) . Now using Equation ( 131 ) and Jensen inequality we can write ( for vector v , the notation v2⊙ is another vector with each entry as the second power of the corresponding entry in v ) : = EΣ ( EWρ , V ρ [ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) Σϕ∗ ( xi ) ] ) 2 ≤ EΣEWρ , V ρ ( 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) Σϕ∗ ( xi ) ) 2 = EWρ , V ρEΣ ( ⟨ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) , Σϕ∗ ( xi ) ⟩ ) 2 = EWρ , V ρ 〈 ( 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) ) 2⊙ , ϕ∗ ( xi ) 2⊙ 〉 ≤ EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) ∥∥∥2 2 ∥∥∥ϕ∗ ( xi ) ∥∥∥2 ∞ ≤ ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ + ( 1− η ) V ′ ) ∥∥∥2 2 ≤ 2ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ∥∥∥2 2 + 2ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( 1− η ) V ′ ∥∥∥2 2 ≤ ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ∥∥∥2 2 + ξ2 ( 1− η ) 2∥V ′∥2F ≤ ξ2EWρ , V ρ ∥∥∥ 1√ m2 aTDV ( 0 ) +V ρ+V ′ , xi ( V ( 0 ) + V ρ ) ∥∥∥2 2 + ξ2 ( 1− η ) 2C22 . Now under the event Ec defined in Lemma 33 we get that ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 , so we can bound the above as ≤ ξ2EV ρ sup ∥V ′∥≤C2 , V ′⊥ϕ ( 0 ) ( xi ) , ∥x′∥≤C1 1 m2 ∥∥∥∑ j aj1 { ( V ( 0 ) j + V ρ j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ( V ( 0 ) j + V ρ j ) ∥∥∥2 ( 180 ) + ξ2 ( 1− η ) 2C22 . ( 181 ) Now defining L2 : = 1 m2 ∥∥∥∑ j aj1 { ( V ( 0 ) j + V ρ j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ( V ( 0 ) j + V ρ j ) ∥∥∥2 , to bound the first term , we want to apply Lemma 23 using the same trick that we did in the proof of Lemma 20 . Note that L2 is the same term as Γ2x′ , V ′ in Lemma 23 except that it is defined with respect to V ( 0 ) + Vρ instead of V ( 0 ) . On the other hand , note that V ( 0 ) + V ρ has Gaussian entries with variance κ22+ β22 m2 and we know κ22 ≤ κ22+ β22 m2 ≤ 2κ22 , which means the argument of Lemma 23 holds true here up to constants : sup ∥V ′∥≤C2 , V ′⊥ϕ ( 0 ) ( xi ) , ∥x′∥≤C1 L2 ≲ ℜ27 . This holds with probability say 1 − δ2 over the randomness of both V ( 0 ) and V ρ . Therefore , with probability 1− √ δ2 over the initialization , then with probability at least 1− √ δ2 over the randomness of V ρ we have the above . Moreover , with a simple Cauchy-Swuartz we get the following almost surely bound : L2 ≲ ∥V ( 0 ) ∥2F + ∥V ρ∥2F . ( 182 ) Now the variable ∥V ρ∥2 is subexponential with parameter ( β41m23 , β21m3 ) . Furthermore , with high probability we have ∥V ( 0 ) ∥2F ≲ m2m3κ22 . Therefore , taking ℧2 : = Θ ( κ22m2m3 + β 2 1m3 log ( ( κ22m2m3 + β 2 1m3 ) /ℜ7 ) ) , then one can easily see by the subexponential tail : E [ L2| L2 ≥ ℧2 ] = Θ ( ℧2 ) , P ( L2 ≥ ℧2 ) ≤ ℜ27/℧2 . Hence , we can apply the same trick as Lemma 20 as E [ L2 ] = E [ L2| L2 ≤ ℜ27 ] P ( L2 ≤ ℜ27 ) + E [ L2| ℜ27 ≤ L2 ≤ ℧2 ] P ( ℜ27 ≤ L2 ≤ ℧2 ) + E [ L2| ℧2 ≤ L2 ] P ( ℧2 ≤ L2 ) ≲ E [ L2| L2 ≤ ℜ27 ] + P ( ℜ27 ≤ L2 ≤ ℧2 ) + ℜ27 ≲ ℜ27 + √ δ2℧2 . Now taking δ2 ≲ ℜ47/℧22 , we finally get that conditioned on W ρ ’ s where E happens , then EV ρL2 ≤ ℜ27 . On the other hand , to handle the case when E happens , we can use the bound in ( 182 ) as it does not depend on the occurrence of E as well : EWρ , V ρL2 ≤ P ( E ) EV ρ ( ∥V ( 0 ) ∥2 + ∥V ρ∥2 ) + P ( Ec ) ℜ27 ≲ m1 exp { −C21/ ( 8m3β21 ) } ( κ22m2m3 + β21m3 ) + ℜ27 . Plugging this back into ( 181 ) we finally get LHS ≲ ξ2 ( m1 exp { −C21/ ( 8m3β21 ) } ( κ22m2m3 + β21m3 ) + ℜ27 ) + ξ2C22 . A.15 BOUNDING THE WORST-CASE SENARIO Lemma 22 Suppose m3 ≥ log ( m2 ) and √ m3κ1 ≳ C1 . We define the sign matrices Dx ′ V ( 0 ) +V ′ , xi and Dx ′ V ( 0 ) , xi with respect to the multiplications ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + x ′ ) , and V ( 0 ) ( ϕ ( 0 ) ( xi ) + x ′ ) . Then , with high probability : sup ∥x′∥≲C1 , ∥V ′∥F≤C2 , V ′⊥ϕ ( 0 ) 1√ m2 aTDx ′ V ( 0 ) +V ′ , xi V ( 0 ) ϕ ( 0 ) ( xi ) ≲ ( ( C1C2 ) 4/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 + ( C1C2 ) 2/3m 2/3 3 ( κ1κ2 ) 1/3 √ log ( m2 ) m 1/3 2 ) ( 1 + log ( m2 ) C 1/3 1 ( κ2 √ m2 ) 2/3 C 2/3 2 ( κ1 √ m3 ) 1/3 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) + κ1κ2 √ m3 log ( m2 ) : = ℜ6 . Proof of Lemma 22 Consider a cover for the euclidean ball of radius C1 in Rm3 with precision ϵ , i.e . BC1 ( ϵ ) . So for every x′ ∈ Rm3 , there exists an x ∈ BC1 ( ϵ ) such that ∥x− x′∥ ≤ ϵ , and |BC1 ( ϵ ) | ≲ ( 1ϵ ) m3 . Now fix x′ and x . We have Γx′ , V ′ : = 1√ m2 aTDx ′ V ( 0 ) +V ′ , xi V ( 0 ) ϕ ( 0 ) ( xi ) = 1√ m2 m2∑ j=1 aj1 { ( V ( 0 ) j +V ′ j ) ( ϕ ( 0 ) ( xi ) +x ′ ) ≥ 0 } V ( 0 ) j ϕ ( 0 ) ( xi ) . Now by a union bound , because each variable V ( 0 ) j ϕ ( 0 ) ( xi ) is Gaussian with parameter κ2∥ϕ ( 0 ) ( xi ) ∥ and using Equation ( 116 ) , with high probability we have for every j ∈ [ m2 ] : V ( 0 ) j ϕ ( 0 ) ( xi ) ≲ κ2∥ϕ ( 0 ) ( xi ) ∥ √ log ( m2 ) ≲ κ1κ2 √ m3 log ( m2 ) . ( 183 ) Therefore , by Hoeffding over the randomness of the Bernoulli variables aj , for a fixed x′ with high probability : Γx′ : = 1√ m2 m2∑ j=1 aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ϕ ( 0 ) ( xi ) ≲ κ1κ2 √ m3 log ( m2 ) . On the other hand , We know that the VC-dimension of the class of binary functions with respect to halfspaces in Rm3 is m3 + 1 . Therefore , the set of different sign patterns in matrices Dx ′ V ( 0 ) , xi is bounded by mm3+12 , i.e . for D = { Dx ′ V ( 0 ) , xi | x′ ∈ Rm3 } , we have |D| ≲ mm3+12 . Therefore , by taking a union bound over all sign matrices in D , we get with high probability sup x′ Γx′ ≲ κ1κ2 √ m3 log ( m2 ) √ log ( m2m3+1 ) = κ1κ2m3 log ( m2 ) . ( 184 ) Now for a threshold r which satisfies r ≥ 2 √ m3κ2ϵ , ( 185 ) we define Jx , r = { j ∈ [ m2 ] | |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | ≤ r } . Now by Equation ( 116 ) and the assumption of the Lemma √ m3κ1 ≳ C1 , we have ∥ϕ ( 0 ) ( xi ) + x∥ ≤ ∥ϕ ( 0 ) ( xi ) ∥+ ∥x∥ ≲ √ m3κ1 + C1 . ( 186 ) ∥ϕ ( 0 ) ( xi ) + x∥ ≥ ∥ϕ ( 0 ) ( xi ) ∥ − ∥x∥ ≳ √ m3κ1 − C1 ≳ √ m3κ1 . ( 187 ) Hence , V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) is Gaussian with standard deviation at least Ω ( κ2 √ m3κ1 ) . Therefore , P ( |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | ≤ r ) ≲ r√ m3κ1κ2 . This implies E [ |Jx , r| ] ≲ r√ m3κ1κ2 m2 . ( 188 ) On the other hand , note that |Jx , r| is the sum ofm2 Bernoulli random variables , so it is subGaussian with parameter m2 . Therefore , with high probability |Jx , r| ≲ r√ m3κ1κ2 m2 + √ m2 . Now taking maximum over all x ∈ BC1 ( ϵ ) and exploiting the subGaussian tail of the random variables , we get with high probability max x∈BC1 ( ϵ ) |Jx , r| ≲ r√ m3κ1κ2 m2 + √ m2 log ( |BC1 ( ϵ ) | ) ≲ r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) . ( 189 ) Moreover , consider a threshold 1 < θ , such that e−θ 2/8 ≤ m2/m3 , and define the following set of indices J ( 2 ) x′ , θ : = { j ∈ [ m2 ] | |V ( 0 ) j x ′| ≥ θκ2C1 } . Then , using Lemma 29 and noting the fact that the standard deviation of Gaussians in V ( 0 ) is κ2 and that ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 , with high probability : sup x′ : ∥x′∥=1 |J ( 2 ) x′ , θ| ≤ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . ( 190 ) Now note that for each j ∈ [ m2 ] , ∥V ( 0 ) j ∥2 is subexponential with parameters ( m3κ42 , κ22 ) , which means that with high probability : max j ∥V ( 0 ) j ∥ 2 ≲ m3κ22 + √ m3κ 2 2 √ log ( m2 ) + κ 2 2 log ( m2 ) . But with condition m3 ≥ log ( m2 ) , we can further upper bound it as max j ∥V ( 0 ) j ∥ 2 ≲ m3κ22 . Now for fixed x , x′ , for j ∈ Jx , r we have |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≤ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) |+ |V ( 0 ) j ( x ′ − x ) | ≤ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) |+ ∥V ( 0 ) j ∥∥x ′ − x∥ ≲ r + √ m3κ2ϵ . On the other hand , for j /∈ J ( 2 ) x′ , θ : |V ( 0 ) j x ′| ≤ θκ2C1 . ( 191 ) Therefore , for j ∈ Jx , r − J ( 2 ) x′ , θ : |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) |+ |V ( 0 ) j x ′| ≲ r + √ m3κ2ϵ+ θκ2C1 . ( 192 ) In a similar fashion , if j /∈ Jx , r , then using assumption ( 185 ) : |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≥ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | − |V ( 0 ) j ( x− x ′ ) | ≳ r − √ m3κ2ϵ ≥ r/2 . ( 193 ) Hence , using the fact that ϕ ( 0 ) ( xi ) is orthogonal to V ′j : ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣ ≤ 1 { |V ′j ( ϕ ( 0 ) ( xi ) + x′ ) | ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { |V ′j x′| ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { ∥V ′j ∥∥x′∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { ∥V ′j ∥C1 ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ≤ 1 { ∥V ′j ∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) | 2C1 + r 4C1 } . ( 194 ) Now by triangle inequality and Equations ( 194 ) , ( 192 ) , ( 191 ) and the fact that ∥V ′∥F ≤ C2 , we can write : |Γx′ − Γx′ , V ′ | ≤ 1√ m2 ∑ j∈Jx , r−J ( 2 ) x′ , θ ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣|V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣|V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j∈J ( 2 ) x′ , θ ∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣|V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| max j∈Jx , r−J ( 2 ) x′ , θ |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) 1 { ∥V ′j ∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) | 2C1 + r 4C1 } |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≤ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| max j∈Jx , r−J ( 2 ) x′ , θ |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) 1 { ∥V ′j ∥ ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) | 2C1 + r 4C1 } ( |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) |+ θκ2C1 ) + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≲ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| max j∈Jx , r−J ( 2 ) x′ , θ |V ( 0 ) j ϕ ( 0 ) ( xi ) | + 1√ m2 ∑ j /∈ ( Jx , r∪J ( 2 ) x′ , θ ) 1 { ∥V ′j ∥ ≳ r C1 } ( C1∥V ′j ∥+ θκ2C1 ) + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≲ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| ( r + √ m3κ2ϵ+ θκ2C1 ) + C1√ m2 √ # ( j : ∥V ′j ∥ ≳ r C1 ) ∥V ′∥2F + 1√ m2 # ( j : ∥V ′j ∥ ≳ r C1 ) θκ2C1 + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | ≲ 1√ m2 |Jx , r − J ( 2 ) x′ , θ| ( r + √ m3κ2ϵ+ θκ2C1 ) + C21C 2 2√ m2r + C31C 2 2√ m2r 2 θκ2 + 1√ m2 |J ( 2 ) x′ , θ|maxj∈m2 |V ( 0 ) j ϕ ( 0 ) ( xi ) | . Now using Equations ( 208 ) , ( 190 ) , ( 183 ) , and ( 185 ) , and the bound on |J ( 2 ) x′ , θ| from Lemma 29 , we write ≲ 1√ m2 |Jx , r| ( r + θκ2C1 ) + 1√ m2 |J ( 2 ) x′ , θ|κ1κ2 √ m3 log ( m2 ) + C21C 2 2√ m2r + C31C 2 2√ m2r 2 θκ2 ≤ 1√ m2 ( r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) ) ( r + θκ2C1 ) + 1√ m2 ( m3 ( log ( m3 ) + log ( log ( m2 ) ) ) ) κ1κ2 √ m3 log ( m2 ) + C21C 2 2 r √ m2 + C31C 2 2√ m2r 2 θκ2 . ≤ 1√ m2 ( r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) ) ( r + θκ2C1 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) + C21C 2 2 r √ m2 + C31C 2 2√ m2r 2 θκ2 . ( 195 ) Now setting r∗ : = ( C1C2 ) 2/3m 1/6 3 ( κ1κ2 ) 1/3 m 1/3 2 . By this choice , from ( 195 ) we obtain |Γx′−Γx′ , V ′ | ≤ ( ( C1C2 ) 4/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 + ( C1C2 ) 2/3m 2/3 3 ( κ1κ2 ) 1/3 √ log ( 1/ϵ ) m 1/3 2 ) ( 1+θ C 1/3 1 ( κ2 √ m2 ) 2/3 C 2/3 2 ( κ1 √ m3 ) 1/3 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) . Now we set θ∗ : = 3 log ( m2 ) , which also satisfies the condition of Lemma 29 and combining with Equation ( 184 ) , we get that with high probability |Γx′ , V ′ | ≤ |Γx′ , V ′ − Γx′ |+ |Γx′ | ≲ ( ( C1C2 ) 4/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 + ( C1C2 ) 2/3m 2/3 3 ( κ1κ2 ) 1/3 √ log ( 1/ϵ ) m 1/3 2 ) ( 1 + log ( m2 ) C 1/3 1 ( κ2 √ m2 ) 2/3 C 2/3 2 ( κ1 √ m3 ) 1/3 ) + m 3/2 3 κ1κ2√ m2 √ log ( m2 ) ( log ( m3 ) + log ( log ( m2 ) ) ) + κ1κ2 √ m3 log ( m2 ) , where ∥x′∥ ≲ C2 and ∥V ′∥F ≤ C2 , ∀j : V ′jϕ ( 0 ) ( xi ) = 0 . We also need to satisfy condition ( 185 ) , which regarding this choice for θ = θ∗ becomes r∗ = ( C1C2 ) 2/3m 1/6 3 ( κ1κ2 ) 1/3 m 1/3 2 ≥ 2 √ m3κ2ϵ , ( 196 ) for which it suffices to set ϵ∗ : = ( C1C2 ) 2/3 κ 1/3 1 2 ( m2m3 ) 1/3κ 2/3 2 , ( 197 ) Substituting this choice of ϵ above and picking the overparameterization large enough to dominate the magnitude of C1 , C2 so that log ( 1/ϵ∗ ) ≲ log ( m2 ) , the proof is complete . Lemma 23 Under the following condition ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ≥ log−7/6 ( m2 ) ( C1C2 ) 1/3 , with high probability we have sup ∥x′∥≲C1 , ∥V ′∥F≤C2 , V ′⊥ϕ ( 0 ) 1√ m2 ∥ ∑ j aj1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ ≲ √ m3κ2 log ( m2 ) + ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ( C1C2 ) 2/3 log1/6 ( m2 ) : = ℜ7 . Proof of Lemma 23 Similar to Lemma 22 , define the helper functions Γx′ and Γx′ , V ′ as Γx′ , V ′ = 1√ m2 ∥ ∑ j aj1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ , ( 198 ) Γx′ = 1√ m2 ∥ ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ . ( 199 ) First we bound supx′ Γx′ . To this end , note that because V ( 0 ) j ∈ Rm3 and the VC-dimension of half-planes is m3 + 1 , then by Sauer ’ s Lemma , the set D = { Dx ′ V ( 0 ) , xi | x′ ∈ Rm3 , ∥x′∥ ≲ C1 } of all sign pattern matrices has cardinality at most |D| ≤ mm3+12 . Now note that with high probability , the entries of the matrix V ( 0 ) are all less than O ( κ2 √ log ( m2m3 ) ) . On the other hand , for each fixed sign pattern Dx ′ V ( 0 ) , xi , we have for the sum with respect to this sign pattern : ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ( 200 ) is ( m3κ42 log 2 ( m2m3 ) , κ 2 2 log ( m2m3 ) ) sub-exponential with respect to the randomness of a , because each entry of the vector 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j is ( κ2 √ log ( m2m3 ) ) subGaussian . Therefore , with high probability we have ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ( 201 ) ≤ Ea [ ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ] + deviation ( 202 ) ≲ m3κ22 log ( m2m3 ) + √ m3κ 2 2 log ( m2m3 ) + κ 2 2 log ( m2m3 ) . ( 203 ) Similarly , if we take a union bound over all sign matrices in D and using the fact that m2 > m3 : sup x′ Γ2x′ = sup x′∈Rm3 ∥ 1√ m2 ∑ j aj1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } V ( 0 ) j ∥ 2 ( 204 ) ≲ m3κ22 log ( m2m3 ) + √ m3κ 2 2 log ( m2m3 ) √ log ( mm3+12 ) + κ 2 2 log ( m2m3 ) log ( m m3+1 2 ) ( 205 ) ≲ m3κ22 log2 ( m2 ) , ( 206 ) which implies sup x′ Γx′ ≲ √ m3κ2 log ( m2 ) . ( 207 ) Moreover , defining Jv , x similar to Lemma 22 and using the similar approach we get with high probability max x∈BC1 ( ϵ ) |Jx , r| ≲ r√ m3κ1κ2 m2 + √ m2 log ( |BC1 ( ϵ ) | ) ≲ r√ m3κ1κ2 m2 + √ m2m3 log ( 1/ϵ ) . ( 208 ) Now for simplifying the analysis , we assume that for indices j ∈ Jx , r we can change the sign pattern with no cost on V ′ , i.e . we can pick any subset of them . Therefore , we first compute a high probability upper bound on the following quantity : 1√ m2 sup S⊂Jx , r , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ . ( 209 ) If we form the matrix V ( 0 ) ( Jx , r ) be the matrix which only keeps the rows with indices in Jx , r , then the above quantity can be computed as 1√ m2 sup S⊂Jx , r , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ = 1√ m2 sup v∈ { 0,1 , −1 } |Jx , r| ∥vTV ( 0 ) ( Jx , r ) ∥ ( 210 ) ≤ 1√ m2 λmax ( V ( 0 ) ( Jx , r ) ) sup ∥v∥ ≤ 1√ m2 λmax ( V ( 0 ) ( Jx , r ) ) |Jx , r| , ( 211 ) where λmax is the maximum singular value of the matrix . Now by random matrix theory , we know for a fixed x and arbitrary t ≥ 0 , the following argument holds : P ( λmax ( V ( 0 ) ( Jx , r ) ) /κ2 ≳ √ m3 + √ |Jx , r|+ t ) ≤ 2e−ct 2 . ( 212 ) Therefore , as |D| ≤ mm3+12 , we get with high probability max x∈BC1 ( ϵ ) λmax ( V ( 0 ) ( Jx , r ) ) ≲ κ2 ( √ m3 + √ |Jx , r|+ √ log ( mm3+12 ) ) ( 213 ) ≤ κ2 √ log ( m2 ) m3 + κ2 √ |Jx , r| . ( 214 ) Therefore , with high probability sup x∈BC1 ( ϵ ) 1√ m2 sup S⊂Jx , r ∥ ∑ j∈S V ( 0 ) j ∥ ≤ κ2√ m2 ( √ log ( m2 ) m3|Jx , r|+ |Jx , r| ) . ( 215 ) On the other hand , as in Equation ( 193 ) in the proof of Lemma 22 , for j /∈ Jx , r we have : |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≥ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ) | − |V ( 0 ) j ( x− x ′ ) | ≳ r − √ m3κ2ϵ . ( 216 ) Picking ϵ∗ : = r 2 √ m3κ2 , we get for j /∈ Jx , r |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | ≳ r. Now similar to the derivation in ( 194 ) we have∣∣∣1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ∣∣∣ ( 217 ) ≤ 1 { ∥V ′j ∥C1 ≳ |V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) | } ( 218 ) ≤ 1 { ∥V ′j ∥ ≳ r C1 } . ( 219 ) Hence , because ∥V ′∥F ≤ C2 , the number of indices for which 1 { ( V ( 0 ) j + V ′j ) ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } ̸= 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x′ ) ≥ 0 } is at most l = ( C1C2 ) 2 r2 . Therefore , we bound the following quantity to use in the analysis : sup S⊂ [ m2 ] & |S|≤l , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ . ( 220 ) But if we define for m2 < j ≤ 2m2 , V ( 0 ) j = −V ( 0 ) j−m2 , then sup S⊂ [ m2 ] & |S|≤l , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ ≤ sup S⊂ [ 2m2 ] & |S|≤l ∥ ∑ j∈S V ( 0 ) j ∥ . Now note that each entry of ∑ j∈S V ( 0 ) j is √ lκ2 subGaussian . Hence , the quantity ∥ ∑ j∈S V ( 0 ) j ∥2 is ( m3l2κ42 , lκ 2 2 ) subexponential . Therefore , we have with high probability sup |S|≤l ∥ ∑ j∈S V ( 0 ) j ∥ 2 ≲ E∥ ∑ j∈S V ( 0 ) j ∥ 2 + √ m3lκ 2 2 √ log ( 2m2 l ) + lκ22 log ( 2m2 l ) ≲ m3lκ22 + √ m3lκ 2 2 √ log ( 2m2 l ) + lκ22 log ( 2m2 l ) ≲ m3lκ22 + √ m3l 3/2κ22 √ log ( m2 ) + l 2κ22 log ( m2 ) . Hence sup |S|≤l ∥ ∑ j∈S V ( 0 ) j ∥ ≲ √ m3 √ lκ2 + lκ2 √ log ( m2 ) . Now using Equation , we can write |Γx′ − Γx′ , V ′ | ≤ 1√ m2 ∥ ∑ j∈Jx , r ( 1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ) V ( 0 ) j ∥ + 1√ m2 ∥ ∑ j /∈Jx , r ( 1 { ( V ( 0 ) j + V ′ j ) ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } − 1 { V ( 0 ) j ( ϕ ( 0 ) ( xi ) + x ′ ) ≥ 0 } ) V ( 0 ) j ∥ ≤ 1√ m2 sup S⊂Jx , r , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ + 1√ m2 sup S⊂ [ m2 ] & |S|≤ ( C1C2r ) 2 , ±signs ∥ ∑ j∈S ±V ( 0 ) j ∥ ≤ κ2√ m2 ( √ log ( m2 ) m3|Jx , r|+ |Jx , r| ) + 1√ m2 ( √ m3 √ lκ2 + lκ2 √ log ( m2 ) ) ≤ κ2√ m2 ( rm2√ m3κ1κ2 + √ m2m3 log ( √ m3κ2 r ) ) 1/2 ( √ log ( m2 ) m3 + √ |Jx , r| ) + 1√ m2 ( √ m3 ( C1C2/r ) κ2 + ( C1C2/r ) 2κ2 √ log ( m2 ) ) ≲ κ2√ m2 ( rm2√ m3κ1κ2 + √ m2m3 log ( √ m3κ2 r ) ) + 1√ m2 ( √ m3 ( C1C2/r ) κ2 + ( C1C2/r ) 2κ2 √ log ( m2 ) ) . Combining this with ( 207 ) : |Γx′ , V ′ | ≲ √ m3κ2 log ( m2 ) ( 221 ) + r √ m2√ m3κ1 + κ2 √ m3 log ( √ m3κ2 r ) + 1√ m2 ( √ m3 ( C1C2/r ) κ2 + ( C1C2/r ) 2κ2 √ log ( m2 ) ) . ( 222 ) Now setting r∗ : = m 1/6 3 ( κ1κ2 ) 1/3 m 1/3 2 ( C1C2 ) 2/3 log1/6 ( m2 ) , we get LHS ≲ √ m3κ2 log ( m2 ) + ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ( C1C2 ) 2/3 log1/6 ( m2 ) ( 223 ) + κ2m 1/2 3 log 1/2 ( m2 ) + m 1/3 3 κ 2/3 2 ( C1C2 ) 1/3 m 1/6 2 κ 1/3 1 log 1/6 ( m2 ) ( 224 ) ≲ √ m3κ2 log ( m2 ) + ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ( C1C2 ) 2/3 log1/6 ( m2 ) ( 225 ) + m 1/2 3 κ2 ( C1C2 ) 1/3 ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 1/3 log1/6 ( m2 ) . ( 226 ) Now under the condition ( √ m2κ2 ) 1/3 ( √ m3κ1 ) 2/3 ≥ log−7/6 ( m2 ) ( C1C2 ) 1/3 , The final term is dominated by the first term , which finally completes the proof . A.16 CONVERGENCE The goal of this section is to prove Theorem 7 . Theorem 7 Letting ℵ = 4B2 , by Corollary 8.1 , we have LΠ ( 0 ) ≤ ℵ . We define the domain Dl : = { ∥w′∥ ≤ C1 : = ℵ+4lψ1 , ∥v ′∥ ≤ C2 : = ℵ+4lψ2 } . For a large enough constant l = O ( 1 ) and function LΠ ( w : = ( w′ , v′ ) ) : RN → R. Moreover , suppose LΠ is ρ1 lipschitz , ρ2 gradient lipschitz , and ρ3 hessian lipschitz in the domain Dl , in the sense that their first , second , and third directional derivatives in an arbitrary unit direction is bounded by the corresponding parameters . Suppose we have access to the gradient of LΠ at each point in Dl plus a zero mean noise vector £ such that σ21I ≤ E££T ≤ σ22I and ∥£∥ ≤ Q almost surely . Also , suppose for a threshold ℵℓ ≤ ℵ , if LΠ ( w ) ≥ ℵℓ for any w ∈ Dl , then we either have ( 1 ) ∥∇LΠ ( w ) ∥ ≥ ν 16 √ C21 + C 2 2 , ( 227 ) ( 2 ) λmin ( ∇2LΠ ( w ) ) ≤ −γ . ( 228 ) Then starting from w0 = 0 , with probability at least 0.999 after at most poly ( ρ1 , ρ2 , ρ3 , Q , ℵ , C1 , C2 , 1/γ , log ( σ1/σ2 ) ) number of iterations , we reach a point wt such that LΠ ( wt ) ≤ ℵℓ . Proof Our proof here is a refined version of that in Ge et al . ( 2015a ) . As we mentioned in section 5 , the key fact that we are using in the other parts of our proof is a uniform upper bound ∥w′∥ ≤ C1 , ∥v′∥ ≤ C2 which is unjustified by only naively using Ge et al . ( 2015a ) . Here , first we restate a refined version of Lemmas 14 and 16 in Ge et al . ( 2015a ) in Lemmas 24 and 26 respectively , and then use them to also bound the upward deviations of LΠ . Moreover , to avoid writing repeated proofs and overwhelm the reader , we mostly treat the arguments in Lemma 16 of Ge et al . ( 2015a ) as blackbox and use them for our purpose here . A point to mention before we start , unlike Lemma 14 of Ge et al . ( 2015a ) where the dependency on other parameters than the step size η is more explicit , Lemma 16 hides the dependencies on all the other parameters ( which is polynomial ) . Here , we follow the same style . We refer to the trajectory of the steps of algorithm by ( wt ) t≥0 . In Lemmas of this section , To avoid introducing new notation and complicating things , we refer to the current point of the algorithm by w0 , while for the next point of the algorithm we use w1 ( in Lemma 24 ) , and wT ( in Lemma 26 ) respectively . Also , similar to Ge et al . ( 2015a ) , Õ and Ω̃ below means we are looking at the dependency on η. Lemma 24 SupposeLΠ ( w0 ) ≤ ℵ+2l , and consider a parameter χ > 1 which can be set arbitrarily . For every pointw0 such thatLΠ ( w0 ) ≤ ℵ+2 , ∥∇LΠ ( w0 ) ∥ ≥ 2 √ η ( Q2 + σ22N ) ρ2ρ 2 1 ( 2χ+ 1 2 ) , then for w1 : = w0 − η ( ∇LΠ ( w0 ) +£ ) and random variable R1 ( depending on w0 ) defined as ELΠ ( w1 ) − LΠ ( w0 ) = −η2R21 , ( 229 ) we have R1 = Ω̃ ( 1 ) a.s. , and almost surely : ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣ ≤ ηR1/√ρ2χ . ( the expectation is over the randomness of £ ) . Proof This lemma is a tuned version of Lemma 14 in Ge et al . ( 2015a ) . First , note that the condition LΠ ( w0 ) ≤ ℵ + 2l assures the smoothness coefficients ρ1 , ρ2 and ρ3 for LΠ by Corollary ? ? . We follow similar to Ge et al . ( 2015a ) ( picking η < 1/ ( 2/ρ2 ) ) : ELΠ ( w1 ) − LΠ ( w0 ) ≤ − η 2 ∥∇LΠ ( w0 ) ∥2 + η2σ22ρ2N 2 ≤ −η 4 ∥∇LΠ ( w0 ) ∥2 − η2 ( σ22N +Q2 ) ρ2ρ21 ( 2χ+ 1 2 ) + η2σ22ρ2N 2 ≤ −η 4 ∥∇LΠ ( w0 ) ∥2 − 2η2Q2ρ2ρ21χ . ( 230 ) On ther other hand , LΠ is ρ1 Lipcshitz , so we have almost surely |LΠ ( w1 ) − LΠ ( w0 ) | ≤ ρ1η ( ∥∇LΠ ( w0 ) +£∥ ) ≤ ρ1η ( ∥∇LΠ ( w0 ) ∥+ ∥£∥ ) ≤ ρ1η ( ∥∇LΠ ( w0 ) ∥+Q ) . ( 231 ) ( To be completely precise , we should justify that we can write the Lipschitz inequality at point w0 , we also need to make sure that w1 remains in the domain that we have the Lipschitz parameter in , i.e . Dl . To see why this is true , see the next Corollary ) . Therefore ( ρ2χ ) ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣2 ≤ 2η2ρ21ρ2χ∥∇LΠ ( w0 ) ∥2 + 2ρ2χη2ρ21Q2 . ( 232 ) Taking η ≤ ( 8ρ21ρ2χ ) −1 , ( 233 ) we get from Equation ( 230 ) : ELΠ ( w1 ) − LΠ ( w0 ) ≤ −2η2ρ21ρ2χ∥∇LΠ ( w0 ) ∥2 − 2ρ2χη2.ρ21Q2 . ( 234 ) Combining Equations ( 232 ) and ( 234 ) , we see that ( ρ2χ ) ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣2 ≤ − ( ELΠ ( w1 ) − LΠ ( w0 ) ) . Hence , if we define ELΠ ( w1 ) − LΠ ( w0 ) : = −η2R21 , we get ∣∣∣LΠ ( w1 ) − LΠ ( w0 ) ∣∣∣ ≤ ηR1/√ρ2χ , ( 235 ) and from Equation ( 234 ) , that R21 ≥ 2ρ21ρ2χ∥∇LΠ ( w0 ) ∥2 + 2ρ2χρ21Q2 ≥ 2ρ2χρ21Q2 = Ω̃ ( 1 ) . Moreover , because the function is ρ1-Lipshitz at the domain point w0 , we get from Equation ( 231 ) : −η2RRR21 = ELΠ ( w1 ) − LΠ ( w0 ) ≥ −ηρ1 ( ∥∇LΠ ( w0 ) ∥+Q ) ≥ −ηρ1 ( ρ1 +Q ) ≥ − 1 ρ2χ , by taking η ≤ ( ρ1 ( ρ1 +Q ) ρ2χ ) −1 , which implies ηR1 ≤ 1 √ ρ2χ . This combined with Equation ( 235 ) and a triangle inequality implies : ∣∣∣LΠ ( w1 ) − ELΠ ( w1 ) ∣∣∣ ≤ ηR1/√ρ2χ+ η2R21 ≤ 2ηR1/√ρ2χ . ( 236 ) Lemma 25 As long as the value of the function at some w is bounded by ℵ+ 2l ( LΠ ( w ) ≤ ℵ+ 2l ) , then η can be picked small enough ( polynomially in other parameters ) so that the change of the function by a step is a.s. bounded by l. Proof Let ψ = min { ψ1 , ψ2 } . First , note that as the function is bounded by ℵ + 2l , we have the lipschitz parameter ρ1 , hence ∥∇LΠ ( w ) ∥ ≤ ρ1 . Therefore , the change in w in a step is bounded as ∥∇LΠ ( w ) +£∥ ≤ Q+ ρ1 . So by picking η ≤ ( √ℵ+ 3l ψ − √ ℵ+ 2l ψ ) / ( Q+ ρ1 ) , we guarantee that the value of w after a step remains in the ball of radius √ ℵ+3l ψ , hence we still have the smoothing parameters even after one step . Therefore , now we can use the Lipcshitz parameter ρ1 to bound the value of the function after one step as it is written in Equation Equation ( 231 ) . Using this Equation , it is enough to pick η as small as : η ≤ l/ ( ∥∇LΠ ( w0 ) ∥+Q ) , ( 237 ) so that the change in the function would be most l as desired . Lemma 26 For a fixed point w0 s.t . LΠ ( w0 ) ≤ ℵ+ 2l , suppose we pick η small enough such that § ( η ) : = 2 √ η ( Q2 + σ22N ) ρ2ρ 2 1 ( 2χ+ 1 2 ) < ν 16 √ C21 + C 2 2 . Then , note that for ∥∇LΠ ( w0 ) ∥ ≤ § ( η ) , condition 228 implies : λmin ( ∇2LΠ ( w0 ) ) ≤ γ . Then , using the notation ET for the high probability event corresponding to Equations ( 36 ) and ( 44 ) in Ge et al . ( 2015a ) , for small enough η ( polynomially small w.r.t other parameters ) , for R2 defined as E [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } = −R22η2 , ( 238 ) we have almost surely ∣∣∣ [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } ∣∣∣ ≤ R2η/√ρ2χ . ( 239 ) Note that in the expectations above w0 is assumed fixed . Furthermore , we can assume P ( ET ) ≥ 1− Õ ( η5 ) . Proof This Lemma is a tuned version of Lemma 16 in Ge et al . ( 2015a ) . We change a couple of things here . First , we consider an implicit coupling that if wt exits Dl we do not move it anymore , i.e . wt′ = wt , ∀t′ ≥ t , which means the noise vectors also becomes zero , i.e . £t′ = 0 , ∀t′ ≥ t. This way , the sequence of noise vectors remain bounded by Q , because if wt is inside Dl , then by assumption ∥£t∥ ≤ Q , while otherwise £t = 0 . We call this event ET for T defined in Lemma 16 of Ge et al . ( 2015a ) . Note that we also have the smoothing parameters ρ1 , ρ2 , ρ3 for all ( wt ) because of this coupling . In fact , we will use a more strict coupling ; we consider the event ET to be the high probability event corresponding to the bounds in Equations ( 44 ) and ( 36 ) of Ge et al . ( 2015a ) holding for all t ≤ T ; We will see that ET ⊆ ET at the end of this proof , but for now we assume it is true . An important point to note here is that in Ge et al . ( 2015a ) , P ( ET ) is bounded by O ( η2 ) . However , the exponent dependency of η in this bound comes from Azuma-Hoeffding type inequalities , particularly used in Equations ( 60 ) and ( 42 ) in Ge et al . ( 2015a ) , in which by considering larger constants one can easily get higher exponents . Unlike their analysis though , a bit stronger dependence of η5 is sufficient for our later use . Also , because the distribution of our noise depends on the point w , our sequence of noise vectors ( £t ) is a martingale instead of being i.i.d , so we apply Azuma-Hoeffding inequality instead of the simple Hoeffdings in Lemma 16 of Ge et al . ( 2015a ) . ( because we are also sampling also a random ( xi , yi ) to compute the estimate of gradient , this could be simplified to the case where we compute the actual gradient and the injecting an i.i.d noise vector in each step , but it is an overhead to compute the actual gradient , so here we choose to analyze the more complicated case . ) Next , notice the definition of Λ and Λ̃ right after Equation ( 66 ) in Ge et al . ( 2015a ) , which in our notation translates to Λ̃ : = ∇LΠ ( w0 ) T δ̃ + 1 2 δ̃THδ̃ , Λ = ∇LΠ ( w0 ) T δ + 1 2 δTHδ . ( 240 ) where δ̃ = w̃T − w0 , δ = wT − w̃T , for ( w̃t ) which is a coupled sequence with ( wt ) as defined in Ge et al . ( 2015a ) . Note that we apply the coupling for the sequence w̃t as well , i.e . if wt+1 = wt , we also set w̃t+1 = w̃t . To show Equation ( 238 ) , we want to use Equation ( 67 ) in Ge et al . ( 2015a ) , though we only use the expansion for the first term which is under 1 { ET } , i.e . E [ LΠ ( wT ) − LΠ ( w0 ) ] 1ET = EΛ̃1ET + EΛ1ET . ( 241 ) First of all , as it is mentioned in Lemma 16 of ( Ge et al. , 2015a ) , in the case where the noise vector σ1 2I ≤ E££T ≤ σ22I instead of having E££T = σ2I for a fixed σ , in order to still get a negative term of order η in Equation ( 68 ) of ( Ge et al. , 2015a ) , we just need the size of Tmax to be as large as O ( 1γη ( log d + log σ2 σ1 ) ) , and it does not change the order of η in any other part of Lemma 16 . Now similar to Equation ( 68 ) of ( Ge et al. , 2015a ) , if w.l.o.g we assume the smallest eigenvalue γ0 corresponds to i = 1 : EΛ̃1ET ≤ 1 2 N∑ i=1 λi T−1∑ τ=0 1 { λi < 0 } ( 1− ηλi ) 2τη2σ21P ( ET ) + ( 242 ) 1 2 N∑ i=1 λi T−1∑ τ=0 1 { λi≥0 } ( 1− ηλi ) 2τη2σ22 ( 243 ) ≤ η 2 2 [ σ22 N − 1 η − γ0σ21P ( ET ) T−1∑ τ=0 ( 1 + ηγ0 ) 2τ ] ≤ −ησ 2 1 2 . ( 244 ) where in the last line we use the fact that P ( ET ) ≤ 1/2 plus the additional log ( σ2/σ1 ) factor . Second , note that our threshold § ( η ) for the size of gradient in Lemmas 24 and 26 has the same order of η compared to that of Lemmas 14 and 16 in Ge et al . ( 2015a ) . Therefore , the arguments in Lemma 16 that considers the order of η and treat the other parameters as constants is true here as well . Hence , we still have Equation ( 69 ) of Ge et al . ( 2015a ) which is under the event ET . Applying it to Equation ( 241 ) , Hence , finally by a similar derivation of Equation ( 67 ) in Ge et al . ( 2015a ) : E [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } ≤ −Ω̃ ( η ) . ( 245 ) Furthermore , combining Equations ( 36 ) and ( 44 ) we get with high probability ( we use the final high probability parameter of Lemma 16 which is the result of a union bound over all the high probability arguments which is equivalent to the occurrence of ET ) , i.e . when ET happens , ∥wT − w0∥ ≤ Õ ( η 1 2 log 1 η ) . ( 246 ) Picking η small enough such that for the bound above we have O ( η 1 2 log 1 η ) ≤ √ ℵ+ 3l ψ − √ ℵ+ 2l ψ , we get for every w̄ in the line connecting w0 to wT : ∥w̄∥ ≤ √ ℵ+ 3l ψ , which implies that LΠ has the smoothing parameters ρ1 , ρ2 , ρ3 along w0 to wT . Therefore , by the ρ2-gradient smoothness property of LΠ : ∥∇LΠ ( w̄ ) −∇LΠ ( w0 ) ∥ ≤ ρ2∥w0 − w̄∥ ≤ O ( ρ2η 1 2 log 1 η ) . Combining the assumption of the Lemma ∥∇LΠ ( w0 ) ∥ ≤ Õ ( η 1 2 ) , we get ∥∇LΠ ( w̄ ) ∥ = Õ ( η1/2 log ( 1/η ) ) . ( The last Õ also hides the dependency on ρ2 ) . Now integrating over the derivative along the direction from w0 to wT : LΠ ( wT ) = L Π ( w0 ) + ∫ 1 t=0 ∇LΠ ( tw0 + ( 1− t ) wT ) T ( wT − w0 ) dt . Therefore , using ( 246 ) one more time , under the event ET : |LΠ ( wT ) − LΠ ( w0 ) | ≤ ∫ ∣∣∣∇LΠ ( tw0 + ( 1− t ) wT ) T ( wT − w0 ) ∣∣∣dt ≤ ∫ 1 0 ∥∇LΠ ( tw0 + ( 1− t ) ∥∥wT − w0∥dt ≤ Õ ( η1/2 log 1/η ) ∥wT − w0∥ = Õ ( η log2 1/η ) . Hence ∣∣∣ [ LΠ ( wT ) − LΠ ( w0 ) ] 1 { ET } ∣∣∣ ≤ Õ ( η log2 1/η ) , ( 247 ) which Now comparing Equations ( 245 ) and ( 247 ) , it is obvious that one can pick η small enough ( again polynomially small in the other parameters ) such that for some random variable R2 , which also depends on η Equations ( 238 ) and ( 239 ) hold . Furthermore , note that the bound in ( 247 ) is an a.s. upper bound on the change of the function value under the event ET for every 1 ≤ t ≤ T . Therefore , by picking η small enough ( polynomially ) s.t . the quantity O ( η log ( 1/η ) 2 ) in Equation ( 247 ) is bounded by l , we again make sure that the value of function during these steps changes by at most compared to w0 , i.e . for every 1 ≤ t ≤ T : ∣∣∣ [ LΠ ( wt ) − LΠ ( w0 ) ] 1 { Et } ∣∣∣ ≤ l ( 248 ) , therefore remains bounded by ℵ+ 3l . This also implies that ET ⊆ ET as promised . A.17 PROCESS FROM A HIGHER VIEW : DEFINITION OF THE ( X ) SEQUENCE The goal is to find a w∗ with LΠ ( w∗ ) ≤ ℵℓ using Lemmas 24 and 26 . For this purpose we define a coupled sequence this way : First , as done in Ge et al . ( 2015a ) , define a sequence of times τi inductively in the following way : To define τi+1 based on τi , if the condition ℵℓ ≤ LΠ ( wτi+1 ) ≤ ℵ+ 2l ( 249 ) does not hold , then just set τi+1 = τi ⋆ ( 1 ) . Otherwise , using the conditions ( 228 ) , we are either in the situation of Lemma 24 or Lemma 26 , setting w0 = wτi . In the first case , define τi+1 = τi + 1 ⋆ ( 2 ) . In the latter case , Let ET be the same high probability event that we consider in Lemma ( 26 ) , which happens when the aggregate behavior of the noise vectors does not behave oddly , so that we remain close to the starting point w0 . Note that from Lemma 26 , we know P ( ET ) ≥ 1 − O ( η2 ) . Now if the event ET happens , define τi+1 : = τi + T ⋆ ( 3 ) , for T also from Lemma 26 and defined originally in Lemma 16 of Ge et al . ( 2015a ) , while otherwise define τi+1 = τi ⋆ ( 4 ) and , moreover , define all the rest of τi′ ’ s equal to τi : τi′ = τi for every i′ ≥ i . At the same time , we define the event Gi , where Gi happens in the case ⋆ ( 4 ) , and Gi+1 happens in case ⋆ ( 4 ) . Also , Gi happens if any of the previous Gi′ ’ s happen for i′ < i ; in other words , Gi is included in Gi+1 . We use these events to bound the probability that the process remains above ℵℓ . Moreover , define the sequence of random variables ( Xi ) as Xi : = LΠ ( wτi ) . Note that by Lemma 25 and Equation ( 248 ) in Lemma 26 , we have ℵℓ − l ≤ Xi ≤ ℵ + 3l . The key idea behind defining Xi ’ s is that we want to bound the MGF of LΠ ( wt ) , without worrying about falling out of the assumptions of Lemmas 24 and 26 . With the definition of ( Xi ) and Gi , we are ready to state the theorem which roughly says the sequence τi will most likely stop after a number of steps . Lemma 27 Let QR : = ⋃∞ i=1 ( { τi ≤ R } ∩ Ḡi ) . Then , for some R = O ( log ( 1/δ1 ) ) ( ℵ+ 3l ) θη2 , ( 250 ) we have P ( QR ) ≤ δ1 . In other words , after R iterations of PSGD , the defined sequence ( Xi ) above has either been in situation ⋆ ( 1 ) or ⋆ ( 4 ) . Proof By Equations ( 229 ) and ( 245 ) in Lemmas 24 and 26 , there exist a constant θ depending polynomially on all parameters except η such that E [ Xi+1 −Xi| Ḡi ] ≤ − ( τi+1 − τi ) η2 . ( 251 ) Now for some constant C that we specify later , define the random time ı as the largest i where τi ≤ C/η2 . Using the fact that Gi−1 ⊆ Gi , for every i we have a.s. : Xi+11 { Ḡi } −Xi1 { Ḡi−1 } = 1 { Gi − Gi−1 } ( −Xi ) + ( Xi+1 −Xi ) 1 { Ḡi } . Now summing this for i = 1 to ı , taking expectation from both sides and using ( 251 ) : EXı+11 { Ḡı } −X0 = ∞∑ i=1 E1 { Gi − Gi−1 } ( −Xi ) 1 { ı ≥ i } + ∞∑ i=1 ( Xi+1 −Xi ) 1 { Ḡi ∩ { ı ≥ i } } ≤ ∞∑ i=1 E ( 1 { Gi } − 1 { Gi−1 } ) ( −Xi ) + ∞∑ i=1 E ( Xi+1 −Xi ∣∣∣ Ḡi ∩ { ı ≥ i } ) P ( Ḡi ∩ { ı ≥ i } ) ≤ sup i sup |Xi| + ∞∑ i=1 E ( − ( τi+1 − τi ) η2 ∣∣∣ Ḡi ∩ { ı ≥ i } ) P ( Ḡi ∩ { ı ≥ i } ) = sup i sup |Xi| − η2 ∞∑ i=1 E ( τi+1 − τi ) 1 { Ḡi ∩ { ı ≥ i } } . Now using Lemma 25 , we know that in except when ET happens ( in which we stop the time sequence τi ) , the increments of Xi are at most l. Therefore , the value of Xi ’ s always remain bounded by ℵ+ 3l , hence : LHS ≤ ℵ+ 3l − θη2 ∞∑ i=1 E ( τi+1 − τi ) 1 { Ḡi ∩ { ı ≥ i } } . Also , by restricting the integration of the second term to the part ⋃∞ i=1 ( Ḡi ∩ { τi ≥ 2C/η2 } ) of the sample space , we know that under the event { ı ≥ i } , Ḡi automatically happens ( it is easy to check ) . Therefore : LHS ≤ ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ∞∑ i=1 ( τi+1 − τi ) 1 { Ḡi ∩ { ı ≥ i } } = ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ∞∑ i=1 ( τi+1 − τi ) 1 { ı ≥ i } = ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ı∑ i=1 ( τi+1 − τi ) = ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } τı+1 . Now by the definition of ı , τı+1 ≥ C/η2 : LHS ≤ ℵ+ 3l − θη2E1 { ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) } ( C/η2 ) ℵ+ 3l − CθP ( ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) ) . But note that Xi ’ s are a.s. bounded between 0 and ℵ + 3l , which implies the LHS above is at least − ( ℵ+ 3l ) . Therefore , we finally get : P ( ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C/η2 } ) ) ≤ 2ℵ+ 6l Cθ . Therefore , by picking C∗ = 2 ( 2ℵ+ 6l ) /θ : P ( ∞⋃ i=1 ( Ḡi ∩ { τi ≥ C∗/η2 } ) ) ≤ 1 2 . ( 252 ) Note that the differences between τi ’ s is at most Tmax = Õ ( 1/η ) Ge et al . ( 2015a ) . Hence , again for η polynomially small in other parameters , ( 252 ) implies that for R = 2C ∗ η2 , there exist R̃ = poly ( . ) such that after R̃ iterations on the main sequence ( wt ) , the corresponding sequence ( τi ) has either been in ⋆ ( 1 ) or ⋆ ( 4 ) with chance at least 1/2 . Repeating this argument log ( 1/δ1 ) times ( using the markov property of the process ) we conclude the proof . A.18 BOUNDING THE MGF OF Xi ’ S Next , we want to exploit Xi ’ s to bound the upward deviation of LΠ ( wt ) . For a fix θ the goal here is to bound E [ exp { θXi } ] ( this is a different θ ! ) . More precisely , let Ft be the sub-sigma field generated by variables wt from time zero to t , and Fi : = Fτi be the sigma field of the stop time τi . Then , obviously , Xi is measurable w.r.t Fi . We prove the following theorem : Theorem 8 For any θ > 0 , the sequence ( Eeθ ( Xi−X0 ) ) ∞i=1 is a supermartingale with respect to the filteration ( Fi ) , Proof We proceed inductively by jointly conditioning on the previous Xi and whether Gi has happened or not , and whether we are in situation ⋆ ( 2 ) or ⋆ ( 3 ) . We have E [ exp { θ ( Xi+1 −X0 ) } | Fi ] = E [ exp { θ ( Xi+1 −Xi +Xi −X0 ) } 1 { Gi } |Fi ] + E [ exp { θ ( Xi+1 −Xi +Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } |Fi ] + E [ exp { θ ( Xi+1 −Xi +Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } |Fi ] . Now by the a.s. bounds of Lemmas 24 and 26 : E [ Xi+1 −Xi| Ḡi , wτis.t . ⋆ ( 2 ) ] = −R21η2 , E [ Xi+1 −Xi| Ḡi , wτis.t . ⋆ ( 3 ) ] = −R22η2 , ( Xi+1 −Xi ) 1 { Gi } = 0 . ( a.s. ) Where R1 and R2 are r.v . defined in Lemmas 24 and 26 and are clearly Fi measurable . This implies E [ ( Xi+1 −Xi ) 1 { Ḡi , wτis.t . ⋆ ( 2 ) } | Fi ] = −R21η21 { Ḡi , wτis.t . ⋆ ( 2 ) } , E [ ( Xi+1 −Xi ) 1 { Ḡi , wτis.t . ⋆ ( 3 ) } | Fi ] = −R22η21 { Ḡi , wτis.t . ⋆ ( 3 ) } . Now we mention the following fact : Fact For a σ subGaussian random variable X we have E [ exp { θX } ] ≤ exp { θ2σ2 } . Using the a.s. bounds of Lemmas 24 and 26 , we get that conditioned on { Ḡi , wτi s.t . ⋆ ( 2 ) } , Xi+1 − EXi+1 is a.s. bounded by 2ηθR1/ ( ρ2χ ) , and conditioned on { Ḡi , wτi s.t . ⋆ ( 3 ) } , Xi+1 − EXi+1 is bounded by 2ηθR2/ ( ρ2χ ) . Therefore , using the above fact E [ exp { θ ( Xi+1 − E ( Xi+1| Ḡi , wτi s.t . ⋆ ( 2 ) ) ) } ∣∣∣ Ḡi , wτi s.t . ⋆ ( 2 ) ] ≤ exp { 4η2θ2R21/ ( ρ2χ ) } , ( 253 ) E [ exp { θ ( Xi+1 − E ( Xi+1| Ḡi , wτi s.t . ⋆ ( 3 ) ) ) ∣∣∣ Ḡi , wτi s.t . ⋆ ( 3 ) } ] ≤ exp { 4η2θ2R22/ ( ρ2χ ) } , ( 254 ) which implies in the notation of conditional expectation on sigma field : E [ exp { θ ( Xi+1 − E ( Xi+1|Fi ) ) } 1 { Ḡi , ⋆ ( 2 ) } ∣∣∣Fi ] ≤ exp { 4η2θ2R21/ ( ρ2χ ) } 1 { Ḡi , ⋆ ( 2 ) } , ( 255 ) E [ exp { θ ( Xi+1 − E ( Xi+1|Fi ) ) } 1 { Ḡi , ⋆ ( 3 ) } ∣∣∣Fi ] ≤ exp { 4η2θ2R22/ ( ρ2χ ) } 1 { Ḡi , ⋆ ( 3 ) } . ( 256 ) Now we write : LHS ≤ E [ exp { θ ( Xi −X0 ) } 1 { Gi−1 } | Fi ] +E [ exp { θ ( Xi+1 − E [ Xi+1| Fi ] ) } exp { θ ( E [ Xi+1| Fi ] −Xi ) } exp { θ ( Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } | Fi ] +E [ exp { θ ( Xi+1 − E [ Xi+1| Fi ] ) } exp { θ ( E [ Xi+1| Fi ] −Xi ) } exp { θ ( Xi −X0 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } |Fi ] ≤ exp { θ ( Xi −X0 ) } 1 { Gi } + exp { θ ( Xi −X0 ) } E [ exp { θ2R21η2/ ( ρ2χ ) } exp { −θ ( R21η2 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } | Fi ] + exp { θ ( Xi −X0 ) } E [ exp { θ2R22η2/ ( ρ2χ ) } exp { −θ ( R22η2 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } | Fi ] ≤ exp { θ ( Xi −X0 ) } E [ ( 1 { Gi } + exp { θ2R21η2/ ( ρ2χ ) − θ ( R21η2 ) } 1 { Ḡi ∩ ⋆ ( 2 ) } + exp { θ2R22η2/ ( ρ2χ ) − θ ( R22η2 ) } 1 { Ḡi ∩ ⋆ ( 3 ) } ) | Fi ] . Now setting θ : = 1 and picking χ ≥ 1/ρ2 : LHS ≤ exp { ( Xi −X0 ) } E [ 1 { Gi−1 } + 1 { Ḡi−1 ∩ ⋆ ( 2 ) } + 1 { Ḡi−1 ∩ ⋆ ( 3 ) } | Fi ] . = exp { Xi −X0 } . Now by hypothesis of Induction we have E [ exp { Xi+1 −X0 } ] = E [ E [ exp { Xi+1 −X0 } | Fi ] ] ≤ E [ exp { Xi −X0 } ] ≤ 1 , which finishes the proof of step of induction . Now using Doob ’ s Maximal inequality for positive supermartingales and R defined in ( 250 ) : P ( sup 1≤i≤R ( Xi −X0 ) ≥ z ) = P ( sup 1≤i≤R exp { Xi −X0 } ≥ exp { z } ) ≤ E [ exp { XR −X0 } ] / exp { z } ≤ e−z . ( 257 ) A.19 PROOF OF THEOREM 7 Finally with the developed tools , we are ready to prove Theorem 7 . Proof of Theorem 7 Starting from w0 = 0 with LΠ ( w0 ) ≤ ℵ , we use Equation ( 257 ) to get P ( sup1≤i≤R exp { Xi − X0 } ≥ Ω ( log ( 1/δ1 ) ) ) ≤ δ1 . Therefore , setting l = Θ ( log ( 1/δ1 ) ) and a union bound implies with probability at least 1 − 2δ1 we should have gotten into situation ⋆ ( 1 ) or ⋆ ( 4 ) without the value of Xi exceeding ℵ + 2l . On the other hand , using Lemma 26 we know that ET happens with probability at least 1− Õ ( η5 ) for every 1 ≤ t ≤ R which is equal to τi for some i and when we are in the situation of Lemma 26 . As a result , the chance that even one of ET ’ s happen along R iterations is at most RÕ ( η5 ) = η3 O ( log ( 1/δ1 ) ) ( ℵ+ 3l ) θ . But picking η small enough with respect to log ( δ1 ) and other parameters , we conclude that with probability at least 1 − 3δ , after R rounds , we should have got into situation ⋆ ( 1 ) and not ⋆ ( 4 ) and not exceeding ℵ+2l , which means that Xi = LΠ ( wτi ) has gotten under the threshold ℵℓ . Note that as soon as that happens , we terminate the algorithm . We elaborate on this more in Appendix A.10 . A.20 GAUSSIAN SMOOTHING In this section , we describe our smoothing scheme and the approximation that it provides which enables us to keep the signs from the case η = 0 . Recall that we use Gaussian smoothing matrices V ρj , k ∼ N ( 0 , β21/m1 ) and W ρ j , k ∼ N ( 0 , β22/m2 ) . Here , we will particularly specify lower bounds for β1 and β2 in order for our sign approximation to be precise . On the other hand , we normally prefer the smoothing noise to be as low as possible so the primary and smoothed functions are close , so we set β1 , β2 equal to their lower bounds , and use this setting in the other parts . To begin fix one of the inputs xi . In order to reduce and simplify the amount of notations , we refer to the sign pattern matrix ( diagonal sign matrix ) of both the first and second layers by D with the appropriate indices . More specifically , for the first layer , we refer to Sgn ( W ( 0 ) +W ′ +W ρ ) xi by D′ , ρ and Sgn ( W ( 0 ) + ( 1− η/2 ) W ′ +W ρ + √ ηW ∗ ) xi by D′ , ρ . Similarly , for the second layer , of course depending on the input vector , we refer to the sign matrix with respect to the matrices V ( 0 ) + V ′ + V ρ and V ( 0 ) + ( 1 − η/2 ) V ′ + V ρ +√ηV ∗ by by D′.ρ and D′ , ρ , η , respectively . We introduce two new notations as well for the output of the first layer with respect to different matrix and sign patterns : x′ ( 1 ) : =W sD′ , ρ ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi , ( 258 ) x′ ( 2 ) : =W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi . ( 259 ) For further brevity , we sometimes refer to x′ ( 2 ) by x′ . Now we are ready to mention our approximation theorem regarding the smoothing and the sign changes . Lemma 28 Under the conditions κ1 √ m3 ≳ C1 + β1 √ m3 and m2 ≥ m3 log ( m2 ) , then for every i ∈ [ n ] : ∣∣∣EWρ , V ρ aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sD′ , ρ ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηV ∗ ) xi ∣∣∣ ≤ ηϱ22β−12 [ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } ] × [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] + η ( κ2 √ m2 + C1 ) ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) ϱ2m3√m3 β1 : = ηℜ8 . ( 260 ) Proof of Lemma 28 We can bound the Left hand side above as LHS ≤∣∣∣EWρ , V ρaTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi ∣∣∣ + ∣∣∣EWρ , V ρ aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sD′ , ρ , η ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sD′ , ρ ( W ( 0 ) + ( 1− η ) W ′ +W ρ +√ηW ∗ ) xi ∣∣∣ . : = A1 +A2 . ( 261 ) We bound A1 and A2 separately . First , we start with A1 . Let P̂i be the set of indices j for which 1 { | ( V ( 0 ) j + V ′j ) x′| ≤ R∗κ2∥x′∥ } happens . Then , from Lemma 31 , we have |P̂i| ≲ R∗m2 . Now for j ∈ [ m2 ] , we write 1 { sign change in thejth neuron } × |amount of change| ( 262 ) ≤ 1 { V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ + ηV ′j x′ − √ ηV ∗j x ′ , −V ( 0 ) j x ′ − V ′j x′ ) } × 1√ m2 ( |ηV ′j x′|+ | √ ηV ∗j x ′| ) ( 263 ) Moreover , note that |V ′j x′| = |V ′j ( x′ − ϕ ( 0 ) ( xi ) ) | , |V ∗j x′| = |V ∗j ( x′ − ϕ ( 0 ) ( xi ) ) | . Also , because ∥V ′j ∥ ≤ ∥V ′∥ ≤ 2C2 plus using Equation ( 129 ) , we can further upper bound the above indicator as : ≤ 1 { V ρj x ′ ∈ ( −V ( 0 ) j x ′−V ′j x′− ( η∥V ′j ∥+ √ η∥V ∗j ∥ ) min { ∥x′∥ , ∥x′−ϕ ( 0 ) ( xi ) ∥ } , −V ( 0 ) j x ′−V ′j x′ ) } × 1√ m2 ( η∥V ′j ∥+ √ η∥V ∗j ∥ ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≤ 1 { V ρj x ′ ∈ ( −V ( 0 ) j x ′−V ′j x′− ( 2ηC2+ √ ηϱ2/ √ m2 ) min { ∥x′∥ , ∥x′−ϕ ( 0 ) ( xi ) ∥ } , −V ( 0 ) j x ′−V ′j x′ ) } × 1√ m2 ( 2ηC2 + √ ηϱ2/ √ m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ . Taking √ η ≤ ϱ/ ( 2C2 √ m2 ) , we can further upper bound as ≲ 1 { V ρj x′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) } × ( √ηϱ2/m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ . Therefore , conditioned on x′ : EV ρ [ 1 { sign change in thejth neuron } × |amount of change| | x′ ] ≤ P ( V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) ) × ( √ηϱ2/m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ . Now notice that for j /∈ P̂i , we have | − V ( 0 ) j x ′ − V ′j x′| ≥ R∗κ2∥x′∥ . Also , note that the variable V ρj x ′ is gaussian with variance ∥x′∥β2/ √ m2 Therefore , conditioned on x′ , for j /∈ P̂i , we have ( note that x′ does not depend on the randomness of V ρ ) : P ( V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) ) ≲ exp { min { | − V ( 0 ) j x′ − V ′j x′ − 2 √ ηϱ2∥x′∥/ √ m2| , | − V ( 0 ) j x ′ − V ′j x′| } / ( √ 2∥x′∥β2/ √ m2 ) } 2 × ( ∥x′∥β2/ √ m2 ) −1 × ( √ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 ) . This equation follows from the fact that P ( a ≤ N ≤ b ) ≲ |a− b| σ emin 2 { a , b } /σ2 . ( 264 ) On the other hand , note that for √ η ≲ C 2/3 2 ( √ m2κ2 ) 1/3 ϱ2 we have : R∗κ2∥x′∥/2 = C 2/3 2 ( √ m2κ2 ) 1/3 2 √ m2 ∥x′∥ ≥ 2√ηϱ2∥x′∥/ √ m2 which implies ≲ exp { R∗κ2∥x′∥/ ( 2 √ 2∥x′∥β2/ √ m2 ) } 2 ( √ ηϱ2/β2 ) ≤ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } ( √ ηϱ2/β2 ) . On the other side , for j ∈ P̂i , we can write P ( V ρj x ′ ∈ ( −V ( 0 ) j x ′ − V ′j x′ − 2 √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 , −V ( 0 ) j x ′ − V ′j x′ ) ) ≲ ( ∥x′∥β2/ √ m2 ) −1 ( √ ηϱ2 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥ , ∥x′∥ } / √ m2 ) = min { ∥x′ − ϕ ( 0 ) ( xi ) ∥/∥x′∥ , 1 } √ ηϱ2/β2 . Therefore , overall using the fact that ∥V ∗j ∥ ≤ ϱ2/ √ m2 , we can write A1 ≲ ∑ j /∈P̂ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } ( √ ηϱ2/β2 ) min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } + ∑ j∈P̂ ( √ ηϱ2/β2 ) min { ∥x′ − ϕ ( 0 ) ( xi ) ∥/∥x′∥ , 1 } × ( √ ηϱ2/m2 ) ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≲ [ m2 × exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } ( √ ηϱ2/β2 ) × ( √ ηϱ2/m2 ) + ηϱ22β −1 2 |P̂i| m2 ] min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } ≤ ηϱ22β−12 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] . ( 265 ) Next , we boundA2 . First we bound EWρ∥x′ ( 1 ) −x′ ( 2 ) ∥ . Recalling the setting c2 = 2 √ nm3C1/ √ λ0 and the definition of in P from Lemma 1 , we obtain that for j /∈ P , we have for all i ∈ [ n ] : |W ( 0 ) j xi| ≥ c2/ √ m1 , |W ′jxi| ≤ c2/ ( 2 √ m1 ) , which means for j /∈ P : | ( W ( 0 ) j +W ′ j ) xi| ≥ c2/ ( 2 √ m1 ) . ( 266 ) Also , we have |P | ≤ c2 √ m1/κ1 . ( 267 ) Now using Equation ( 105 ) in Lemma 5 , we can write for every i ∈ [ n ] : valj : = 1 { sign change in thejth neuron } × ∣∣∣amount of change∣∣∣ ≤ 1 { W ρj xi ∈ ( −W ( 0 ) j x−W ′ jxi + ηW ′ jxi − √ ηW ∗j xi , −W ( 0 ) j xi −W ′ jxi ) } × 1√ m1 ( |√ηW ∗j xi + ηW ′jxi| ) . Using the fact that ∥W ′j∥ ≤ ∥W ′∥F ≤ C1 , and Equation ( 105 ) ( ∥W ∗j ∥ ≤ ϱ √ m3 m1 ) and picking √ η ≤ ϱ √ m3 C1 √ m1 , we obtain ≤ 1 { W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − η∥W ′j∥ − √ η∥W ∗j ∥ , −W ( 0 ) j xi −W ′ jxi ) } × 1√ m1 ( √ η∥W ∗j ∥+ η∥W ′j∥ ) ≤ 1 { W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − 2 √ ηϱ √ m3√ m1 , −W ( 0 ) j xi −W ′ jxi ) } × 2√ m1 ( √ ηϱ √ m3 m1 ) . Now for j /∈ P , because W ρj xi is Gaussian with std β1√ m1 : EWρ [ valj ] ≤ P ( W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − 2 √ ηϱ √ m3√ m1 , −W ( 0 ) j xi −W ′ jxi ) ) × 1√ m1 √ ηϱ √ m3√ m1 ≲ exp− { min { | −W ( 0 ) j xi −W ′jxi − 2 √ ηϱ √ m3√ m1 | , | −W ( 0 ) j x ′ −W ′jx′| } / ( √ 2β1/ √ m1 ) } 2 × ( β1/ √ m1 ) −1 × ( √ηϱ √ m3√ m1 ) × 1√ m1 √ ηϱ √ m3√ m1 . Now from Equation ( 266 ) and by picking √ η ≲ c2 ϱ √ m3 so that 2 √ ηϱ √ m3√ m1 ≤ c2/ ( 4 √ m1 ) , then LHS ≲ exp { −c22/ ( 32β21 ) } η ϱ2m3 β1m1 . ( 268 ) On the other hand , for j ∈ P we have Evalj ≤ P ( W ρj xi ∈ ( −W ( 0 ) j xi −W ′ jxi − 2 √ ηϱ √ m3√ m1 , −W ( 0 ) j xi −W ′ jxi ) ) × 1√ m1 √ ηϱ √ m3√ m1 ≲ ( β1/ √ m1 ) −1√ηϱ √ m3√ m1 ×√ηϱ √ m3 m1 = η ϱ2m3 β1m1 . ( 269 ) Now define the following random variable with respect to the randomness of W ρ : V al : = m1∑ j=1 1 { sign change in thejth neuron } × |amount of change| then for every k ∈ [ m3 ] , we have |x′ ( 1 ) k − x ′ ( 2 ) k | ≤ V al , which implies ∥x′ ( 1 ) − x′ ( 2 ) ∥ ≤ √ m3V al . But Combining Equations ( 268 ) and ( 269 ) : EV al ≤ ( exp { −c22/ ( 32β21 ) } + |P | m1 ) η ϱ2m3 β1 ≤ ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) η ϱ2m3 β1 , which implies EWρ∥x′ ( 1 ) − x′ ( 2 ) ∥ ≤ ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) η ϱ2m3 √ m3 β1 . ( 270 ) Now we can write∣∣∣aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 2 ) − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 1 ) ∣∣∣ ≤ 1√ m2 m2∑ j=1 | ( V ( 0 ) j + ( 1− η ) V ′ j + V ρ j + √ ηV ∗j ) ( x ′ ( 2 ) − x′ ( 1 ) ) | ≤ 1√ m2 m2∑ j=1 |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ ( 1− η ) |V ′j ( x′ ( 2 ) − x′ ( 1 ) ) |+ |V ρ j ( x ′ ( 2 ) − x′ ( 1 ) ) |+√η|V ∗j ( x′ ( 2 ) − x′ ( 1 ) ) | = 1√ m2 m2∑ j=1 |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ |V ρj ( x ′ ( 2 ) − x′ ( 1 ) ) |+ 1√ m2 m2∑ j=1 ( ( 1− η ) ∥V ′j ∥+ √ η∥V ∗j ∥ ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ ≤ 1√ m2 m2∑ j=1 |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ |V ρj ( x ′ ( 2 ) − x′ ( 1 ) ) |+ ( ( 1− η ) ∥V ′∥F + √ η∥V ∗∥F ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ . Now by Equation ( 127 ) in Lemma 10 ( i.e . ∥V ∗∥F ≲ √ ζ2 ) and the fact that ∥V ′∥F ≤ C2 , and by taking √ η ≤ C2√ ζ2 , we have ( ( 1− η ) ∥V ′∥F + √ η∥V ∗∥F ) ≲ C1 , ( 271 ) so we can bound the above as LHS ≲ 1√ m2 m2∑ j=1 ( |V ( 0 ) j ( x ′ ( 2 ) − x′ ( 1 ) ) |+ |V ρj ( x ′ ( 2 ) − x′ ( 1 ) ) | ) + C1∥x′ ( 2 ) − x′ ( 1 ) ∥ . Furthermore , using Lemma 32 and noting the fact that the entries of V ( 0 ) are normal with standard deviation κ2 , we get with high probability over the randomness of V ( 0 ) : ≲ 1√ m2 m2∑ j=1 |V ρj ( x ′ ( 2 ) −x′ ( 1 ) ) |+ ( κ2 √ m2+κ2 √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) +C1 ) ∥x′ ( 2 ) −x′ ( 1 ) ∥ . Now note that V ρj ( x ′ ( 2 ) −x′ ( 1 ) ) is normal with standard deviation β2√ m2 ∥x′ ( 2 ) −x′ ( 1 ) ∥ . Hence , taking expectation with respect to V ρ : EV ρ ∣∣∣aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 2 ) − aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) x′ ( 1 ) ∣∣∣ ≲ ( κ2 √ m2 + κ2 √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) + C1 ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ . ( 272 ) Finally , Combining Equation ( 265 ) and ( 272 ) and applying it to Equation ( 261 ) implies with high probability over the random initialization : ∣∣∣EWρ , V ρ aTD′ , ρ , η ( V ( 0 ) + ( 1− η ) V ′ + V ρ +√ηV ∗ ) W sx′ ( 2 ) − aTD′ , ρ ( V ( 0 ) + ( 1− η ) V ′ + V ρ + √ ηV ∗ ) W sx′ ( 1 ) ∣∣∣ ≤ A1 +A2 ≲ EWρηϱ22β−12 min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] + EWρ ( κ2 √ m2 + κ2 √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) + C1 ) ∥x′ ( 2 ) − x′ ( 1 ) ∥ Now notice that under Ec , using the assumption κ1 √ m3 ≳ C1 + √ m3β1 and Lemma 33 we have ∥x′∥ ≥ ∥ϕ ( 0 ) ( xi ) ∥ − ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≥ κ1 √ m3 − ( C1 + √ m3β1 ) ≳ κ1 √ m3 , and ∥x′ − ϕ ( 0 ) ( xi ) ∥2 ≤ ( C1 + √ m3β1 ) 2 , ( 273 ) which implies : EWρ min { ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥ , ∥x′ − ϕ ( 0 ) ( xi ) ∥ } ≤ EWρ1 { Ec } ∥x′ − ϕ ( 0 ) ( xi ) ∥2/∥x′∥+ 1 { E } ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≲ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + EWρ1 { E } ∥x′ − ϕ ( 0 ) ( xi ) ∥ ≲ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } . Substituting this above and further applying the result of Lemma 33 and Equation ( 270 ) and the assumption that m2 ≥ m3 log ( m2 ) : A1 +A2 ≲ ηϱ22β−12 [ ( C1 + √ m3β1 ) 2/ ( κ1 √ m3 ) + [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } ] × [ exp { −C4/32 ( √ m2κ2 ) 2/3/ ( 8β22 ) } + C 2/3 2 ( √ m2κ2 ) 2/3 ] + η ( κ2 √ m2 + C1 ) ( exp { −c22/ ( 32β21 ) } + c2 κ1 √ m1 ) ϱ2m3√m3 β1 , which completes the proof . A.20.1 SETTING β1 AND β2 As we mentioned , to minimize the amount of deviation of the smoothed function compared to the original one , we prefer to choose β1 , β2 as small as possible . ( The benefit of such choice , indeed , can be observed more explicitly in other parts of the proof , e.g . Appendix A.14 . ) Observing the bound in Equation ( 260 ) and noting that we can easily make the exponential terms orders of magnitude smaller than the poly terms , it is easy to find the following optimal setting for the smoothing parameters : β2 : = Θp ( ( κ1 √ m3 ) −1 ( √ m2κ2 ) − 23 ) , β1 : = Θp ( m3 √ m3/ ( κ1 √ m1 ) ) . Using this setting , we still can make ℜ8 arbitrarily small . Here , we remind the reader that Op only cares about the non-logarithmic dependencies on the overparameterization , i.e . m1 , m2 , m3 , κ1 , κ2 . A.21 BASIC TOOLS In this section , we introduce and prove some lemmas that we use in our analysis as basic tools . Lemma 29 Suppose V ( 0 ) ∈ Rm2×m3 has standard normal entries and a is a random sign vector . Suppose theta > 1 , R < 1 are given thresholds , such that m2R ≳ m3 ( log ( 1/R ) + log ( m3 ) + log ( log ( m2 ) ) , e−θ 2/8 ≲ m3/m2 . Then , for the following quantities : N1R ( x ) = # ( j ∈ [ m ] : |V ( 0 ) j x| ≤ R ) N2θ ( x ) = # ( j ∈ [ m ] : |V ( 0 ) j x| ≥ θ ) , with high probability we have sup ∥x′∥=1 N1R ( x ′ ) ≲ m2R , sup ∥x′∥=1 N2θ ( x ′ ) ≲ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Proof of Lemma 29 Suppose B1 ( ϵ ) is a cover for the Euclidean ball in Rm3 with precision ϵ . We know |B1 ( ϵ ) | ≲ ( 1/ϵ ) m3 . Now for a fixed ∥x∥ = 1 , we have P ( W ( 0 ) j x ≤ 2R ) ≲ R. Therefore , using Bernstein , with high probability we have # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≤ 2R ) ≲ m2R+ √ m2R+ 1 . Hence , using union bound , we have with high probability sup x∈B1 ( ϵ ) # ( j ∈ [ m ] : |V ( 0 ) j x| ≤ 2R ) ≲ m2R+ √ log |B1 ( ϵ ) | √ m2R+ log |B1 ( ϵ ) | = m2R+ √ m2Rm3 log ( 1/ϵ ) +m3 log ( 1/ϵ ) . By picking ϵ ≲ R/ ( √ m3 log ( m2m3 ) ) , The assumption implies m2R ≥ m3 log ( 1/ϵ ) , which implies LHS ≲ m2R . On the other hand , note that with high probability we have sup j∈ [ m2 ] , k∈ [ m3 ] |V ( 0 ) j , k | ≤ √ log ( m2m3 ) . ( 274 ) Now for ∥x′∥ = 1 which is not in the cover , if x is the closest point to it in the cover , i.e . x ∈ B1 ( ϵ ) and ∥x− x′∥ ≤ ϵ , then for every j ∈ [ m2 ] we have ||V ( 0 ) j x| − |V ( 0 ) j x ′|| ≤ ∥V ( 0 ) j ∥∥x− x ′∥ ≤ √ m3 log ( m2m3 ) ϵ ≤ R , by picking a small enough constant . Therefore , for a j that |V ( 0 ) j x| ≥ 2R , then |V ( 0 ) j x ′| ≥ 2R−R = R. Therefore , we get that with high probability , for every ∥x′∥ = 1 : sup ∥x′∥=1 # ( j ∈ [ m2 ] : |V ( 0 ) j x ′| ≤ R ) ≲ m2R . For the second part , note that for ∥x∥ = 1 , by the tail bound for normal vars : P ( W ( 0 ) j x ≥ θ/2 ) ≲ e−θ 2/8 . Hence , again using Bernstein , we have with high probability sup x∈B1 ( ϵ ) # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≥ θ/2 ) ≲ m2e−θ 2/8 + √ log |B1 ( ϵ ) | √ m2e−θ 2/8 + log |B1 ( ϵ ) | ≲ m2e−θ 2/8 + √ m3 log ( 1/ϵ ) √ m2e−θ 2/8 +m3 log ( 1/ϵ ) . By picking ϵ ≲ 1/ ( √ m3 log ( m2m3 ) ) , and using the assumption m2e−θ 2/8 ≲ m3 , all terms are dominated by the third term so we can bound the above as sup x∈B1 ( ϵ ) # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≥ θ/2 ) ≲ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Now for ∥x′∥ = 1 not in the cover , for the new ϵ we can write ||V ( 0 ) j x| − |V ( 0 ) j x ′|| ≤ ∥V ( 0 ) j ∥∥x− x ′∥ ≤ √ m3 log ( m2m3 ) ϵ ≤ 1/2 ≤ θ/2 . Hence , with high prob . sup ∥x′∥=1 # ( j ∈ [ m2 ] : |V ( 0 ) j x| ≥ θ ) ≲ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Lemma 30 For x ∈ Rd and W ( 0 ) ∈ Rm×d which has standard normal entries ( and a is a random sign vector ) , we have with high probability : sup ∥x∥=1 f ( x ) : = 1√ m aTσ ( W ( 0 ) x ) ≤ √ d. Proof of Lemma 30 For the first part , we first compute an upper bound on E sup ∥x∥=1 1√ m aTσ ( W ( 0 ) x ) . To do so , we use Dudley ’ s chaining . Note that the for x1 , x2 ∈ Rd , the variable σ ( W ( 0 ) j x1 ) − σ ( W ( 0 ) j x2 ) is subGaussian with parameter ∥x1 − x2∥ , so the variable f ( x1 ) − f ( x2 ) is also sub- Gaussian with parameter ∥x1 − x2∥ . Hence , by Dudley ’ s integral : E sup ∥x∥=1 1√ m aTσ ( W ( 0 ) x ) ≤ ∫ 1 0 √ log ( N ( Bd1 , ϵ ) ) ≲ √ d. Now for a fixed x , note that 1√ m aTσ ( W1x ) − 1√ m aTσ ( W2x ) ≤ 1√ m m∑ j=1 ∥W1j −W2j∥ ≤ ∥W1 −W2∥F . Hence , the function f ( x ) is 1-lipchitz with respect to W and l2 norm , so is the function sup f ( x ) . Hence , by Gaussian concentration , sup f ( x ) is 1-subGaussian around its mean , so we finally get with high probability sup f ( x ) ≲ √ d+ 1 ≲ √ d. Lemma 31 For R∗ : = C 2/3 2 ( √ m2κ2 ) 2/3 , we have with high probability over the randomness of V ( 0 ) : sup x′ , V ′ : ∥V ′∥≤C2 # ( j ∈ [ m2 ] : | ( V ( 0 ) j + V ′ j ) x ′| ≤ R∗κ2∥x′∥ ) ≲ R∗m2 . Proof of Lemma 31 Note that obviously the condition of Lemma 29 is satisfied with this choice of R = R∗ . Therefore , with high probability we have for an arbitrary x′ : # ( |V ( 0 ) j x ′| ≤ 2R∗κ2∥x′∥ ) ≤ m2R∗ . On the other hand , note that for j ∈ [ m2 ] such that |V ′j x′| ≥ Rκ2∥x′∥ , we have ∥V ′j ∥∥x′∥ ≥ |V ′j x′| ≥ Rκ2∥x′∥ , which implies ∥V ′j ∥ ≥ Rκ2 . Therefore , there are at most C 2 2 R2κ22 . Therefore , setting aside m2R+ C22 R2κ22 of j ’ s , for the rest we have | ( V ( 0 ) j + V ′ j ) x ′| ≥ |V ( 0 ) j x ′| − |V ′j x′| ≥ 2Rκ2∥x′∥ −Rκ2∥x′∥ = Rκ2∥x′∥ . Setting R∗ as defined above balances the terms m2R and C22 R2κ22 , which completes the proof . Lemma 32 If V ( 0 ) ∈ Rm2×m3 is a matrix with standard normal entries , then with high probability sup ∥x′∥=1 1√ m2 m2∑ j=1 |V ( 0 ) j x ′| ≲ √m2 + √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . Proof of Lemma 32 LetB1 ( ϵ ) be a cover for the unit Euclidean ball with precision ϵ , for which we have |B1 ( ϵ ) | ≲ ( 1ϵ ) m3 . Now for a fixed x ∈ B1 ( ϵ ) , note that because V ( 0 ) j x is a standard normal variable , the random variable |V ( 0 ) j x′|−E|V ( 0 ) j x ′| isO ( 1 ) -subGaussian , which means 1√ m2 ∑m2 j=1 ( |V ( 0 ) j x ′|−E|V ( 0 ) j x′| ) is also O ( 1 ) −subGaussian . Now from the tail of maximum of subGaussian variables : sup x∈B1 ( ϵ ) 1√ m2 m2∑ j=1 ( |V ( 0 ) j x| − E|V ( 0 ) j x| ) ≲ √ log ( |B1 ( ϵ ) | ) = √ m3 log ( 1/ϵ ) . On the other hand , note that E|V ( 0 ) j x′| ) = O ( 1 ) , which implies w.h.p : sup x∈B1 ( ϵ ) 1√ m2 m2∑ j=1 |V ( 0 ) j x| ≲ √ m2 + √ m3 log ( 1/ϵ ) . Moreover , note that again by the tail of subGaussian variables , we have w.h.p : max j∈ [ m2 ] , k∈ [ m3 ] |V ( 0 ) j , k | ≲ √ log ( m2m3 ) , which implies with high prob for every j ∈ [ m2 ] : ∥V ( 0 ) j ∥ ≲ √ m3 log ( m2m3 ) . Now by picking ϵ : = ( √ m3 log ( m2m3 ) ) −1 , we get with high probability sup x∈B1 ( ϵ ) 1√ m2 m2∑ j=1 |V ( 0 ) j x| ≲ √ m2 + √ m3 ( log ( m3 ) + log ( log ( m2 ) ) ) . ( 275 ) On the other hand , for an arbitrary x′ with ∥x′∥ = 1 , if x ∈ B1 ( ϵ ) is the representative of x′ , we have by definition ∥x′ − x∥ ≤ ϵ , which combined with ( A.21 ) implies∣∣∣|V ( 0 ) j x′| − |V ( 0 ) j x|∣∣∣ ≤ |V ( 0 ) j ( x′ − x ) | ≤ ∥V ( 0 ) j ∥∥x′ − x∥ , ≲ √ m3 log ( m2m3 ) ( √ m3 log ( m2m3 ) ) −1 ≤ 1 . Therefore ∣∣∣ 1√ m2 m2∑ j=1 |V ( 0 ) j x ′| − 1√ m2 m2∑ j=1 |V ( 0 ) j x| ∣∣∣ ≤ √m2 . ( 276 ) Combining Equations ( 275 ) and ( 276 ) , we conclude the result . A.21.1 DEFINING THE RARE EVENTS Ej Lemma 33 For x′ ( 2 ) defined in Equation ( 259 ) we have EWρ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ , EWρ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 + √ m3β1 , EWρ∥ϕ ( 2 ) ( xi ) ∥2 ≲ C21 +m3β21 . Moreover , for the events Ej = { |W ρj xi| ≥ C1/ √ m3m1 } , E = ∪jEj , we have under Ec : ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ , ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 . Furthermore , E happens rarely : P ( E ) ≲ m1 exp { −C21/ ( 8m3β21 ) } , EWρ1 { E } ∥ϕ ( 2 ) ( xi ) ∥ ≤ [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } . EWρ1 { E } ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ ≤ [ √ m3m1β1 + C1 ] exp { −C21/ ( 8m3β21 ) } . Finally , we have the following almost surely bound : ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 + √ m3 m1∑ j=1 1√ m1 |W ρj xi| . Proof of Lemma 33 We start by writing |x′ ( 2 ) k − 1√ m1 W skσ ( W ( 0 ) + ( 1− η ) W ′ +√ηW ∗ ) xi| ≤ m1∑ j=1 1√ m1 |W ρj xi| . ( 277 ) Now notice that by Lemma 1 , we know for every j /∈ P̃ : |W ( 0 ) j xi| ≥ c2/ √ m2 , ( 278 ) | ( 1− η ) W ′jxi| ≤ c2/ ( 2 √ m1 ) . ( 279 ) In addition , by Equations in ( 105 ) from Lemma 5 , for every j ∈ [ m1 ] : ∥W ∗j ∥ ≤ ϱ1 √ m3 m1 , so by picking η ≤ c2/ ( 4ϱ √ m3 ) we obtain η|W ∗j xi| ≤ c2 4 √ m1 . ( 280 ) Combining this with Equations in ( 279 ) , we see that the signs of ( W ( 0 ) j + ( 1− η ) W ′j + √ ηW ∗j ) xi and W ( 0 ) j xi are the same for j /∈ P̃ . Moreover , the matrix ( 1− η ) W ′ +√ηW ∗ satisfies ∥ ( 1− η ) W ′ +√ηW ∗∥ ≤ ( 1− η ) C1 + √ η √ 2ζ2 ≤ C1 , by picking √ η ≤ C1/ √ ζ2 . Hence , the conditions of Lemma 6 are satisfied and we get : ∥ 1√ m1 W sσ ( W ( 0 ) + ( 1− η ) W ′ +√ηW ∗ ) xi − ϕ ( 0 ) ( xi ) ∥ ≤ C1 . ( 281 ) Combining Equations ( 277 ) and ( 281 ) : ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ ≤ C1 + √ m3 m1∑ j=1 1√ m1 |W ρj xi| . ( 282 ) In exactly similar fashion , one can derive ∥ϕ ( 2 ) ( xi ) ∥ ≤ C1 + √ m3 m1∑ j=1 1√ m1 |W ρj xi| . ( 283 ) Now first of all , note EWρ m1∑ j=1 1√ m1 |W ρj xi| ≤ β1 , which proves the first part of the claims . For the second part , note that by the Gaussian tail bound P ( |W ρj xi| ≥ C1/ √ m3m1 ) ≲ exp { −C21/ ( 8m3β21 ) } . Therefore , P ( E ) ≤ ∑ j P ( Ej ) ≤ m1 exp { −C21/ ( 8m3β21 ) } . Moreover EWρ1 { E } m1∑ j=1 1√ m1 |W ρj xi| ≤ EWρ 1√ m1 ∑ j2 ∑ j ̸=j2 1 { Ej2 } |W ρ j xi|+ 1√ m1 ∑ j EWρ1 { Ej } |W ρj xi| , = [ 1√ m1 ∑ j2 ∑ j ̸=j2 EWρ |W ρj xi|+ 1√ m1 ∑ j EWρ [ |W ρj xi| ∣∣∣ Ej ] ] P ( Ej ) ≲ [ m1β1 + C1/ √ m3 ] exp { −C21/ ( 8m3β21 ) } . Plugging this into ( 282 ) finishes the proof . Also , under Ec by Equation ( 282 ) we have ∥x′ ( 2 ) − ϕ ( 0 ) ( xi ) ∥ , ∥ϕ ( 2 ) ( xi ) ∥ ≲ C1 . Finally , exploiting Equation ( 283 ) : EWρ∥ϕ ( 2 ) ( xi ) ∥2 ≲ C21 +m3 1 m1 EWρ ( ∑ j |W ρj xi| ) 2 ≤ C21 +m3EWρ ∑ j |W ρj xi| 2 ≤ C21 +m3β21 . A.21.2 BOUNDING THE VALUE OF f ′ The following Lemma provides a reasonable bound on the value of the smoothed function . Lemma 34 We have the following general bound on the values of the smoothed function : With high probability over the initialization , for ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 and ∀i ∈ [ n ] ( having small enough choices of β1 , β2 described in Appendix A.20 ) : |f ′W ′ , V ′ ( xi ) | ≲ ( κ2 √ m3 + β2 ) ( √ m3κ1 + C1 + √ m3β1 ) + C2 ( C1 + √ m3β1 ) , which is O ( C1C2 ) for large enough overparameterization as described in AppendixA.3 . Moreover , we have the following almost surely bound ( with respect to the randomness of W ρ and V ρ ) : |fW ′+Wρ , V ′+V ρ ( xi ) | ≲ ( κ2 √ m3 + ∥V ρ∥F ) ( √ m3κ1 + C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) + C2 ( C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) . Notably , with slightly higher overparameterization , the high probability bound in ( 34 ) holds even if we take supremum over x . Proof of Lemma 34 Using Lemmas 30 and 33 and using the fact that ϕ ( 0 ) ( xi ) is orthogonal to the rows of V ′ ( recall x′i = ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ( xi ) ) : |fW ′+Wρ , V ′+V ρ ( xi ) | ≤ 1√ m2 aTσ ( V ( 0 ) x′i ) + 1√ m2 ∑ j | ( V ρj + V ′ j ) x ′ i| ≤ ( κ2 √ m3 ) ∥x′i∥+ 1√ m2 ∑ j |V ρj x ′ i|+ C2∥ϕ ( 2 ) ( xi ) ∥ ≤ ( κ2 √ m3 + ∥V ρ∥F ) ∥x′i∥+ C2∥ϕ ( 2 ) ( xi ) ∥ ≲ ( κ2 √ m3 + ∥V ρ∥F ) ( √ m3κ1 + C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) + C2 ( C1 + √ m3 ( 1√ m1 ∑ j |W ρj xi| ) ) . ( 284 ) Note that above , if we apply the stronger worst-case norm bound of the first layer ’ s output presented in Lemma 46 , we would get supx , ∥x∥=1 |fW ′+Wρ , V ′+V ρ ( x ) | is bounded by the RHS , which in turn proves a stronger uniform bound on f ′ . Similarly , this time by taking expectation with respect to W ρ and V ρ : |f ′W ′ , V ′ ( xi ) | = |EWρ , V ρfW ′ , V ′ ( xi ) | ≤ EWρ , V ρ |fW ′ , V ′ ( xi ) | = EWρ ( κ2 √ m3 + β2 ) ∥x′i∥+ C2∥x′i − ϕ ( 0 ) ( xi ) ∥ ≲ ( κ2 √ m3 + β2 ) ( √ m3κ1 + C1 + √ m3β1 ) + C2 ( C1 + √ m3β1 ) . Corollary 8.1 If we set C1 = C2 = 0 above , we get |f ′0,0 ( xi ) | ≤ ( κ2 √ m3 + β2 ) ( √ m3κ1 + √ m3β1 ) , the point being these terms go to zero by an order of O ( ( √ m2κ2 ) − 23 ) . Therefore , taking ( √ m2κ2 ) − 23 < < B , we make sure that |f ′0,0| < B , so by the 1 smoothness of ℓ and B boundedness of the labels we get ℓ ( f ′0,0 ( xi ) , yi ) < 4B 2 . A.21.3 BOUNDING THE DIFFERENCE BETWEEN ORIGINAL AND SMOOTHED FUNCTIONS The following Lemma bounds the difference between the smoothed function and original function of the network . Lemma 35 Bound on the smoothing change under the assumption m2 ≥ m3 log ( m2 ) : with high probability over the initialization , for any ( W ′ , V ′ ) with ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 : |fW ′ , V ′ ( xi ) − f ′W ′ , V ′ ( xi ) | ≤ β2 ( κ1 √ m3 + C1 + √ m3β1 ) + ( C2 + κ2 √ m2 ) √ m3β1 . Proof of Lemma 35 We write |fW ′ , V ′ ( xi ) − f ′W ′ , V ′ ( xi ) | = |fW ′ , V ′ ( xi ) − EWρ , V ρfW ′+Wρ , V ′+V ρ ( xi ) | = ∣∣∣EWρ , V ρ ( fW ′ , V ′ ( xi ) − fW ′+Wρ , V ′+V ρ ( xi ) ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣fW ′ , V ′ ( xi ) − fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣ . In the following , σ means we apply Relu activation to the vector in front of it ( entrywise ) : LHS ≤ EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ + V ρ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi − 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi ∣∣∣ +EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi − 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ ) xi ∣∣∣ . Now for the first term above , using the previous notation of x′i representing the output of the first layer and using Lemma 33 : EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ + V ρ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi − 1√ m2 aTσ ( V ( 0 ) + V ′ ) 1√ m1 W sσ ( W ( 0 ) +W ′ +W ρ ) xi ∣∣∣ ≤ EWρ , V ρ 1√ m2 ∑ j |V ρj x ′ i| ≲ β2EWρ∥x′i∥ ≤ β2EWρ , V ρ ( ∥ϕ ( 0 ) ( xi ) ∥+ ∥ϕ ( 2 ) ( xi ) ∥ ) ≲ β2 ( κ1 √ m3 + C1 + √ m3β1 ) . ( 285 ) For the second term , by starting off with a simple triangle inequality : EWρ , V ρ ∣∣∣ 1√ m2 aTσ ( V ( 0 ) + V ′ ) x′i − 1√ m2 aTσ ( V ( 0 ) + V ′ ) ( ϕ ( 0 ) ( xi ) + ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣ ≤ EWρ , V ρ 1√ m2 m2∑ j=1 ∣∣∣ ( V ( 0 ) j + V ′j ) ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣ ≤ EWρ , V ρ 1√ m2 m2∑ j=1 ∣∣∣V ( 0 ) j ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣+ ∣∣∣V ′j ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) |∣∣∣ ≤ C2EWρ∥x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ∥+ EWρ 1√ m2 m2∑ j=1 ∣∣∣V ( 0 ) j ( x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ) ∣∣∣ . Now using Lemma 32 : ≲ ( C2 + κ2 √ m2 ) EWρ∥x′i − ϕ ( 0 ) ( xi ) − ϕ ( 2 ) ̸ρ ( xi ) ∥ ≲ ( C2 + κ2 √ m2 ) √ m3EWρ ∑ j 1√ m1 |W ρj xi| ≲ ( C2 + κ2 √ m2 ) √ m3β1 . ( 286 ) Combining Equations ( 285 ) and ( 286 ) we conclude the proof . B APPENDIX CONTENTS B.1 Smoothness coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 B.1.1 Computing the Lipschitz Coefficients of f ′W ′ , V ′ . . . . . . . . . . . . . . . 98 B.2 Representation Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 B.2.1 Representation Toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 B.2.2 Some Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 B.2.3 Bound on the norm of x̄i ’ s . . . . . . . . . . . . . . . . . . . . . . . . . . 106 B.3 Coupling for ∇̂W , ∇̂V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 B.4 Handling the Injected Noise by PSGD . . . . . . . . . . . . . . . . . . . . . . . . 113 B.4.1 Bounding the Norm of the first Layer ’ s Output in the Worst Case . . . . . . 114 B.1 SMOOTHNESS COEFFICIENTS Recall that for a function f ∈ C3 on Rd , we say it is µ1 lipschitz , µ2 gradient lipschitz , and µ3 hessian lipschitz at point x if for every unit direction v , | ddλf ( x+ λv ) | ≤ µ1 , | d2 dλ2 f ( x+ λv ) | ≤ µ2 , and | d 2 dλ2 f ( x+ λv ) | ≤ µ3 . The aim of this section is to bound the lipschitz coefficients of the loss ℓ ( , y ) and objectiveL ( W ′ , V ′ ) in a bounded domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 . The following is our main Theorem in this regard : Theorem 9 For given values C1 , C2 > 0 , in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , for any label |y| ≤ B , the loss function ℓ ( . , y ) isO ( ( C1C2+B2 ) ) -lipschitz ( having enough overparameterization ) and 1 gradient-lipschitz x = f ′W ′ , V ′ . Moreover , the loss function L ( W ′ , V ′ ) is ( O ( C1C2 ) +B ) Ψ1+ 2 ( C1+C2 ) lipschitz , Ψ21+ ( O ( C1C2 ) +B ) Ψ2+4 gradient lipschitz , and 3Ψ2Ψ1+ ( O ( C1C2 ) +B ) Ψ3 hessian lipschitz , where Ψ1 , Ψ2 , Ψ3 are defined in Lemma 36 . Proof of Lemma 9 As in the proof of Lemma 36 , let ( W̃ , Ṽ ) be a unit direction , i.e . ∥W̃∥2 + ∥Ṽ ∥2 = 1 . Then , using Lemma 34 , we know that for every i ∈ [ n ] : |f ′W ′ , V ′ ( xi ) | = O ( C1C2 ) , so by 1-smoothness of the loss and B-boundedness of the labels , we get that ℓ ( . , y ) is ( O ( C1C2 ) + B ) lipshcitz at point f ′W ′ , V ′ . The gradient smoothness parameter of the square loss ℓ is bounded by 1 and its third derivative is zero . Now using these coefficients , we can easily compute the coefficients for L as well by simple differentiation : | d dλ ℓ ( f ′ W ′+λW̃ , V ′+λṼ ( xi ) , yi ) | = |ℓ̇ ( f ′ , yi ) d dλ f ′| ≤ ( O ( C1C2 ) +B ) Ψ1 . | d dλ2 ℓ ( f ′ W ′+λW̃ , V ′+λṼ ( xi ) , yi ) | = |ℓ̈ ( f ′ , yi ) ( d dλ f ′ ) 2 + ℓ̇ ( f ′ , yi ) d2 dλ2 f ′| ≤ Ψ21 + ( O ( C1C2 ) +B ) Ψ2 . | d dλ3 ℓ ( f ′ W ′+λW̃ , V ′+λṼ ( xi ) , yi ) | = | ... ℓ ( f ′ , yi ) ( d dλ f ′ ) 3 + 3ℓ̈ ( f ′ , yi ) d2 dλ2 f ′ d dλ f ′ + ℓ̇ ( f ′ , yi ) d3 dλ3 f ′| ≤ Ψ31 + 3Ψ2Ψ1 + ( O ( C1C2 ) +B ) Ψ3 . Moreover , note that d dλ ∥W ′ + λW̃∥2 = 2⟨W ′ + λW̃ , W̃ ⟩ ∣∣∣ λ=0 = 2⟨W ′ , W̃ ⟩ ≤ 2∥W ′∥ = 2C1 , d2 dλ2 ∥W ′ + λW̃∥2 = ⟨W̃ , W̃ ⟩ = 2 , d3 dλ3 ∥W ′ + λW̃∥2 = 0 , and similarly for ∥V ′ + λṼ ∥2 . Combining these results finishes the proof . Above , we used parameters Ψ1 , Ψ2 , Ψ3 , the lipschitz coefficients of f ′ in domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , which we bound in Lemma 36 below . B.1.1 COMPUTING THE LIPSCHITZ COEFFICIENTS OF f ′W ′ , V ′ In this section , we bound the lipschitz coefficients of f ′W ′ , V ′ in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 by poly ( m1 , m2 , m3 , β1 , β2 ) functions . Lemma 36 For every point ( W ′ , V ′ ) in the domain ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , we have the following bounds on the lipschitz coefficients of f ′W ′ , V ′ ( ( W̃ , Ṽ ) is a unit direction with ∥W̃∥2 + ∥Ṽ ∥2 = 1 ) : ∣∣∣ d dλ f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ ≲ m2 β22 ( β2√ m2 √ m3 ( κ1 + C1 + β1 ) ( κ2 √ m3 + C2 ) + β22√ m2 √ m3 ( κ1 + C1 + β1 ) ) + m1 β21 ( √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ( κ2 √ m3 + C2 ) + β2 √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ) : = Ψ1 , d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) √ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] : = Ψ2∣∣∣ d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 3/2√ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] : = Ψ3 ( 287 ) Proof of Lemma 36 Let ρ ( W ρ , V ρ ) : = 1 ( √ 2π ) m2m3+m1d ( β1/ √ m1 ) m1d ( β2/ √ m2 ) m2m3 exp { − ∥W ρ∥2 2β21/m1 − ∥V ρ∥2 2β22/m2 } , be the density function of the law of W ρ and V ρ which is a joint Gaussian . Then to compute the derivative and second derivative of the function in the unit direction ( W̃ , Ṽ ) , s.t . ∥W̃∥2F + ∥Ṽ ∥2F , we can write the value of the smoothed function as an integration with density ρ , change variable , and then take derivatives : d dλ f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 = d dλ EWρ , V ρfW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) = d dλ ∫ fW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) ρ ( W ρ , V ρ ) d ( W ρ , V ρ ) = d dλ ∫ fW+ , V + ( xi ) ρ ( W + − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) . But one can easily see that for fixed V ′ and Ṽ , the set of functions fW+ , V + ( xi ) ρ ( W+ − ( W ′ + λW̃ ) , V +− ( V ′ + λṼ ) ) for a small neighborhood of λ can simultaneously be upper bounded by an integrable function . Hence , the Leibnitz rule holds here because of dominated convergence theorem , and we can change the order of integration and derivation : = ∫ fW+ , V + ( xi ) d dλ ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = − ∫ fW+ , V + ( xi ) 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉 ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) 〈 ( W̃ , Ṽ ) , ( m1 β21 W ρ , m2 β22 V ρ ) 〉 ρ ( W ρ , V ρ ) d ( W ρ , V ρ ) = EWρ , V ρ ( m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉 ) fW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) ( xi ) ∣∣∣ λ=0 = EWρ , V ρ ( m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉 ) fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) . ( 288 ) Similarly we can compute the second derivative : d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 = d dλ ∫ fW+ , V + ( xi ) 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉 ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉2 −〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1W̃ , ( β22 m2 ) −1Ṽ ) 〉 ] ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( m1 β21 W ρ , m2 β22 V ρ ) 〉2 + 〈 ( W̃ , Ṽ ) , ( m1 β21 W̃ , m2 β22 Ṽ ) 〉 ] ρ ( W ρ , V ρ ) d ( W ρ , V ρ ) = EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 − ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] fW ′+λW̃+Wρ , V ′+λṼ+V ρ ( xi ) ( xi ) ∣∣∣ λ=0 = EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 − ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) . ( 289 ) Similarly for the third derivative : d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 d dλ ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉2 −〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1W̃ , ( β22 m2 ) −1Ṽ ) 〉 ] ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = ∫ fW+ , V + ( xi ) [ 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉3 − 3 〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1 ( W+ − ( W ′ + λW̃ ) ) , ( β22 m2 ) −1 ( V + − ( V ′ + λṼ ) ) ) 〉〈 ( W̃ , Ṽ ) , ( ( β21 m1 ) −1W̃ , ( β22 m2 ) −1Ṽ ) 〉 ] ρ ( W+ − ( W ′ + λW̃ ) , V + − ( V ′ + λṼ ) ) d ( W+ , V + ) = EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ] . Now for first derivative , exactly similar to the derivation in ( 284 ) , we can write∣∣∣EWρ , V ρ ( m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉 ) fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣m1 β21 〈 W̃ , W ρ 〉 + m2 β22 〈 Ṽ , V ρ 〉∣∣∣ ( κ2√m3∥x′ ( 2 ) i ∥+ ∥V ′∥F ∥x′ ( 2 ) i ∥+ EV ρ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) ≤ EWρ , V ρ ( ∣∣∣m1 β21 〈 W̃ , W ρ 〉∣∣∣+ ∣∣∣m2 β22 〈 Ṽ , V ρ 〉∣∣∣ ) ( κ2√m3∥x′ ( 2 ) i ∥+ ∥V ′∥F ∥x′ ( 2 ) i ∥+ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) = m2 β22 ( EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣EWρ∥x′ ( 2 ) i ∥ ( κ2√m3 + ∥V ′∥F ) + EWρEV ρ 1√m2 ∣∣∣〈Ṽ , V ρ〉∣∣∣∑ j |V ρj x ′ ( 2 ) i | ) + m1 β21 ( ( EWρ∥x′ ( 2 ) i ∥ ∣∣∣〈W̃ , W ρ〉∣∣∣ ) ( κ2√m3 + ∥V ′∥F ) + EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣EV ρ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) ≲ m2 β22 ( EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣EWρ∥x′ ( 2 ) i ∥ ( κ2√m3 + C2 ) + EWρEV ρ 1√m2 ∣∣∣〈Ṽ , V ρ〉∣∣∣∑ j |V ρj x ′ ( 2 ) i | ) + m1 β21 ( ( EWρ∥x′ ( 2 ) i ∥ ∣∣∣〈W̃ , W ρ〉∣∣∣ ) ( κ2√m3 + C2 ) + EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣EV ρ 1√m2 ∑ j |V ρj x ′ ( 2 ) i | ) , where the last line follows because ∥V ′∥ ≲ C2 . But notice that because ∥Ṽ ∥F ≤ 1 , ∥W̃∥F ≤ 1 , then 〈 Ṽ , V ρ 〉 and 〈 W̃ , W ρ 〉 are Gaussian variables with variances at most β22/m2 and β 2 1/m1 . Hence EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣ ≲ β2/√m2 , ( 290 ) EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣ ≲ β1/√m1 . ( 291 ) Similarly , using the same derivation as in ( 33 ) , one can also get the following a.s. bound ( over the randomness of W ρ ) : ∥x′ ( 2 ) i ∥ ≲ √ m3 ( κ1 + C1 + 1√ m1 ∑ j |W ρj xi| ) , ( 292 ) therefore EWρ∥x′ ( 2 ) i ∥ ≲ √ m3 ( κ1 + C1 + EWρ 1√ m1 ∑ j |W ρj xi| ) ( 293 ) ≲ √ m3 ( κ1 + C1 + β1 ) . ( 294 ) Moreover , for every j ∈ [ m2 ] : EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣|V ρj x′ ( 2 ) i | ≤ √ EV ρ ∣∣∣〈Ṽ , V ρ〉∣∣∣2√EV ρ |V ρj x′ ( 2 ) i |2 = β2√ m2 β2√ m2 ∥x′ ( 2 ) i ∥ = β22 m2 ∥x′ ( 2 ) i ∥ , EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣|W ρj xi| ≤ β21m1 . ( 295 ) Similarly , using Equation ( 292 ) we bound EWρ∥x′ ( 2 ) i ∥ ∣∣∣〈W̃ , W ρ〉∣∣∣ ≲ √m3 ( EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣ ( κ1 + C1 ) + EWρ 1√m1 ∑ j ∣∣∣〈W̃ , W ρ〉∣∣∣|W ρj xi| ) ≤ √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) . ( 296 ) Now applying these bounds ( 291 ) , ( 294 ) , ( 295 ) , and ( 296 ) to ( 290 ) and using the fact that EV ρ 1√ m2 ∑ j |V ρj x ′ ( 2 ) i | ≤ β2∥x ′ ( 2 ) i ∥ , we get LHS ≲m2 β22 ( β2√ m2 √ m3 ( κ1 + C1 + β1 ) ( κ2 √ m3 + C2 ) + EWρ β22√ m2 ∥x′ ( 2 ) i ∥ ) + m1 β21 ( √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ( κ2 √ m3 + C2 ) + EWρ ∣∣∣〈W̃ , W ρ〉∣∣∣β2∥x′ ( 2 ) i ∥ ) ≲m2 β22 ( β2√ m2 √ m3 ( κ1 + C1 + β1 ) ( κ2 √ m3 + C2 ) + β22√ m2 √ m3 ( κ1 + C1 + β1 ) ) + m1 β21 ( √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ( κ2 √ m3 + C2 ) + β2 √ m3 ( β1√ m1 ( κ1 + C1 ) + β21√ m1 ) ) . To make it easier for handling the second and third derivatives , we first bound the expectations of f2W ′+Wρ , V ′+V ρ ( xi ) ( xi ) which enables us to use Cauchy-schwartz . Again using similar derivation as in ( 284 ) and Equation ( 292 ) : EWρ , V ρf2W ′+Wρ , V ′+V ρ ( xi ) ( xi ) ≤ EWρ , V ρ ( κ2 √ m3∥x ′ ( 2 ) i ∥+ C2∥x ′ ( 2 ) i ∥+ 1√ m2 ∑ j |V ρj x ′ ( 2 ) i | ) 2 ≲ ( κ2 √ m3 + C2 ) 2EWρ∥x′ ( 2 ) i ∥ 2 + EWρEV ρ 1 m2 ( ∑ j |V ρj x ′ ( 2 ) i | ) 2 ≲ ( κ2 √ m3 + C2 ) 2EWρ∥x′ ( 2 ) i ∥ 2 + EWρ 1 m2 ∑ j EV ρj |V ρ j x ′ ( 2 ) i | 2 + EWρ 1 m2 ∑ j1 ̸=j2 EV ρj1 |V ρj1x ′ ( 2 ) i |EV ρj2 |V ρ j2 x ′ ( 2 ) i | ≲ ( κ2 √ m3 + C2 ) 2EWρ∥x′ ( 2 ) i ∥ 2 + EWρ β22 m2 ∥x′ ( 2 ) i ∥ 2 + EWρβ 2 2∥x ′ ( 2 ) i ∥ 2m ( m− 1 ) m2 ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) EWρ∥x′ ( 2 ) i ∥ 2 ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) EWρm3 ( κ1 + C1 + 1√ m1 ∑ j |W ρj xi| ) 2 ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) m3 [ ( κ1 + C1 ) 2 + 1 m1 ∑ j EWρ |W ρj xi| 2 + 1 m1 ∑ j1 ̸=j2 EWρj1 |W ρ j1 xi|EWρj1 |W ρ j2 xi| ] ≲ ( ( κ2 √ m3 + C2 ) 2 + β22 ) m3 [ ( κ1 + C1 ) 2 + β21 m1 + β21 m ( m− 1 ) m2 ] = m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] . ( 297 ) Now for the second derivative , we can proceed by applying Cauchy-Swartz : d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ≤ EWρ , V ρ ∣∣∣ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 − ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ EWρ , V ρ ∣∣∣m21 β41 ⟨W̃ , W ρ⟩2 − m1 β21 ∥W̃∥2 + m 2 2 β42 ⟨Ṽ , V ρ⟩2 − m2 β22 ∥Ṽ ∥2 + 2m1m2 β21β 2 2 ⟨W̃ , W ρ⟩⟨Ṽ , V ρ⟩ ∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣ ≤ √ EWρ , V ρ ∣∣∣m21 β41 ⟨W̃ , W ρ⟩2 − m1 β21 ∥W̃∥2 + m 2 2 β42 ⟨Ṽ , V ρ⟩2 − m2 β22 ∥Ṽ ∥2 + 2m1m2 β21β 2 2 ⟨W̃ , W ρ⟩⟨Ṽ , V ρ⟩ ∣∣∣2√ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 . Now note that the cross terms have expectation zero , so we get = √ EWρ , V ρ ( m21 β41 ⟨W̃ , W ρ⟩2 − m1 β21 ∥W̃∥2 ) 2 + ( m22 β42 ⟨Ṽ , V ρ⟩2 − m2 β22 ∥Ṽ ∥2 ) 2 + 4m21m 2 2 β41β 4 2 EWρ , V ρ⟨W̃ , W ρ⟩2⟨Ṽ , V ρ⟩2√ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 ≲ √ m21 β41 ∥W̃∥4 + m 2 2 β42 ∥Ṽ ∥4 + 4m1m2 β21β 2 2 ∥W̃∥2∥Ṽ ∥2 √ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) √ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ( xi ) ∣∣∣2 . Now applying Cauchy-shwartz and Equation ( 297 ) to above and combining it with Equations ( 289 ) : d2 dλ2 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) √ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] . Similarly for the third derivative : ∣∣∣ d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ = EWρ , V ρ ∣∣∣ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ∣∣∣∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣ ≤ √ EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] 2 √ EWρ , V ρ ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣2 . ( 298 ) But note that EWρ , V ρ [ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 3 − 3 ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) ] 2 ≤ 2EWρ , V ρ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 6 + 18 ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 2 EWρ , V ρ ( m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ ) 2 Now note that m1 β21 ⟨W̃ , W ρ⟩+ m2 β22 ⟨Ṽ , V ρ⟩ is a normal variable with variance m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 . Therefore , by the bound on the moments of normal random variables : LHS ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 3 . ( 299 ) Plugging this into Equation ( 298 ) and also using Equation ( 297 ) : ∣∣∣ d3 dλ3 f ′ W ′+λW̃ , V ′+λṼ ( xi ) ∣∣∣ λ=0 ∣∣∣ ≲ ( m1 β21 ∥W̃∥2 + m2 β22 ∥Ṽ ∥2 ) 3/2√ m3 ( ( κ2 √ m3 + C2 ) 2 + β22 ) [ ( κ1 + C1 ) 2 + β21 ] . B.2 REPRESENTATION LEMMAS In this section , we prove lemmas mostly related to the representation power of the network , which we mainly use in Appendix A.12 . B.2.1 REPRESENTATION TOOLBOX Lemma 37 Recall the definitions of W+k , and x̄i , k from Equations ( 66 ) , and ( 62 ) . For all k , i ∈ [ n ] : |x̄i , k − trace ( W+k Z i k ) | ≤ √ n/ ( m1λ0 ) ∥Vk∥H∞ . Proof of Lemma 37 trace ( W+k Z i k ) = trace ( Z i k n∑ j=1 Vk , jZjk ) = n∑ i′=1 Vk , i′⟨Zik , Zi ′ k ⟩ . ( 300 ) But note ⟨Zik , Zi ′ k ⟩ = 1/m1 m1∑ j=1 W ′k , j 2 1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } ⟨xi′ , xi⟩ = ⟨xi′ , xi⟩ m1 m1∑ j=1 1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } . Now note that ⟨xi′ , xi⟩ ≤ 1 , and 1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } is a Bernoulli with E1 { W ( 0 ) j Txi } 1 { W ( 0 ) j Txi′ } = 1/4 + arcsin ( ⟨x , y⟩ ) /2π . Therefore , by Hoeffding inequality we get |⟨Zik , Zi ′ k ⟩ −H∞i , i′ | = O ( 1/ √ m1 ) . Hence , because obviously ∥H∞∥2 ≤ 1 , we get trace ( W+k Z i k ) = n∑ i′=1 Vk , i′H∞i , i′ +O ( 1/ √ m1 ) n∑ i′=1 Vk , i′ = x̄i , k +O ( 1/ √ m1 ) n∑ i′=1 Vk , i′ , which implies |trace ( W+k Z i k ) − x̄i , k| ≤ O ( 1/ √ m1 ) √ n∥Vk∥2 ≲ O ( √ n/ ( √ m1 √ λ0 ) ) ∥Vk∥H∞ ≤ √ n/ ( m1λ0 ) ∥Vk∥H∞ . Lemma 38 ( Bounding the rows norm ) For every 1 ≤ j ≤ m1 , we have ∥W+j ∥ ≤ √ nm3/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ . Furthermore , for every k ∈ [ m3 ] , we have ∥W k+j ∥ ≤ √ n/ √ λ0m1∥Vk∥H∞ . ( 301 ) For the ease of notation , because here we want to work with row sub indices of the matrix W+k , we refer to it by W k+ . Proof of Lemma 38 For a fixed 1 ≤ j ≤ m1 we have with high probability over the randomness of the sign matrix W s : ∥W+j ∥ = ∥ m3∑ k=1 W k+j ∥ = ∥ m3∑ k=1 W sk , j1/ √ m1 n∑ i=1 Vk , ixi1 { W ( 0 ) j xi ≥ 0 } ∥ ≤ √ m3/ √ m1 √√√√m3∑ k=1 ∥ n∑ i=1 Vk , ixi1 { W ( 0 ) j xi ≥ 0 } ∥2 ≤ √ m3/ √ m1 √∑ k ( ∑ i |Vk , i| ) 2 ≤ √ m3/ √ m1 √ n ∑ k ∥Vk∥2 ≤ √ m3n/ ( √ m1λ0 ) √∑ k ∥Vk∥2H∞ . Furthermore , for every k ∈ [ m3 ] , we have ∥W k+j ∥ ≤ 1/ √ m1 ∑ i |Vk , i| ≤ ( √ n/ √ m1 ) ∥Vk∥2 ≤ ( √ n/ √ λ0m1 ) ∥Vk∥H∞ . Lemma 39 With high probability over the initialization , we have ∥W k+∥2F ≤ ( 1±O ( n/ ( λ0 √ m1 ) ) ) ∥Vk∥2H∞ . Proof of Lemma 39 Recall from the definition of W k+ in Equation ( 66 ) : ∥W k+∥2F = ∥ n∑ i=1 Vk , iZik∥2F = n∑ i=1 n∑ i′=1 Vk , iVk , i′⟨Zik , Zi ′ k ⟩ = n∑ i=1 n∑ i′=1 Vk , iVk , i′ ( H∞i , i′ ±O ( 1/ √ m1 ) ) ≲ VTk H∞Vk ±O ( ( ∥Vk∥1 ) 2/ √ m1 ) = ∥Vk∥2H∞ ± ∥Vk∥2H∞O ( n/ ( λ0 √ m1 ) ) = ( 1±O ( n/ ( λ0 √ m1 ) ) ) ∥Vk∥2H∞ B.2.2 SOME LINEAR ALGEBRA Lemma 40 For n ≤ s , let r1 , . . . , rn be s-dimensional vectors that are approximately normalized and orthogonal to one another , i.e . given some δ > 0 , for every 1 ≤ i ̸= j ≤ n : −δ ≤ ⟨ri , rj⟩ ≤ δ , ∥ri∥2 ≤ 1 + δ . Then , for any vector v we have n∑ i=1 ⟨v , ri⟩2 ≤ ( 1 + δ + n ( n− 1 ) δ ( 1 + δ ) 2 ) ∥v∥2 . Proof of Lemma 40 Define v1 = n∑ i=1 ⟨v , ri⟩ri , v2 = v − v1 . First , note that n∑ i=1 ⟨v , ri⟩2 ≤ ( 1 + δ ) n∑ i=1 ⟨v , ri⟩2∥ri∥2 = ( 1 + δ ) ∥ ∑ i ⟨v , ri⟩ri∥2 − 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ = ( 1 + δ ) ∥v1∥2 − 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ . Next , we write ⟨v1 , v − v1⟩ = ⟨v , n∑ i=1 ⟨v , ri⟩ri⟩ − ⟨ n∑ i=1 ⟨v , ri⟩ri , n∑ i=1 ⟨v , ri⟩ri⟩ = ∑ i ⟨v , ri⟩2 − ∑ i ⟨v , ri⟩2 − 2 ∑ i ̸=j ⟨v , ri⟩⟨v , rj⟩⟨ri , rj⟩ = −2 ∑ i ̸=j ⟨v , ri⟩⟨v , rj⟩⟨ri , rj⟩ . Therefore n∑ i=1 ⟨v , ri⟩2 ≤ ( 1 + δ ) ( ∥v1∥2 + ∥v − v1∥2 ) − 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ . ≤ ( 1 + δ ) ( ∥v1∥2 + ∥v − v1∥2 + 2⟨v1 , v − v1⟩ ) + 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ⟨v , ri⟩⟨v , ri⟩⟨ri , rj⟩ ≤ ( 1 + δ ) ∥v∥2 + 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ∥v∥2∥ri∥∥rj∥δ ≤ ( 1 + δ ) ∥v∥2 + 2 ( 1 + δ ) ∑ 1≤i ̸=j≤n ∥v∥2 ( 1 + δ ) δ = ( 1 + δ ) ∥v∥2 + n ( n− 1 ) δ ( 1 + δ ) 2∥v∥2 = ( 1 + δ + n ( n− 1 ) δ ( 1 + δ ) 2 ) ∥v∥2 , which completes the proof . In the following lemma , we state a trivial bound on the norm of x̄i based on ζ1 . B.2.3 BOUND ON THE NORM OF x̄i ’ S Lemma 41 For every i ∈ [ n ] , we have ∥x̄i∥ ≤ √∑ k ∥Vk∥2H∞ = √ ζ1 . Proof of Lemma 41 By definition : x̄Ti = ( H∞i , Vk ) m3 k=1 . Now consider the Cholskey factorization H∞ = KKT . Because of the assumption ∥xi∥ = 1 , we know that the diagonal of H∞ is all 1/2 . Hence , for the ith row of K we have ∥Ki∥ = 1/2 . Now by Cauchy-Swartz , we have x2i1 = ⟨ ∑ i Vk , iKi , Ki1⟩2 ≤ ∥ ∑ i Vk , iKi∥2∥Ki1∥2 = 1/2∥Vk∥2H∞ . Summing over i and noting Equation ( 61 ) completes the proof . Lemma 42 In the context of Lemma 10 , for ζ2 ≤ 2nB2 , one can substitute f∗ by f∗1 such that Rn ( f̄ ∗ ) ≤ 2Rn ( f∗ ) + B2 n , f̄∗TA−1f̄∗ ≤ f∗TA−1f̄∗ , and furthermore , f̄∗ is in the subspace of eigenvectors of A with eigenvaue larger than Ω ( 1n2 ) . Proof of Lemma 42 For an arbitrary i ∈ [ n ] , define δ = |f∗i − f̄∗i | , and suppose the slope of ℓ ( . , yi ) at point f∗i is equal to c. Then , using the convexity , the fact that ℓ ( yi , yi ) = 0 , and the 1-smoothness of ℓ ( . , yi ) , it is not hard to see the following poincare inequality between the value and derivative of ℓ ( . , yi ) at point f∗i : c ≤ √ 2ℓ ( f∗i , yi ) : = 2ℓ . ( 302 ) where from now on , for brevity , we refer to ℓ ( f∗i , yi ) by ℓ . Also , from the definition of δ and again using 1 smoothness property , it is easy to see that ℓ ( f̄∗i , yi ) ≤ ( c+ δ ) δ + ℓ ( f∗i , yi ) = ( c+ δ ) δ + ℓ , ( 303 ) Plugging Equation ( 302 ) into ( 303 ) and using AM-GM inequality : ℓ ( f̄∗i , yi ) ≤ δ2 + cδ + ℓ ≤ δ2 + √ 2ℓδ + ℓ ≤ δ2 + ℓ+ δ2/2 + ℓ ≤ 2ℓ+ 3δ2/2 . Summing above for i ∈ [ n ] , we obtain Rn ( f̄ ∗ ) ≤ 2Rn ( f∗ ) + 3∥f∗ − f̄∗∥22/2 . ( 304 ) Now we write an eigendecomposition for A as A = ∑n i=1 λiuiu T i for orthonormal basis { ui } , and let f∗ = ∑ i γiui be the representation of f ∗ in this basis . Then , from our assumption , for arbitrary ω > 0 ∑ i γ2i λ −1 i = f ∗TA−1f∗ ≤ 4nB2 , which implies ω−1 ∑ i : λi≤ω γ2i ≤ 4nB2 , or equivalently ∑ i : λi≤ω γ2i ≤ 4nB2ω , ( 305 ) where notice that ∑ i : λi≤ω γ 2 i is the squared norm of the projection of f ∗ onto the directions whose eigenvalue is at most ω . Now taking ω = 112n2 and defining f̄ ∗ by keeping only the directions for which λi > ω completes the proof . B.3 COUPLING FOR ∇̂W , ∇̂V In general , because the gaussian smoothing matrices ( W ρ , V ρ ) can become unbounded , the gradient estimates ( ∇̂W , ∇̂V ) = ∇W , V ℓ ( fW ′+Wρ , V ′+V ρ ( xi ) , yi ) also become unbounded . However , in analyzing the stochastic behavior of SGD and showing that it can escape saddle points , it is convenient to assume the gradient ’ s noise vector is almost surely bounded . The goal of this section is to introduce a coupling between ( W ρ , V ρ ) and another random variable that is a.s. bounded polynomially in other parameters . As that the coupled random variables take different values is exponentially small while the number of iterations in our algorithm is only polynomially large , without any concern we instead work with this new random varaible , and with an overload of notation we also denote it by ( W ρ , V ρ ) . Lemma 43 For an arbitrary parameter χ > > 1 , On any pair for ( W ′ , V ′ ) with ∥W ′∥ ≤ C1 , ∥V ′∥ ≤ C2 , there exist a mean zero random vector Λ̄ with respect to the randomness of the uniformly picked data point ( xi , yi ) and the smoothing matrices W ρ,1 , W ρ,2 , V ρ,1 , and V ρ,2 which define ∇̂W ′ , V ′ ( meaning it is a function of those variables ) , such that with probability at least 1− 2 exp { − ( χ2 − 1 ) dm1/4 } − 2 exp { − ( χ2 − 1 ) m3m2/4 } : = 1− δ1 , we have ∇̂W ′ , V ′ = ∇W ′ , V ′L ( W ′ , V ′ ) + Λ̄ , ( 306 ) and finally Λ̄ is a.s. polynomially bounded , i.e . almost surely we have ∥Λ̄∥ ≤ poly ( m1 , m2 , m3 , C1 , C2 , B , χ ) . Proof of Lemma 43 Remember that x′i was the output of the first layer ( by considering the smoothing matrix W ρ ) . Now with high probability over the initialization , ∥∇W ′fW ′+Wρ , V ′+V ρ ( xi ) ∥F = ∥∇x′ifW ′+Wρ , V ′+V ρ ( xi ) T D ( x ′ i ) dW ′ ∥ ∥∇x′ifW ′+Wρ , V ′+V ρ ( xi ) ∥∥ D ( x′i ) dW ′ ∥ ∥ 1√ m2 aTDV ′+V ρ ( V ( 0 ) + V ′ + V ρ ) ∥∥ 1√ m1 m3∑ k=1 diag ( W sk ) DW ′+Wρ , xix T i ∥F ≤ ( ∥V ( 0 ) ∥F + ∥V ′∥F + ∥V ρ∥F ) ( 1√ m1 ∑ k ∥diag ( W sk ) DW ′+Wρ , xixTi ∥F ) ≤ ( κ2 √ m2m3 + C2 + ∥V ρ∥F ) ( 1√ m1 ∑ k ∥diag ( W sk ) DW ′+Wρ , xixTi ∥F ) ≤ ( κ2 √ m2m3 + C2 + ∥V ρ∥F ) . ( 307 ) On the other hand , using the final bound in Lemma 33 : ∥∇V ′fW ′+Wρ , V ′+V ρ ( xi ) ∥F = ∥ 1√ m2 diag ( a ) DV ′+V ρx′i T ∥F ≤ ∥x′i∥ ≤ κ1 √ m3 + C1 + √ m3 ∑ j 1√ m1 |W ρj xi| ≤ κ1 √ m3 + C1 + √ m3∥W ρ∥F . ( 308 ) Denoting ℓ̇ ( fW ′+Wρ,1 , V ′+V ρ,1 , yi ) ∇W ′ , V ′ℓ ( fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) , yi ) by ∇̃W ′ , V ′ , then combining Equations ( 307 ) and ( 308 ) and using the 1 gradient lipschitzness property of the square loss , : ∥∇̃W ′ , V ′∥F = |d ( ℓ ( f , yi ) ) df | √ ∥∇W ′fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) ∥2F + ∥∇V ′fW ′+Wρ,2 , V ′+V ρ,2 ( xi ) ∥2F ≤ ( |fW ′+Wρ1 , V ′+V ρ1 ( xi ) |+ |B| ) ( κ1 √ m3 + C1 + √ m3∥W ρ,2∥F + κ2 √ m2m3 + C2 + ∥V ρ,2∥F ) . ( 309 ) Finally , applying Cauchy-Swartz to the second a.s. bound in Lemma 34 we have : ∣∣∣fW ′+Wρ , V ′+V ρ ( xi ) ∣∣∣ ≤ ( κ2 √ m3 + ∥V ρ∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ∥ ) + C2 ( C1 + √ m3∥W ρ∥ ) . Combining this with ( 309 ) : ∥∇̃W ′ , V ′∥F ≤ [ B + ( κ2 √ m3 + ∥V ρ,1∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ,1∥ ) + C2 ( C1 + √ m3∥W ρ,1∥ ) ] × ( κ1 √ m3 + C1 + √ m3∥W ρ,2∥F + κ2 √ m2m3 + C2 + ∥V ρ,2∥F ) . Therefore , using the lipschitz bound in Theorem 9 : ∥∇̃W ′ , V ′ −∇W ′ , V ′E ( xi , yi ) ∼Zℓ ( f ′ W ′ , V ′ ( xi ) , yi ) ∥F ≤ ∥∇̂W ′ , V ′∥F + ∥E ( xi , yi ) ∼Z∇W ′ , V ′ℓ ( fW ′+Wρ , V ′+V ρ ( xi ) , yi ) ∥F ≤ [ B + ( κ2 √ m3 + ∥V ρ,1∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ,1∥ ) + C2 ( C1 + √ m3∥W ρ,1∥ ) ] × ( κ1 √ m3 + C1 + √ m3∥W ρ,2∥F + κ2 √ m2m3 + C2 + ∥V ρ,2∥F ) + ( O ( C1C2 ) +B ) Ψ1 . ( 310 ) Now we define the following events Ξ1 : = { ∥W ρ1∥F ≥ χ √ dβ1 ∨ ∥W ρ2∥F ≥ χ √ dβ1 } , Ξ2 : = { ∥V ρ1∥F ≥ χ √ m3β2 ∨ ∥V ρ2∥F ≥ χ √ m3β2 } , where recall we assume χ > > 1 . Then , as we know the variable ∥W ρ∥2F has mean dβ21 and is subexponential with parameters ( dβ41/m1 , β 2 1/m1 ) . Hence , by a union bound and Bernstein ( Note that W ρ,1 , W ρ,2 are independent ) : P ( Ξ1 ) ≤ 2P ( ∥W ρ∥F ≥ χβ1 √ d ) = 2P ( ∥W ρ∥2F ≥ χ2β21d ) ≤ 4max ( exp { − ( χ2 − 1 ) 2β41d2/ ( 4dβ41/m1 ) } , exp { − ( χ2 − 1 ) β21d/ ( 4β21/m1 ) } ) = 4max ( exp { − ( χ2 − 1 ) 2dm1/4 } , 2 exp { − ( χ2 − 1 ) dm1/4 } ) ≤ 2 exp { − ( χ2 − 1 ) dm1/4 } . Similarly for ∥V ρ∥F : P ( Ξ2 ) = P ( ∥V ρ∥F ≥ χβ2 √ m3 ) ≤ 4 exp { − ( χ2 − 1 ) m3m2/4 } . Moreover , because of the subexponential tails of ∥W ρ∥2F and ∥V ρ∥2F , for each of W ρ,1 or W ρ2 , V ρ1 , or V ρ2 : E ( ∥W ρ∥F | Ξ1 ) ≲ χ √ dβ1 , E ( ∥W ρ∥2F | Ξ1 ) ≲ χ2dβ21 . E ( ∥V ρ∥F | Ξ2 ) ≲ χ √ m3β2 , E ( ∥V ρ∥2F | Ξ2 ) ≲ χ2m3β22 . Now Defining Ξ = Ξ1 ∪ Ξ2 and combining the above equations : E ( ∥W ρ∥1 { Ξ } ) ≤ E∥W ρ∥ ( 1 { Ξ1 } + 1 { Ξ2 } ) = E ( ∥W ρ∥| Ξ1 ) P ( Ξ1 ) + E ( ∥W ρ∥ ) P ( Ξ2 ) ≲ χ √ dβ12 exp { − ( χ2 − 1 ) dm1/4 } + √ dβ12 exp { − ( χ2 − 1 ) m3m2/4 } = 2 √ dβ1 ( χ exp { − ( χ2 − 1 ) dm1/4 } + exp { − ( χ2 − 1 ) m3m2/4 } ) , and E∥W ρ∥21 { Ξ1 } = E ( ∥W ρ∥2| Ξ1 ) P ( Ξ1 ) ≤ 2dβ21χ2 exp { − ( χ2 − 1 ) dm1/4 } . Similarly E ( ∥V ρ∥1 { Ξ } ) ≤ 2 √ m3β2 ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) , and E ( ∥V ρ∥21 { Ξ2 } ) ≤ 2m3β22χ2 exp { − ( χ2 − 1 ) m3m2/4 } . Applying these equations to ( 310 ) with Cauchy-Schwartz to get the upper bounds E1 { Ξ1 } ∥W ρ,1∥∥W ρ,2∥ ≤ E1 { Ξ1 } ∥W ρ∥2 and ( EWρ∥W ρ∥ ) 2 ≤ EWρ∥W ρ∥2 ( for terms with only one W ρ , i or V ρ , i , we simply write them as W ρ and V ρ ) : EWρ , V ρ∥∇̃W ′ , V ′ −∇W ′ , V ′E ( xi , yi ) ∼Zℓ ( f ′ W ′ , V ′ ( xi ) , yi ) ∥F1 { Ξ } ≤ EWρ , V ρ , ( xi , yi ) [ B + ( κ2 √ m3 + ∥V ρ,1∥F ) ( √ m3κ1 + C1 + √ m3∥W ρ∥ ) + C2 ( C1 + √ m3∥W ρ∥ ) ] × ( κ1 √ m3 + C1 + √ m3∥W ρ∥F + κ2 √ m2m3 + C2 + ∥V ρ∥F ) + α ( O ( C1C2 ) +B ) Ψ1 = [ B + ( κ2 √ m3 ) ( √ m3κ1 + C1 ) + C1C2 ] × ( κ1 √ m3 + C1 + κ2 √ m2m3 + C2 + √ m3EWρ1 { Ξ } ∥W ρ∥F + EV ρ1 { Ξ } ∥V ρ∥F ) + ( B + ( √ m3κ1 + C1 ) √ m3 + ( κ2m3 + C2 √ m3 ) + √ m3 ( κ1 √ m3 + C1 + κ2 √ m2m3 + C2 ) ) × ( EWρ1 { Ξ1 } ∥W ρ∥FEV ρ∥V ρ∥F + EWρ∥W ρ∥FEV ρ1 { Ξ2 } ∥V ρ∥F ) + ( B + √ m3κ1 + C1 ) ( EV ρ1 { Ξ2 } ∥V ρ∥2 + P ( Ξ1 ) E∥V ρ∥2 ) + C2m3 ( EWρ1 { Ξ1 } ∥W ρ∥2 + P ( Ξ2 ) E∥W ρ∥2 ) +m3 ( EWρ1 { Ξ1 } ∥W ρ∥2EV ρ∥V ρ∥+ EWρ∥W ρ∥2EV ρ1 { Ξ2 } ∥V ρ∥ ) + √ m3 ( EV ρ1 { Ξ2 } ∥V ρ∥2EWρ∥W ρ∥+ EV ρ∥V ρ∥2EWρ1 { Ξ1 } ∥W ρ∥ ) + ( O ( C1C2 ) +B ) Ψ1P ( Ξ ) ≤ ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) poly ( m1 , m2 , m3 ) = negligible . ( 311 ) But note that ∇̂W ′ , V ′ = ∇̃W ′ , V ′ +∇W ′ , V ′ ( ψ1∥W ′∥2 + ψ2∥V ′∥2 ) , ∇W ′ , V ′L ( W ′ , V ′ ) = ∇W ′ , V ′E ( xi , yi ) ∼Zℓ ( f ′ W ′ , V ′ ( xi ) , yi ) +∇W ′ , V ′ ( ψ1∥W ′∥2 + ψ2∥V ′∥2 ) . Applying this to Equation ( 311 ) , we get that if we define Λ : = ∇̂W ′ , V ′ −∇W ′ , V ′L ( W ′ , V ′ ) , then E∥Λ1 { Ξ } ∥ ≤ ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) poly ( m1 , m2 , m3 ) . On the other hand , note that using again Equation ( 310 ) , we have the following a.s. bound : ∥Λ1 { Ξc } ∥ = poly ( m1 , m2 , m3 , C1 , C2 , χ ) . Defining Λ1 = Λ1 { Ξc } , Λ2 = 1 { Ξ } E ( Λ|Ξ ) , Λ̄ = Λ1 + Λ2 , we get that with probability at least 1− P ( Ξ ) : ∇̂W ′ , V ′ = ∇W ′ , V ′L ( W ′ , V ′ ) + Λ̄ and also note that EΛ̄ = EΛ = 0 . Finally by the a.s. bound for Λ1 , we have a.s. : ∥Λ̄∥ ≤ EWρ , V ρ , ( xi , yi ) ∥Λ1 { Ξ } ∥+ ∥Λ1 { Ξ c } ∥ ≤ ( exp { − ( χ2 − 1 ) dm1/4 } + χ exp { − ( χ2 − 1 ) m3m2/4 } ) poly ( m1 , m2 , m3 ) + poly ( m1 , m2 , m3 ) = poly ( m1 , m2 , m3 , C1 , C2 , B , χ ) , which completes the proof . Corollary 9.1 It is easy to check that running PSGD with unbiased gradient estimate ∇̂W ′ , V ′ is equivalent to running SGD after our change of coordinates , with unbiased gradient estimate ∇̂w′ , v′ : = Ῡ∇W ′ , V ′ , where Ῡ is the matrix for our change of coordinate , which is equal to Υ defined in Appendix A.10 for the coordinates in V ′ and simply identity for the coordinates in W ′ . Therefore , projecting both sides in Equation ( 312 ) of Lemma 43 onto Φ⊥ by multiplying Υ implies that with high probability for all iterations of the algorithm ∇̂w′ , v′ = Ῡ∇W ′ , V ′LΠ ( W ′ , V ′ ) + ῩΛ̄ = ∇w′ , v′LΠ ( w′ , v′ ) + ῩΛ̄ , ( 312 ) where £̄ : = ῩΛ̄ ( using the properties of Λ̄ in Lemma 43 ) is a mean zero noise vector with almost surely bounded norm , i.e . ∥£̄∥ ≤ Q′ for some Q′ = poly ( m1 , m2 , m3 , C1 , C2 ) . ( we dropped the χ parameters by considering constant high probability argument ) . Finally , note that injecting noise ( Ξ1/∥Ξ1∥ , Ξ2/ ( √ m1∥Ξ2∥ ) ) by PSGD results in adding an extra zero mean noise ( Ξ̃1 , Ξ̃2 ) : = ( ῩΞ1/∥Ξ1∥ , ῩΞ2/ ( √ m1∥Ξ2∥ ) ) to the gradient ∇w′ , v′LΠ ( w′ , v′ ) . Therefore , overall running SGD on LΠ ( which is equivalent to PSGD on L ) observe an unbiased noise vector defined as £ : = £̄ + ( Ξ̃1 , Ξ̃2 ) . Now it is easy to check that the moment matrix of Ξ̃1 and Ξ̃2 are σ′1 2I and σ′2 2I for σ′1 2 : = 1 m21d , ( 313 ) σ′2 2 : = m2 ( m3 − n ) mm2 3 , ( 314 ) which implies the moment matrix of £ is upper bounded by σ22I : = ( Q ′/ ( m2m3 +m1d ) + max { σ′12 , σ′22 } ) I , and lower bounded by σ22I : = min { σ′12 , σ′22 } I , i.e . σ21IE££T ≤ σ22I . ( Note that we look at the new coordinates ( w′ , v′ ) as a vectors , so the term E££T makes sense . ) Moreover , ∥Ξ̃1∥ = 1/ √ m1 , ∥Ξ̃2∥ ≤ 1 almost surely , which implies the following almost surely bound for £ : ∥£∥ ≤ Q : = Q′ + 1 + 1/ √ m1 . Lemma 44 Let g ( x ) be a second order differentiable function over RN such that at point x , there exist a random direction y and deterministic direction z and fixed positive real r with : Ey = 0 , Eyg ( x+ ηz + √ ηy ) ≤ g ( x ) − ηr . Then , for the gradient and Hessian at point x , we have either ∥∇g ( x ) ∥ ≥ r 4∥z∥ , or λmin ( ∇2g ( x ) ) ≤ − r 2∥y∥2 . Proof of Lemma 44 We write the second order tailor approximation of g around x : g ( x+ w ) = g ( x ) +∇g ( x ) Tw + 1 2 wT∇2g ( x ) w + o ( ∥w∥2 ) . Now substituting w with ηz + √ ηy and taking expectation with respect to y , as we send η → 0 and using the fact that Ey = 0 : Eyg ( x+ ηz + √ ηy ) = g ( x ) + Ey∇g ( x ) T ( ηz + √ ηy ) + 1 2 ( ηz + √ ηy ) T∇2g ( x ) ( ηz +√ηy ) + o ( ∥ηz +√ηy∥2 ) = Eyg ( x ) + η∇g ( x ) T z + 1 2 η2zT∇2g ( x ) z + η 1 2 yT∇2g ( x ) y + o ( η∥y∥2 ) = Eyg ( x ) + η∇g ( x ) T z + η 1 2 yT∇2g ( x ) y + o ( η ) . Combining the assumption with the above Equation , we get that for small enough η , we have η∇g ( x ) T z + η 1 2 EyyT∇2g ( x ) y ≤ −ηr/2 , i.e . ∇g ( x ) T z + 1 2 EyyT∇2g ( x ) y ≤ −r/2 , which means we should either have ∇g ( x ) T z ≤ −r/4 , which implies ∥∇g ( x ) ∥ ≥ r 4∥z∥ , or EyyT∇2g ( x ) y ≤ −r/2 , which implies λmin ( ∇2g ( x ) ) ≤ − r 2maxỹ∈support ( y ) ∥ỹ∥2 . B.4 HANDLING THE INJECTED NOISE BY PSGD In this section , we prove that having SGD injecting noise into our gradient estimates mostly does not change the sign pattern of the first layer , namely among the set of rows in P defined in Lemma 1 . Lemma 45 Having enough overparameterization , with high probability , at every iteration of the PSGD for W ′ ( 2 ) defined in Lemma 1 , we have for every j ∈ [ m1 ] : ∥W ′ ( 2 ) j ∥ ≤ c2/ ( 4 √ m1 ) . Proof of Lemma 45 Let Φ′ be the subspace of the first layer weight matrices which is zero in rows j ∈ P ( P is defined in Lemma 1 ) , while in other rows it is the span of Zki ’ s , i.e . using our notation Z̃ i k introduced in the proof of Lemma 1 , we can write Φ′ is span ( Zki ) i , k . Recall from Lemma 1 that we decompose the first layer weight W ′ as W ′ ( 1 ) + W ′ ( 2 ) , namely the parts in the subspace Φ′ and subspace Φ′⊥ respectively . Moreover , let Ξ1/ ( √ m1∥Ξ1∥ ) = Ξ ( 1 ) +Ξ ( 2 ) be the decomposition of the injected noise at some iteration of PSGD into subspaces Φ′ and Φ′⊥ respectively . Now recall that the current W ′ is the value of the previous iteration moved by the gradient plus the injected noise : W ′ =W ′ − η ( ∇̂W ′ + Ξ ( 1 ) + Ξ ( 2 ) ) =W ′ − η ( ∇̃W ′ + 2ψ1W ′ − , ( 1 ) + 2ψ1W ′− , ( 2 ) + Ξ ( 1 ) + Ξ ( 2 ) ) , whereW ′ is the weight of the previous iteration andW ′− , ( 1 ) , W ′− , ( 2 ) are again its decomposition to Φ′ and Φ′⊥ , where ∇̃W ′ , V ′ is defined in Lemma 43 . Applying Lemma 24 for the previous iteration of the algorithm , we get ∇̃W ′ ∈ Φ′ since the bad events E defined in Lemma 33 occurs only with probability exponentially small ( hence union bound across all the iterations rules it out ) . Hence , the decomposition for the current iteration becomes W ′ ( 1 ) =W ′ − , ( 1 ) − η ( ∇̂W ′ + 2ψ1W ′ − , ( 1 ) + Ξ ( 1 ) ) , ( 315 ) W ′ ( 2 ) = ( 1− 2ηψ1 ) W ′ − , ( 2 ) + ηΞ ( 2 ) . ( 316 ) We handle the W ′ ( 1 ) part in Lemma 1 and prove that as long as ∥W ′ ( 1 ) ∥2 ≤ ∥W ′∥2 remains bounded by C21 , then the sign pattern of the first layer , when only considering the W ′ ( 1 ) part , is specified by the initialization except within set P ; here we handle the W ′ ( 2 ) part as well . Note that for every row j ∈ [ m1 ] , the variable ∥ ( Ξ1/ ( √ m1∥Ξ1∥ ) ) j ∥2 is ( O ( 1/ ( m41d ) ) , O ( 1/ ( m 2 1d ) ) ) -subexponential with mean 1/m1 . Therefore , with probability that is exponentially small in m1 , ∥ ( Ξ1/ ( √ m1∥Ξ1∥ ) ) j ∥ is bounded by O ( 1/m1 ) . It is not hard to see the same argument holds for the projection of Ξ1/ ( √ m1∥Ξ1∥ ) onto Φ⊥ , i.e . Ξ ( 2 ) . Applying a union bound for all iterations , again using the fact that we run PSGD for poly iterations while the chance of error is exponentially small in m1 , we can then argue that with high probability over the noise of gradients , at every iteration and for every j ∈ [ m1 ] : ∥Ξ ( 2 ) j ∥ = Õ ( 1/m1 ) . ( 317 ) But applying trinagle inequality to Equation ( 316 ) and writing it in a telescope form , particularly for the jth row , and further using the assumption in 317 , we get that ∥W ′ ( 2 ) j ∥ grows at most to O ( 1/ ( m1ψ1 ) ) ; as we set 1/ψ1 = O ( poly ( n ) ) , assuming polynomially large enough m1 concludes the claim . B.4.1 BOUNDING THE NORM OF THE FIRST LAYER ’ S OUTPUT IN THE WORST CASE Lemma 46 Suppose W ′ satisfies the assumption of Lemma 1 , i.e . ∥W ′∥ ≤ C1 , and ∥W ′j∥ ≤ c2/ ( 2 √ m1 ) except possible for indices in P , also defined in 1 . Then , with high probability over initialization sup x , ∥x∥=1 ∥ϕ′ ( x ) ∥ ≲ ( 1 +O ( m23d2 log ( m1 ) 2/m1 ) ) C1 + √ m3 C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 , which is O ( C1 ) for large enough overparameterization . Proof of Lemma 46 Note the because the VC-dimension of the class of binary functions with respect to halfspaces in Rd is d + 1 , the number of different sign patterns DW ( 0 ) , x for different x can be at most md+11 . Now similar to Equation ( 318 ) , for k ∈ [ m3 ] define Zk ( x ) = 1/ √ m1 ( W sk , j1 { W ( 0 ) j Tx } x ) m1 j=1 . ( 318 ) Then , for k1 ̸= k2 , as ∥x∥ = 1 : ⟨Zk1 ( x ) , Zk2 ( x ) ⟩ = 1 m1 m1∑ j=1 W sk1 , jW s k2 , j1 { W ( 0 ) j Tx } ≤ 1 m1 sup x m1∑ j=1 DW ( 0 ) , xj , jW s k1 , jW s k2 , j . But for each fixed DW ( 0 ) , x , using Hoeffding bound , we have with probability 1− δ : 1√ m1 m1∑ j=1 DW ( 0 ) , xj , jW s k1 , jW s k2 , j ≲ √ log ( 1/δ ) m1 . Applying the above for all possible sign patterns with δ < O ( 1/m1d+1 ) and a union bound , we have with high probability sup x , ∥x∥=1 ⟨Zk1 ( x ) , Zk2 ( x ) ⟩ ≤ 1 m1 sup x m1∑ j=1 DW ( 0 ) , xj , jW s k1 , jW s k2 , j ≲ d log ( m1 ) / √ m1 . We can even state the following stronger bound with respect to two adversarially picked vectors x , x′ : sup ∥x∥=1 , ∥x′∥=1 ⟨Zk1 ( x ) , Zk2 ( x′ ) ⟩ ≤ 1 m1 sup x m1∑ j=1 DW ( 0 ) , xj , jDW ( 0 ) , x′ j , jW s k1 , jW s k2 , j ≲ d log ( m1 ) / √ m1 , ( 319 ) because eachDW ( 0 ) , x′ j , j has at mostm d+1 1 cases as we discussed above , thenDW ( 0 ) , xj , jDW ( 0 ) , x′ j , j has at most md+11 possible cases , and applying a similar Hoeffding bound for each of them and a union bound as we did will imply ( 319 ) . We will use this generalized version in another section . Now combining Equation ( 319 ) with the fact that ∥W ′∥ ≤ C1 and applying Lemma 40 : sup x , ∥x∥=1 m3∑ k=1 ⟨W ′ , Zk ( x ) ⟩2 ≤ ( 1 +O ( m23d2 log ( m1 ) 2/m1 ) ) C21 . ( 320 ) On the other hand , setting m2 = m1 , m3 = d , and R = c2/ ( 2 √ m1κ1 ) in Lemma 29 , we get with high probability # ( j ∈ [ m ] : |V ( 0 ) j x| ≤ c2/ ( 2 √ m1 ) ) ≤ m1c2/ ( 2 √ m1 ) = √ m1c2/ ( 2κ1 ) . Noting that ∥W ′j∥ ≤ c2/ ( 2 √ m1 ) for j /∈ P , we conclude that with high probability , for any x , 1 { ( W ( 0 ) +W ′ ) Tj x ≥ 0 } and 1 { W ( 0 ) j Tx ≥ 0 } can be different in at most √m1c2/ ( 2κ1 ) of the j ’ s outside of [ m1 ] \ P . Therefore , as we have also |P | ≲ nc2 √ m1/κ1 from Lemma 1 , we conclude that with high probability , for any x , there are at most O ( nc2 √ m1/κ1 ) sign changes by adding W ′ to W ( 0 ) . This further implies : |ϕ′k ( x ) − ⟨W ′ , Zk ( x ) ⟩| ≤ 2/ √ m1 ∑ j : Sgn ( W ( 0 ) j T x ) ̸=Sgn ( ( W ( 0 ) +W ′ ) Tj x ) |W ′jx| ≤ ∥W ′∥2 √∣∣∣ { j| Sgn ( W ( 0 ) j Tx ) ̸= Sgn ( ( W ( 0 ) +W ′ ) Tj x ) } ∣∣∣/√m1 ≲ C1 √ nc2 √ m1/κ1/ √ m1 = C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 . Combining this with ( 320 ) , we conclude with high probability : sup x , ∥x∥=1 ∥ϕ′ ( x ) ∥ ≲ ( 1 +O ( m23d2 log ( m1 ) 2/m1 ) ) C1 + √ m3 C 3/2 1√ κ1 ( n3m3 m1λ0 ) 1/4 , which completes the proof . You may include other additional sections here .

Summary: The paper presents a theoretical analysis of a 3 layer neural network with ReLU activations. The main result is introduction of adaptive generalization bounds that are defined for a modified SGD algorithm. These bounds are robust to noise and data dependent.   The idea is to define a product kernel for the second layer of the form $K^{\infty}\odot G$, where $$K^{\infty}$ is fixed and $G$ is adaptive and can be modified so that $g_k$ has minimum possible norms. This leads to a definition of a complexity measure that together with a simple modification to the SGD algorithm defines a bound on the population risk.