Deep Networks and the Multiple Manifold Problem
Authors: Sam Buchanan
Dar Gilboa
John Wright
Venue: ICLR 2021 Poster (2021)
Keywords: deep learning
overparameterized neural networks
low-dimensional structure

Abstract: We study the multiple manifold problem, a binary classification task modeled on applications in machine vision, in which a deep fully-connected neural network is trained to separate two low-dimensional submanifolds of the unit sphere. We provide an analysis of the one-dimensional case, proving for a simple manifold configuration that when the network depth $L$ is large relative to certain geometric and statistical properties of the data, the network width $n$ grows as a sufficiently large polynomial in $L$, and the number of i.i.d. samples from the manifolds is polynomial in $L$, randomly-initialized gradient descent rapidly learns to classify the two manifolds perfectly with high probability. Our analysis demonstrates concrete benefits of depth and width in the context of a practically-motivated model problem: the depth acts as a fitting resource, with larger depths corresponding to smoother networks that can more readily separate the class manifolds, and the width acts as a statistical resource, enabling concentration of the randomly-initialized network and its gradients. The argument centers around the "neural tangent kernel" of Jacot et al. and its role in the nonasymptotic analysis of training overparameterized neural networks; to this literature, we contribute essentially optimal rates of concentration for the neural tangent kernel of deep fully-connected ReLU networks, requiring width $n \geq L\,\mathrm{poly}(d_0)$ to achieve uniform concentration of the initial kernel over a $d_0$-dimensional submanifold of the unit sphere $\mathbb{S}^{n_0-1}$, and a nonasymptotic framework for establishing generalization of networks trained in the "NTK regime" with structured data. The proof makes heavy use of martingale concentration to optimally treat statistical dependencies across layers of the initial random network. This approach should be of use in establishing similar results for other network architectures.

Published as a conference paper at ICLR 2021
DEEP NETWORKS AND THE MULTIPLE MANIFOLD PROBLEM
Sam Buchanan Columbia University scb21 570col ummbi edu
Dar Gilboa
John Wright Columbia University jw2 66 umbi a edu
Harvard University
dar_gi 1boa fas. harvard.edu
ABSTRACT
We study the multiple manifold problem, a binary classification task modeled on applications in machine vision, in which a deep fully-comnected neural network is trained to separate two low-dimensional submanifolds of the unit sphere. We provide an analysis of the one-dimensional case, proving for a simple manifold configuration that when the network depth L is large relative to certain geometric and statistical properties of the data, the network width n grows as a sufficiently large polynomial in L, and the number of i.i.d. samples from the manifolds is polynomial in L, uudooll-initializz the two manifolds perfectly with high probability. Our analysis demonstrates con- crete benefits depth and width in the context of a practically mootivated model problem: the depth acts fitting resource, with larger depths corresponding to smoother networks that can more readily separate the class manifolds, and the width acts as a statistical resource, enabling concentration of the randomly- initialized network and its gradients. The argument centers around the "neural tangent kernel" of Jacot et al. and its role in the nonasymptotic analysis of train- ing overparameterized neural networks; to this literature, we contribute essen- tially optimal rates of concentration for the neural tangent kernel of deep fully- connected ReLU networks, requiring width 2 L poly(do) to achieve uniform concentration of the initial kernel over a do-dimensional submanifold of the unit sphere Smh-1 and nonasymptotic framework for establishing generalization of networks trained in the "NTK regime" with structured data. The proof makes heavy use of martingale concentration to optimally treat statistical dependencies across layers of the initial random network. This approach should be use establishing similar results for other network architectures.
gradient descent rapidly learns to classify
INTRODUCTION
Data in many applications in machine learning and computer vision exhibit low-dimensional struc- ture (Fig. Taa. Although deep neural networks achieve state-of-the-ar performance on tasks in these areas, rigorous explanations for their performance remain elusive, in part due the complex interac- tion between models, architectures, data, and algorithms in neural network training. There is a need for model problems that capture essential features of applications (such low dimensionality). but are simple enough to admit rigorous end-to-end performance guarantees. In addition to helping to elucidate the mechanisms by which deep networks succeed, this approach has the potential to clarify the roles of various network properties and how these should reflect the properties of the data.
These considerations lead us to formulate the multiple manifold problem (Fig. [b]. binary classi- fication problem in which the classes are two disjoint submanifolds of the unit sphere Sho and the classifier is a deep fully-connected ReLU network of depth and width trained on N i.i.d. samples from distribution supported on the manifolds. The goal is to articulate conditions on the network architecture and number of samples under which the learned classifier provably separates the two manifolds, guarantecing perfect generalization to unseen data. The difficulty of instance of the multiple manifold problem controlled by the dimension of the manifolds dg, their separation A, and their curvature allowing to study the constraints imposed by these intrinsic properties of the data on the settings of the neural network's 's architectural hyperparameters such that the two manifolds can be separated by training with gradient- hased method.

---

Published as a conference paper at ICLR 2021
M.
1/K
A
Hedgehogs
Hairbrushes
S
(b)
Figure 1: (a) Data in image classification with standard augmentation techniques, as well other domains which neural networks are commonly used, lies on low dimensional class manifolds- in this case those generated by the action of continuous transformations on images in the training set. Tangent vectors at a point on the manifold corresponding to an application translation illustrated in green. The dimension of the manifold is determined by the dimension of the symmetry group, and is typically small. (b) The multiple manifold problem. Our model problem, capturing this low dimensional structure, is the classification of low-dimensional submanifolds of a sphere gro-1. The difficulty of the problem is set by the inter-manifold separation A and the curvature k. The depth and width of the network required provably reduce the generalization error efficiently are set by these parameters.
rotation or
Our main result an analysis of the mi-dimensionall case the multiple manifold problem, which reduces the analysis of the gradient descent dynamics to the construction of certificate-showee that a certain deterministic integral equation involving the network architecture and the structure of the data admits solution of small norm. We construct such a certificate for the simple geometry in Fig. 3 guaranteeing generalization in this setting.
Theorem (informal). Let do = 1. Suppose a certificate for M exists. Then if the network depth satisfies L Z poly(R, Cp, log(n))). the width sattisfees 2 poly (L, log(Lno)), and the number of training samples satisfies N 2 [L)y randomly-initialzzed gradient descent on N i.i.d. samples rapidly learns nelwork that separates the two manifolds with overwhelming probability. The con- stants Cp. K depend only on the data density and the regularity of the manifolds. In addition, if 2 then a certificate exists for the configuration of M shown in Fig.
Theorem I gives a provable generalization guarantee for a model classification problem with deep networks structured data that depends only on the architectural hyperparameters and properties of the data. addition, it provides an interpretable tradeoff between the architectural settings necessary to separate the two manifolds: the network depth needs be set according the intrinsic difficulty of the problem, and the network width needs to grow with the depth. Our analysis gives further insight into the independent roles played by each of these parameters in solving the problem, with the depth acting as a 'fitting resource', making the network's s output more regular and easier to change, and the width acting as "statistical resource', granting concentration of the network over the random initialization around a well-behaved object that we can analyze. Moreover, the sample complexity of Theorem iis dictated by the intrinsic difficulty of the problem instance which is set by the geometry of the data. As a consequence, we avoid any dependence of the width of the network on the number of samples, which is common deep network convergence results in the literature (e.g. (Allen-Zhu et al. 2019b Du et al. 2019), (Chen et al., 2021 Theorem 3.4)). As is the case in practice, given a fixed architecture, more data doesn't have detrimental effect on fitting !
Theorem is modular, in the sense that generalization guarantee is ensured for any geometry for which one can construct a certificate. The key to our approach will be to approximate the gradient
When using data augmentation, for example, the number of samples is effectively infinite yet highly struc- tured, enabling convergence and generalization.
2

---

Published as a conference paper at ICLR 2021
descent dynamics with linear discrete dynamical system defined in terms of the so-called neural tangent kernel ((x, x²) defined on the manifolds. Due to the structure in the data, diagonalizing the operator corresponding to this kernel is intractable in general, but show that constructing aertificate-aruuably an easier task, because it requires producing a bound on the norm of a solution an equation rather than producing the solution itself- suffices to guarantee that the error decreases rapidly during training given a suitably structured network.
We summarize the primary contributions of this work below.
Generalization in deep networks: There are few generalization results for deep networks trained efficiently with gradient descent available in the literature Theorem provides such guarantee that does not depend on any property of the trained network (c.g., norms of final weights) that is not readily available before training. In this context, the certifi- cate condition is equivalent to the initial network function having a controlled norm in a certain RKHS; this condition is natural in the training regime we consider, and appears ubiquitously in works on generalization in shallower networks (Ghorbani et al.) 2020 Ji &
Telgarsky 2020
Nitanda Suzuki 2021
Uniform concentration of the neural tangent kernel for deep ReLU networks: As an inter- mediate step in the proof of Theorem [ we establish essentially optimal rates of uniform concentration for the neural tangent kernel of an arbitrarily deep network (Theorem ( using martingale concentration, where we require the width grow only linearly with the depth. We expect this martingale approach to be applicable to essentially any other compositionally-strucii
network architecture. Our uniform result generalizes prior re- sults on pointwise concentration (Arora et al., 2019b) Ahlen-Zhu et al. 2019h), analogous to our Theorem B.3 and proves useful establishing generalization.
Strong regularity estimates for random ReLU networks: As a further consequence of the uniform concentration framework we have developed, obtain depth-logarithmmic Lips- chitz estimates for random ReLU networks of arbitrary depth and linear width, as well as (for still wider networks) a uniform approximation for the network output by a constant which improves with depth, both with overwhelming probability (Section 3.3). We also control the evolution of the Lipschitz constant during NTK regime training (Lemma B.7 showing that it scales polynomially in the depth. These results may be of interest in appli- cations where guaranteeing Lipschitz property for networks is important, such as GAN training (Miyato et al.) 2018) or denoising (Ryu et al.
2019 Sun et al. 2020).
1.1
RELATED WORK
Deep networks and low-dimensional structure. The notion of modeling data as low-dimensional submanifolds has been widely considered in the context of clustering (Wang et al. 2013) and man- ifold learning (Donoho & Grimes 2005 Fefferman et al. 2016). Goldt et al. (2020) independently proposed the "hidden manifold model". model problem for learning shallow neural networks for binary classification of structured data with motivations very similar to ours and which admits mean-field analysis (Gerace et al. 2020). The data model consists of gaussian samples from a low-dimensional subspace passed through nonlinear function acting coordinatewise in the stan- dard basis; although this models statistical variations around a base domain, a feature of real data that the model we study here lacks, we believe that the study of arbitrary density supported on two Riemannian manifolds lends our data model increased structural generality. In the context of kernel regression with the kernel given by the NTK of two-layer neural network, Ghorbani et al. (2020) study a data generating model that consists of uniform samples from a low-dimensional sub- sphere corrupted additively by independent uniform samples from a subsphere in the orthogonal complement, and target mapping that depends only on the low- dimeeniinii part. The authors ob- tain asymptotic generalization guarantees for this data model that reveal conditions under which the corruption degrades the performance of neural tangent methods.
"The closest result we are aware of is Chen et al. 2021 Theorem 3.4); this result involves a a-priori assump- tions on the trained network weights, which are only resolved for two-layer networks, and entails unnatural relationship between and N and possible exponential dependence of N on which Theorem avoids.
3

---

Published as a conference paper at ICLR 2021
Analyses of neural network training. To reason analytically about the complicated training pro- cess, we adopt the neural tangent kernel approach (Jacot et 2018). The first works to instantiate these ideas in a nonasymptotic setting obtained convergence guarantees for training deep neural net- works on finite datasets (Allen-Zhu et al. 2019b; Du et al. 2019). By exploiting more structure
in the data, generalization results have been obtained (Allen-Zhu al. 2019a Arora al. 2019a
Ji & Telgarsky| 2020; Oymak et al. 2019; Cao & Gu 2019] Suzukij 2020 Li et al. 2020 Allen- Zhu & Li 2020) that apply shallow networks, teacher-student learning scenarios, and/or hold conditional on the existence of certain small- interpolators. Other works have obtained gener- alization guarantees using generalization bounds for kernel methods (Ghorbani al., 2019] Liang et al., 2020: Ghorbani et al.) 2020; Montanari & Zhong. 2020) using the fact that the linearized predictor in the NTK regime can be linked to kernel method (Arora et al., 2019b). A parallel line
of works (Mei et al., 2018: Tien & Raginsky, 2020 Mei et al.) [219; Chizat & Bach, 2020) Fang ctt al. 2020) approach the problem by studying an mnfnttt-width limit of neural network training that yields different training dynamics. Approaches of this type are of interest because there is no restriction to short-time dynamics, and the limit of the dynamics can often be characterized in terms of a well-structured object, such max-margin classifier (Chizat & Bach 2020). On the other hand, it is often difficult to prove finite-time convergence to the limit.
2 PROBLEM FORMULATION AND MAIN RESULTS
2.1 DATA MODEL AND NETWORK DEFINITIONS
We consider data supported on the union of two class manifolds = =MM where M2 and M_ are two disjoint smooth regular simple curves taking values Sno-1, with no 2 We denote the data measure supported on M that generates our samples as 1t80, with corresponding density P, and write Pmnnn = inf p(æ) and pmax = Su p(æ)). We denote by k a uniform bound on the (extrinsic) curvature of the two curves, we write A = miDMEM ..EM Z(xr, x²) for the separation between class manifolds, where L(x, x²) cos" " x²) for unit vectors, and to have a quantitative characterization of 'how simple' the curves are, we assume there exist constants < cx I 1, Ky 2 such that for every 0 S ca/K k and every x, x' in common connected component of M, one has dist (x. x²) I KA$ if L(x, x²) I 8, where dist.M denotes the Riemannian distance. Our target function is the signed indicator for each class manifold fx( L,,
M
The model we consider is a fully- colneccon neural network with ReLU activations and access to i..dd samples from Sx and their corresponding labels. We parameterize our neural network with weights W1 E Ruxno, W E Raxx if f E (2....,
and WL+1 E R plxn, which we collect as 0 = (W1 .... WL+1, and write the iterates of the forward pass as a (x) = [W' ( (xx) for é .,..., L with (x)
...,,,,, T.).
(a) = x. which we also refer to as features or activations, with the network output written as So fa())= = W L+1 orl (z),, and the prediction error as Co(x)) fo(#) (x))
For an i.idd. sample (æI,.... xN from p we write μ.a = + Ei 8z, for the empirical measure associated to the sample, and we consider the training objective ,, (0) 1 JM (Go(z))² (x), ic. the empirical risk evaluated with the square loss. We train with vanilla gradient descent with constant step size > 0: after randomly initializing the parameters 0, W -ii.d. (0, 2/n) if É E [L] and W M..... N/ 1) we consider the sequence of iterates 0 = O - (0 where VL,,, represents 'formal gradient' of the empirical loss, which we define in detail in Ap- pendix A [. We say the parameters obtained at iteration à of gradient descent separate the manifolds M if the classifier implemented by the neural network with the parameters 0,² labels the two man- ifolds correctly, i.e. if f= (ax) signSoo
(x:) = 1 for every & E MM. As a shorthand, we will denote quantities evaluated at 0% with a subscript k; an omitted subscript will denote k 0, and we will write explicitly On 0. Additional notation is provided in Appendix A.5.1
2.2 ERROR DYNAMICS AND CERTIFICATES
Because it is difficult to endow the network parameters generated by the gradient iteration with a particular interpretation, we prefer reason about how the network error G evolves under gradient
4

---

Published as a conference paper at ICLR 2021
descent. We calculate (in Lemma B.8)
= = -T -T/ 0f x2) (x') du (z'),
(2.1)
where we have defined the integral kernel O (a, x²)== = So l (x'), Vo -Iree (0g) ( (x)) dt,
where fe, denotes a formal gradient the initial network function with respect to the parameters, which is defined in detail in Appendix A.1 We then define nominal error evolution by
CE (2) Ck (z) - -S// //x 0(x, (x)) dxx)
(2.2)
with identical initial conditions _0- = ( and where ( x²) = (e. (2)), So (xx)) is the so- called neural tangent kernel with associated integral operator @ We prove that the error evolution 2.T is well-approximated by the nominal eπror evolution under suitable conditions on the network width, step size, and number of samples, which together ensure that training proceeds in the "NTK regime" where 0 stays close to e. As for the nominal error evolution (2.2). we note that this system linear, time-invariant, and stable when T set appropriately small, so the norm of the nominal error is guaranteed to decrease rapidly if the initial error Ç aligns well with eigenfunctions of @ corresponding to large eigenvalues. However, computation these eigenfunctions is intractable for general data geometries and distributions because the operator θ is not generally translationally invariant on M. To overcome this issue, we prove this alignment implicitly by constructing an approximate solution to the linear integral equation O[g] such that ||9l| is sufficiently
small. To be precise, g T will be called 81, d2-certificcat for the dynamics (.2 if
|||[g]
Sllez S Sd1; | S S82.
(2.3)
2.3 MAIN RESULTS AND PROOF OUTLINE
Our main result is that conditional on the existence of a certificate of suitably small norm for M, gradient descent provably separates the two manifolds in time polynomial in the network depth.
Theorem 1. Let M be one-dimensional Riemannian manifold satisfying our regularity assump- tions. For any 0 < 8 < 1/e, choose
L 2
C1 max Cnu log (1/8) log Co ,x2 x233//2)
n = 7 log" (1/5) log" (T))))
N 2 L 10
and fix such that
Co STS nL
C4 m..
Then if there exists a certificate in the sense of 2.3 with 6, = CsC///2 Vlo((1/8) =C,C3/2/Tog(1/// log(nno)) L and 52 Ca./oooo /1)))18)) log(nno)//npp)) (n))))) with probability least 1-8 over the random initialization of the network and the i.i.d. sample from n the parameters obtained at iteration [L 3,99/4 1//4/((r)) /((r() of gradient descent the finite sample loss E,x yield a classifier that separates the two manifolds.
The constants Ci...., Co are suitably chosen absolute constants, the constants K, Kx ex are respectively the extrinsic curvature constant and the global regularity constant defined in Sec- tion 2.1 the constant Cp is defined as max{pwin- Pmin., and the constant Cyoe is defined as (M-)))
5 ) (min
For one- -ie-dimeniionll instances of the two manifold problem with sufficiently deep and overparame- terized networks trained in the small-step steull-teep-aaee step-size regime, [ completely reduces the analysis of the gradient iteration to the certificate problem. From a qualitative perspective, the network resource constraints imposed by Theorem are natural:
(i) The network depth L is set by geometric and statistical properties of the data with only mild polylogarithmic dependence on the ambient dimension no, which reflects the role of depth in controlling the capability of the network to fit functions.
5

---

Published as a conference paper at ICLR 2021
(ii) The network width is set by the depth L: the inductive structure of the network causes quantities that depend on the initial random weights 0g to concentrate worse as the depth is increased, which can be counteracted by setting the width appropriately large.
iiii) The sample complexity of N ' reflects the capacity of the network via the depth, and is in particular independent of the width n, which can thus be interpreted as purely a statistical resource.
In addition, the conclusion of Theorem I implies not just that the expected generalization error with respect to n of a binary classifier is zero, but the stronger separation property, i.c. that the generalization error will be zero for any choice of test distribution supported on .M simultaneously. We give a brief sketch of the proof of Theorem in Appendix [A.4] To obtain a generalization guarantee from Theorem [ it only remains construct certificate for M. We demonstrate this for the family of simple, highly-symmetric geometries shown in Figure 3 and leave the case of general one-dimensional manifolds for future work.
Proposition 1. Let M be r-instance of the two circles geometry studied in Appendix C.I.1
and shown in Figure 3 with r 2 1/2. For any 0 < 5 S 1/e, if L. 2 GA and n. 2 C2L log" (1/5) log (Lng log(1/8)), then there exists a certificate in the sense of 2.3 satisfying the requirements of Theorem] 7 with probability at least 1 36 for some absolute constants Ci C2 O.
Taking union bound, Proppssition[sheeeee shows that under the hypotheses of Theorem I with probability at least 46 a certificate exists for the geometry shown in Figure 3 as soon as L is larger than constant multiple of the inverse separation A even the separation approaches zero. We conjecture that a similar phenomenon holds for more general geometries, possibly with additional dependencies the curvature and global regularity parameters of M. The dependence of I. on the geometry is due to the "sharpening" effect the depth has on the kernel θ governing the dynamics and thus the fitting capabilities of the network, illustrated Figure 2a
20
50
100
500
10
100
100
1000
( )
((a,,ee) fm
(2,x²)
M
xf2 (xax²)
(a)
(b)
Figure 2: (a) Depth acts as a fitting resource. As L increases, the rotationally iaaaioonlly-invaiiann invariant kernel ê (a slight modification of the deterministic kernel in Theorem [ decays more rapidly as function of angle between the inputs L(x, x') (n is held constant). Below the curves we show an isometric chart around point MM Once the decay scale of θ is small compared to the inter-manifold distance A and the curvature of M_, the network output can be changed at æ while only weakly affecting its value M_. This is one mechanism that relates the depth required to solve the classification problem to the data geometry. (b) Width acts as a statistical resource. The dynamics at initialization are governed by θ, a random process over the network parameters. As increased, the normalized fluctuations of θ around é decrease (here L 10). These two phenomena are related, since the fluctuations also grow with depth, as evinced by the scaling in Theorem 2
To prove that the nominal error evolution 2.2 decreases rapidly and approximates the actual error evolution 22.I) throughout training, it is essential to have precise characterization of the 'initial' neural tangent kernel @. One of our main technical contributions is to show concentration of θ in the regime where the width scales linearly with the depth L.

---

Published as a conference paper at ICLR 2021
Theorem 2. For any do N. let MM be do do-dimensional complete Riemannian submanifold of gho-1 Then if n 2 C₁ C. log" (Cmnno.) one has with probability at least 1 - -n- that for
every (x. x') E M
L-1
L-1
( x²)
Σ
cos (0(0
III F'ml
II
S Cay CnynL log (oghooCoo)
1=0
where we write v (xx x²) with an abuse of notation, gl denotes the e-fold composition of
(p()) cos-1 ((1 - ") cos V + = the constants C1 C2 > are absolute, and the constant CM > 0 depends only on the diameters and curvatures of the class manifolds (Lemma C.4
For networks of uniform width that are wider than they are deep by a certain constant factor, believe that the scalings in Theorem - are essentially optimal: the variance calculations of Hanin & Nica (2000) give some heuristic evidence here, and believe the idea of using diagonal con- centration to prove deviation lower bounds could be generalized to rigorously establish optimality. Figure 2bjilusttsttes the phenomenon underlying Theorem 2 We discuss the proof of Theorem 2 in
more detail in Sections I.n 3.3
3 KEY PROOF ELEMENTS
3.1
CONCENTRATION AT INITIALIZATION, MARTINGALES AND ANGLE CONTRACTION
The initial kernel θ is a complicated random process detined over the weights (W1 ..... WL+1). To control it, we first show for fixed (æ, x²) that ( x²) concentrates with high probability, and then leverage approximate continuity properties to pass to uniform control of O. Here we describe our approach to pointwise control; uniformization is discussed in Section 3.3 The kernel can be written the form
O (a))
Σ az)) 3 f
where (x(x) = (yylt1PPe(0)) ... W P will be referred to as backward features, and Pte(m) is a projection onto {a (a)(x) ())) 0}. We consider (3) (3 as representative example: sum of martingale differences.
up small residual term, this random variable can be expressed Formally, for & E [L]. let J denote the o-algebra generated by all weight matrices up layer L, with F denoting the trivial o-aleebra. We can then write
L+1
1 (x)) - 90((9) &
ge ge
W WI, [[g:(W" W!,,)) F+
R
(3.1)
for some functions 9r and controllable residual R, where ~ Lax, x²). If fix all the variables in Jt-1, the fluctuations in the é-th summand will be due to W alone. Intuitively, since each weight matrix appears at most once (x))) [ will appear at most twice in gt, and therefore ge will have subexponential distribution conditioned on F(-1 and concentrate well around its conditional expectation. This property stems from the compositional structure of the network, with independent sources of randomness introduced at every layer, and is essentially agnostic other details of the architecture. The concentration of the summands in (3.1) implies concentration of the sum: even though the summands are not independent, they can be controlled using concentration inequalities analogous to those for sums of independent variables (Azuma 1967 Freedman 1975).
Showing that terms of the form (a())) (æ), a (xx') concentrate in the linear regime gives rise to ad- ditional challenges. Here we exploit an essential difference between the concentration properties
Since we do not use the "NTK parameterization" the norm of our NTK scales like nL rather than L. Due scaling of the weights (Section 2.1) the contribution of the final layer to the NTK is negligihle and can he dropped. This leads discrepancies between the expression above and similar expressions found in the literature-we we show essential equivalence hetween our NTK others in Appendix A.3
"Technically, the features a (w) depend on all the weights up layer E and hence so does the projection matrix P (o)( but our analysis shows that this dependence has only minor effect the statistical fluctuations.
7

---

Published as a conference paper at ICLR 2021
of the angles between features v' = (1) relative to those of the correlation process
(a (a'(x)) a (x')) studied in prior works on concentration of θ: when 2 = 0, we have that v' = deterministically, whereas the correlation process behaves like subexponential random variable with small but nonzero deviations. 'oggtherr with smoothness, this clamping phenomenon allows us to show concentration of the angle at layer l' around the function (1²). which no larger than constant multiple of g-!_ This contraction of the angles with depth is the key to establishing Theorem 2 in addition, it gives the invariant kernel Ô (see Figure 2bb its sharpness at zero and localization properties, both of which increase as the depth is increased and which we exploit in the
proof of Proposition I
We provide full details of our approach in Appendices Dand E
3.2 CERTIFICATE CONSTRUCTION: GENERAL FORMULATION AND A SIMPLE EXAMPLE
By simple argument that relies on positiveness of O we show that if we can solve the certificate problem (2.3). then for suitably chosen learning rate and number of iterations & (Lemmaa B.6
M_
v Vdlog
P
I CCp
2 21-e
L
Figure 3: The coaxial circles geometry.
If the network sufficiently deep, the norm of the nominal error can thus be made arbitrarily small in number of iterations that scales only polynomially with the problem parameters.
Because formulation of the certificate problem 2.33 accommodates approximate solutions, der a minor condition on the network width n (see Proposition [ it suffices to solve an auxiliary system ② [g] Ç. where and Ç are amalytically.covveienn convenient approximations to @ and Ç produced by our concentration analysis, including Theorem 2 For the simple geometry in Fig. [ we show in Appendix C.I.I how to solve this auxiliary system using Fourier analysis, where we require 2 -1 The depth of the network is thus determined by the geometry of the data, and specifi- cally by the inter-manifold distance which intuitively sets the "difficulty" of the fitting problem. In Section we discuss approaches to constructing certificates for general smooth curves.
3.3 UNIFORM CONCENTRATION AND ITS CONSEQUENCES
To uniformize the pointwise estimates of Section 3.1 we must overcome the issue that the back- ward features 3 (x) are not continuous functions of the input, due to the matrices (m)) Our approach is discretize the input space, control the number of features that can change sign near each point in the discretization, then extend the pointwise estimates of Section 3.1 to the setting where a small number of features have changed sign-again again, we find martingale concentration necessity to achieve linear width-depth scaling. We give full details in Appendix D.3
Although Theorem the main application of these estimates- with uniform control of θ we can prove operator norm bounds on its corresponding integral operator θ, which is of great help in proving generalization results-they also imply useful regularity estimates for the initial ran- dom network fe,- For example, we prove that networks of uniform width x niLL are with high probability (log no) Pn))( (log L) L)-Lipschitz functions on Ro (Theorem B.5 in particular, the Lipschitz constant depends only logarithmically on depth, in contrast to existing results in the lit- erature (Nguyen et al. 2020). For networks of larger width Z d3 prove that with high probability the network Je, is approximately constant on the domain M C ghe-1 (Lemma D.11
- for (x) du du (a')
1
sup (x)
$
#EM
We find this result to be useful in simplifying the certificate construction problem of Section 3.2
4 DISCUSSION
Certificates for curves. The most urgent task toward expanding the scope of Theorem T is the construction of certificates for geometries beyond the coaxial circles of Proposition [ The proof
8

---

Published as a conference paper at ICLR 2021
of Proposition I relies heavily translation invariance of the intra- and inter- inter-manifold distances in the coaxial circles geometry in order to avoid the need for certain sharp technical estimates for the decay of the kernel é. With sharper control of the decay of the kernel Ô, it is possible to select the network depth in way that grants appropriate worst-case control of the magnitude of the cross- manifold integrals in the action of Ô (as in Figure 2a), allowing us reduce what is essentially a one- mne-manfolld certificate construction problem that can be solved with harmonic analysis. Beyond these considerations, it is important extend Theorem to manifolds of dimension dg > 1, which should be relatively straightforward. Our concentration results, notably including Theorem 2 are already applicable to manifolds of arbitrary dimension.
Convolutional networks and non-differentiable manifolds. Although we have motivated our data model in the multiple manifolds problem using applications in computer vision, it is important to note that the spatially-structum do not carry a differentiable structure (Wakin et al., 2005), so the assumption of bounded curva- ture may not be realistic here. On the other hand, in these applications it is standard to employ convolutional network architecture. We anticipate that our martingale concentration framework can be extended to these architectures, and beyond establishing analogues of Theorem [ in this setting, we believe it should be possible to obtain similar guarantees for models of image articulation man- ifolds. In particular, one might expect randomly-initialize, convolutional networks enjoy local invariance properties, like the scattering networks of Mallat (Mallat 2012: Bruna & Mallai 2013), which could achieve a degree of invariant classification without expending additional network re- sources computing convolutions over general LCA groups (Cohen & Welling) (016))
image articulation manifolds that arise data in these contexts
The importance of being low-dimensional. Ghorbani et al. (2019) show that kernel ridge regres- sion with any rotationally invariant kernel on S (including that of a deep network) is equivalent to polynomial regression with a degree polynomial if the number of samples is bounded by d and consider here, would expect less
d + c0. For data lying on a low-dimensional manifold, pessimistic rates; indeed, in a subsequent work (Ghorbani et al.) 2020) the authors establish similar guarantees to Gohorrraini et al. (2019) for linear data model with low-dimessional structure in terms of a smaller "effective dimension" " In comparison, although our present certificate construction ar- gument only implies dynamics for the restrictive coaxial circles geometry of Figure 3 for which one can obtain guarantees for kernel regression with a shallow NTK by the results of Ghorbani et al.) (2220), the general multiple manifold problem formulation allows one to model nonlinear structure in the data, and measures fitting difficulty through intrinsic parameters like the curvature and sep- aration. The guarantees in Ghorbani al. 2019 2020) depend on the degree of approximability of the target function by low-degree polynomials, and although this achieves additional generality over our model, seems more challenging to relate this to geometric or other types of nonlinear low-dimensional structure.
The NTK regime and beyond. In recent years there has been much work devoted to the analysis of networks trained in the regime where the changes in ON remain small and the dynamics in (2.I) are close to linear (Jacot et al.) 2018; Lee et al.) 2019) Arora et al. [019b] AAllen-Zhu & Lij wwwpeerprrametrzzeeeeeeeeeee
2019) (referred to the
regime). Concurrently, there have also been results highlighting the limitations of this regime. In (Chizat & Bach 2018) the authors coin the term "lazy training" in referring dynamics where the relative change in the differential of the network function is small compared to the change in the objective during gradient descent. While the dynamics we study indeed fall into this category, the analysis makes it evident that not all lazy training regimes are created equal. Our performance guarantees depend on the structure of the kernel é, and on controlling the fluctuations of O2 around it. We are able to control these only if the width of the network is sufficiently large compared to the depth. In contrast, lazy training can also be achieved in homogeneous models by simply scaling the output of the model (Chizat & Bach [2018), in which case one cannot argue that the kernel has the decay properties that enable it to fit data.
Our analysis hinges on staying in the NTK regime during training. We obtain suboptimal scaling of n with L in Theorem ! because we treat all changes that occur in O during training as being adversarial to the algorithm's 's ability to generalize. It is likely that if an improved understanding of feature learning can be incorporated into an analysis of the dynamics, the resulting scaling require- ments would be more realistic.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work was supported by the grants NSF 1733857, NSF 1838061, NSF 1740833, NSF 1740391, NSF NeuroNex Award DBI-1707398 (DG), the Gatsby Charitable Foundation (DG) and Swartz fellowship (DG), and by fellowship award (SB) through the National Defense Science and Engi- neering Graduate (NDSEG) Fellowship Program, sponsored by the Air Force Research Laboratory (AFRL), the Office of Naval Research (ONR) and the Army Research Office (ARO). The authors would like to thank Ethan Dyer, Guy Gur-Ari, Quynh Nguyen, Jeffrey Pennington, Sam Schoenholz, Daniel Soudry, and Tingran Wang for helpful discussions/feeddhck..
REFERENCES
P-A Absil, R Mahony, and R Sepulchre. Optimization Algorithmss Matrix Manifolds. Princeton University Press, April 2009.
Zeyuan Allen-Zhu and Yuanzhi Li. What can resnet learn efficiently, going beyond kernels? In Advances in Neural Information Processing Systems, pp. 9015-9025, 2019.
Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep
learning. arXiv preprint arXiv:2001.04411. January 2020.
Zeyuan Allen-Zhu Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. In Advances in Neural Information Processing Syxtems, volume 32, 2019a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via Over- Parameterization. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pp. 242-252. PMLR, 2019b.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-Grained analysis of opti- mization and generalization for overparameterrzze Two-Layer neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 322-332. PMLR, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, pp. 8139-8148, 2019b.
Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, Second Series, 1933)3357-377, 1967.
Stéphane Boucheron, Maud Thomas, et al. Concentration inequalities for order statistics. Electronic Communications in Probability, 17, 2012.
Stéphane Boucheron, Gábor Lugosi, and Paseal Massart. Concentration Inequalities: A Nonasymp- totic Theory of Independence. OUP Oxford, February 2013.
Haim Brezis. Functional Analysis, Sobolev Spaces and Partial Differential Equations. Springer, New York, NY, 2011.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell., 35(8) August 2013.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Zixiang Chen, Yuan Cao, Difan Zou, and Quanquan Gu. How much puempttrattiiiiitoonnn is suffi- cient to learn deep ReLU networks? In International Conference on Learning Representations, 2021.
Lénaïc Chizat and Francis Bach. A note on lazy training in supervised differentiable programming. CoRR, abs/181 81 82.07556 2018.
10

---

Published as a conference paper at ICLR 2021
Lénaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 1305-1338. PMLR, 2020.
Youngmin Cho and Lawrence K Saul. Kernel methods for deep learning. In Advances in neural information processing systems, pp. 342-350, 2009.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2990-2999, New York, New York, USA, 2016. PMLR.
Donald L Cohn. Measure Theory. Birkhäuser, New York, NY, 2 edition, 2013.
M Corless, G H Gonnet, D E G Hare, D J Jeffrey, and D E Knuth. On the LambertW function. Adv. Comput. Math., 5(1):329-335) December 1996.
Herbert David. Order Statistics, pp. 1039-1040. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 978-3-642-04898--
Victor H de la Peña. A general class of exponential incqualities for martingales and ratios. Ann. Probah., 27(1))537--33 January 1999.
David L Donoho and Carrie Grimes. Image manifolds which are isometric to euclidean space. J.
Math. Imaging Vis., 23(1):5-24, July 2005.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learming Representations, 2019.
Lawrence Craig Evans and Ronald F Gariepy. Measure Theory and Fine Properties of Functions. CRC Press, December 1991.
Cong Fang, Jason D Lee, Pengkun Yang, and Tong Zhang. Modeling from features: a mean-fiell framework for over-parameterized deep neural networks. arXiv preprint arXiv:200.001552, July 2020.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. J. Amer. Math. Soc., 2944)9933-104, February 2016.
David Freedman. On tail probabilities for martingales. Ann. Probab., 3(1):100-118, February 1975.
Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. General- isation error in learning with random features and the hidden manifold model. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3452-3462. PMLR, 2020.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. CoRR, abs/1904. 12191, 2019,
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? In Advances Neural Information Processing Systems, volume 33, pp. 14820-14830. Curran Associates, Inc., 2020.
Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Phys. Rev. X, 10(4): 041044, December 2020.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kemnel. In International Conference on Learning Representations, 2020.
Christopher Heil. A Basis Theory Primer: Expanded Edition. Birkhäuser Boston, 2011.
Roger A Horn, Roger A Hom, and Charles R Johnson. Topics in matrix analysis. Cambridge university press, 1994.
11

---

Published as a conference paper at ICLR 2021
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gener- alization in neural networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbi- trarily small test error with shallow ReLU networks. In International Conference Learning Representations, 2020.
Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: Isoperimetry and Processes. Springer, Berlin, Heidelberg, 1991.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear mod- els under gradient descent. In Advances in Neural Information Processing Systems, volume 32.
Curran Associates, Inc., 2019.
John M Lee. Introduction to Riemannian Manifolds. Springer, Cham, 2 edition, 2018.
Yuanzhi Li, Tengyu Ma, and Hongyang R Zhang. Learning Ever-Parametrized Two-Layer neural networks beyond NTK. In Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 2613-2682. PMLR, 2020.
Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of Minimum-Norm interpolants and restricted lower isometry of kernels. In Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 2683-2711. PMLR, 2020.
Stéphane Mallat. Group invariant scattering. Commun. Pure Appl. Math., 65(10):1331-1398, Octo- ber 2012.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proc. Natl. Acad. Sci. U. S. A., 115(33):87665-7767) August 2018.
Song Mei, Theodor Misiakiewiez, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Proceedings of the Thirty-Sccond Confer- ence on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pp. 2388- 2464, Phoenix, USA, 2019. PMLR.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
Andrea Montanari and Yiqiao Zhong. The interpolation phase transition in neural networks: Mem- orization and generalization under lazy training. arXiv preprint arXiv:2007.1282 /2826,, July 2020.
Robb J Muirhead. Aspects of Multivariate Statistical Theory. Wiley Series in Probability and Statistics. John Wiley Sons, Inc., Hoboken, NJ, USA, March 1982.
Quynh Nguyen, Marco Mondelli, and Guido Montufar. Tight bounds the smallest eigenvalue of the neural tangent kernel for deep ReLU networks. arXiv preprint arXiv:2012.11654 December 2020.
Atsushi Nitanda and Taiji Suzuki. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. In International Conference Learning Representations, 2021.
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint (90,::9966.0992. June 2019.
Mark Rudelson and Roman Vershynin. aon-asymptotic theory of random matrices: Extreme sin- gular values. Proceedings of the International Congress of Mathematicians 2010 (ICM 2010),
pp. 1576-1602. June 2011.
12

---

Published as a conference paper at ICLR 2021
Ernest Ryu, Jialin Liu, Sicheng Wang, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. Plug- and-Play methods provably converge with properly trained denoisers. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 5546-5557. PMLR, 2019.
John M Sullivan. Curves of finite total curvature. In Discrete differential geometry, pp. 137-161. Springer, 2008.
Sun, J Liu, and U Kamilov. Block coordinate regularization by denoising. IEEE Transactions Computational Imaging, 6:908-921, 2020.
Taiji Suzuki. Generalization bound of globally optimal non-convex neural network training: Trans- portation map estimation by infinite dimensional langevin dynamics. Advances in Neural Information Processing Systems, volume 33, pp. 99224-19233 Curran Associates, Inc., 2020.
Michel Talagrand. Concentration measure and isoperimetric inequalities in product spaces. Pub- lications Mathématiques de I'Institut des Hautes Etudes Scientifiques, 81(1):73-205, 1995.
Belinda Tzen and Maxim Raginsky. A mean-field theory of lazy training in two-layer neu- ral nets: entropic regularization and controlled McKean-Vlasor dynamics. arXiv preprint arXiv:2002.01997, February 2020.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge University Press, 2018.
Michael В Wakin, David L Donoho, Hyeokho Choi, and Richard G Baraniuk. The multiscale structure of non-differentiable image manifolds. In Wavelets XI, volume 5914, pp. 59141B. Inter- national Society for Optics and Photonics, 2005.
Xu Wang, Konstantinos Slavakis, and Gilad Lerman. Multi-Manifold Modeling in Non-Euclidean spaces. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pp. 1023-1032. San Diego, California, USA, 2015. PMLR.
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of em- pirical measures in wasserstein distance. Bernoulli, 25(4A) November 2019.
Shunhui Zhu. The comparison geometry of ricci curvature. In Comparison Geometry, volume 30 of MSRI Publications, pp. 221-262 Cambridge University Press, May 1997.
13

---

Published as a conference paper at ICLR 2021
CONTENTS
Extended Problem Formulation
16
A.1
Regarding the Algorithm
16
A.2
Regarding the Data Manifolds
16
A.3
Regarding the Initialization
17
Proof Outline for Theorem
19
A.5
Notation
20
A.5.1
General Notation
20
A.3.2
Summary of Operator and Error Definitions
21
B
Proofs of the Main Results
23
Main Results
23
B.2
Supporting Results Dynamics
27
B.3 Auxiliary Results
46
Skeleton Analysis and Certificate Construction
Certificate Construction
61
C.I.I Two Circles
62
C22
Auxiliary Results
65
Geometric Results
65
C.2.2 Analysis Skeleton
Concentration Initializatinn
83
D. Notation and Framework
83
D.2 Pointwise Concentration
83
D.2.1 Forward Concentration
83
D.2.2
Backward Feature Control
94
D.3
Uniformization Estimates
101
D.3.1
Nets and Covering Numbers
101
D.3.2
Controlling Support Changes Unitrmlyy
101
D.3.3
Uniformizing Forward Features Under SSC
108
D.3.4
Small Support Change Residuals
120
D.4
Auxiliary Results
148
E
Sharp Bounds on the One-Step Angle Process
159
Definitions and Preliminaries
160
Main Results
161
Supporting Results
162
E.3.1 Core Supporting Results
162
14

---

Published as a conference paper at ICLR 2021
E.3.2
Proving Lemma E.6
164
1.3.3
Proving Lemma
179
General Properties
186
3.5
Differentiation Results
190
E.3.6
Miscellancous Analytical Results
202
Deferred Proofs
232
Controlling Changes During Training
235
Preliminaries
235
Changes in Feature Supports During Training
235
Changes in Features During Training
237
F.4
Changes in O During Training
241
Auxiliary Lemmas and Proofs
242
Auxiliary Results
249
APPENDICES: SUMMARY OF CONTENTS
We briefly summarize the contents of each of the subsequent appendices.
A. We discuss the contents of the problem formulation section from the main body, Sec- tion 2.1 in more technical detail, in particular giving technical definitions for formal gra- dients, regularity conditions, and so on. We provide a proof sketch to offer some intuitions about the proof of the main result. We also summarize notation and the key operator defi- nitions that appear throughout the paper.
B. We give proofs for our main results. We provide supporting results on the NTK regime dynamics gradient descent and other relevant technical lemmas, discussed in the proof sketch of Section 2.3
C. We give technical definitions relevant to the cross-manifold perspective on certificate con- struction, construct certificate for the two circles geometry Figure and provide tech- nical estimates on the kernels th and u that remain after applying our measure concentra- tion arguments the NTK θ.
D. We collect results centration result for the NTK, Theorem 2 Some of these results are also relevant for controlling changes during training.
concentration relevant proving our main uniform
E. We collect results relevant to proving certain concentration result for the angles between features as they propagate across one layer of the initial neural network. The main results of this section are fundamental to the study of the concentration of angles Appendix D and we provide them in a separate appendix due to their length.
F. We establish results on uniform control of the changes during training of the NTK O from its "initial value" of O These are a key ingredient in dynamics arguments in Appendix
G. We provide statements of general technical lemmas that are a classical nature, which we rely throughout the other appendices.
15

---

We reiterate here our notational conventions for quantities evaluated at these iterates: we denote evaluation of quantities such as the features and prediction error at parameters along the gradient descent trajectory using a subscript k. with an omitted subscript denoting evaluation at the initial k = 0 parameters, and add superscript N to parameters such as the prediction error to emphasize that they are evaluated at the parameters generated by (A.I. For example, in this notation we express Say as C In addition, use 0, to denote the initial parameters 0g We emphasize Published as a conference paper at ICLR 2021
A EXTENDED PROBLEM FORMULATION
A.1 REGARDING THE ALGORITHM
We analyze gradient-like method for the minimization of the empirical loss C,N. After randomly initializing the parameters 0g as W Wiid NO 2/n) if f E [L] and WL.+1 Lidd ((0,, 1), independently of the samples æ1,.... UN we consider the sequence of iterates
0 1+1 0 -VL )
(A.1)
where т > 0 is a step size, and VG represents a 'formal gradient of the loss LuN, which we define as follows: first, we define formal gradients of the network output by
fo = 3 (z)) { (x))
for E [L] and M, where have introduced the definitions
0 (a) = (WL+1) (an W Pl.-.(=).... W (()))
for 8 1,,,...,,. 1, and where we additionally define
T Tx( = supp (lag(e))))
Pl((#)
Σ e;e;
SET((m)
for the orthogonal projection onto the set coordinates where the l-th activation at input æ is posi- tive. We call the vectors By (a: the backward features or backward activations-the correspond to the backward pass of our neural network. We also define
Vwitsfole - az(x)).
We then define the formal gradient of the loss f by
4 (0) = / M Violo)oo(o)
Let us emphasize again that the expressions above are definitions, not gradients in the analytical sense: we introduce these definitions cope with nonsmoothness of the ReLU [-] On the other hand, our formal gradient definitions coincide with the expressions one obtains by applying the chain rule to differentiate N at points where the ReLU differentiable, and we will make use of this fact to proceed with these formal gradients in a manner almost identical to the differentiable setting.
the dependence of certain quantities on these random initial parameters notationally, including the initial network function foa-
A.2 REGARDING THE DATA MANIFOLDS
We now provide additional details regarding our assumptions on the data manifolds. For background on curves and more broadly Riemannian manifolds, we refer the reader to (Lee) 2018 Absil et al. 2.09). We assume that M MMMMM, where M4 and M_ are two disjoint complete connected Riemannian submanifolds of the unit sphere guo-1, with no 2 3. In particular, Mi are compact. We take as metric on these manifolds the metric induced by that of the sphere, which we take in turn as that induced by the euclidean metric R We write u and n for the measures M
'Certain parts of our argument, such as the concentration result Theorem B.2 are naturally applicable to cases where Mi themselves have finite number of connected components with a mild dependence on this number, and we state them such. We skip this extra generality in our dynamics arguments to avoid an additional juggling act' that would obscure the main ideas.
16

---

Published as a conference paper at ICLR 2021
and M_ (respectively) induced by the data measure μ²= and assume that 1 admits density p with respect to the Riemannian measure on M. writing p+ and p- for the densities on M induced by the density p. When do 1, add additional structural assumptions to the above: we assume that M2 are smooth, simple, regular curves.
Concretely, that M admits density P with respect to the Riemannian measure means that
= S/a (x))
JmL (+(x)dV+(a)) + S/a P (-(x)dV-(a)) (x).
When do 1, because Mr smooth regular curves, they admit global unit-speed parameteriza- tions with respect to are length 7+ : + + -"0-1 where 4 are intervals of the form [0, leMl(e)n))),. In this setting, the curvature constraint is expressed
[n"()) NET
| K.
max
sup
sup Se
and we observe that the fact that M sphere curves implies x 2 Exploiting the coordinate representation of the Riemannian measure and the fixed inherited metric from R", we thus have
Jmn P= = p+ o 7 (t)ll = =f{- P= o (t) dt.
We will exploit this formula in the sequel to compare between (M) and ("(M) norms of func- tions defined on the manifold. More generally, we will frequently make use of similar reasoning that leverages the existence of unit-speed parameterizations for the curves.
For elarity we rewrite the global regularity condition: we assume there exist constants 0 CA S 1, Kx 2 1 such that
vs E (0, cx/k), (x, E M. x M., * (+,
<(x, x') S = ( S KA8,
(A.2)
where dist denotes the Riemannian distance between points in a common connected component, and we define Cs K3/c2. Because M+ are simple curves, they do not self-intersect; the as- sumption (A.2) gives a quantitative characterization of how far the curves are from self- self-intersecting. We illustrate how the associated constants can be obtained from the assumption that the manifolds simple curves: for either {+, -), consider connected component M, C C,, and for any < 8 S leneee)),, define
inf r.(s) o. X.M.: dist
<x)
(s) 0, by compactness
construct sequence of pairs of points that converges to r. (s). but this would imply that M, is self-intersecting, contradicting our assumption that it is simple. follows that r. (s) > for any value of s. we now define K2 = r. (s) 8, it follows that for any (x,x") x²) M. x M..
<<x,x') <
dist x') < K,s.
Our regularity assumption implies that a single such constant holds for a range of scales below the curvature scale, which is mild assumption since K, approaches 1 as approaches O.
A.3 REGARDING THE INITIALIZATION
The manner in which we have defined our initial random neural network fe, sometimes referred to as "fan-out initialization" in the literature- guarantees that feature norms preserved from layer to layer in the network, and thereby avoids the vanishing and exploding gradient problems. The difference between this initialization and the called "standard" or "fan-in" initialization is only in the first and last layer weights, yet in sufficiently deep network trained in the NTK regime the effect of any single layer is negligible and the dynamics of our network will be essentially identical to one with standard initialization. On the other hand, following the work of Jacot et al.
WW point out that the curvature of the manifolds does not enter into the proof of the concentration result Theorem B.2 there is ambiguity in discussing curvature only in the context of curves.
17

---

Published as a conference paper at ICLR 2021
2018), it has become common in the theoretical literature to consider different construction of the neural network called "NTK parameterization", which is some ways more convenient for theoretical analysis. In particular, Arora et al. (2019b) prove their results NTK concentration using this parameterization; to facilitate a comparison between our concentration result (Theorem and theirs, discuss the connection between fan-out and NTK parameterization in this section. This material is well-known and no doubt can be found already in the literature, but we believe it may be helpful to translate into our notation.
Recall our definitions for the weights and features in our neural network: we have W wi.id. N 2/n) if E .0.,...,42 L and Wh+! Mid. (00,, 1), with features defined for 0, .....,.. L by
x [W'a'' otherwise,
-=0
a'
and output (x=) = WL+1 00-11( Within this section-and only within this section-we we shall define auxiliary weights by ( E TRuxnn G E TRuxn for integer < < < I 1, and GLL+1) E
RIx=, with distributions ( ridd (N0,,) 1) for E {0, ..., + 1}. independent everything else in the problem. As before, for l E {0,1,.... D}} we a (x) to denote the layer- features:
(8)
f=
NTT ②
G
aNTK (z)
(L-1)
otherwise.
This network's output will be written
fNTK(W)=
2 II
G(L+1) a
By I-homogeneity (absolute) of a, follows that foo SNIK- As the notation suggests, the network fNIK corresponds to "NTK parameterization" network network-although this network and fe, are equivalent in terms predictions, their "gradients" are not equivalent. The NTK for the NTK parameterization network is obtained by differentiating (at points of differentiability):: after caleu- lating (as in Lemma B.8). we introduce notation as we did for the fan-out parameterization network in Appendix A.1 so that
ONTK(O,2)) (x) fNTKk(')).
with (for f= 1, 11
(oOoNTT(2)) =
IIV v
L
2
BKiTK (x) ( (x))
where
( (x)((x)=
(uttt) PITT (L)) PPTKK()) ... ((++2) PI PINTE((())
6= 1,..,...... L { L,
and
IN NTK x = supp (1 (e))(())))))
We shall relate the NTK parameterization NTK θNTK to our fan-out parameterization NTK using homogeneity of the ReLU, First, let us observe that
{i e [n] (a (x)), > 0} d i [m] (a)tk(t)), > =0].
because [-]+ is 1-homogeneous and we have ( d / vn/2W' when e & L. For e E (0, 1,..., L.).
we note that both a axxx) and (x) depend only on the parameters (W) ..... W and
(G .....,G()). respectively. If we write = (W...., Wh) and θNTK = (G ( ((D)),
then it follows that a (xr) is E-homogeneous function of 0 (and likewise for ITK (æ)). In
(6)
18

---

Published as a conference paper at ICLR 2021
addition, the projection matrices (z) O-homogeneous functions of 0, and taking l {0, l..... L - }} and counting parameters in the definitions of (x) and (xTR(2) (2) implies that
these two functions are (L- f -)-homogeneous functions of 0 and θNIK, respectively. Of course, for & L, they are 0-homogeneous. Thus, using that G 1 Vn/2W for = L again, we obtain
GNT oh 2/ So t= 0=0.1..... Wo
t= I=1+1
Although we have argued equidistriutiionaity above for each index l separately for simplicity, the elementary nature of our arguments (we are just moving scalars around) and the statistical dependen- cies across gradients allows us to apply the same argument 'in parallel' the sum of inner products between gradients, yielding
@NTK(Z (') = (a (x), (z). + aa)
This expression makes immediately clear that our concentration framework proves sharp con- centration of the NTK of a uniform- wiith NTK parameterization feedforward ReLU network that improves over the results of Arora et al. (2019b) when the data are on the spherd a simple adap- tation of the proof of Theorem B.3 will suffice.
A.4 PROOF OUTLINE FOR THEOREM
In Appendix B we prove a slightly more general version of Theorem Din Theorem B.1 Here, we give brief outline of the proof of this result.
Proving the separation property essentially requires us to obtain control of lIck" I (m()), and by an interpolation inequality (Lemma B.14) it suffices to control the generalization error ||C 1|, and
the smoothness (measured through the Lipschitz constant) of GN We start with the generalization error, picking up from where we left off at the end of Section 2.2 the triangle inequality gives
|| I|r; S 112, + +|
1
(A.3)
which allows us to divide the analysis into two subproblems: characterizing the nominal dynamics (Lemmas B.6 and B.12), and the nominal-to-fin transition (Lemma B.7p. Beginning with the (2.2) to write
nominal dynamics,
C (Id- [c].
where O denotes the operator on - corresponding to integration against the kernel Q and Id denotes the identity operator. The definition of θ and compactness of M imply that O a posi- tive, compact operator (Lemma so these dynamics are stable when 7 is chosen larger than the operator norm of @ However, the rate of decrease of ||<E" with could still be extremely slow if the initial error has significant components in the direction of eigenfunctions of O corre- sponding small eigenvalues, and because acts roughly like a convolution operator, we expect there to exist eigenvalues arbitrarily close to zero. By solving the certificate problem (2.3), we can assert that misalignment does not occur. To solve the certificate problem, as we describe Sec- approximations for θ and Ç: the exact definitions of these approximations Ô and ? are given in Appendix A.5.2 and prove their suitability as approximations in Theorem B.2 (a slightly more general version of Theorem 2 and Lemma D.11 respectively. As we have discussed Section 3.1 our rates of concentration for θ about e are essentially optimal-the poor rates that end up appearing in Theorem B.1 are set by later parts of the argument.
tion 3.2 we work with analytically-coneniie
With our approximation to θ justified, we show that for any sufficiently small step size and number of iterations k; solving the certificate problem (2.3) guarantees appropriate decrease of the
? 'The results of Arora et al. 2019b) apply to data of norm no larger than 1. but it is straightforward to extend results for spherical data to this setting, using the 1-homogeneity of θ in each argument (as a kernel on the entire ambient space to "o write ( ||>]||||~>||| (() (z/ ///l/////aaaa/ a /||ar|[22)
19

---

Published as a conference paper at ICLR 2021
nominal generalization enror; additional details are discussed in Section 3.2
The key property that we use in constructing certificates in Proposition B.4 (the "appendix version of Proposition [T] is that as the depth L increases, the kermel é sharpens and localizes (Fig. 2ap: the conditions on L in Theorem B. guarantee that the sharpness is sufficient ensure that the cross-manifold integrals in the certificate problem are small in magnitude, which leads to rapid decrease of the nominal error.
Our precise characterization of this phenomenon is presented in Appendix
To complete the proof, we will justify the nominal-to-finte transition in A.3).. Starting from the update equations (2.1) and (..2). subtracting and rearranging gives an update equation for the differ-
ence:
(Id -) [4t -- N |<-- + + [4--
In particular, if T is chosen less than the operator norm of θ we can take norms on both sides of the previous equation, apply the triangle inequality, then exploit telescoping series cancellation to obtain the difference bound
&
|| - -sh -1 ST
/
6% (,,)') (x')dn" (x') - 00. (0))""" (x') d (a')
12500
(A.4)
There are two obstacles to controlling the norm terms the RHS of (A.4) the kernels O distinct from the kernel θ due to changes in the weights that occur during training, and the empirical measure n incurs sampling challenge, we measure the changes to the NTK during training in a worst-case fashion as
relative the population measure u To address the first
A :E(0.1.....k) ||l," - -
and train in the NTK regime, where the network width is larger than a large polynomial in the depth L and the total training time kt is no larger than L/n. These conditions imply that with high probability A is no larger than a constant multiple of n n poly(I, (o.yy[.,d do) for small constant 8 >> O, so that the amortized changes during training k can be made small by sufficient overparam- eterization. We provide full details of this argument in Appendix [F] By the preceding argument, we can use the triangle inequality and Jensen's inequality pass from the norm term in (A.4) to difference-of-measure term which integrates against O, and by Theorem B.2 we can replace the integration against by an integration against a smooth, deterministic kernel, which leads to bound
||CE - - 162. S RR do) + Σ
01(4(-, x2)) (x') ( dy (2'))
M
19,00
where R& is residual term that we argue is small in the NTK regime with high probability, and for concision we write d/ to denote the function of L(x, x') that appears in Theorem B.2 To control the remaining term, we make use of basic result from optimal transport theory, which states that for any probability measure on the Borel sets of metric space X and corresponding empirical measure S one has for every Lipschitz function f
{ ((x)) (du(x)
(du(t)))pppp d (x)) s ll W (M,
where W(., denotes the 1- I-Wasserstein metric, and concentration inequalities for empirical mea- sures in the 1- 1-Wasserstein metric (Weed & Bach 2019). To apply this result to our setting, it is necessary to control the change throughout training of the Lipschitz constant of CN" and one must also account for the fact that the metric space in our setting is M, which has two distinct connected components. We treat the first issue using an inductive argument, and our treatment of the second issue (Lemma B.13) leads the dependence on the degree of class imbalance demonstrated in the constant Cp in Theorem B.1
A.5 NOTATION
A.5.1
GENERAL NOTATION
nn E N. we write [n] = (I.....n). We generally use bold notation æ. A for vectors, matrices, and operators and non-bold notation for scalars and scalar-valued functions. For vector matrix
20

---

Published as a conference paper at ICLR 2021
A, we will write entries as either 2 or Agj. or (z), or (A);;; we will occasionally index the rows or columns of A similarly as (A); or (A),, with the particular meaning made clear from context. We write [x]+ max(x, 0} for the ReLU activation function; if æ is vector, we write [x]+ to denote the vector given by the application of I++ to each coordinate of æ and we will generally adopt this convention for applying scalar functions to vectors. If æ a² IR" are nonzero, we write <(x, x') = (( x²)/ ||=/||/||=/] for the angle between æ and æ'.
The vectors (e:) denote the canonical basis for R". We write (ar, y) E; ziyi for the euclidean inner product on R", and if 0 < p +00 we write ||>||p (E, |z,
1)1/ for the 8 norms (when p 2 1) on R". We also write ||0]|lo |( E [n] at # 001 and ||||>>> max4e(1) [aa]]- The unit ball in IR" is written B" = {x E I= || - 11, and we denote its (topological) boundary, the unit sphere, as 5--1, We reserve the notation I| for the operator norm of a m x n matrix A, defined as All = su |a || more generally, we write || = sup ||]|| -1 ||Ax]lg for the corresponding induced matrix norm. For m X n matrices A and B, we write (A, B) tr( A B) for the standard inner product, where A" denotes the transpose of A, and || (44, A) for the Frobenius norm of A.
The Banach space of (equivalence classes of) real-valued measurable functions on measure space (X, ()) satisfying If 1/2 < +oo is written (X) or simply L if the space and/or measure is clear from context; we write || ||ا> for the associated norm, and (., L) for the associated inner product when p = 2. with the adjoint operation denoted by * For an operator T : L. - L3. we write T[] to denote the image of f under T, T to denote the operator that applies T i times, and rg ||T
= Supllsll $1 [[T[f We use Id denote the identity operator, i.e. Idig] [g = g for
every 9 E D.. We say that T is positive ((f)) 12 2 0 for all f E L2; for example, the identity operator is positive.
For an event& & in a probability space, we write 1g to denote the indicator random variable that takes the value if E & and 0 otherwise. If 0, by 9 NO0, x² I) we mean that g TR is distributed according to the standard i.i.d. gaussian law with variance o², i.e., it admits the density (2nO²) -2/2 exp( - (=ll==//22(22) with respect to Lebesgue measure on I"; we will occasionally write this equivalently as g "Lid. NV(0, We use to denote the "identically- equivalence relation.
d
We use "nuuerrcal constant" and "absolute constant" interchangeably for numbers that are inde- pendent of all problem parameters. Throughout the text, unless specified otherwise we use c, d' d", C, C", C", K, K', K", and so on to refer to numerical constants whose value may change from line to line within a proof. Numerical constants with numbered subscripts C1, C2.... and have values fixed at the scope of the proof of a single result, unless otherwise specified. We gener- ally use lower-case letters refer to numerical constants whose value should be small, and upper case for those that should be large; we will generally use K, K' and so on to denote numerical constants involved in lower bounds on the size of parameters required for results be valid. If and g are two functions, the notation ʃ S 9 means that there exists a numerical constant C > 0 such that f S Og: the notation f Σ 9 means that there exists a numerical constant C > such that f 2 Cg: and when both are true simultaneously we write f > g. If f is a real-valued function with sufficient differentiability properties, we will write both f' a'and ff for the derivative of f. and when higher derivatives are available we will occasionally denote them by f(x), with this usage specifically made clear in context. For metric space X and a Lipschitz function f X + IR we write [ Lip to denote the minimal Lipschitz constant of f.
will
A.5.2 SUMMARY OF OPERATOR AND ERROR DEFINITIONS
We collect some of the important definitions that appear throughout the main text and the appendices in this section. We begin with the NTK-type operators that appear in our analysis. Recall from Appendix A. Jour definition for the backward features: we have
(x(x) (x) (WL+1) Pi,
( W Pi)-(()).... ... w Plets(a))*
for t 0, 1, 1,.., ,L- L- - 1, and where we additionally define
T (xx = supp (1o6(e)20):
Pl(())
Σ e
(x))(m)
21

---

Published as a conference paper at ICLR 2021
for the orthogonal projection onto the set of coordinates where the 2-th activation input is positive. "The" neural tangent kernel is defined as
((x, = ( v (x), Voo(
= {a a (x))) + L-1
with corresponding operator _z (M)
J O(x, x' )g(a') du (xk).
+ [Q](())
As shown in Lemma B.8, this is not exactly the kernel that governs the dynamics of gradient descent: the relevant kernels this context are defined
O (xx x0) S( fory (x'), Vog -1- CON dt.
We define operators @ L (M) corresponding integration against these kernel in manner analogous the definition of A
= I/ ON (a. (')2))((z)) (x').
+ (gl(()
We then move to the deterministic approximations for θ that we develop: we define
(p)) cos" ((1='((1-)))) - ((1-vTT) cos + (1/T)) sin sin
which governs the angle evolution process in the initial random network, studied in Appendix B and write ((() to denote f-fold composition of &o with itself. We define
1-1 Σ
L-1 II
か (v)
cos
1
t=
f=t
which is the "output" of our main result on concentration, Theorem B.2 and
1-1
(1 (()
(b))
II (=t
which is at the core of the certificate construction problem. We think of ab as an analytically-simpe version of th, with an approximation guarantee given Lemma C.IT Throughout these appen- dices, we will make use of basic properties of th and v that follow from properties of without explicit reference; the source material for these types of claims is Lemma E.5 which gives ele- mentary properties of (for example, that it takes values in [0, x/2]. which implies that t and wi are no larger than nL/2). For derived estimates, we call the reader's attention to the contents of Appendix C.2.2 will make explicit reference to these results when we need them, however. Although have mentioned approximations Ô and @ in the main text, will prefer in these appendices to explicitly reference & and ú to avoid confusion; as exception, we will the O notation in Appendix
C as discussed there. Our approximation for the initial prediction error is
-(x)==)(()) / fa, (x))
(A.S)
where recall fe, denotes the network function with the initial (random) weights. In particu- lar, this approximates the network function with constant, and the enror as a piecewise constant function M4. This approximation is justified in Lemma D.11
22

---

Published as a conference paper at ICLR 2021
B PROOFS OF THE MAIN RESULTS
B.I MAIN RESULTS
Theorem B.1. Let M be one-dimensional Riemannian manifold satisfying regularity assump-
tions. For any 0 < 5 Σ 1/e, choose I. so that
L 2 C1 max C₁ log" (1/5) 24 (C log(1/8)) , C2),
let N 2 L 10 set = C2L log (1/6) log (Lno), and fix T > 0 such that
C3
C4
nL? S r S
nL
Then if there exists function 9 E - (M) such that
Vlog(1/0) (og(nooo)
/log(1/0) log(nna)
|||[g] C1123-0 (M)) I Cs SOs, I. min Punur
Puin grert) }³ [191l S Cg focrs
Tupmin
(B.I)
with probability at least 1 over the random initialization of the network and the i..d sample from , the parameters obtained at iteration |139/44| [nT]] of gradient descent on the finite sample loss L yield a classifier that separates the two manifolds.
The constants Cr...., C4 > O depend only on the constants qcert, C5. Co > O, the con- stants K. Cx respectively the extrinsic curvature constant and the global regularity stant defined in Section 2.1 and the constant Cyse is defined as max Pmnin {1 + Panax) (min (1((+)) n 1t (M-))) where ==11 8ffert-
Proof. The proof an application of Lemma B.7 with suitable instantiations of the parameters of that result; to avoid clashing with the probability parameter 5 in this theorem, we use E for the parameter ó appearing in Lemma Define C2 max Pmin- Pmin) }. We will pick q 39/ 44 and & 5/47, so that the relevant hypotheses of Lemma B.7 become (after worst-casing in the bound on N somewhat for readability)
K (og(nngCCm)
1/3
n K' max dd log L. 2/5
ex
L 2 K" max Ca C²}
N K" Cp (133/18+(152/2775) (1 Pmaxx) 133/54
18/3 log L,
min (Mxx(M)))) 1 (M-)) 19/18
and the conclusion we will appeal to becomes
CC2 +2 (1 + Penax) )12
2/4 d//4lo 4/3 L
Le
P s 10 / LM(M))
> 1
min (100 (M
L1/11
71T
Under our choice of 7 and enforcing
L (2 + Pmax) 11/2,33/1 log 11/3
(B.2)
(min {u 00 (M-))) 11/2
we have the equivalent result
s / /wn
< 2
1
> 1 - 133-c0
21 d'd
where the last bound holds when d 2 K Klog L, which is redundant with the hypotheses and d
required to use Lemma B.7 Thus, when in addition d 2 (1/d)) 1og(1/5), we obtain
TP I mi (o (M))
1
<
-
2 -
(B.3)
2
23

---

Published as a conference paper at ICLR 2021
Therefore conclude, need only argue that choices of N, L. d, and in the theorem statement suffice to satisfy the hypotheses of Lemma[..7 We have already satisfied the conditions on and q. We notice that (B.2) implies that it suffices to enforce simply N L 'o,, and following
Lemma C.4 we can bound CM as in (B.62) in the proof of Lemma B.7 by
len (Mm) len(M_) 2 21+ Pmaxx
CM 1+
11000
(M+) (ML)
Pmin
eliminate CM from the lower bound on d while paying only an extra factor of 2 in the constant. In addition, because & 2 1 and Cx 2 max 11,1/1 1/xx),
Because 2 7,99 and I 1 Cp(1 + Pmaxx),
1/3
we can remove x²/5 and
lower bounds on n, since they are enforced through L already the bound L 2 K=* worsening the absolute constant if needed. These simplifications lead us to the sufficient conditions (plus the certificate existence hypotheses)
z K max(log(1/O)) log(nno))
n 2 L9 d log") L
Cp 22 13 (1 + Pmax) ) 11/2 233/ log 44/3
1o/3
L 2 K" max
(min (M+ 1 (M (M- n²Css
N L
We ignore the condition on N below, since it matches with the theorem statement. When ó 1/e, given that no 2 3 we have mng 2 e and max [log(1/5), log(nno)) S 1og(1/8) log(nno). For the sake of simplicity, we can also round up the fractional constants in the lower bound on L. We can eliminate d from these sufficient conditions by substituting the lower bound into the conditions on and L, and this also implies that our conditions on certificate existence in the theorem statement suf- fice for the certificate existence hypothesis for Lemma B.7 Thus, we have the remaining sufficient conditions
> KL 99 log"(1 (1/8) (og/)))oognnmma) (nno) L
Cl 1 8 (1 +1+ Pmaa)) log (1/8) log log (nno) log
x² 2C }
L z K' max
(min (~ (M
Using Lemma B. and choosing L larger than sufficiently large absolute constant and larger than log(1/ (1)), obtain that suffices to enforce for
KL log (1/6) log (Lng).
In the hypotheses of the theorem, have chosen the equality the last bound. This implies log(nmo) S C loog(Lno) so it suffices to enforce the L lower bound
KL log (1/8) ((n((( in
Cp C11+8quure (1 + Prnax) 1 log (1/5) log (ogg((mnoo)
L K" max
nx² .2Ox
(min (x
Defining, as in the theorem
C!1+8quam (1 Pmaa)
Cya
(min (n00 (M+), 1x (M (M-)))
and using C,== 2 1, we can worsen the absolute constant K' in order to apply Lemma B.15 once again, obtaining the simplified condition
L 2 CK' max { Cr (1/6) log?* (C log , CC))
These conditions reflect what is stated the lemma.
Theorem B.2. Let M be di-dimensional Riemannian submanifold of $0a-1 For any d 2 Kdg log(nng if > K d. then has event of probability at least 1 c-d
L-1 II 11
π 2
Σ cos (p)99() E=0
S &vdnnt
sup (M.="))MMM
P-t
where write Z(x,x') in context with abuse of notation, C. C, K' O are absolute constants, and CM > depends only on the number of connected components of M and their diameters and curvatures (Lemma C.4).
24

---

Published as a conference paper at ICLR 2021
Proof. We have by the definition of +
L-1
( x (a+ (a)), a (x')) (x²)) Σ a (a')) (x), (x')).
(B.4)
E=0
Under the stated hypotheses, Lemmas D.10 and D.13 give uniform control of each of the terms appearing in this expression with suitable probability to tolerate 2L + 1 union bounds, which gives simultaneous uniform control of the factors on an event & with probability at least 1 e Starting from B.4), we write with the triangle inequality
L-1 n Σ cos (v))(())
L-1
(x, x')
II
1-
- az))))
2
π
(B.5)
1-1
(p)
(a (x), a (a')) (x), (
n
Σ
cos
(w)
II P==
1
2
t=0
t=U
By the triangle inequality, have
1-1
(0(
(a)), (x')) ('))) -
n
(v)
II
cos
t'=8
L-1
so(0)(())
(z)), < (z),, a (x')))
(x)))
n
II
1
===
1 (1)))))
(v))(())
(v)
cos
2 é'=
Under the conditions on n, L,, and d, we have on the event & that for each
(n)) ( 2,
sup (x,'')M
can conclude that on 3
L-1
so(()(())
(a (x), a
n cos (v)
II d'tt
1
<3vd'nt.
The conditions d, and L imply that this residual is larger than that incurred by the level-L features, which is no larger than 2. Returning to (B.5), we have shown that on E
(1-
I--1
(l0(()) (1--)
O x²)
Σ cos (g()(w)
II n==
<CVdLLL.
π
E=U
After adjusting the other absolute constants absorb C into d, this gives the claim.
Theorem B.3 (Pointwise Version of Theorem B.2 submanifold of gno-1_ For any d 2 K Klog n. (x, x²) M M
Let M be a dn-dimensional Riemannian
K' max(11 d L) then one has for any
P 0 xd)
Σ
cos
(v(0(()
II E'Et
< ddn'L
> 1-
where we write v = L(x, x') in context with an abuse of notation, and c, K, K' are absolute constants.
Proof. Follow the proof of Theorem B.2 but invoke the pointwise versions of the uniform concen- tration results used there (i.e., Lemmas D.I and D.4 after rescaling to relocate the log n terms.
Proposition B.4. Let M be an r-instance of the two circles geometry studied in Appendix C.I.1 with r 2 1/2. For any 0 < 5 < 1/c, if 2 KL KL* log"(1 (1/5) log' (Lng log(1/5)) and L 2 K' 12)-1/2 then there exist absolute constants C5 Cg > and a function g such that (B.1) is satisfied with the choice qorrt 1/2 with probability at least 36. The constants K, K' 0 are absolute.
25

---

Published as a conference paper at ICLR 2021
Proof. Given 2 1/2 and L z max K, (T/2)(1- (1)))(1-)2)-1/2) we have by Lemma C.1 that there exists
g such that JM v o 2( , (²()') du (a') = with
|, - (64/VT) 1K
(B.6)
1/2
M2mml
By this bound, the triangle inequality, the Minkowski inequality, and the fact that jr a probability measure, we have
||[[]] - <l12 - 110 3 4 (M) + +15 +115-Elle3-
[1(lL=((M) + 11s (((
I c v/ 4 (MMMM))
(B.7)
np
An application of Theorem B.2 and Lemma C.11 gives that event of probability at least
-
||0 o 1 (M S CCn/L
if d 2 Kdo log(nnoC(m) and K' k'd An application of Lemma D.I gives
P
$ <
( $
V2d
2 1 - el-e -ed
and
P P[llllo= Val -
as long as n 2 Kd and 2 K'do log(nmoCy)) where we use these conditions to simplify the residual that appears Lemma D.I1 In particular, combining the previous two bounds with the triangle inequality and union bound and then rescaling d, which worsens the constant and the absolute constants in the preceding conditions, gives
P I| S v 21 21-2e-as
Combining these bounds using union bound and substituting into (B.7). . we get that under the preceding conditions, on event of probability at least 3e we have
Cvd
1
[ [g] Slla,.
1
L
1/2 Prnin
Cvd
max (eax(Pra)
(Pmint
Pmin
-1/2
L
where we worst-case the density constant in the second line, and addition, on the same event, we have by (B.6)
Vd 1/2 mP
|لاlا S (64/VT)
To conclude, we simplify the preceding conditions on and turn the parameter into a parameter
8 > 0 in order to obtain the claimed form of the result. We have in this setting dg = 1, and also that
C. is bounded by an absolute constant; since no 2 3, we can thus eliminate the parameter CM from our hypotheses by adding extra absolute constant factor. Choosing d > (1/c) log(1/5) we obtain that the previous two bounds hold on an event of probability at least - 35. When 8 < 1/e, given that no we have nR 2 and max (log(//og1//// log (m)))) (og(1 log(nnu),
that suffices to enforce the requirement 2 K log(1/8) log (mn)) for a certain absolute constant K > 0. We can then substitute this lower bound form claimed in (B.1) with foert = 1/2. For the hypothesis on n,, we substitute this lower bound on into the condition on n to obtain the sufficient condition n 2 K' L log (1/6) log"(nm)) (nmp). Using Lemma B.15 and possibly worsening absolute constants, then get that it suffices to enforce K' 7,5 log' (1/5) log pogg(TTm (Tog log(1/5)) which is the hypothesis in the result.
d into the certificate bounds above to obtain the
26

---

Published as a conference paper at ICLR 2021
Theorem B.5. There exist absolute constants C, K, K' 0 such that for any d 2 Kno log n, if n 2 K'd then on an event of probability at least E -od the natural extension of for to R is 3v d.iiscciii.z
Proof. The proof a simple application of Lemma B.17 which (because Sa. I-nonnegatively c homogeneous and so are all its intermediate feature maps 0 (x)) implies that it suffices to control the Lipschitz constants of the maps and bound them on the unit sphere, together with Lemmas D.1T and D.12 In particular, for any d 2 Ko log(n) and any K'd L, we have that there exists an event probability at least e on which
llfel L00 1) S Vd.
and
I 800 IlEAp S Vd.
Applying Lemma B.17 it follows that fe, _8 + RR is 3v d-Lisshhizz on event of probability at least 1 - e-od
B.2 SUPPORTING RESULTS DYNAMICS
Lemma B.6 (Nominal), Suppose Cerry Ceorts qoert O are absolute constants. Then there exist absolute constants c, c C² C", Ct > and absolute constants K, K", K" such that for any d > Kdg Iog(nngC(m) and any 1/2 S q S 1, is 2 K'd if L 2 K" dC and if additionally there exists g L2 Iy satisfying
| - -1lla (M) S Cen Crear Vd L
Yeart d
| 13 S CoertPmin
and T > 0 chosen such that
d S
nL
then one has
2 21-e-d |15 L2 (M) S Vit
P
OSkSLa/Aay
and in addition
C" VVCggorevilloog villg L nkt
n
1s 11 5
2 2
(orr Meart SkgTa/
Moreover, one has
& Σ ==0
dood L
P
1150 12 (M) S 22
21-e-d
TTT
OSkkLaa
The constant Cp = max max Pmlo)
Proof. We will combine Lemma B.12 with various probabilistic results obtain a simple final form for the bound from this result.
Invoking Lemma B.12 we can assert that for any step size
satisfying
1
(B.8)
O
12,00 ( (M))
and for any le satisfying
/ee 1911
kr 2 V 2 [()))
(B.9)
27

---

Published as a conference paper at ICLR 2021
the population dynamics satisfy
3119ll2300011
log (V2
l I
Ic< 11 (M) S v3| - <11 (M)
(B.10)
kt
|| [sll
We state the bounds we will apply to simplify this expression. An application of Lemma
D.11 gives
/2d
$ P $ (llmcmm)
e
(B.11)
L
and
P P|lSlle S vd 21-e
(B.12)
long as n 2 L5 and K' log(nn)C.M). where we these conditions simplify the residual that appears in the version of (B.IT) quoted in Lemma(D.11 In particular, combining (B.IT) and (B.12) with the triangle inequality and union bound and then rescaling d, which worsens the constant and the absolute constants in the preceding conditions, gives
P [IKill S vv 2 2e 2e-od
(B.13)
In addition, we can write using the triangle inequality
[Kll) 2 ( - -||s -1|-- 5/ -ileocan
and
1 sup xEM fz()) / for (z))
|
= max (1/ Sor (2' (e - -1:1// So (2' du du (c')
1,
so that, by (B.l)). we have if 2 2
P (Klll
(B.14)
Because 0 is probability measure, Jensen's inequality, the Schwarz inequality, and the triangle inequality give
+ (M)-
sup (x,')eMM
((xx
sup (x.x))MMMMMMMMM (( x') th1 ((x, x')l (" |th o L(x, x'),,
and an application of Theorem B.2 and Lemma then gives that on an event of probability least
c -ed
| 1, 12 (M) S CCL
(B.15)
and > K'd L. We will write E for the event consisting of the union of the events invoked for the bounds B.11 (B.15). which has probability at least e
provided d 2 Kdy
by union bound and choice of d 2 K. We will conclude by simplifying (B.10) on E. First, we note that by (3.15. the step size condition
B.8 satisfied on provided
(B.16)
nL
which holds under our hypotheses. Next, on E, we write using decreasingness of x + - log x and B.
3 3||و|| 13 (M)
3 llol 1 (3)
3 3/9 13 (M) (M)
(v3
log
V [<
log
Kt
Kt
kTvd
V311gl1ze (M))
(13 (1919112 (M)
-
log
1/2 kr
/2 Vd
(B.17)
28

---

Published as a conference paper at ICLR 2021
By the hypothesis on g. have & and so follows that on &
Vd
|19 19112g (M) S Cpmin
Yoorr
(B.18)
1991 199113 (M)
2
kT kd
nke Pruen
The function æ log x is strictly increasing function on [0, e0,e_1,, when chosen such that
Ce
S
k;
(B.19)
Piin
we have on & by B.17
3/9 11gll1,, (M)
1 12 (M)
CV6d Roest
log
log (C nkTTP Plen
(C''nkT
(B.20)
Kt
nkt
min
Additionally, in the context of the condition
B.9 notice that by B.14 and (B.18). on & we have
3e
9 (o (M)
v Cevd
2 TIlSlle
PTPii
so that given 2 1, we have that the choice
Ceva Brart nT PTPpiin
(B.21)
implies both conditions (B.9) and B.19). We can simplify B.20 using the hypothesis kr 1 L9 L9/n with 1/2 < g < 1: get
nk' fecert
p
I. 1+
C
where the last inequality requires T. 2 Paunn /C, which implies
3 3|g 10
l99 (99l
C² C'Volog
log
(B.22)
lT
2 || 1. (==))
nkT groer
The conditions we need to satisfy on kt can be stated together as
Cevd
k S L9/n,
np Jeort
and it possible to satisfy these conditions simultaneously as long as
1/9
CeVd
L 2
Rears
c for the quantity on the RHS of this inequality from q 2 1/2; Patn
We obtain an upper bound suffices choose L larger than this upper bound instead. The other simplifications easier: using the assumption on the norm of θ [g]] C. we have
CVd
O [g] Clleg (M)) S Cyeer 1/2
Lp
Worst-casing terms using our hypotheses d and L to obtain a simplified bound, on E, have thus shown that when (B.2 is satisfied, we have
1
log nkT
1 || 12 (M) S &CC vdl
29

---

Published as a conference paper at ICLR 2021
We have
1 log L
L Llog I 2 KT,
L nkT
which is implied by the hypothesis Kt I L9/n nn/nan as long as L 2 e. So we can simplify to
11 (133)0 CC vdlog
nkT
We also need a bound that works for that do not satisfy (B.21). From the update equation for the dynamics in the proof of Lemma B.12 and the choice of T, we also have
I| (MM) S (5lll1g.0 S Vd,
where the last bound is valid on &. Finally, we can obtain the claimed sum bound by calculating using our 'small-k' and 'large-k' bounds:
Lova/(nTP (
c Il (M))
Σ
11 +
1155 11 (M)
4=0
s=
[CVd/ (nrp )
C'Vd
C"Cygan vd Vdlog L
S <v vdl
1
+
nt Boort
Cdd
C" C"Cycco Vallgg T.
P Ovd
ds
nT P
nn
(Va/(nnT Acort
8
Cd
1) log I. + + C' C'max Mfaas (Pmin
C" C"CCcon vdlog L
Soert
where the second inequality uses standard estimates for the harmonic numbers and C' C'Vd/(nn (nrppnn Scart) 2 1, which follows from S <d/(nI.), d and 2 Ky for suit- able absolute constant K; and the third inequality integrates and simplifies, using kr S L/n and again d 2 and T, 2 Cpliass Worst-casing constants and using nt S 1, we simplify this last bound
to
4
Cdlog* L nT
E1 I max
Pmmn
2quare
Pauin
To see that the conditions on L in the statement of the result suffice, note that we have to satisfy (say)
L 2 K p feort and L 2 K' Pmin pmuur the first of these lower bounds is tighter when Pmin 2 1, and the second when Pminn < 1, and it suffices to require L 2 K 2 and L 2 K' Pmin instead. O
22tert
Lemma B.7 (Nominal to Finite), Let do 1, and suppose Cer,s Ccert: 9cert > are abso- lute constants. Then there exist absolute constants C,,, C" O!M! > and absolute constants K, K" K" K" > such that for any 2 Kdg (g(nnCCC) any 1/2 S < and any 0 < 5
if L, > K' mmxxxx2 CCagem d, xe xx
1/3
K" max 252/6 604 log L, 2/5
Cx
and if
N1/22-0) > K C. /7 (1 + Pmnax) 17/6 119((3))
5/4 5/2424 log L,
min FA00 (/2) 00 (M_)
and if additionally there exists g E 12 (M) satisfying
[1|[]] IIle, (M) Cerr Cqeas Vd T
Bcert Vd
|1 101113,0000 (M) S CeertPmin
and T > 0 is chosen such that
d
T S
nL
30

---

Published as a conference paper at ICLR 2021
then one has generalization in Ly (M):
TP [s 13 (M))
C'C vdlog
5 Le-od
2
L
117
and in addition, one has generalization in (M))
I (~( x C" (1 Pmax) 1122 24/138) d3/4 log 4/3 L
C Le
min ( (M-)) 11/4-3166
12
The constant Cp max (Pains Pmin)-
Proof. The proof controls the _ norm of the error evaluated along the finite sample dynamics using an interpolation inequality for Lipschitz functions on an interval (Lemma B.14), which relates the - norm to a certain combination of the predictor's Lipschitz constant and its 2 norm. We can control these two quantities at time zero using our measure concentration results: to control them for larger times 0 the generalization error of the finite sample predictor and the Lipschitz constant of the finite sample predictor, and use the fact that kT not large argue by induction that not much blow-up can occur. Along the way, control the generalization error of the finite sample predictor by linking it to the generalization error of the nominal predictor as controlled in Lemma B.6; the residual that arises shown to be small by applying Corollary B.1 and applying basic results from optimal transport theory adapted to our setting, encapsulated in Lemmas B.13 B.16
S L (2/((T),, we set up system of coupled 'discrete integral equations' for
To begin, we will lay out the probabilistic bounds will rely for simplifications, that the rest of the proof can proceed without interruption. We will want to satisfy
1
(B.23)
max
O WuN
L2 (M 1
بااا~ (1) (M)
following the notation Lemma triangle inequality, we have for
10 Using Jensen's inequality, the Schwarz inequality, and the
1/1 θ x()))()9(()) du (x')
|| p 10 11,,, (M)
=
sup
lall ep (1)
13 (M,
10(x,x')
sup (MMM
a (0 x¹) 1/1 o / (n. x')l
(B.24)
+ (uP( |1 L(x, x )
where the notation 2) follows the definition in Appendix C.2.2 The first term in (B24 can be controlled using Theorem B.2 we obtain that on an event of probability at least 1
| wh o 211 S Vd3n13
(B.25)
d > Kdo log(nn)CMm and n L' The second term in (B.24) be controlled using the triangle inequality, Lemma E.3 and the definition of the we obtain that it is no larger than nL/2. Combining these bounds, have event of probability at least -cod
max |10 (lee (M)- (M)) 1 pm 112, (M)) < CnL
(B.26)
provided d 2 Kd Log(nnoCM) and > K'd Thus, with probability at least 1 e-d our choice of step size 7 S c/(n)) satisfies (B.23p. Under our hypotheses on the function g in the statement of the result and taking union bound with the event in (B.26). we can invoke Lemma B.6 to obtain
|KF (1 (M)) 5 C'Oyter Vdlog L
C. Le ced
IP
|1
> 1
(B.27)
nke
117
CVd/(nnrpoon Menet ((nr)
31

---

Published as a conference paper at ICLR 2021
and
k Σ ==
C"d L
C Le
ll 12 (M) S Cpeen
2
(B.28)
11T
OskSLa/(ne
provided d 2 dgg Cn)(nn)Cm) 1/2 = 1. L'd and L 2 K" C" d. We have by
Lemmas B.6 and B.10, a union bound with (B.26). and our condition on T that
and
{ |1c lre s svae
CLe
IP
11e S vajon
n
21
12T
OSkSL2/(rr)
(B.29)
as long as d 2 do log(nno and n 2 L log" L, and where we used our conditions
on and q to obtain that L9/nr 2 1 and simplify the probability bound: and, following the notation of Corollary [B.11] we have by this result (again under our condition on and union bound) that there is an event of probability at least CLe -d/ ((T) on which
( S (n 28884 d log" L) 1/12
(B.30)
under the previous conditions on and d. addition, applying Lemma D.12 and union bound gives that on an event of probability at least 1 - Ce-od
max [|<t 1ham || 1lam S Va
(B.31)
provided d > Koo (m and n 2 K' max L, 11/322//3 x2/53. Finally, we have by Lemma B.13 that for any < 5
((x) dx) (x)) - Im J- ((x) (
}
P
2 Se (B.32)
SELip(M)
N1/(2431
as long d 1 and N 2vd/ min (M,, (MM
We let E(g. denote the event
consisting of the union the events appearing in the bounds bound and the previous observation that L /// 1, we have
B.25 to (B.2 hold; by a union
C'Le
P 2
In the sequel, will use the events defining E to simplify our residuals without explicitly referenc- ing that our bounds hold only on E to save time.
We start from the dynamics update equations given by Lemma B.8 which
write c Ci = (Id -T) [ -h + |---1] - 7 [<
where + is defined as in Lemma B. 12 Under the choice of and positivity of θ (Lemma B.9 we apply the triangle inequality and telescoping series with the common initial conditions to obtain
||<F - CR (11)
<
Σ
ON [cN1 - o [<c | (M0)
(B.33)
0=0
We can write
N (x) / (2, (x') dx
I'm (ON (x, x') 1b <(x, x')) C:" (x))))
+ + 21 / (x')dp" NN (x'),
and analogously
+ [<. (x) = =/ (0(z,x') 31 L(x, c.: (z') dp
+ I tha ( CC
M
32

---

Published as a conference paper at ICLR 2021
Using Jensen's 's inequality and the schwaaziiequallin we have
1/ (0, - 1/ <(x, x')) C, (a')
(x)) du (a')
L220 (ML)
_ / 10 (,,x') - 1/1 ( x ) riw (an) (2x') (xx)) dx(((())) (x")
M
x |0 - 1 o < L b (Mx 115." I1 (M)))
x |0N 1 o 4 [ (M))
since N² is a probability measure. Repeating an analogous calculation with μ for the other term and applying the triangle inequality, we have
||5:0 - -Ma (m)
| [s:]] - θ [5: (M) S $100 - 1/1 o <1le~(MXM
+||s l1 (M) +||5." 12 (
+ +1100" - - - -= Is leeg,
+ *1/ wi o <( x' 2' (xd) (d') du (z'))
1, (M)
(B.34)
We detour briefly to simplify residuals appearing in (B.34) before using the result to update (B.33)
Using B.25 and (B.30), get
||0 - 11 o ( (163- -5 lلllاy (M)) + 11550 11 Leg + 1|5." 1|s (mssss
+||0N - 0|1 =mx
(M)
Vd (1K38 - co2 11 ملاا (M) + 11<5 1e (M) + ||<." llenccm
((n"1 L 188 log" L) 1/12 ||55"1 (155ll2excc 1
G.N Illll (2o) + 11: 16 (1)))))) (M) + 2 11 log 1)1 [1:30
x
(B.35)
where the final bound holds when 2 d Using (.29), we can further simplify the RHS of the last bound above to
(n 11 18 l log L) 1/12 (I550 - --55 llllee (M) + 1155 11,2 (M) + +2||5h
L,3 legn(m))
2 48480 log" L) 1/12 + (n I 484480 & log L) 1/12
||:00 - -500 &o"ll1z (10)
With this last bound and B.34p, we can use kt S T9/n simplify B.33) to
c( 100420111100 log" 1/12
||520 Si 18lllg (M)) 5
+r (n T,148848 log 1.) 1/12 k
E C3 (M))
2 o 20-,,,00 x x' (ae') (du)(x))) dr dz'))²))
L, (M)
(B.36)
To control the remaining term in BB36 we split the error < into a Lipschitz component whose evolution is governed by the nominal kernel v/1 / and nonsmooth component which is small in
33

---

Published as a conference paper at ICLR 2021
L Formally, we define @om (M) + (M) by
nom [g] (a) = / vi Lxx, x )g(e")) du (x'),
and use the update equation from Lemma B.8
write
52
[s]]]
nom [<
nonm O [sM
82
so that C. = -.N. + 5% and So Lip = Ç. 6 It is straightforward to control 8 L we have (as usual) by the triangle inequality, Jensen's inequality, and the Schwarz inequality
| (
1//1 ((. (-. (m (f (z') du
==0
8-1
ST
1/2 < 6, ( L (M)
(M))
and then the triangle inequality together with (B.55). B.29 and B.30 yield
(M)) S sTVd
(Vd'n133 (n 1 I, 184 a log" I.) 2/122
sTV L8+ log" L)
1/12
(B.37)
where the second line applies the same simplifications that led us to (B35). The triangle inequality gives
1/ v1 <(- 2')8N (x') (dd) dp (x'))
L/ (M))
1/,, 2 <(,, x')0 (z') du' x0)
Σ
13 (M)
and simplifying as usual using Jensen's inequality and the Hölder inequality, obtain
1/ 1/1 <1 x 2') (2') (d dy (x=))
$ 5 (1
M
(M)
S ST (n 00 L 8q 25 log L) 1/12
where the last bound uses (B.7).. Then using the triangle inequality and kr S 1 to simplify in B.36). we obtain
L 604432 15 log L
1/12
15 Si" [11 SC
9 1) 1/12 A-1 El( S.N 3."ll12 I1 (M)
(n 22 1 S log
1/ o ((. N. (d') - -dp -dp (x'))
1/ Nz (M)
(B.38)
To simplify the remaining term in B.38 we aim to apply (B.2 ; to do this we will need to justify the notation and establish that C. E Lip(M) regardless of the random sample from 8 and the
34

---

Published as a conference paper at ICLR 2021
random instance of the weights. Because 53 T. is sum of functions, bound its minimal Lipschitz constant by the sum of bounds on the Lipschitz constants each summand. We always have for either {+.-}
N. M.
c M. Lip
+
Σ
J
vi 00
x x'){ (x') d
(a')
(B.39)
Lip
2=0
Lip
We note that because the ReLU
[.] is 1-Lipschitz as map on R", we have
|sh.. 11.pp W 12 IIll < +00,
t=1
so we need only develop Lipschitz property for the summands the second term of BB.39, To do this, we will start by showing that + 1/1 cos" (r,, (4), x') is absolutely continuous for each æ'. Continuity is immediate. The only obstruction to differentiability comes from the inverse cosine, which fails to be differentiable at =1, and because M C 3 have (y+ (t), x') =1 only (t) =x'; because 7/- are simple curves, this shows that there are at most two points of nondifferentiabiliy in [0, [cn(M)) At points of differentiability, we calculate using the chain rule the derivative
(1) V1- x²)2
ト (v/ cos co (t),
x
and because Y, is sphere curve, it holds (1=)(())(()))) (t) for all t, whence by Cauchy-Schwarz
(1), 2:'2 x )
1I (t)) (1(1)) )l
{())
V1 xx'))
7. (t) (t) )a'll2 < 1,
(B.40)
(1) x;12
where we also used that y, are unit-speed curves. In particular, the derivative is bounded, hence integrable on [0, len.M))] and an application of (Cohn) 2013 Theorem 6.3.11) establishes that t + th cos cos-t((r (1), æ²) is absolutely continuous, with the expansion
1/1 cos" cos' (t,(t), - th o cos" 1os-'(10 (tt,,,,
(rx
- (v/ o cos' (x, (Tx") (t)),))
(t")
dt"
(Yr (t"), x0)
which gives an avenue to establish Lipschitz estimates for & ei cos (t) Because x' + Ci (x') is continuous and i < < k < L9/(nr) < +oo, an application of Fubini's theorem enables us also use this result to obtain Lipschitz estimates for the summands examined in (B.39). to wit
1//m 2 o <( ,a)) (2') a4> (x')
Lip
sup weM. I Iw (xx, az)l (a!)) (x'))
sup ([n(s ((xx 21)) (1) dn ap))
1/2
$ "|G 1e
(B.41)
WEM.
after using the bound (.40 in the first inequality and the Schwarz inequality for the second. Before note that the C22 property of th, continuity of C.V. bound-
proceeding with further simplifications, edness of i, and compactness of M let assert using [.41] and (B.39) that <N.Lip e Lip( whether or not we are working on the event E. Continuing, we develop a bound for the RHS of [B4T) that is valid E. Using the triangle inequality and the Minkowski inequality, have for
35

---

Published as a conference paper at ICLR 2021
the second term on the RHS of the last bound in (B.41)
(Jn (wi (1b) ( ((x, du (x')) 1/2
sup #EM.
(1ha (w' <(x,)'1)) (d'x
(x') du d2)))))) 1/2
I sup weM.
(B.42)
(N (00 2(xx du (a')
1/2
sup w.M.
For the first term in B.42). use Lemmas C.7 C.22 and C.24 to obtain that æ' + (v/ <(x, x²))²
bounded by Cn² and O'n² L"n2LS-Lipschhee Lippschitz for every æ. and then applying (B.32) gives
(0- (w' <(x, x)) sup sEM,
(1/
du dx)((()) )
1/2
1/2
CnLLAVd N
N1/(246)
(1 + Ca.o
CC- N1/(4+26)
nL 5/2 1/4
(B.43)
For the second term (B.42) we apply Lemmas C.8 and C.22 get
together with the choice L 2 K
(M (1) 1/2
du (x))
1/2
sup MEM.
I CnL sup =EM+
(
I CnL3/2 p (len(M+) + len(M-)))
S Cppla G2/2 n m/
(B.44)
Combining (B.43 and (B.44) to control the RHS of (B.42). obtain from (B.4]
1/4 1/ (( x' x')ci" (x'') dx'"(a))) (ax)
Lip
< <c||s* =CIs:"/leg (11g (1 CN 11/2 N1/(4426))
nL nI.5/22 + Pauxx Cwe MMnI332
< <||ss -cllsiille,,, Ilee (1 ++C Ca= 1/2 7/6 + Pmax) 1/2 //2
(B.45)
where in the second line we used N > 4426 Plugging B.45 into B.39) and applying in addition we get
B.3I).
( N.pp
M. w..l
S <v 7/6 (1+ Cp= (1 + Pinax) 1/2 1/2
1/2
1/
3/2
Ells
(M)
1=0
(B.46)
Let us briefly pause to reorient ourselves. We do not have control of the empirical losses appearing (B46)) by an outside result, so we need to make some further simplifications to this bound. We will control the sum of empirical losses term in (B.46) by linking it to the difference population error, which we last saw in (B.88.. and the population error using the triangle inequality and a change of measure inequality. Mcanwhile, with the Lipschitz property of Ç. CN.Lip will be able to obtain bound in terms of simpler quantities for the last term on the RHS of (B.38) using B.32). The resulting bounds will give system of two coupled 'discrete integral equations for the difference population error and the Lipschitz constants of C.N. Lip which we will solve inductively.
have shown,
First, we continue simplifying (.46).. measure give
The triangle inequality and the fact that INN is probability
||</|
||
1. N. (M)
s:
N.Lip
12 (M) +
(B.47)
36

---

Published as a conference paper at ICLR 2021
and we have by the triangle inequality and Hölder- continuity of 1 VI
1 (
$ 1|5 (
||s
- -|(s)iiiio)
+
u (M)
(M)
LRN N (M)
12 (M)
+ VV (())L)(())) (duro()) du (z)).
$ s1
(B.48)
(M))
We have shown that Si N.Iip Lip(M) and Si N.I.P E (M) above, and so (s: E Lip(M well, with
(GNLAD)
<:-ht
N.Lip
Ci
N.Ipp
(M)
M. I.
M.
T.ip
Applying the previous equation with
(B.32
to control
B 4888
we get
<: N.N.L. Lip
2. (M))
L.
12000 (M)
1 (ex +
1o0 (M) max.El-,-1
L.
+ -aav
M. Lip
N17(0-6)
|
M.Lpp
(M)
7160 100 mollss
1/2
Lo (My mas.cti.-)
c
L.p
1/2
+Cdl/1
(M0
M.
21
where the second line applies the Minkowski inequality. Using the triangle inequality and that 2 is probability measure, we have
Lip
$ ||<:"1 12 (Ma)) + 1|o.51 11
11 (M)
(M)
1 11 12 (M) + ||5t" (200 1,,000 (M)
+1oo8 L=x)
(B.49)
Substituting B.49 into B.47 and using (B37)) to simplify gives
1| 12
< || - -c.0
+ 1|
+K<1911; (M) + 2ir Vd (n)""" 884 log" 1,) 1/12
(
No.M (.omM|sMaam NN..ip 1/2
N.Lipp
1/2
L*(M) max max.e/+.-)
+ C/1/4
(M)
M. Lip
N1/(1426)
(B.50)
Following
B.46
we need to sum the previous bound over i. To simplify residuals, we use
B.28
get
C& (8 log 1.) 1)
1/12
Σ
11
1
Cz O'd L
Cs2rv/l (n 48 log 1. 1/12
+
nt
20 C'd L
nt
37

---

Published as a conference paper at ICLR 2021
where the second bound uses the control ST < kT I L9/n and holds under the condition n 2 (C/C")12
( L8 Summing (B.50) and using the previous bound, it follows
SEE 11:; 122 (M)
CCpa dlog²
L
1-1
Σ
-G; -c7 (2p
+
nt
1=0
e/66 / NN 1/2
1/2
(.V.Lip))
max.e
}
51 N.Ip
+ //
1a (M)
(M(M)
M.
Lip
VN
N1/(4+26)
(B.51)
Plugging
B.5l into (B.66). we obtain
N Lip
M. Lip
d dlog L + NT Ela coel 11 (M) (M)
s 4/d//4/312
L 20000
||c, M 20 ( max
N.Lip
1- +nTd//4
M. Lin
2i=
VN
N17(4423
(B.52)
where for concision we have defined
C, = CC 24/5 (1 (o (1 Pmax) )
(B.53)
and simplified the v residual in (B.46) by worst-casing with the larger residual from the population error term in [B.5T). and made other simplifications by worst-casing some constants. We simplify
B.38) next: we have shown that N.N.Lip E Lip(M) and C.N.ip e L (M) above, and for every w M, we have
1 L(x, NNLi Lip(M)
as well, with
|/ o ((x, N... M. I
CnI. max <e(+.-)
(NN.Li
M. 11ap
C'nI? ( N.N.pp
(M)
(B.54)
using the definition of wh, Lemmas E.5 and
C.22 and
|/ L(x
)g.
N.ip
(M)))) I CnLll(sN.Lip (M))
(B.55)
The bounds
B.54 and
B.55
retain no z dependence. Applying BB32 and integrating over æ, we
obtain from
B.54 and B.55
1/ vn <0- S5
N. (x)) (d)) (x'))
L L2
COLVdl<s. L>(M)
N
CnLel4/EC C, MVd www.mmaaxee..... (+.-)
N.ppp
M.
Lip
N1/(2+))
d. N.Lip
(M
+
N1/(2+5)
38

---

Published as a conference paper at ICLR 2021
and we combine the first and third terms the RHS of the previous bound by worst-casing, giving
M/ 1 a1),
(d - du (x')
12 (M)
OvdnLe (wOeMM
N.N.ip
M.
wull1an Tip
+ L bmmmn))
max +E(+,-)
N1/(2+8)
Plugging the previous bound into (B.88) we obtain
1 - (1
L 1,00+324d 225 log" L
1/12
1/12 k-1 Σlls - S/ 12l1 (M)
+ (n 1 B. log L.)
s=0
CTvdnLe (1 M)
M. 11.
+ -
+
N1/(2+8)
max +E(+,-)
(B.56)
To finish coupling BB55 and (B.56). we need to remove the (M) terms. We accomplish this
using Lemma B.14 which gives
|| Lip
(M) S 1/0//11/550
(M(MM) (M)
+
Pinin
1/3
C
||<." N
Lip
2/3 2 (M)
max Eet,--)
N.Ipp
M.
Lip
1/3
(B.57)
where we have defined
C2(
Pmaxx
(B.58)
Pmin min w (M+), 1 (MM))
For coupling purposes, it will suffice to use version of (B.57) obtained by simplifying with some coarse estimates. Using B B.49). (B.37) and B.29 we have
N.Li Si
Vd ir (n 88 log" L) 1/12 + || -2--0 -220 (0
(M)
<2dd+ || - 17 (11
using iT < L9/n and n 2 48 20 log L in the second line, and plugging this into B.57 and using the Minkowski inequality gives
||5 N.N.pp (M) 5 sCC,// Vd CC
-
47 17 +
Cdl/3
1/3 max Pmin
N.Nipp
1/3
+E(+.-
M.
Lip
1/3
0:0
2/3
12 (o (M)
||550 | +e(t--)
max
M. Lip
Pmin
(B.59)
To make some of the subsequent bounds more concise, we introduce additional notation
NN..ipp
A, = max
+
M.
Lip
39

---

Published as a conference paper at ICLR 2021
Plugging (B59 into (B.52) and using the Minkowski inequality, obtain
C22
1 C2 22/2
A, S CC/d dlogg I. +
Σ |s - G
Gil11
+ noT
VN
VN
nTd 8-1
Cy nTd
e-1 22/(442))
1/3
1/2
VN
P Pmin k=0
1m
5/12
//d
2/3
1
Ci 2/3
(MD) A, 1/3
Pmin N1/(4+26)
1/6
1/3
1=0
Pmin
==0
2//4 n/
11
c:
A./2 (MD)
1/2 1
21/(4+26)
T/d
1|5
1:
(M)))))
2/3
1?/3
1/6 N1/(1+26) Pmin
(B.60)
To simplify (B.60), we use ST S L9/n, C2 2 1, and q S 1, and so if additionally we choose
N 2 C2 max Vd. ] we obtain
A. S CC/d dlog L + Σlli - -270 12000 (M)
2=0
nTd1/1
7/12 s-1
||
-5
2/3
(M))
A2/3
A
1/3
1/6 NI/(1+28)
12 (M)
1/3
VN
Puin'
Pmin
1=0
C/44 1/2
s-1 21)(4+26)
1512
1/2
2
1/6 N1/(4+28)
Puin
nrd//4 8-1
11 N
5:0
2/3
2/3
1, (M) A
VN
Punin
==0
C//nndl/4 1/4 14
a-1
1:-
11 1/2
1/12
21/(4+26)
1=0
(B.61)
Meanwhile, we recall
Cya,M len(M+) (M + len(M-)
(M))
and an integration in coordinates gives
Mon(M-)
1 (M+)) = 0
P+ (+) dt 2 Pmin len(M=),
so that
2 Pmin
Cye,M 5
(B.62)
40

---

Published as a conference paper at ICLR 2021
Using B.62
and plugging (B.59) into (B566.. obtain
Sk
-1 (M)
L 200+320d d log L
1/12
0
n
11488 l log L) 1/12 C.C, VdnL2e14/6
k-1
+
Σ 520 -
Pmin P N2/(2+6)
C'VVdn
Ag + LC, Vd
Pmin \1/226)
+LdL/3 Priin
Punin
A//3 + pm
1/3 c00
co 1
2/3 1 (M)
1//3
(B.63)
In (B.61) I and (B.63), we now have a suitable system of coupled discrete integral equations for IICE &h 1 - (M) and Ag. We will solve these equations by positing bounds for each parameter that are valid for all indices 0 S k S [L4/(nT] based on inspection of (B.61) and (B.63), then proving the bounds hold by induction on Positing the bounds is not too hard, because each term in (B.61) and (B.63) with factor of N in its denominator can be forced to be small by requiring N to be large enough. For all 0 < k < L [3/(n=]],
I. we claim
Cip 1/22 2 CA/3 / log"
|| - C5 (1:3000)) (M) S Caitf max
L 1.000300 log" 1)
N1/1--1)
1/12
(B.64)
A S Cnp / log2 L,
(B.65)
where Caitr and Clip are two absolute constants that we will specify in our arguments below. We prove B.64 and (B.65) by induction on k. The case of k 0 is immediate, since C co" for B.64): and by construction Co Lip C. and [B.31 and d 1 then gives (B.65) if L 2 e. We therefore move to the induction step, assuming that (B.64) and (B.65) hold for k 1 and showing that this implies the bounds for k. We begin by verifying (B.64). Applying the induction hypothesis
for k - via (B.65), we can write
dAg
1/3
A₂ LC L
Pinin
s cd
/ / 3/2
log L LC Vd+
1/2
(ChpC1 Pmin
1/3
d 18/213/2 3/2 log log2/3 L
2/3
I Ci. 1/2 C.. /3/ log L.
where we worst-cased the second line using Cip and C1 2 1, C2 2 1, which follow from B.53) and B.58). We use kt < L9/n with the last bound to note that
C'TVdnLe 14/8 k-1
log? L CI C4/3 d ///41//24g log L
Punin MhiinN//10 (1/(2+5) Σ
11/(246)
where 0" 2 1. Using this bound and (B65 once more, we can simplify (B63) to
|< si (f'lleg
/2 ///////4"3"///////2111 log L
I, I,60422d d log" I
1/12
S CC" C2/2 2/2 CA13
+ + N1/(2+))
(n. 1844888 log L) L 1/12, C² VdnL2el4/6
k-1
(B.66)
Σ C.N
Pmin Mnin.V// 11/(2+))
n/5/ log
2/3
k-1
Σ E1(00
<2N
2/3 (M)
4/3 Pmin
N1/(2+6)
(M))
==0
41

---

Published as a conference paper at ICLR 2021
Noticing that the RHS of the bound (B.64) does not depend on k, let us momentarily denote it by Cai (i.e., the part of the RHS of this bound that does not involve Cair is denoted as M). Plugging into (666)) and using kt s L /n, we obtain
||< - Sh 1111 Cipp C d 1.5///1/19900 L
1,00+32 log" L
1/12
+ +
N1/(266
(
[ [ af log" L
1/12
C'C2' VAL22Ag222
Caitf
M
Pmin N1/(2+))
2/22/3 c/4/6 1 log 2/3 1og dift
M2/3
(1/(2
Puin
In particular, if Can 6 max (C, C") (for the constants in the first line of the previous bound), we can bound the RHS of the previous bound and obtain
M
(Tr:s--00m loo" T.
1/12
C'22/22 Var M
||<200 56 13 (M) S
11
+ Can
3
n
Pmin N1/(20))
C'C C 22/32/22 /11/125//249 log
log2/3
+
diff lip
M2/3
4/3
N1/(245)
Pauin
(B.67) We can conclude (B.64) from (B.67) provided we can show the second and third terms are larger than C/.M/3. For the second term in B.67). if we choose N such that
N2/(2+5)
6C'C2 1/2 p
P e 11/8 22 1.249
and n such that
2 g log L
then we have
L L88 o log" L
1/12
C/C./2 VaL
Cair
Can
M
n
Pmin N1/(2+6)
3
For the third term in (B.67). , we proceed in cases: first, when
Ci /2/2 ca/o /4 12/22- L
L 604324 log L
1/12
(B.68)
(1/(2+))
we have by (B.64)
M 6660+322 log"
1/12
7 and if we require additionally Cait 2 1, follows that
C'2 C// C 14/8 1 15/2+4 1og
2/3 2233 M2/3
2/3
diff
A/3
N1/(2+6)
Pmii
d 14/8
15/2t log²
C' C'Can Ci C
1/2
14/3
N1/(2+8)
< C'Cune 14/8 1
using C > 1, 2 1, and Cip 2 1 and worst-casing exponents on d and log L in the first line, and B.68 in the second line. In particular, by the value of M in this regime, if
2 (3C"e11/6/18 60+322 d5 log" L
then we obtain for the third term in (B67])
C' 2//2/3 C1/3 c1/3 11/6 211/12
log 2/3 I. (1/(2+))
Can
M2/33 <
4/3
3
P Pmiu
42

---

Published as a conference paper at ICLR 2021
as desired. Next, consider the remaining
C2 C/13 /2// 1,/22-0//00000 L
1,00+320 16 log" L.
1/12
(B.69)
N1/(26)
which by (B644 implies
M = Cin CA// 1//22 log L
N1/(2))
With this setting of M, the third term (B.67) can be bounded as
C' Chp C c/4/6
1/ log 2/3
22/3
4/3
N2/(2+6)
Puin
(//8//440 /3/41113 1,/21///,,//2 log" T.
C' Cip C1C2 1/3
C4/348/9 11/8 C
diff
2 Caann 24/8 24/8 /1/3
N2/(6436) M,
and using the RHS of the final bound in the previous expression, we see that we choose
22/(2+)) 2 (3C")3/2 22/2 /2/2 5/249,
then have for the case B.69
C' Ch 113 uip
11/3 61166 (2//22///+0 log 2/3 L
Cai
22/3
A/3
N1/(2+6)
3
Punn
Combining the bounds on the third term in (B.67) over both cases BB68 and
B.69
we have shown
|| c
(M) S Can
1.g (M)
which proves (B.64). Next, to verify B.65). we proceed with a similar idea: the bound claimed in (B.65) corresponds constant multiple of the first term in parentheses in (B.61), to establish (.65)) suffices to show that each of the other terms in (B.61) is no larger than a certain constant. To work with the maximum operation in (B.64), we will again split the analysis into two cases. First, we consider the case where (B.69) holds, so that the maximum in (B.64) is achieved by the second argument. Plugging (B64) and (B.65) into (B.6l) and using Kt S L9/n, we get
dlog² L Ca C4/3 d / 1,5/2+2 log" T. N1/(+66
A S 1/4 3/22
1/6 / 1116+44//3 log²
C1/3 Cip 111/18
C.C.
N6/(1045))
/249 1og 1/ L
1/2 0//20//2 C//2 9/8 L// 13/4+0 log
d//0//8 11/(4+)))
C /1 01/3
+
+
VN
/ L 4/3
2/6 2/3
NI/(4+25)
d 13/645b// log L (15+6)
Ca
C 5/3
2/3
Cip C C 1/2 ca/s 4/442++//2 log"
C1/2 diff
N1/(245)
1, and C2 > 1, we can worst-case constants in the previous expression to simplify. We can then do some selective worst-casing of the exponents on d, N, and in all except the first term: we have evidently (to combine the first and last terms)
Using Cip > 1, >
T 3,234/22 log² L
1// log² L
N1/(2+6)
N1/(2+6)
43

---

Published as a conference paper at ICLR 2021
and (to combine the first and second terms)
//3 L 11/0+4//3 log" L
/2/2 log? L N1/(2+6)
N6/(10+36)
and because < à < 1, we have 1/(2- - 6) I 1/2 and (5+ + +0) 26) 2 1. and N 2 d this
implies (to combine the first and second-to-last terms)
1 L log" L
dT/4 5//+22 log L
N(6+6)/(4426)
N1/(2+6)
We can worst-case the remaining three terms, and we thus obtain
As S LC dlog² + Ci C/ 1//4//1,5///2/2 log
1714466
+30 C C2/4
lip C2 C// 1 21 1og
N
We can then pick Ciip 3C, and if
1//(4426) 2 3(3)) 1, 149
and
N1/2+6)) 2 12CCCnnCCC 1/2
///3/////////////////////// // 1//2420
then it follows from the previous bound
A 3CC 251 log L.,
which establishes (B.65) in the first case, where B.69) holds. Next, we consider the remaining case where (B.68) holds, so that the maximum in [B.64) is saturated by the first argument. We start by grouping some terms in (B66)) that it will be slightly easier simplify later: write
A S 14 3/2 dlog L nt Σ
6,
n/d//4
||5.5 -3 (3 (2) d'/0 A2/3
1/6 N1/(1+26) Pmin
(B.70)
ntdl/4
||s.5 - -501 1112 12.00 + dd/3 A!/3
1/3
Pmin
C 1/4
k-1
[| - c50 l1 (i (M) + d1/4 A!/2)
21/(4428)
By the case-defining condition B.68 and B.64). enforcing
n > 112 60 60-324 d log" 9 L
implies
||s." -00 l1e (M) (M) dl/2 1/2
and we can use this simplify (B.70), obtaining
5/12
As S CC 1,1//22 dlogg L + NT Σ ElIc. 6.00 (1)
2//3
Pmin N1/(1+28)
(B.71)
2nT of /12 k-1
2C//4nTdd// N1/(1428)
2/2
k-1
1/3
Σ
1/2
1/3
Pmin
==0
8=0
44

---

Published as a conference paper at ICLR 2021
Plugging B.64 and B.65 into (B.71) and using KT L9/n and BB68 get the bound
L 100+4 log" L
1/12
As (Ci 13/2 dlog² L + Cai
n
20 2/3 C. 2/13 0 g1 d L+a 2/3
1og N1/(4+26)
(B.72)
dI, /T,2/2t9 1/g2/3 I. C
20
up
VN
d (/1
20 c /
//
NI/(4426)
From B.72
that if we choose such that n 2 (2Cait) 6004440
and we choose N such that
/ > 16C c C2 22/2 /2/
then (B722 implies the bound
A < 3CC // log" L.
which agrees with the previous choice Cip 3C and thus proves (.65 in the remaining case of B.7I). By induction, then, we have proved that (B.64) and B.65 hold for each index s f [*/[n]]]
We can now wrap up the proof: we will obtain the desired conclusion by plugging the results we have developed into (B.57) and simplifying. Plugging (B.37). (B.27) and (B.64) into (B.49) and bounding the maximum by the sum, we get
< ( L2 (M)) +
N (1Lx/(ar)) - -Sii 12 (MD)
+ +110% +100Ee//oonllmmmmmm
Cogea Vollog (((La/(ar)]
(100+320d' log" 1/12
+ +C"
+C"C,02 11/2 C
A//3
// Ing
N1/(248)
CC valog L
L60432ad
log L
1/12
+C"CCh C4/3 (4 N1/(2+5)
log? L
+0
La
n
CC /d
(B.73)
L
where in the third inequality apply [Lz((nT] [2/(2n)), which follows from choice of step size, and in the fourth inequality we simplify residuals using n 2 /C" 2 1604444 and 12/(225) > C² Cp /5/4 log I.. Applying B.73), the triangle inequality (with B.37) and the fact that e is probability measure) and our previous choice of large n, we get
CC vd
( I
(B.74)
(M)
La
ie. generalization in - (M). We can bootstrap generalization in (M) from (.73) using the triangle inequality and B.57): get
( (M(
$ sCOL/2
N. ((19/(ar)1
C
2/3
(M) (Lo/(or)) + +1000nntnrull (-(
A
1/3
17 (M)
1/3
Pmin
CC2120 Cyeur VdlogL
C/C//3 /3/4 (3 (3-14(4)))))) 49)/6
log
4/3
+
1,9
Pmin
1/3
min
Pminn
1/3
45

---

Published as a conference paper at ICLR 2021
where in the second line apply (B.37) and previous choice of large to absorb the residual from & [[r//(rr] and apply (B.65) to bound the A 1/3
[2*/(wr)) term. Worst-casing the errors in the
previous bound, we obtain
834 log 4/3 L L (4g (44-33) 3)/6
( (m)))) S (Cp + +
To conclude, will tally dependencies and make some simplifications to show the conditions stated in the result suffice. Recalling (B.53) and (B.58) and using (B.62), we have
C S CC CCpaamtt 2gamet (1 + Pmax) ) 1/2 24/5
so we can simplify
1/2
+CC2+2 13 (1 + (max)) g/6 (4/((36) p // + 0///0 CC
Pmax
min (M+ 1 (M-)3
CC CC++2 22 /3 (1 + Pmax) 112 g2/24/32)))
min (100 (M+), (1 (M²))
We can use this to obtain a simplified generalization in L [M bound from our previous expression: becomes
CCP 2Roan /3 (1 + Pmaax) 1/2 2/2222/2220))) d8/4 log 4/3 L
/ II (M)) min (MM), 11 (MM
(B.75)
L(44-3)/6
which can be made nonvacuous when q > 3/4. Tallying dependencies, we find after worst-casing (and using q > 1/2 and some interdependences between parameters to simplify) that it suffices to choose N such that
N1/(2+)) 2 22/1/////2/// 21/6 8// 10///115// //52+22 log L.
21/6
the depth such that
L M C C2 d, x C²),
the width n such that
1/3
2 C max
252/6 1,04400 1 log" L, 2/5
and d such that d Cdo Cm Unpacking the constants in the condition on N, we that it suffices to choose N such that
CC2/2+ (3 + Pmmax) 7766 (7/6/119/(3)) 119/(38)
/ 2
min (1 (M-)) d 5/2422 log L.
B.3 AUXILIARY RESULTS
Lemma B.8. Defining kernel
@ (z, (2,x') = S.' (Vio (x'), Virr - -er (o2) dt
and corresponding operator L22N (M)
O (2 (x0', /
+ No)((
(xd),
M
have that ON is bounded, and
Sh41 (Nd- [<ta]
46

---

Published as a conference paper at ICLR 2021
Proof. By the definition of the gradient iteration, have that
Ck Sox - Sop-
The total number of trainable parameters in the network is M = n(n((L - 1) + no + 1), and the euclidean space in which 0 lies is isomorphic RM,, For & E No. define paths T [0, RM by
Th (1) 00 - LL (0
so that
c5 = (1 f-)((0)
We will justify a first-order Taylor representation in integral form based on the previous expression by arguing that for every æ E M, t + faz (e) (x) is absolutely continuous on [0, 1]. by checking
the hypotheses of (Cohn 2013 Theorem 6.3.11). Because 7 is smooth and f(- (2x) continuous, f-2'(0) (t) is also continuous. Continuity of the features as a function of the parameters and of 7i
implies that for every 2 0, the image of [0, 1] under the map
t +
(1) (1) (x)
compact. By repeated application of Lemma E.21 we conclude that t + (n (o) is differen-
tiable at all but countably many points of [0, 1]. Following the proof of Lemma E.21 we see that the points of nondifferentiabiliyy of t faz"" (e) (x) are contained in the set of points of [0, 1] where
(a:) vanishes. Applying the chain rule at points differentiability of the ReLU [-]+ and assigning otherwise, follows that the derivative of ( +> (1)()) at l E j0, 1] is equal to
there exists a layer é at which at least one of the coordinates of o (.)
-7 ( (0 (1)
all but countably many points E [0, 1]. We finally need check integrability of this derivative on [0, 1]. We have by linearity
LW N (0 (x - ((((0(0)))=--
-//////
Son Soy (2 (2e) (a)), F ( (B.76)
and by definition
(V V5o (1)
= = ab (a') + Σ a xh co (x), agy (2))(( (B9 (x)
(x), (6) (xy))
By construction of the network, the feature maps (t,, x)) + a (1) (x) are continuous. For the back-
ward feature maps, we can write for any 0 (W) ....., w(+1) and any 02 = (W2 ........ WL+1) using Cauchy-Schwar
(10% (o),)))
II e=(+1
W +1
W
and the RHS of this bound is continuous function of (0.)). Because our domain of interest [0, 1] x M is compact, we have from the triangle inequality, the previous bound on the backward feature inner products and the Weierstrass theorem
K ( ( (o')( +00,
sup 1e.
(B.77)
so that particular, we can bound our expression for the derivative of + fo using the triangle inequality as
T ( Vlggo(()) s Cr /// (o) (2') (a²)
47

---

Published as a conference paper at ICLR 2021
for some constant C 0. The RHS of the previous bound does not depend on t, so by application
of (Cohn 2013, Theorem 6.3.11), follows that + fa (z) is absolutely continuous, and we have the representation
(x) - (2) (FC (froo)))) dt.
Using (.76), we can express this as
(o V ( dyc (xt) dt.
(z) (z) - (x)=
To conclude, it will be convenient to switch the order of integration appearing in the previous pression. Applying (B.77). we have
Son (zx) ( for (x'), V f (c) (z)) - c S0x (z')
and the RHS of this bound is integrable over [0, 1] x M because the network is continuous function of the input. By Fubini's theorem, it follows
S (x:) = - Jm ( (x'),
at cor dz)
(B.78)
Defining
ON (x. x') = [. [( (a'), ( di
and using B.77
define bounded operators ON Lax (M) LaN (M) by
[g(())) = /
(a'),
and with this definition. (B788) becomes
SNN Gr = = [
claimed.
Lemma B.9. For any network parameters 0, define kernels
Og(x, x') = fe(x'), (
for (0), define corresponding bounded operators on E2 (M) by
D (gl(z) = =sna Oo(a, x')g(()) (x').
For any settings of the parameters O, the operators Qo, self-adjoint, positive, and compact. In particular, they diagonalize in countable orthonormal basis of L (M) functions with corre- sponding nonnegative eigenvalues.
Proof. When + = N. an identification reduces the operators Og to operators on finite- dimensional vector spaces, and the claims follow immediately from general principles and the dinensoiinnsi spectral theorem. We therefore only work out the details for the case * oo. Boundedness follows from argument identical to the developed in the proof of Lemma [B. in particular to develop an estimate analogous to (B777. This estimate, together with separability and compactness of M, also establishes that O compact, by standard results for Hilbert- Schmidt operators write for any 92.92 (M)
$B). In addition, this estimate allows us to apply Fubini's 's theorem to
(g1, 10.00 (ge)) 12 (M)
Oo(x, x' (z) (2) d du (x') = (g2, Og [g1]}
48

---

Published as a conference paper at ICLR 2021
since (,(x,x²) = 0o(x', A similar caleulation establishes positivity: have for any g L (M)
(g. 00 12 (M)
(ill)) ((())((()))))))))))))))))))))))) ( (z)
(Vio(2'), Vfo(z)g(z) (z), /.x (z
dp))
where we applied Fubini's theorem and linearity of the integral. These facts and the spectral theorem for self-adjoint compact operators Hilbert space imply in particular that the operator O can be diagonalized in countable orthonormal basis of eigenfunctions (Ui)ieN corresponding nonnegative eigenvalues (Ai)ee C +00)
L (M) with
Lemma B.10. Write EN for the operator defined in Lemma with the parumeters O set to the initial random network weights and the measure set to p There exist absolute constants K, K' >> such that for any Z O and any d z Kog log log(nno if
12 un
(M) L (M)
and if in addition L84 d log" ! L, then one has
I|sailleswo s vaj 21 21-
2L9
e c-d
10
L5k5L2/(MF)
NNnom defined iteratively as Proof. Consider the nominal error evolution Sk
S6+1
N.nom
NN Nooom
T N
Ck
N Nnom
NNnom
So
for step size T > 0, which satisfies
@ L2xx (M)- (M)
We will prove the claim by showing that this auxiliary iteration is monotone decreasing the loss, and close enough to the gradient-like iteration of interest that we prove that the gradient-like iteration also retains a controlled loss. These dynamics satisfy the 'update equation"
G N,nom = (Id x,,)** [c].
Because M is compact and Ç continuous function of the input, we have E (M) for all values of the random weights. Because NN is a probability measure, this means has finite L (M) norm for every > 0. Mcanwhile, the choice of t and positivity of the operator (by Lemma B.9 guarantees
Td-rO ,M 11 (M) 1M) $
from which it follows from the update equation
N.om
L2x (M, & (M)) & I1lll
(B.79)
where the last inequality uses that V is probability measure. In particular, this nominal error evolution noninereasing in the relevant loss. Now, we recall the update equation for the finite- sample dynamies
<i (Id |
which follows from Lemma B.8 Subtracting and rearranging, this gives an update equation for the difference:
Sk+1
= (Id ,,xx
[st" Si
N,ao
(0 - -w,,) [CE
(B.80)
49

---

Published as a conference paper at ICLR 2021
Under hypothesis 7, (B.80) and the triangle inequality imply the bound
S
N.nom
Sk+1
L2 L2N (M))
Sk
Using Jensen's ss inequality and the Schwarz inequality, we have N
(M)) (MM)))
/ 10 (,,)) - 0(
-0(-, (ur) la(az) d (x' (a')
sup
19ll oN (M)
|00 θ (lL
sup 1 (M)
< 1 ZM
since w is probability measure. Defining
A
|| - -
max (o.........k
by telescoping series and the identical initial conditions, thus obtain
|
N.nom Sk-1
STAR
1 N l1
(m)
2=
and the triangle inequality and (B.79) then yield
k || Ii, (M) S I1 (M) + T
| Σ
Il
(M))
1=0
Using discrete version of (the standard) Gronwall's inequality, the previous bound implies
k-1
h-1
|| l1ee cun) $ [[]l + +|Slll
I exp
Σ 1=++
1=0
|1 (1+ AT exp (KTA
(B.81)
To conclude, we will use Lemma F.5 and an inductive argument based on B.8I). that by Lemma D.II (with rescaling of d, which worsens the absolute constants), we have
Let us first observe
Vd
I [S]
2 l od
(B.82)
2
as long as n 2 K&t and d 2 K'do log(nnoCMz). Define events & by
EN - =<1K3>
(1st"llegwam) Vdl vd
where d > 0 is sufficiently large to satisfy the conditions on d given above. We are interested in controlling the probability of U &K for l E No. We can write
Tk-
P
E U Li=0
P
N
(EN)
1=0
and unraveling, we obtain
i-1
P
E3
=
P
N
(gN)
K=0
2=0
1=0
50

---

Published as a conference paper at ICLR 2021
In words, it is enough to control the sum of the measures of the parts of EN that common with the part of the space where none of the past events occurs. First, note that (882) implies
P P Se
and so assume > below. For any 9 > O. kT S L9/n, 2 KL d and d Z gives that there are events BN that respectively contain the sets ( CT, /// log L). and which satisfy in addition
K' d'dg log(nnoCMM) Lemma F.5
1-1
B o n (c)))
Se
==0
We thus have by this last bound, partition, and intersection monotonicity
i-1
P
E n
n (eM)
S S od + +P(eN n (BN)),
==
and by construction, one has A S CL4+24/3 11/12 log 3/4 on (B,)). Another partition and B.82 give
[kll - P n (B:) se-d + 3 n (BN)
vd
2
When the two events on the RHS of the last bound are active, we can obtain using
B.81
| (M)
Vd (1
3/4 Lexp 23/ 11/12 log 2/1 1.))
Given that KT S L 12,n, we have
212 log L 1)
1/12
KT 0/4, 11/12 3/1 log
$1/e,
n
where the last bound holds provided n 2 K8,48+20g d log L. Thus, on the event
vd 2
(BN) n [6ll <
we have and thus
|s (Ellera S VVd
vd Na n (BN) n [KllLoRE S 2
}
= 0.
By our previous reductions, we conclude
i-1 n
P
E
n
(EN))
2e
==
and in particular
P
E 1=
< (2k + ced
-cd
The claim is then established by taking l large as L /(T)).
51

---

Published as a conference paper at ICLR 2021
Corollary B.11. Write ② for the operator defined in Lemma B.9 with the parameters 0 set to the initial random network weights On and the measure set to Jc" and define for No
A
max (o. R
|| - -
There exist absolute constants C, C² K 0 such that for any q 2 0 and any d 2 K Kdo log(nngCMM) if
|uNN 13 (M) (M)
and if in addition n 2 1"1,184200 d log" L, then one has on an event of probability at least (1(1+L//(nr)
A (1 Clog
2 11/12
Proof. Use Lemma t. remove the hypothesis about boundedness of the errors from Lemma FS then apply this result together with a union bound.
Lemma B.12. Write @ for the operator defined in Lemma B.9 with the parameters O set to the initial random network weights and the measure set to μ Consider the (population) nominal error evolution &% defined iteratively as
RR+1 Ck - - (5h So = C
for step size T > 0, which satisfies
o
(M)
Then for any E 13 (M) and any k satisfying
3e
|gll 1, (M)
Kt 2 2 (
we have
31 17000 (M)
(13 3 lg L2 (M)
115 152ll112 (1,3000)
[3]|O[g] - -Sll23 (M)
log
(V2 ||
KT
Proof. The dynamics satisfy the 'update equation"
G = (Id (Id-r0)*
Because M is compact and is continuous function of the input, we have ( (M) for all values the random weights. Because 1 probability measure, this means Ç has finite L. - (M) norm for every p > 0. Using the eigendecomposstiin of @ as developed in Lemma B.9
we can therefore write
20
<=
(uss 5) (M)
the sense of I2 (M). Because θ and Id - - diagonalize simultaneously, we obtain
11 1123 1. (M)
(1- (vn, 12 (M) (M)
2kT (vi, S) (M)
i=1
where the inequality follows from the elementary estimate 1 - S - for x 2 O and our choice of r, which guarantees that T'; > for all E N so that the elementary estimate is valid after squaring. We can split this last sum into two parts: for any À E R, we have
|| 1 (M)
Σ AAA,ZA
-kT (vs, 4) (m) (M)
Σ
-2kTd, (vi, L..))
AAAAAA
52

---

Published as a conference paper at ICLR 2021
Because ② is positive, have further that X; 0 for all i, can take 2 0. The first sum consists of large eigenvalues: we use exp(-2krX}) I exp(-2ktA) to preserve their effect, and then upper bound the remainder of the sum by the squared _ norm of The second sum consists of small eigenvalues: replace use bilinearity, elff-adjinntness of @ and the triangle inequality to get
s and then plug in θ [g] (o[g) () and
111. (m)
(KUi.0)),..0 (mD)
(us, [
1 (m0)
We then square both (nonnegative) sides of the incquality and use Cauchy-Schwen to replace the squared sum with the sum of squares times a constant, obtaining
||< 1 I1i (M) Se -k-kk> IKII; (M) + 32 (M) + 3 [gg] -Clliz (M)
after re- rddadding indices i to the sum to obtain the third residual. We will choose X 2 0 to minimize the sum of the first and second terms. Differentiating and setting to zero gives the critical point equation
(N)) = (2tA)e2RTA,
3
||91|
22 (M)
which can be inverted to give the unique critical point
|1 Li (M) (M)) (kT))
2
A
W
2KT
3
1 (M) (M)
where W is the Lambert W function, defined as the principal branch of the inverse of 1-> ze"; we know that this critical point a minimizer because the function of À we differentiated diverges as A + cc. Plugging this point into the sum of the first two terms gives
||<291 N (M) S [ <11i2 (MM
||.11 (M)
[Ksl 2 (Mn) (KT)) (1T) (3/2) [[ L (M)
I|ll 122 (M) (kTT) (3/2)
1 w 1+
+ (KT)
W
(M)
For 2 the function Wxx) is strictly increasing, the inverse of y ye ; by definition W (e) 1; and we have the representation W(z) log (z) = log = (Corless et al. 1996), whence
((x) log x if e. Because n R0 is probability measure, have
| 12 (M) (M) S 11
and therefore if
3e | (
kr 2
2
[
can simplify the previous bound
91lg|1 12
( l1gl12 folliz (M)
(M)
1 11 S 33/0 - Cl (M) +
log"
4(kT)²
2 |[
[| (m)) (kT))
using also properties of the logarithm. Taking square roots and using the Minkowski inequality then yields
1999l1. (M) log
(v2 lg Iا (M))
I 11 S v3| - <II (M)
KT
2 ||(|| [|[||
where we used the previous lower bound on kr to determine the sign that the absolute value of the logarithm takes. This gives the claim.
53

---

Published as a conference paper at ICLR 2021
Lemma B.13 (Kantorovich- Rubinstein Duality). Let Lip(M) denote the class of functions f M + R such that both f' M,, are Lipschitz with respect to the Riemannian distances M For any d 2 L. any 0 < 8 S 1 and any N 2 2/dj min (M+), 1 (MM)) one has that event of probability at least
Se-a, simultaneously for all f E Lip(M)
1 ((x) du (x)
SM (())
e/4/6 Cu .M V mwx.maxeeeee
(
N
N1/(2+5)
where
len (M+) (M,
len (M-) (ML)
CHN-M-
Proof. The proof is an application of the Kantorovich-Rubine Wasserstein distance (Weed Bach 2019 eq. (1)), which states that for any two Borel probability measures on M+. one has
duality theorem for the 1-
sup / d(x)) IIfI M2
-S f(z) du(x)) M
W (1GD)=
where M denotes either of M_ or M_, and I-llup is the minimal Lipschitz constant with respect to the Riemannian distance on M+. Therefore for any f M_ + R Lipschitz, we have
U (1))))
S.y f(x) s Il (1, m
(B.83)
where one checks separately the case where I| = 0 to see that this bound holds there as well. To go from (B.8)) the desired conclusion, we need to pass from the measures μrth and μ both supported on M. to measures 11 (with E {N, b. supported on the manifolds M_ (which we will define detail below); the challenge here is that the number of 'hits' of each manifold M that show up in the finite sample measure yo is random variable, which requires small detour to control. Let us define random variables N4. N_ by
N = NN (M4);
N_ = NN (M-),
so that N+ have support in {0, 10. NJ. and N4 + N_ N. Define in addition
p+ = 1r (M+);
p- μ 00 (M-),
which represent the degree of imbalance between the positive and negative classes in the data. By definition of the i.i.d. sample, we have that N4, ~ Binom(N, p.). Using N_. and N_,, we can define "conditional" finite sample measures μ and N by
Σ # [1 :
6
Σ
[(:))
max (1, ]
i
so that (N+NN) + (N- NNNN) N when N4 E (0,N), in which case exactly one is probability measure. By the triangle inequality, we have for any continuous f M R
B and μ and n are both probability measures except
1L f(x)
/ f(x) (1))duc( (x)
Σ
/ f(x) f(x) (x)
p.
N.
N J f(x)
P.
*E(+-)
N, p +++ +1 f(x)) P-
d2)((x))
/ f(x) du dp) (a)
(1 +E(+-)
Σ
N p+
(B.84)
Herr treat the empty the appropriate 'zero element' of the space of finite signed Borel measures Ma, namely the trivial measure that assigns zero every Borel subset of M
54

---

Published as a conference paper at ICLR 2021
By Lemma G.1
we have
N.
Vd
2 2e 2dd 2d
(B.85)
N
N
Using that N N2 = N_ and 1
p+ = p-, the bound (B.85 implies if N 2 2/d/ min mpn,p+
P+
N. 1
p+
P
2 2e
(B.86)
N
Now fix an arbitrary Lip(M). For either E (+.-), we can write
M. SM f(x) duy (x:) ma N
1
Σ f(x))=
Σ 1 M.M. (z:).
max [1, 1, z....))
K.
and since MM and M- separated by positive distance A O, have that m; 1 E.M, continuous functions on M. Since f is continuous on M by the same reasoning and the fact that M is compact, follows that the functions (xI..... ØN) + SM, f(x) du,," (a: are continuous on M x M as well, and in particular for any the sets
{ M.
(x)
S. dp) >
f(x)
dp,
P+
are open in M, and so is their union over all f Lip(M). By conditioning, we can then apply (.86 to write
P
U
U.S.A.
f(x)
du
S. d(x)
zt]
SELip(M)
De
[N(1--)//2
{
SM. f(x) (xF(() -JM. ((x) did (ax)
< 2e 2d +
Σ
I
U SeLip(M)
>t
N. = k P[N. = k).
k= [Pp.22
(B.87)
Conditioned on {N. = k} with 0 < <<<,, N. the measure n is distributed as an empirical measure of sample size from the probability measure μ [N(1 p2)/21, any > 0 and any d 2 we have for both possible values of *
pL/p.. /p. supported M., For [Np./2] s s
Vide31/0 len(M.)
Vide446 len(M.)
k1((2+0)
([ Np.
1/(248)
v 24/ (en(M.)
(Np.)
1/(2)) 1/(2+0)
and so an application of Lemma thu6 gives for any 0 < 8 S and any d 2 2
du (x)
Vdel4/8 len(MM)
P
W
du.")
N, = k < ad
(Np.) 1112+))
P.
Combining this last bound with (B.83)
and (B.87 gives
SM. f(x) du (2))
JM. (x))
P
U fELip(M)
P. vaclaro|
sl
S 3d
Ii len(M.)
N1/(11)
P.
where we used max (p+.p3} < to remove the exponent of 1/(2 + 5) on these terms. Taking max over the Lipschitz constants and combining this bound with (.84 and (B.85) and union bound, we obtain
IImm (2) f(x) f(x) dj (m:)
P
U
< 8e-d
SELip(M)
N1/(24))
where the constant defined as in the statement of the lemma.
55

---

Published as a conference paper at ICLR 2021
Lemma B.14. Let dg = 1. There is absolute constant C 0 such that for any function f M with fl M2 Lipschitz with respect to the Riemannian distances on M one has
2122 I1 (M0)
Punn (min (n (M+ ) (M-)))1/22
|f I C Cmmxx
[I L 1,00 (MD max.g(+.)) 1
1//3
Lip
173
Proof. For any T' > 0 and a nonconstant Lipschitz function f :0, + R. we will establish the inequality
|| C Cax
[| VT
2/3 1/3
(B.88)
where the constant C > 0 is absolute. We can this result to establish the claim. We start by writing
||/||
max +E(+.-)
and for * E {+,-}, we have
| A0
=
(B.89)
where Y (M..) - M. the smooth unit-speed curves parameterized with respect to are length parameterizing the manifolds. Similarly, the curves' parameterization with respect to are length implies
If o 1oo%lllp Iapp s $1
(B.90)
Applying (B.88) with B.89) and (B.90). we obtain
1l |,-- a0
s Cax
Il 1 Vlen(M..)
1lf 121///3
fl..
1/3 Lip
For the first term in the max, we have
If 7a 12.
leM
Mea(M.)
1
fo
len(M.)
leu(M.)
Ya()) lea[M.]
foya(t)² P+ (t)
dt
P. (t)
(lm((M.)
+
(on_
using the triangle inequality. For the second term in the last bound, we note that
/mme (mmlM.))
for,(t)3p dt)
)oa()4+)
8oy+(1))pr p+ o 7 (1) plau(M))
<
p (7)
< 11/1112 (1)))
(B.91)
56

---

Published as a conference paper at ICLR 2021
and for the first term, have
P. Y.()) leu[M..]
E [e.
P+ 07)
Pr (+(t) (1)
p.o
(M.)
(M(M.) len(M.)
max rE (e(MM)
Px 0 Ya())
(M.)
Pinax
1 (M+)
1100 (M+) Panin
2Pmax
(B.92)
1 (M.)Pmi
where in the first inequality used the triangle inequality, and for the second used that 0, in- tegrates to n² (M.) over [0, len(MM.),, )., which implies that there exists at least one t E [0, (en(M, (M.) which P. %. (t) 2 yea (M.)/ Ie.,.... so that the maximum of the difference in the second term on the RHS of the first inequality is bounded by the maximum of the density term. Thus, by Hölder's inequality and (B.9 and (.92 we have
len(M.)
soy(11 o 7+(1) (t)
Men de (t(())
3pmax
|1 22 (M)
Pmin min {pro (M+), nx
Similarly, for the second term in the max, we have
[I o 1211233
(11 (en(M.)
1/3
(en(M-)
1/3
(0)= de
(1)- dt
Mon(M1)
len(M }
1/3
<
f foy4(t) (t)
1/3
Pmin
|| 1.3.. (M)
1/3 Puin
Thus, we have
1/2 fl 12,00 (M)
1/3
1/3
| S max
12
(Mi)
ráp
Piin
(min (eE² (M+), p (M (M-
Prim
and taking maximum over * E (+. -} establishes the claim.
To prove B.88 consider first the trivial case where ||
here the LHS and RHS of (B.88)
are identical, and the proof immediate. When || || > 0, the Weierstrass theorem implies that
there exists [0, 7' such that [f() []ff] we consider the case sign(f(t)) 0. For any
t' [0, 17, we can write by the Lipschitz property
f(t) ||fl|z= || -
and the RHS of the previous bound is nonnegative on the intersection of the interval [| I| 1+ [| [ with the domain [0, 7] (with standard extended-valued arithmetic
57

---

Published as a conference paper at ICLR 2021
conventions when |
0). This gives the bound
minu
(+
||
(11f
-1flllllllpppp t²1)² ae
max
T31
1
.0}
min
1/ 11
TT -T-
(lffl [1fllcpplt()) dt',
max
1/ Irliiap
-
where the second line follows from the changes of variables 1' - 1' + L. The integrand on the
RHS the second line in the previous bound is even-symmetric, and max ||
-1)
117
min || / so we can discard one side of the interval integration to get
11.0000 T-1] TJ
min
(1 - -1/
(B.93)
1f Millii
-t]
max
11 1/71
min
(I1fll>> llfll dt'.
(B.94)
We proceed analyzing two distinct cases. First, if llfllew I maxft, +} Lip, then we must have [ 0; integrating the RHS of (B.94). obtain
1I 2
fl
or equivalently
|lfle 131/3 13/1111110/111111111
2/3 I|
Lip
(B.95)
Next, consider the case |//| max(1, ftll/|/iip We split on two sub-cases: when [ = 0, integrating (.94 gives
1 2 1 m -3
(B.96)
2
where we used max(t, - t} 2 T/2. When || > 0, integrating (B94 gives
Iffli 311fIIT (159
(Ilfllz= Ilf max(t, T +3)²)
max(t,
t
11 (I1fll - max(t, -1))
T 6
(B.97)
where the second line uses standard algebraic identity, and the third line uses max(t, T- T/2 together with the definition of the case to get that [/ 1,00 - [|[lTap max (1, () > 0 in order to
discard all but the & = 0 summand. Combining B.97 and (B.96), we obtain for this case
/6]lflle
|
<
(B.98)
VT
and combining B.95 and (B98)) gives unconditionally
Vellflle
If S max
2/3
1/3
VT
which establishes B.88). For the case sftn(f(t)) < O, apply the preceding argument to -f to
conclude. See (Brezis 2011 Exercise 8.15) for a sketch of proof that leads more general versions of (B88)..
58

---

Published as a conference paper at ICLR 2021
Lemma B.15. For any p N. if 2 4r
then one has
C' loog""
if Cog'
Proof. We first give a proof for p 1, then build off this proof for the general case. Consider the function f(z) cz log z. We have f') c 1/z, which is nonnegative for every x Z 1/c, so in particular
increasing under this condition. By concavity of the logarithm, we
have log Z log(2/c) + (c < 2/c), whence
f(x) + cx/2 log(2/c).
The RHS of this bound is equal to zero at (log(2/)) 1), and
2
2
log
S 2e 3/2 -3/2
In particular, we have f()) 2 for every x 2 (2g(/))) Rearranging this bound, we can assert the desired conclusion that if C' 2 3, then C'log for every 2 2C log 2C. Equivalently, we have for such n that Cn logn S Next, we consider the case of p > We will show
C Jog'"1 n
S
"
under suitable conditions, Let us consider the choice = KC log" KC, where K >> 0 is a constant we will specify below. Consider the function f(x) Cx - log" x, which satisfies
f( ((=)=clo8"
log -1 (z) (p- - -oo""'(m)) (x))
x²
In particular, f is decreasing soon p S log (x:). Now, we can calculate
1 (
ploglog KO)" log KC
f (oCllog KC)
K
and by our result for the case p 1, have for all p 2
p log log KC
$ 1 if log KC 4p 4p.
log KC
This condition is satisfied for KC 2 (r, so we set K we obtain the above conclusion when C' 2 (4p)4r. Under these conditions, we then get
S(KClog" KC) 1.
Similarly, we have log"-'(K than because 4p 2 It follows that f(x) < 1 for every x 2 KCClog K KCC which completes the proof.
'KC' log KC) > loo"""o"(""" (P) (4p) log" (4p), (4p), which is larger
Lemma B.16 (Concentration of Empirical Measure Wasserstein Distance (Weed & Bach 2019
Let do = 1. For either E -}, let μ be a Borel probability measure on M., and write μ for the empirical measure corresponding to N i.i.d. samples from μ. Then for any d 2 and any 0 < 6 < 1, one has
P W( < /del (cn(MM)
21-c 2d
N1/(2+5)
where the 1- 1-Wasserstein distance is taken with respect to the Riemannian distance.
Proof. The proof is direct application of the results of (Weed & Bach 2019) on concentration of empirical measures in Wasserstein distance. For the duration of the proof, we will work on the metric space
(M,, (eM, dist (-)), i... the same metric space scaled to have unit diameter; we will then obtain the result in terms of the unscaled metric by the definition of the W-Wasserstein distance.
59

---

Published as a conference paper at ICLR 2021
Because do = 1 and M, can be given unit-speed curve parameterized with respect to are length, we have for any Borel S C [0, 1] and any ε > 0
NC(S) - E
where (S) denotes the s-covering number of S by closed balls in the rescaled metric. Following the notation of (Weed& Bach, 2019, $4.1), we then obtain for any
log inf (N) p(S) 2 c-(ss2)
de (m, (0,,="((= (=((=)))
3
log e
Invoking (Weed & Bach 2019 Proposition 5), we obtain after some simplifications of the constants that for any 0 ò (putting s ô in the previous estimates)
E[W (1,, " 31122=11(2t) 3 N S c24/6 N-///++1//00))
where the final inequality worst-cases constants for convenience. Using (Weed & Bach 2019 Proposition 20), we have
P W(a, ) E((We,,
N
S e
2d
N
and hence
Vde
221/8
d
P W(HN
P
N1/(200)
N1/(2+6)
d
<P W(H, E N)
Se-2d
N
if 1.
Lemma B.17. Let n, m E N. Let R" - I be 1-nonnegatively homogeneous, and suppose
there exist M, L such that
1. g-- [lall S M;
2. ga-1 T-Lipschitz
Then for any x. xx R2, one has
[F(x) - F(x')l22 $ (2L + +)]l= - x'll2:
so that F is (2L + M) -ipssschhit
Proof. For any numbers a, b 2 0 and any u, v E R.", one has by the triangle inequality ||au bull2 I minfallu - oll2 + la b|l|e|1,, bl|2u 21|2 la bb]lu||2)) Using an elementary property of the min and the max, we thus have
| -ou-belly bull2 min(a, |/ ø11a max [1/u]22 || - b/.
(B.99)
Now we proceed to show the claim. Noting that the case where both æ, x' are zero is trivial first consider the case where æ is nonzero and x' is zero. By nonnegative homogeneity, it suffices to proceed as
F(x) F(z'Il2 [[F())]2 ||||||
(1
x ||00||/
M| M x'1l2
to conclude; for the inequality used the boundedness assumption on F. Now fix æ. x e R" nonzero. The inequality (B.99) can be applied to get
F(xx) F(x')l22 = |100|/2 F
||00|122 11 F
||
< min [|
22
|| (12)
x
x
||00|| 1
||00
12
max
x
F
x
|1
||
x
x2'122
[00
| 11
60

---

Published as a conference paper at ICLR 2021
where in the inequality we also applied the l² triangle inequality. Using the assumed properties of F, we thus have
x²'
[F))) (22
L min
[|00 22,
ac
112)
.MM
x
11
|
ac
22
By classical inequality (e.g. proved in (E.15)
has
x²'
|
x'
122
|
x
21
max
||3|22
11
whence
[(())
F(x')l22 & (2L + M) - x x'112:
as was to be shown.
C SKELETON ANALYSIS AND CERTIFICATE CONSTRUCTION
In this section, we construct certificate y for the certificate problem (B. the context of a simple model geometry. We also collect technical estimates relevant to the analysis of the skeleton th. We point to Appendix AA.32) for summary the operator and function definitions relevant to the certificate problem that we will use below. We will use the notation
+ (g](2=) = S/x 1h (ax, x. ()('))))))) dic"(x))
in this section; call explicit attention to this notation to avoid confusion with the kernel é xhy 2 that we have defined in the main text for convenience of exposition.
C.1 CERTIFICATE CONSTRUCTION
To construct a certificate, suffices to solve the integral equation
( θ [g]]
(C.1)
for function g E L2 (M) and obtain estimates the norm of g. It useful to consider separately the contributions of integration over the class manifolds M in the action of the operator @: we can write for any g
x M vo 2'ggga (elleeddpf(ff (x') //x vi L x' (()))(" (x)),,
J.
+ [x)
and it then makes sense to further subdivide based whether the evaluation point lies in M., or M_, and to introduce the density explicitly by a change of variables. With slight abuse of notation, we will write dæ' to denote the Riemannian measure on M4 and M_, for concision. Because the kernel & L is symmetric, if we define an operator θ [.M - (22 by
+(+++]<=) So M. vh ((x (1)+ dz',
operator := - 122 by
(g- / 1 (xr, (ar" da',
and an operator θ (M² + (22 by
θ (g+))(²²)
b (xx x )g+ (x') da',
M
then we can write the certificate system
C.1
equivalently the 2 x 2 block operator equation
θ +
p+9+ !-
θ_
where write 2+ and p- for the restriction of the density P to M4 and M_, respectively, and where the adjoint operation is viewed as occurring with operators between L2(M,) and L (MMM) (both Hilbert spaces). We will make use of this notation in the sequel.
61

---

Published as a conference paper at ICLR 2021
C.1.1 Two CIRCLES
The two circles geometry a highly-symmetrie geometry where M+ and M_ are coaxial circles in the upper and lower hemispheres of S², each of radius 0 < r < 1. Here we note that since the skeleton & depends only on the angle between points of S the particular embedding of this geometry into - is irrelevant, and it is without loss of generality to consider the geometry in & once we have restricted ourselves to this configuration. We have unit-speed charts, for E [0, 2xr)
r cos sin t/r /1 ,²
cos t/r r sin sint/rr v1
(+))
7-(1)= =
which implies specific forms of the spherical distances
< (+ cos'1
22 cos
l'
+ (1 -1-))
(C.2)
and
(n2 cos
t-t t'
- (1 - )))
(())) cos"
(C.3)
with the analogous results for the remaining possible combinations of domains, by symmetry. Be- cause ? is piecewise constant on each connected component M, there are constants C+, C__ such
that C+ = on M and C- = ć on M_. The block-structured system we are interested in solving
then
C- C_ = _ 6_
θ
p+9+ p-
(C.4)
where subscripts used to denote the domain of each component of the certificate. The coordinate representations (C.2) and (C.3) show that each of the operators appearing in the 2 x 2 matrix in (C.4) invariant the circle; we can obtain some useful simplifications by identifying these operators
with their coordinate representations. Defining
(+) cos 11 (r² cos + (1 +²))
gr (t) cos (r² cost (1 r²))
and (self-adjoint) operators on 2n -erpeooii functions g by
A[gi(t) /o 2n
s o ft t')g(t') dt',
|00 2n
(t)(())
S gt (')g(t')dt dt',
by a change of coordinates, is equivalent solve the system
r C+ -
A
x
P+9+
(C.5)
x
A
p-
where we have identified p+ and p- with their coordinate representations, and with an abuse of notation used the same notation for the certificate in (C.4 We can use symmetry properties to determine
g+(t)= ग ft TT,
so for purposes of analysis we need only consider fr. Each the invariant operators in (C.5) diagonalizes in the Fourier basis, and because the target Ç is a piecewise constant function, we only need use the first Fourier coefficient, In other words, we solve this system by first inverting the invariant operator, which responds to only the constant component of the target, and then inverting the density multiplication operators. This approach is made precise in the following lemma.
Lemma C.1. There is an absolute constant K > such that if 2 max(K, (T/2) 12)-1
1 and r Z 2 then the system C.4 has a solution that satisfies
6a||<|| LM(M))
64
9
1. 00
no 1/2 1/2 1/2 P
62

---

Published as a conference paper at ICLR 2021
Proof. Following the discussion by CC.3) it is equivalent solve the system in the Fourier basis, with only the DC component. We thus start by solving the system
r FCc =
2 l vo f+(t) dt wo (t) di [
2 2 bo g)) de
G_
2 wo d( [
where G+ and G- are constants that we will show exist. This is a 2 x 2 system, and the matrix is symmetric, with minimum cigenvalue 2 f. (v - v o (t) dt. Using Lemma C.2 we have L 2 max K, (m/2)(1 - r2) 1/23 and r 2 2
J (bo - v (t) dt 2
32r
so the matrix is invertible, and by an operator norm bound on its inverse we have the regularity estimate
(G + G G2)
1/2
32
(c + c²) 1/2
nn
follows that the function
9+ g-
G
solves the system
C.4
We conclude
2t
G+
2x
(1
G_
) p- oz)
g+
P+ 04 (+)
++ (7+(1))
9
12,00
2
(c + C2).
xn
Punin
Taking square roots both sides of the expression resulting from the last inequality will give the claim, after simplify the expression
/ 02 + C². Since
0 C2 (= [=
we can conclude after adjusting constants.
Lemma C.2. There exists an absolute constant K > 0 such that if 2 ma (1/2) and if 31 one has
2
( 1 (t) [U.=]
32r
Proof. Write a, o fr o gr for brevity, which is nonnegative, by Lemma C.3 We consider the tangent line to the graph of at 0; by Lemma C.3 this line has the form t + o (0) tnrL(L 1)/4x, and its graph hits the horizontal axis (0) nL/2, we see that this point of intersection is no larger than 2 /rLr(L + 1), which can be made less than by choosing 2 K", where K > the absolute constant appearing in the convexity bound of Lemma C.3 and K' >
(0) + 1). Using that a, S
0 is an absolute constant. Under this condition, we
obtain using Lemma C.3
(t)()) (0) tnr L(L + 1)/4T,
and so
J (t) dt 2 [0.a]
(0/44o,(00)/nnI((--)) (o,(0) - ntr L(L + 1)/47) dt
2TT (0)2
Lr +1)
We have a0) nL/22 (bcos" - (2r² - 1)), and using the estimate of Lemma C.20 we get
((v) < nL 1 + Lv/2a
2 1 Lv/T
63

---

Published as a conference paper at ICLR 2021
Together with the estimate cos (2r² - 1) 2/1 - ,², obtain
nL 2
nL
LV p2
nL
or (0)
viccos (2)2 1)) 2
2L
where the final inequality requires the choice 1 π 2
r².
Thus, we have shown
m
(f) 2
[0.m]
64r
claimed.
Lemma C.3. There is an absolute constant < K S T/2 such that if L 2 3, one has for all E (0,
(i) vb o Jr - 3 o gr 2 0 on [0, m]; (ii) (ip o fr v gr gr) = LL + 1);
(iii) v/ o f. - s o gr is convex on [0, K].
Proof. this proof, we will make use of basic results on the skeleton v, namely Lemmas B.5
C.17 and C.18 without making explicit reference to them. Property (i) follows from the fact that is decreasing, cos is decreasing, and the definitions of fr and gr- We note that fr is smooth on
(0, m); to prove property (ii), it will suffice to show that fr admits right derivative at and and apply the chain rule. We have if 1 E (T,
72 sint
rsint
(00)
(y22 cos + (1 22)² V (cost 1) v1 cos f.
after some rearranging, and by periodicity and symmetry properties of fr, we have lim gf (t) lim (t) 0.
2
'/x
We Taylor expand sin cos in neighborhood of zero to evaluate the derivatives there. We have sin f. = -t- 16+666+0 and cos /2(1=1//2+((('))) by the binomial series, we have =2//(L++//4+0(+*) whence sin sint(1- (
cos t) 112 (1 co- and
22+
lim f2(1) =r. 8 5
Thus
fw - o g.) = 3 (0())((0)) nr LLL(L 1)
4x
For property (iii), now consider E [0, x when necessary. The chain rule gives (w fr s o gr)" I(V' fr) (1b gr)9" + 1(w" f.)(F;) (w" (g))))))) and we have if (T,
(* ra(1 cost) (cost (r² cost + (1 - (1-r2))
(1- (r2 cos (1 -2)) 3/2
after some rearranging of the numerator. We have 1 cost O, and so the estimate r2 cosss (1 r?) 2 cos (with equality only at t = 0) yields (t) < (with strict inequality if < t < σ). By
symmetry, this implies that of 0, and using that 4// < O, obtain
(w o fr - 1/ o 9. 2 (16" o f.)(f))² (8" o (o))
By symmetry, we have (t) = fT t) on [0, nT, and because fr is strictly concave we know as
well that fr. strictly decreasing: it follows that fe g also strictly decreasing, and its unique zero satisfies the equation
1 cost t
2 2 + cost)
+ cos t
2 2 1 cos t)
64

---

Published as a conference paper at ICLR 2021
Noting that t 7/2 satisfies this equation, conclude that f 2 gf on [0, 1/2], that on this interval we have
(+ fo s o g.)" (g/)² ((v) fr) (v gr))
By Lemma C.19 if L 2 3 there an absolute constant K > 0 such that iه < 0 on [0, K]. The previous bound then yields
o fr + gr)" 2 0,
as claimed.
C.2 AUXILIARY RESULTS
C.2.1 GEOMETRIC RESULTS
Lemma C.A. Let M be a complete Riemannian submanifold the the unit sphere She (with respect to the spherical metric induced by the euclidean metric on R with finitely many connected com- ponents K. If do 1, assume moreover that each connected component of M is a smooth regular curve. Then for every 0 & S 1, there a E-net for M in the euclidean metric || ||| having car- dinality no larger than (Cm/s))o, where CM 2 is constant depending only the diameters EMM, M, (x. x²) and, when dg 2 2. additionally on the extremal Ricci curvatures of M,. Moreover, these nets have the property that if .M is given, there is point in the net E within euclidean distance E of x such that a lies in the same connected component of M as x.
Proof. Consider a fixed connected component M; with E [K]. We write the Riemannian distance of M; dist M;; because M; is Riemannian submanifold of IR"o, have dist,, (at, y) > || - yll2 for every x; y in M Because Mi m: (2t, y) 2 || yll.. it suffices to estimate the
covering number in terms of the Riemannian distance. We will consider distinctly the cases do
and do 2 2, starting with do = 1.
we have assumed that M, are regular curves, so without loss of generality to assume they are moreover unit-speed curves parameterized by are length, with lengths (c,.M.). It follows that we can obtain an g-net for M, in terms of dist.M, having cardinality at most len (M,)/E when 0 < 8 < 1, and by the submanifold property these sets also constitute e-nets for M; in terms of the 12 distance. Covering each connected component M, in this way gives a e-net for M by taking the union of each connected component's 's net.
When do
When do 2 2, we make use of standard results relating the covering number to the curvature and di- ameter of M. Let diam(M,) sup w' E.M, distM, (æ, x'), and let Rie; denote the Ricci curvature tensor of M, (recall that we assume the metric on M the one induced by the euclidean metric). Then because M is compact, (1) [ diam(M;;
+oo; and (2) because Ric, is moreover
continuous, there are constants k > 0 such that Ric; 2 - (do 1)k for each E [K]. Applying
Lemma C.5 it follows that for any = > 0, there is a s-nct for M, in terms of dist,, with cardinality no larger than (e), where CM: S diam(M,)) diammM..)
Thus, for any i [K], any do and any 0 < S 1, we can conclude that there is e-net for M; in the euclidean metric having cardinality no larger than (CM,/E)do, where
(M)) 16 diam((M, diam(M))
do =
CM.
do 2.
Taking the union of these nets and applying Lemma G. 10 for simplicity, we conclude that for any do 2 and any & < 1. there is z-net for M in the euclidean metric having cardinality no larger than (C, where
1 DK len(Mi)
do = 1
Cm
1+ ( (M)) do 2 2.
The additional property claimed is satisfied by our construction of the nets.
Lemma C.5. Given l > and integer d 2 2. suppose that M is a d'dimensional complete Rie- mannian manifold with Ricci curvature tensor satisfying Rie -(d 1)k. Then for any r,e
65

---

Published as a conference paper at ICLR 2021
and any p E M, there exists E-met (measured in the Riemannian distance dist.M) of the metric ball fa E M dist M(p x) I r) with cardinality at most (CM/E)", where CM > is a constans depending only & and r.
Proof. The proof is essentially an application of Zhu 1997 Lemma 3.6) together with some cal- culations volumes of geodesic balls hyperbolic space that we record here for completeness, although they are classical. For any 0 and any p E M, write
(p) {æ E M dist, x) rr.
Fix E M and r,e 0. The hypotheses of the lemma make Zhu 1997 whence
Lemma 3.6) applicable,
vol(B*(2r)
inf card(S) S (,),
U Be('') p'Es
(ol(4()))))
where (S) denotes the cardinality of a set S, and for all > O, (E)( denotes the volume of geodesic ball of radius in the d-dimensional simply-conneeted hyperbolic space of constant sectional curvature -k k;; these spaces are homogeneous and isotropic so the base point does not matter (c.f. (Lee 2018 Proposition 3.9)). particular, we can calculate these volumes in any model of hyperbolic space and anchored at any base point; we choose the Poincaré ball model and the base point where the maximal unit-speed geodesics take the simple form
(1) A 1/2 tanh Vkt
2
for v E 8d and Lee 2018 coordinates, then get for any >
Theorem 3.7, Proposition 5.28). Integrating the volume form in
[
2/k
vol(BB(e)
dæ
(k-1// tanh 2k)/2))BBd
1/k
|| 2
= kd22
dæ
(tanh /ke/2jBd
where the second line changes coordinates + k- 1/2 expression, we get
Changing polar coordinates in the last
2
vol(B² - 4422 vol(s)-1)
dx,
[,,tanh vke/2]
x.2
and then changing coordinates - tanh x, obtain after applying several trigonometric identities
vol(B((E)) k vol(su-1)
2sinh (2x) dr
10,
= d vol(3d-1) /
sinha= sinh"=l(x) dx,
[0.
whence
v2r(
(0.2A sinh d- (x) dæ
(=1()) Ja.evE/a sinh (x) dr
We have bounds x < sinh r S xe for nonnegative which gives after integration
vol (2r))
Jo.2rvey xd old 1)x dr
vol( vol(B*(e/4)
Jjo.evkra xd dr d
(2r) Sooi1 -d arrkidd dx
(eV//4)²
16re2rv
The lower bound is implied by cosh x: 2 1; the upper bound follows from writing sinh 0.5e*
22 and using 2
66

---

Published as a conference paper at ICLR 2021
where in the second line change coordinates (2rvk)ær and then use L control of the (monotone increasing) integrand the second line to move to the expression the third line. [
Remark C.6. The constant Cм in Lemma C.5 can be sharpened if more is known about the cur- vature of M: if Ric O, the exponential dependence on curvature and diameter can be removed (intuitively, taking & > O "recovers" this from the proved result), and if Ric >> 0, the dependence on diameter can be completely removed using Myers' theorem (Zuu 1997
Theorem 3.4(1)).
Lemma C.7. For any a., x', Tr wt' in Sna-
has
<ax x - ( x')| - v 2 x 2'122 a E'||2|-
Proof. Writing (ax, x') = cos (ax, x') cos - (1 (1- x 1 consider the function f(z)
cos" (0s-1(1-1 - (1/2)x²) for x E 1- v21, which is differentiable except possibly at O. We calculate
fx) = V1 -1 202
x
sign x
and taking limits at 0 shows that f admits left and right derivatives on all I- V2). f is even- symmetric, so by checking values at 0 and v2 we conclude that [f'l s v2, which shows that f is v2-Lipschitz. The claim follows.
L
Lemma C.8. Let do = 1. Choose L so that L 2 CR where k. and Cx are respectively the curvature and global regularity constants defined in Section 2.1 and K, K' >> absolute constants. Then
du (xr))
CPmax(len(MMp + len(M-))
sup BEM+
M (1+ (Zx
where C is an absolute constant and M2 denotes either M or M.
Proof. Recall that 7 7- denote unit-speed curves parameterized with respect to are length whose images are M4, M_. For convenience, define g(v) = 1/(1 + Lv/T). We have
sup f (o(<(x, ((,xx)x)))) 2²))) du (x') =
/
sup = M
(C.6)
+ sup I (o(<(a, x')) (x)).
3EMai
First, we note that [g| is strictly decreasing. We claim that for any E M_. there is a E M. such that L(xx, x') S L(x,, x²) for all x' E MMM; is easy to see this is the case by choosing æ, to achieve the minimum in min [(x, x') and arguing by contradiction. By monotonicity of the integral, this implies
sup / (9(<(x,2²))) (x') s sup EEM+
/
2'))
(C.7)
xEM+ M+
M
and similarly for the term involving integration over M.. Therefore
I (o(<(x,x'))) du
sup *EM+M,
sup / x((<(x,x1))) dz (x)) sEM.J M
(C.8)
+ sup I (o(<(x, d/ #EM-M
and it suffices to analyze these two terms. We bound the first term, since the second can be bounded by an identical argument. By compactness, the supremum in this term is attained at some M,. Taking such that (1) = æ. we can write
sup / (()( du (xr)) S Pmax 10 s(((6()))()))) ds. M M+
(C.9)
67

---

Published as a conference paper at ICLR 2021
We split the interval [0, S+] into two disjoint sub-intervals [t.
/ and
[0, ..]]] \. [t. K2 K + K+/VT,, corresponding to "large scale" and "small scale" behavior,
where X is the global regularity constant defined in (A.2) If we now assume
VL. S then from
A.2
we obtain
1 VL
K^
(ax x') s
dist, di
VL
and hence
Kx
1
dist dis x')
+ L(x, x')
VL
VL
From the definition of it follows that
L+VI/T
VL
Since [g] a monotonically decreasing function we can bound the second integral in C..9 obtaining
(o(4()(s,, ds len(M+)C'/L.
(C.10)
+E(U,S+IVE 14
We next consider the remaining interval of integration in C..9 Defining
Kx
S min (+i
5 = min
S+ -
VL
and (s) L(7+ (t* s), Y+ (t)),, the integral of interest can be written as
/
((((7)(s) ds=
(s(v+
Pmax
Pmax
<E(0.S+]n[(-- "/ +
+
==0
(C.11)
(g(v) ds.
+
It will be sufficient to consider the first integral here since the second one can be bounded in an identical fashion. We aim show that the integral above not too large. This will be the case 2+ (s) stays very small for a large range of values of s. To show that this is does not occur, we will use our bounds on the curvature of M to bound (s) uniformly from below, which will in turn provide upper bound on the integral. We will require an application of Lemma C.9 which will N² k?
be applicable S m If 2
we have
K
S
K
It follows immediately that Lemma C.9 applies to any restriction of 74 of length no larger than z. Next define by Ý: [0, S goo unit-speed are of curvature and i(s) ((((())((s)))
We claim that
Vs [0, [0]
(s) 2 (s))
(C.12)
The proof is by contradiction. Assume there is some such that
(r) < ((r).
(C.13)
Now define by Yr [0, -0 restriction of 7+ such that (0, (t())), (s) (t" + s), by 7r an are with curvature x and the same endpoints as 7r, and by 7, a restriction of Ý with
len(ā,) len(
Note that ((9,(0), = D(r). However, an application Lemma C.9 gives
len(x,) & len(*+) len(fr.)
68

---

Published as a conference paper at ICLR 2021
where the second inequality is because 7r and 7r have identical curvature every point, and by assumption (C.13) the endpoints of Yr are a greater geodesic (and hence euclidean) distance from each other than the endpoints of 9, (which are distance 2+ (r) apart). This inequality contradicts
the equality above it, and we conclude that no such r exists, and (C.12) holds.
write for the first integral in We have that [g] is monotonically decreasing function, hence
st
(+))) ds =
/ (1(T())))) ds.
s=0
==0
We now bound this integral. Since is an are with curvature n, from the proof of Lemma C.3 have that D is concave, and since (00) = 0 we can write
we
((S
E(8) 2 S+ 8,
and since |g| is monotonically decreasing
st
B(st)
2
D(ST)
St
(o(((s)))2 ds S
9(
8)
ds
(g(s)))
ds
St
D(st)
St
BSST)
st
D(S
1+ Lū(S)\\
LŪ($t)
where we used the definition of 9. It remains show that S and Ø(St) close. Since is an are with curvature and length ST. if we additionally assume L 2 Kk²C2 for some K chosen so that [|]+(())-($T))) < TST <
xC 2VD
we obtain
2'
||<(0) - 11
sS
k: 1=/5700) 5($t)12
2 sin
-1
2
<||5(0) -(5$) + n² ||((0) - 11
<
x² |||(0)) ( St| < |5(0) 7($$) 11 v 4 (st)² 3 - n²2 K2 St
4
4 L
where in the first line we used sin (x) < x x for Since
15(0) - 7 112 - ((8)) { S
we obtain
n² K2 K2
(((t)) -
4 1
and hence
S
S
S
2 K2 st
1
K -
We now choose L KR² K? for some K, so that the above term is smaller than 2. We therefore have
8,
(s
for some C. We can bound the second integral in C.11 in an identical fashion. Combining this result with 1C. and recalling (C.9), we obtain
(o(<(x, x'))) du (ad) C' C'Pmax (ln((4)) + len(M_)))LL
sup m M
69

---

Published as a conference paper at ICLR 2021
for constant, which completes the proof.
Lemma C.9. Given a smooth, simple open curve in IK oength with unit-speed parametrization
Y [0, S] R" such that for some
1. 1 S
2. S S
define by 7 an are of any circle of radius 1 such that 5(0) = -(0), ((5) = 7(S), I then have
k.
S
Proof. This result is generalization of well known comparison theorem of Schur's higher dimensions following the proof in (Sullivan 2008), where we additionally specialize to the case where one of the curves is an arc.
Given a curve y satisfying the conditions of the lemma, we first consider an are 7 of circle of radius and length S, with unit-speed parametrization. At the midpoint of this are, the tangent vector ) is parallel to 7(5) 50)), hence
$ /
IIS(S) 7(0)l22
R(
|
Similarly, for the curve 7 have
5 ((1))
7' 2) f (t)ctt
I|T(S) >
Denoting the angle between tangent vectors (7'(a),7((b)) any smooth curve with unit-speed parametrization [t) for any & [0, S/2]
cos a(, b). we use the fact that for
2e s)ame 10'(t)) This gives
7' S 2 +t)
7 s
o' (t')dt'
cos
T--
{++t
008
(1')))))))
cos
1 (1 de
$ ++
g+
1 (1)l de - ( 5 s
2 0008
di!
= cos
+1)
where we have used monotonicity of cos over the relevant range which is ensured by assumption 2
and similar argument follows for E (0, -S/2.. Combining these inequalities gives
[|>(S) (0)1l2 2 IIF(()) - -(0)]22-
We have shown that, unsurprisingly. if the curvature of y is bounded and it is not too long, then the distance between its endpoints is greater than that of curve of equal length but larger curvature
namely the are ?.. We now consider the are ý defined in the lemma statement. If S Š, due to assumption this would imply
(S) - (0))l, > 11((5)~(00) x = -|/(S)) - -
contradicting the inequality proved above. It follows that S S.
1oFor any circle and choice of endpoints there will be two such arcs, and the last condition implies that choose the shorter of the two.
70

---

Published as a conference paper at ICLR 2021
C.2.2 ANALYSIS OF THE SKELETON
Notation. where
Define (0)) Id, and for E define po()) as the Ê-fold composition yp with itself,
( = cos" ((1 -r ( COS + - sin v)
is the heuristic angle evolution function. We will make use of basic properties of this function such as smoothness (established in Lemma E.55 below. In this section, we will study the skeleton
L-1
L-1
1 (V) - 12
Σ
cos &p
(v) (00
-
DE [0,
8=t
where we have not included the additive factor cos (v)) as it is easily removed along the lines of Theorem B.2
We define
L-1 (x) = II -
(1-00))((0)))
6= f --1,
/'=
so that
n L
()(
Σ cos up (0) (v)
(C.14)
We will also establish a convenient approximation to the skeleton. Define
(1))
Σ (v). E=
Lemma C.10 implies that vb is convex; less trivial obtain the same for th We will prove several estimates below for the terms g(f) and their derivatives that be used to immediately obtain useful estimates for ve and its derivatives.
Lemma C.10. For each f 0, 1,... L. the functions e(() are nonnegative, strictly increasing, and concave (positive and strictly concave (0, T)); if < L, the functions g(), are nonnegative, strictly decreasing, and convex (positive and strictly convex on (0, z)).
Proof. These claims consequence of some general facts for smooth functions that articulate here so that we can rely on them often in the sequel. First, we have for any smooth function f (0, m) R
=f (ʃ (f'o f ( of)
These equations show that if f 0, > 0. and I" < O, then f also satisfies these three properties. Lemma F.5 shows that şp satisfies these three properties on (0, v); conclude from the mean value theorem and simple induction the same for (( claimed. Meanwhile, if f. smooth real-valued functions on (0, π), we have
(fg) s'g g'S.
= f"" 9"
Thus, if ff and g are both positive, strictly decreasing, strictly convex functions (0, π), then fg also satisfies these three properties. Lemma E...iimppiss that 0 - - ((()) < on (0, T), and the first and second derivatives are scaled and negated versions of those of (0); we conclude by another induction that the same three properties apply to the functions {
Lemma C.11. There is an absolute constant C > O such that if L 2 12 and 2 L, then one has
Cn
1 - ~ I
L
71

---

Published as a conference paper at ICLR 2021
Proof. We have from the triangle inequality
cos (0) (v) -1(((())
11 2 z sup
=E
L-1
1 n
cos (v) -1 (1))
sup vE(
where we use Lemma C.10 to take g(t) outside the absolute value. Notice that (cos pp()) 1)((() < O, so to control the L norm of this term it suffices to bound it from below. We will show the monotonicity property
(cos (0) - 1) - (cos vf (f-1) L)g(E+1) 20. 20
(C.15)
from which it follows
| 2 S 2 vE(U.m]
nL
sup cos sp (L ()) -1/-
using also (v)) < 1. Since cos x 2 1 - (1/2)x², and since Lemma C.12 gives that glt-1)) <
C/(I--1 (and also estimates the constant), we have as soon as I, 2 C/v2
C2nL
11 V S
4(L 1)²
which gives the claim provided L >. and n 2 L. So conclude, we need only establish (C.15)
To this end, write the LHS of (C.15) as
(0)) (cos (0) - 11) g(e+2)
(cos (00) - 1)g (cos (((+1) 1) (e+1))) = (cos p0()) - cos ((+1)))))
to notice that it suffices to prove nonnegativity of the bracketed quantity. In addition, since / 2 0
and (pv) by Lemma E.5 we can instead prove the inequality
(cos x cos (())) (cos 1) 20
π
for all x E 10, Using the closed-form expression for cos (xx) in Lemma E.2 we can plug into the previous inequality and cancel to get the equivalent inequality
sin 2 0.
But this is immediate from the concavity estimate sin x S. x. and CC.55 is proved.
Lemma C.12. I No, has the "fluid" estimate for the angle evolution function
(0) S +lv
V
where is absolute constant. In particular, if L E N one has (( 1/cl.
Proof. The second claim follows from the first claim and + clv 2 clv, so we will focus on establishing the first estimate. The proof is by induction on E N, since the case of l 0 is immediate. By Lemma E.5 there is constant Ci > O such that (pv) S v(1 CIP), and using the
numerical inequality x z) S xx1 x)- valid for x 2 we get
p(v) I+cp'
(C.16)
which establishes the claim in the case l - 1. Assuming the claim holds for e - 1, we calculate
(v)
((0 S 1 ((p((()1(())) 1 C1 (1o
(ta 11
where the first inequality the relation x(1 + -2-1 - (1 +1+33 x)- to see that x + x(1 + f1x) increasing. Clearing denominators in the numerator and denominator of the RHS of this last bound, see that it is equal to v/(1 + LV/T), and the claim follows by induction.
C.16 and the second inequality uses the induction hypothesis and
72

---

Published as a conference paper at ICLR 2021
Lemma C.13. I E No, the iterated angle evolution function satisfies the estimate
1 (v) 2 LV/T
V
Proof. The proof is by induction on & E N. since the case E = 0 is immediate. The case E =
follows from Lemma C.14 Assuming the claim holds for e - 1, we calculate
(v)
(V) 1 Cole D(V/TT
1)p-
where the first inequality applies Lemma C.14, and the second uses the fact that the RHS of the bound in Lemma C. is strictly increasing and the induction hypothesis. Clearing denominators in the numerator and denominator of the RHS of this last bound, we see that it is equal to v/ (1++1v/TT) and the claim follows by induction.
Lemma C.14. It holds
(v)) 2
1 D//
Proof. After some rearranging using Lemma it suffices to prove
(1-22 COs v + sin
TV
I cos
(C.17)
πT +
Using Lemma E.5] we see that both the LHS and RHS of this bound are nonincreasing. We will prove the estimate in three stages, using "small angle", "large angle", and "intermediate angle" estimates of the quantities on both sides of (C.17). Since TV/( + v) E [0, 7/2). we can use standard estimates for cos to get RHS estimates
TV
TV
COS
2
1
(C.18)
+
2
+
and
TV
cos
2 + v
(C.19)
T + v
As for the LHS, we can obtain an estimate near V = π straightforward way. Transforming the domain by v it suffices to get estimates sin Using cos v 2 (1/2)²² and sin // S v, it follows that sin v/ - v cos v S (1/2)r². We conclude
cOS near = 0. then divide by n.
(1 (1-") cos V π
sin
v/3.
(C.20)
π
2
We will develop a second-order approximation the LHS near 0 for the small-angle estimates. The first, second, and third derivatives of the LHS are (1-1) (1-v/T) sinv, (1/T) sin (1 - v/T) cos v, x sin 1/, respectively. To bound the third derivative, we will use the estimate cos < 1 - v² /3 on [0, / /2. To prove this, note that Taylor's formula implies the bound cos v < v² /3 on 00, cos 12 (2/3)]: because cos is concave on [0, 2/2), we also have the tangent line bound (v) < 5535 + (2/3 + cos ( on [0, π 7/2]. We can then solve for the zeros of the quadratic polynomial 1 - u²/ + (v5/3)u (2/3 + 55 coss (2/3)/3); a numerical evaluation shows that both roots are real and outside the interval |cos ' (2/3), n 1/2). Since the tangent line touches the graph of cos at v = cos (2/3), this proves that cos v < 1 v² on [0, π 7/2]. We can therefore write
and (2/ ②)) cos v + (1 2/4))
cos + (TT v) sin v S 2(1 -2/3) -( V E T/2..
The RHS of this inequality is a concave quadratic; we calculate its maximum analytically as 2 + 3x² /20. Meanwhile, if v E [m/2, π]. we have cos v/ < 0, and (IT - 2) sin v S x/2. We conclude
that (2/T) cos + 1/1 sin v < 2 2 2/0 on [0, x]. Writing an estimate
1/(3m) + 4400, this implies
cos v + sin
<
+ 3
(C.21)
2
73

---

Published as a conference paper at ICLR 2021
Finally, we will need some estimates for interpolating the small and large angle regimes. We note that the second derivative (1/TT) sin - -11 1//T) cos v of the LHS of (C.17) is nonnegative T 7/2, because cos 2 0 here; meanwhile, the third derivative (2/MT) cos (1 v/T) sin of the LHS of (C.17) is nonnegative if 0 < v S 7/2, since cos 2 0 here, and it follows that the second derivative is increasing on //2-- Checking numerically that the value of the second derivative at 1.42 is positive, we conclude that the LHS of (C.17) is convex on 1.42, T.. In addition, we use calculus evaluate the first and second derivative of the RHS of (C.18) as -// /(T + 2) and -T² 2v)/(m + v)", respectively; this shows that the RHS of (C.18) is convex for v 2 x. /22, and
concave for v S x/2. Taking a tangent line to the graph of the RHS of (C.18 at и 7/2, follows that the function
1 ((T
x x/2
g(x)
(C.22)
(4T + (1++//54)
x x n/2
a concave lower bound for the RHS of (C.18) on [].
We proceed to using the estimates developed in the previous paragraph to prove CC.17) argue that for in a neighborhood of 0, we have
We first
v² /2 + cv S (T² /22 / v)²
which will in turn prove (C.17) in the same neighborhood. Cancelling and rearranging, it is equiva- lent to show
(2/T n - 2c) - (4c/ - -
(2c/7²) 2 0.
The LHS is a concave quadratic, with value 2/TH 2c > 0 at O; we calculate its two distinct roots numerically as lying in the intervals [-5.1, -5] and [1.42, 1.43), respectively. It follows that (C.17) holds for [0, 1.4].. Next, we argue that for in neighborhood of r, we have
2T (T ~)
which will in turn prove C.17 in the same neighborhood. Transforming with +> V and rearranging, it is equivalent to show v²(2T- v) I 4T²² in neighborhood of O. The LHS of this last inequality 0 at and nonnegative on [0, T]: its first and second derivatives are 2(4T 3v) and 4x - 6v, respectively, which shows that it is strictly increasing function of on [0, x]. Verifying numerically the three distinct real roots of v3 2T +1= and transferring the result back via another transformation v +
V, we conclude that C.177 holds on i 1.1, v.. To obtain that C7177) holds on [1.42, π - 1.1]. we use that the function g defined in (C222) is a concave lower bound for the RHS of (C.18) so that it suffices to show that the LHS of (C.17) is upper bounded by g on [1.42, ; - 1.1]. The LHS of (C.17) is convex on [1.42, n), so it follows that it is sufficient to show that the values of the LHS of (C.17) 1.42 and at π 1.1 are upper bounded by those of g at the
same points. Confirming this numerically, we can conclude the proof.
Lemma C.15. If E No, one has
100 (v)
1
S
It (c/2)EE>
where e >> O is the absolute constant also appearing in Lemma (property 4), and in particular c/2 is equal to the absolute constant appearing in Lemma C.12 we have the estimate
In particular, if E Nand v E [0, x]]
1 (v)
2 of
Proof. The case of = follows directly (as an equality) from (v) v. Now we assume E N. Smoothness of (f) follows from Lemma E.5 Applying the chain rule and an induction, we have
f-1
3 (8o ((1-1)
II
3 p(t)
(C.23)
and applying the chain rule also gives
( = (ss(t-1)) / (po
+ ((B1E-1) (g0
(C.24)
74

---

Published as a conference paper at ICLR 2021
By Lemma E.5
have & > 0 [0, w]], and the formula (C.23) then implies that 36()) > 0 on [0, TT) as well. Considering only angles in this half-open interval and distributing, it follows
( = 3 o ((t-1)
(3(-11
+ o plt-11
(5)00)
(6 lbopte p((-
(c((-11)2
(3((-1)
3 p((--1)
s o pot--1
((b(E-11)² 2
Applying an induction using the previous formula and distributing in the result, we obtain
(300)
o
(C.25)
(50())
2
- & yller)
x
===0
By Lemma E.5 we have < < 1 on (0, T)
and & s Thus
(000
pl())
(3)10)
P=
When l' > O, we have w(t) S T/2, and by Lemma E.5 we have
x 1n - < 0 on [0, π //2]; thus,
+ pl()) 2 c if > O. When l' = O, we can use the fact that
3 in 0 on [0, T] to get bound
の I -cl We conclude
50 2 cl - + el /.a/2)--
(C.26)
(E)
Next, we notice using the chain rule that
(00
(5(E)
(3)(1)2 2
and using C.23 and Lemma E.5
we have that ()) 1. For any v E T). we integrate both sides of (C.66) from 0 to v to obtain using the fundamental theorem of calculus
1 2 oll - 1)v +c : //"
10,1/22 (t) dt
())))
= cl 1)v + cmin(u,
clv
2
where in the final inequality we the inequality min(u,x/22 2 v/2, valid for [0, r]. Rear- ranging, we conclude for any 0 < <
1 (c/2)
and noting that the LHS of this bound is equal to x and the RHS is positive, we conclude the claimed bound for every [0, n]. The second estimate claimed follows by multiplying this bound by a on both sides, and using + (c/2)ev 2 (c/2)lv.
Lemma C.16. IféE N. one has
(100()) < [c/888)E (1+ (c/8)v log (1 + (c/8)((-1))))
where C 0 is absolute constant, and O is the absolute constant also appearing in (property 4), and in particular c/2 is equal to the absolute constant appearing in
Lemma E
Lemma
C.12
If E (0, n). the RHS of this upper bound is decreasing function of and moreover
we have the estimates
|3(0 < Ce,
1P300()) 5
I+ (c/8)Ev (1- Sloge
OTv
S
STC cl
+
log 2l
CT
75

---

Published as a conference paper at ICLR 2021
Proof. Smoothness follows from Lemma E.5
make use of some results from the proof of
Lemma C.15 in particular (C.23) and C225
We treat the case of & = 1 first. By Lemma E.5
have | S C. for an absolute constant C
0, and since 1/(1 (c/2)e) 2 1/(3/2) = 2/ by
the numerical estimate of the absolute constant > 0 in Lemma E.. it follows
3C/2
[p(x)] - 1+ (c/2)v'
which establishes the claim when (after worst-casing constants if necessary). Next, assume
l > L. Multiplying both sides of C.25 by (zb(0)² and cancelling using (C.23) we obtain
( p((t))
2
/ p(())
-e 10 sp(en)
(50 a po(er)2
6-1
/ 0 (op
3 o ple"))
30x(0")
(C.27)
#"=0
8"=6+1
(3()) S p(())
P==0 0 plle)
where the last equality holds at least on [0, x). by Lemmas E.5 and C. C.15 and where empty products
are defined to be 1. If L" > O, we have (((") < π 7/2, and by Lemma E.5 we have that [g] < C and
e 2 d >> 0 on [0, π /2] for absolute constants C, C" O. Separating the 2 summand, this gives
a bound
sc
S o p(()) 8=1
II
+
sle)
(C.28)
P=1
we have (p()) < v. hence (v) S Using concavity of y, nonincreasingness of yp and nondecreasingness of (f')) (which follow from Lemma E.5) and a simple re-indexing, we can write
By Lemma C.15 we have (p()) S 1/(1 (c/2)u), and by Lemma
E.5
2-2
8-2
II 9 o x0 (0))
(v)
(v)
II
+ o s
(f' (1)+
II
3 o s (t))
o p((()
=0
2-2
-
II
(f)
(v/2))
P=0
(v/2)
1 (c/4)(e-1)
1Iv 1)p
1+ (c/8)Ev
where the third-to-last line follows from C.23 the second-to-las line follows from Lemma C.
and the last line follows from the inequality 1 > f/2 if & 2. Following on from (C.28) we conclude by an application of Lemma C.I5
(w0
C
C/d
Σ
<
1+ (c/8)
I+ (c/2)E (c/2))00
4
Σ
1+ (c/8)Ev
1+ (c/8)eVv 8=0
where the last line simply worst-cases the constants. For any l' E Ng. the function x + 1/(1 + (c) is nonincreasing, so we can estimate the sum in the previous statement using integral,
76

---

Published as a conference paper at ICLR 2021
obtaining
C/d
1+ J
(10
-
dx
1+ +c(88))1
1 cov
cor
C/d
1+ (c/S)/ log (1+ (c/8)(e- 1)2)
1+ (c/8)ev
after evaluating the integral- we define the quantity inside the parentheses on the RHS of the final inequality to be l - 1 when V = 0. which agrees with the integral representation in the previous line and with the unique continuous extension of the function on (0, म| to [0, whhch establishes the first claim.
We now move on to the study of the bound we have derived. For decreasingness, we note that the functions
Clc
2 +++++
(c/8)v log (1 (c/8)(e 1)m),
(C.29)
+ (c/8)ev'
whose product is equal to our upper bound, are evidently both smooth nonnegative functions of v at least on (0, .7. that by the product rule for differentiable functions it suffices to prove that these two functions are themselves decreasing functions of v. The first function is evidently decreasing as an increasing affine reparameterization of r 1/v: for the second function, after multiplying by the constant & - and rescaling by positive number (when é = 1, the function is identically zero on (0, T), and the function's continuous extension as defined above equals 0 at 0 as well), we observe that suffices to prove that v
log(1 (1+)) is a decreasing function of (0, oo). The derivative of this function z + (x (1 + z) log( 1 x))/ + z)). so suffices to show that (1 log(1+ x)) < O. Noting that the function + log x is convex (its second derivative is 1/x), it follows that x (1 + x) log(1 + +x) is concave as a sum of concave functions, and is therefore has its graph majorized by its supporting hyperplanes; its derivative is equal to log(1 (1 + x). which equals at O, and we therefore conclude from our previous reduction that the second function in (C299) is decreasing, and that our composite upper bound is as well. For the remaining estimates, we use the concavity estimate log(1 1 + x) I x to obtain from our previous result
(0)
[c(88)) S CC.
Ci
since the function + C/(1 cx) is nonincreasing for any choice of the constants. Next, the expression we have derived in the first claim to obtain
1000 (1) < 1+ (c/8)EE
Cv
(ut
1 (c/8)
log (1 + (c/8)(E- (c/8)(E-1)u)
For any K O. the function x x/(1 Kx) nondecreasing, and using the numerical estimate
(c/88) < 1 that follows from Lemma E.5 Thus
we obtain in addition + (fc/8) - 1) & & for E E N.
2 (5)
Cx²
1+
log l en
1 (c/8)eT
STC
64C log! +
cl
c29
as claimed.
Lemma C.17. One has for every E E {0, (0.,,...,I) D}}
28 3m
( 0;
((0)()) =1;
(L)
for every & E [L]
(10 = (x)
Finally, we have (T) and (0)) (T) 0.
77

---

n. from which it follows Published as a conference paper at ICLR 2021
Proof. The claims consequences of Lemma E.5 when & = 1, and of (p()) Id for smaller é; assume f >> 1 below. The claim for sp(0 (0) follows from the fact that so(0) 0 and induction. For the claim about ((0) (0), we calculate using the chain rule
()(((0) (0) = (0)) (0)
((0)))(-))())
= -3((-1)(0)
By induction and Lemma E.5 we obtain (0) = 1. The claim about (T) follows from the same argument. For the remaining claims about (((), we calculate using the chain rule
( (1b(1 )) 3 (o( (z((-1)) و pltt-1)
whence
(10) (0)) (tt
Using Lemma B.5 to get ((0)) = 2/(32/33)))))) this yields
2l
( (0)
3m
Similarly, since we have shown ftth 1) (T) = 0, we obtain (z = 0.
Lemma C.18. For first and second derivatives of ((o), one has
5( = -T
10-1ppp 10(")))
and ((()
(C.30)
(le")
II
((()))
- (000)
(3()")
II
(1 -1-a) n ((("))
E"
e" +l 1
+8"
(C.31)
where empty sums are interpreted as zero, and empty products as 1. In particular, one calculates
L
(L (L - - L(L 1 1) l(e
(0)) 1;
(0)
(()
n²
3x
and
1 -x) (T) 0.
(0))
(T)
Proof. The two derivative formulas are direct applications of the Leibniz rule to g()) The claims
about values follow from plugging the results of Lemma C.17 into our derívative formulas and the definition of (()), For values π. we first note that yo (0() (()(
c(0) (T) O. Next, we use Lemma C.17 to get that (())
for all {0, ...., L} and E (I) Lez. For g()) (IT). we have
(00) ( Le=o to get ((9 π
{(0)
((())
II (1- π lol (T))
p=
II
(1- 1-
1- (T))
P=1
If L 1, the sum in the last expression is empty, and this quantity is 0. If L 1, the sum is nonempty, and every summand is equal to zero by Lemma C.17 We conclude (0 (πt) (()) n
78

---

Published as a conference paper at ICLR 2021
Lemma C.19. If 2 3, there exists absolute constant 0 < C S T/2 such that on the interval [0, C) one has for every 0, 1...., L
0.
Proof. We consider functions only on [0, 7/2] in this proof. Following the calculations in the proof of Lemma C.15
have the expression
(po
(C.32)
= =
(3111) +3 p(e (10-1) (5)0 + ((to)i))))) (1)6e-1))²
(C.33)
Using as well Lemma E.5 we have first and second derivative estimates
5 5 S
and
-O2 60 -c2t.
By Lemma
E.5
X extends to continuous function [0, /22, in addition there exists a 5
such that on 0, 5 we have
1
(C.34)
2T
We lower bound C.33 [0,8] using these estimates. For l. = do better than (C.34 For > 1, we can write
(po
0 o olt-1) + 36(0-1)
((0 (((-11)
(16(--1)²
6
(((-))))) v0(e-1) + 3p(c-1)
Lgll- - 6T
We have the numerical estimate C2 = 0.14 from Lemma E.5 and we check numerically that
(0.14)² 1/6=2. This implies that on [0, and for every f
2, (7(0) is lower bounded by
positive number plus a scaled version of (f-11), We check precisely using the original formula C.33 and Lemma E.5 for l =
2
(0) (0)( + 38(0)² = 2 3r
> 0,
so that in particular
(2) (0)
= 3π
1
> 0.
By continuity, it follows that there is neighborhood [o. 8'] on which we have Thus, on [0, min 8')], we guarantee that simultaneously
> 0.
0 if e 2
(3 + 20) > 0.
Now consider the third derivative of the skeleton summands (()) Following the calculations of Lemmas C.10 and C.18 particular applying the Leibniz rule, we observe that every term in the
sum defining { that does not involve third derivative of one of the factors (1 - (f")
will be nonpositive, because (1 (1/ (L/=)/ (e())
(ttt) > O, and (3(*) 0. Mcanwhile, by our calculations above, on the interval j0, min 8' H. the only terms that can be positive are those with
0 or where we differentiate the factor three times, i.e., the l 1 term the sum
L-1
(e("))
π
=/
79

---

Published as a conference paper at ICLR 2021
with / or é = 1. We will compare the l' summand with the l' = 2 summand: we have that the sum of these two terms equals
so(e") π )
(1-1
(o(2) (3(2) (1- ))
1
(C.35)
π
π
#1,2
At O, the quantity inside the right parentheses is equal to ن + (3(2) 0, by calculations above.
Thus, by continuity, there a possibly smaller 8" >> 0 such that on [o. 8"), the sum of terms (C.35)
negative. We conclude that [0, m 8', J. we have for every 2 0
()))) S 0,
and since we have chosen the neighborhood sizes 8, 8 8 independently of the depth L, we can conclude.
Lemma C.20. For all & E {0,.... L - 1}, one has
5 (1) - 1+ 1+/T
1 Lu/T
Proof. We have
1-1
5 (v)
1-1
II
1
(v)
1
1
(o1))())
π
π (L 2)
(C.36)
Σ 8=
1-11
(1))(())
S exp
where the first inequality applies the AM-GM incquality, and the second uses the standard exponen- tial convexity estimate. Using Lemma C.I3
we have
Σ (00
(v)
Σ
E1te
Ev/ S
[
L
de,
8=28
where the last inequality the fact that 6' w/(1 w//x) is nonincreasing for every [0,z] together with a standard estimate from the integral test. We calculate
J' I+ll/ de' x log
1 LV/T
1+ Lv/T
which gives the claim after substituting into (C.66).
Lemma C.21. For & {O...., L - 1), one has
L-t
(x)) <3
1+ Lu/T
Proof.
Using Lemma C.18 we have
c(1)
(())
L-1
(3(0)
{(0)
Σ
16=0
so(t")
(=mma((f11
where we directly treat the case avoid dividing by zero at v n. The triangle inequality
and Lemmas E.5 and C.20 then give
(z)9 <
1+ Lu/a Im + + {(
1
Σ
(00
8 (=maa((.1)
80

---

Published as a conference paper at ICLR 2021
Using Lemma C.15
we have
(v) S
Σ
2 +
It clvv de'
P=t 1+ 1 clu
where the last incquality uses the fact that l' + 1/(1 + cl'v) is nonincreasing for every v E [0, together with standard estimate from the integral test. Evaluating the integral, we obtain
Σ é=t
(0)0 (V)
1
log
1 + 1+oL--1)
1hv
1 clv
where the second term on the RHS defined at v = 0 by continuity. Using the standard concavity estimate log(1 (1 + x) S we have
+ c(L - 1)v)
log (1 1+ (L- - 8- 1)ev)
(-1ee
S L-6-1
log
CV
1 + clv
CV
1 clv
1 clv
whence
6((0) (v) (v)
L-8
(C.37)
1+ clv
Combined with the result of Lemma
C.20 we conclude
1 L-
(00
CT 1 + Lv/T
The numerical estimate c = 0.07 in Lemma H.5 then allows us to conclude
--6 (5(0 (v)
1 Lu/m
as claimed.
Lemma C.22. One has
5nL
(x(()) 5 1+ LV/T
and
(3/2)mL2 1 L2/T
1 (v)l <
Proof. We calculate using the chain rule
((e)
cos - sin (t)
2
t=
and the triangle inequality gives
1
71
2
+ g00
t=
Applying Lemmas C.15 we then obtain
C.20
and
C.21 I
and Lemma
E.5 estimate the constant e Lemma C.I5
T-
1+/u/x
[x)( < 2(1 LV/T)
3(L 2) + 1 (v/(5T)
E=
ev (v/(5T)
< 2(1 + LV/T)
3(L 2) + 1+
(0/(5T)
3112
2 + 5L
2(1 + LV/TT)
5n
1 Lu/T"
The proof of the second claim is nearly identical, since in this need only the bounds (E()
81

---

Published as a conference paper at ICLR 2021
Lemma C.23. There are absolute constants f) + (1 (1+clu)( + ctv)
C > 0 such for all E {O...., L 1}. has
S LL
(L-1)
210)
Proof.
By Lemmas
E.3 and C.18
we can write
(3(0)
((0)
p00
π
E' #mmaa [1:0]
x
1-1
sp(ee
(310)g200)
1
(5(0 Il 1=
+
+g(0
(0)
Σ
Σ
n²2
(1
(=
(10) 8 ==axx(1,e)
a
e" "+
Focusing first on the second term, we have using Lemma
E.5
CC37 and Lemma C.20
(ole")
1-1
(3((') (,(("))
25 1
+ c(t)
[ so(1")
[1 pl60?
F=1
8 Tamax max(1.f) l" (1)
E"
L-1
<40(1/100
(((0)
+ 4g
Σ
F=1
(=,) 8" m1)
E"
We can then write using nonnegativity
ΣΣ
L-1
Σ
6-1
20(0"))
Σ
4-1
)
and using CC37 and Lemma C.20 we obtain thus
((1) 1
((*)
{(0
Σ
Σ
(10)
3
(L-
CT (1 Tw/ + clu)
P=1
mm (1.6 r" ("=max(La (1,)
C"
Regarding the first term, we have using Lemma C.16
L-1 SC
SC L(L- 1 (c/4)Lv*
E e=t
1+
8=t
because the function 8' + f/((1 + cl'v) is nondecreasing. Applying also Lemma C.20 we obtain using the triangle inequality and worst-casing constants
sc, LL f) (v/ + C2 (1 + clv)( + clu)
(L (1-
(1+cl
Lemma C.24.
One has
[+((w)] 5 CnL3
+ cTu
and
CnL3 [0)( < + cLu
where C, C are absolute constants.
82

---

Published as a conference paper at ICLR 2021
Proof. We calculate using the chain rule
5 cos ( 2(00) sin (p!))) - -0000(0 sin w0 - gg( (1511)) cos (
t=
and the triangle inequality gives
n
(00)
+
(0) 5 + 5l (00) + 5 (1)0)2
2
Using Lemmas C.I5
C.16
C.20
C.21
and and worst-casing constants for convenience,
obtain from the last estimate
T-1
LL-001+2/mm
(L-F)² (T+c 14/0/T
OnL 1 cLu'
(0) & Cn
(I+cLa)
S
(I+ +
(I+LP/a(L+cb))
I+L
where in the second line we made some estimates along the lines the proof of Lemma C.22 and worsened the constant C. The proof for db follows from the same argument, since in this case we
have the same sum of g(() terms but none of the extra residuals.
D CONCENTRATION AT INITIALIZATION
D.I NOTATION AND FRAMEWORK
We recall the expression for the neural tangent kernel, as summarized in Appendix AA..2
O x²)
v fer (x), (x))
L-1
= (aa"
The objective of this section is to establish supporting results for the proof of Theorem B.2 which gives uniform concentration ( x²) over M x M around the deterministic skeleton kernel. We take -unirormie approach to proving this result: Appendix D.2 establishes concentration results for the constituents of ((x =²) when x, x' are fixed, and Appendix D.3 develops results that control the number of local support changes near points in discretization of M M in order to provide a suitable stand-in for the continuity properties necessary to uniformize these pointwise results. We collect relevant technical results and their proofs in Appendix D.4 D.4
D.2 POINTWISE CONCENTRATION
We fix (æ, w') in this section, and generally suppress notation involving the specific points for con- cision. We separate our analysis into two distinct sub-problems: "forward concentration", which consists of the study of the correlations (a'(x), ae' (x')), and "backward concentration", which consists of the study of the backward feature correlations (3 (æ), 3 (x")). Forward concentration is prerequisite of our approach to backward concentration, begin there.
D.2.1 FORWARD CONCENTRATION
Notation. For l = 0, ,,...,,. L, define random variables 2f = || (æ) and 25 = ||a((x'))2
With the convention 0 +oo = O. we define for l 0,... , L, random variables oe by
Lf c1 (1./20 11 ( a |aa (z) ||a'(("))
a'(x')
-1 (=)-0))((350)))
These definitions guarantee that of π whenever either feature norm of vanishes. These random variables significant toward controlling O x²) because, for each
(z)) o (xx))
=f cos L
83

---

Published as a conference paper at ICLR 2021
Let us define pairs of gaussian vectors gi g widd No (2/n) that are independent of every- thing else in the problem. For & 2 1, we have by rotational invariance of the Gaussian distribution and the probability chain rule
W (x))+ - -
Since a²(x) - ar and ||a||| = 1, we have by an induction with analogous definitions
2
9i
p=
Similarly, have
2/
9i
P=1
As for the angles, we have by rotational invariance
=
[ W a (x))
[ 11 cos wt +
2 sin v
2
1-1 E-1
-2
2
so that an inductive argument gives
111,))
i
gi cosv + sin 1/ - [1ll))
2/26
II
9f
We will write
of cosv g2 sin v
gf
+
+
and similarly
[gi] +
[g] cos v 9 sin e-1
of cos-1
Lagazso
1 [18==0)U(2$=0)
[of] [of cos ve-1 + J sin
so that we obtain for the angles by similar inductive argument
(D.1)
For technical reasons, it will be convenient to consider an auxiliary angle process, defined for { 2
[91],
[g] cos 0--1 + g sin xe-1] 11+
8 cos-
(9(
(D.2)
[9f]]
off cos p + g sin 0e 11]
+
where we define with notation from Appendix E.
E = = n ( ((1.
Vv E (0,2m],
4oj (g cos v + sin $2).
kE(n)
and 00 = --= (ar, x²). We then observe
II ae (of
II $
since the inductive structure of E implies that all feature norms are nonvanishing if and only if the top-level feature norms a/ are nonvanishing, and since the statement [ l .
84

---

Published as a conference paper at ICLR 2021
implies by definition that of 2 2- and 3 2 2- By Lemma E.16 long as n 2 21 the event has overwhelming probability, and in particular a union bound implies
P
11 13f3520
1
2 IP
18
of 95 = 2 - CLE
(D.3)
1=11
so that
- = 1,2, ..., ,,, -- CLe
(D.4)
We can therefore pass from if to of with negligible error.
From the expression for ie, and we will control them using martingale techniques. For = 0, ..... 0.1, L. we write J to denote the o-algebra generated by the gaussian vectors (gi 9 9 9 ................................ I (igh) so that (J) ........ F2)
that the angles 500 81
uL form Markov chain,
is a filtration, and the sequences of random variables (i) (FF (")... Moreover, with these definitions we have
8²) and (1 D²) are adapted to
E[
[- = ((1
where y is the angle evolution function defined in Appendix E.T which well-approximated by the function
(11-
sin
(pv) = cos
the sequel, we will employ the notation sp()) to denote the f-fold composition of to with itself. By Lemma [E.5] the function jo is smooth, and the chain rule implies the same for ep ; we will employ the notation (8()) and ((()) for the first and second derivatives of (t) respectively.
(see Lemmas E.1 and E.2
Main results.
Lemma D.1. There absolute constants c, C' 0 and absolute constants K, K' 0 such that for any d 2 K, in 2 K' max(1, dt log' n, d L log" n3 then one has for any E = I,.... L
P (a(()) - cos
(4x, (00
sN
d3 Bog"
< nod
Proof. We have
(a'(x), a (x')) d =12 cos v
and the triangle inequality (applied twice) then yields
Ka' (a(x (x')) cos 6 (09) < cos e/ 12//0 - 1/ + |oos vz cos cos
==111 1| |/2 - 1| - (10)
where we also use |cos| < 1 and that cos is 1-Lipschitz. Since 2 _ oी for i = 1, 2, we obtain using and
Lemma D.2 and the choice 2 KdL
df n
2 - 1
C
s dd
and as long as 2 Cdd,, we obtain on one of the same events
2 2 > 1
C'ee
By union bound, we obtain
P
2 1|
-
< 30
2 1 - -d
85

---

Published as a conference paper at ICLR 2021
so that if put d' d log n and therefore choose 2 Cdd log n, we have
de log n 1| |2 1 - 3C
2 - 2C-20"nn 21--2'n
n
with the second bound holding if d 2 1 and n L. For the remaining term, have by the triangle inequality
10-000 - <1p' -y'-0"
(0()) (z9)
By (D4), the first term on the RHS of the previous expression is equal to zero with probability at
least I CL as long as n 2 21. The second term can be controlled with Lemma D.3 provided select n, L, d satisfy the hypotheses of that lemma. We thus obtain via additional union bound
d log" n + C'y
P [( (a(),, cos (10) 3C n
de logg
SC" Ce
nf.
n 2. (2/c') log J. and n 2 (2e/d)ddog we have C"n ced + C" S (C" + C" The previous bound then becomes
P (a'( cos (00) > 3C1
dl delog n
d3 log n CO
S (C" In")naa
nel
and if worst-case the dependence on l and d in the residual in the previous bound, we obtain
p (a (o<( ax(')) - cos (0 (3C + + d log n
S (C" C"")n=a
claimed.
Lemma D.2. There are absolute constants c, C, C" > 0 and an absolute constant K > 0 such that
fori 1,2, every f 1,.... I, and any 0, if max maxx(Ktt [Kd,, 4}, then one has
de
I
1
CV
-od
Proof. Because 2f zf, it suffices show
dl CV
P
S C'ee-
(D.5)
n
The proof will proceed by showing concentration of the squared quantity I[
gi
around
1, so that we can appeal to results like Lemma D.26 and then conclude by applying an inequality for the square root pass the actual quantity of interest. To enter the setting of Lemma D.26 it makes sense to normalize the factors in the product by their degree, but we must avoid dividing by zero. We
have II
and only if I
gf
O, and whenever I
*
write
1/2
2
II
gi
(D.6)
1/2
1/2
2
II
V
n
9
(D.7)
2
2
86

---

Published as a conference paper at ICLR 2021
using O-homogeneity of the 6 "norm". This leads to extra pfoduct-of-degress teri;; will make use of Lemma (D27 to show that the product of degrees itself concentrates. We will also show that the event where a degree is zero is extremely unlikely and proceed with the Pegree-normalizer main term by conditioning. By symmetry, the random variables
[ llo are i.i.d. sums of n Bernoulli
random variables with rate 2.2 By Lemma
G.1
we then have
n/2 $ cc-2//10
and
oof
< <n/2 - 1 P 38 P
IP
min
E {1, (I,...,,?
< <n/2
<ep 9 1
< <n/2 - S Le 22-
where the first inequality applies union bound. Putting
11/4, conclude
96
< n/4
< 80/88
[gf]]
so that whenever n 2 16log l, we have
llo 2 n/4 for every l' < l with probability least
1 - e n/16 This gives us enough to begin working on showing concentration of the squared version partitioning, we can use the previous simplified bound write
of D.5
de
P
CV
(D.8)
de
$ e n/16
min
2 n/4,
(D.9)
#=1
Using
D.7 and the triangle inequality, write whenever terms in the product vanish
1
2
1
9
2
9
+
2
2
(D.10)
f=1
Moreover, we have by Lemma
D.27
de
P
4
S 4le 4le-od
as long as n 2 128de. Choosing in addition 2 4dd and using nonnegativity, this implies
IP
2
4le ed
occurring on the same event. Combining the previous two bounds with (D.10) and D.9 via another partition, we get
de
P
C
S n/16
46c 4ed
n
==1
ming-
g
2 n/4
+
+ P
2
(D.11)
1 1
Tr-=
V g
(C/2
2)
de
87

---

Published as a conference paper at ICLR 2021
where we use here that on the event {minee=t,....
of
|, llo 2 n/44). the quantity II
9
+ is nonzero almost surely, which allowed us to invoke the identities D.7p. For (KI...., ke) [m] we define events Ex ...., ohe by
oky Ee
ke }
Conditioning and then relaxing the bounds, we can write
de
min
2 n//4 4,
C'
2
9
2
2
1
9i
C
gh
oki
Σ
+|P
1m/4]
Conditioned
on
cki
gki
0, the random variable llo is distributed as a product of independent degree-normalized An application of
with
kp
Io standard
gi
x²
random variables with minimum degree min(ki..... kz).
Lemma
D.26 then yields immediately
de
IP
91
C'
cki
Eh
S Ce
2
1=1
as long 2 d"de, whence
1
de
min
gi
2 n/4,
-1+
1
9i
> C'y
2/2
n
I Cod
Combining this previous bound with D. yields
2
dt
P
C
S --/16 + C'ee-oa
where we worst-cased constants in the probability bound. If we choose 2 4C we have ddde/n S 1/2, and we obtain on the event in the previous bound
1 2
P
< e
n/16 + ood
In particular, the complement of the event in the previous bound, the product lies in [1/2,3/2] To conclude, we can linearize the square root near 1 to obtain an analogous bound for the product of the norms. Taylor expansion of the smooth function x + 1/2 about the point I gives
Vx - - 2 (x - 1) 8
3/2 (rr - 11)
where l lies between and 1. In particular, if 2 2. we have
x 1) (x- 1)² S Vo 15 (x 1),
V2
so that
(VT - 1) -
1
(x - 1)
(r - -1)2
2
/2
88

---

Published as a conference paper at ICLR 2021
Thus, when 2 we have by the triangle inequality
I - 1| -
v2 (x
1)2
11.
from which we conclude based on a partition and our previous choices of large
P -
91 1[1
20 de
< 20-11 2C'
which yields the claimed probability bound when 2 16d.
Lemma D.3. There are absolute constants c, C, Co > 0 and absolute constants K, K' > such that for any Lmuxx N and any 2 K, if K' max d log" n, d Imax log" n), then one has
d3 (olo"" n nL.
P EL E [Lmax] - (0(0) >Cop
S Cn -nd
(D.12)
Proof. The proof uses a recursive construction involving E [max] Before beginning the main argument, will define the key quantities that appear and enforce bounds the parameters to obtain certain estimates. For each I. E [mmaa). we define the event
f110- -1000)000)
d3 log >Co
nL
where Cu > is an absolute constant whose value we will specify below, so that &L F1, and our task to produce an appropriate measure bound on ULe((mms)) Er. For notational convenience, we also define & Ø. For each L E [mmaa] and each E [L.,, we define
A (
that for every L, A) L A L is adapted the sequence Fl ,-.., FL, and we have the decom- position
ot - (10 = Σ A
In particular, have
d3 log" nL
E,
Σ 2 Cov
The sequences (At)eEL not quite martingale difference sequences, but will show they very nearly so: writing
A - - -A +
+E[AL FT11),
we have that (EEL is martingale difference sequence, which can be controlled using trun- cation and martingale techniques, and the extra conditional expectation term can controlled analytically. In particular, we have the following estimates: by Lemma D.24 we have if n 2 max K2 log" n, KoLmax that for every L E [Lmax] and every & [L]
log
Øt-1
1
B[A F "
+ (cL/64)(L fjze1 (1 + log L) + C2
(D.13)
n?
n
by the first result Lemma D.25 we have for every 2 max (K3, 6/c1) that if n 2 Kad log n, then for every L [mmax] and every & E [L] (and after worsening constants)
dlgg
2C, n²
PP | > 203v
14 (co (4)(L E)ot-I +
xt-1
< Crn-arn
(D.14)
89

---

Published as a conference paper at ICLR 2021
and by the second result in Lemma D.25 for every L E [mmax] and every E [L] (after worsening constants)
have by previous choices of n, d, and Lmax that
T (At)))) F
403
dlogg
(T+ (co/66/)) e)oe-1
+
C1 7
(D.15)
n
The main line of the argument will consist of showing that measure bound of the form (D.12) ULE(L-1) & implies one of the same form on UEE(c) & For any L E [kmax], on the event ε7 we have
d log" y nL.
8¹ S (6)) Coy
2
d3 Golog
(D.16)
+Col
nL
To
where the second inequality follows from Lemma C.12, and the third follows from the choice 2 (Coco) L log In particular, if we make the choice > cd log we have (D.6)) on & for every [Lmax] Accordingly, for every L E [Lmax] and every [L] we define truncation events 9 by
| s 2C3
d
2C₂ 1 (co/ 64) pjpe- + n²
nE
(D.11)
n
We have of E F², and union bound and 4D14 imply
(
oi
p4-1
P
< CsIm -add +P
U Ey
[E(T)
pE(t--1]
CGLn Ureenn Ege
where the second line uses the fact that Eer F In particular, taking expectations recovers
(
1
P
sf.
CaLn +P
U Ee [EELL11]
(D.18)
(EIL)
In addition, by (6. we have on E
i8-1
S
1 + (co/64)(L -())t111
Co (6 - 1) + (3 (L E)
3
co (3/64)L (61/64)
Co
128
To
where the final inequality requires L 2. Thus, when L > have gf that
256C3
d dlog 2C²
L <
Cn L
n?
512Cs
d nL2
(D.19)
Co
2Ko
90

---

Published as a conference paper at ICLR 2021
where the final inequality holds when d 2 1 and (50o)/18Css)//313 L2/3. Similarly, when 2 2, on E we have by (D.15)
2 C3
dlogg n
C4 n
(AL)²
6
2
nI?
(D.20)
2 Cg
d doog n
d log
2Kg
%
nI?
nI?
where the second inequality holds when d 2 and n 2 (C1ch/ (3) and the same setting have by (D.13)
12801 (1 + log L) log
C2
B [
Cp
nL
n12
(D.21)
256C1 (1 + log L)) log
C0
nL
where the second inequality holds when 2 (C2eg/ 128C L. In particular, if we enforce these conditions with Lmax in place L, we have that (D.19 to (D.2T) hold for all 2 S Lmax (with (D.20) and (D221 holding E
We begin the recursive construction. We will enforce C. = max max 6ku} for the absolute constant in the definition of &g. The main tool the elementary identity
U & = [E[L]
P
U &
P +
+
n & BE(L-1]
(D.22)
[E[L-1]
which allows us to leverage an inductive argument provided we can control PIEz n
ketL-1 E
the probability that the L-th angle deviates above its nominal value subject to all prior angles being controlled. The case = can be addressed directly:
(D14) gives
d dolog n
2C² n
P | > 2xC3y
< Csn
and as long as d 2 1 and n 2 (C2/TC3)2/3
this implies
d logg n
PP
A > ATCs
Csn -iid
(D.23)
This gives suitable measure bound on ε,, after choosing d 2 1 and 2 that d log" > d log n. We now assume L 2 2. By the triangle inequality, we have
Σ
<
Σ ETAL
(D.24)
t=1
and we therefore have for any
1
>
t
1=1
LE(L-1]
L
<
P
Σ
B[AL
Fー >
n
E
(D.25)
EE(L
I. Σ
P
1 ir E3
Ni
1 +
1
58
E E[A
t=1
By D.21
we have
L
256C1 (1 log L.) log n
1 NEE 8f
Σ
A
(D.26)
E=1
91

---

Published as a conference paper at ICLR 2021
For the remaining term, have by the triangle inequality
E
A L + Et Ls of
IE D 0%
xt ']
A
(D.27)
Σ 1- E o -
B[AL|
1=1
By (D.17. an integration of (D144, and union bound, we have
necte 28
Σ a A619; 0
>
dlogn
D-
2C
(D.28)
A 2Cs/
+ n2
1 (L f)pe-1
n
[EIL]
< CgLn -id
and we have
Σ A
1 oi
xt-1| -E AE F+11]
E
| 11 (02)²
x
8=1
((f)) : xt-1]
$ x
1=1
|
=20s
dlog
Dl-1
202 n²
U Ft-1
1 (LL/64)(L f)pe-1
L-1 Σ LE
S SaCILn ad + π
where the first line uses linearity of the conditional expectation and the triangle inequality for sums and for the integral; the second line uses the worst-case bound of x on the magnitude of the incre- ments Ae: the third line (D.17): and the fourth line uses union bound, & J and D.14). Multiplying both sides of the final bound by Neetc 11 ES. we conclude
L Σ
AL
xt-1]
EE FC-1]] < 1 M C5TCoL -dd S TT3LL -iid
(D.29)
For the remaining term in D.27 we first observe
E
(Aillo
E
A 1i
54-1 (z--1)²
< E (Ai)) 1o [(-1]
2 F(-1].
where the first line uses the centering property of the L norm, and the second line uses (A/)²² 2 0 to drop the indicator for G For notational simplicity, we define
VL Σ
(AiLo:
E |oAiLoo
so that our previous bound and D.20 imply
& c { yv < 2k3dlog
n LE[L-1]
nI
92

---

Published as a conference paper at ICLR 2021
This implies that for any > 0
1
P
Necta
Σ
Af.lo 0f
E - F(-1]
>1
1=1
n &
n
Σ
A 1g. - -B(A/LO
54-1| >
KE|L-1]
olgg
< P
v4
<
2Kg
Σ
Aill
- -B(2 of
x(11
nT.
The previous term can be controlled using Lemma G.5 and (D19):
P yh
nL 22kg
222 dlogg
A og
E A
Fz11|
12/2
$ 2exp
2K3
d logn nL
(2Ko/3)ty
dlog nL²
Setting / = 3K0v d9 log" n/nL, we obtain
Dal =BAALo x(-1]
d3 log""
Neciz n f
3Ko
nL
9 d log
(D.30)
< 2 cxp
4 41+ d'log
< 2n (9/8)d
where the last line uses the bounds L 2 and dol n/(1 d dolog) 2 1/2 d 1 and n 2 e.
Combining (D.28) to (D300) in (D27) via union bound, we obtain
d log Inn Ia add
P
Neete-11
E
Σ A
kKo
C, cid + 2n (9/8)/
nL.
t=1
Applying this result and (D26 D.25) via union bound, obtain
L
d3
log
256C1 (1 log L) log
3Ko
+ Tad
n & eE[L-1]
nI
Cu
n
S CsLn aid 2n -99/8d
If 2 2/e, and 2 Lmax, we have -id S Csn under these condition on d and n, we have xCs Tm-ost & TCsn -1 and so aosn- (250C)/oo) (1)) + log [)(log nn/n s
C(1 + log [log //; and if 2 and n 2 Tamax we have
d log nL
( + log L) log
3Koy
provided
2 (C(C3/oo) Liaa log Lmaax-
Under these conditions, our previous bound simplifies
to
L
d logg n 6Ko
P
1
n
n E (E(L-1]
< (2+ Cz)n min min(s// (c/22.9888
nn
93

---

Published as a conference paper at ICLR 2021
In partieular, applying this bound to (.22
have shown that for any 2
PP
&
=
P
&
+ (2 + Cz)n min((c/22,9/88)
FE
rE[1.-1]
!]
Unraveling the recursion with (.23) (and worst-casing the constants there), we conclude
& LE|L
$ (2 + Cs)Ln min/es/2,9/884
which proves the claim, after possibly choosing be larger than another absolute constant multi- ple of Lmax to remove the leading L factor.
D.2.2 BACKWARD FEATURE CONTROL
Having established concentration of the feature norms and the angles between them, it remains to control the inner products backward features that appear in θ. The core the technical approach will once again be martingale concentration. We establish the following control the backward feature inner products:
Lemma D.4. Fix x. x' E S and denote v Lax, x²). If 2 max (KLlog K' K'Ldb, K"), de K" log L. suitably chosen K, K', K" K" then
P
h-1
n
(1S)
cnn]
2 2
t=
additionally n, L, satisfy the requirements of lemmas 1..3 and
E.16
we have
L-1 n { (0
n L-1
- (1)0(()))
(1
II
S (n)
21 ood
2
π
inf
where pl denotes applications of the angle svuttt defined lemma E.2 and c 0, C are absolute constants.
Proof. For l E [L], write J for the o-algebra generated by all the weights up to layer & in the net-
work, i.c., W1 ....; W with F given by the trivial o-algebra. Consider some (3 (x), 8 (x'))
for < l' < Defining
plll' (x: Pie(a) W PI 1<x) ... P(y W
Bed = pt:l+ (x) Pl Pirz1 (m) PI Pi pél pel'+2(
(x)
for 6 {E" + L..... L). and setting 22 BB:'' I we define the event
En = ={1 Blae"
EL+18
F S CnII.
Blat' ea'
sCL.
n
[
B BLaE Fe
S $Cn].
Since (3r (x), 3 (302)))) = Wh+1 I.a W Whl is Gaussian chaos in terms of the Wh+1 vari-
ables (and recalling WL+1 ~N I)) and EL is Fu-measurable, the Hanson- inequality
iid
(lemma G.4 gives
P
1 Eeln
(13" (x)
BLii
CVtnL
$ 2exp
-emin
tn
< 2c ct
L
94

---

Published as a conference paper at ICLR 2021
Using lemma D.28 to bound TP (&gthie) from above gives
8 (2)), (z), - tr B
CVVtL
<P Egtnr 13"" (2),, (30),
(0),)((('))) (a'))
BLiE" axx
2 C.ttT.
+ ( ((eeueee (38" (x) (a') -
BLit ooxr
2 CVInL.
(D.31)
SP Lggtree 38" (x), (x). (2') tr BLit"
CvtnL IP (8/-1:))
2e + Cn-d S C'e-e"t
for appropriately chosen constant, if / E n/T. Choosing / = n/T. in the bound above and using the bound on tr
B Lit
from lemma D.28 we obtain
1100 (z) ||3
$ IP 1 (a) (x) tr BL 2000
2 2Cn
> Cn + P tr
BL ae
>
Cn]
Ce + C'nnTee s Cn'nI2e-
for appropriate constants. Taking union bound over the possible values of e' proves the first part of the lemma.
using martingale concentration (in similar manner to the control of the angles established in previous sections). We write
We next control
B
BL.
II (1 (1- p
(1-
E=
(v)
tr
Blat Lip mxx
n
(D.32)
2 i=pp'
1-1--11 1-11
( 1- (10)
) (1r B
- ( (1( (1--0(((v))
- II
tr
B L:t man!
Σ
A
(D.33)
(-tt=(+1
6=t'+1
(note the change in the indexing). Consider the filtration
TO C
C FL and adapted sequence
A A E
A
(D.34)
so that
Σ
A
Σ
E
Ae+
Σ t=1+1
(D.35)
E=
t=
We begin by considering the first term in the sum since it takes distinct form. Denoting W the i-th column of W rotational invariance of the Gaussian distribution gives
(s.))
tr Bf*+1s4 +1:0"
=tr P Pl Plp PIplite))
tr Pwe (m) Pw Pw 1t (m))20
P tr P P wee cos + sin >0
and hence
E
tr
B
1.f")
Fe
E tr
B +1:6" ax'
E 1 >0 cos cos tor sinvel 20
wi'+1
$1.92
where (g1,92) NOO I). Moving spherical coordinates, obtain
00
n
De'
B +1:8
-- 12
E we
axx
2TT
2 π
tuer
95

---

Published as a conference paper at ICLR 2021
We now note that conditioned on F tr P a. Pwe.npoPWr w dependent variables taking values in {0, 1). An application of Bernstein's inequality for bounded random variables (lemma
Pwo
(..) cos w W. slav> is sum of n in-
G.3) then gives
1 (o(((())
D
P
De+1 > Vnd
P
II
tr
+I-e
12
1
> Vnd
2
L==++1
nd
eex
(-c n2 + vnd
S 2e-cd
(D.36)
for some where we used the fact that the angle evolution function ip is bounded by x x/2. Note also from Lemma D.3 that
L-1
(1
log (n)n
P
E
II
L
i=
L
(1
so(((w))
ve
so(e)(u)
d3 log" (n)
(D.37)
n
=Ip
2
L
6=14+1
Se Senod od
for some constant c, where we assumed d > log n for some K.
Having controlled the first term in D.35 events
proceed bound the remaining terms. We define
(1|a--(<>)12 || |aa-'(x")11 > 0} n {lete- - -1 SO
d3 log
ne
{IBEStee S SChnl
{t B
1 |Son]
wax
a
{ B B sce)
n
(which from lemma D.28 hold with high probability). Note that as a consequence of the first event in séil' the angle of is well-defined. Note that gê:2" is -me-surase..l
We will first control (D35 by considering each summand truncated on the respective event Ege".
Our task therefore to control
I 1 Σ
Σ t=
Ae +
EAI
Egg
Since
1 c
=
E
F
E F1 =0
the first sum is over zero-mean adapted sequence and hence martingale, and can thus be con- trolled using the Azuma-Hoeffdin inequality. We will first show that the remaining term is small. We begin by computing
1 gew Etr
E eeee BÊ-1d 8 Peo
Wi
where we used the fact that Etil' F(-l and is thus independent of W
There exists matrix R such that
Ra R- (z) | ê Ro Ro R ||a"=(a")\] cost + (t
Rotational invariance of the Gaussian distribution gives
(x) & w W& (a)))((
w'a d | We
cos f-1 + (1.)) sin v-11)
(11)
(1.2
96

---

Published as a conference paper at ICLR 2021
where we denote by W
(5,)) the i-th column of W Defining B
B
RB
R we have
Etr
B que
F
B
w
Pw
oo P
Wt
612 cos +W sin We
W
ax.
= 1 c
1
zx'ji
E
E wt su W/, cos
W 2 12 W
W
=1 gld
E nn w0 cosed 1 Wi2 sinve 120 W
Σ
Qiy
s-=1
If (1,2) we get (with the square brackets denoting indicators)
0 dej Dt-1
(1-11
0 -
20g
[W 0] [Wi cosv"=1 W sin
W
π
ifi (1 then the Q/ + only if j E (1, these cases we have
= =
(W
W o] [Wi cos ve f-1 W sin e- x0
W
2 E of 191
0] 91 cos e- + 92 sin
0]
W
where (g1,92) ~ (O, I). Moving to spherical coordinates, we obtain
x/2
1
e 22 cos Adrdd = v8-1 sin e-1 cos 8-11
Q
0
and similarly
x/2
sin f 8-11 11-1
-- 12,3 sin? 0drd0
π
tv
0 = 0 /-1
E W
W > 0j [Wi cos
W sin e-1 > 0] W f-1
n/2
x/2
sin ve-1
12,3 sin O cos Adrdd
sin O cos 0d0
2m
2n
$ 4yd-1
Combining terms and using tr B Be-1:e"
tr
B B 1:e"
obtain
ve-
B l:f
π
ax
1. Agger E
B
[
1 e
sin 8-1
cos e-1
(Bb-1il
Be-1:e
π
sin 2 ee-
B88-11
1:8'
Bo--11e"
2x
hence
1 (x)
(2)
1
geer E W
tr Blil"
B B 1:10
r² (v)) 60 tr B
1.0"
=1
+ sin x cas -1
Bst BBe-1i"
Be
5e-1:P
22
sin 2 Af1
130-1:1
snae 12 + Be-1-
2x
97

---

Published as a conference paper at ICLR 2021
On E the bound on
|0 1 (V) -ve-1 and lemma C.12 give vf
C
F a.s.. Additionally, this
Be B Br= Cl a.s.. It follows that
event max AJE(m]]
(1-
16 (V)
Loge E W
Bua
B BL-li6" we
(D.38)
π
d (onlog's log
20
C
d log
I 0²
S
O'
(D.39)
π
al
E
almost surely, and hence restoring a constant factor with magnitude bounded by 1, we have
(D.40)
(1 (1- (1)
II
oeier E W
Bic mxx
B--1:e" 1:0"
(D.41)
π
2x
n n log n
< C²1
(D.42)
[(
Using lemma D.28 to bound
from above then gives
d log n
ct (ett)) S S'n
P
CVV
P
(D.43)
An application of the triangle inequality and union bound then give
EA/
> C' C'vd&Ln
<IP
EA,
Fe-1 C'v log n
12
L(=l'+2
dn log n
IP
EA > CVV
L
t=
}
IP
t=/42
<CLn -ed
(D.44)
for some constants C.
L
We proceed to control the remaining terms in (D35 namely
Σ Ae. Aiming to apply martingale
(-
concentration, we require an almost sure bound on the summands, which we achieve by truncation. Towards this end, define event
ge Dcl A dyy
en log
Combining (D.43) and the result of lemma D.29 (after taking an expectation) we have
dn Inlogg
P G > 1-P
EAd Ft CV
P
A
CV
de
2 1 C" ced
d'd ".
c""
(D.45)
98

---

Published as a conference paper at ICLR 2021
for appropriate constants. We now decompose the sum that would like to bound:
A
1=1'+2
A St
Σ (=t'+2
D 1
(D.46)
F=
Eg
E
A
[ E [A
t=t+22
Since each summand in En are equal zero on the respective truncation event, a union bound and D.45)
give
PP
Σ A. - A 1/-4+2
>
SP
Ü 9i
S
[9
LCe -ed
(D.47)
12=F+2
E=
for some constants. The term Σn is of almost surely bounded martingale differences. We can apply the Szuma-Hoeffding inequality (lemma G.8) directly to conclude
P
Σ
A.C.c Ed [
d VVVLL VLLlog" log L
(D.48)
d''Llog nlo L
S exp
Se Scd
(D.49)
Σ
CVvil + C'V dinn log
t=t+2
Considering single summand in Eg, Jensen's inequality and the Cauchy-Sehwarz inequality give
E Aelle - Aol -11 = E [AAlc:
x-11]|
E [E[1 F- < (E 1 gg -ーー 1/2 (E =(
(D.50)
a.s.
This is an J -massurblle function, and we can show that it is small on the event E E F(-1 To control the first factor, we note that
ge [1og] Ft-1]
E oggneta Fi-1]
|d'n
Drl Ovde+ + +.'
n n otil'
Ft-1
[ > Cvde +0
n DE-E"
J--1
+P [| Arl
DE 0 nE
F--1
(D.51)
d3nlog"
1
Jgge D
> CVde CV
FP-1
P
1 oae A
CVdl Ft-1
+P
1
EAg[Fe-1
> C"v
Pe Pon log n
J(-1
P
1
o
A
Cvde
< Cod
0.8.
0.8.
99

---

Published as a conference paper at ICLR 2021
where to obtain the second to last line we used the definition of Ae, then used Lemma D.29 and (D40) to bound the first and second term almost surely.
We proceed to control the second factor in (D50), by bounding
[ Ft-1]
ege
L-1
gol
(1
=1 gad
II
1
Buir"
B re
B
1:ee
we
7
i=
sp(e-1)(u)
27
Be-lic") weee
< 1
gise f W
2
1
BeLe aaa
E W
tr
Bése" mrar
< 4 E
0.3.
B 200!
wi 22 (v)
tr
B 1:0")
1)
W
axx
Using D.38 and (.148) to bound the integrand above, we have
IP
1 egg
E
[
F+
C
de dd++ d'n log"
") S C'e-od
)]
l
for appropriate constants. Combining [D.51 and the above bound gives
d log n -od
I Deg | - > Cy de
S C''e-c'
for some c, d', C², and using lemma D.28
E 11.. A1|=t cyde d' Inlog"n n P
cd
S SCe c'd
+PP [EE
b
< CCe c'd + C' CCm ccd
for appropriate constants. An application of the triangle inequality and union bound (and intro- ducing some slack to simplify the expression) then gives
P
Σ
E [1gAA. -
CLty Fn log ne od S Le od
1==++2
for some constants. Combining this bound with (D.47 and (D48) gives
P
Σ Ae > d nL nlog L + CLy On ne
I L' ded +e Le -cdd < cC" Le""
P
Σ Dr
Cdl Ln logo log L S L'e
l=t0+2
where in the last inequality we assumed K log L S d. Combining this with (D44 we obtain
-edd Σ A > C& CiP log² n log L < Tm e'd + C. Le -o---a d'd < Le eed
2=
for appropriate constants. This bound all the terms in the (D322) aside from the first one. The first term is bounded in (D.36), (D.37). and the fluctuations due to the last layer are bounded in D.3I). Combining all of these gives
n L-1
P
1 (x), 3 (2'2) (z1))-²
II
1
> Cd2 ( In n log L
i=
100

---

Published as a conference paper at ICLR 2021
SP
(
(x),
(x)) -
C
de
v
Ln log log L
max
4
C dp
+ PP
A
S 3
La Log n log L
+P
A
S - dl
Ln Loloog logo L
1=#1+2
+ P
Σ 1m#1422
EA
C & I 4
Ln n log
L' oo
after worsening constants. A final union bound over and assuming 2 K log L gives
L-1 n
((kKer O8 (0) o (z)
L-1
V Ln n log } 2 21-C'
II
1
Cd CdV
L--
π
for uppropriately chosen e, C² K. If we additionally assume n > L we obtain the desired result.
D.3 UNIFORMIZATION ESTIMATES
D.3.1 NETS AND COVERING NUMBERS
We appeal to Lemma C.4 to obtain estimates for the covering number of M, which we will use throughout this section. In the remainder of this section, we will use the notation N₂ denote the E-nets for M constructed in Lemma C.4) and for any 2 E Ng. we will also use the notation (x) = B( e)nMa, where D E {+,-}is is the component of æ. to denote the relevant connected neighborhood of the specific point in the net are considering. Here we are implicitly assuming that M are themselves connected, but this construction evidently generalizes to cases where M themselves have positive number of connected components, treated in Lemma C.4 Focusing on this simpler case in the sequel will allow us to keep our notation concise.
D.3.2 CONTROLLING SUPPORT CHANGES UNIFORMLY
The quantities we have studied in Appendix D.2 are challenging to uniformize due discontinuities in the support projections Plç(.). We will get around this difficulty by carefully tracking (with high probability) how much the supports can change by when we move away from the points in our net Ne. It seems intuitively obvious that when is exponentially small in all problem parameters, there should be almost no support changes when moving away from our net; the challenge is to show that this property also holds when not relatively small.
Introduce the following notation for the network preactivations at level E, where E E [Z]:
p'(x) a-
so that (x) [p< (x))+ +- We also let F denote the olllebrra generated by all weight matrices up level in the network, and let F denote the trivial alagebra..
Definition D.1. Let A > 0, and let æ No. For E E [L]. a feature (a (æ)); is called A-risky if (p (x) < A; otherwise, is called A-stable. If for all Nc( we have
VÉ' E [e]. [I' p² (+) AA.
say that stable sign consistency holds up layer f. We abbreviate this condition as SSC(E, A) at æ. with the dependence on C, s, and A suppressed when it is clear from context.
If SSC(E) holds at E and if (a (x)); is stable, we can write for any x E N (x)
sign (0" (z)). sign (p" (+)), + (I0" (x)); (0" (o)):) sign (B))..
so that no stable feature supports change on (ā), and we only need to consider changes due to the risky features. Moreover, observe that
(((o(()) E (+4)] (w) (+A) F(-1] 0,
(D.52)
101

---

Published as a conference paper at ICLR 2021
where 9 (O (2/n)I) is independent of everything else in the problem, since A 0. It follows that when considering the network features over any countable collection of points Z E M, we have almost surely that the risky features are witnessed in the interior of +-]
Below, we will show that with appropriate choices of E and A, with very high probability: (i) each point in the net has very few risky features; and (ii) SSC(L) holds uniformly over the net under reasonable conditions involving n, L, d. We write Re (z, A) C [n] for the random variable consisting of the of indices of A-risky features at level with input æ E Ng.
Lemma D.5. There is an absolute constant > 0 such that for any & E M and any d > 0, if max [Kd,, 4) and A I d log n/(6n//21) then one has
I
Σ
Re A) d logn S 2n d
Proof. For any NN. Lemma D.2
(with suitable choice of d that context) gives
1 2
P
| (
1
C
so that if additionally a 2 (2/c)e e log(()) one has
P (+) 1| 2
S le -0
(D.53)
Let Ge {1/2 ( S 2). so that Ge is F'-maauurable, and G = n then by L We also let Go Ø For i E [n] and E [b]. consider the random variables Xxt = (p (x)).) and moreover define
D.53 and union bound, we have P[g]
Xie
Ia | 1
We have Ei,e IxuSA = (T),, which is the total number of A-risky features at ă, and the corresponding sum with the random variables Xie is thus an upper bound on the number of risky features at σ. Notice that Xie and X, are -e-maauurall,, and additionally notice that on G. we have Xae/ I Xxx S 2X For any K E {0, (0.1....,nu nL 1, nLJ, we have by disjointness of the events in the union and partition
P
Σ IxuSA>
K
S L--e-en/l -nnL
P
gn-
Σ
IxuSA =
k=
< 1.2e-enfle on/L
Σ
P
gn
Σ
Xus2s 1
k=
so it is essentially equivalent to consider the Xee. By another partitioning we can write
= }
L II
p gn
Σ
1
II
1
Su
i.
SE (0 |||
Using the tower rule and where (0, 1pnx1 is the set of L matrices with entries in {0.
Fl measurability of all factors with É L, we can then write
L
E
19
1 8
[=1
==1
1
S2s
Sie
Fh--1
i=1
(L-1
1g
520
=S
19,_
x s2a
Stn.
102

---

Published as a conference paper at ICLR 2021
We study the inner conditional expectation follows: because p'( Who (x),, we apply rotational invariance in the conditional expectation to obtain
1
E
I 1=1
1 & &
8iz
Flim1
E
I1
How w 52a Siz
Fliml
Li=1
IE
II
1 1E -
Fb-1
where ~ (O..
(2/n)T) is the first column of W and the last equality takes advantage of the presence of the indicator for I l multiplying the conditional expectation. We then write using independence
E
1 Ho b
JL-1
((w2)) Stcc FL-1
P (w) NSsA S.t F1-1l1 /
and putting PL P[](w))1 & 24], we have by ddeenically-diistrbutteeeee
II IP ((oo))ss Siz F1-1|
II
(
S..
i=1
==1
After removing the indicator for 9L-1 by nonnegativity of all factors in the expectation, this leaves us with
L-1 II
II 11
Sie & I
P2 (1 PL) 1-2 i=1
1 1=1
II
1
52a
Su
Su
This process evidently be iterated L additional times with analogous definitions we observe that the fact that all weight matrices W have the same column distribution implies that p1 = Pt, write p Pi henceforth and by this obtain
L
E
19
11 x II 1=1
S.L (1 p) 8.1, 18,1,
Am t=1
and in particular
1x =k | Σ 1,8
n
p
Pnn
Σ
p Su (1 -)
Sef0,1 =[
f=1
For i e [n]] and [L]. let Ye denote nL ii.d. Bern(p) random variables; recognize this last sum as the probability that Ei Yu = k. In particular, using our previous work we can assert for any > 0
P
Σ i.8
IxuSA
SP
SP
>
+ LenLL
conclude it suffices articulate binomial tail probabilities and estimate p. We have
2A
nt.
P[[(wr) 2A
exp
dt
2T
Jo
(D.54)
2n
and we can write with the triangle inequality and a union bound
P
Σ Yue
>
t
< P
Σ
Yu
> t
+
2
E|Y
i.8
i.
1.8
103

---

Published as a conference paper at ICLR 2021
By D.54 we have Exet BlY
LA.
We calculate using independence
E
Σ
Yue
B|Yad
Bi S n LA,
so an application of Lemma
G.3
yields
12/2
P
Σ
Yu - 1
>t
S 2exp
12/22A t/3
2,6
For any d > O, we choose / log n and enforce S dlog n/ (6nnn//2222) L), we obtain
P
Σ i,
Yu > dlog
2n
from which we conclude as sought
1l Ix.sa > dlogn
< 2n -d + 1
The next task is study the stable sign condition point 2 function of a and A, assuming A at least satisfies the hypotheses Lemma [.5] In particular, we will be interested in conditions under which can guarantee that SSC(E 1) holding implies that SSC(E) holds. Let Sr(æ, A) [n] \ ( A) denote the A-stable features at level with input æ. and define for 0 < SEST.
The PS((a) Pi(()) W' W'P ( Ple- W tíe' -
Ps ( Plent W W
(D.55)
so that &he x carries an input a (a) applied at the features level e' (in particular, corresponds to (x) x. the network input) to the preactivations at level L in network restricted to only the stable features at T. We can write
(2)) W a <())= (z)( W' (m) w (m) W
p (x) = W Ph (z) W
disregard all levels with no risky features: let (a Ao) be the number levels the network with risky features, and let E1 &2 < l, denote the levels which risky features occur. If no risky features occur at level l, we of course have Pss(#)) I. Assume to begin that é > ly, and start by writing
which gives us a useful representation 1l El_
p((x) -b (Ps.. PRe, (w)) Plr (w) bh (Pst.-(()) + PR... (2)) P
T 69.6
(Ps (x) + PRey (2)
P (xe) $..0
Now we distribute from left to right, and recombine everything to right on the term corresponding the projection onto the risky features at e,; this gives
p² (x:
L.C. (a) (x) + e. Ps (()) + PR.. (0)) Pie, 1(m)
(Ps (x) + PRe, (0)) Ple (z)) B4.0 '.0
We can write
t.l PR. ax)))()())) E.lr (x)
a (xx))) (z))
104

---

Published as a conference paper at ICLR 2021
where the restriction notation emphasizes that considering column submatrix of the transfer operator induced by the risky features. Iterating the previous argument, we obtain
pa(x) T x
+
f. Re,
(2)
(x)
(a)
Re, (x)
1=1
is clear that an analogous argument can be used in the case where & S e by adapting which risky features can be visited: we thus assert
p'(x)
0p000 x +
Σ
D= Re,
(
(D.56)
Furthermore, we note that under SSC(L - 1), no stable feature supports change on (z and so one has for every (x)
f.
so under SSC(e- 1) we have by (D.56)
p'(x) p((z)
4 e.0 (x
tlli L
a Rr, (a) (
(x)
(E(r) RECE
The ReLU [+] is 1- I-Lipschitz with respect to ||-1|xo. and by monotonicity of the max under restriction and SSC(E - have
aa (,,()) (a) a ax)))) Rr, (a)
s p' Re, (a)
p p'((z) 12 (0)
I p (x) p ( 2l S A.
Thus, by the triangle inequality, we have under SSC(E - 1) a bound
p' s se $
A
Σ
4.4
(D.57)
Rey (2)
:
4 <E
This suggests inductive approach establishing SSC(E) provided have established it at pre- vious layers-we just need to control the transfer coefficients D.57].
Lemma D.6. There are absolute constants e, C, C", C², C"" 0 and absolute constants K, K' > O such that for any S ll < l < L,, any d 2 K log n and any a E gno-1, if A - cn 5/2 and n 2 K' max (d' 1}, then one has
P
SC(L+
70
2 cod
18
f any fixed S C [n], one has
IP
$ <C
2 c"c
s
Proof. We will use Lemmas D.14 and D.23 to bound the transfer coefficients, let first verify
the hypotheses of these lemmas. our setting, the transfer matrices differ only from the 'nominal' transfer matrices by restriction to the stable features æ; we have Sz (ō) n lx(T) = [n] Re(w),
which is an admissible support random variable for Lemmas D. and D.23 and particular
(Ps((e) Pie(#) Pie(a)) (x))l22 = ( (a) 112 S Vnd
by Lemma Cammma/G.10 and the definition of Re( Additionally, using Lemma D.5 we have if 2 1, KdL, and < ed/n2// there is an event with measure at least - - LenLL on which there are more than d risky features at æ. Worsening constants in the scalings of n if necessary and requiring moreover d 2 K' log n and n 2 K"dd, follows that we can invoke Lemmas D.14 and D.23 to bound the probability of events involving transfer coefficients multiplied by 1.. Let us also check the residuals we will obtain when applying Lemma [D23 in the notation there, the vector d has its f-th entry Re (.), and have bounds ||d||1/2 Σ | [ and ||d|]]
Ee Re(
105

---

Published as a conference paper at ICLR 2021
which means the event &, the residual is dominated by the Cvd'mL term in the scalings we have assumed.
The l + f operator norm of matrix is the maximum l norm of row of the matrix, and the
+ l operator norm the maximum ll norm row. Thus
4.0
e 2
max ==
(W
max ==.....
where (Wt); is the i-th row of W which is ng-dimensional when 1 and n-dimensional otherwise. In particular, we have
Φ 1.0
(W*))ll> y
and so taking a square root and applying Lemma G.2 and independence of the rows of W we have
1.0
2(1
2ne
12-480
for c an absolute constant. When > 1, we can write
(W';
TT PI(()) W 2 El...
max E Ty
max
W
(W P (
max im Teoeh
where the second line applies Sauuchy-Schwarz Using, say. rotational invariance, Gauss-Lipschitz concentration, and Lemma E.48 (or Vershynin 2018 Theorem 4.4.5)), we have
|| > C(1 +
S mm
for absolute constants c, C > 0. On the other hand, note that (W"), Pr, (a) llz has the same distribution the square root of one of the index-0 diagonal terms studied in Lemma [.233 in network truncated at level é - instead of and scaled by 2/n; and so applying this result together with union bound and the choice max K'
K'dzL) L) for absolute constants K, K" > 0 gives
P lg max
(W*); WT
||, > C²
Ce d'd
(=
where C², C.C,C"" C > O absolute constants. We conclude by another union bound
P r||05
1 1= <
c
no
- on - C'ne-d'd - C²d L'eeeed'
We can reduce the study of the partial risky propagation coefficients to similar calculation. We have
[Ww)) ( II,
max
s 10 j=I...,m
where by construction have that E > have
l". In the case f = ll + the form is slightly different;
+1.6
(we+z)
2
( () I
max ==
max s=I....m
10 100
where the inequality uses Lemma G.10 The classical estimate for the gaussian tail gives
'2d
( (2
P
< 2e
(D.58)
for each [n)], so a union bound gives
(we'+1)
P max
< 2ne
jl
106

---

Published as a conference paper at ICLR 2021
and conclude
/ 2d
+1.1
2e 2e-a/2
4/2
s sippa
where the final bound holds if d 2 2 2logn. n. Next, we assume gives
E> + 1. In this case, Lemma G.10
(wi) (T 1))
max
9 g0
f=1....f
[ ((T 1))
max jl....mm
For the second term the RHS of the inequality, we write
(W') TT
(W (W') TWi ek
max J=
max max J= RES
9
then apply rotational invariance of the distribution of (W*), and F -easurabbiityy of T Té-1. to obtain
TT 1.6
1. (WC);
max 2
max RES
max j=m Ly
max TT 1,0"
Ek
kEs:
20
Th 1.6'
max
max KES
ex
J=
where 9 (0, (2/n)T) is independent of everything else in the problem. We have by Lemma D.14 based on previous choices of n and d
p P[Le| T
S c 21 C'e on/it
ex
and (D58 applied to g controls the remaining term. estimates and partitioning with E, we conclude
Taking union bound over [n]] in these two
PP
max
(W')
(T-11' 1.6" 13)1 (Th
C
< 2ne d + n'ne--nn/ + 2e + I2 -en/L
and thus
P
ele' s slppp-pp
cls|y
C'e 10e
where the last bound holds if 2 2 log n. Choosing n 2 log n for a suitable absolute constant K > 0 allows simplify the residual terms in the probability bounds to the forms have claimed.
Lemma D.7. There is absolute constant 0 and absolute constants k. K, K, 0 such that
for any 2 K logn, if 2 K' max(dd L, 1), A < kn 5/2, and e < A + has for any & E Ne
then one
Σ LIR(E E d
(SSC(L) holds at Φ}
2 21-e
Proof. We start by constructing high-probabilltty event on which we have control of every possible propagation coefficient. For any > K log n, choosing A cn- and K' K'd' and applying the first conclusion in Lemma D.6 and union bound, we have
78 E [Z].
2.0
C1
Ce-
(D.59)
12-2
and under the same hypotheses, for any 1 5 l' < < L and any S C [n]. the second conclusion in Lemma D.6 givee
P
> C2|5
< C -6d 16d
pp
107

---

Published as a conference paper at ICLR 2021
Using Lemma D.5
we have if n max 4) and A < /2/n//2
P Σ RR A) d 2e + 1,2 on/L.
(D.60)
Denote the complement the event in the previous bound as E. On &, there are no more than d levels in the network with risky features. There are at most -==0 (x) ways to choose subset C [n] with cardinality at most [d]; using 2 and d 2 1, we have
dd
< 11 [d|ndd S 4dn
In addition, there are at most L ways to pick indices l < L. Using n 2 L, this yields at most 2dn 242d < n 8d items to union bound over, i.e.,
P
U SChnl (isIT
U
D
sel > C2lsly n
I
Ce
(D.61)
d log n and n 2 max K'd no). Denote the complement of the union of the events in the bounds (D.59) to (D.61 and & G; taking additional union bounds and worst-casing absolute constants, we have shown
P[G] 2 1 - Ce
we enumerate the levels the network that have risky features as 1 S 6 < .... < l, < L, it follows from our previous counting argument that G, we have transfer coefficient bounds (for any i E [L] and any & < E)
d
S C1
C2 Re, (3)
Rs, (m)
Now we begin the induction. Let E (T). For SSC(1), we have from the definitions
(z)))) Se s
I Ci(+ )r,
1400000
where the last inequality holds on G. So we have SSC(1) G if S ( V
Continuing, we suppose that we have established SSC(E - 1) g. We can therefore apply D.57 together with our transfer coefficient bounds get
To ci(( V
+ +2A
Σ &4
RR, (+)
V no E + C2A1
d3
< cG((1
Notice that the last bound does not depend E. Thus, if we choose E S A(2C ) and n 2 4C3dd, we obtain p²(x)) - p (a) < A; by induction, we can conclude that SSC(L) holds on G, which implies the claim; we obtain the final simplified probability bound by worsening the constant in the exponent.
D.3.3 UNIFORMIZING FORWARD FEATURES UNDER SSC
Under the SSC(L) condition, we can uniformize forward and backward features. prerequisite of approach, which also used establish SSC(L) in the previous section, is control of certain residuals that appear when a small number of supports can change off the nominal forward and backward correlations. These estimates studied in the next section, Appendix D.3.4
In previous sections, most of our results (e.g. Lemma D.I. feature lower bound of the type n 2 K in their hypotheses. After uniformizing, will discard this hypothesis using extra assumption
108

---

Published as a conference paper at ICLR 2021
that no 2. 3, which gives lower bound the logarithmic terms of the form log(Cnno) that appear as lower bounds on d after uniformizing. and the fact that our lower bounds on n always involve polynomial in d. Thus, by adjusting absolute constants, we can achieve the same effect as previously.
Lemma D.8. There are absolute constants G, C and an absolute constant K, K' > 0 such that for any d z Kdo Ogg(nngCM)) if 2 K'd then one has
P
[SSCLL 3no 3n0 -1/2 On 3) holds a n
Σ Rela, Cn Cn- {d]
aEN 3mg
1/2
2 21-
Proof. Following the discussion in Appendix D.3.1 if < S then |N.l e log(CM//E); to apply Lemma D.7 we at least need A < kn 5/2 and s K'A (1 v 1 so it suf- fices to put A = Cn when is chosen larger than an absolute constant, and require & I Fixing & = =n 2 1/2 which again is admissible when n is sufficiently large compared to an absolute constant, for any d Kd (og(nn)Cm) we choose
min 1, 1// 5/2 (1 + V
K max dt L and take union bound to obtain the claim (using here that CM 1).
O
Lemma D.9. There is an absolute constant e > O and absolute constants K, K', K" > 0 such that for any z Kdg log(nn)C(m), in 2 K& then one has
- 1|5 1) 21 21-e-ad
WR E (T),
LEE
Proof. Let σ E N mno Lemma D.2 and union bound give
1
P E ||)a(()) - 4
< C'T.2e-oup S se-d
if n 2 KdL and d 2 K' logn. If additionaly d > K'do (og(nnoCM)) obtain by the discussion in Appendix D.3. and another union bound
e [4] |||)/()]|2- - -1| 1)
P
U
S -
8EN
3Po
Let E denote the event studied in Lemma
D.8 choose d > Kdo log(nngC'm) and n sufficiently large to make the measure bound applicable. A union bound gives
PEU
U
3 [T] [ -1/>
11 4 Se-
8EN 1/2
Let G denote the complement the event in the previous bound. For any x E M. we can find a point T E N A 1/2 N. - (x)) On G. SSC(L, 1 -1/2 On -n- holds at every point in the N
a-an, 1/2, which implies that on
C n3
vl [4] |p(x)) p')
(D.62)
and by the 1-Lipschitz property of [.] and Lemma G.10 this also implies that on G
A4 [L]. [])((x) a
a'()))l < 5//
109

---

Published as a conference paper at ICLR 2021
Choosing 2 (4C)2/5 the RHS of this bound is no larger than 1/4. We write using the triangle inequality
| - - |||aa())) (z)1l2 - || (z) +
- 1/.
Using the triangle inequality again, the first term on the RHS is no larger than 1/4 for any on G. The second term is also no larger than 1/4 g by control over the net. We conclude that g
1 vl E [2], [|)aa()))) - 1 2
This implies that the event is contained in the
VA E [L], (x) 1| 1| 11
n
-5
3EM
which is closed, by continuity of || |/a and of the features function of the parameters, and is therefore also event. The claim follows.
n
Lemma D.10. There are absolute constants c, C. > and absolute constants K, K' > 0 such that for any d 2 Kdg log( Mg) if 2 K'd L,, then has
d3 s sCV
P
vP E [2], (a (a(()) (a') cos
2 1
Proof.
Let a, I' E EN n Lemma and a union bound give
P p
cos 60 (< و'1)| > Cv. d3 (a'()) (
S Le Le S se-dd
if d > K logn and 2 max(K' K" K").. If additionally d 2 Kdo log(nnno we obtain by the discussion Appendix D.3. and another union bound
d
P
U
36 e [4]
Ka' (a) (())) cos
-os (L.))) CV
(
where with abuse of notation write Sx2 to denote S x S for set S. Let & denote the event studied in Lemma D.8, and let &g denote the event studied in Lemma D.9. choose n sufficiently large to make the measure bounds applicable. A union bound gives
P P
U
U
[ (a(()), (a cos cos (<0 ON
d
nh
N
0e odd
after adjusting constants. Let denote the complement of the event in the previous bound. For any (x, x²) E M M, we can find point æ N 11122 Nn-an=17
O N (/-2 and point
1/2 On 3) holds at every point in the a' E N. // NWW (=-1/a((x)) On g. SSC(L, n - net N nn 11/2, which implies that on
C
C
A [L], [p((x) - (z)
and Vl E [L], ||p((x²)) p'
n
no
and by the i-I-ipsciitz property of [. and Lemma G.10 this also implies that on G
C and
C
3A E (7), [a'((r) a<)))12 <
v8 E [4], ||a - a (('))) <
(D.63)
5/
5/2
110

---

Published as a conference paper at ICLR 2021
For any & [Z], we write using the triangle inequality
(a) (xd') a
cos 9 (L(0, ))
- (z)) (a)))
(a) (a' (a')) (a ( (σ') cos
(D.64)
cos (0 (<(2
cos の (0 (<(x, cos x (< xx))
Using Cauchy-Schward we have on G
2C'
(a''
(2))l2 < o (d)) < a<(()) - - (1())
n2//2
with the same bound holding for the second term (D.64) by an analogous argument. For the third term, we have on g
(a' cos -0 CV d
For the last term, we use I-Lipschitzness of cos and 1-Lipschitzness of the (l,, which follows from Lemma E.5 and the chain rule, to obtain
cos ( (L(ā, x)) cos cos at) a²)) s |[ x') L x²))-
Using Lemma C.7 and several applications of the triangle inequality, we get
(x) (xx, x S / | x' 1l2 - || x' 1|2
- v ( x²) (a -x')1l2
2v2 n2¹
S V x 2112 + V21l0 æ'1l2 1
and so returning to (D.44). we have shown
de
C' n5/2
di
<a< (x) a (z')) cos (<(x, S CV
sc+
S
for every f E [L]. This implies that the event G is contained in the set
n
ve E (4], (a' (a), a (a')))
cos t0 (<(x)
SC
d
n
(M,a))MMMM
which is closed, by continuity of the inner product and of the features as a function of the parameters, and is therefore also an event. The claim follows.
Lemma D.II. Assume L, d satisfy the requirements of lemma D.10 and additionally d 2 1, n 2
K L some K. Then
P P[llfoulle> I va 2 cd
[ S val 2 add
Define
{(x) = -((()))())) + (oo
M
Then under the same assumptions
PP
|
d sV Lm
+ d5/2
2 1 - c
L2
for some numerical constant c.
111

---

Published as a conference paper at ICLR 2021
Proof. At some M, we note that
Sa (z) w
1+ (xx))=)))))))))))())) - y |1)a(())|12
(D.65)
where g is a standard normal independent of the other variables in the problem. Similarly
fo, (2) fSoo (x')dx²(("") = WL+1 (a((m)) -
M
M1
(D.66)
o' ax)) (a'dd')
M
where g' is also standard normal. With respect the randomness of WL+ these two objects are Gaussian processes with variances | 1 and We next note that
[²(xx) fo [ (a')d respectively.
M
at (x) f af (2'))
/ aa(()) - -a')d'
=
M
M
S [x)(x) a" (x d du (x))
M
/
a (x)) 11 aa (x') )|ا (a+( du
||a + aaa (x || (at( al))))))())))
$ sup aEM
a (z) 2
+ +Ila"(x')) (xx) 2
$ sup #EM
(2 - 2os sp (L) (v(ar x ))
+ 2-220 cos s (v(2,
where the first inequality comes from an application of Jensen's inequality. Assuming n, satisfy the requirements of lemma D.10 and denote the event defined in it by G. On G, angles between features concentrate uniformly around a simple function of the angle evolution function ye, in the sense that, for all E M simultaneously.
a - (w) 1
(x') (12 2 (a)( (x) - (2 cos s (D ((xxxx)))) ||
| (x)
||aa" +2 (x)) (x')) cos (1(xx
(D.67)
L3
<C'v
for some constant C'. From lemma C.12 there exists constant Co such that for all v E [0,
1
0 ss s (M) < CoL.
Using cos x 2 1 - x²/2 and the above bound gives
1 cos (1) ((0()) (v))
1
20
and thus
1
2 2008 4 (v (x,,xx))))
S
2..2
112

---

Published as a conference paper at ICLR 2021
Combining the above bound with D.67 and recalling the probability of g holding, have
d L
I
sup
|||aa(x))
(x) l| + ||a" (a [22) 2 2( (a"(x), (x),
+ +C
S od
GL22
M
(D.68)
On the same event we have
sup .E
aa (
(a:) (2: 122==
ee
Thus on & the variances of the Gaussian processes [D65
(D66)) are uniformily bounded by 2 and
1 C' LL respectively. Writing for concision in the subsequent expression
(e) - / (x" )du (x')
vd
1
L
E.
S 2
12
M
laking a union bound over all points on the net N
n 1--n11/2 1/2 gives
vd
P
[e, (x) S
nE,
g
2
TEN
Bug
1/2
(D.69)
21
N
3no
1/2
od 21
21-
for some constants, since chosen to satisfy the conditions of lemma D.10
In addition, we see from (D.33) that on the same event, for every æ E .M there exists æ E N n- such that
(x) - [z)]
= fo)
(zd(2)) fo. fo.
(2d
M
M
Wh+1 (a'(x) - aa'
C WL41
WL+1
a (x) - a² (
5//
Bernstein's inequality also gives P complement this event by we have that on Eng, for every a E M there exists æ E N such that
W ++1|| > Cvnl -n for constants. Denoting the
n-
C²
Vd
(x)
I n2
S
2
[oo for (2:)
f M
/ (0 (dp))d))((
C²
Vd
- So, (x)
I S
+
n
2
L2
M
where we assumed d 2 1, 2 KVL for some K. Combining the above bound with (.69 taking a union bound over the complements of & gives
and
P
{ [1. (x) s vajn n
(o)(()) - (2e (2"))
d
T d5/2
1
m
ØEM
21 e - P Pg P B
> c d'
c n'
c
-
From the first result, we also obtain that with the the same probability || worsening the constant in the tail simplify this to |[|| Vd.
$ 1 Vd. By
113

---

Published as a conference paper at ICLR 2021
Defining
{(x) -aa(r)) + / soo (ad))
M
since ((x) ((x:) (2)) S So (20' (d (x'), it follows that
M
P
| - < 1m $
d
L
2
cd"d
1
n
Lemma D.12. For some integer do, assume n. L, satisfy the requirements of lemmas D.8 and
D.14 meaning that there exist absolute constants K, K' K" K" K!" such that for any d Z max do M)) K" log L.), if 2 max{K" d.,, K" I lo n. K then
1. do 1 and n 2 K"" max (LL
1/3
2/5
where K"" is some absolute con-
stant. K and сλ are the extrinsic curvature and injectivity coefficient defined in Section 2.1 then on an event of probability at least 1 - - one has
| IMollMall Lip
<v
for numerical constant c, and where the Lipschitz seminorm is taken with respect to the Riemannian distance on M
2. If M -2o-1 so that do = ng then on an event of probability least 1 e -cd one has
fl - S vd
for numerical constant
Proof. We recall
fa. (x) w
Let M. denote connected component of M. Let æ1 M.. and fix smooth unit-speed
geodesic : [0, dist., x2)] + M. such that 7(0) Φ1 and (dist (æ1, x2)) = æ2. The absolute continuity fea M y follows from an argument almost identical the one employed in the proof of Lemma B.8 and we obtain in particular the bound
Ife. (x1) - (x2)) distM. (m1.20))
("'''""'''
Because y is unit-speed geodesic, we have for all
= P,, = (+((t)
and in partieular, by the triangle inequality and Cauchy-Schwarz
[o, (m1) - for (x²2) & dist M. (æ1, x2) sup mEM.
Pr.M. (W!) )0)))|
(D.70)
This implies
Nool.v. Lup < sup
PILM 3'
sEM.
We next write
xx=x=="=++" H W W xx' W xxx) =
M,M can be identified with a linear subspace of Rn of dimension do- Since it also a subspace of T. g%0- PraM. and hence
PraM, W PraM, H PraM. (I axx'' Wi Pri.M Wi
114

---

Published as a conference paper at ICLR 2021
where W is copy of W that is independent of all the other variables in the problem (since (x) depends only on G!) ).
We first consider the dg no 1, and subsequently the case do 1. We note that
PraM, W W (() W
Σ (W:"
(D.71)
Considering single summand in the above expression, repeated application of the rotational invari- ance of the gaussian distribution gives
W./h (w))) 2
_ 2
2||30
(x) 11 Ii
2
W p2:2 (x) ) Pr, (cxil (eiiill2
2
W
p
11 )iz(())
71
d 2
| (w) ( W 112 11 (
2
| (x) 1| w 112 112 )i
(D722)
where (i,T(m) is standard normal variable that depends on and the support pattems I(x) (Ti (ar)...., To (a=),, since (az) depends only through I(æ:) Similarly, the dependence
in the first factor (.72) is only through [(x). We now show how to control such terms uniformly on M..
Define a net N pn as in Appendix D.3.1
According to Lemma C.4
S
3pp
e3 (ogg(( nna)do Assume n, L, d satisfy the requirements of Lemma defined in that lemma by We also define sets of support patterns
D.8 and denote this event
E
[(x) J [Jr (JI....,
J2
J
(D.73)
J(N 33 11/2) =
U
J(z).
TEN
On E, U (I(x)) C J(N - (12 and additionally for any E M., then there exists some aEM.
E E N Nn-=nn01 1/2 such that N, - T and I(x) E J We now show that on E, the with 5, = d, K2 552 with the anchor point in the statement of that lemma chosen to be æ. The value of 8, is satisfied by the supports at every point the manifold on & from the definition of this event. From the definition of the stable sign consistency property (SSC) in Appendices D.3. and D.3.2, the only features whose sign can differ between T æ are the risky features, and from the bound on their norm in the definition of SSC(L, n -nn
supports I(x: satisfy the requirements of Lemma with
1/2 -1/2 Cn 3) we obtain for all
C n
C
(P PI(T)) p
T
S
(Pss
PI p ( (T) S
1.0
n5/2
where in the last inequality we used Lemma G.10 It follows that if 2 K Ld for some K, the requirements of Lemma D.14 are satisfied if we make the choice E = Ešk-
We would next like to apply Lemma D.14 for every possible support pattern in J(N -2 1/2)
which requires that we first bound the cardinality of this set. Note that J (x=) is the Thus
d
[4]
Σ 1=0
Ln
s [d]
eeLn" [d]
$ CL
i
115

---

Published as a conference paper at ICLR 2021
for some C. Using the bound on the cardinality of the net from Lemma C.4 the size of this set can be bounded, giving
[
J(N
a-8 1a2)
N n
1/2
i
I c log(CM, uno do + log(Ln)d
(D.74)
1=0
We can now apply Lemma D.14 with & = εS to bound union bound over all possible supports. Bernstein's 's inequality and an exponential tail bound can be
( 11 for all é on &, taking
used to bound the second factor and third factors in (D.72) respectively. Using (D74) to bound the
number of supports we need uniformize over (since on 3, U {I(x)) - JNV a/////) and
sEM.
the bound on the size of the net in Lemma C.4 and Appendix D.3.1
we obtain
YE E M. P
1s
||
L:2
(x)
2 W
2
I S
=P
VI N
n 1/22
11
2
| (x)ee
W L+1 12 12 S
Vo E N n- (2)
(D.75)
21
N Nn-onm-/1 1/1
Clog(nid
N ~ 1 @loggnndd dd'n
21 Blog(CMM noo)do Clggn)dd-
wooooo Clo(n))d-ddd c"n
21
where assume d 2 K Klog(CM, and n 2 K'La for some K, K' Since according to
Lemma D.8 the event & holds with probability greater than - edd for some c, we can remove the indicator in the bound above by assuming d 2 K for some absolute constant K and worsening the constant in the bound.
We can now complete the proof for do no 1. Since we are interested in bounding the sum (D.71)
uniformly, we can bound Σ 9 (()) as well using Bernstein's 's inequality and uniformizing as above
obtain
P
Vo E M.
2 1
(2)e
Wb+1 b+1 w
2
c T(x) S C(no + d)
1=1
=P
Va E N
Vo N " 3 1/122
n-3n 1/2 1/)(2)
2 2 -s (2:2(2))
2
W W L+1
113
Fr S C(no + d)
1=1
21 N Na-3p
c (og(n)dd-- -07 - N n 1/2
c Clog(nndd -e'd c -cnn
21 Blog(CM, ano)d +C log(a)d-e4 Coggaa- 3log(CM, ano)d C log(n)d- -c''
21 - 1--c-e" dd'd
where we assume d > K llog(MM mon)do n 2 L'L for some K, K'. As above, we can remove the indicator in the bound above by assuming dl 2 K for some absolute constant K and worsening the constant in the bound. Worsening constants in the failure probability, we can replace the residual in the above expression by d. Using (D770), we obtain that with the same probability the Lipschitz constant of fe, on S is bounded by Vd.
We now consider dg 1. Recall that for any æ E M.. then there exists some N that æ NV... the form
- 1/2 such n-3ng 1/2 is the net defined earlier. The gradient vector at takes
-n 11/2 ( where N
PR.M W 3d (x) PI W 3d + (Pr.M PI W 3x()..
(D.76)
We proceed to bound the first term in the above equation uniformly over M,. Since in the do = 1 case T,M, can be identified with linear subspace of Rnne of dimension 1, write the
116

---

Published as a conference paper at ICLR 2021
projection operator for some unit norm Ug. We then obtain from rotational invariance of the gaussian distribution that
PixM. = 1
(D.77)
Pram W (
W
3 (3xx)
U U
2 (xx))
11
n
2
r 2 (z)e 2
w L+1
2 -
where IR is standard normal variable and the last bound comes from calculation identical to D.72p. Under the same assumptions on d, L, n as before, we can bound this expression uniformly
using D.75). and additionally take union bound over the net account for all possible choices of
Σ. This gives
P
VI E N a 3nD 1/
21 L n
| (x))))) |
2 S
We E N nn 1/2 (3)
21 Nº 8ng 1/2 e edd
21 e 3oog(CM, mmo)do-e'd
21 -cdd
where we assume d 2 K log(CM,, nmo) for some K. This gives control of the first term in (D76) uniformly on M.
We now turn controlling the second term in (D776. For some æ, choose E N 12 n-3nn 1/2 such
that a E 2 (z). Define a unit-speed curve 7 [0, s] + MM such that 7(0) = V, 7(s) x.
Since the curvature of M is bounded by K, have
vs [0,s]: || (22) k.
Denote by the geodesic distance between and æ'. Since the euclidean distance between them is bounded by n = 1/2 assuming n > K for some K implies that C'n C'n-3ni 1/2 for some C².
If we now demand n 2 C" which implies
C' 174
S
(A.)) gives
C
n
1/2
for some C > O. For T., vd defined as in D.77 we have y'(0) = ,0)))(s))))) = -² Combining the previous two results, it follows that
/ x" (s')
Cs
10 x (10)
S S
1/2
n
straightforward calculation then gives
|| P Pi 00 -0
|/000
v011/2 1//1000 + t
Cn
3 1/2
we now use Lemma D.13 to control the norms of the backward features uniformly, a standard bound on the norm of Gaussian matrix to give W C C(+ <m and assume
n 2 2/5 we obtain that
PP V E 3n-
N -1,/22 &
-1/2,
n3 11/2 1/22211))) (T)
(Pr.M- - P W (x)))(()) sc]
- 1--e-od cd -o'n > c
-c'd d'
117

---

Published as a conference paper at ICLR 2021
Combining the above with (D.55) and using (D.76) and taking union bound over the failure prob- ability of which results in a worsening of constants completes the proof. We can additionally rescale d to obtain a final bound on the Lipschitz constant vd instead of CV d, which also results in worsening of constants.
Lemma D.13. There are absolute constants c, C and absolute constants K1...., K4 > 0 such that for any d 2 Kdo log(nngCM). if 2 K' a L, then there exists an event & such that
On E, we have
L-1 II
gl ((x,
ve [2], (x),
<CVd'nL.
E=t-1
simultaneously for every (x, x') ) M M;
2. [P(E] 11]21-6-0
Proof. Let E denote the event studied in Lemma D.8. with Co denoting the absolute constant ap- pearing in the SSC(L) condition there; choose 2 K do log(nngCM)) and n sufficiently large to make the measure bound applicable. We will need to apply CemmmaD.23
with a derandom- ization argument to prove the claim; we appeal to the same residual checks at the beginning of the proof of Lemma D.6 see that on & the dominating residual Lemma D.23 under the scalings of d and n we enforce here is of size CVd'nL.
For any subset S c. [L] [n]. write S ii [n] (e, S}. and define S(S) (-1,+1)15:) x...X {-1,+111ss for the set of "lists" of sign patterns with sizes adapted to these
projections of S, with the convention {-1, +13 = {0). If = for.... to....,, oL} E S(S) is such list of sign vectors and A 2 define
(x, E, A) = supp (1) ((o)):A)e:)
which is sort of two-sided robust analogue of the support of a (x): notice that when S = have (x, S, A) Tt(x)) We also define for e= 0.1.....
( ( (wL+) (II(E,,,EE))) W. Pi PI
4 Pi
generalized backward feature induced by these robust support patterns. Writing for concision
( G.n. (0)) S.E. con-nonn
Ja = AE E [L]
TT-l (1- 21
Ci Ci
where Ci > is an absolute constant we will specify below to make the event hold with high probability, we then define the even [T]
Eg
U
U
BEN zEN -a a' N'EN
Sc[L|x(s] DES(S) s'c[L]x(n] Y'S(("
There are no more than E (n a fixed there are no more than 2d ways to choose the sign pattern E. Thus, there no more than
4d ways to choose the subset in this union, and for
"To see that this set is indeed an event, use that 3 (xr) is continuous function of the network weights except with respect to the support projections; hut x: 1 2>0 is increasing, hence Borel-measurable, and so the consists of finite union of Borel-measurable
118

---

Published as a conference paper at ICLR 2021
exp(10d log n+ 12d, C))) elements in the union, and under the condition d this number is no larger than n ldd For concision, write
1 (((x x¹))
(( x')
n
2 F
For any instantiation of these parameters, Lemma D.23 and union bound give
P 3 E [4] (......m==( // D.''.CCnn- con-s ( x') > Cva'nI
< P[Ef] + P [L] Me ( (2)), 8 (0,x"CCnnn Er- 1 (2, x¹)
CVd'nnl
Sea
for any d 2 K llog and n 2 K'd+L. Thus, if we C1 C and enforce > Kd log(nngCm)/ log and max K' Log' have by union bound
Pl [&]
S
Let G = = Es. For any (x,x²) E M M, we can find a point I E N Nn- N. 1/p
C.CCn holds at every and a point x' E N Bn NNW, an ( On G. SSC(L, 2nj 112
point in the net N a -n3 1/2, and there are no more than d On 1 8-risky features at any point in the net N Nn-00g1/2, and in addition, following (D52), we have almost surely on G that all risky features are realized for magnitudes in (-A,+A) +A). This implies that on G, the support sets LkeErr) point æ E N - (x) differ by the support sets LiEE[ Iz(ā)) at the base point in the net by no more than d entries, consisting only of subset of the risky features at æ; the analogous statement is of course true for x' and ø'. At the same time, notice that on the event & we have constructed, we have control of every possible backward feature inner product obtained by modifying the supports at the base points æ, s' at no more than d risky features (each), since, for example, if (pf (æ)); is risky, then I (x)())) corresponds to "turning off" the feature, and I (()()))) corresponds to "turning on" the feature. Formally, we have established that on G
Ie(x)) at any
D-1
so(t)) (0()) 1
(xx
Vl E [.]
(x), (x')) (x)))
II
SC dn n.
π
p'=-11
We can use differentiability properties for the remaining link: following the proof of Lemma D.10 we have
2v2
<( <(x, 2 S V2lz x ø112 + /21lz' - x'112 - n so we just need Lipschitz property for the function q(v) (1/2) -- For this we appeal to Lemma E.5 which shows that the function & is smooth, increasing and concave;
therefore by the chain rule, the functions (d are increasing and concave. and by the Leibniz rule,
decreasing and convex. It therefore suffices to calculate q (0); this is done in Lemma C.18 which gives of (0) [n (2)(, and in particular [4'(0) < cnT. It follows
L- 1
pll)) (((x,z'))
n
T-11
(p(()) (4(x,
cL S
π
2
π
n
#'=t
='
so that by the triangle inequality
(L() x'))
vl e [4],
II #=
1
< 201 dLLLLlog n,
π
where the residual simplification valid when n 2 KL. We conclude that the set
{
N
Ve E [4], (x),
E (σ, x') < 20 v (d'nLlog'
(H.
contains the event G, which satisfies the claimed properties and completes the proof (after rescaling d by log n, which updates the lower bound d).
119

---

and Published as a conference paper at ICLR 2021
D.3.4 SMALL SUPPORT CHANGE RESIDUALS
In this section, prove generalized versions of our pointwise concentration lemmas for back- ward feature correlations and the matrices defining the propagation coefficients used in our study of SSC(L).
Lemma D.14. Assume 2 max L n, K'Ld, K" d > K" log L for suitably chosen K, K', K", K and integer L. and choose 1 l' I v L. Define an anchor point a E M and denote [,(x)= supp (ci (x) > 0) for E' sisl.
Choose some 8g, K2 > O and let - = ,Je....,JJe Jtl denote a collection of support sets such that each J; C. [n] depends on the network parameters only through the pre-activation (z) We define events implying that the supports at I are close to those at æ:
N (IJ [.())) S 5,).
esist
εK
N {Il(Ps, PI((a)) p'(2))l22 S K.)
esise
ESK = E3 Nεk.
Define
rf Ps.
PJ W
and fix unit norm vector v.. K, S I, 8, S 7, then
IP LEsK
[if u/|
0 2 -
and
|Esx | SCVT. 2 -c
For vector 9.9s ~ua (o, 1). defining H = W (I - a
(z)aa aI-aa-(z)a""
for [L]
r&e' = PJ, H', Ps H
we have
P [Lec
(
T
r&e 8/1,
CvdL se-a
for some numerical constants e,
Proof. In the following, will denote by vy g=-1 fixed unit norm vector and by v. S"-1 random vector uniformly distributed on gn-1 When there is no need to distinguish between the two we will denote either by Up.
r Vy with sufficiently high probability, and then apply e-net argument to uniformize the result (lemma D.20) and get control of the operator norm. In achieving the first goal, we will rely heavily on a decomposition of the weight matrices into terms that are conditionally independent given the pre-activations, We will also utilize martingale concentration to control the terms that result from this decomposition.
Our strategy in bounding L M will be first bound 1
Denoting S span { a (x)} for i [L],, we decompose the weight matrices into
W w Ps'P.-1 + w' G H'.
Note that (H ..... H4 are conditionally independent given o (G' ..... G (by which denote sigma algebra generated by G²...., G Since the pre-activations obey
p²(x) W arra-11p ( G'a
120

---

Published as a conference paper at ICLR 2021
and the features deterministic functions of the pre-activations, both (x)....,.a(())) and (z)).) , (x) are measurable with respect oG ,...,G',
We define events
E M (|le'(>)112 sc), Es EsK E
(D.78)
and aim control Ess, (T)((()) (a)) Since the supports depends on the weights only through the pre-activations and are thus also (G' Ghi-mmeaurabble, this truncation does affect the conditional independence of H ,...,, H It will often be convenient utilize the rotational invariance of the Gaussian distribution replace all occurrences of H' in given expression by W Psi-11 where W is fresh copy of W independent of all the other variables in the problem, which will not change the distribution of the original expression.
For E' sis l, S S ii+ it will also be useful denote
TH = PJ. H3
H
p GJ = Py. ..........
where we use the convention gives
rds+1 63
p
= Decomposing the weight matrices at every layer HJ
|1 up/12 = Ps, (G' + H²) ... ... ( + +H1))
||
E
P ... Ps. M Up 1p
(D.79)
MC......M. 8 8( HH)
We next define
(xxx) Pj. Pii(m))
(D.80)
In accounting for all the terms in the decomposition (D.79). there will be two simplifications that we repeatedly. One is
H Ps,p'(() witl (1 ( (--a()))()a))) +
H 1 (x))
(D.81)
where we used Pa Par()) (x) = (p(()) = axx), from which it follows that
H G =wi+1 I a'()))'(()))
(Pa + Q((z)) p p'(x)at-1(x)
|
(D.82)
H Q'
We also have
G Ps
-G (Pl.)) + (()x)) G'
-Wi+i (x))
(P1(2) + Q'(z)) w f- (z)a (x) (x)
|<<(x))11
l |a)))())11
||((2))11
1+ ('))'
(a--((x)];
|
| -
wit a a'
[x(()) -x)122)112
and thus
PJ W o (z)
P
GJ
8k
||o (x)]l
(x)]l2
8k
| (x) a |
(D.83)
k=j
121

---

Published as a conference paper at ICLR 2021
We refer to such product G-chain. We proceed to expand (D.79) into terms with different combinations of matrices 1 and I H.. There will be 2 terms in total, and we denote the set of terms with G-chains by Gr.p (with the subscript E {u, f denoting the choice of vector U,)).
can clearly label each term by the start and end index each G-chain, which may not be distinet. TP We denote each such term by 9(6 (io) where
S is
2 < 41 - 2 < is - 1 ... 12m 2m + 2 2 i2m 2m + +
I 12 2r
+ S 12r 2r + < 2r
(D.84)
The constraints above ensure that every two G-chains are separated by least one H' matrix. To lighten notation, we denote a set of indices obeying the constraints by (81,..., (2r) E (e, E"). The maximal number of G-chains possible is bounded by r - [e (()--()/3 /2/
Since the TP (2.22....)2.))) are non-negative, we have
((-)))/2
11 Eeso
r Up LESK
1//2
+ Σ
(D.85)
Considering first the form of the terms in Gy.p: using (.83) and D.81 and recalling that mm- 1.m I, we have
7.P Jac)
Priie HJ
p
152--2+1
GJ TAJ
Tee
r lif
Up
HJ
Priee pma- (2))
LEAP
p1 +1 HJ
a (x)
122 12
f2m42-1
122m+1 (x) T kemms 1i2um +1
Pr p 2m (x)
1 Ea KP
Sk
HJ
a fm (x)
a 2m (x) 2
k=12m 11+1
8 86001 (a:)
HJ U
k=
LEs
| (x)1l2
biam 2m 1:22m 22.5
(D.86)
The magnitudes the factors in this expression are bounded in the following lemma:
Lemma D.15. For āg, bagj. Ehe defined D.86.. R. V V2.RA Rg.R5
and l' s k <e, +25
j j+2sis < <sSise
āz
C a.s.,
aR
K.]
C²e
P
K2
C'e
VI
CR
-dd + ce-n
P
d
-c -d'd
for some constants d, C.C' C" and 2 0.
122

---

Published as a conference paper at ICLR 2021
Proof: Deferred to D.3.4
We will use these results in order to bound ESK
most F Up/ using (D.85 While the
of these terms can be controlled using the triangle inequality and the lemma above, there is subset which will require special treatment since they are typically larger. These are the terms where the leftmost rightmost chain is a G-chain (meaning ř2r = l or i l' respectively) and they will be controlled using martingale concentration. The above is exclusive, since we can bound terms with 12r l and ii !' using a triangle inequality. We denote these three sets of terms by G..ps G Gr.p. G rp respectively, and elements in them by TP, T.P "g for clarity when needed. Arranging the remaining terms into sets denoted Gr.p. the sum in D.85 decomposes into
[(--))/2]
Σ
Σ E
P. goPe@ppp
(EC....))))C.((..)))
Q Gr.p T
G.p.
G r.p
(D.87)
We consider first terms in & r.p (and hence with ägr = f). We denote such terms by
T.P
9
(11.2.... = aebe,
m=1
We show
Lemma D.16. For E {f.u] and Ry = V =
i
/2
P
Σ
Σ
9
< Ce cd -Ce--4
Ce-
(D.88)
9 g
(22
PP
Σ
Σ
CRp
Ce + C'e--''
(D.89)
7
for absolute constants C, C", and where d 2 K log for some constant K.
Proof: Deferred to D.3.4
Turning next to bounding the terms in & r,ps we first define an event
E [lāel S n
n
Kg (irstas 5
n
{leeel s CRp}
VT.
#4156 Sie
251s 256-2
and from lemma D.15 and union bound obtain
2d < L'C'e c + (2e -od =e-c'n) < -c-e-'t -c'd
123

---

Published as a conference paper at ICLR 2021
assuming
Kog L,
K' log L for some K, K'. It follows that
12
Σ
9
& T rip
p
18
Σ
biam 2 2m 1m Cigil
(er 1/EC. (0
(1-)))2)
p- 2
Σ
Σ
1 be
biam+ fam+ 1.zm Cia.
(22 1)EC, 1(9
m=1
[1--))
K 22
Rp
VL
(K,L3/2) 1/2
Σ
(k,3
Rp
1 -
RR S 2Rp
K,I3/2
where we used / Kg S 1.4 We also bound the number of summands in
E
by
(2
2 which is tight for small It follows that
P
Σ
2p
P
13
Σ
g TD
2np
17 r
G
mp
E C rm
+P P
(1 - 18)
Σ
>
T.P
T E G mm
P
3
< Ce c + 2e c'd
(D.90)
for appropriate constants. It remains to bound the terms in Gr.p by similar argument. Defining
K.
E
n PSi
(lō,, S K.)
N
S
VL
#422h
truncating on this event gives
11
Σ
Dr.
S
g
Σ
Trpp
<
(13/2K))
)
dL
dL
n
I..
EGr.p
Enpppr
and bounding the probability of this even from below using lemma D.15 and union bound gives
dL
dL
P
Σ
Drp
>
21
S PP
1
Σ Drp
>
21
+ PP
ErppGrpp
S Ce-
+ 2c o'd
124

---

Published as a conference paper at ICLR 2021
Combining the bound above with (990) and the results of lemma D.16 and worsening constants, the sum of all terms containing matrices G* is bounded by
Σ
PP
Σ grieE
gf
C
dL
dI. (((
Qnp
T Gro)
P
Σ
grip
RRp R₂)
Pr.p
Σ
Σ grie@np
grt
CRp
Qn Grpe Gnd)
SC - C Che-d'a
(D.91)
Bounding the first term in (D85 using lemma D.18 setting f aabve and choosing d = L gives
P L
| us
c s C'
We then apply lemma D.20 to obtain
P P[
> CVT
< c'e-dd
Recalling (D78). to obtain our final bound on the operator norm it remains to control the probability
of Ep- We consider some l' < i < and assume a²' (x=)(()) * 0 (otherwise |
|| S C with
probability 1). Defining orthogonal matrix R such that Ra = ||a-- (x) 1| ēl, rotational invariance of the Gaussian distribution gives
||0'(x))112 a' "''
o' d (x) || W
(11)
W. Il =
Since | (.00 2 is a sum of independent sub-exponential random variables with sub-exponential
norm bounded by C. Bernstein's 's inequality (lemma G.2 and n
D.2
give
P <[|p><>)11 > c] P [|0)
a (a-1(x))
(x) 1
C
(x))
C 2
+P +
(m)) (1)
C
< cen + C'e < C" -d
for appropriate constants. Taking union bound over gives
I 1 Le c > 1 C" -c"
for a suitable chosen constant c", where we used n 2 KL log for some K.
We then have
P LEa |TGj us
cl P 1ew -,
| 1//1 > cc + [Desnne; | 1//11 > 0]
< TP Leveee, D By cl
+ +P +P(Ea] s CC- + C'e-d12 s cc"6-""
for appropriate constants, and similarly
P
LEaK
CVI c""
This concludes the proof of the first two statements. For the final result, we consider a vector g with gi riid (0, 1). Bernstein's inequality gives
||g|| > 2n e en and since
d Uu ||91|2
g
125

---

Published as a conference paper at ICLR 2021
where vu is uniformly distributed on s--!, we can use D.91 setting p obtain
P P[I(TT'(() -H)( Cval.
SP P
(TT Let (a) ( tu Bull2 >
C dL
2 V
n
P [ || > 2/
Se m + Cee -' + C²e c S C. c-ddd
for appropriate constants, where we assumed n > LL for some K.
Corollary D.17. Defining
(x)
Pı, PI ... Pie W
.......P,.
under the same assumptions on n, L in D.14 we have
pe: till (w))o|
sc] 2 1-
P
[|| SCVT]
21 2
for some numerical constants c,
Lemma D.18. Fix a collection of supports J = (Jp... Jt} for 1 S E E that satisfy the assumptions of lemma D. 1.14aadd and denote by Up a unit norm vector. Define an event
ell' = 18s
-
Lill H To
o0|1, s ccnnnns
|Tw | c n
|
18s
pe:e" 12 HJ F
S
cCn
n 2 logn for some constant K then
P
21-C' CHH
where c, C, C' are absolute constants.
Proof. In the following, we denote by Wi an independent copy of W and by W (:g) the j-th column of this matrix. Note that due to rotational inyariance of the Gaussian distribution we can replace every occurrence of H' in an expression by W PS without altering the distribution of the expression, which we will do presently. We can repeatedly use this rotational invariance to give
T E:l' T HJ H. o H PJp
D +
I T PP, H Up
HJ
HJ
o P -1 We P,, p T P,, W P1
and rotational invariance of the Gaussian distribution gives
TT H
| 12 W (.11)
Ps
Ps W (:1)
HJ
HJ
|P se upl
P Psetn W
HJ
HJ
P3pe1
+1 se Pspy W (.1)
|P se' 1p112
Prr+u P.,, W. (..)
W
WC
P1 W
(s1)
HJ
H3
and continuing in similar fashion we obtain
2-1 | 12
Lit' up , ST
II
Ps PI. P,, (..1) |
Ps, W(,)) 2
im
Pı, W
(:.1)
0.8.
126

---

Published as a conference paper at ICLR 2021
where in the last inequality used the fact that multiplication by Pat cannot increase the norm of a vector. Denoting by {xi] a collection independent standard chi-squared distributed random variables where Xi has |J,] degrees of freedom, we have
e
II 1=
PJ.
(1.1)
Xi.
==
Define
[1((())]
E1
min
[1(x)]
2 0
Pssise
Denoting 8² = 1 4(x) on & we have
71
11
7
min
|/
2
2
(D.92)
el PSSs
4
L
8
2
2
(1T:) + 5t)
(174)
7)
II
[7:
(1+
L
L
f=
1=8
8=
i=
2 141
2 14
1
S 2
n
n
Y=
t=
where we used the assumption
8
and assumed n 2 8T. It follows that
L
P
II 2 X
11- i=
2
=P
Xi
Eg E
J
J
X
E3 n
J
26
An application of lemma D.26 and (D922 then gives
P
X
E
S CTe s -6-
(D.93)
J.
2
for appropriate constants, assuming > KL L for some K. Using D.30 to bound P [Ey] have
we thus
2
2
Jul
P
Ss o I i TEE
H. u
1 22 2
P
X
i=
[J]]
$
Xt
1+
11
Eg
71
1=
1=
2
< P
II
Xi
1+ II
Es E
+ [[E]]
i=
1=
< Ce Ce + -e--d' < C" -c
for some constants. Having shown
P
T off' U > cl -
H3
for some fixed up 2 apply lemma D.20 to obtain
L
rest' HJ
>
CVL < Le - < -d'
127

---

Published as a conference paper at ICLR 2021
where we used n 2 KL log L for some K. Choosing dp é, for i [n] and taking a union bound, one obtains
1
phe' HJ
2
>
On
P
Pleettr [ HJ
> Cn =
Σ
HJ T >
Cn
le, T H of S nC"e - Σ 1=1
for some constants, where we used n 2 K n for an appropriate constant K. A final union bound over the last three events gives the desired result.
Proof of lemma D.15 We first consider the terms äg. For l = e, the definition of EsKp in (D788 gives
D ())l((
(a)p((a))>l
ā 1EsK>
HT
1le || (
I C' a.s.
(D.94)
||at- (2x)l2
In order to handle the case 8' s k l, use (D8I) and obtain that for any 2 ssisish L,
p (x)
p' (x)
Tit Hi PJ,-1
- Pj, H P.
(()))
(x)2((x)122
(x)
miitl
Pj
H
(ar)
HJ
|
pitt1
PJ, W Psp 33-11 Q (x))
H3
P 101((2)()) (x) (a) 12
HJ PJ,
W
(:.1
(x))2
where W is independent copy of W and we denote by W used the rotational invariance of the Gaussian distribution. Truncating on the event E which does not affect the distribution of W (s,1)* we have
(s.1) the first column of W and
W (11)
E
p Ps, Depp-
W
(..1)
2
2
11
EH
tr
r 1 PJ,
HJ
m
2
pds++
b HJJ
oi
tr
HJ
71
2
p
C
H3
71
almost surely for some constant C", and the Hanson Hanson-Wright inequality (lemma G.4
gives
pig+1
PJ,
W
> 1+
HJ
(111)
n
n
< 2exp
-emin
4
1
pij+1 HJ
mis+1
H1.
1 HJ
T
p
< 2exp
-dd
j + 1)
for constant c', where we used
|ejp HJ
i HJ
HJ
HJ
s
Cn(i jj 1), and the fact that multiplying matrix by P,, cannot increase its norm. Writing for concision in the subsequent expression
pis++ HJ
P,, W (:.1)
11 1eg
pis+l HJ
PJ W (:.1)
128

---

Published as a conference paper at ICLR 2021
it follows that
PP
HJ Ps, W
Winnll > 0r] SP Leg
T Ps, W //1111 C"
+ PI > 0]
2 (-o")
11 I
+ C
11 Cexp (-m"Z) I C' exp (-o%)
for appropriate constants, where used D.I8 Since EsKi have
p'-1(a a
q p (z)
Psua Q2(x)
$ 2K,,
11002
i 2
| - (2)
obtain
P LEsKo
T H. P3
p (z)
20" CC'exp (-c7, 12
(D.95)
|00 x 11
hence
? (-d
P f > K,] Cexp
(D.96)
L
for appropriate constants.
We now turn to controlling the byej. Note that
(x))l
t (x))" || (x))
EaKp k=1
= EaKp
| (x>"l(>)ll
II
1
Q² (x) Wz)
(D.97)
k==
3 2K ) S 322K5L <.a.s.
where in the last inequality we used 2k L 1. Additionally, we have
a'(x)*
p'(x)
1ESKP
T PJ
|<''))*11
l12
[ l2
a (xxx'' W Psi-21
(x)
d 1EaK
2-2:1 PJ;
HJ
(x))l2
a'xx)' P3,-1 wi-1 d (xx"|1))
LEKK
u
og
where Wi- is copy of Wi-l that is independent of all the other variables, defined
p (x)
u = Pgi-2
2:j+1 2ij2++1 PJs
HJ
a (x) (x)
and g is a standard normal variable. In the above expression,
0² = LsKE 2 |el(a))111 11ll LESK
2 2| 2
Note also that r 1 is well-defined since 2 H.
We therefore have
(x))') 1--1: P.,
p (x) | (x)
CK₃ VL
P
LeaKe
(2)(1))11
HJ
2 K,
K2
P [1ESK] |/u1/2 > K.]
+ P
0
VL
< <C'e + 2c -c < cc
(D.98)
where we used (D95) and the Gaussian tail probability to bound the first and second terms in the second line respectively. Combining the above with (.97) we obtain
K,
< c
(D.99)
L
129

---

Published as a conference paper at ICLR 2021
We now turn to controlling e.. If 2 l' we have
j-1
II s
(x)
G,4 LESK
k=++2
rif H
(x))112
(D.100)
j-1
II 940 kmi+2
shaath
it (x)) Ps.
d LEsE
W Ps: 11 ri-le d
Up = og
1latt1()) (2))11a
where g is standard normal and
j-1
2
o² = LEsK
II k=1+2
Ske
Ps:
I1
r
HJ 2/112
S 1
2C
THJ Up
| a.s.
n
for some constant C' where we used (D97).
We also have
HJ Up 1
2
S Ö on E 1,
Lemma
D.18 and Gaussian tail bound then give
j-l
1-1 (a)) T 7
P
net
II
8k
a
(x))
k=
P
(Ieino 1
π Sk a (a)
+P
||o (a)*
k=++2
<
P
2
Og
+P (E
2e 2e- Ce -e
(D.101)
n
for appropriate constants.
Additionally, if l'
we have from D.97 for some fixed Up Uf
j-1
a (x))
S.00
LESKP
II
sk
(x)]l2 la"())l
r - 1:f.
Vs
k=
a (x))
LEsK
Sk l| L (x)
Uf
(D.102)
(2)
k=
< 9 a.s.
k=P++
If however Up v. is drawn from Unif(Sn-2), if we denote by vector with independent standard Gaussian entries we have
a (x))
d
I E |
(x)
Va
ggl1
From Bernstein's 's inequality it follows that IP tail bound gives
|| 1
n
S c
Combining this with a Gaussian
P
<
P
|g
n
+
P
Igil
d
< e
cn
2e 2e-c'd
-c'd
1gl12
2
2
for some constants c, c'. From (.97 it follows that
d
j-1
a (x)
d
II
sk
(2)22
Vo
>
S e
+ 2e -ddd
n
for appropriate constants.
130

---

Published as a conference paper at ICLR 2021
Proof of lemma D.16 Part (i). We denote the set of all such terms with G-chains by T r.p. Con- sidering first the contribution from the terms with single G-chain, denoted Gip, we have
Σ
Σ
2
9 T1PEE
jm841
where
35.0e (x)
(T)
H.J Up
Σ
EsK
1+
a (z)112
==
j=f++
Denoting by (A 1 ..... A&) the sigma-algebra generated by a the random variables 41... Ag, we define a filtration
x (Up p' (z) ..... (z(x))
(D.103)
FJ
Ups p (2) (a:)...... pp H ,..,..,,") ......,))) je......
The sequence (Xi}
Σ
is adapted to the filtration, and since the summands are linear
j=H+1
in the zero mean H the sequence is martingale (E X Fi = X2). The martingale difference sequence is
πI k=
Spa
(x) H3
A X; -X-1
2,,+1
1EEMP
||at| (x)]l2
giving
E
f-1
E-1
Σ j=P+1
T. X
Σ j=8+1
A Xx
Σ
A
2,411
1=
We cannot control this directly because do not have almost control of the martingale differences. To remedy this, we recall the event ci-1,e" defined in lemma D.18 and decompose the sum of interest into
H
A
S
Σ
10-18411 A +
Σ
(D.104)
j=f'+1
j=''+1
-=
Notice that the second sum is also sum of zero-mean martingale differences. Using (D.100). have
Ae-1d-10g Eq
E/
|
Up 1-f' 1/
where g (0, and 1l E/
n 1 1.0
II Sk
almost surely
for some constant C. It follows that
E
exp
Al A A.)
زーー exp (enx²) VX, a.s.
and we apply Freedman's ' inequality for martingales with sub-Gaussian increments (lemma G.7 to conclude that for some d > 0
dn L
TP
A Vd < 2exp
-c
As for the first term in (D.144. using lemma D.18 we have
t-1
P
Σ
A
1
1e
A
>
0
< P (ct-1.C4))
LC'e-'t CCe
E
131

---

Published as a conference paper at ICLR 2021
for appropriate constants, where assumed DD94). and using (D.IOT) to give IP [ > the triangle inequality, we have
K log L for some K. Combining the above with
Ö I C'e-ci for some constants and applying
IP
Σ
T
> CVd
SP
T [
+ lTel
Σ A
cvd
(D.105)
17 PPPG
j=++1
S Ce + ---
for appropriate constants.
Having controlled the sum of terms in G lps we next consider a sum over the terms G Grp for 2 2. The argument will be very similar the Gip case, with some additional technical details.
Note that since different G-chains must be separated by an H' matrix for some i and we consider only terms with 1 2 + 1, the minimal starting index of the r-th chain (indexed by j below for clarity) is L' + + 2(r 1). The sum of all possible terms is thus
Σ
T =ū
Σ
II
di am f1 12.51
e' 2r - S j S 2) C (L' + 1,j -2)
(i
PE.
f=
The constraints on the indices i1. reflecting the constraint i l' in the definition of Gr.. We once again define the filtration J as
22 are similar to those in DD84 with the starting index
k-1
at [m)"""" T 1om+1 P" (m)
in (D103) for l - i S. l. Noting that āe, be,L,m = 1EAP
II
Sq
(x) (2))lal1ll (2)]2
and
g=1+1
11 8, (a) (T) vy w-(11
k,1 = LEEP
are all Fi -mesuurabl, the index constraints imply that
(w))22
PC.
is -e-aauuuarble and thus the sequence {X]) is adapted to the filtration. be is linear func- tion of the zero-mean variables H' for any choice of i2r-2, and we can replace H' with W' Ps--11 where W an independent copy of W without altering the distribution of XÇ. Since beim for < 12r-2 independent of the Wi for any choice of I, m, it follows that Pi,++ is also a linear function of the variables in W which have zero mean. Consequently
E EX
PE, =
22
hence (X,) a martingale sequence. Defining martingale differences
A X X Pi,4+
we have
Σ A i=f'+r-2
Pej
(D.106)
j=('+2-11
We define an event E E Fi by
Kg 1
L s c/
(lāel < C7n n
n
vT.
E =
i 2
i Sigsi
(D.107)
n
pish P
p-1-1(0) (x)
< <CK.}
132

---

Published as a conference paper at ICLR 2021
for 1 2 l' + 1 and decompose the sum in (D.106) into
Σ
Σ
é-1
A 1 E
+
Σ
t-
(D.108)
1
+
22--22
In order to control the second term, note that
8/ - =
y-
=1 g/-1 01
Σ
1
II
biam 3m kmm Rg
(1 2)C
m=1
è-1
a (ar)
TT
Gdgr -2 (22r-2(a))
HJ
=1 d
Σ
1S
g=++2
Sq
a
(2))ll) | (2--2
(1 (
22 122
m
and using 2 PJ, -i
Hi
d Ps W W'P
p 1 HJ
where W is an in-
dependent copy of W gives
gg A og
where g ~ (O 1) and
Pst 11
2- (2))-2()))
-V
2 2
II Sig
dt
Σ
2r-2
(x))
g=i+2
12 22 1mmm
12,11
EC,-1(f' 11---)
m=1
MT
1/2
(x))
2 Eiconie
II 8g
at
Σ
(m)
g
(i1...) EC,, 2C,+1(E'+1,i1)
110 130+3.52mm- Jm 12.11
m=1
8-1 II 8q g=++2
p P 10'1 (x))
2 EsK
2
max 115--1
ESKINE
aa [<]-ll
x-2
1 [eex
max 1 S
Lox,
1122512588i11
< Vdl. dL
2r-2
Kg
r-1
=C n
VaLL (L3R2K)
r-
11
In the last inequality used the definition of E and the assumption 3 Kg < follows that
en2
E exp (4
2r-2 xー exp dL (L3/2K.)
a.,
and we apply Freedman's inequality for martingales with sub-Gaussiaa increments (lemma G.7 to conclude
E-1 Σ +2r-2
r-1 A > (13/2K))*
dL
2 exp (-c
(D.109)
n
It remains to bound the first term in (.108) Using lemma D.15
(.95) and taking union bound
over é1, i2, ia in D0. we have
ci 2 CL- - C' (e (c te-dd) 2 C" cce c""
133

---

Published as a conference paper at ICLR 2021
where assume 2 KL log L and d 2 K' log L for some K, K' An additional union bound over gives
8-P'
Σ 22rr22
A - A
>0 S Σ P
(
==1
S LC
(c
- -c
- 0² (c-0r e-
for appropriate constants. Combining the above with D.109 and recalling (D.106) gives
P
Pii
(1,/2K.)*
dL
2 + C'e S Ce
+=
n
for some constants, L3/2K, S 1/2 implies
[(2-
[(e-t)
(1312K.)* -L312K,
Σ
(L312K))
1- (L3/2Kk) (t--))/2]]
= =L3/2
2.
1 1 --3/2K,
A final union bound over and rescaling of d gives
((-(0)/2
(=
P
Σ
dL.
=P
Σ
Σ
Peg
>
dL
F
j=t'+2r-1
SCLe + Le d'd e-'
C e-""d
(D.110)
for appropriate constants, again assuming assume KL log and d K' log L for some K, K' Combining the above with equation (D. and worsening constants gives
dL
P
Σ
Σ
Se C'e
n
T.E..
for appropriate constants.
Part (ii). We consider terms in the sets G (with i = l' and 12 S e- 1). In contrast to the previous section, the bounds on these terms will differ based on the value of the p subscript (denoting whether we use a fixed vector vy or a random vector v.). We first consider G G..p- noting
1-1
Σ T
L.P
Σ c
18.
1=8
bp
Lemma D.15 and union bound give
t-1
{-1
E-1 scn
do
n a < K, n
3.0'
s
C
3=
j=m
n
2 1 LC'e- - (22 (2e-d's - =dnn))
2 1 C -c" 20-c" do
134

---

Published as a conference paper at ICLR 2021
for appropriate constants, where we assume K, K". With the same probability we have
log and do K' llog L for some constants
g Eg'
Σ gorE
- CLK, S CR,
(D.11I)
ThEEGi
where we defined
dp
R
R9
n
and used _K3 1.
We next consider sums of terms in Grp for 1. In controlling the sum of these terms, the proof will proceed along similar lines to the previous section. The main tool we will be utilizing is martingale concentration. Recall that since i2r S l - and every G-chains are separated by matrix H the starting index of the final G-chain is no larger than l 2r + 1. We thus have
Σ
Σ
Biyr
b Di dam +1 fem js.
g
S <
1- 2r +1,
m==
(i3.....,
v2)) C +2,
2-2
Jr. Dj
j=
Define filtration
F = (Up,
f (Up: p² (x)..... p² (x), H H j (e- l' +1
(note the reversed indexing convention compared to the filtration defined in (D.103). Since / is -e-i-measurable (as can be seen from (D.86)) can define
X
Σ
P
and it follows that XT.P is
F' Fimmasurabll.. Recalling from (D86 that
Sk a (x)) by 1:+1 HJ Pys (x)
LESK
II kmi++1
la's (z: 11 a (x) 1|
and hence p is linear in the zero-mean variables HJ+1 have
8 Σ j=(-3
EXPIFF
Σ MJ
Σ -=
D
E H
1-1--
1 = X;
H
j=t-i-1
and thus the sequence {X. T:p is a martingale with respect to this filtration. Defining martingale the sum of interest can be expressed
differences A X x
1-11
2r+1
Σ
P
A T.
(D.112)
135

---

Published as a conference paper at ICLR 2021
We now define an event which will shortly show holds with high probability:
(lötar
| I K5)
n
r-1
biem
2. 11.am
S
VT.
Kg
m=
s scJ
d1
c2 |
1 CV
ci-1
n
II (x
T 1:8-112 ( Pr
HJ
R=ig11
Eska
SCVL
P Q pc px>1p-1<>)11
n
LEsKo
< 2Kg ||tt- (x:)
(D.113)
Truncating the martingale difference on such an event gives
11-1 A
gg
P
Σ
Ōizr
b b
(2r (E 1err-11l--i-22-
m=
1 gl
a (x) ( p)--ip--(10)
Dizr
bigm 22 5m 1am
II 8k
| |aa-i-1 (xx)22 Ge-i
HJ
(ar)
oog
for standard normal g, where we used
)-=1(x)) (x)
T 14 11
P
HJ
a
e-i--1 (x)
0 6 (x)
PJ H 4-it1
P
HJ
1
xx)
1ig-1:2-42
P
P H Q
(1)
HJ
Ta
(x)) ||
p
(x)
11--1:8- Pse
W
P Ps--i
Q jr-
( 1|
with Wi--i-1 an independent copy of W and we have defined
Σ
Tige m=2
bigm 2. 1. G2-1.
oP
(x) #migh1
paa-te-- P
Pge- Qsc-ip (2)) ax (2))la 2
|
Note that from (..13). if define
dg V
Rs 1,
n
136

---

Published as a conference paper at ICLR 2021
the standard deviation ol can be bounded as
CK?
K₂
CRp (1,/2KK,
1. 3/2 Rp S VLn
vn
VL
where in the first inequality we used a triangle inequality, bounded the number of summands by _ In the second inequality we used L S
Writing the sum in D.112
L-f' Σ
Σ
A TP
(1-18-1)
A S
A
ATP
i=
/=2r-1
and recognizing that the second sum is over a zero-mean adapted sequence that obeys
exp (A ..)) زーー
en
E
exp
(13/2K2)))) 20
AY,
a.s..s a.s.
RG
an application of Freedman's 's inequality for martingales with uub-Gaussian increments (lemma G.7 gives
P
Σ i=2r-1
A
Rp (13/2K.)
S exp
2T
Turning now to controlling the probability of E holding, we use lemmas D.15 tion of K and union bound conclude
D.18 the defini-
P ( & C& - + d'd
d'd + +enmn) + L''C' 4 < ce" t cdds
for appropriate constants, where assumed 2 KI.log I, d1 2 K Klog L for some K, K'. Com- bining the previous two results gives
P
Σ
> Rp
[1312K]) Kg
=P
T.P
Rp
(1,12KK)
i=
i=2r-1
8-v'
E-1'
SP
Σ
1 ri
A.
> Rp [3/2KKg
+
Σ 2=2r-1
(1 11
A:PP
>0
2=2r-1
< 2e
+ LP
(EX
$ Ce =' -dd
[(r-))/2]
for some constants. Noting before that
Σ
1-1 S 2, using D.111 (T,321K,)
to bound the
2=2
terms with r 1, and setting dg = d1 we obtain
/2
PP
Σ
Σ
TPP
CRp
< LC (e-e ( +e SC te
for appropriate constants, where we used again n 2 KL L, d1 2 K log L for some K, K'.
Lemma D.19. (Horm al. 1994) Given semidefinite matrix A, for any partitioning
A 11
A12
A1b
A21
Az2
A
A
Abb
b
we have ||||| < Σ
A
t=1
137

---

Published as a conference paper at ICLR 2021
Lemma D.20. Civen semidefiiite matrix A and unit norm if
P Piv Av C* 2 1 CP cxp (-a "1
and n > ( then
|I S C 1-111-10 - exp (-c 12
for some constants c, C. C
Proof. We partition A into blocks of size E for appropriately chosen c2. There are 4 such blocks, and we similarly partition the coordinates {l...., n into A sets K; = {1+ su
1 for i [
We proceed to bound the operator norm of the diagonal blocks using standard e-net argument Vershynii 2018). The set of unit norm vectors supported on some K; forms sphere S * We can thus construct --net M this sphere with at most elog(9)ez points. A standard argument gives
|| | |l I C sup || Agæll-
wEN.
We control the RHS by a taking a union bound over the net, finding
P sup LENN
A, S C² = PP
sup
||
AxIl C
L
2 1 CO exp (-01 2 1 CP exp (log(9)02- - (1)
11
We now choose 02 to satisfy log(9)o2 = -, and the blocks will still have non-zero size because we 2 C1
assume
Taking union bound over the Co blocks and using Lemma D.19 gives
A
| Ce
w.p. P 2 1 C' exp (-c7) for some constants c, C.C
Lemma D.21, Assume n 2 max [KL log K' K"), do 2 K log L for suitably chosen K, K', K" K!" Define I as in Lemma D.14 For Sno and
By (WL+1 1+1 Ps, WL. W PJen))
denote
di 4= e Jıl, d = (di,...,din .d.)),
and dunn = mind: We then have
DESK |og - -0(2))) CVdgL+ P
||
2|17|
Lds
d
d
1/2
1/2
I (e-mmax(dmmlll c + dd
d'
for absolute constants c, d, ddd C,C',C"!, C"", where the event ESK is defined in lemma D.14 Other useful forms of this result are
|
Ldb
P 1 [| - (20)))) Cvd.L+ C, Lns 128 +
+ +CL (8,
2
-e + te -c' +e -c'ds
max(dh.1)
i=!
where 8; 1.
138

---

Published as a conference paper at ICLR 2021
Proof. Denoting by { } the weight matrices projected onto the subspace orthogonal to the fea- tures as in lemma D.14 we define
B1 (x) = H (xx(2) (WL+) H H PI H²*
BH - H 3a (WL+1P. PJ. HH H PJ H
for 2= ..... 1- Note the additional matrix compared to the standard definition of the backward features. Control of the norm of the difference between them can then be used to control the back- ward features and Lipschitz constant of the network. Note also that H may not be square matrix (and indeed in the case of the Lipschitz constant it will be rectangular). We denote the number of columns of H by nk-1- Writing
B3g - (3(2))
S 1sx AHs - -1x((0) + 11 B3g (H)
1 EA
(D.114)
+ L [[((xz) Bir (2)((x)) (2=)|l2
we begin by bounding the first term. For D
H (a:), D
HJ defined as in D.14 and
('(x) Ps. PI((m).
(D.115)
we have
1
BAA3 - B3 (z)
r
(x))
=1
WL+1
Σ
bii+ H
16+11 HJ
b
1m8+1
i=t+1
We first bound | 12-- similar manner to the proof of lemma D.18
Repeated use of the rotational invariance of the Gaussian distribution gives
||b:l12
Esx ||
Li+ (2(()(((((())))))( (x))) ( 1
d n
2
Esx Pie 1
(1.) ESN Q'(x)
i-1 II
H (1.)
CSK
Psr
12
H
H&+1
(1.9
2
k=(+1
-61,(=))
where we defined invariance gives
H4t1 v WL+1 Denoting by W an independent copy of W rotational
2
E,
LE Ps
-X
where X* is standard chi-squared distributed random variable with |suppdiag Psy degrees of freedom that is independent all the other variables in the problem, and similarly for (la(2).
A product of such terms was bounded in lemma D.18 from which obtain
P
II (L)( k=itl
II EJ. On < k=f+1
(D.116)
We similarly have Recalling
H Les (((0) (1.)
('xr
(1.)
(D.115)
and since
= = (suppdiag
recognize that
NEEN (kxx)
1 Xi n
(1.:)
139

---

Published as a conference paper at ICLR 2021
where Xi is standard chi chi-squared distributed random variable with d, degrees of freedom. If this variable is identically Otherwise, d; 2 and Bernstein's 's inequality gives
P Xx d; > Csdi] & 2e cad. + P Xi > Cdsd < 2e cadi
< 2e cs
2e-ce (dx.1)
(D.117)
for some constants and 8 2 Clearly this result also holds d, = 0. Similarly,
H (1,3
bounded almost surely by 2 2 Xe+1 where Xe+1 has nz-1 degrees of freedom. Bernstein's 's inequality
gives P
t e for t > Kne-1 for some K. Combining these results with
D.116 and taking union bound, obtain
1b:ll > n
Σ s:ds
S 2
Σ
cos max(d:.)) 2L(e + C)
imt+1
(D.118)
< 2
max(di.)) + te- +oc+x c-
i=f+1
for appropriate constants, assuming 2 K log L, n 2 L'L L for some K, K', which can be simplified to
L
L c -ssmaa -axmax(d;:)) + e-e" t
P
Σ E
bbll2
t Σ di
2
(D.119)
1=++1
i=
< 2Le + ned + -e-
S Se-c's + ndd 4 te- +
assuming 8 2 K" log L for some K"
We next bound ((b, by)l for 8 & < i S L. Once again using rotational invariance starting from the last layer weights, we obtain
(b;, bj)
-1 W
Lat+ H
Q
1:4
1-
H
(x) Lijt1 (x)
HJ
HJ
L II
LES k=1+2
EL
(13
HJ
(x)
(where we interpret an empty product as unity). As before, we find using lemma D.18 that
LESK n II Ela(m) > Cnl -7 k=++2
P
(D.120)
2
We proceed bound the remaining factors in (b,, by)) by first writing
b,) = 1Es
II En (a) Hit) Q ('
( H
2
(:.)
k=i+2
di d
d EAA
II
(a)
Σ
Hi+l
(1.k2) 8k, (k..:)
1- H (1,k Sky
Ha-lo || ||
(k).1)
Ammi2
Ru
where 2² #+1jt+ = P1 p
ii and 9km. {-1,11 are the signs of the elements in (²) (s.))
H
for E ji In the above expression, km index the entries on which diagQ" is supported, and we denote dm = |suppdiag (m and use the permutation symmetry of the Gaussian distribution to these be [d....
If f 1, defining H P where Wi+1 is an independent copy of Wi+1, with fi-1-8+1 denoting the matrix HJ
pi-1:611 with #1+1 in place of H/+1 and writing for concision (1.)
HJ
=
Hith Hi..,)
(1.k.) (k..) HJ
P P3)4((11)
(1.) (1.))
Hi+1
$35-1H
(.2,) (k),1)
Hi+l || ++15+11
140

---

Published as a conference paper at ICLR 2021
and
(ik) ((k.))
HJ
T
1--14++1
HJ
H Hitt
(k5.1)
1 +
we have
L
di
d2
18
II (m)
Σ
Σ
ki.ss
(b) =
k=i+2
k=
d: Σ
+ Esk
(lx(a)
1p ki
(D.121)
k=t+2
k=
n A + 12 2
where we used the invariance of the Gaussian distribution to reflections around the mean, (H & Wm Psm- j. and the independence between the { Wwm variables and the sign variables {sk... to absorb the latter into the former. Making a separate definition for concision
E R:
(1,kg)
(kr.))
1i-1:j2
HJ
P P)n((al)
first consider the term
1
H ++1
(1:3)
(c.ke)
(k).2)
ky=1
A d Bid
B
1 | |1
d
1
H +++
(:.kg)
(k).)
B
k
Lemma D.22 gives
IP
Bi.
c
-P
[
|u (+++t:+11
12 >
C - c""
(D.122)
We next consider B Writing
B.d 1EsK
H
H (.dky)
1exx
H +1
Φ
H Ps With
(1:))
(icky)
(1
ky=1
First, since the variables
W f+1 (1) are independent of { {LEax
H
H
" (Pse)
(1.))
Gaussian tail bound gives
d
P
(1.)) Cンン- H
Ho+t- (ky,1)
2d
1 1s
H (1.)
T - H
(Par)
(ikk)
(:A,))
72
5 -od
(D.123)
for constants and d 2 K for some K.
141

---

Published as a conference paper at ICLR 2021
Two applications lemma
D.22 give
P
11 Eax
H H
H (onk)
(Podasssss
> Cdy >d,
=t
(1,3)
2 c-11 (Poe)(esiss)) Hil (1)gg)
Σ
Ees
H H
(1::)
n
k
$ dj (e - te d
Se sed E te
assuming & 2 K no K'L log n for some K.
Combining this bound with D.123 we obtain
ddst
Bij 3
Se ed - + -c-
(D.124)
for appropriate constants.
We now turn to bounding B.d. Define by Q' matrix such that Pos Q (2))
Then
B
d LE
P a H Q P3:
W++1+ (:11
(1.)
In order to bound this term using the Hanson- Wright inequality, we first note that since
Ps .- H
Q
Pss
s | H
<
N++++ HJ
HJ
Hi*
Peil $ H
P
| H ||
Q²
dz.
use lemma D.14 standard nenet argument to control the operator norm of a Gaussian matrix and union bound to obtain
I 1 Psr 1H
Q
Psi
sCT.
P
2 - - +e -cn 2 1 =
EE Psr
H Pssi
sCId,
We also have
With 2
E With
Peu
- 1 H²Q Ps:
Psiiiis H Q Psi-
(1.3)
(:11)
Wa11
n
dj
P ni+++1 1 - H
łky St
HJ
HJ
ky=1
and using lemmas D.14 and D.22
gives
P
P
s:
r HJ
1--1-441 1.841+
HJ
Hiteky
c n
2 de
2
-o
assuming KL log for some K. Denoting the union of these two events by G, application
of the Hanson-Wright Wright inequality (lemma G..4 gives
2Cd,
n2 n52
))
P
Lg
B
> 82 +
< Ceepp (-e
(D.125)
T.2dy L
for appropriate constants and $2 > 0, and an additional union bound gives
2Cd, < Ceexp (-emin
nn²83 1182
P
Bij > 82 +
) + E
(D.126)
n
I2d L
142

---

Published as a conference paper at ICLR 2021
We next turn to bounding
Rotational invariance of the Gaussian distribution gives
Bld
E
++ (1,bg)
1f-1++2
P34+16.1)
HJ
k,=1
di
ES
H+ W
(1,bg) (k..1) Pa. 1-+1
1,=1
since PJ, I HJ
1:j+2 1: PJ (1))) and ((,,1) (k.,1) are independent.
Since Hit1 (1,k)) Wi (1, are both sub-Gaussian with sub-Gaussian norm bounded by Ch the prod- uct of two such variables is sub-exponential variable with sub exponential norm satisfying for some constants. Thus the first sum above a sum of independent, zero-mean sub-exponential random variables, and Bernstein's 's inequality gives
H (1.b.)
( S
di
-omin n2yl
W (ke.1) (1..)
> 81
< 2e
(D.127)
for $1 2 1 and some constant c.
Since tain
LESN Ps.,
D PJ, Po I [Ene 12-1
we can apply lemma D.14 to ob-
P DI
P T
1i-1:+2 HJ
Paintaa Post2Pasnnaa
c| s C'e
for appropriate constants. Combining the last two results gives
s eemin nfat
B > Css
(D.128)
for some constants.
Combining the above with (D.122)
D.124
and D.126), we have
2d
ddyt
p
C
$2
81
n
n1
(D.129)
2
-d" min
n2 1.ady
$
-emin
mmmin
man)
+ c t + d" +
+ te -e" "
In the above proof we assumed i > + 1. If instead = + we simply set 1:8+1 in the expression for A in (D.12T) and have (b,+1, b,) A Ai+ld
1-81
HJ
HJ
We now turn controlling the term A.d Since
H H (ka))
W ((k. Ps-- and
Ps PS--1 1-1 1:611 H is independent of W: (k...)) rotational invariance of the Gaussian distribution gives
HJ (skky)
d: A/
d
Hitl (1,k.)
Wi
k==1
=0j
(D.130)
dy
1s
P
11
HJ
1 HJ
1 1:1+1
H
(1kk)) 1
Hi+ls
(k,,2)
||
ky=1
We bound [Ci using (.127).
143

---

Published as a conference paper at ICLR 2021
remains to control
C
Since H H
d (ky.1) (Psi-)(a,n))) W and W
(1,kg) are independent of
(1.k))
(Pss+ ((k),ds))
Ps--
fi-1:4+1
HJ
HJ
(.k,)
the second factor in D.130 also zero-mean,
and it follows that
E
P (k)))))))
PS--
r T
H
W (1,5))
HJ
HJ
(ciks)
P
ky=1
C'e
d
2d
1 Esk
(Ps
Pst
1:44
1-1.-t1 1:6+1a
H (crks)
HJ
for
some
constants
and
d
0.
Since
Pgi
ni
HJ
1:6+
HJ
(1.kg)
r HJ
H
2
applying lemmas
D.22 and D.14
total of dj times and taking
HJ
a union bound gives
dj Σ
P TH 1:841 r 10-11++1 H ddLL
P
LES (Psr
(.kg))
n
8-1
S dy ( +e te-dt) Se-d - ee-et
where we assumed > KT, 2 K" log for some constants, Combining the above three results with (D.122 and taking a union bound, we obtain
> Cs Vd,dne-1t La
n2 -e- +e -c"h min( d, .ns
P
A
I Se +e te
for appropriate constants.
Taking union bound over this result well (D.19 and 1.12 allows to bound the inner product by
82 d v ddddne-LLt Lt
P Kb, b,رl 2 Cnsi
(D.131)
22? d, nos)
+ +e d + ad " min n²
S Se-mming
for some constants, again assuming 2 KLd At this point we obtain a bound on the sum of these inner products that will be useful an application where the {d,] are expected be small. Subsequently, we will derive a different expression that will be useful when they are large.
We now choose 51
de ,2
L,
for some 2 1, which gives
a2-1
P [bi, by 2 Cd,s
dgLs
dj Lddj n
< <C'' C''e- -emindd,ddd + - + -cd'
n
n
for appropriately chosen constants. Note that if d, 0 or d, = 0 then (b,, b,)l identically zero, hence we can replace the term min d:) in the tail above by max (1, will still hold d, = or d, = Lower bounding the second expression by dmin mind, gives
and the result
Ld
Ldd
PP
(b;, by 2 Cd.s
+
Cee -cmxx(dmmm11 - C -
144

---

Published as a conference paper at ICLR 2021
Recalling the definition of d in the lemma statement, additional union bound over the values of 2, in the expression above combined with (D.119) gives
P
LEA
|31 - 3r (x))
Cs
d
+
2||d|
Ls
Ld
d
|
1/2
11/2
Exx AHH (20(20) Cs Σ di
Ld + Lddy
+ Cs
Σ di
i=++1
< I? (C'e (C'e-cmax(ddmm comax(dmun e + C.M
C'e
C'e-d" d" max (Musiss e e"""T C e
C'e-d"s + C. e" + C. e
(D.132)
for appropriate constants, where we assumed d 2 K log L, 2 max K' log L}. n 2 K" Log L for some K, K", K". Taking a square root gives a bound on the first term (DI11).
We next consider a different bound for this term that will be useful when the {d:) are large. Our starting point will be D.131 If we set 81 = 8, 82 = Lss and use (D.1T8) we obtain
LEsK
AHHJ - (1)( > CLns
L's
|d]l n
+
VLdt
1/2
+
C
Σ
L
syd
1/2
n
i=t+1
<
P
LEAR
Bir(
Ons Σ
Ls + dj +
Tdddzt
+C
t
Σ
L
s;d;
1=8+1
n²
<
2
C'exp
-êminf
ns}
+ C
-0-7
C'll -C"l-
b
Zss max((d;;)) 46 -2"
+
i=f-1
< e one + - edd
Σ
-- max(drll) + -
t=t+
(D.133)
for appropriate constants under similar assumptions n, L, d.
To bound the remaining terms in (D.144, since
1ESK
S 1l W (T
LL HJ
|e+1|
d Dax
[|
pL:++1)
W L+1
H
and we can apply lemma tively, to conclude
D. and an g-net argument to bound the first and second factors respec-
PP DEaR
B3 -2yB/ Cvdl < C'e -ed + Ceen -cd
for some d such that d 2 K Klog L and assuming > K'd. An identical result holds for the last term in (D.I14) where we simply choose J; I,(x:) for all In conclusion, using (.132) we have
LEN B0y - -3(()))) > CvaLL
a
1/2
1/2
CCe cx e C! c""e'""a
(D.134)
145

---

Published as a conference paper at ICLR 2021
for appropriate constants, while if we (D.133 instead we obtain
P LESK 33 (((x) (,0 CVdL+ O Lns L2s ||] Late
+ CLt
(s
0 te +
E o- max((..1) e -
i=/+
(D.135)
remains to transfer control from ||an - B/ (z)
|| to ||5 - -B/ (x) (z) Note that
|in - (x) (-) H (0'H - 31)(2))
Pgrl
We++- (:.1)
1
||8 - f//(2))1
Pgrl W 2
where if l we define Pgoa I Ing x Since E
Lr [Peca]
2(n-1)
(111] 2
W
Bernstein's 's inequality gives (assuming n >> K for some K)
P
Wett+ Ps
(t.1)
< C
S c
on
and hence with the same probability
pin - B1r(2x) |
110'r5 01n(2))1
I sc -
(1)(
|PSLL 10
(G.1)
The bounds (D.134). probability up to e term which we can absorb into the existing tail by demanding n 2 K L for some K.
(D.135 also apply to 11 - FIH (a>)|l
|| up to constant factor, with the same
Lemma D.22. For any É + 1 < m S + 1,
[n]] if 2 K for some K then
p |
H T
> c v -0
(k.i)
HJ
and if E. ++1
P
ll
H (in))
T
CV 1
S -0 te-dr
(k.))
assuming t 2 K' K'ne-1ff for some K'
Proof. If 1 > m,
1ssx
H i+1 Jum (1::) HJ
= 1E
W (1.))
Pss
1m++1
Ps Hm
HJ
Esx
W 1i--Pssu (1,3)
HJ
P.m. Wm
Psm 11
||
<1EsK
Witt (1,)
Pss
pim++
HJ
P,,,
wm
2
1
W
Pssa a pim++
HJ
Psm
2
Wi.))
2
< S1Es
Psua P
pim+
HJ
W
(1.:)
2
W
(1.9
P
pim++le
+ (1.))
W
(1.))
2
If on the other hand j ++1 1 m, we have 1
H J:m ||_ 1Es
H ++1
S
W
(1.:) HJ
(1:))
(1.)
2
Bernstein's 's inequality gives TP
+1+ (1.1)
c
< -oe on and IP [|| (1.)
C !
s 2e
for 1. 2 1, while lemma D.14 gives
P
EA
mim++s HJ
c
C.
2
Taking union bounds gives the desired results.
146

---

Published as a conference paper at ICLR 2021
Lemma D.23 (Generalized backward features inner product concentration). Fix æ. x' E Sae-1, V y= Z(x, x'). Define a collection of support sets J. generalized backward features B a constant 8g and event Esk in lemma D.14 Assuming 2 max {KL log K'), d K" log n suitably chosen K, K', K", we have
Lea (B',
12 F=
1-11
11
II
(0
P E
S CCe C
c
(LVL V
vā + d3/28, (1+ 3/1
L
for absolute constants e, C, C'. If d'additionally we have P P[ExK] 2 1 1-e-de then the same result holds without the truncation LEg,, with worse constants.
Proof. Note that
L-1
1 (v()
1sx (B'g
n
II
2 1=
Esx (06 B² (z)),
+
Esx
3 (x')))
1-1
(ol0)
+ NES (3r B3 (x'))
II
(D.136)
1x | (a 112 |13 3² (ax) (xx) (zx)12
11
|1:3 |1 - -3(x')ll )())
+ LE (3)(2), B² (2')
II
In order to bound the first two terms, we use rotational invariance of the Gaussian distribution twice obtain
LESM 1|
1 ESN
w
WL+1I
LiL++11 1 J(x')
P
s3((n))
|
< 1 Esx
L: w WL+IrLe
| W
1 2d
WL
1 Es
| é 1|
g 2
Bernstein's inequality gives IP bounded using D.14 to give
s en while 1sx W Cnl
p 11 can be
P Leex || 1| 2 Cnl s Cne C² < C" c""e
for appropriate constants. Using lemma D.21 to bound LEs 1|3r B3(x))112 we obtain
P Ls (2)( 11 (l(x)ll CVdIn+ C² dd. - d3/28. (1
6a
Vn
< + cdd s cddd
for some constants, assuming n 2 KLd for some K. Bounding the second term in (D.136) in an identical fashion and the last term in (D.136) using Lemma D.4 we obtain
1 II F=t
1 ENN (Bg
P
C
dl vn + v |60.L + d3/28, (1+ 5
L5/2
1-1
1xxx (B)
II
1-
SP
pm
(vatm++ dō.Tnn + d3/28, (1
> C¹
1,5/2
+ C' &C VT
147

---

Published as a conference paper at ICLR 2021
1 Esx
|18 1111111188 - (2))l + 1 |1 113g. - (2'))1
P
(vdLm+ / /doLLm + /d//20g
(1+ fn
Vn L/
+P
1EaK (B'',
II 1.-1
11 (000)(
(1 1
>
LL
F'=l
S SCee -od_ + c
S C""
for appropriate constants assuming d 2 1. Taking union bound over all possible choices of E [L] and using 2 K log L for some K gives the desired result. If we additionally have 1 - e-e"d for some c"", we can write
P|EsK] 21
L-1 (1 (v)
11 L-1
(1-
sole) (v)
(B3.
Esx (Bg,3')
II
1
π
2 ='=
f=
+ (1 1xsx)
(B'g
and since the last term is zero w.p. 2 1 - - c" we obtain the same result as in the truncated case, with possibly worse constants.
D.4 AUXILIARY RESULTS
Lemma D.24. There are absolute constants C12 C,C' > 0 and absolute constants K, K' >> 0 such that for any E N. if 2 max K log" n, K L), then for every E [L] one has
F 9 (D*) ( F
C
log n
- E)pe-1 (1+ log L) +
C'
n 1+
n²
The constant 01 is the absolute constant appearing in Lemma E.
Proof. The case of = L follows immediately from Lemma E.I with an appropriate choice of d 2 K" for K" > 0 some absolute constant. Henceforth we assume E E [L - 1J. We Taylor expand
(with Lagrange remainder) the smooth function p(4-() about the point ((2b-1), obtaining for any t t [0, T
(1) bll (((08-)))))) (1 (1-(((1))) + ((4-
( ())
where & is some point of [0, m| lying in between t and (p(r-1). In particular, putting t E7, obtain
(1-)))11-+ () ( 1)) (Et - (p(tt11)) + ((4=0(() (Ee ((0
pll (2") ++1)
2
where & is some point of [0, x lying in between of and (12--1). By C.23 and C.26 we have that (((-() < 0, whence
(ll-0)(0 (1) s 3 1 (1(()11)) (0t (((e-1))
(D.137)
Using Lemma E.5 and an induction, we have that ((T--1) is decreasing, and moreover by the con- cavity property we have c(it-1) > 12-1/22 An application of Lemmas E. and C.15 then yields
E
(oll--1) (1(-Et1) (
CDD-1 logn n
C'n -ndd
1 (co/4) - Ejpe-1
n
< C log
je-1
n 1+ (co/4)(IT E)ott + C'n--id
as long as 2 K and n 2 d' log* n. In particular, we can choose d = max K, c/ to obtained the claimed error for the upper bound. Next, for the lower bound, we make use of the estimate
1
()L-0 1 - -)1 1+
(co/8)z log (1 + (co/8)(L l 1)v)
f(x)
148

---

Published as a conference paper at ICLR 2021
which follows from Lemma C.16 and ((1--1) 0; by that lemma, we have that f is increas- ing. By Lemma E.3 as long as 2 K' log' 2 n. there is event & on which ji - ((( <<
CD--ly Vognn/lognnn n/ + C'n and which satisfies P(E [ 2 1 - C"n 3 In particular, on the event & we have of 2 at 1--14-C C² /nnn provided n 2 16C log n, and so on the event & we have
& 2 min(()((e-1). ) bt-1/4 /4- C² C²/nn" 2 14 - C²/n" //n.. We can thus write
(0)) 2l1
2 S (p 1) (i' - ()( + f(5)
(pt - 1(
2
= (10 1) (0' - 1(( + +111g + 1gx) ((5)
(0e - -0(6t11))
2
6 (1000-1) 1) (Et - 1) + 11
f( 1
C
n3
(i' - LIDE
-120"
2 (010-0 1) (Et 1(( + T
(BL 18--p0 1)( =220" L) -
C
2
where the inequality in the third line follows from boundedness of the angles and the magnitude estimate on if in Lemma C.16. together with our estimate on & on E, and the inequality in the final line is consequence of f U, which allows to drop the indicator for E and obtain a lower bound. Taking conditional expectations using the previous lower bound and applying jrl -easurraabbbll of bt-1 and boundedness of the angles together with our conditional measure bound E obtain
م (D)) ((1-et1)00-1) Ft-1|=
-0 logn
DC-1
C'
C
E
n 1+ (co/ 4](L 6)pe-1
n²
n²
or Co E (Dr 1)( 2
F
where we also apply the complementary bound obtained by our previous work following D.137)
Since the CL estimate in Lemma C.16 applies also to f. and since f I 0, an application of Lemma E.4 with an appropriate choice of d and the choice 2 K' n then yields (with larger absolute constant C")
E p (D)) - -ll- ze-1] 2 -C logn
C'
CoL
1+ [co/4][L - C)pe-I
n2
n2
C7 Co n ( 2
C4
n13
If we choose 2 (C,/C")L, we can simplify this last estimate to
(00 ((l- st-1]= 2 - log
logg n
De-1
2C"
E
n
1 1+ (co/ ()o/4)([. - -EjBe-T
n²
O7 n.
(DE-)22 2 (br-1
C4
n
4
3
To conclude, we divide our analysis into two cases: when se-1 > 8C./n², we have 82-1/4 Cnn 3 2 jf bf-1/8, and so
(b(-1)²/ f ( (e-1
C4
2
2
4
n
8
= 64
f
8
8
8Crit-1
8 8log(I. [. ())
11
(co/
(6)
E)pe-1
COT
where the last inequality follows from Lemma C.16 C'L estimate in Lemma C.16 immlies
On the other hand, when et-1 5 1880//n the
(DC-1)² 2
C1
64CC3L
64C03L
20"
(4
>
> n²
3
n
n2
149

---

Published as a conference paper at ICLR 2021
where the last estimate holds when n 2 (L2 Adding these two estimates together, obtain one that is valid regardless of the value D and choosing n 2 C7 log n to combine the residuals, obtain (after worst-case adjusting the constants)
B p0l-000
x
4C
Cslogn D- (1+ log(L 1 -() (cn/64)( f)De-I
n²
n
Combining with our previous work, we obtain
E 1oll ( - ((0-1) xe-1] < C
logn (1+ log
n 1 (c) f)De-1 C²
1 n12
after worst-casing constants.
Lemma D.25. There are absolute constants CL, C, C", C" ,"' 0 and absolute constants K, K' O such that for any d 2 K, if n 2 K'd log n, then for every LE N and every { E [L] one has
P pll ( (1)- 1
dlogn
1+ (co/8)(L - -))or1 + +2'n -1d/2 n jtt
20
n
2 1 n""nnaau
and
E
(oll-0 - (2L-)))
Ft-1
s 402 dlogg
DP-1
1 (L E)
C=n--14/2
The constant 01 is the absolute constant appearing in Lemma
E.
Proof. We will fix the meaning of the absolute constants C, C" C" > throughout the proof below. By Lemma E.3 we have if K and n 2 K'd log" n that for every E [L]
dlog
P of - ((p
S SCD-1
C'n -ad
xt1
1 - n aad
(D.138)
By Lemma C.15 we have the estimate
10 (t)
_ 1 (co/2)et'
valid for any l E No. Writing E = oe ((0--1) so that Es (p(t-11) + + we have that (E is adapted (Ft), and by the fundamental theorem of calculus
( ( -00-)+t1(00))))-----11111101)))-
dt D(0e-1)) 1 (co/2)(L f)t
The integrand is nonnegative, so by Jensen's 's inequality we have
p(0--))
(10) pll ( s |
/ /000-11440 (1 (Lo/2)(LL - t)t)2*
dt
and an integration then yields
( wl1 (()-1+))(0-11)
(E
(D.139)
1 (L) ()p((t))))1ll)) (co ()o/2)(L 11(--0(6 (L (((())))((())++3)))
Choosing d 2 1/C1, we can guarantee that whenever v 2 C -cd/22 one has
dlg n.
dlogn
Cup
+C'n-aid < <2Cv
(D.140)
n
n
150

---

Published as a conference paper at ICLR 2021
and choosing 64C2dlog n, we can guarantee that
d dlog 2 2C
(D.141)
4
n
In particular, the last condition guarantees 20
doog S 1/4. By concavity of via Lemma E.S
we have (((e-1) ) 2 vf-1/ /2, and using (D.188 to obtain
d logg
P E
CD
C'n
xt 21 11-C"n
we have by (D.140) and (D141) as well as the concavity lower bound on yp
P
P( = 2 -' /4 F( 2 C" -ord
/4|="=1
as long D- (C' (C) -od/2 In particular, plugging these bounds into D.139 and taking square roots, we obtain by a union bound
P (oll-0(0 - ((1--41((0-) S 1 dlog n
Ft
1+ (L e)ūt-T
2 1 nC"hn
whenever et (C' -dd/2 Meanwhile, when DE (C' (C'/C))==ddd if we choose 2 dlog n we have
CD |dlogn C'n S 2C'nnoa/122 -2//2
and we can use the 1-Lipschitz property of p((-1), which follows from Lemma B.5 to obtain using D.138
pol (Dt) - pll ((l- 22C'n
xt-1]
P
2 P
(04) (1 S cCD-11
dlog n
+ +'n -ddd 56-1
2 P 10- (((t-1) 1)| s CDi-1 dlogn n
+ C'n -ai
2 1 n"nn-r -nd
Because
(p(l s(b- 1| 2 0. we can then obtain using union bound
P (oll-0 -ll- (o--++1)) (1 (bC-1) S dlo
208811 ++ (co/8)(L - (()p))
2C'
F
+
neid/2
n
2 3C" n -iid
which holds regardless of the value of De-1, We can then obtain the second bound using this one, via partition of the expectation: let
E= f1(p0 1 - (11-) S
dlog n
2008-1
20'n 22''n-cid/2 -21d22
1+ (co/8)(L 0)pe-1
n
so that & E Ftt and TP| F 21 1-3C" - by our work above. Then we have
(olLL-0)(0)) - (1) (c--1)) 2 xP-1
E
$ E
6 (z) - plt ((1-
1g F(
+ xTE[1
Fe-1]
2
20
dl dlogg v
it-1
+ +20' -cdd12
F4-1
+ -oid -idd
1 + (L - p)pe-1
dlog
DC-1
2
2C
+ 22'nn
-ann
+ (co/8)(L 0)
n2
151

---

Published as a conference paper at ICLR 2021
where the first inequality uses the triangle inequality to obtain (102-))) ((1.-E1)
2 the second inequality applies the definition of &, uses nonnegativity to drop the indicator in the first term, and applies the conditional measure bound on E and the final inequality inte- grates. Using the fact that our previous choices of large imply 2C / /dlogn/n S 1/4, and that 6t-1 1 (0)0)))-- I n, we can distribute the square in this final bound and worst-case constants
to obtain
I (16L-0 (C
J4-1
40 <402dlogn logg
Dt-1
2 + C."
(co/8))) (1+ - C)
n
as claimed.
Lemma D.26. Let XI..... XL. be independent chi-squared random variables, having respectively di,...,d, .d. degrees of freedom. Write dmni = minset di and let & d, X, Then there are absolute
constants C > 0 and an absolute constant < K S
such for any 0 < t S K, one has
P
E
t
S CLe Coduint edmint//1
In particular, there absolute constants C' C
0 and absolute constant K' 0 such that
for any > 0, if dmin 2 K'dl. then one has
dl. dmin s Le
P
- -1+
II
V'
1=1
Proof. For any t 2 O, we have by the AM-GM inequality
1/L
P I =
III
> (1+
S P
Σ & (1 + 12)11/1
By convexity of the exponential, we have (1 + 1) 2 1 + 1 log(1 + t), and by concavity of the logarithm we have log(1 (1 + t) tlo 2 ift S 1. This implies
L
P
II
& > +t
<
L
M Kt.
1=1
where K
(2g. Decomposing each X; into a sum of d, ii.d. squared gaussians and applying we obtain
Lemma G.2
L.
t2
J })
PP
E
t
< 2exp
-emin
C E
''C
di
$ 2 exp
(-d'dmin min ( (P/CL,t))
(D.142)
(-or dmmint" 12
< 2exp
where the last inequality holds provided S CT,, where C > O an absolute constant. Thus, as long S CL/K, K, we have suitable control of the upper tail of the product II, &.. For the lower tail, writing (0) = - -c, we have for any 0 < <t<
L
log & x-]]
P
I
IP
Σ log & log(1 -
< PP
where the inequality uses concavity of t + log(1 - t). By Lemma G.2 and every < C (where > 0 is an absolute constant)
we have for each E [4]
IP[E: 1 < 2c ed,
152

---

Published as a conference paper at ICLR 2021
so that by union bound and for t Σ CVL, we have with probability least 2L addmntt
that 1 - t/vL & 11+t/vL for every [4]. Meanwhile, Taylor expansion of the smooth
function I + log a in a neighborhood of gives
log = (r - 1) 24 (2 1)²
where & a number lying between and æ. In particular, x 2 2 we have log 2 (x- - 1) - 2(x 1)², whence for t I min[CvI, }}
P <
$ 2L Amina 12/L P
-L Σ
E
-
21
i=1
1 2
S 2Le comint -he + L
E1
where the final inequality requires in addition / S 1. An application of (D.142) then yields the claimed lower tail provided t S CL, which establishes the first claim. For the second claim, we consider the choice = / /dL/cdmin for which we have t < K whenever dmin 2 dL/ck², and
cdmintt 12 L d.
Lemma D.27. Let XI,...,Xp Xt be independent Binom(n, 2 random variables, and write & x Then for any S 3. one has
L
S ALe Lnt /8L
>
In particular, for any d > 0, nf 2 128dL then one has
L
dL
P
&
4
S 4Le
Proof. The proof is very similar to that of Lemma inequality
D.26
For any
0, we have by the AM-GM
1/L
L
P I =
|I
&
>
(1+8)
<
Σ & (1+ t)
By convexity of the exponential, have (1 + 1/2/2 + 1 log(1 t). and by concavity of the logarithm we have log(1 + 2 tlog 2 if < 1. This implies
L
L
P
III
& > 1tt
< P
Σ E Kt.
where K log(2). Decomposing each X; into a sum of n i.i.d. Bern()) random variables and twice, we obtain
applying Lemma G.
L
L+
E >
< 2c nt /22
(D.143)
This gives suitable control of the upper tail of the product I1, Ec. For the lower tail, writing log(0) - oc, we have for any 0 < <
L
P
I & < <
=P
Σ log & llog(1 -
Σ log & -1 -t]
where the inequality uses concavity of t + log(1 - t). By Lemma G.I
have for each i E [4]
Plls 1| 1) < 2e -2t
153

---

Published as a conference paper at ICLR 2021
so that by union bound, we have that 1- -t/VL S S 11t/vVL for every i E [L] with probability least 2Le 2Le peee-tt/21, /2L, Meanwhile, Taylor expansion of the smooth function x log x
neighborhood of gives
logz = (x 2 (x 1)²
where k is number lying between 1 and æ. In particular, if x 2 1/2 we have log 2 (x 1) 2(x 1)?, whence for t s 1/2
L
& <
2Le 2L /2
+P
L +
& -
21
L
S 2Le nu
+ P
L
4
2
where the final inequality requires in addition / S 1/4. An application of (D143) then yields the claimed lower tail, which establishes the first claim. For the second claim, we consider the choice t / /8dT./n, for which we have & S _ whenever n 2 128dL, and nt2 18T.88T-- d.
D
Lemma D.28. For < l' < L - define events
Ge. B
B 1:8
B
p s Cnn(( ( {3)
n
B
C(E - C((--))
t B 1:8
I sCn}
and
ct:l' = -11aa--100)
a 1 (x''l =0}
(w) - sC
d log nl
Gt.e' BB
If n, L satisfy the assumptions of corollary D.17 then
P
2 C'n(l E1 -
If n, L additionally satisfy the conditions of lemmas D.3 then
E.16
and n 2 L'Llog(n) (log() + d),
staf B
2
C'n
where c, C,C'C,C''' C" are absolute constants.
Proof. Since
n +--220222 +2)) P1) Pleniee Pe (2-
B 11-1 axx
i=1
applying corollary D.17 2n times gives
| (2))e svo] 21 Ef2
=E(mra")
16- cn 2 C'n(l 13
P tr
With the same probability also have
max ((,
p (2) (2)
= max tr (2)
$ Cn
ze (0,,m))
154

---

Published as a conference paper at ICLR 2021
and
P
max
r
+2
S
2 21 2
from which it follows that
B Bi-1i10 6x
1:f
and
B Bl- 1: mxp
S C'n(l-
r (2))))
r (2)1)))(00) | S Con(e--e) (2)
2 P
max
max
)E(2,0²)
$E(2,011
Ce -1)8 C'n(l -
2 1 nl 1)
follows that ded holds with the same probability. B
From lemma E. 16
P [1]a - (z))l1 (a')) > = on
for some constants c, C" Here D is the auxiliary angle process defined in
D.2
Using D.3
obtain
d log n P - $ CV
(v)
P (v) -e- sCV
d log n E +P
nl
nl
6e -n"n 2 C'n-
for appropriate choice of c, C
We conclude that
P
B
Coll'
CT -c"'d 21 C'e CI-Ceetn-C"neee nle -
21- -1--n--o
for appropriately chosen constants, where we used n 2 ("""Elog(r) (log(E) + d).
Lemma D.29. For D defined in D.34j and st.t' defined in lemma D.28 we have &g
P
1 o
Ae
Cvd
F+ < Coe
a.s.
for some constants C, C".
Proof.
1 Lege Dr =1 gege (A, - EA(|F(-1)
L-1
(v0((v)
1
tr
E.
B
W
i=t
and denoting pf_ Pl((m) Ple(ar)) we have
Blel gxx
tr
BE-1:
(w'nprs
per W
[ pt
We
200
axx
W
Defining Se = ( a (xx') we decompose We into sum of two independent terms as
W W'P W' G + H
155

---

Published as a conference paper at ICLR 2021
Note that each H is independent of every other random variable in the problem conditioned on the features. Legertr
B&E' thus decomposes into a sum of four terms, which we proceed to consider mare
individually and show that they concentrate.
The all G ierm
dim S
1 eg BBew" G P
1 B u u1
1- 1--1 1-1 L-
p
W 2x
W'u
xx
where {uā- is orthonormal basis of 5t-1, 1 If a (ffat-1((n (xx) & a (x:' we choose
Pa Pa)- (x0 (uf-1 f1 = = (z) Par 1 a
(a)
(which are well-defined on B
et
).Using rotational invariance of the Gaussian distribution, we have
pd-1+ u w
W
U
22
1 R
W Pwi-iRal Wi- Rea- W Ru,
u
P cos g22 sinvt 1>09 Pg.>00 009)
ax) (x') then dim S and simply choose f-1 a (a) and end up with an identical expression, with r O. Since Pai0gg and uj
where gi midd (0, 2 I).
P
Si cos + sin >00 are vectors of independent sub-Gaussian random variables with sub- of independent sub-exponential vari-
Gaussian bounded by V n their inner product ables with sub-exponential norm bounded by for some constant C. Momentarily abbreviating 91 cos 2-1 92 sin t-1 Bernstein's inequality then gives
Posogg Paoog))
E
(Pozogi, Po Poi099))
Jt-1
<
2e
cd
(D.144)
9r.92
0.s.
for some constant c. Since on ct.e"
Bt-l:r" ex
< Cl, we obtain
dim 8
I Aegetr tr
B n
p
Ge) ow
CE
Σ
u.
E-1
1-
w
pe. ux
W' l t-1
u
i-=1
almost surely and thus
P Legerr Blu
1: G G uu
GO)
> C'
F(-1
< 2e cd
0.5.
for some C², and hence
Bl-1:f G G G¹ Go
Legp tr
P
2a0
xx
< 20d Ht-1
< 2c -0
(D.145)
E 1 peart
B B8-1-l ax
G pl G²) 2xx
a.s.
we
also contains the terms
B pp awe'
H
BF-1:f' ax'
H G zm
tr
+ tr
Considering the first of these (since the second can be treated in an identical fashion), we recall that H' is independent of all the other random variables in the problem conditioned on the features, and we thus have
1 Lggertr B l.f' G si H
p
1 ege tr
B lif' G''PÉ aan
W Pge-11
dim sf-1
Σ
Lege WW'w
i=1
156

---

Published as a conference paper at ICLR 2021
where W is independent copy of W of P u w Pst- B lef u. Hence conditioned on all the other variables, of W'wf is zero-mean Gaussian with variance 2|| ege Again from the bound
B
l:f" implied by E have
C
dim g
almost surely. Noting that W
E tr
Ba
B 1:
G
pe am
Hd
E E tr G'WI
0 (e-Wers
0. a
Gaussian tail bound gives
P tr Bl
p
H
E 11 gegrr Bo g pe
H²]
> v8 x(-1
xx
We
xx
(D.146)
26 07
0.3.
The final term tr
B&L'
oxx
BL-1d wa
H
pt aa
H
B ax
Pse
W
p W
W'Pst Pst
Due to the independence of W from the remaining random variables, this is simply a Gaussian chaos in n² variables. The Hanson-Wright inequality (lemma G.4) gives
P
Megerr
B H pt. H wxx
E ll see W
B 1: H p Ha uxx"
2 F(-1
< 2 exp a.s.
-ont min
Lege Pse-11
B P Ps--11 1 eew Ps--ni
2
Bi-1.l' Psc- ax
F
n1
< 2exp
-c- min
where in the last incquality we used the definition of gi:e'
follows that
T
B H
Pd. wa
Hd
E. We
septr
H P [BL
Pax Hd
2 de
Ft
IP
ogart B H pd Hd) E sg BB- H pt Hd
Vdl
F
a.s.
+PP
E 1 H
seartr
B H P aa H
E W
centr
B H ax
p
H¹)
IP
Be-1d H p tr
pi H²
E 1
gg
BB-1:e H pt pan H¹
ddl
xt-1
H
< 0.8.
2tr
Pse l1
B l:f' axx
Pse 11
+P
1 geg
tr
pt. ax
Etr tr H
P aa
> vdll
Ft-1
< Ce -od od
4.3.
(D.147)
where in the last inequality we used (D.144) and the properties of sli Collecting terms and using
D.145).
D.146). (D.147) we obtain
TP
oeer
tr
B faf xxe
> Cvdl
Ft-1
< C'e-od
(D.148)
W
a.s.
157

---

Published as a conference paper at ICLR 2021
and hence
P
L oee
De > CVde Ft-1]
L-1
(1-
(o((0(()
= P
1
II
tr
mar
E tr
g wir!
> Cvde F
n
W
i=
C'e-aad
a.s.
(D.149)
Lemma D.30, For @ E 1 and E [L], denote Lt(x) = supp(a'(x) 0). If 2 K then
n 4
P
min [[(x))]
2 1 2LCe
andfor any 0 S 1
2 [1(())
(-o7+12
TP
2
2exp
I''
n
where c, d, C, K are absolute constants.
Proof. Consider the activations at layer f. From lemma E.16 if 2 K we have
P [[laf- >
0| 2 Ce
Rotational invariance of the Gaussian distribution gives a'(x) It follows that
[w'a--(<o)]_
2
w (.11)
E [Z<())] |
> 0
E
af(2)>0
a 1a=~(()))) 0
E
We
| a (z) 11 > 0
From the symmetry of the Gaussian distribution
E
[[(=))]]
a
(x)
2
Since this variable is a sum n independent variables taking values in {0, 1}, an application of Bernstein's 's inequality for bounded random variables (lemma G.3) gives
P
[e(m)]
n 2 4
<P
[e(a=)]
n
n
| la (z)
>0
+P +
2
4
(dd
n² /6
< 2 exp
+ Cen C'e--''
+ n/4
for appropriate constants. A union bound gives
L
[[((x)]
> m1
< 2LC''-e"
2
4
158

---

Published as a conference paper at ICLR 2021
from which
P min [((x)) 2 n 21
21-22'''-''s
4
follows.
To prove the second inequality, we use the AM-GM inequality which gives
1/h
[()a]
4 2 L
(x0)
II
<
n
-=1
and hence
1/2
L 2
[=(x))
2 [1)2
2 (1+0)//2
P
1+
P
1
[[(2))
< P
2 L1 (1+
Convexity of the exponential gives (1+1)1/22 2 1+ +. log [1 +t) and for t s we have log(1 t tlog2. 2, giving
2 [x()))]
2 [x()))
P
1+ SP
L 2 1 log
We note that
[[(((=)1 d n
Σ lgeb
n
where by
2 of n2
mid Bern( (t) and g maxb; e-1 + #0} is the event that the features at layer
e- are not identically O. Since
Leeb
bf a.s. we have
2[((())
P
L
> log
P
b
PILLO log 2
[=1i=]
Since this is of independent bounded random variables, application of Bernstein's 's inequal- ity for bounded random variables (lemma G.3
gives
(-
2 2exp (-c
P
L2t
< 2exp
- B(bs)2 21
L
for some absolute constant c', where we used L 2 2 Hence
2 [e(x))
126 t < 2exp (-c
P
E SHARP BOUNDS ON THE ONE-STEP ANGLE PROCESS
In this section, we characterize the process by which angles between features for different pairs of points evolve as they are propagated across one layer of the zero-time network. This section is self-containel and such it will occasionally overload notation used elsewhere in the document for different local purposes. In particular, we will use the notation (xx) = [x]+ for the ReLU this section (and only in this section), and (g) 1920 for its weak derivative.
159

---

Let E N, with 2 2. Let and 92 be ii.d. (O0, (2/n) random vectors; we use to denote the joint law of these random variables. We write E Raxx for the matrix with first column gi and second column 92. and .'.".."""""".............. for the rows of G. I C [n] is nonempty and A E R=x we write As |l to denote the submatrix of A consisting of the rows indexed by S in increasing Published as a conference paper at ICLR 2021
E. DEFINITIONS AND PRELIMINARIES
index order. In such situations S will always denote the complement relative to [n]]
For 0 < 2x, define random variables
vr 92) a (g) cos + g2 sin
and
Úg(g1 (g) cos v + 92 sin 1/) @ (g2 cos v 91 sin
Because 82 separates over coordinates of its arguments and has each of coordinates the product of nondecreasing function and continuous function, it is Borel measurable. A key property that we will use throughout this section is that the joint distribution of (g) rotationally invariant; in particular, it is invariant to rotations of the type
cos v
sin v/
G G
sin
where v E [0, 2T.. Since we can write
G cos 1/
(G cos 2/ sin
3 (G
sin v/ cos
8, à
where all of the 2 vectors appearing above are elements of S) it follows by applying rotational invariance and the specific rotation given above that
(Vur ün) d (vo, 00)-
This equivalence is useful for evaluating expectations and differentiating with respect to v.
If 0 < e 0.5 and m E No with < define an event
Ecrm
n
(g),92) c (g S ".
SCIa] vE[0,25] |8|=
For each m, the set &c,m is closed, since linear map A. We further define
|| is continuous function of (g1. 92) Em for any
Ea,m U E E1/(2)).m KcN
so that
Ea,m
n (g) 10 < (g)))(9))))))
SC [n] wc (0.2m) =m
and E,,m. is Borel measurable. If is omitted, we take the constant i the definition to be 0.5. On Eam we guarantee that ||w, 2 uniformly on [0, T). Define function X,, by
V0
X, = 18
|
On ε,, we guarantee that v, # for every v, so X,, well defined; because EL Borel measurable, have that X, is Borel measurable, and moreover [x,] < 1, so X. Li for every p 2 1. Finally, define for < π
((v)= = E [cos-1 Xy),
(pv) = cos-1
E Koo, v,))
91 .22
9. .02
160

---

Published as a conference paper at ICLR 2021
E.2 MAIN RESULTS
Lemma E.I. There exist absolute constants c, C, C' > 0 and absolute constants K, K' > such that if d K and n 2 K'd log' n, then has
(x()) x(x) Cv log n
+ C'n -ad
Proof. Using the triangle inequality, we can write
(p(v) - p(v)) (Ø()) - cos |[X,]| |cos [[X,]] (pz)|
Choose sufficiently large to satisfy the hypotheses of Lemmas E.6 and E.7 applying these lemmas to bound the first and second terms, we conclude the claimed result (after choosing n larger than an absolute constant multiple of d dllog n so that the n -d error dominates the e error).
Lemma E.2. One has
( = cos-1 (11-*) 4)
COS v + sin
π
Proof. See (Cho & Saul 2009
Lemma E.3. There exist absolute constants C, C², C > O and absolute constants K, K' such that if 2 K and n 2 K'd log" n, then one has with probability at least n oo
>
dlogn
Joos x, 1 x.-,(u) s Cuyy
Con
The constant c the same the constant appearing in Lemma E.1
Proof. Under our hypothess, the second result in Lemma E.6 together with Lemma E.1 and the triangle inequality imply the claimed result (after worst-casing multiplicative constants).
Lemma E.4. There exist absolute constants c, C, C' 0 and absolute constants K, K' 0 such that if d 2 and n 2 k'd log" n, then one has
[[(sss' x, -(])(()))] Cv2 dlogn C'n
The constant is the same the constant appearing in Lemma E.1
Proof. Under our hypotheses, Lemma E.3 is applicable; we let denote the event corresponding the bound in this lemma. By boundedness of cos"1, nonnegativity of X and & s n/2 from Lemma B.2 we have |ccos Xx - ((vv)lze I n. Thus
E (cos' X, - x(v)) 2 < le (cos-) X x
+ CC" -
2
dlogn
Cuv
+ T² + -Cn
n
2 2d
+ +0"
as claimed.
Lemma E.S. One has
1. ip E C (0, T),, and i and & extend to continuous functions on [0, π);
2. (0) = 0 and (T)) = 22: (0) = 1, ((() = -37 and (0) 11(33²²); and
(p(T) ((T)) = 0:
3. yp is concave and strictly increasing [0, m| (strictly concave in the interior);
161

---

Published as a conference paper at ICLR 2021
4. ø -c for absolute constant on [0, T/2]:
5. 0 & and 0 > CC on (0, π for some absolute constant 0;
6. (1 - Ciu) - ( S v11 - civ) on [0, n for some absolute constants C1, 01 >
Proof. Deferred to Appendix E.4
E.3 SUPPORTING RESULTS
E.3.1 CORE SUPPORTING RESULTS
Lemma E.6. There exist constants c, C, C', C" """",, , 0 and an absolute constant K such that for any 2 1, if n and d satisfy the hypotheses of Lemmas E.9 and E. and moreover Kd log n, then one has
log C'n
cos"t X cos" [Xx]
< Cv
91.92
Si 192
n
and with probability at least Cn -ed, one has
dlogn n
cos"1 X, - EE cos' X,] S vv
+ C." ned
n1
Proof. Fix y E [0, π|. The function cos is smooth on () if < 8 < 1, and Taylor expansion
with Lagrange remainder this domain about the point [[X₂] (by Lemma E.23 E|X,] < v > O; we will handle v = O separately below) gives
have s
cos (x) = cos- [[Xx]- V1 /1-EXX,
(x E[Xx)) X
2 - 22)3// (x E[Xx])²
where & lies between æ and E[X,]. Using the fact that X. + almost surely if v > 0, which is
established in Lemma E.23 we plug in X. to get
((Xp)
cos [[x,] cos (X,)
(a E(X,))
2(1 E(X,)2)3/2 (X, E(X,])?
V1 X
(E.1)
where express & as function of X,. From Jensen's 's inequality it is clear
Elcos X₂] I cos E[X,],
(E.2)
so all that remains is obtain matching upper bound for the righthand side of (E. We will make use the following facts, proved in subsequent sections: there are absolute constants C, > 0, i [6], and e, 0, é E [5], such that
1. E[X,] - cgp2 + Cie (Lemma E.g
2. For each v, Var[X,] S CaeAlog n/n + Cam
can (Lemma E.9
3. With probability at least C4e-sm (Lemma E.
Cyn eaad one has X, E x CgU C /dlog n/n +
Let E denote the event on which property 3 holds. Combining properties 1 and 3, we obtain with probability least Cyn -ad
X, < 1 (cs/2)u² Cie -an
provided chosen larger than an absolute constant multiple of dlog n. Thus, defining
4
(Cie-ain ccn)
cs
162

---

Published as a conference paper at ICLR 2021
obtain for 2 Mu
c5 52 B[X,] S
C5 v2
X, -
(E.3)
4
4
with the second bound holding with probability at least - C3n rzd obtain using the triangle inequality, Lemma E.20 and property
Considering first 0 S v S Vo.
coos' E[X,]
E cos
cx,)) - E 1 cos E[X²1 - cos" (X,)
E gg cos" E[X₂] cos" (X,)
- E 1l X E x
- Ce + C'n C'n-o'd c'd
(E.4)
with the final inequality following from the triangle inequality for the e norm and the fact that Vo- Meanwhile, if v 2 Po. we have by E3.
0 S E(X,)
< max X E[X,]] - v²
with probability at least Czn we have under this condition on y
add Cad Using 1 - 2²= (1+ x) z) and B[X,] 2 ((E())) 2 0,
1
2
(E.5)
V1 - 22X,]
V1 E[x]]
CSU
and similarly
{(X,)
4
18 + π
1g.
(E.6)
2 - 3/2
Applying (E.6) and taking expectations in
we obtain by property 2
log n -Ce + C" -ad
cos" 1 x
E[cos'
x2)
< Cv
(E.7)
n
Together,
and (E.7) establish the first claim provided n is chosen larger than an absolute constant multiple of d dloog.. n.
For the second claim, we begin by using the triangle inequality write
|cos X, E[cos" xu] < loos" X, cos E[X.]] + /cos" E[X,)] - E E[cos"* Xu]|:
and then observe that our proof of the first claim implies suitable control of the second term. For the first term, if v < PD we use Lemma t. immediately obtain with probability at least 1 1- Can ogd that this term at most Ce-on, For 2 Mo. we apply property and the bounds (E5) and B.6 in the expression EE. to obtain with probability at least Cyn
dlogn
+ C'v dlogn
Jeos X. - cos E[X
Cup
n
n
which is of the claimed order when n chosen larger than an absolute constant multiple of d log n.
Lemma E.7. There exist absolute constants c, C.C, C"" > 0 such that in 2 C log n, one has
p(v)) cos"1
C'e- =0" E [Xo]
91.92
n
Proof. Write f(v) cos p(v), and
h(v)) = E[X)= - [(v),
so that h is the residual between the terms whose images trying tie together. We will make use of the following results:
1. The function cos is Höldder continuous on [0, 1]. so that |cos - cos- y/ S VIx if x. y 2 0. (Lemma E.20
163

---

Published as a conference paper at ICLR 2021
2. For E [0, m]. we have - 2222 s f(v) 11-08 1 (Lemma E.14
3. For all 0 $ v n, [h(v)] & ar + C2v2/n. (Lemma E.15
We choose large enough that the hypotheses of Lemma E.15 are satisfied. Define vo
2 C1/c2e-sin/2, We split the analysis into two sub-intervals: 4 [0, vo]. and 12 == [Po, m]. Choosing n larger than an absolute constant multiple of log n, we guarantee that I and Iz both have positive measure.
On 4, we proceed as follows:
|cos cos" (f +1) Vhll
$ VCeeenn + C2v2/n /n
$ v C 42,C20
S - 20in
The first inequality uses Hölder continuity of cos the second uses our bound on the residual, the third uses the definition of J, and the fourth worst-cases the constants.
On I2, we calculate
D2
/ + h/ S 1 |M] S Cie
C2
+1 +1-+-p0 c2u2
using the triangle inequality and our bounds on [b] and f. Using the conditions No and choosing 2 4C2/ca. we can rearrange to get
Cie + C2 1² C20²
n
which implies If + 1-111-000-
/2 By the control f(v) < -222, valid on 12, we get that both and f are bounded above by 1 02v² /2 12; moreover, because f > and h > can apply local Lipschitz properties of cos on 1z. This yields
[4]
|cos'! cos 'os'( + //+1
V 1 (supr, mf f h1)²
Cee an + C2v? /n
(1 (c2/2)u2222
O212/n /n
V302u/(2 - 2c202)
(2 - 222012
$ Cv-te-m + +Cvv/n
Se -0mn + C'v/nn
z01n
Above, the first inequality is the instantiation the local Lipschitz property; the second applies our upper and lower bounds on f and f f+ h derived above, and our bound the residual |/]; the fourth c2222 to conclude
applies the bound 0 < (v)
1 2 on I2, and cancels factor of v in the second term; and in the last line, we apply v E I, to get v 2 /C2e/ which allows us to cancel the ~ factor in the first term of the previous line.
To wrap up, we can choose the largest of the constants appearing the bounds derived for I and l2 above and then conclude, since I U l2 [0, π| under our condition on n.
E.3.2 PROVING LEMMA E.6
Lemma E.8. There exist absolute constants c, c, S, C' C > such that if n 2 C' and if n is
sufficiently large to satisfy the hypotheses of Lemma E. 15 one has
1- C'e--enn -e'n < E
X
< - c2² + -'enenn
9.
164

---

Published as a conference paper at ICLR 2021
Proof. By the triangle inequality, we have
Icos (p()
[[,,] cos ((v) _ [[x,] & (cos + [XX, cos (())).
Applying Lemmas 1. and E. 15 with 0, get
1 C² Ce C²v² [[X,] < 1 co2 + Ce Ce CCvu2/nn
which proves the claim if we choose n 2 2C' /c.
Lemma E.9. There exist absolute constants c, C, C² such that if n satisfies the hypotheses of and E. then one has for each E [T,
Lemmas E. E.lee
Cv log Cre
Var Var[X,] v
Proof. We use the following elementary fact for random variable with finite first and second mo- ments, easily proved using Var[X,,] B[x E[X₂]² and Fubini's theorem: in this setting has
Var[X.] Var (g) )[I Var ( .292)]--
gi
92
By Lemma E. there is event & of probability at least Ce-en which Var[X C²v / C"
Invoking as well Lemma E.12 we obtain
C""v"oog C.M
Var [x,] E[( + Var)Var(XX, ))]]
n
CvA logn Cne-on + FF E Var (g -)1
n
CoAlogn C'e-on
n
claimed, where in the second line applied nonnegativity of the variance and the Schwarz for any random variable X inequality, and the third line we used the fact that |||||a> I || in 1
Lemma E.10. There exist absolute constants c, d, C, C² C > 0 and absolute constants K, K' > such that for any d 1 such that n and d satisfy the hypotheses of Lemmas E.1l and E.13 2 max K log n, K'd d), for any E 10, T), one has
and
dlog n
P
X,
E [x,] 91.82
s v²
+ +Ce
2 21 -1-C'n C'n-dd
n
Proof. By Lemma E.1T
have
P
X,
[[X,]
< v²22
+ Cn
2 1 C'e-c'd
g2
Let 1/- = 1/.22 denote the cutoff function defined in Lemma E.31 and write
volgi, 92)
(g1.
(g(1
(1lvolg1 92 212) 4/ (11 92)
By Lemma E.13
we have
d n
P
E
EY
E [Yy]
< C²
One 21- C'n-o'
g2
g1.92
We have X, inequality
Y, on the event E,, by Lemma E.16 and we thus calculate using the triangle
E Y
E X2]
< E
[X, - Yu
E [1 Yg)
S Cne
91 .9
91 .9
91 .02
91.95
165

---

Published as a conference paper at ICLR 2021
where the last inequality uses Hölder's inequality and the measure bound in Lemma E.16 using the triangle inequality, we have
Again
E[X,]
E E| Yu
and so using our previous calculation and Markov's 's inequality, we can assert
E[Y]]
E
E[ S Cne cnn/2 Cne-on/2
2 21-6-m12
The claim then follows from the triangle inequality, union bound, and choice of n larger than an absolute constant multiple of log n and an absolute constant multiple of
Lemma E.11. There exist absolute constants c, c' cd, d d" C,C², ,,",, C Cs,Co constants K, K' > O such that for any d 2 1, if 2 max Kd, K' log n), then for every v E [0, π| one has with probability least -Ce-a
Cg > O, and absolute
Var Var[Xu(g1: }] Cav' C'e
and with (g1. (2) probability at least
C" c""e one has
S Ogpu V + + cCme
X.
E[Xp] 92
Proof. Fix T]. Let E E0.s.1 denote the event in Lemma E.16 which is in the definition of Xy. start by treating the case of or v = m. We have X² O deterninistically, so
variance is zero and it equals its partial expectation g2 with probability one. For the other case,
one has Xo = e we have
E [1,,, E S (1
Var Par(Xo(g1, )]
and since [E[1,,,] 1 One on by Lemma E.16 we obtain by Markov's inequality
P VarlXo(g1. )] 2 Cne een/ S <enon
This gives a suitable bound on the variance with suitable probability. For deviations, we note that
E
Xo
B[Xo] 92
and following our previous variance inequality but taking expectations over both Φi and g2 gives Var[Xo] Cne on, which implies by Chebyshev's inequality
P
X,
E[X,] 2 VCne-cu/2 -cn/2 g2
which is suitable deviations bound that we can union bound with the event constructed below, which controls deviations uniformly for the remaining values of v. We therefore assume below that
0 << < π.
Let (xx) maxfæ, }) which is continuous and differentiable except at x 1. with derivative
(x) 1/21/8. We note in addition that x S v((x)) and since up 2 we have for x 2 0 the bound
x/v(x) < 1. Define
(vo(g), g2). Un (g(g)))))
(g1gg22) ([(vo(o) .22) 2) v (g), (g1.92))))))
We first show that enough to prove the claims for Y., which will be preferable for technical reasons. On E,, we have Y, Xy. We have [Y,] < 1, and calculate
E (YD xx) = =E 1s; (Yo xx)
< E [ss
1/2
E (Y,
x
S C'
1
1/2
92
92
92
92
92
166

---

Published as a conference paper at ICLR 2021
where the first inequality uses the Schwarz inequality, and the last inequality uses that [X,] S 1 and the triangle inequality, and where > 0 is an absolute constant. We have by Tonelli's ' theorem and Lemma 16
1/2
E 90
E5
Cne
so Markov's inequality implies
E
[
1/2
2 Cne-onja
I e e0n22
Thus, with probability at least e-on/2, we have
E [Y, - x,) S C'ne-en/2
so that an application of Lemma E.32 yields that with probability at least 1 enn
Var[X, ) Var[Y>(g1. )] + C ne nen
where we have worst-cased constants and the exponent on n. For deviations, we write using the triangle inequality
X, - [[X,] s |x, - Yol+ Y, [[[]]
+ E|y] [[X,]]
and then note that the first term is identically zero on the event E,, which has probability at least Ce -an, whereas for the third term, we have
- E (Y,-x,)) 1/2 S C''ne ga
E 92
Y
E g2
where the first inequality uses the triangle inequality and the Lyapunov inequality, and the second inequality holds with probability least 1 e -cn/2 and leverages the argument we used control the difference in variances. Ultimately taking union bounds, we can conclude that it sufficient to prove the claimed properties for Y,
With 0 < < fixed, we introduce the notation
Ug vo(gi);
Ug
u, (g (g1.g2). 92).
so that
Y,
Ug
91 .a
1/ (1 gg
gz
For fixed g1, we will write (g2) = (g( g2) with an abuse of notation. For g E IR" arbitrary and 92 fixed, we consider the function f(t) Y, + tg) for t E [0, 1]. Writing f for the derivative of ff where it exists, at any point of differentiabilltyy we calculate by the chain rule
f(() (V (2 tg).g).
where
sin
e Ug az 92l 12) az 2 g1.92
V (g2) = 15( Uu (2)
(I az >0 C Ug))
og1. l12)
# 191 02 191. 122
Using the fact that
Unn >
Ugi
P(r. az 00} Ugi
where P Pl 20 is the orthogonal projection the coordinates where ug is positive, gether with the fact that
i 99 Ui g. Pto = Ptes 200 "g
we can also write
V 92
Yx(g2)
P P(ogr
(I
II-
25 (1(1/v (1/lvg .z (12) o2 2gr.02
Ugg
(1 U2
l22) Ug .2 l2)
wb) Mg or (221)| 1 oe ll2
(E.8)
167

---

Published as a conference paper at ICLR 2021
We next argue that f does not fail to be differentiable too many points of [0, 1]. Because &b >> 0, it will suffice to show that (i) + 1g -2-1 and (ii) t + (1 gg, (22) are differentiable at all but a note that at any point where "ggt.2--tglle <
set of isolated points in [0, 1]. For the latter function, [2] is locally constant, and therefore differentiable
8. by continuity we have that t - (1 get at such points. At other points, by Lemma E.21 it suffices to characterize t + (1 as differentiable at all but isolated points, and because ||0g). +1gll2 2 8. by assumption, the norm is differentiable and by the chain rule it suffices to characterize differentiability of each coordinate of L. ++ #9 which settles the question of all-but-isolated differentiability of (i) as well. We have gg aggi cos 92 sin v + ig sin v), again by Lemma F2.2 conclude from differentiability of + 91 cos + g2 sin v/ + tg sin v that t + o2--0 is differentiable at all but isolated points, and consequently is f. In particular, f is differentiable at all but countably many points of [0, 1]. Next, we show that f' has suitable integrability properties. Indeed, we calculate using E.8
V ga (g2))l22 - 80 801(
*/ (1 .2 (12)) sn Vo1.00
Ugg ug (12) 12
2 e= [|2)] Ug 9: 112
+
=8v -80/1 e (1 .= (g2))
(E.9)
where used Cauchy-Schware and &b in the first inequality and distributed and applied (w/)22 v/ and the estimate (x) < in the second inequality. In particular, this implies that (f) < |22 which a t-integrable upper bound for every g. Because Y, Yg(g. :) is continuous
by continuity of a sb, and the fact that t becomes constant whenever 1| .ga ll $. we can apply (Cohn 2013 Theorem 6.3.11) to get
(g + + = -Y(g))) + / (V Y2(2 go + tg). dt,
and since 9 was arbitrary, for any g2 E RR." we can put g = 922 -2 to get
Y(gg) (g2) + [ (V Yo + (1-1)) (1-2))2),00 g22 92) de.
Performing the expansion with g2 and g reversed and applying the triangle inequality and Cauchy- Schwarz then implies the estimate
(g) (g))) S 1192 -2lla L IV (tgl + (1- (1=g)) dt.
(E.10)
This relation is enough to conclude the result for angles satisfying co. where 0 < <0 S T/4
is an absolute constant. Indeed, (E.9) and (0) imply that Y, is C-Lipschitz, where C' > 0 an absolute constant; so the Gaussian Poincare inequality implies
C"
E 92
Y,
E [Y,] 92
n
and Gauss- concentration implies for any d 2 0
2 "M dd
p
Y,
- E YY
< 2e d
Because v 2 co. we can adjust these bounds to involve s and x² (respectively) while only paying increases in the constant factors. We proceed assuming 0 < v S co.
Let < Tg. < 1 denote a median of (, .). i.c., a number satisfying Pg. 2Tg.l 2 and
(Y, < Tg. Σ 2. and for each < < Tg. define
w. (g2) = max ( Tm - 8).
For any 0 < < Ta. notice that w. 2 Y,, which implies that PPW,, 2 T 2 PLY, 2 To. 2 1.
because Ty. is a median of Y;: and similarly P(w, < T 2 FLY < Tg. Σ 1. so that Tg. is also a
median of w.. The fact that w, > Y, implies for any > 0 that PYY Tgi > [) P(w. Tg. 1).
168

---

Published as a conference paper at ICLR 2021
and additionally if Y, I Tgi we have w, '9i -8, so that P(Y, - I S P s}. In particular, the tails of Y can be controlled in terms those of w, for appropriate choices of s. Additionally, by Lemma E.21 have that for each s, t + (g(g2 tg) is differentiable at all but countably many points of [0, and has derivative there equal to / +> (g. V g2 wa (g2)), where
w. (g2) 1 (, - g2 (g2),
which, following from satisfies a strengthened gradient norm estimate
V go Ws w. (g2)
I 8v1 wo(o)>> - Swllwaoo)>>
/ (1 .x (ll (2)))
(E.11)
- 8u (' 92 (2) s)?.
In particular, we obtain a
estimate of the form (E.10P.
(w=(g²) - w. (g(g2)l S | g2ll2 (. 8u Bu
+01 (1n-))))ll))a - 8)2 dt. (E.
For each g1. we define a set Sg. = {g2 11 ga 11112 2 1. Noting that the function 92 H [l (g) cos + 92 sin mn]l2 is convex 1-Lipschitz function (given that |sin 1), we have by Gauss-Lipschitz concentration
P Ug 0o
S E 1l ael.n..llll 112l t s e -ent?
and by Jensen's inequality
| ll2
E 92
9. 02
[2] 2 (cosw/l/ug 112
V2
where the last line holds because v S x /4. By Lemma E. there is gi event & having probability
least 1 Ce -en on which | || 2 2. so that for any 9i E, we have by a suitable choice of
our Gapss-Lipsehitz bound P92 [S,,] 2 en.een Thus, using the first line of (E.IT) the Gaussian Poincaré inequality and the Lipschitz property of w, (which follows from (2. after bounding by
an absolute constant) and Rademacher's theorem on a.c. differentiability of Lipschitz functions, we have whenever 91 E
2
128
Var[w.] <
E. [ 92 [g+)1 <
E
[
v/ (l 12){
g2
92
128
E [1s (1
('(1 az 12 (2)))
92
+
E 1 ge (1 きه (1 U1 (22))
ga
256
E (g on
(E.13)
g2
where we also make use of the fact that 0 < Y, < 1. Now, we calculate for gi E E and g2 Sg.
2
Y = 1-
Ugg
g
2
|
1 o
.9
> 1 2
|1.gg
Ug
ga
||u
> 8|| Ug
Ug
92
N 8l|gi (g) cos go sin v)
(E.14)
where the second inequality uses gi E, the third uses nonexpansiveness of a, and the first requires a proof; we will show that for any nonzero vectors æ, E IR.", one has
y
< æ y
(E.15)
|
|
169

---

Published as a conference paper at ICLR 2021
To see this, write 0 for the angle between æ and y, and distribute to obtain equivalently
1 | + cos 0) = | l2 2 y cos 0.
Divide through by ||:12. write K = || - 2 and rearrange obtain the equivalent expression
k² + cos 0)) 4K cos 0 + 2 0.
It suffices to minimize the LHS of the previous inequality with respect to K subject to the constraint
K > 0 and then study the resulting function of to ascertain the validity of the bound. Given that + cos & 2 O, the LHS is convex function of K, with minimizer K cos 0(1 + cos 0)- and therefore for any 0 2 x /22, the LHS subject to the constraint K > is minimized at = O, where the inequality is easily seen be true. If x/2, we have that the minimizer is positive, and we verify that after substituting the bound becomes
+ cos 0 2 2 cos 0,
which also seen be true for 7/2. for example by showing that the polynomial 2x + xx + is nonnegative on [0, 1]. This proves the inequality, so returning to (E.14, we have
y, 2 1 8((1 cos v) || l1 sin 2 22 2(sin 21 cos 1)) 92))
2 1 8((1 cos -) || 112 sin // ga 2(sin v1 cos v) 9 ll g2l12)
using Cauchy-Schwarz in the second inequality. By Gauss-Lipschitz concentration (e.g. following proof the third assertion Lemma E. there a gi event 8 and 92 event 8" each with probability least 1 Ce (sin (1 cos v) 2 0, we obtain that when gi E E n E' and when g2 E Sg. n
which we have (respectively) lIg ll I 2 for 1,2. Then using
y 2 1 - 32((1 - cos v/² + sin' v) = 1 1411 cos 2 1 - 32
where the final inequality uses the standard estimate cos y 2 1 - 0.5v², which can be proved via Taylor expansion. By union bound, conditional (in g2) probability at least C'e-c'n have Y, 2 1 32v2, that in particular, by nonnegativity of y,, and choosing n larger than an absolute constant, we guarantee with 91- probability least 1 Ce cn
assert that with 91-probability at least 1 Ce-en with
E [Y,) 2 g2
1
2 C²e
Tg. 1-3222²
(E.16)
Plugging the mean estimate into
E.13 113 we conclude with probability at least C"e-a'a
Cv
Var[w:]] <
(E.17)
We could have just as well applied this exact argument to Y, instead of W.s so we conclude the claimed variance bound from this expression. We have stated the result in terms of the truncations w, so that can be applied towards deviations control in the sequel. As an immediate application, use the fact that any median is minimizer of the quantity E[[ cl) for any integrable X and E
IR get with probability at least ccnn
Cv²
[w]] Tgx $ [[l,, - To s
w.
[w.]
S
Var[w,] S
+ a" (E.18)
Vn
where also applied Jensen's inequality for the first inequality and the Lyapunov inequality for the third. In particular, the same argument yields
Cv2
Yy) Tg. 82
R.
S
+ -nn
(E.19)
Vn
We turn to removing the t dependence E. 12) without sacrificing the dependence on Tgi- To
obtain Lipschitz estimate the subset Sg. need to control the norm of V Vgz W. the line segment between 92. g E Sg. For this, write a (x) = max{ı x - y, 0} for any E R, and make the following observations:
170

---

Published as a conference paper at ICLR 2021
1. g2 -9: cot (gg), that
Yo(g2)
Ugn
(sinv)o - 91 cot v (g2)
wi (1 21)) v (Il(sin via en - (g:)
2. for any z. (x()) = max (x. y) y: x max (x, y is the projection onto the convex set for 24 2 y vi). so in particular x + (2) is nonexpansive, has convex range, and satisfies (y(o,( + y) (x)) and thus
(x,
3. for any g2. (g(g)) Yu cot (g2) 91 cotv).
We write 92 = 0
9n cot v (g2) 91 cot v 82 = 0 gi cotu(g)) - 91 cot v, so that (2.12) becomes
(g,(g2) w. (g)) (w,(g)) w/ (g2)
10 ø2lle
8u - - (T 8)2 dl
1 92ll2
8V BV
v' (1l (g (T) s)² dt,
where the second line follows from nonexpansiveness and translation invariance the distance. Having reduced to the study of points along the segment between 92 and 92. we now observe
o gi cot v (1g2 + (1 t)ğ2)= (to gn cut (g2) + (1 1)0-9 cot (92))
ta an cot d (g2) + (1 ()a-g) en cot (g2);
because ent v has image included in the nonnegative orthant, which is convex. then follows from (1) above that
V (10.000--1-00:111 = (sin (g2) + (1 1)o--) (g2)
= ||t + +(1 - t)
and in particular
(0gg.. .22 + (1 - t)
1| 1 .z 12 +-+20 ++2(11 - 1) (0 02 Ug )+ + (1 - 1) ..
1 12 (1 1)3) 2
1
16
32
where the first inequality uses that a 2 and 92.g' E Sg. and the second minimizes the convex function of in the previous bound. We conclude that 92. Sg, implies that [[ (1 llz 1 for every t [0, and consequently (E.12) becomes (after an additional simplification of the quantity under the square root using Tg. S 1)
[w,(g)) w. (92)) < 16 - (Tg. s)llg' g2 l|2
(B.20)
so that w. is 16 / (Tg. -ipsichciitz on Sn. Then by an application the median bound 32 with 9I probability least Ce-on we have that w, is 16 /202 s-pschiit on Sg. For the previous assertion
in (.. if
nonvacuous, we need to take small: in particular, we have 32,2 1 if 1/8, which we can take to be the value of the ab- solute constant we left unspecified previously. Then for each such define L, = / and define
2
32 +
w. ( = sup {ws(g') Lsllg's 92ll2).
Then w. is L. La-Lipschitz on R", and satisfies úh, = w. on Sg, Evans & Gariepy 1991 $3.1.1 Theorem 1). By the Gaussian Poincaré incquality, we obtain immediately Var Lg, and using i. w. Sg.. we compute
F.. [w, V. .92
E S
[w+
w.)
<
R 92
s5, Iw-
vira
TP [st 1/2 (/||.s Wxlle2
91
Ce (llw)ll22 + l1bbll12)) S onn
(E.21)
171

---

Published as a conference paper at ICLR 2021
where the second inequality follows from the Schwarz inequality, the third holds given that 91 E and by the Minkowski inequality, and the final uses that W, and vú are both Lipschitz with Lips- chitz constants bounded above by absolute constants together with the Gaussian Poincaré inequality. Meanwhile, by Lipusst.ipschitz concentration, we obtain Bernstein-type lower tail
W - E [ - E exp (- (32222 +
ens²
(B.22)
and for the upper tail, it will be sufficient consider wo, which satisfies subgaussian tail (for any t 2 0)
- S exp (- c'.itt
P
S
E
[uo]
(-
(E.23)
24
92
Using the results
E.18
E.
E.21)
and the fact that Wa W. on Sog we get
u²
P
Y
E [y)
S
P
i.
E. [W S C
Vn
+'e C'e
+ P
(E.24)
g2
g2
92
92
92
Using d 2 1, we put s 2Cv² V /d/n + C'e-on this bound; using that < 1/8, and in particular
32.² 1. we can choose n larger than an absolute constant multiple of to guarantee that for
all 0 S < 1/8, this choice of s is less than 32v² and that Cv² dd/n S 32v². Together with the lower tail bound (E.22), these facts imply
d F
d
IP 92
Y
BY) - 2C V
E
C'e-c
S
P
Cu? d'd + cén
E
[
+ Cde
93
92
S se-ct
e
Mcanwhile, for the upper tail, we have for any 2 0
2t - C 12 -Ce
Y [YY,, 2 S 0p - Elool
C'e
P 9L.
(E.25)
vn
92
and if we put 2012 vd/n + e" our previous requirements on and the upper tail bound
E.23
yield
2 V -Ce
p
Y,
E
> 20
C'e
d -dn
g2
92
Combining these two bounds gives control of absolute deviations about the mean. By independence, we conclude
IP 91
Y2
E E[yo]
s
2Cv2
V + C'ee
2 (1 - 2e -od - -Ce -1
.22
92
21 2 -C -c'n
To conclude, we have shown that for every E [0, π] one has with probability least -en
Var[Xu(g1, 5) C²v² and with (g): 92) probability at least 1 - 2e- OO" one has
n"
1 d X, [ < C.'"
V +=+00 + nh
To simplify these bounds, may in addition choose n larger than an absolute constant multiple of log n, and n larger than an absolute constant multiple of d, to obtain that with probability at least 1 Ce -en
Var Var (e : 5 Cav + -dn-
and with (g1. 92) probability at least 1 - ccdd one has
d V
X, [[Xp]
+ C. -c""
n
which to be shown.
172

---

Published as a conference paper at ICLR 2021
Lemma E.12. There exist absolute constants c, C, C' and absolute constant K 0 such that if n 2 K log" n, then for every v E [0, π. one has
C log
Var E [[x(( 92)] -
+ C'e
.2
n
Proof. Define
(volg), 92) U. (g.
(g 92)
+ (1 (o (g), g2 2 ( va (g) (1)
where = 25 is in Lemma E.3 Then by Cauchy-Schwas and property 2 in Lemma E.31
(the case where either ||voll2 or 0 l2 0 is treated separately, since this case Y, = 0), we obtain |Y,] I 4, and
27
(uu v.)
1)
27
E
E[X,]
= E
ge <b( (2l2)))))) (
91
82
(vo: vo)
27
S E
S 16 S One
where we use the fact that if (g1. 92) E Em then ||w0||2 2 2 for every S S π and hence (((2))) = || in the first line, apply Jensen's 's inequality in the second line, and combine our bound on Y, with Hölder's inequality and the measure bound Lemma E.16 the third line. An application of Lemma E.32 then yields
Var E [ ge)]
< Var E
.22) + Cne-on -o Var E ( 92)]
Ce-on/e
where the last inequality holds when n is chosen to be larger than an absolute constant multiple of log It thus it suffices to control the variance of Y,. Applying Lemma E.26 get for almost all gi E R
10o13
E (91.
+ +E + +3. [s) 91, 92)] ds dt,
( 22222
where we follow the notation defined Lemma 13
We start by removing the term outside
integral from consideration. We have as above Y v 4. that [Eg. [Y,]| 4. Moreover, following the proof the measure bound Lemma 1. but using only the pointwise concentration result, assert that if n C an absolute constant there is event & on which 0.5 & ||voll2 < 2 with probability at least 1 - 2e- with e > an absolute constant. This implies that 91 & we have
1
1,
<
No
2
and since
S 4
< [[ vo 11
by the same argument used for Y,, calculate
6 (2oo))) -1| 2. (11ooll2) -1 18
||
<
1
11gg S Cn 5
12
by the Minkowski inequality and the triangle inequality. An application of Lemma E.32 implies that it is therefore sufficient to control the variance of the quantity
f(r 11 J/ S E(E, + +E + +,+Ea E Es) 91, 91. 92) ds
By Lemma E.37
the Lyapunov inequality, and Fubini's theorem, we have
(f( - E[f(ugg))) ( /
Σ e ( g1.g2)
E E, 91, 92)1 ds ds
ga
gi .9a
173

---

Published as a conference paper at ICLR 2021
Using the elementary inequality
(L
S g(s) ds SV f
f td [ (ss)
valid for any square integrable g 10, + and proved with two applications of Jensen's inequal- ity, and Lemma E.37, we obtain
2
S L
(f(v, -
Σ E (E, -E [.(s, 92)1 ds dt.
91
Thus, again by Lemma we have
E.37 the Lyapunov inequality, Fubini's theorem, and compactness of [0, m],
Varlf(z, 5)
Var Σ [ 91 92)]
ds dt.
(E.26)
We can control the variance under the integral using combination of Lemmas E.35 and E.37
together with the deviations control given by Lemmas E.39 E.41 to E.44 and E.46 since we have
chosen n according the hypotheses of Lemma E.13 particular, these lemmas furnish deviation bounds of size at most
C. ((V dlog + n nna + One-
that hold with probabilities at least 1 1-C"'nn-'' -'e--""nnee"""" for any d > 1 larger than absolute constant and suitable absolute constants specified above. We can simplify these bounds as follows: first, choose n such that 2 (2/c,") log for each i, which guarantees that the bounds hold with probability least 1 - C. n0 -nn-ccd -CMH d22"n/2, Next, choose 2 (2c /c"") "d log for all i, which 2 max C. In -nddd Similarly, we
implies that the bounds hold with probability at least also choose such that 2 (2/4) logn for each i, which guarantees that the error terms that are exponential in in the bounds upper bounded by C/e-dn/2 and, choose (2a//4)d (d)/c))log for all i, which implies that for all
C, c.(V logn +n + Cine S Cip ddlogn + max(Cr, C.
Finally, we make the particular choice d = 4/ min,,(c,,, e'),, or the minimum required value of d, whichever is larger, so that there are absolute constants C, C², C" such that with probability at least 1 n we have for all
log C'n -4 20
log
E (v. gi..g. g2)l
E (E.(v.
S scv
n
where the last inequality holds when n is larger than an absolute constant. With these bounds, we can now invoke Lemma with with Lemma E.37 to get
C log
log
Var
E (s. 92)]
n
n2
n
for different absolute constants C, C", C" > 0. and where the last incquality again holds n is larger than absolute constant. Plugging back into (E.26) and evaluating the integrals, get
Varls(u, )] s Cuslogn logn
which is enough conclude.
Lemma E.13. Write
(vo 92). (g(gi
Ye 92) ((looo() 22) (1 (1)
where 25 25 is in Lemma E.31 There exist absolute constants c, c. C, C', C" > 0 and
absolute constants K, K' >> such that for any 2 1, if n Z Kdt log n and if d 2 K', then there an event & such that
174

---

Published as a conference paper at ICLR 2021
1. One has
dlogn
VV E [0,
E[Ya]
E [X]]]
< v²
+ on
92
Ø1 92
if g1 E E;;
2. One has
P[ C'n-ed
Proof. Fix d > 0, and write
f(v. gi) [g 92)l
Applying Lemma E.26
we get for almost g E R
10
[(u,9))
+
E
+ 2 + E +4 + (s) 92)) dsdl, (E.27)
(1 22
92
where
n
σ (gns) )a p( 911 cot.)
(s..91.92))
(1 (2) ( 2 (122 sin"
(Do, v.) U. 12 1.lle
(wo Uz)
E2(s,91992)
(1lloollee 22))) 0. 12) & 20 12) 22112)
(wo. v.) (w,,,
l12)
Ea(s,91992)
(1l (2) (6 U.
(vo. a.) (u.,
v v2112)
(8.gg11992)
* ((1oolla u.
(wo, ns) b. 12 11
Es (1.911991. 92) = ((1vol12)) (1 0
(vo v.) (U., 0.) // (1 (12)
(vo, (v) 2,)2
(1) U. [2)
Ea(8, (1,.91922 92)
(1 0. 12 1 11
< 2) v>, (2) 12
Here we put Ei (0.
9l 92) (1 = O. which does nor affect the integral and which is
equal the limits lim,> E (v, lim, 7 (1, 91, 92) for every (g1, 92).
Momentarily ignoring measurability issues, it is of interest to construct gi events & of suitable probability on which we have
(Velogn
sup [E(0,a]
(v, 92)1
(.((v,r,
s C
+ aid
C'ne-dn (E.28)
g1.92
for each = 1,..., 6, and 9i event E, on which we have
1
2 2)
< Che
E 91
We can then consider the event = n & possibly minus a negligible set on which (E.27) fails to hold, which has high probability via union bound and which have simultaneously for all E [0, T.
((u,g)) - E[[(v,9))] <
E
91
91
(
2g
2
R.
(s (s. (5. 92)]
E (s(s.gi.
ds
1=1
g2
g1.92
d dlgn
CD2
+ -ed
con
175

---

Published as a conference paper at ICLR 2021
by Fubini's theorem and Lemma E.37
the triangle inequality (for | and for the integral), E.28
and using 1/² Σ n² and worst-casing the remaining constants.
To establish the bounds (E.28), will employ lemma Lemmal E.48 which shows that it is sufficient to obtain pointwise control and show a suitable s-Lipschitz property for each E [6]: following the lemma, these properties also imply Lebesgue measurability of the suprema immediately.
Reduction to product space events.
Fix
By the triangle inequality, we have for each
1,....6 6
R
E 91.
E [1 91, 91)22))
S E ga
E.(1.9..22)
E. [E, 91 92)]
(F.29)
91-92
91-93
Suppose we can construct g2) events E such that
1. If (gi.92) E E then
(V + 200)) n
dlogn
(v,
[v 92)1 sO,
Cne -in;
01.02
2. One has P[ 2 1 n
ddd C.
Then for each such i, can write
E
E.
(v, 92) - E
[. (.((,
92
91.82
E
(1g ((gy)) E1
E
(E (1.())11
82
91002
d dlog
C
+ add
+ Ch din
Y
+ EE 1 (ey (v, (1))))92))
E (. 91 92
(E.30)
92
using nonnegativity of the integrand and boundedness of the indicator for & in the second line. The random variable remaining in the second line is nonnegative, and by Fubini's theorem (with Lemma E.37 for joint integrability) and the Schwarz inequality we have
E
1 [gg
E, (1.,9.99)))
E (1.
< [1 [err
1/2 E
[-
(
E,
27 1/2
- Egi.. (1)))))))
g1.92
1/2
< C Cf d/ C..l re-e"""" d('n)
where the second line applies Lemma E.37 and the Lyapunov inequality. We can replace this last inequality with one equivalent to the measure bound on (E/) using subadditivity of the square root and reducing the constants d4 and ef by factor 2f 2. Using this last inequality, Markov's inequality implies for any 2 0
On 1dda + 122 22'"n
TP
E
1 (Eg
E (v.
E g1.82
(E((v,1..9))
Cn 3dd + C' 1/2 "
which, together with E.29 and after worst-casing some exponents and constants, implics that there 9i event that satisfies (the constants C and C' are scoped across properties 1 and 2)
176

---

Published as a conference paper at ICLR 2021
1. If gi E then
dlogn + (C, + C)n 3ad
E (v.gv.
E [, 91 92)l $ s..
n
(C) + Cn)ne" - min(c/,e?"))
2. One has PP[E] 2 1 Cn 3er'd O'ne 24'
Thus, we can pass from (91.93) events to g1 events with only worsening of constants, and suffices to construct the events E
Additionally, we can leverage this same framework pass v-uniform control from the product space to gi space. Suppose we can construct (g1, 92) events E such that
1. If 92) E E,, then
dlogg n
Av E m]. E,(1.91992)
[,, g1, g1.
SC,
+ 12 Gd
9..92
n
Cine-in
2. One has P[ 2 1 Cf''"" 1o'n-/f- cdd
ne
Then following (E.30) we can
Vv E [0, T],
E 92
(v.
E (E((vg91922))
91.92
dlgn
< <C
+ Cine
+
E 92
(eg
(v,
E (1))
91.92
To get uniform control of this last random variable, we can use Lemma E.37 which tells us that we have bound
Yv [0, m]
(1,
IE (v.
8a
gh.92
(E.31)
d dlog + n nedd + Cine- -cnn
C
+ (L
n
ga
where fa is in [R" R²), and has I norm bounded by absolute constant C. 0. Then Fubini's theorem and the Schwarz inequality allow us to assert
E [1 (1)=/((91 9z)] C. E [1e)))) 1/2
g1.92
which can be controlled exactly as in the pointwise control argument. In particular, application of Markov's inequality gives
Leg 92)) Cn dd 1/2 f < Cn 3 C'' 1 3n
P E 92
have uniform control of the quantity |Eg. (v,gg.199 Egg.22 (v, gi g2)]l on an event of appropriately high probability. In particular, we have incurred only losses in the constants compared to the pointwise case.
that, returning E.31
177

---

Published as a conference paper at ICLR 2021
Approach to Lipschitz estimates. We will use this framework for controlling the E and Es terms only. Accordingly, the sections for those terms below will produce results of the following type, for absolute constants C₁. c,s di c, Cs C²2, Ci C",, CttH > 0 for i = 1, 1,2, and parameters 2 1, 5 > O such that and are larger than (separate) absolute constants and n satisfies certain conditions involving d:
1. For each v E [0, л] fixed, with probability at least 1 -CC"n d/d - ne we have that
E 92)]]1 & Ci CI /llgg///n Cin and + Cinne" 4";
I ga
2. With probability least Che CC''n we have that (v..919)))))) (V, 92)lll is (C2 + Cin + J-Liscchiz.
We show here that we can use these properties to obtain uniform concentration of the relevant quanti- ties. Write M/ C₁ / log n Ci Ci Cfine-ii we are interested in showing that uniform bounds of sizes close to M hold with probability not much smaller than that of the pointwise bounds. By Lemma
E.48 it follows from the assumed properties that for any 0 < & < one has
P
sup
B ( 01, - E,E,,,, S SM+ +e (C2 + C2n
ME [0,]]
992
2 1 (c''c CCT"" nT""ne-a'n) Ke - + C2" ())
where K 0 is an absolute constant. To make the RHS of the bound on the supremum of size comparable
M, it suffices to choose & = Ci n/n// (C2 + Cyn! We have C2 + Chn < K'n2+5 for K' > an absolute constant, and so we have e S K'n3/2+0 for K' > O another absolute constant. This gives
o'd + CT.H ne (I') g-3 K'13/246 (CTI e CTT (CT -old
K' K diddoog
K'n 4/4/2
where K" > O is absolute constant whose value changes from line to line; and where the first inequality assumes that n 2 (2/c"") log n, the second inequality assumes that n > (2c²) /c4" /c"")dd log n, and the third assumes that 8 S d/d/2 - 3/2. Choosing d so that the value cf 3//2n is larger than the minimum value for 5 (i.e., larger than an absolute constant), then choosing 8 = c/d/2 3/2, and finally choosing d 2 6/cl/, we obtain
E. E. 91. 92) E 92)1 < 2M 2 2- 21-K----42-00
C" - Chth - d//4
sup <E[0,x]
82
where K > 0 is an absolute constant, which is acceptable level of uniformization.
Completing the proof. To obtain the desired control, we apply the uniform framework for the terms E, i = 2, 3, 6; and the pointwise with Lipschitz control framework for the terms E, i 1,5. We also establish high probability control of the zero-order term in Lemma E.38 The events need for the pointwise framework terms are constructed in Lemmas E.39 The events we need for the uniform framework are constructed Lemmas
E.40
E.44 and E.45
to E.43
and E.46
Because and d are chosen appropriately by our hypotheses here, can invoke each of these lemmas to construct the necessary sub-events and obtain an event which satisfies
1. One has
(Valossu + -n-odd -od + C'ne--c-
dlogg
VD E T]
Y g2
E Yc]
S Cr2
(V
g1.00 gg
if gl E E;
178

---

Published as a conference paper at ICLR 2021
2. One has
P 2 C" -e"d n"n-e"d
C ne"
We can adjust d and n slightly obtain an event with the properties claimed in the statement of the lemma. Indeed, choosing n be larger than an absolute constant multiple of log n, we can obtain C"ne-cnn constant multiple of d'log n, we can obtain C -cdd + +C "nn n/2 I 2C =edd, and choosing to be larger than an absolute constant, we can assert d'logn n/n + n -dd < 2 log n/n. This turns the guarantees of into the guarantees claimed in the statement of the lemma, and completes the proof.
c'n < C'e2-'n/2 and C" ne ne"ne"""n I e n/"n choosing n to be larger than an absolute
E.3.3 PROVING LEMMA E.7
Lemma E.14. One has bounds
1/2 S c08 <008p(u) 51- ou? v
Proof. Write f(v) = cos (p()) = cos v - cos where the last equality follows from
Lemma E.2 We start by obtaining quadratic bounds f(v for [0, 0.1.J. In particular, we will
show
1 1 1 S f(v) 11 WE 0.00.]]
(E.32)
We readily calculate
f(v) sin v + π v ("(v) - - ++++ π (v cos sinv).
Taylor expanding at == gives
1 infrep0.0.1) (t) 2² < ((v) < suPre(o.0.1 (t)
v2
2
2
We have f"(0) = 11 and sin I sin 0.1 on our interval of interest by monotonicity. The derivative of is cos v vin vı; sin is increasing the product of two inereasing functions (given P < 0.1), and one checks that cos(0.1) 0.1 (0n(1.1) > O: therefore v cos v S 0.1 cos(0.1) on our domain of interest. One checks numerically
ccos (0.1) + (0.10 sin(). < 0,
and this establishes f(v) < v/2 on 0.. I < x/2, we have cos 2 0 and sin 2 0, so that cos sin v 2 0 on this domain. This implies f"(v) 2 cos v 2 - for 0 < v S 7/2, which
proves infreju,n/21 (t) 1, and establishes the lower bound on x/2/.
To obtain (possibly) looser bounds on O0, т]. we use bootstrapping approach. The lower bound is more straightforward; assert the lower bound on [0, æ]. we evaluate constants numerically find that the lower bound's value at T/2 is x² /8 < 0. and given that f 2 0 by Lemma E.5 and the concave quadratic bound is maximized interval.
= follows that the bound holds on the entire
For bootstrapping the upper bound, we note that the equation
" ( ( -11
f'(v) = sin + - sin v
sin v/
π
shows immediately that f is strictly decreasing function of on (0, T). Therefore f(v) < f(0.)) on [0.1, T]. and so the quadratic function + 1 - π ((.(()))) which is lower bounded maximized 0 and the verification 1 x² 7/44 < 0 < f(0.1), is an upper bound for f on all of [0, T]; so the claim holds with π - ((-1)))))))
by - 1/2 on [0, r| by the fact that both concave quadratic functions
179

---

Published as a conference paper at ICLR 2021
Lemma E.15. There exist absolute constants c, C, C", C" 0 such that n C. C'logn, then one has
I
[xy] - cos cos))(()) S on + /n.
Proof. Write h(v)) = 0os sp()) B[Xx) By Lemmas E.24 and E.25 we have a second-order Taylor formula
h(v) h()) + //" [1(00) + / hs)s) ds dt.
We calculate (0) 0, since (v0, dn}1 (g))) 92)l 0, and P 0. We also have h(0) E voll -B[[1 (lE%) (writing 1), so this formula yields
v² suphh" 2 [E
[k(e)| S p(E
and we see that it suffices to bound h". We will use the (Lebesgue-a.c.) expression
1
U
[V(v] ((, bbo)] E c.
I
1 la
Distributing over the inner product and applying rotational invariance to combine the two cross terms, then using the triangle inequality, we obtain the bound
(00, 02)
(00, v.)) (Uu, (U,, Go)
[V(D)] S Bbb do)]
E e....
+
2T
1 am
|| 10 2a /0, 112
| 2| 1/
El(e)
B2(1)
(oo vo) (w),
Ue)
+
E
1 E
|
Ea(v)
We proceed by giving magnitude bounds for (v),), i 1,2,3. Because we are working with expectations, it suffices to fix one value v E [0, TJ and prove pointwise u-independent bounds; we will exploit this in the sequel to easily define extra good events without having to uniformize, and we will generally suppress the notational dependence of E, on v as a result. We will also repeatedly use the fact that we have ((EF) <. Cne for some absolute constants C > 0 by Lemma E. 16 We will accrue a large number of additive C/n and C' -en errors as we bound the E, terms; at the end of the proof we will worst-case the constants in each additive error and assert bound of the form claimed.
E, control. Let E i1 s 2} n { follo S 2}. By Lemma E.17 and union bound, have (uE9) < Ce-on, Define an event & = Em NE. The first step is to pass to the control
1 e (t.. vo) (1
1
E
E.
Do 100ll2//0.l1z 1
The triangle inequality gives
E - E < [[[e; (Uv, d0)}
|+
+
E Lem's 18
8 |ooll2
11
The first term is readily controlled from two applications of the Schwarz inequality, union bound, and rotational invariance together with Lemma F.29
E [1 (bv, do)] < E [[e
1/21 E
[10,,
[1
1/4
[[
1/4
- (u(gS,) + Ce Ca)
2/2
E 00 1/2
One + CCee
1/2
1+
0"
1/2
< 122122-on
180

---

Published as a conference paper at ICLR 2021
where in the last line require to be at least the value of large absolute constant. The calculation similar for the normalized term, except we also apply the definition of &m to get some extra cancellation:
(0 úo)
kiv, d0)l Un 122 vp
1 em V0
Em 18
- 41R
Em vel (ov, dn)
| 12 2oll2
4E| [E[ 1/2 E1l f1l [1]] 115) 4/] E |1
1/4
C"
1/2
S Ce on
1
s Ce on
where in the last line we apply our bounds from the first term and use n 1 to obtain the final inequality. Next, Taylor expansion of the smooth convex function + 1/2 on the domain about the point 1 gives
1 -x-1))+
3 J,, (x-t)t-- -)t dt.
x 1-2 =1 -1-
1)
(E.33)
4
Given that &m guarantees
|/22122 2 2, we can apply this to get a bound
18 (1-
1
|11oll2 Nw 112
1 -1) 3 | 2
(100
(1
1, 12
t)t t))- 5/2 dt
On E', we also have
21002=11 24, so we can control the integral residual as
11 w.ll
3 0 < 11 4
(110ol/110>11 1//- di s 18,384 384 (1lvol 1 1)²
where we replace the tighter bound that we get in the case || 10ull2111 10 2 1 with the worst-case bound from the other case. This gives bounds
18: (5 (1 -1) 384 (1lvol13 1 - 1)²)
< 11:
| 12
1
s و 2
(1001|2110
1
1).
Given that | 1l S on E", follows (60o tz)) S 4 on ε₁, so that (0o, v.) + here. Writing
L (wo. (10.00) (n) (1- |
(100, tn)) + 4) (1-
|| (
-4
| l2 / 1l
we can apply nonnegativity to obtain upper and lower bounds
E < IE 1c, (wo, )ي)
(1 v 1 1) + 4C (110o131 20113 1)²²
12 1) 50 (11m8113) 1)² }
E E 1e, (vo. (x)
(12g
2
where C 384.
181

---

Published as a conference paper at ICLR 2021
We continue with bounding the quadratic term arising in the previous equation. We have
IP E (0o, iz) (1 1
1)
4E
(1| |
4E [||vo|12] 1 21 121 +1]
s (1 - 2 1 [[1l0111)
< 1 - 2(1 - -(Cn + Cee= + 1 C")
S SCn'le + C'e-d'n C" T2
The first inequality applies the triangle inequality for the integral, the definition of & and Cauchy- Schwarz, then drops the indicator for Er because the remaining terms are nonnegative; the sec- ond line is just distributing; the third line rearranges and applies the Schwarz inequality; and the fourth inequality applies Jensen's inequality and Lemma E.18 to control the second term (to apply this lemma, we need to choose n larger than an absolute constant; we assume this is done), and Lemma E.29 to control the third term. Since n 2 1, this gives C/n + C"e -an bound on the quadratic term.
Next the linear term; our first step will be to get rid of the indicator. By the triangle inequality, suffices to get control of the corresponding term with the indicator for Ef instead; we control it as follows:
|B| Eey (6o. in>) (1|vol
1 113 1)]l
[1gg]] E (ir,, )² (1lvol 2
o2 11 1)²] 1/2
1/2
22 12 (110,111le,, 112 1)²]u² 1/2
One + coen
E
|
(One-on on + C'ee-dn))
1/2
|1
1
6
|| 0 112 1/2
(One + +C'e Cde)
1/2
LE
01000
1/4
E
00
11
1/4
+ E 01 8o [6 12)
Onne on + C'e--o")
1/2
(1+
1/4
(1+
1
C2
1/4
+
1+
C3 n
1/2)
n
n
I Cn'/2e + CCCe-d'n
The first line is the Schwarz inequality; the second line is the good event measure bound and Cauchy- Schwarz; the third line distributes and drops the cross term, given that all factors are nonnegative; the fourth line applies subadditivity of the square root function, then the Schwarz inequality to the resulting separate terms; the fifth line applies Lemma E.29 and the last line again uses square root subadditivity and treats the remaining terms as multiplicative constants, since n 2 1. Therefore passing to the linear term without the indicator incurs only an additional exponential factor. Pro- ceeding, we drop the indicator and distribute to get for the linear term
[ bx) (11001221 - 1)] = E (to. úز)) |001|| - (o.. ùm)]ن;
it is of interest to apply Lemma E.30 to these two terms to get the proper cancellation, and for this we just need to check that the coordinates of each factor the product have subexponential moment growth with the proper rate. For even powers of f2 norms of v,, this follows immediately from Lemma G.11 after scaling by v2/n; for the inner product term, the coordinate functions are à (g14)92 ( cos + I2s sin (g2 cos v- 91, sin sin v/), and we have from the Schwarz inequality and rotational invariance
E à ( cos + g2i sin v) (92s cos 914 sin }}]
< ((d((11))3)) 2k
which has subexponential moment growth with rate On 1 by Lemma E.17 and Lemma G.1l after rescaling by 22/n. These formulas also show that when 1, we have a bound of precisely n -1 This makes Lemma E.30 applicabb, so we can assert bounds
E (bo, ) (I|eal1 10 1)] - ( ( (E((
nE((bo)
C
n
182

---

Published as a conference paper at ICLR 2021
Because IE (g( )²]² n- this is enough conclude C/n bound on the magnitude of the linear term. Thus, in total, we have shown
C - +Ce "C" n
Eil
where we combine the different constant that appear in the various exponential additive errors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
Eg control. The approach is similar to what we have used to control E We start with exactly the same & event definition, and as previously define
Eg = E.
(0o, vv) (u,, 0.)
vo
2
100
and then calculating
(0o, v.) (u,, 8.) 1.)
E Ezl E Le ie
1100 ,, 12
< E em em 22) (ow ا 10
< E I g< (00; Ue) (w,, 0
0o [1a]
< 2 g 1/2 (00, )
1/4 E (er, )ي) 66 1/8 E
| 1/8
s 2 1/2 [1 [|]
EE |>>
[b]]
1/1
[1l
[|2]]
1/10
[[l
1/16
E
1/8
Se on + C'n 122n
using the same ideas as in the previous section, plus several applications of the Schwarz inequality and a final application of Lemma E.29 We can therefore pass to 5N with a small additive error. Next, we Taylor expand in the same way previously, except that larger powers in the denominator force the constant in our residual bound to be 227, and the event & now gives us a bound (éo, (ue, ,,) B | s 2 on the numerator, which we add and subtract as before to exploit nonnegativity. We get
< E L. (wo. vx) (w., 1o
2 (3 10.1 119) + (2 + 1)C (111o1 1 11 1)
2 (3 |0 |001||10~~11) 2 (112o 1)²) )
E, 2 E 1e (to, v2) (wv. ( 11
with C = 3 227 Proceeding to control the quadratic term, we have
E c (to, vw)( ).)) 10
12 l 1
I 4 (1|00
10 - 1)²]
2 [1v0
| 11 11 + 1]
I 2 (1 2E
112
I 20 2(1
(Cn
C + (1 ("n)")))))
S Cn- + Σ
3
6
)
(C'n + con 2k-1
2k -1 1)
k=1
2k-1
$ Cn + C²
Σ
(2k 1ak-1-5le-om cenj
Cn -o
The justifications for the first four lines are identical to those of the previous section. In the last three lines, we use the binomial theorem twice expand the sixth power term, and we assert the final line
183

---

Published as a conference paper at ICLR 2021
by the fact that O, that each term in the sum corresponding to j 0 has positive inverse power of n attached, and when j = 2k - 1 we pick up an exponential factor. Moving on to the linear term, in the previous section start by dropping the indicator. We control the residual as follows:
E Mep (o: (o )x) 11 (1
1 3)]
[[,
1/2
0o 12 è
12
2 (1l v012 - 3) 3)²|1/ 2
E [[4]]] 1/218 (1
vo
11
+ 31 vol 2 1/2
E| E[
1/2 [1
do
11
+ 3E |1 11 7z 11:1 11 12)
Ce + 222
The justifications are almost the same as the previous section, although we have compressed some steps into fewer lines here and we have omitted the final simplifications which follow from applying the Schwarz inequality to each of the two expectations in the second-to-lass line 3 times and then applying Lemma E.29 Dropping the indicator and distributing now gives:
= EE [[ U2)) (Vv, 0.)) vo 1ol co 12 v. 1
(vo. w,) (vz, (.)) 1/
(1 |
3)]
- 3E (0o, un (2 ). 1 12]:
to apply Lemma E.30 we check the two new coordinate functions that appear in this linear term: for (oo; 0,), we have
E [ cos + 924 sin r]] s B( 2k1 1/2
(((((n))2) 1/2
(E.34)
and for (év. u2), have likewise
E [[ cos v + 924 sin (g2, cosy 91, sin y* & ((((g1))) 1/2
(( 2k 1/2
(E.35)
both by the Schwarz inequality and rotational invariance. As before, an appeal to Lemmas G.II and E.17 implies that these two coordinate functions satisfy the hypotheses of Lemma E.30 so have a bound
E. [60, v.) oc)
voll
n ((>))
$
Noticing that
(((u, أ,)1 = ( اn)) -o(((o g2)l =0
by rotational invariance and independence, we conclude by dinates of v, and 6,
of the coor-
n n ( [( (1. -n ((
which establishes the desired control on Eg. Thus, in total, we have shown
C + C'e-en 122een
E2
71
where we combine the different constant that appear in the various exponential additive emrors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
E3 control. The argument for control of this term is very similar to the previous section, since the degrees of the denominators now match. We start by defining
(o. Up) (Vo. u (v ú
E =
1
184

---

Published as a conference paper at ICLR 2021
with the same & event as previously, and then calculating
(oo; vo) (u (x)) (vo, Uny) 1 1 1
[ Eal= = E. em
s E e veloel (o2, ;.) (vu, vy))l
< E e- (vo, vo) (un, 8x) (wo, w.
x - 1/2 (00. vo) "] 1/4 E (e,, أزحر 1/8 _ (vo. 22 25)881/8
< <2 1/2 00
1/8
E ||
1/8
B [1
1|
1/16
EE [1000012]]]
1/16
E
[1
261 1/8
I Cn /2 -m + Ce c'n
using the same ideas as in the previous section. We can therefore pass to Eg with an exponentially small error. Next, we Taylor expand in the same way as previously, obtaining
E3 S E
L (wo, vy) (oo, vo) (w,, 02)
(3 1 1 + (4 + +)C (1100l1@l - 1)²)
2
(3 | - c (11vol@@1 11))
Eg 2 E
L (wn, ve) (0o. (wu, v2)
with C= 3 3- 3,22 Proceeding to control the quadratic term, we notice
E 1 (io,, vo) (wv, iz) un) |101111100
1)
- 4 (1 0101 1)²]
Cn + C
since the final term was controlled the previous section. Moving on the linear term, as in the previous section we start by dropping the indicator. We control the residual as follows:
E eeg
(oo. vo) (uz, u.) (Vo. v.) (1loo1@ 1 - -]]]
[[Deç]
1/2 E
du 1 8 vo 2.1
@l /1 1 - 3) 3) 1/2
[[]e]]
1/2
[|
1
No
+ 3 00 11 1o 11 10.0 1/2
s El 1gy
1/2 E 60
12
1011 + 3E || 0o 11311 1 101l 11 12)
S Ce + Cn
by the same argument as in the previous section. Dropping the indicator and distributing now gives:
E (0o, vo) (wv, (vo; un) (1lvol1 w, 11 - 33] TE[ (to. vn) (Uv, in) (Vo. v.)) || | 1
-3o vo) (w oy)) (vo. e;))]:
to apply Lemma E.30 we check the one new coordinate function that appears in this linear term: for (vo, vn), we have
E (((o(g1)))r(gg: cos 92 sin z) < B[a(g1:))*
(E.36)
by the Schwarz inequality and rotational invariance. As before, an appeal to Lemmas G.11 and E.17 implies that this coordinate function satisfies the hypotheses of Lemma E.30 we have bound
((0, vo) (wv, iz) (vo, vn) (11vol1 - 33]
n (E)( (2)())))))))))))))) 11te://001011011111////////AC/D0
In-
+3n E((o)) (vo) (oo))
As in the previous section, using that (,, .. then allows to conclude the desired control on Eg. Thus, in total, we have shown
Eal C +C'-- 122 n
where combine the different constant that appear in the various exponential additive emrors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
185

---

Published as a conference paper at ICLR 2021
To wrap up, take the largest of the scaling constants in the estimates have derived, and the smallest of the constants in-iinn-eeeeeee- the-exponent that we have derived, order to assert
C' C'n
[W(v)) =
n
Matching constants in the exponent and choosing larger than an absolute constant multiple of log n, follows
+ C² V2
[n(v)] S Ce
which was to be proved.
E.3.4 GENERAL PROPERTIES
Lemma E.16. Consider the event
Ec,m= N
N (2.
cs (g) sc J
SCIn| ME[0.2x] |
Suppose n 2 max 2m, m 20}.
Then we have the following properties:
1. m(E,,)) Cnme--'n
2. We have &n,m = Ex.m for every Q E 0(2,, so that in particular L (GQ)
1 E (G).
Above, (n)) denotes the set of n x a orthogonal matrices.
Proof. We will show the second property first. For each
0, if Q 0(2), notice that
EormaQ Q
n
GQ < Isco (G
[oos v sin
<c-1)
SCinl |
vE[0,2x]
n
(G c<
I (GQ*u)lla <c
Sc(M]
wgst
|s|=m
Ee,m,
since the vector [cos v, sin E St, and (2)) acts transitively on $1, This proves the second property when e > 0; the result for set in the infinite union defining the c = 0, m event.
is obtained by applying the preceding argument each
For the measure bound, observe that &c,m C Ee, . if c', so it suffices to bound the measure of the complement for the particular choice 2. We start by controlling pointwise the measure of the complement of the event
ε0.6,m,u
|.6 I (Gu)ll2 5/33
SCIn
[S|=m
for each u E Si, then uniformize over the mee-dimensional manifold Si; we need to begin with = 0.6 instead of c = &/2 to survive some loosening of the bounds when uniformize. We have
U (G o (Gu)lla S 0.6) (G | (Gu)llz 2 5/3),
606,m,,
ScIn) [s|mm
that union bound implies
l (...6.m..) <
s (Gu)ll2 s 0.6] s (Gu)lla 2 5/3]
(c(n)
Is]=m
n1
(P I (m]= (91) - 0.6] [P[] (91))12 5/3)), (E.37)
m
186

---

Published as a conference paper at ICLR 2021
where the final inequality follows from right-rotational invariance of and identically- distributedness of the coordinates of 91- Let g E Rn be distributed as (o (2/n) so that
a(ğ) has the same distribution [mnoor (g1) By Gauss-Lipschitz concentration (Boucheron et al. 2013 Theorem 5.6), we have
(g)l( 2 (9)l + t S e ont",
Pill(())l2 s (ill - t] - c entt
since is 1-Lipschitz and nonnegative homogeneous. After rescaling, we apply Lemma E.19 to get
m7
m 5
< [ -
Vnya
Plugging these estimates into the Gauss Lipschhi bounds gives
(])l 2 1 + S ontt
P 1 - v m
2
onn?
t
S e
Vn
Putting 2/3 in the upper tail bound gives the control we need for one half of lower tail, we note that the assumption n 2 max m + 20} yields the estimates
F.37
For the
m
1
2
1
<
V Voyn n
10'
so that
m
1
t,
n
Vn 12 m
V2 10
(1/10) > 0.6. Putting therefore t 2-1122 (1/10) - 0.6 and one checks numerically that 2-2 1/2 in the lower tail bound yields
P[) - 0.6] S cn
Plugging these results into (E.37 gives the pointwise measure bound
(E5.6,m.u) S
for constant 0. For uniformization, fix S C [n] with |S| = m and consider the function fs RR? - defined by fs(u) s (Gu)ll2-
By Gauss-Lipschitz concentration, have
P||| > E||| + S eent
and by (Rudelson & Vershynin, 2011
Theorem 2.6), we have
E|| [|]] S V Vn S 4.
Let & [ < 5); then it follows that μuE) 2 e en On E, for every S, we have that fs is a 5-Lipschitz function of u. Let T. C S be family of sets with the property that E S' implies that there u' Te such that || u l2 < for each e >> O: by standard results (Vershynin, 2018 Corollary 4.2.13). Te exists and we have [T_l < (1 + 25 -122. Define
n
E0.6,m.u-
uET,
Then union bound together with our pointwise concentration result gives
(...6,,,)) <
n1 m
2
On & n E..6.m.r. for any u E St and any S, there is u' E T- such that [ss(u) - -fs(w')l < 5e. But since on this event 0.6 < fs(u') 5/3, we conclude 0.6 5E S fs(u) S 5/3 + -5e, and therefore the choice E = 1/50 gives 0.5 < fs(u) < 2. This implies
E n E 6. 1,500 C E0.5.m-
187

---

Published as a conference paper at ICLR 2021
Thus, by union bound and our previous results, we have Ia (m).s,m)
S μ (Ec & m.,.
S μ E (6.6..11/50) te
$ 2- 1502
c c-c c'n
+
m
which is the desired measure bound.
Lemma E.17. We have for each fixed v E 10, that:
1. The coordinates of b, have subgaussian moment growth
2p
p/2
E (
S
n
2. The event fll $ 2} has probability at least - eo::
3. The event (Vv [0, x || S 4} has probability least -
Proof. We have that the coordinates of i, i.i.d., and
(é.); (g11)922.
by rotational invariance. By independence of 91 and g2. we compute
E | = ((r
1
Eigh
2np/2 p2/2
2p/2
2
for each 1; the last inequality follows from Lemma G.I
This shows that the coordinates of 8,
independent subgaussian random variables with scale parameters most C C2/2n, have tail bound (Vershynin) 2018 Theorem 3.1.1)
P ا1lاأا12 2 1+ S onn
also taking into account that IE| (+))] 1/n. This shows that the event E" {11ò>112 S 2] has probability least 1
For the third assertion, we use the triangle inequality to get ||òy|22 < || + ||ga|12. which has RHS independent vı; then applying Gauss-Lipschitz concentration gives for 2
| 2 2 + S -ent"
using that E[llg < / g
Putting t = 0.5 this bound and applying a union bound, we
conclude that there is an event of probability at least 1 - -nn on which 1 S 4 uniformly in v.
Lemma E.18 There exists an absolute constant C such that if 2 C, one has
C"
1-
e
|1l1ol/22 [2] 1,
$1.92
where c, C², C > 0 are absolute constants.
Proof. For the upper bound, we apply the Schwarz inequality to get
Tollool22ll < B[] vo |0 1/ 1/2 <
by rotational invariance and Lemma G.Il For the lower bound, we will truncate and lincarize the product using logarithms. Let & Eo.s.o: by Lemma E.16 long Ce-en Define X /ooll/// v,, g ge, so that
20 we have 1u(9) S
|o [[a[G)]allo>(])] GEE, otherwise.
X(G) 1
188

---

Published as a conference paper at ICLR 2021
We calculate
E vo 1 [2]]
-[X]] S ((8)) + 11 1//2//2
C'e C² C2/n
using the triangle inequality, the Schwarz inequality, rotational invariance, and Lemmas E. and E.29 follows
-
so it suffices to prove the lower bound for X instead. Factoring as X = (1loolllg e (1le, 1el gs), we apply concavity of x +> log Jensen's 's inequality, and convexity of a +> to get
E[X] 2 exp (E [log (1l to 1221 + Læx)] E [log (1|>> 12 1s)
Σ 1 + E log Uy 21 + [g + E log 11,,l2l. + 1s:)]
2 1 + 2E loo 0o 2o11211 + s=)]]
where the last equality is due to rotational invariance. Now write Y = || 1g + Ige, so that by the definition of have Y Taylor expansion with Lagrange remainder of the logarithm about Y 2 1 gives
log(Y) log E
(Y -E
2((Y)² - B[]))
E|y]
for some E(Y) between E[y] and Y. Using 2 1 and taking expectations on both sides, we get
Bllog Y] log [[Y] 2Var[Y]].
Moreover, we have
E [ B|||o|||a]| Ce B[Ler || s Ce C'en
by the Schwarz inequality, and this extra exponential error can be rolled into the exponential error accrued via our use of X. In particular, we have
on S E| < + Cn
Since 2 20, if we also enforce > C₁ c- log(5//2) have 2/n + Ce 3 it follows by concavity of x + log(1 - x) that we have bound
by Lemma E.19
[
2 Ce- 2 -2ogg(2 (2 Ce Ce-mn)
log
which has the form claimed. It remains to upper bound Var(Y]; using that y = 1|1u11211 + 1es, we have
Var[Y] = E| 2 11 Ce on -11 11-2- Ce Ce-on)
2
n
- -
2
Ce +2
+ -e
+ Ce Ce-cn
m
4
+ -on
which is sufficient to conclude.
Lemma E.19. One has
2 < 11
1
91.92
Proof. By rotational invariance, is equivalent to characterize the expectation of [|a()))]22 By the Schwarz inequality, we have
||||o|||2| E
E
1/2 1,
189

---

Published as a conference paper at ICLR 2021
by Lemma G.I1
For the lower bound, apply the Gaussian Poincaré inequality [oucchron al.
2013 Theorem 3.20) and the 1-Lipschitz property of g + || (g) to get
E (110ol22 - E [|]] - 1,
so that after distributing and applying |||| vo [| = 1, we see that
2
1
E|| vo volla]
Because n 2 2, it follows
2
B Blllvolle [2] 2
2
where the last bound holds because 2m 1.
Lemma E.20. If0 S x, y S have
cos x cos"
yl S V12 yl.
Proof. Let 0 < x < 1, and assume to begin that y. We apply the fundamental theorem of
calculus and knowledge the derivative of cos" to get
cos" y [
cos
vI dt
The integrand is nonnegative, so cos cos y 2 0. Writing -T- -1- + t and using x 2 0, we get
1 1 y S
1
cos x - -oss
dl
v1 -
x
v1 x V1
This shows that /cos" x-cos yl S M y) when S y. An almost-identical argument
establishes the same when v, via the inequalities 2 cos'1 x c-coss - (V1 - y). So we have shown
lcos-! 1 cos yl s 1V1--22 - --
for arbitrary 0 S and 0 S S 1. Now notice
M x
y 2 < VI x
u IIVI 1 11
< (1 x)
9) x- y/.
which establishes Jcos x cos vl < v - vl.
E.3.5 DIFFERENTIATION RESULTS
Lemma E.21. For a < b, let J [a, b) + R be continuous function that is differentiable on (a,, b) except set of isolated points in (a, b), and let E R.. Then max J c) is differentiable except at a set of isolated points in (a, b).
Proof. Let C (a, b) denote the set of points of differentiability of f. and let B C (a, b) denote set of points of nondiffereniaabiiity of max(f.c). Because finite unions of isolated sets of points in (a, b) are isolated in (a, b), it suffices to consider only points x E
Fix A. and consider the case f(z) * Then because is continuous, there is a neighborhood of æ on which f * c. If f e on this neighborhood, then we have max(f,c] = f on this neighborhood; if f < c, then have max(f,c} In either case, this implies that max c) is differentiable at ø. and thus r is not B.
Next, consider the where f(x) First, suppose f'(x) 0; then by Rolle's theorem, find a neighborhood on which f('') > if x' > x and f(²') < c if x' < æ. Possibly shrinking this neighborhood, assume every point of the neighborhood is point of differentiability of
190

---

Published as a conference paper at ICLR 2021
f. Thus, for x' in this neighborhood, we have max(f(x')) = c. We conclude that max f. cf is differentiable at all points of this neighborhood except 2, and in particular is isolated point in B. A symmetric argument treats the case where (((x) < O, with the same conclusion.
f(z'), and for x' > 2,, we have
write f(x') = = o(12' =1) for x' in neighborhood of x, which implies max (fxx), = max(c, o(I2' =1)] = c o(I=' - x:)). In particular, Imax{
On the other hand, if f((x) O,
f(x)) c} of x - =l), which shows that max x, and thus x is not in B. This shows that every point of A n B isolated in A n B, and we can therefore conclude that max f. c} differentiable except at isolated points of (a. b).
differentiable
Lemma E.22. For 0 S π,, consider the function
((v)=
E
(v,
9.8: 11..
where
11ollllles 2
(v0 v.,)
ølv, (1) = cos"
Then & is absolutely continuous on [0, n and satisfies the first-orler Taylor expansion
P0
/"
10
(vv) (0)
E g1.82
1
dt,
2
and moreover & is 1-Lipschitz.
Proof. At points of (0, σ) where each of the functions composed in ø is differentiable, the chain rule gives for the derivative of the integrand as function of y
8' (V(, 91.
12
a
(E.38)
2
10
where we have used the result
1
|a0|2
valid for any + O. Because & guarantees that u2 o 0 for all v E [0, n/. we see that the integrand is continuous. Similarly, given that 11v/l2 2 on E,, we note that there are just two obstructions to differentiabilityy
1. The inverse cosine is not differentiable at [+1}:
2. The activation a is not differentiable at 0.
First we characterize the issue of nondifferentiabiiity with regards to the inverse cosine. We note that cos d(v, 91 92) to and volgi. 92) and Un, (g. g2) are linearly dependent. Because two vectors Uj u2 E R" have o and a (42) linearly dependent if and only if (w)) and (u2) have the same support and are linearly dependent on the support, and given that ||æ: > for each v, we have that there is
if and only if the Cauchy-Sehwarz inequality is tight, which is equivalent
being linearly dependent. Suppose we have (g1, 92) E ε and Do E (0, α) such that
submatrix of GMr, having positive entries and rank 1 (since the rank is zero if and only if the submatrix is zero), where
[
cos 1/
M, =
sin v/
191

---

Published as a conference paper at ICLR 2021
Write the corresponding 2 2 submatrix of G as X. Because rank M, 2 by no (), have rank X 1. On the other hand, if G midd ((0,2/n) we have
Gi G1y
G2i G2f
I has singular 2 x 2 minor]
Σ
P
rank
igi<jsn 0,
where the first line is union bound, and the second line uses the fact that x 2 submatrices of G i.i.d. (O 2//n)/n), and that the complement of the set of full-rank 2 x 2 matrices is positive- codimensional closed embedded submanifold of I It follows that the subset of & of matrices having singular 2 minor has full measure in E1, and conclude that for almost all (g), 92), we have cos (vV, 91 92) < 1 for every (0, F). Next, we characterize nondifferentiabiliy due the activation σ; by the chain rule, suffices to consider nondifferentiability of v, as a function of and then Lemma E.21 implies that for every (91,92) is differentiable at all but at most countably many points of (0, T... Next, we observe that whenever ny nonvanishing, one has
KToolee
Uu
(I
Un 2
P u
12
00
112/122
1
||
||
Pul. E l|2
|
1a
1
11a
where the first inequality due squaring the orthogonal projection and Cauchy-Schward and the second equality follows from distributing to evaluate the squared norm, cancelling, and taking square roots. Using the fact that orthogonal projections have operator norm 1, we thus conclude
P. d
l g1. 92)1
< C/
(E.39)
|
where the last inequality is valid whenever (g1, 92) E &1. Since
|| 1/2122 (gr cos + 92 sin v) @ (g2 cos 91 sinv)]le
Ig2 cos 91 sin 2/|2
g2 || + 1|
and this upper bound is jointly integrable in and (2) over J0, Ra x R", apply (Cohn 2013 Theorem 6.3.11) to obtain that whenever (g1,92) E & minus a negligible set, we have for every [0,
= (0.g..g))) + L g'(,.gr.g))d
In particular, multiplying by the indicator for E,, taking expectations over (g1, 92). and applying the previous joint integrability assertion for o' together with Fubini's theorem yields
I"
((v) = ((0) +
E g1. dt,
Jo
9: .2
192

---

Published as a conference paper at ICLR 2021
so to conclude the Lipschitz estimate, it suffices to obtain a suitable estimate on Egi. (a (v' 92)] In light of (E.39) we calculate more precisely
P
uul12 E2
Pelp do 22
E
E
12y
|
(I (I-
06gn (o
(d(g)) o (d(9)) [
E
(9)
-
(t(g)) o (222 [|o()))
E
1l |1o0
|
(0)) (
(ō(g)) 0g2)
n
Σ k=2
2 n
E
||o = k:
k
(
2
E
E (~)(()
X~x(k-1)
k=2
In the first line, we apply rotational invariance and unpack notation; in the second line, we use non- negativity of the integrand to pass to the containing event where Uh at least 2-sparse; and in the third line, we condition on the size of the support of 91. In the fourth line, we use several facts; first, we note that Pol (ā (g)) P (() P.l, implies that the operator no ((o(o))30) is itself an or- thogonal projection, with range equal to the (l]voolo- -)-dimmnsio subspace consisting of vectors with support supp(ve) orthogonal to vo. In particular, (91)) and P (m.200)) independent gaussian vectors, and conditioned on the size of the support of (gg)) the quantities l|o ll2 and [ P(. (c..>0)9)) are distributed as independent chi random variables with (respectively) and k degrees of freedom. An application of Lemma
④ g2) P Pt (()0)))=0)99 for any 92 RR. and that the commutation
relation P.
Do P(o
G.9 then gives
P
82112
11.
S 1,
(E.40)
|0
which is sufficient to conclude.
Lemma E.23. The random variable X, satisfies the following regularity properties:
1. If0 n, we have X, < almost surely.
2. (f11992) E E, then X,, is absolutely continuous
[0, TT) with a.e derivative
X,
U0
U v U
10,11
and moreover have Egg. oz..gzll Egg. (Xu).
[Xx1]]
1, the analogous differentiation result applies to
Proof. The first claim is corollary the proof of differentiability the inverse cosine part of f in Lemma E.22 and the observation that X, = O. The second claim is also direct consequence of the proof of Lemma E.22 and Fubini's 's theorem.
Lemma E.24. Consider the function
Un
=
[X]]
R.
1
|uoll22
91-92
91.82
Then f continuously differentiable, with derivative
f'(v)= = 11
U0 |
U 2
>}
|
13
91 .93
193

---

Published as a conference paper at ICLR 2021
Moreover, f' is absolutely continuous, with Lebesgue-da a.e. derivative
1
U0 VD
(²) - E 91-92
11
Ou:
I
1o
10
120l22
2o
Proof. The expression for f' is a direct consequence of Lemma E.23 continuous, apply rotational invariance of the Gaussian measure and E, by Lemma Lemma/E.1jj get
To see that f is actually
V
1
((())
= E
||
1
gh.9a
then notice that this expression an integral of continuous function of which therefore continuous. Moreover, the y dependence in this expression for f' mirrors exactly that of f; in particular, the integrand
1
Uy
1ew2122 1voll2
vo
2
20
absolutely continuous whenever (g1, 92) E E by Lemma
E.23
with a.e. derivative
U2 2
Wo 0
Oy: |ooll
8
voll
We can therefore conclude the claimed expression for f" provided we can show absolute integra- bility over & of this last expression, using Fubini's theorem in way analogous to the argument in Lemma E.22 But
11
I
U2
I
D vo
>>|
4
P
u
P
do
1|00/12
1 1
1
9. 92
91
4E
4,
using, in sequence, Cauchy-Schware and the lower bound in the definition of E the operator norm of orthogonal projections being 1, the Schwarz inequality, nonnegativity of the integrand, and rota- tional invariance; and Lemma E.17 We can therefore conclude the claimed expression for f" and complete the proof.
Lemma E.25. For the heuristic cosine angle evolution function cos s(v)
E 91 .9
[(vo, vuY),
we have the following integral representations, for its continuous derivatives: (cos
E [ d.}1
(cos
E (10, d.)) g1
Proof. The proof follows exactly the arguments of Lemma E.24 but with simpler integrand and
different integrability checks; the continuity assertion relies on Lemma E.3 Indeed, this approach gives that (Vu, vy) is absolutely continuous, with Lebesgue-a.e. derivative (No, by); we check
E (o bz) < E [1 91.92
E g
[1
1/2 1
91.92
by Cauchy-Schwarz the Schwarz inequality, rotational invariance, and Lemma E.17 This verifies the claimed expression for oyp.. For the second derivative, apply rotational invariance to get
(cos (p)
.92 (o)),
which has an absolutely continuous integrand, with Lebesgue-a.e derivative (vo. Cz).
Checking absolute integrability, we have as before
E ( r <
E 1 <
g1.92
g1.92
by Schwarz, the Schwarz inequality, rotational invariance, and Lemma E.17 This estab- lishes the claimed expression for (cos o
194

---

Published as a conference paper at ICLR 2021
Lemma E.26. Let dh R - R be defined by (xx) (2)) where th.25 is the function
structed in Lemma E.31 Then the function
( Un)
[(ugi))
40 No
92
satisfies for all E [0, m and Lebesgue-a.e. 91 the second-order Taylor expansion
11001
o(mn) p(-m)) 9i col
f(v.g1)
E
* ((1voll2))b (1 sin
9a
(Uo, Uz)
(vo. / 2 2
(1 volloolllll (2) 112
(1 22)
12)
(vo, u.) uu 6,) " (11
v.
2 (wo. v.
(vo. v.) 0e 3 [2,12))
+ E
((lvoll2)
(([voll2) ee |
(wo v.) (v, ú, )
(wo. u.) (u,( i 2 2
+ E
ds.
(((1ooll2)) 12) v.
((
where previously notation in this expression introduced in (E.44)
Proof. Take 91 E RR such that f(v, )) exists and is gi-integrable; by Fubini's theorem such have full measure in R". Because v > 0 and (((l==ll)) is locally function of v constant when- ever || < 7. we need only consider nondifferentiabilit of a when assessing differentiability of f(,,9)). By Lemma E.2T we conclude that (() is differentiable at but at most countably many points of (0, : since v 0 and e is smooth, f is continuous, and we therefore ap- ply Lebesgue differentiation theorems (Cohn) 2013 Theorem 6.3.11) to f provided we satisfy the standard derivative product integrability checks. Writing
(vo, vv)
@bv. (V.91,92)
the chain rule gives (at points of differentiability)
(wo. ve (11e: 8, (1 (b (b 12
o' (v..1.92)
(1(1lool2)) , 12) 8
this expression, we follow the convention 0/0 to account for the possibility that | (in this case, the s term handles the denominator). For product integrability, we Lemma E.31 to get | < C' for some absolute constant C' > together with inequality get
Schwwrz and the triangle
(V. 16/]00||2||@ 64C 64C||/oo|allee.ll (
and applying the Schwarz inequality, rotational invariance (to eliminate dependence in the sulting expectations) and Lemma 1. we conclude that o' is jointly absolutely integrable over [0, n] (g a & μ). We have therefore a first-order Taylor expansion [(v.g)) ((0.91)
No (1)
E
1 wollo )o (1l ll2 12 dt.
(((1voll2))1 0e
92
92
(M)
3²(1)
We have
1o 1
f(0,91)
=
1
22
(b 10 )
195

---

Published as a conference paper at ICLR 2021
since tu depends only 91- Next, show t-differentiability of the inner expectation. Our aim to apply Lemma E.27 to differentiate E1 and Ez. We first focus on E₁; distributing and applying linearity, have
(12))) cos Mli sin (0[ voll)) 1/ (
Ei(e) Σ
(gin cos + 92s sin v)
ò
92
We have shown absolute integrability of the quantity inside the expectation above; we can therefore apply Fubini's theorem and the previous definition write
o (gr)) cos v I1 sin v) (g cos + 92s sin v) (2))1))))
[]
B1(v)
E 1=1 (925)0
(E.41)
15 (o 92) 192c
For each i E [n.]. write T. : R" pn for the linear map that deletes the i-th coordinate from its input, and let R; R R=-
R be the linear map such that ū (g))) = g. With ga fixed (in
the context of (F.41) ), if we define
a ( (g cos In sin f1 (v, (llvo (g n. (122 (1 a. (g2))
then we can write
E E1( (925) Jig
E [i, (ale cos v + I2e sin
19
Thus, to differentiate E1. it suffices check the regularity of fI g) and apply Lemma E.27 As before, & > 0 and vé smooth implies that fi is continuous on [0, T] x I.. For integrability of J. we appeal to the Fubini's theorem justification that we applied previously. For absolute continuity, we apply Lemma E.2I to get that the derivative of f with respect to is, by the chain rule,
91s cos + 9 sin (g cos V- 91, sin v (1 (2)) 8x)
(v, (91+)
*11 v> e, 112) 1
at all but at most countably many values of v; and the triangle inequality, Cauchy-Schwand and Lemma B.31 yield
[(( (g
+ (00000000)0000000000000000011))1
S [gl + (91)) (16 64C + l122ll2))
{ [9n.)) (16 64C + [|T: Ig))),
(E.42)
(we apply square root subadditivity in the last line) which is jointly integrable over [0, æ| x IR and moreover over [0, T x R". We conclude absolute continuity of (.(-)) and the integrability property of f/ Finally, for the growth estimate, we obtain an estimate for fi similar to the one we just obtained for f follows:
Ui (u < 16/g + lons));
(E.43)
the RHS of the final inequality above a linear function of [g), and when [g| 2 1 we can therefore obtain [1, g) I 16([91/ 19g), which is suitable growth estimate with p Then as long as 9ix # 0 for all i (such form a set of measure zero, which we can neglect), we can apply
Lemma t. get
( ( cost I2 sin 1)] -1 filt, cot t)p( cot
(p)
(2)
dt
sin
The estimates E.42 and E.43)) show, respectively, that f and f1 are absolutely integrable functions of (v.g2). We have
(
fi (t, cot
w1 vo (0iigg2)la (2)))2)))((((v(lll)))) R. (ot cot (
so that Lemma
E.3] and nonnegativity give
fa (t, -9 cot t) pl 91, cottt sin²
< 16 (gu)
914
sin pl 914 cot.).
196

---

Published as a conference paper at ICLR 2021
As in the proof of Lemma E.37 in particular using the estimates (E.52
(E.53)
control the magni-
tude of the RHS for all values of t, we can conclude that the Dirac term is absolutely integrable over [0, T x R". An application of Fubini's theorem then allows to re-combine the split integrals in the previous expression:
E fi (0 ("(
(-
-91 (Eor
Egg [[, g2s)) cos 92s sin [)]
Eg. lfi (t, -91 cot ) dt.
E1(D)
Σ
21 2n cot
92
sin
We notice that
ogi
cos 1001100s1
921 sin v)
(91((111) COs v + 92(1-1) sin
(g f1(-11: cot cotto (T(g2)))
0
( COS + 12(;++) sin v)
Ogin cos + I2n sin v)
and thus motivated introduce the notation
g't (t..g.992) = f 91, cott, (g()));
(E.44)
0g( = v. (t..9g9(tt.
We can then write
fi (t, -1 cot p1-
I1: cot
o ( ) p(p((1p 91, cot
91
sin²
(1 (1 sin
Finally, apply linearity of the integral move the summation over back inside the integrals, obtaining
(wo: io)
E (v) g2 (b100)))2)
1'"
p(=))"(--) (( (i sin
(n ve
) de.
Σ
M ve ae 2)
(Po. (m) bn (1 1a)
ga
Noting that, in the zero-order term, the only 92 dependence is in do o(g)) @ g2. we apply independence of 91 and 92 to obtain finally
( p( 01 cot
(1)
dt
(1Ivoll2))(11 l22) sin
(vo, v.)
(vo, úe) (ve, úz) 2 ((ellll)))) ( (2) v [ 122) |
Jo
E
(
di
We the same type of argument on Eg next. Distributing and applying linearity, we have
E2(1) =
E Im (g1s cos + 92v sin v))
82
where in the previous expression
A (vo, (g cosv + 92: sin (g2 cos v 9n, sin v) (((vll))
v (1
By the preceding (product) absolute integrability check when taking first derivatives, we can apply Fubini's 's theorem to split the integral as we did with E, We define, with fixed, the function
(vo(g)), un (g # (gg))))
(v. = ((l1oo())lla)v(((e
(g), #e m (i(g))))12
197

---

Published as a conference paper at ICLR 2021
where in the previous expression
B (g cos v + g sin si cos v/ I1 sin
(g)))
so that
n Σ
B2(1)
E
fz(v. 92s (91) cos v + 92r sin sinv)]
(ony):ifa
Now check that the hypotheses of Lemma E.27 satisfied for f2. The continuity argument is identical to that employed for f1, as is the joint absolute integrability property of f2. For absolute continuity, again use db > smooth, and Lemma E.21 to obtain the derivative at all but finitely many points of [0, π (by the chain rule and the Leibniz rule) as
(wo, (g cos + gin (g cos 9l, sin & (1100=122)
(2
volvol122
,, 12,
(vo. (g + sin ())
914 sin v) (12,
(1
(vo. ( cos c oon
cos
(1(211
cos I htto (1 i.)
(wo. cos v ou
2n)
v0 v0
(wo. vn)a cos v + gsin (3) cos gu ")) (e,, 0n)
<5( 1b )b 0. [,
Because v or v and our convention handle cancellation in the case where ||u,|22 = 0, we can proceed when necessary with the convenient estimate
a (gi: cos + gin 1
0. (g n; (g2)))ll
which follows from the fact that | < ||u|| for any E RR As with E₁, we then estimate the magnitude of f/2 using Lemma E.31 Cauchy- Sehwar,, the triangle inequality, and square-root subadditivity (skipping some steps that wrote out in the E estimate):
C" C
(f(,gg)) < 64 + lolll
(lóulla
2
+ (1a) + gi (2 +8|| 811bb)ll))
(1gg)
lla + (gm + (el) 2 -
I 64C + 19ell
+(gg (g (2 + 8(1gg1|2 + (g2) (gl))
(E.45)
which is jointly integrable over [0, x] x IR, and moreover over [0, π] x R". We conclude absolute con- tinuity of f2( .g) and the integrability property of fz. For the growth estimate, we argue similarly to our bound on f' to get
152(2,)) < 64C (oollellll + 1911))
< 64C ll (lg)2 2 [lg| + 191:13);
(F.46)
the RHS in the final inequality is a quadratic function of [gl. and we therefore obtain suitable growth estimate with 2 and C' = 664''|91110 [1 + 2) 19asl") 12) soon as lgl 2 1. We can therefore apply Lemma E.27 to get that for all but a negligible set of g1 that
[."
(Egz (5 92: (o cos + Ize sin t)] -1 falf.-guu gi cot f)p(- cot
)) di
E2(1)= =
E (92; k
E [2 g2))((1)))
.921
sin't
The estimates E.45) and (E.46) show, respectively, that f/ and fa are absolutely integrable functions of g2). Because a (91 cos 91 cot sin v) O, we have (fortuitously)
fa (1, Iie cot 0,
198

---

Published as a conference paper at ICLR 2021
that there is no Dirac term in the derivative expression for Eg. An application of Fubini's theorem then allows us to re-combine the split integrals the previous expression:
E[f (2)))())))))))))) /."'
E2(1) = Σ
Elff(t, gx) cost gos sin dt.
g2
We have by linearity of the integral
(vo. 0o) 1| moolllo l (I O,
E E(f2(0,0a))tgnn)) (2)))( 82
+b ((1o))))
where the last equality applied independence of gi and g2. as in the zero-order term of E Finally, we apply linearity of the integral move the summation over back inside the remaining integrals, obtaining
(vo. ú.) (we, de) ( (vo, vx) l BB (1 (20l2))
B2(u)=
E .92
4 Nn (2) ox |1 ([11ool2))bb we [|2) 1l
(wo. vo) (1 (l2 1all2 (wo. (e) ie) 2 (1
+
IE
(((1ooll2) (1
(11
g=
(wo, v)) (1r, 2 (11o))))))
(wo. wx) (wr, by) 2 (1ee 1 (2)
EE
dt.
(1(1 |0 1122) 1 (1 (ll))l))))))) (0 o 122 1
g2
Since ((V,9:) 5(0, - 1o Egg (1) (2) dt, the claim follows.
Lemma E.27. Let denote the distribution of a (0, (2/n)) random variable, and let denote its density, Let u E R and + O, and let f [0, x R satisfy:
1. f is continuous in its second argument with its first argument fixed;
2. f is absolutely continuous in its first argument with its second argument fixed, with a.e. derivative f";
3. f and Ir are absolutely integrable with respect to the product of Lebesgue measure and μ
over [0, π] x R;
4. There exist p 2 and C O constants independent of x such that (f(v,z)) S Clai whenever |x| 2 1.
Consider the function
q(v) = J (v. x:) x sin v) du(x).
Then is absolutely continuous, and the following first-order Taylor expansion holds:
q(v) = 9(0) [ (- (u, p(-
cot f((,,z)ō(ucoss + ssin dp(z)) dt.
sin2t 2
Proof. For E N, define
0 m.
< 0
dm (x)
0 x S m x 2 m
Then 5m < Fm is continuous, hence Borel measurable; o. + & pointwise as m - oo; and om is differentiable on E except at E {0, m m.),, with derivative o.. = mllssssm Morcover, we have
J m mlloosm-idd dr
and the first-order Taylor expansion
om (x: = /" ml mloswssm dz
199

---

Published as a conference paper at ICLR 2021
Define
Im (m( (v) -
sin du(x).
Then at every v [0, T.. we have by assumption
J I(), cu + æin v)] du(x)) & [((exx)(du(()) < t00,
so that the dominated convergence theorem implies
lim Im (v) = g(v). m-00
By the chain rule, the expression (2) = max - m max 0}, -1}. and Lemma E.21 v H Fm (u + sin is absolutely continuous function of v E [0, m). and we therefore have by the product rule for AC functions on an interval (Cohn) 2013 Corollary 6.3.9)
4m (v) (0)
(alls)) at (((t:))dmm(ucost arnn
(z u sin -ssns
We have
S (( + dt J (f((tttt))ddd to0, by assumption, and
(1 (1((,x)((z usin (iint)luse costto 15m- dulz))
(J
1/2
f(t,z))
(1 (r - usin ()) du(x))
1/2
$ Ct (lul ((aduul))
(fe du(x))
< +00,
by the growth assumption on f and the Schwarz inequality. Applying compactness of [0, œ and the lack of v dependence in the final inequality above, an application of Fubini's 's theorem therefore yields
4m (v) qm (0)
+ 1-"
I du(x)) dt f((tx)) cost - sin
+ fmf () cost - u sin t) cos
o uSm
By dominated convergence and the first the preceding two product integrability checks, is clear
lim ((, + sin () du(x)) di = J." / ['11, + rsin () da(x) di. m-
For the second term, need to proceed more carefully. For A E N sufficiently large for the integral to be over a nonempty interval, we consider
(m, f/ dt S. mf(t, cost sin t)
k-
2²
2nd
u f+r sin fSm m dx,
K-
V2rc²
which a truncated version of the integral constituting the second term in Jm. with a change of variables applied explicitly show the density corresponding fi, and where write e² 2/n.
200

---

Published as a conference paper at ICLR 2021
In particular, by the caleulation used to apply Fubini's theorem in this context previously, have by dominated convergence
lim (m.((v) 1/" (t) cost sin /) og oga
x:
t+ t5m (1(x)) dt.
k-+00
By the product integrability assumption on and Fubini's theorem, integral for fixed i, and due to our truncation we have 0 < L < ı,; we therefore change variables x sin" in the inner integral to get
consider the inner R-
{ mf (t. s
COS sin
8m.k(1) = Jn-1
x
u
sin 1 ttr dæ.
2
0 < t and x E R., define
cost I sin
g(t,2)
2n sin
sin
V2Te2
so that, after an additional change of variables + x - u coss t, we obtain
p- di J (1.x uos () 1 oosesm-1 dr.
Qm,k (V) m
Using the growth estimate for f, we have
10(1, x - cos ()l S C u cos || cos ul exp (-6x u cos t)
(x
sin
2c sin?
where C 0 depends only
We are going to bound this quantity under the assumption that |x|| I /u//2, where we use the assumption u/ > 0. First, note that when r7/4 S t < 3 /4, we have
sin 2 1/v2, and we always have sin / < for 0 S S क; so in this regime
(x uco0s l)2
lg(t, x - cos t)] S C2P/2+1 u cos f| cost us| exp
2ca
which is a continuous function of (t, x), and is therefore bounded by a constant depending only on e, J, u over the compact set [=/4, ///43/3/// x [-u/2, u/2]. Next, we consider the case < S a/4; by the symmetry sin(T π 1) = sin i, controlling (g(t, x: u cos /)] in this regime implies control of it in the regime 3 /4 S < и. Here, we note that by our assumption on and the triangle incquality
Ix u t| 2 /u|/coss |x]] 2) 2 Klu),
2 [[oos
2-1 -1x > 0. Applying the triangle inequality and the condition on |ar] where we can take K = 2-1222 - gives
sin" cxp (- 2c sin
|u|p+1
k2
u C(3/2)++1
which only depends on t. For any constants c', C" > 0, the continuous map + C7 Clylpt2e-cd dy is bounded function of y E R by L'Hôpital's rule applied to determine limy Pe -2² 0 for any > 0. It follows that there is constant M 2 0 depending only on u such that 1g(1, cos 1)] < M whenever 0 < /4; obtain the result for
by the previous limit calculation. Applying symmetry and taking the sum of our two bounds then yields (g(t, cost)) < M' for
M' > 0 not depending on k m whenever (t, x) [0, x] [-u/2, u/2].
Now, we have after one additional change of variables x + rm
dt / (t, cost) Losasi da.
(m,
We can invoke our M' bound when rm -m < |u]/2, and the indicator enforces |a|| S 1; thus, taking m 2/ |u| (here we use uu] > 0 critically) implies
fTa v dt / (t, cos Dosess dæl s M'
p
J=- dt < +00,
201

---

Published as a conference paper at ICLR 2021
so that by dominated convergence, have
lim (v) S dt ( (t, cosst l0s=51 dr.
R-000
By the same estimate together with recond-argument continuity of f, hence of g, have by the dominated convergence theorem
lim lim (m,x(M) = J/' gtt, dt -u [."
f ft, cottt) 1
v cot
-u Combining with results Im and the first term, conclude
2n
dt.
sin?
V2Te²
q(v) (0)
1 dt (u (-u ft, (t.-ucott)
u oot
/ (
20
sin2t V2Tc2
as claimed.
E.3.6 MISCELLANEOUS ANALYTICAL RESULTS Lemma E.28. If > then & is 1-Lipschitz.
Proof. We recall
n...amii.a.... N(0. ((v)
E
ccos X,].
Considering instead the related function & defined by
(
E Mr.
[1, olv, 91. g2)l.
where
(vo, vv)) (B0.00))
Φlv, 91.92) = cos
110ll2|1u012
we notice
( = -(v)) + (T/2))((E)
It is therefore equivalent to show that e is 1-Lipschitz; but this follows from Lemma E.22
Lemma E.29 (Even Moments). If A E N and h I one has
E || 11 1 s C,n
| 26 1 S C1n
where Ck - - 1
(2k 1
1)!!.
Proof. First notice that the claim is immediate if 1, since E assuming > 1. Also notice that Lemmas G.11 and E. show that ú, and u, have matching even write
|| 1. We therefore proceed
moments, suffices to prove the claim for v.. By rotational invariance,
24
E U.
Σ
(o))
n
((.1)
2
Σ
E
II
(oi,))
nk
where the last sum is taken over all elements of [n]* We split this sum into a sum over terms whose expectations contain no repeated indices, and sum over all other terms. There are exactly k!( ways to choose k-multi-index from alphahet of size without repetitions-sele select the k distinet
202

---

Published as a conference paper at ICLR 2021
indices, then arrange them in every possible way-and multi-indices without repetitions correspond to terms in the sum where the expectation factors completely, by independence, so we can write
1)
2 n
B 1
B( kk)
k!
11
Σ
E
II ola
Si Sn repeated indlcees
We will prove the elementary estimate
n² k!
S (k
(B.47)
Assuming it for the time being, we use that E (3)) 2 to conclude
(2/n)*k1
k El Elo( -1 - (I. 1)22"
1
Next we study the expectation of of-products arising in the sum. The expectation factors over distinct indices; we can classify repeated indices multi-indes by partitions j1 ..+ Jm k, where each jt is a positive integer. Formally, for each multi-index (i1....,is) there is partition ji +... Jm such that
A
m
II o O,,))
II ((a
where p: [m] [k] injective. We can evaluate these expectations using the result EE (g)
(2k - 1)!!, because the coordinates of g are i.i.d.:
II 1=1
E F(o
220 = 2
[2 (2j. 1)!!.
11
We claim that
1 2
II (2j 1)1!
(2k - 1)!!.
(B.48)
which is the expectation obtained from a term with all indices equal, whence
2
2 nit n - (
Σ
E 1=1
n
1sh Sn only repeated indlces
-(k - 1)2222 1)!!)n
by E..47
which gives bound the number of terms in the sum. Noticing that this constant is larger than (A: - 1)222-2 we can conclude the claimed estimate on Ck provided we can justify E.48 For this, it suffices show
(2k 1)!!
< 2
I1 (2j, 1)!!
Observe that m for any partition, 2 righthand side. We write this term as
2 1 and need only study the second term on the
(2k 1)!!
II (2i 11-11 1)
ThC (2),
(21 1)
The fact that ji +... + Im implies that there are factors in the denominator, so we can put the factors in the numerator and denominator into one-to-one correspondence. Consider the ordering of the factors in the denominator (I (21 1))
( pm (21 1)). Then
I - (2i
II (2i 1).
P (2i - 1) (=1++1
203

---

Published as a conference paper at ICLR 2021
If f1 kt then this product is empty and m 1, so the claim is established. If not, then proceed to the next group of factors in the denominator: we get
1L ش (2i 1)
11
(26-
(22--1 1)
because ji > implies that every term in the numerator (ordered in ascending order) larger than the corresponding term in the denominator. This gives the claim in the case m 2; for > 2, we conclude the claim by induction.
To close the loop, we prove
(E.47) Using simple algebra, we observe
n1 = n nn(n - 1)... (n k + 1)
n2
kd
k)
= t
1
k-1
1 5) m
#))
j=1
and we note bounds
(1-A-1))
1
II
1
1 1. n
n
j=1
Working on the upper bound first, we obtain with the help of the binomial theorem
k-1
2)
2 2
11
1 h -
k-1
n
n
k 1 (-1)²t)
h - 1 1
j
2
h
k
where the last expression removes cancellation by making each term in the sum nonnegative, then applies a change of index. With the identity (++1 = (k: 1)/(i + 1) 2)= we proceed as
k
2 k- 1)
(+
k (K-1)²
(k- 1)2
k 2
k:
1
n Σ j + 1
n
n
j + 1
n
j=
(k- - 1)3
k-2
kk - 2)
j )
k- 1
n
(k - 1)2
- 1
k-2
1
n
n
given that 1/(j 1/j++1) E 1. Since 2 k. this gives
n²
kl
S (k - 1)2 k- 12k-2
The upper bound on the product gives immediately n² - (?) 2 0, which completes the proof.
Lemma E.30 (Mixed Moments). Let g². ...... gg" denote the (i.id, according to (o, (2/n) I2)) rows of the matrix G. Let ls E [m], and for each 1 S j S k let S3 B2 - R be a function such that
1. E(I < Cnn-pp. with C > 0 an absolute constant and p 2 1;
2. (g) 5n
204

---

Published as a conference paper at ICLR 2021
Consider the quantities
A = E II Σ f
B n
E
Then one has |A- B| S On 11 with the constant depending only on
Proof. Start by writing
A
Σ
E
II )موو²t)
Se Sn
k!
II
[f
B
E
E
Sn
only repeated indices
n
k!
B
Σ
E "I
k
1Sh1..... Sn
only repeated indices
as in Lemma E.29
Applying the triangle inequality and the first moment assumption on the func-
tions f3. we get
n
k!
11 B
B
S <k 1)22k
with the last inequality following from the estimate triangle inequality
E.47
For the remaining term, we have by the
&
Σ
E
1 n
k!
sup
k:
1 En only repeated indices
j=1
S 2-2
sup i Cnj
E
fy 5g
using again (E.47) control the number of terms in the sum. To control the supremum, we apply the Schwarz inequality & times to get
1/2
E
(
<
I
( 2-
pe
By the subexponential assumption on the functions fj. we have moment growth control, and we therefore have bound
&
< II 1=1
Cin-12t
-12---
chi - (k- (t-1)(++
C k2(k 104--1)+22 1)+ E/ー
205

---

Published as a conference paper at ICLR 2021
and consequently
Σ
E
II
S -k)
SS Cn only repeated indlces
which proves the claim.
Lemma E.31. For any 0 < C 1.2 there exists smooth function d. IR R satisfying
1. (c(x)) = x i 2 2c and (x)) = if x S c. and ve is between and 2c if S S 2c;
2. vo(x)) 24:
3. There are constants M1, M2 depending only such that | S M1 and k/' S M2.
Proof. The function f(x) = 12200 is smooth on R, and satisfies f f and f = 0 x $ The function
f(x)
Φc(x) f(x) +fc)+f(c-0) x)
therefore smooth, satisfies 0 S de 1, and satisfies 2c(x) 0f and dc(x) if
Simplifying using the definitions, we can write
x 0 0
dc(x)=
1+EXD
Z(C-F)
z6 c.
It follows that (x:) is zero when x S when 2 c, and in between otherwise. Thus, the function (x( c (x c) c) satisfies property 1.
For property
note that (x) c (I c) c) implies that sb since 6. c) whenever x I c and Φ. 2 0. Since (x) = x when x 2 2c, we can then conclude vi(x)) 2 D.,
since z & c when x I 2c and 1/2
when x 2 2c.
For property we note that by property 1, (x(x)) 1 2 2e and v((x) = if x < 0: consequently (x) 0 if x & j0, 2c), and it suffices to control vo and v in this region. By translation equivariance of the derivative, then suffices to control the derivatives h(x) = (Φ) for 0 < < c. We calculate
M'(x) = dc()), h²(x) (z) 20.(z),
(E.49)
and
6x(x) - (x)
(E.50)
x))²
-
(x(x) -
+ x))
(E.51)
- ()(((()) -C)00)=----))((0))]]-)-----00
f(c-)((c=x)) 3)
Completely ignoring possible cancellation, we see that it suffices to get lower bound on f(x) + f(c - æ) and upper bounds on I' and fr to bound [N'] and [7']. We calculate
(((x) + I'(c x)
22 = 1
11.00
(c x)²
c-n lace:
and since f(x) > 0 if x 0 and 0,
that any solution of f( ( x) =0 must
occur for x E (0, c), which implies as well c - x E (0, c). Writing g(x) x²c and using
c-1 x1 co for x E (0, c), note from our previous work that ('(x) f'(c x) 0
206

---

Published as a conference paper at ICLR 2021
-(=-1)=00 = =[((==()))))) =)- We calculate g' (z) xe =e2 z), so that if 2 then g'xx) < 0. which implies that is injective on (2, xo). By assumption, we have c= > 2; consequently there is at and given that &e is solution, there
most one solution to fz) - fc z) 0 in 0
is exactly one solution. check
2f(c/2) < f(0) + f(c) = log? < 1/c.
where the first RHS the value of f(r)) + f(c x) at both x and c, and since 1/c 2 2, we conclude that f(x) + f(c - x) 2 (c/22) > 0. Next, we
1
(()) x²
x>00
2
1
('(x)
1
18>0.
x
together with the bound x-e- < P for p > which is proved by differentiating x xPe-"
equating to zero, and comparing the values of the function at x. = 0, x = p. and x + co, to obtain with the triangle inequality
((z))=4//2
If"())) - 4'e + 2
Combining these bounds with our lower bound f(x) f(c x) and repeatedly applying the triangle inequality and modulus bounds in (E.50) and (E.5T), then subsequently in (E.49) (using
also |x]| S c), conclude the claimed bounds on & and 1-
Lemma E.32. Let Z, Z E L2 be square-integrable random variables. Suppose that Z S C a.s. and 2 La < M. Then
Var[ S Varl + CM M
Proof. This is simple consequence of the triangle inequality and the centering inequality for the L norm. We have
1
211 + 1 - E( E E 12 1 7 7 F[Z
and additionally
1 2 - E[Z 1 1 -11 S M,
so that, after squaring, get
Var [2] S Varl [2] + M 7 B[ L² M²
s Var [] + M | + +
S Var [2] CM + M
by centering and the a.s. boundedness assumption.
Lemma E.33. Let X, be square-integrable random variables, and let d > 0. Suppose |X| S Mi
a.s., and suppose P[[Y - 1| 2 Cvd/n/] S -oe and || 11lza S M2. Then one has with probability at least C'e-od
XX - X - + 2CM + M, -dd/2
Proof. We apply the triangle inequality:
XY E[xy]| < XXY X| E|X]| + [[ E[XY]
< M - M, 1l] + |X - B[X]],
where the second inequality also applies Jensen's inequality. We have
B||Y 1|] =
1 Y 11-122000 + + 1x Y
< <C
1 20 va/n 8-11
1/2
1/2
<Cv
1 1
E 18(( - 1)2
d/n
< <C
+ Vo'ee cd/22 -cd/
207

---

Published as a conference paper at ICLR 2021
where apply the Schwarz inequality in the third line. Consequently, with probability at least Cee mod we have
d XY [[XY] S|x - B[X] + 2CMV VO' cd/2
as claimed.
Lemma E.34. For i = 1,....,h. n, let X Y; be random variables in LA, and let d > 0 and >
Suppose X; 2 0 for each and Σi= [allla I Ms. and suppose P|vi E [n], IY; 1| 2 and for each i, IY; 1 S Mz. Moreover, suppose that C dd/n S 1. Then one has with probability at least 1
I Cyd/n)=
/ d + +2CMM
X,Y B[X,Y] 2
X, B|x,]
M,M.
Proof. The proof is minor elaboration on Lemma
E.33
We apply the triangle inequality:
n
n Σ
n
X,Y, E[X S
X,Y,
X;
x; E[X,]
+ Σ
B E[X
n
" Σ
" - 11]
c
Σ Xi]
Xx x, X
where the second line holds with probability at least d. Another application of the triangle inequality together with nonnegativity of the X; gives
X
X
Σ
X;
B[X,]
+ Σ B[X.]
i=1
M3
+
Σ X₁ E[X.]
i=1
where the second line applies the Lyapunov inequality. By the Schwarz inequality and the Lyapunov inequality, we have
d n CV
a
B|x - 5
E[Ix,|] Σ
11 11: X,I13, -1] 11]
n
CMM
+ 8 M2
Consequently, with probability at least
we have
Σ x,Y - E[X,V]
< 2 Σ X, B(X.)
+ +CCM3
82/
as claimed, where we use that C dd/n & here.
Lemma E.35. Let k E N, and let XI...., Xx be integrable random variables satisfying [|XX; E[X,]| La < M, for some constants M; > Suppose moreover that with probability least 1 8; one has X - E[X,]] & for some constants N; > O. Then one has
Var
X
N:N, Vo 6,,M
f11=1
208

---

Published as a conference paper at ICLR 2021
Proof. We start from the formula
de
n
Var Σ
X
Var [X]] 2
Σ cov X,, X,},
E E[X; one establishes this formula by distributing in the definition of the variance. By assumption, there are events & on which [X; - R [X] I N; and such that P > òz. Partitioning the expectation, we therefore have
where cov[X;; X,] -XX Bi
Var[X,] = E[ - E[X,))*
- N + - B[X.))
- N₂ + [
E[X,))
< N? VoM..
where the first line uses nonnegativity of the integrand to discard the indicator after applying the deviations bound, the second line applies the Schwarz inequality, and the third line uses fourth moment control. For the covariance terms, apply Jensen's 's inequality to obtain
|cov[X,,
[[ |[X,] I B[|X; E[x,]||-; B[X,III.
so that, again partitioning the outermost expectation and applying assumptions, we get |cov[X, X,]l s N²² se [X; E|X, B[X,
< N? + squey [2, IX - E[X,D*V < N,N, + R. 6,
where in the first line we again use nonnegativity of the integrand to discard the indicator after applying the deviations bound, in the second line apply the Schwarz inequality twice, and in the third line we use a union bound to control the indicator. Since ó; < 26,, we conclude the claimed expression.
Lemma E.36. Cf > and p the function g(t) CCr for 2 satisfies the bound g(t) S (p/(2C)))
Proof. The function is smooth has derivatives gt) = f--e- - 20F) and g = - (p(p 1) 2(4p 1)Ct + 4C2().. It therefore has most two critical points, one possibly at 0 and one t (2C) and these points distinct when p > 0 and C 0. We check the sign of g" the second critical point; since vp/(2C) > we need only check the value of (p(p - 1) 2(4p - 1)Ct² + 4C evaluated at = / (2C), which is 2p² < O. Then since lim,- (() and g(0) 0, conclude that g(l) & o (2C)), which gives the claimed bound
Lemma E.37. Following Lemma E.26 consider the random variables
a (gnx) p( -1e c)
(s.gg..92)
(1(voll2) sin
(wo, v.) (1|,,|12)]] (((eolla)))((1)))
(wo, us)
Ez(v.g1.22)
(1
(vo, v.) (v,, 8.) ",),,l
Ea(u.gi,92)
(([vooll2 mb
(wo, 6 (o.,
(10x)12)
Ea(v.g1.92)
(1 ( v. 1 1,,12
u.) 0 Ba (1[x,112)
(Vo, v.) (v., 0.) 3 (1)
(vo, v.) (v:, 2 (11222112)
Es(v.g1.92) 92) ( (2(1)
+5 ((leolla) v1 (2) 1
209

---

Published as a conference paper at ICLR 2021
where Ei(-,g) 91. 91.92) is defined {0, x by continuity (following the proof of Lemma E.27 it is here). Then for each i = 1,....6, 6, one has:
For each i, there is a jnjuuintegrabll function Si R2 x x -> R such that [E( ..,, 92) [ 91, (2)]1 S f 92);;
2.
There is absolute constant C; 0 such that for every 0 S n, has Il I C..
so that in particular ||E:(u; : ( )lll S Cj.
Proof. First we reduce to noncentered fourth moment calculations. If X is random variable with finite fourth moment, we have by Minkowski's inequality
| [X]]| L+ & || [E[X]],
so that the triangle inequality for the expectation and the Lyapunov inequality imply
| E ||
We can therefore control the noncentered fourth moments of the random variables E, and pay only an extra factor of 2 in controlling the centered moments. For the proofs of property 1, we have similarly J.X ]| - [X] + [|| from the triangle inequality, so that it again suffices to prove property 1 for the noncentered random variables E
E₁ control. If
or x, the integrand is identically zero; we proceed assuming 0 < v n.
Using &b 2 4. we have
cot
0 (v, S 16
sin v
For property 1, by elementary properties of cos we have for S v S 7/4 and 3 / S v S that
cos 2 2 so
no 8sin
n
pl p (l-g <
P.
4m
This gives
σ (g11)² (g1))"((991) cotv) 91- p(-91:110000
1( 914 v)
2 K
1/2
911
Q11 sin V
sin v
sin v
sin
where we define = n/8. By Lemma E.36 C > 0 is an absolute constant. We conclude
we have that g S g( 0(33/2K)
CK-3/2
where
(g1.) p( ILs cot v)
S CC/n,
(E.52)
sin
provided is not in [T/4,33] 44]. On the other hand, if T 4/45vv <V 23/4, we have sinv 2 1/v2, so that
( p(-)))p(9911 cotv) < (
(E.53)
sin'
where > is an absolute constant. Since these v constraints cover [0, T). we have for v and all gi (by the triangle inequality)
(v.g..92)) C C'
where C, C are absolute constants, and by Lemma G.11 we have
E(C 3/2 3/ ) C C
where C" > is an absolute constant. This proves property 1 with fi (00gg)) C+2'n3/2 (91n)) with different absolute constants, and property 2 follows from Lemma G.II after applying the Minkowski inequality and calculating the integral, which has the necessary cancellation the 3/22 factor.
210

---

Published as a conference paper at ICLR 2021
3g control. By Lemma E.31 Cauchy-Schward then implies
have | C' for absolute constant C > 0 and 2/v(x)) 2.
(o, v2) v/ v. |22 |2)
8C.
* ) R/ 12,l2)2
In exactly analogous manner, have
Uo v.)
4.
Both bounds satisfy the requirements of property 1, with f2 16C + 8. The triangle inequality and Minkowski's inequality then implies (v, : )l S.
E3 control. By Lemma E.31 we have |al"| < C for an absolute constant C > 0, vb 2 1 and
1
x///(x) S 2. Cauchy- Schwarz then implies
(vo, (u. 2 2 ( (1 v2
I
16C'
and the triangle inequality gives || / S 11g1 12 bounded by 4, by the Schwarz inequality and Lemma C++ ('llill + (1gel12) and we have
2 2 2l | g2ll2, whose expectation is We therefore take f3(g1.92)
G.11
[|(191 112 + 11g2122) I Illl llz + 112212l12 & (Ill99lllllz= + lll 1221 sc
where C >> 0 is a (new) absolute constant, by the Minkowski inequality and lemmas Lemmas G.10 and G.11 This establishes property 2.
E, control. By Lemma E.31
we have |al'| -
C for an absolute constant C 0, +b 2 i and
</v(() 2: Cauchy-Schwarz then implies
(vo, ú.) (Ua, 82) (1 112
S 64
(5( (oll2))(((())) n, 21) 12
Following the argument for 3g exactly, we conclude property 1 and from this bound with a suitable modification of the constant.
E, control. We have
(wo. u.) 12
I |
(1[[ (> u
following exactly the setup and instantiations in the argument for E,. Following the argument for Eg exactly, we conclude property and from this bound with a suitable modification the constant.
Ee control. The triangle incquality gives
(wo, U.) (u,,, iz) ) (1
(wo us (u,, 3. 2655 2.
Es(s.gi g1. gx)l S
(1 ( 11 n.
(1 (12) 2, 2
and following the setup of 3. and E, control gives [e, 91 92)l < 128C ë 11 32C 12 Following the argument for E3 exactly, we conclude property and from this bound with a suitable modification of the constant.
Lemma E.38. In the notation of Lemma E.13 there are absolute constants c, c, > 0 and an absolute constant K > such that if n K. there an event with probability least 2e on which one has
1o < 2012))
E
S -ennnn
15 10|l2)²
Proof. There is no v dependence this term, so we need only prove a single bound. Following the proof of the measure bound Lemma [6 but using only the pointwise concentration result, we assert that if n 2 C' an absolute constant there is an event & on which 0.5 < ||oo|22 < 2 with probability least 1 2e -rn with e absolute constant. This implies that g1 we have
1,
Po 1
211

---

Published as a conference paper at ICLR 2021
which we can use together with nonnegativity of the integrand to calculate
|100l
27
E
E[1g]
+
E
Lge
2
12)²
2b/1/voll2)
2c on
whence
2
E8
I 2e
No 21
2/
00 22
whenever @1 & Similarly, calculate
vo
100l22
E
E[1e] E 1
)2
(20ll2)
1/2
2o Dull2
< 1 + 11++ 1/2E
2
(10012)
< 1 + 16Ce
applying the Schwarz inequality, property in Lemma E.31 and the measure bound on E, with C² >> 0 absolute constants, whence
1
|00
E
16C'e-c'n
+b Pu
12)
+b
12)
whenever 91 E E. Worst-casing constants, we conclude
1
E
I Ce 2
-on
(ool))
+ vo
when 91 E, which is sufficient for our purposes.
Lemma E,39. In the notation of Lemma E.13 if d 2 1, there are absolute constants e, de,e, ddd,e C,, C,C", C" C"", C"" Kdd log" n and 2 K', there is an event with probability at least O. -e"dd"d C!" nee" on which one has
and absolute constants K, K" > such that if
dlog n
E1 BB, 91. 92)ll CV
n + C''n Ce
Proof. If v/ {0,T), then E, (v.91.92) 0 for every (g1.g2), therefore assume 0 below.
We will apply Lemma E.34 to begin, with the instantiations
X; ((91))3 ppppppp(ppp)) g1s c)t
Y
sin v
4[|v0 12 (2)
since then (v. E, X,Y, We have X; 0; writing 122 2/nn, caleulate
g
1
g²
E[X,]
v S V2nk² JR sin v
exp
21 sin
2 y
dg
2
sin
(B.54)
212

---

Published as a conference paper at ICLR 2021
where the second line uses the change of variables g 9 sin and Lemma G.11 have
Additionally,
k+
1
go
B[X?]
29 9 cot? w) dg
4m
v 2m
JR sin
6
exp
kt sin
[ 9 exp
- (1+ + +cos² 2 v) dg
4
2
k^t ssn
1
g 80--8222 dg
4T + cos2 2/77/2
2T
J
15 V
(E.55)
T cos 1/7/22
where in the second line we change variables +> giin v. in the third line we change variables g + 9/ /I cos" v. and in the fourth line we use Lemma G.11 We can calculate the derivative of the map g(v) = (1 cos? 2))=/2 sin v as g'v = cos(v)( + cos² v) v)- + cos² w) + sin (1) + cos? v)5/2), which evidently has the same sign as cos(v); so g is strictly increasing below x/ x/2 and strictly decreasing above it, and is therefore maximized at g (2) We conclude the bound
15 T
E[X}] S
(E.56)
which shows that Σ, 1X = 0(). Next, we have Y; S 16 by Lemma E.31 so by the Minkowski inequality || 1 L S for each i, and remains control deviations. We consider the event E = E0,5,1 in the notation of Lemma E6 which has probability at least 1 -ne-on and on which
we have 1 < ||ot12 s 2 for all i [m] and in particular 3 <
||oll2. and thus by Lemma E.31
Vo 2
e 112
for all i E [n]. By Taylor expansion with Lagrange remainder of the smooth function æ + x= on the domain about the point have
= =1 1x
g3 (x 1)²
where & lies between 1 and x. If g2) E E. then for all i | 2 (1/64), and we can
therefore assert
(Ilo||a||e[[1
64 (1 o/ 1)² - Y, < (11 1f 1)
(E.57)
By Gauss-Lipschitz concentration, we have P[ll Moll2 - B||l vo voll 2 0 S -ent and P /o 2 t] S 2e ent?? Lemma E. implies that 1 - 2/ (//nnn1)) -1) < Ellloi - 1 and 1 - 2/n S
E B
|a]]
vo 2] 1, so can conclude when and when is larger than constant that
d n
||10ol12 - 1
Vi [n]. || - s sVv
with probability at least 1- -C'ne--4 by union bound. Using then the fact that ||0 < 2 for all i on the event & together with the previous estimates and (E.57), we obtain with probability at least n" an C"" ne (via a union bound with the measure of E) that for all i,
v - SI-KE
As long as 2 d, we conclude that with the same probability, for all i we have -111| CCC//// We can therefore apply Lemma E.34 to get that with probability at least 1 - On n" we have
o (gn)) (( ((-91ncoLv) (( sin? v
E (v. 92) (l, g2)] 2
sin
1=1
C + (C" -nn/4 + (C"')//4 ne d/4 (E.58)
213

---

Published as a conference paper at ICLR 2021
where we also used the triangle inequality for the & norm to simplify the fourth root term, together with n 2 For v E [0, n|, we define fv IR + by
fu()) 2m sin
ag))
exp (-
21 (2ogooot))
1
cot? cottv) v)
so that the task that remains is to control E; fulg))) E[fu(gn))l- Lemma B.36 obtain estimate
We start by applying
C
fu(9) - c/cosv/ 2/31
where > is an absolute constant. When S V S 7/4 or 3 SUST, n, we have therefore conclude folg) I C/n + C" (g) for all v, which shows that fu(g) is not much larger than C" (n)(
fulg) s C/n. Meanwhile, if / S 34/4, we have fu(g) I C" "na((g)
Next, let g ~ NO 1), so that g d kg: we have for any 2 0
P(C' 2 1 P ( C" (n() S exp (- ((n))1)))))))))))
where the classical estimate Plg 2 e valid for 1 2 1, and accordingly require t 2 ((")~=n" In particular, there is an absolute constant C" > 0 such that we have
PP P
C"
PP
(
1/6
exp (-
1/3
se
Vnd_
where the last inequality holds in particular when n 2 8d (and this condition implies what is necessary for the second last hold when d 1). Returning to our bound on fus we note that when n 2 (C/C")P we have that
2C" C
2C"
C"
fu(g)
Vnd
Vnd=
Vnd
from which we conclude that when our previous hypotheses on are in force
20"
IP f((g) 2
Vnd Se-
(E.59)
We are going use this result control fr ([fv(gi))) using a truncation approach.
Define M 2C" //nd, where C" > the absolute constant in (.59).. We write using the triangle inequality
m
Σ fo(gu)) E[fv(g1)))
Σ fo(i)) f
folgne (oonsM-- E[fo(gr)) ( [ew
E [[fe())))) ((fe(
By E.59 and a union bound, we have with probability least 1 ne-d
folgn)) - (9rs) so(ow) = 0.
Moreover, we calculate
E [[s((on)) (oww)))) ((Jo( 5
B [(oona)))
P(fa(ge)) > (/
Cc-4/2
214

---

Published as a conference paper at ICLR 2021
for absolute constant
0, using in the second line the Schwarz inequality, and in the third line (E.56) and (E.59). The second term can be controlled with Lemma G.3 together with the observation that
C >
(L E ((der
E [Lle))ssm((m))) - (ooos
==1
I C/n.
where the last inequality is due (L56).. Lemma G.3 thus gives for any 1
a
f2/2
Σ fulg)) (OWNNN E (o (( 2 t
<
2exp
-n Mt/3
It follows that there is an absolute constant C' > 0 such that
(llnn) [ (.
2 CV
S 2e
and therefore with probability at least 1 - -nne (by union bound) we have
fL(n)) B[fu(on) s cCy. d
v cCee
Combining with E.58
using union bound and worst-casing constants in the exponent, we con- c""n, we have
clude that with probability at least 1 C n" peeeeed
EI( 91. 92) - (M 9r, g2,g2)11 S C C' +C"ne
Aggregating our hypotheses on n, there are absolute constants C₁ C2. C2.C3. > such that we have to satisfy n 2 max(Cd, C'd', C"). Moreover, to able to assert ne -c"d < e ec" we have to satisfy d 2 2/c" log Introducing an auxiliary d >> 0 and setting = dlogn, C'd log* n, C"J and d > 2/c". Choosing n in this way, we can finally have
have to
satisfy > max{C'd
conclude that with probability at least C""nn" d/2 C ("""ne-'""
dlog n
(v, g1. 92) (V, g1. 91. ge)ll a SC
+ +C'n -C'n-ed + Cen
n
which is the desired type of bound.
Lemma E.40. In the notation of Lemma E.13 there absolute constants c, C, C² that for any 5 2 3/2, we have
C" C" 0 such
P
E. E (1....11.921)) 92)]
E (v. 91.82
is C + 1'n 1' -Lipschitz 2 C"
Proof. Write f(v, Egg.. (v, 91. 91. to v. bound the derivatives on an event of high probability, and apply the triangle inequality to obtain highhiigbbar sigg-probability Lipschitz estimate for [gg. (v. g1)2 Egg. [ (v. g1. g1)ll)
9: it will suffice to differentiate and E[f with respect
Define is 22/n. For fixed (g), g2), the function
qVv.g1 9z) = Σ
215

---

Published as a conference paper at ICLR 2021
is differentiable all but most points of (0,v). v),, using Lemma E.31 to see that the only obstruc- tion to differentiability the function o in the term |u 12; and there has derivative
of cos
F a sin o C1l (al Kol.et e lla sin cos
(- - 2k Ri cot?
(v, - 2 (1ool)))))
exp
The triangle inequality and Lemma E.3 yield
4
4
160'|bš11
12 sin a
(v.
exp
21 Ri cot
2
122 sin
sin v
(E.60)
for > 0 an absolute constant. We have || l22 S 11 + 1g2ll2 by the triangle inequality, so to obtain (v, 92) g2)-integrable upper bound it suffices to remove the dependence from the previous estimate. We argue as follows: 0 S S T/4 or 3T/4 < S n, we have cos² v 2 2. and so for any p
exp (- 25 cot? v)
gh. 4k
$ exp
sin Pv.
(E.61)
sin' v
sin
By Lemma E.36 where put C
gis /4k2 and therefore have require that gi + for all i E [n] (a set of measure zero in RR), this yields
(1, ((v,9..92) ) S C2192112,
(F.62)
where C > 0 is a constant depending only on n and gi- In cases where 9ıs = 0 for some i, we note that the bound (E.60) then equal zero, which also satisfies the estimate (E.62). On the other hand, when æ //
3 /4, then sin f 2 2-1/2 and we can assert for any p 3
exp (- cot2 v)
< 2P!2
sin V
By the triangle inequality, this too implies
(1' 92)1 S C21192212
we conclude that o' is absolutely integrable over [0, T] x R", so that an application of Fubini's theorem and (Cohn 2013 Theorem 6.3.11) gives the Taylor expansion f(v,gi) Next, we show also that q' is absolutely integrable over [0. T R x R which implies that El ([ g1)] + fo Blg'(1, (1,91.92)) di as well. Starting from E.60
where C" > O is constant depending only n and 91. Invoking then Lemma G.9
((0gg)) + fo Eg (q' (1,, 91 92)] di.
have
Ellg' (1.91112))ll
4
4gf k22 sin
16C 12 + (9/1l2)
12 sin't
1 22 ois cot col2v)
< E
3
exp
2
sin v
and the expectation factors over gis, gi gh. so we can separately compute the 9iv integrals first. For the first of the three terms on the RHS of the previous expression, we have
19
(-
1 2229 cot² w)
km / lgl
(-2229 ssinn dg
E
exp
sin exp
sin 9 v
[ 1oleexx
2 dg.
(E.63)
22
after the change of variables g + sin in the integral, which is valid whenever O
To take care of the case where v = 0 or и = I, we can use the estimate (E.61). valid for v/ sufficiently close conclude that li (1..1.922 g2)
0 or æ, and the assumption Ø1: * 0 for all such fixed (g1 92), and by symmetry the analogous result lim, 7 o' (1, (v,91 92) = 0; and whenever for some we have 91: O, we use (E.60) to see that the term the sum involving 91i poses no
0 for any
problems as D Y 0 or v / æ because it is identically O. Returning to the integral (E.63). we have after a change of variables
1
exp exp
2 dg
k5
101 exp (-20²) dg Ck.
22
V2T
216

---

Published as a conference paper at ICLR 2021
where C 0 is absolute constant, and where Lemma G.11 for the last equality. The remaining two terms can be treated using the same argument: we get
91. sin
(- 1-22 cot' cot'v) v) C²
1
exp
Si
v
(after using [sin v S 1) and
exp (-22296 oi cot² cot?v) ) C²
E
sin
R1s
for absolute constants C" C" O. Combining these estimates gives
C Σ E
Elld' (v, 91, (1)
1 + [
and using Lemma G.9 (or equivalently Jensen's 's inequality) gives finally i= 9:
Elld' (v. 91. 92)l S CV n SC
To conclude, we need to show that Eg, Ig' (v: g1, (2)92) is uniformly bounded by polynomial in n with high probability. For this start from the estimate (E.60) and apply the argument following that, but with more care in tracking the constants: if v is within T/ of either 0 or æ,, we can assert
3 C.ka k
C
Czkı
Ig' (v. 9v.91.9221
S
+ C2
+ (1g5112) +
191d
191sl
whenever I1s # for every i set of full measure); and when within x/4 of πT 22, we can assert
Ci + C¹ ou (llgilla + (1 + gl
(v.
where C₁ C/ are absolute constants. By the triangle le inequality, independence, and Lemma G.9. when we consider [Eg, (4' ditionally, by Gauss-Lipschitz concentration and Lemma G.9 we have that simultaneously for all i lggst l2 E l ll2 < 2 with probability at least 1 2e on Moreover, since ||g1ll lo S ||gil22 we also
the term ||lاوا|||] bounded by an absolute constant. Ad-
have control of the magnitude of each 9ial this event, with probability least 1 2e have for every v
C3
Elg (u, 92
C2k3
+ Ca E22
191a
i=1
for absolute constants C, C, 0. If ~ ~O, 1), we have for any t 2 0 that PI 2 1 1- Ct, where > 0 is an absolute constant; so X; i..d (0, 1), we have by independence and is less than an absolute constant P(vi, | > uses the numerical inequality e 51-ts
" 2 1 - C'ntt where the last inequality valid for S S 1. From this expression,
conclude that when 0 < t S en 1/2 for an absolute constant c > O, we have
Plvi E [m]. 1ora 2 f] 2 - 3/3/2
so choosing in particular& t en (6+ for any 5 > 0, we conclude that P(vi E [n], Igvd 2
en 3/2 > C'n-5. Consequently, for any 5 0 we have with probability at least 1
C'n 2e
Elg'(u (v. 91. ge'l| g2
C E <
C + C2k² Ca 12
C
k 5=1
and since k- 22/n, this yields [Egz (g, (v. 91)g2)) < Cin + C2 + C3n C/n3/2 with the same probability. Consequently we can conclude that for any 8 2 3/2, we have
E E (v.9..99))))) (i(v. is C + 1 -Lischitzz 91.92
2 1 - -1-2 26-
C n
92
217

---

Published as a conference paper at ICLR 2021
Lemma E.41. In the notation of Lemma E.13 d 1, there absolute constants c'C, C' and an absolute constant K 0 such that if n Z K, there an event with probability at least 1 - Ce- which
VV E 0, m]. [2 91, 92) ( 91 91.92)] C'e
Proof. Let & denote the event E0,5,0 in Lemma E.16 then by that lemma, & has probability at least Cn-on as long as n 2 C", where c, C, C' are absolute constants, and for (g1,92) E &, one has for all y E [0,
(v, 0.
This allows to calculate, for each v,
El 91, 92)] = E[Ig 91 92)) - B[1 >> S C'e
after applying Lemma E.37 and Lyapunov's inequality and worst-casing constants. We conclude that with probability at least Ce
Av E [0, x]. [2(v. g1.92) E| gr. 92)]l C'en
Lemma E.42. In the notation of Lemma E.13 if 2 1, there absolute constants c, c.C, C' and an absolute constant K > 0 such that if n 2 K, there is an event with probability at least 1 Ce cn on which
VU 00 m.. Ea(u, 9a 92)
(V, 91 92)]] C'e-d'
Proof. The argument is identical to Lemma E.41 Let & denote the event E0.5,0 in Lemma E.16
then by that lemma, has probability at least Ce mn as long as n 2 C², where c, C, C' > are absolute constants, and for (g1.g) E, one has for all [0, T
Es(v.91.92) 0.
This allows us to calculate, for each
(5 91, 92)] = E[ 91. 92)1 s E[1e=] |3s(,, 11 S C'e
after applying Lemma E.37 and Lyapunov's inequality and worst-casing constants. We conclude that with probability at least
Ce Ce-on
Vv E [0, n). Ea(u, E(Es(v 92)))) $ d'n
Lemma E.43. the notation of Lemmal E.13 if there are absolute constants e, C, C' C" 0 and absolute constants K, K' 0 such that if 2 Kdlog and d > K', there is event with probability at least Ce-en n on which one has
dlogn Vv E [0, [1 g2. (v. C"
n
Proof. We are going to control the expectation first, showing that it is small; then prove that [E]] small uniformly v. Let & denote the event E0.5.0 in Lemma 1. then by that lemma, & has probability at least - Ce en as long as n 2 C", where c, C, C are absolute constants, and for (gi.ge 92) E, one has for all E [0.
Ev g1. -2 (vo, G.) (vv 2c)
10 12 12 1
Thus, we write
(vo. ú, (v 2,)
(v.911992) = -211 (g1 92)
1 11,00
218

---

Published as a conference paper at ICLR 2021
have E. E, for all whenever (91. 92) &, so that for any
[ | 91 92) ( 91
(v, 91. g2)]l + Ce
where the second line uses the triangle inequality and the Schwarz inequality and Lemma F.37 together with the Lyapunov inequality. We proceed with analyzing the expectation of E.. Using the Schwarz inequality gives
1/2
E (v.
< 2E (vo. , " ú, )
|
U.
$ 32E (vo. )²)) (w,, 85 1/1
and the checks and around (E.34) and (.35 in the proof of Lemma E.15 show that we can apply
Lemma E.30 to obtain
E (o,, i.)² (u,, )
( (l cos 921 sin 1/) (921 cosy 911 sin v)] sim nn
)
$ C/n.
( (glo cos + 921 sin 1) (921 cos 9n1 sin
But we have using rotational invariance that (g cos 921 sin s cos In sin 0, which implies
[R (No, 2 { $ C/n,
from which we conclude
(V. S C/VV.
Next, we control the deviations 3 with high probability. By Lemma E.17 there is an event & with probability at least 1 - cn which //rll2 I for every E v]. Therefore the event & = & n Eg, which has probability at least 1 - Ce Ce-en by a union bound, we have using Cauchy-Schwarz that for every
(v(v.11.9)) S (u, ا.ازن
The coordinates of the random vector u O i are (911 cos + I2s sin (g2x cos v 91 sin v), and note
(g cos v + 92i sin (g2 cosv - 9ns sin v)] = (lo(g)))2 = 0,
by rotational invariance. Moreover, the calculation (E.35) together with Lemmas G.Il and E.17
demonstrates subexponential moment growth with rate C/n, so Lemma G.2 implies for f 2 0
PlKvr d,} 2 4 S 2e-ende de- minfeet..))
For large enough this gives
llg n
I (u dr) 2 CV
< 2n 2d
We turn to the uniformization of this pointwise bound. The map V + E, cos v + 92i sin (g2 cos v 91, sin v) is continuous, and differentiable
but finitely many points of [0, r] (following the zero crossings argument in the proof of Lemma E.22 with derivative
μ Σ (gr cos 92: sin (g2: cos v 91 sin sin (81: cos + 92s sin v)
which is evidently integrable using the triangle inequality and Lemma G.11 In particular, write the derivative as || - Voll Thus, by (Cohn 2013 Theorem 6.3. 6.3.11). to get a Lipschitz estimate v + (Vn, 02) suffices to bound the magnitude of the derivative + | /o
But this is immediate, since on the event E, we have ||| 11 ||vo| < 20. It thus follows from Lemma E.48 that with probability at least Ce-cn- C'n 2d+ we have
dlog
Yv 10, x). (Ve, bn) S c"y
(E.64)
n
219

---

Published as a conference paper at ICLR 2021
As long d 2 2. have that this probability is at least 1 - Ce - C'n and the triangle inequality and union bound yield finally that with probability least Ce-on C'n-d
AV E [0, m] Ea(v.gn.92) 92) - (1)) sC") dllog
Lemma E.44. In the notation of Lemma E.13 d 2 there are absolute constants e, d,,,d, C, C, C C"", C > O and an absolute constant K > O such that if n 2 Kd log n, there is an event with probability at least Ce en + C'e on which one has
d Es(u, 92) - ElEs(Eu, gi, 91)))1l S VV + C eed + c e"'n
Proof. Fix [0, r|. Let & denote the event E0.5.0 in Lemma E.16 then by that lemma, & has probability at least Ce-on on as long as n 2 C", where c, C,C are absolute constants, and for (g1.2 92) E, one has for all v E [0.
(vo, vy 1
Es Ev(
1100|12 |
Thus, we write
(0o: v. 1
Es (v,91 92) (g1.99)2))
|| 2
we have Es Es for any v whenever (g1.92) 92) E E, so that by the triangle inequality, for any
(s g1, 92) - (1, 92)]1 < (1 14, 9 (2) - E Es..
E E (1, 91 92)] E (v......9))
( 91 92) - E Es (v.
+ E 1 Es(v.g. 92)
E v4 91 (1,92)
- E - (v, g 92)
92)
+ Ce
where the second line uses the triangle inequality, and the third line uses the Schwarz inequality and Lemma E.37 together with the Lyapunov inequality.
So, can proceed analyzing Eg. First, aim to apply Lemma E.33
with the choices
(vo. vy))
X - 1
Y
1
||
since XY Eg(r, )): square-interrabiiity of X and Y evident from the definition of Igg, and have X by Cauchy-Schwand To control Y, we start by noting
Y 111z> 1 1 + || 1 + 4E j [8] 1/2 S 51+4/1+C,
where an absolute constant; the first inequality is the Minkowski inequality, the second uses the property of & and drops the indicator by nonnegativity, and the third applies Lemmal E.29 and discards the n factor. For deviations, we start by noting that E,],,, = 1, and that by
Lemmas G.2 and G.1l we have
1|] 1 2 t s 2c eut (Cn,))
It follows that there exists absolute constant C' 0 such that, putting t C' C'vd//n d/n and choosing 2 (C'/C) we have
P 1100111 - C²
< 2 -d
(E.65)
220

---

Published as a conference paper at ICLR 2021
Moreover, by Lemma E.17 constant multiple of d
run similar argument | to get that if is larger than
11 1| C.
S
24
(B.66)
Next, Taylor expansion with Lagrange remainder of the smooth function a + x-1 on the domain J: about the point gives
1- ( {3 (x 1)²
(E.67)
where & lies between 1 and æ. If gg)))) E &, then ||0y 1 2 (1/64), and we can therefore assert
- (111r112 - 1) x
S 1 (112,112 1) + 64 (1 1))
with probability least Ce en Using a union bound together with (E.65) (and changing the constant C), we have with probability least 1 2e-od ed C'e o'n that
6402 d S - n
1
C
1
Given that 2 d, it follows that with the same probability we have
d -C(1 + 64C)64C)
1
<
1
which implies that with probability least
2e-d Co
on we have
d
S
1|
n
Now, the triangle inequality gives
b 12
11
| 11
| 11
1
12
When (91.92) 92) E E, we have C'e-ot we have
1|
2 so, by union bound, with probability at least 1 - 4e-
1
4CV n
and since
(g1.g2) E
another union bound and the measure bound on & let us conclude that with probability at least 4e -cn we have
Y 1 < 4CV
If we choose n 2 (1/c)(d log C² /4), occurs with probability at least Sc least 1 - Se
we have 4e 4e-d + C'e-cn < 8e so the previous bound
We can now apply Lemma E.33 to get with probability at
)||+ + c// d
1
Vo
V,
E
Un
v.
+C'e 4/22
vo
12
|
112
1oll2
1 11,
Next, we attempt to apply Lemma
E.33 again, this time to = Le(Up, vn)) and y
ee de([oollal pall) -1. Using the definition of E, we have |X| < 4 and || -1/l2a | + & 5.
221

---

Published as a conference paper at ICLR 2021
where the second bound also leverages the Minkowski inequality; need only establish devia- tions Y. Applying again (E.67). and using (91.92) E implies |/ 1
2 1. i' we get
(1wool2l|oww/2 1) 64 (1vool2lllw 1)² < 00 1olle21 1
S (1) 1 1) (E.68)
with probability at least - Ce Using Lemma G.1l and /Vesshynin 2018 can assert for any / E [0, x and any 2 0
Theorem 3.1.1), we
Fill muu12 2 1J < 2e ont
which implies that there exists an absolute constant
> such that for any
P 11w 11
C
2e
In particular, when d, assert that ||,, la s C with probability at least 2e- By
the triangle inequality and a union bound, it follows
||| 2oll - 1 - |1 mollllllellle ll2 1 + 1oll22 - 1
d <CV
n
with probability at least 1 1-6e- -e Then union bound gives that with probability at least 1 1-66 -e Cee e (E.68) leads to
CV Va
'd
| S C VV
1 64C
S
voola
and using n 2 d and worst-casing constants implies that with the same probability
< C
12
Then since (g1.92) E = Y (1 (12) another union bound gives that with prob- ability at least 1 6e -n we have YY 1| s Cv/dnn As in the previous step of the reduction, we can choose previous bound occurs with probability at least 1 12e -d We can thus apply Lemma E.33 union bound, and our previous work to get that with probability at least 20e
(1/c) log C" /6) get that 6e-d C"e-en 12e-d. that the
l (wo, v.) E(1 (vo, w. + /V dCe--d/2
E
Whenever (gr.g) 92
E E, we have by the triangle inequality, the Schwarz inequality, and Lem-
mas E.16 and E.29 that
(Vo, vy)) E[ (vo, w.)1l S (1o. un)) El(vv, w.)) + [ w.)] B[Le v,)il < (vo. ux)) eo, w.)1 + cnn
allowing us drop the indicator. We have (No, v.)) Σ, (g1s (g cos + O2: sin v), which is sum of independent random variables; following the argument at and around (E.36), we conclude moreover that these random variables are subexponential with rate C/n, where C 0 an absolute constant. We therefore obtain from Lemma G.2 the tail bound
(P u²)
E( e, 2 f < 2e 2e-cne -oe (Ct,1).
which, for suitable choice of absolute constant C" 0 and choosing C'd, yields the deviations bounds
d
PP (0,, - B[( w.)il 2 CV
< 2e -d
222

---

Published as a conference paper at ICLR 2021
Taking final union bound (since assumed throughout that (91. E) gives that with proba- bility at least Ce on + C²e edd one has
d (v, 91.92) - ( gr 92)ll S C" V tc + -C"l
+d
which is sufficient to conclude pointwise concentration claimed for sufficiently large after put d = log n and include extra log n factors in any points where we need to choose n larger than d.
Lemma E.45. In the notation of Lemma E.13 there are absolute constants c, C,C', C" CM > such that for any 8 2. one has
E Es(
E [s(u.gr.92))
C + 2 1 c n
g1.82
as long as 8 2
Proof. We will differentiate with respect to v the function
(wo. uz) d2 1 21
f(v =
g2 45( 1o 2) )oe (1 1
and construct an event on which I' has size poly(n). We need to also differentiate the function E[f( ( ]:: for this we will additionally show that f'(v, -) absolutely integrable over the product [0, T) x RD x IR", which allows us to apply Fubini's ' theorem to move both the 91 and g2 expectations under the v integral in the first-order Taylor expansion we obtain. particular, the derivative of [[f( 9.)]] will in this way be shown to be (- (.)]]. that linearity and the triangle inequality imply poly(n) magnitude bound for the derivative of Eg, E(Es]
Define
(((v91,92)) (wo, ve) (1/2,/111)0) 11) cos gle sin s)n
10 vol 2)
so that, for almost all g1.
(( Σ (v. cos + g2s sin v)].
For each fixed (g1, (2) and each the only obstructions differentiability of qi in v arise from the function a (using smoothness of & from Lemma E.31 and the fact that is constant whenever ||w!||| is small enough that nondifferentiability of || 22 could pose problem); following the zero- crossings argument of Lemma E.22 gi fails to be differentiable at no more than n points (0, s], and otherwise has derivative
(v, (V,911110 92) =
(vo. G. G.) (11u,) (g): cosv - 9 sin 1²
* 1 2
16( [ 12) 2 wul12
+ (oo, vn)
(Mer Cn)) 26 (1 (1)) (925
(92s cos Ins sinz
un 12) 1
2 (oo. v.) ,, 22) 122 cos D In (91 cos + g2s sin
(110=112)² 1
(1(11//112) (v (x)) (wp, ).)) (J2x cos IN sinz
(1 1a 1
('((1un112) (wv, (y)) (wo, u (92s cos - Ins sin
(E.69)
+1 0(10 1
by the chain rule and the product rule. To conclude absolute continuity of (.(...1..0 we need to show that of is integrable; this follows from
the integrability of | 1
223

---

Published as a conference paper at ICLR 2021
| l2 (Lemma E.17 the triangle incquality, and the Lemma E.31 estimates 2 2 | C, | S C", and [ (=))/ /x| & C" for any z E R (to see this last estimate, note that 1 is bounded on R. and use that is constant whenever S }). Then (Cohn 2013 Theorem 6.3.11) implies that 2:( 9 92) is absolutely continuous with a.e. derivative T,- Next, we can write
f(v.g))= - Σ
( 91. 91092)9100000 (21 cos v + 92s sin sin 925 12
E
that Fubini's theorem can be applied. Our aim is now to apply Lemma E.27 so we need to check its remaining hypotheses. First, continuity of qi follows from continuity of smoothness of 1b, and the fact that the denominator never vanishes. Joint ab- solute integrability of and of follows from our verification of absolute integrability of of above, which produces final upper bound that does not depend on (which is therefore integrable over [0, π as well); the corresponding result for follows from Lemma E.37 Last, we need the growth estimate, We have from Lemma|E.31
using Lemma E.37
(v: 91 (2)1 < (22 cos I1 sin v) < 32C + 19vx)) < 32C + l92sl)²
which is evidently quadratic in 192e once 192a 2 1. Consequently we can apply Lemma differentiate f() we get at almost all gi
E.27 to
[(u,91)= = Σ
E 9050ti
(0, 9n .1..2)2200000 (2)
92
I' de (
Egg (t, 91 91 (gu cost t 924 sin t) (t.gg (1-g 91 P2)p( I1, cot t) sin )
92
where 9 is the vector 92 but with its i-th coordinate replaced by -1: cot t, and where p the pdf of (0, 2/n) random variable. The changes in 32 drive updates to the terms in qi as follows: have a (gin cos v + g2s sin v) becoming O, and f2: cos v - g sin v becoming -9 sin v. Thus, we have
o (b). of (2)p(( (p)pp(((9)))))) Ih cot
gragitt 91. Pp( Ine cot sin = t
(b vo ( b ( ll sin*t"
where the notation of is in use in the E₁ control section and is defined in Lemma E.26 and v is de- fined here similarly (the R=-1 vector which the projection of Un onto all but the i-th coordinates).
Using Lemma E.3I we can further assert
| p( gis cot 1)
19. (n. 9Is cot sin 26 s 16C
(E.70)
sin
where we use that || S |100|12- For each fixed 91 having no coordinates equal to zero, we write if t x r/4 or 3x t I m, have cos? t 1.2 and
K, gial
pl 01: t)
K?n 1 sin
sin I eexp
sin't t
4x
Using Lemma B.36
we have
pl 914 t) sin t
16 2
4T
Kin
On the other hand, when 7/4 S < 37/4 /4, then sin > 2112, and we can assert
1/2
A p cot 1) sin't
< <8/n/T.
We conclude for any
(1..t1 cott sin =2
C/(Kk3313) C' C'VK33
(E.71)
224

---

Published as a conference paper at ICLR 2021
for absolute constants C. C" O, and this upper bound is integrable jointly over and O2. We have checked previously the joint integrabilityy of the of terms when applying Lemma E.27 so we can therefore apply Fubini's theorem to get 91 -a.s.
f(v.g)) =
E
(0 gr. (1.
92l-)gg2)0 cost 92 sin sin -1. g2) (11))(((1111)) 91s cot t) sin sin-2 dt.
E
Consequently, to conclude a Lipschitz estimate for ((. 9)) it suffices to control the quantity under the integral the previous expression. We will start by controlling the second term using Markov's inequality. Following (E.70). calculate
n gi. cos" sin?
Loss cxp
E [1 91, (1,00)1((0(0000 cot sin =t = gi .02
< 8C,
sin
g1
4Cn
1g
n2 g
π
J sin
a
exp
sin
dg
40n
/
gl
exp
(-7
n2
dg.
where the last line follows from the change of variables + 9 sin in the integral. We can evaluate this integral with Lemma G.1l which gives bound
128C
E 91 .9
(t. -1 cot t) sin - < nn
and therefore bound of C > an absolute constant the sum over i. As a byproduct of this estimate, we can assert that the second term is jointly integrable over [0, x IR" x R", which allows us apply Fubini's theorem and obtain the same differentiation result for E[f( .g)].. Meanwhile, beginning from (E.7I) we can write using the triangle inequality
n
C
E Σ (t.. cot sin -2,
91 + C² Vnlllgn
By Gauss-Lipschitz concentration and Lemma G.9 ability at least
we have that I|gıll2 S 1191 S 2 with prob-
2e 2e-cn, and since ||g| loo I g 2. we conclude with the same probability that
19rd < 2 simultaneously for all i. Mcanwhile, if X _N 1). have for any f > that
P|X| 2 f] 2 Ct,, where C > 0 is an absolute constant; so if X; miid W(0, 1). we have by independence and if t is less than absolute constant PIvi, [X, 2 t 2 Ct) 2 1 C'
where the last inequality uses the numerical inequality e - < - S e- valid for 0 S L < 2-
From this expression, conclude that when 0 en en-11/2 1/2 for an absolute constant > 0, we have
PIVi E [m]. [01a
2 1 2 1 Cn 3/2
so choosing in particular en (6+ for any > 0, conclude that Plvi E [n], Igval >
en 3/2-6) > 1 C'n-6 Then with probability least 1 C'n 2E en we have
E
9ng9 (t, g1. (t.g ())(p((((pp)))) 9ie cot t) sin 2
< Cn 145 + 3/1
ga
so as long as 2 1 we have
P
E
Σ
(t. g2, (2).))((--n)) cot sin 2t
> 1nô
C'n 2e mm
g2
225

---

Published as a conference paper at ICLR 2021
Proceeding to the qf term, from the expression (E.69) we get
d. 91 91. (21 cos 92s sin
(vo, B.) 11 [121 8u113 + (vo: m. (w 1/" 1155((11 (11 (2) |
2b( ((l1voll2)
(111 2
(1 12) | 2ツ 11
(E.72)
(vo. un)
( (11))122)
(e vn) ( (2) )x)) (vo, w.t أ
villew (12) 1
v/ R2 12) 2
(1(1lzll2) (w,, ú.))) (vo, w 11 (1 12)
Using the triangle inequality, Cauchy-Schwarc and Lemma E.31 we obtain
(Dp. ).)) v (1 [|
+ (wo. (2) ).) v
vollo) ( 112) 2ul12
(1 ( 1
112) e
(using also the fact that (x) 0 and (x)) 0 whenever is sufficiently near 0); and
No v. w (())) u.)
(11va112)2 v.
x (11v)l1)) (wn, (x)) (wo, un 11,
I Cll/ul12 C*
2 (11 dx) 00. un) 1
21 22
from which we conclude
dlv.g1.92) cos + 92 sinv) S Cll + C"
for some absolute constants C, > 0. By Lemma E.17 there is event & of probability least 1- Cee-n on which we have |/1,122 S for every v. Moreover, we have from the triangle inequality that || l|g1|2 + Ig2lle which is independent of y; and in particular we have
2
(lv.9i.9)) (gu cos v G2: sin v)
s (C(l ll + (gall2) (9(lg ll2 + (1
which is polynomial in [g. ||2 and llgzle by the binomial theorem. Thus, applying independence,
Lemma G.10 Lemma G. yields that there is an absolute constant C" > such that
(C(ll l12 + l122l22
C"(ll 112 1192ll2))) C".
01.02
Therefore, as the framework section of the proof Lemma E. 13
we can use the inequality
(. 91.92)0 cos + 92 sin
<
(v. 91. (210 + 92e sin v)
g2
82
(B. 73)
together with the partition
E
Σ
dil 91, (9w cos v + 924 sin
1=1
I Cd
+ g2
1 (gr)
Σ d(v, (1. (2) cos g2s sin v)
(E.74)
and this last expression can be used to obtain gi event of not much smaller probability 1 Ce -mn on which the LHS of (E.74), and hence the LHS of (E73), is controlled by absolute constant
226

---

Published as a conference paper at ICLR 2021
uniformly in (in particular, using Markov's inequality in the framework section of the proof of Lemma E. 13 Consequently, one more application the triangle inequality gives that
P
(v, 91 g2)l
91. is C+ C'n' " -Lipschitz 2 c" n" g1.92
as long as 2 2-
Lemma E.46. In the notation of Lemma E.13 if there absolute constants C, C' C" O and absolute constants K, K' > 0 such that if 2 Kdlog and d 2 K', there is an event with probability least Ce-en en C'n
which has
Vv E [0, m], (6 g2) (V. S C"1 V"V doog
Proof. The argument is extremely similar to Lemma E.43 since both terms have small expectations and deviations essentially determinable by the same mean-zero random variable.
We are going to control the expectation first, showing that it is small; then prove that [Esl is small uniformly in v. Let & denote the event E0.s,0 in Lemma E.16 then by that lemma, has probability at least 1 Ce- as long 2 C", where C, > O are absolute constants, and for (g), 92)
E, one has for all v E [0, s
(vo, U.) (Un, 0.)²
Eo(vgg199)
2 22.12
Thus, if we write
(0o, Uy) 1r) (1 2z,)
E6(u, Go 91.99 92) = (1. g2)
11 1201l2112113
have Ee Eg for all whenever (g1 92) E, that for any
(E E E 91 92) E 91 91
[ g1. g2)ll + Ce
where the second line uses the triangle inequality and the Schwarz inequality and Lemma E.37 together with the Lyapunov inequality. We proceed with analyzing the expectation of Eg. Using the Schwarz inequality gives
E Eo(
$ 3E 3E (vo. v2) (w,, bx) 1/2 1/2
1g
1/2
|
10
< 192 (wo. w2 (uu, B.) ) 1/2
and the checks at and around E.36 the proof of Lemmal E.15 show that can apply Lemma E.30 to obtain
E (vo, v2) 2 bx)"
-n - cos + sin Bij cos + 921 sinv) cos - 911 sin
<
But we have using rotational invariance that (g( cos v + g21 sin (92) cos v gn sin v) = 0. which implies
| [(0,, (v,, [1 < C/n,
from which we conclude for all
E EE < ClVR.
Next, we control the deviations Ee with high probability. By Lemma E.17 there is an event E, with probability at least 1 -on which | I for every E [0 m]. Therefore the
227

---

Published as a conference paper at ICLR 2021
event E = n Eas which has probability at least 1 Cn by union bound, have using Cauchy-Sehward that for every y
gl, g1 92)1 - (1,4| e.)l
Using the high probability deviations bound established in (E.64). it follows that if n. is large enough then with probability at least 1 - Ce-Ceer C'n 22d+1/2 we have
Yu E m], Eo(u, 02.92) S CVV dllog
As long as d 2
2' we have that this probability is at least 1 - Ce - C'n and so the triangle
incquality yields finally that with probability at least - Ce C"n
dlogn
Vv E [0, [6(v. 91. 92) - 91. C"V
Lemma E.47. Consider the function
g(v) = -nm ( cos v +i v) [[[ v) cos v -in + ((t--))f (t) -(( cos + v/ sin" v,
which is the negated numerator of G. Then if0 < v S 0/2, one has a bound
2x² 2/
83
/ - g(v).
24
and the lower bound is positive if0 < v S 1
Proof. To see that the lower bound is positive under the stated condition, write
2T 33
83
22T²
83
24 2d v3
;
3 24
the quantity in parentheses is positive in neighborhood of zero by continuity, and in fact one calculates for its unique zero 48T² 4 /249, and one verifies numerically that 48T /249 1.9 > 7/2. We conclude that the bound is positive for v 1.9 by continuity.
To establish the bound, we employ Taylor expansion of the numerator. which is a smooth func- tion on (0, .) with continuous derivatives of all orders on (0, T). in neighhorhood of zero. In our development in the proof of Lemma E.5) we showed that the analytic function -g(v) -(2m² /3)p² + ((4)) near zero, so Taylor's theorem with Lagrange remainder implies
2TT
1
3/
inf
(v) S sg(v).
24
vE(0.T/2)
and it suffices get suitable bounds on the fourth derivative of g. We will develop the bounds rather tediously. Start by distributing in to write
g(v) = 3 (- v) + 2 (3xr cos v + sin v) +u (cos - 2n cos v 2x sin cos v))
(M)
g2(x)
(1(M)
+ t 3 v 2 sin sin
COSP)
(M())
Using the Leibniz rule, we have for the fourth derivative
((4) v3 (v) + 12 (v) 12g (3) (v)) + 1 (g(((()) 8g2 (3)(()) +369 (2) (v))
(90 (x) 4gl (3) (v) 12g (2) (v) + 21)g0(())
228

---

Published as a conference paper at ICLR 2021
To calculate these derivatives, we just need to differentiate sin, cos, and their third powers. Write c(v) = cos" (v) and s(z) = sin (v);; using the elementary calculations
(v) 3s(v) 3 s v,
(1 cos v 9c(v),
dl) 21 sin v 27s(v), (v) 60 cos // + 81c(p);
(E.75)
s (V) cos 3c(v),
c(2) (v) 6 v 9s(v),
( (v) 27c(v) 21 cos
(4) (v) = 60 si + 81s(1),
one can calculate the results
93 (1) (v)
92 (v) 3T + sin
(4)
(4) (v) (61 - 2 cos y 2n sin 81 cos
91
0 ('(v)) (v) = (2m² 60) sin v 50m cos v + 81 cos 81 sin vs
and
93 (v) sin V,
(3) (v) = 3r sin CoS 92
91 (v) = (7- - (7-2 sin + 2n cos v/ 27 cos² v sin
and
(2) cos
92 (v) 3n cos sin
(2)
93
and finally
93 (1) (v) sin
Plugging back into (F.75) and canceling, we get
g (4) (v) = v cos v) + (3T cos v slsin v) +u (22m sin v + +(9 - 2nm²) cos v - 81 cos v)
ha(P)
h2(")
(y(v)
+ (27 sin v + 81m 2 + 31T cos (6x² + 128) sin
ho(w)
Since v > O, we can leverage lower bounds on each h4 term. We have trivially [ha] S 1, so that hr (v)l s 7 7//8. We will study phj (1) + ho(v) together to get better bound. We have
(v) + ho(v)=)))))))) (22TT - (6x² + 128)) sin 27 sin v + ((89 - -2m²) + 31T) cos v
+ (81x 81v) cos"
(22 (6m² + 128)) sin 27 sin + ((89 -2m3)2 + 31T) cos v,
a(v)
(E.76)
using v I x/2 and cos 2 0 on this domain. We will show that the RHS of the final inequality, denoted q, is decreasing function of v, and is therefore lower bounded by its value at π 2 on our interval of interest. We calculate
('())= 97 sin v + (42 8x²²) cos v + 22 (89- -2772 22T²)) 81 cos
Reordering terms,
write
3 v + d'(v) = -81 cos
9T
(89- 2n 27
sin
(8n
42)
22 v
cos v.
(E.77)
C3
C₁
We can estimate numerically
69 < C2 < 70;
69 < C4 < 70;
C2 >
C4.
which shows that C₁, C2, C3, C4 > 0 and both of the linear prefactors are decreasing functions of We have all of (0, n/ 7/2) by concavity of sin
(C1 - Ciu)sinu SV (G
2C2
229

---

Published as a conference paper at ICLR 2021
using in particular sin S and sin v 2 (2/x)v. Using similarly concavity of cos on this domain, in particular the inequalities cos v I 7/2 1/ and cos v/ 2 - (2/ we have
-Css Cav) cos v - Cav²² (2C' +
CAT
v + CiS
2
In total, we have bound
( 81 cos
20
C4 Ca)()
2²
(2Cs
220s
TC's
+ +.i v C3.
π
2
We calculate the maximizer of the concave quadratic function of in the previous bound via differ- entiation; plugging in, we get
(2Cs
+ CCas +G1)
d(v)s- 88 cos +
2
C3. + +Ca)
20
A numerical estimate gives
2Cg
xC4 2
+
CG)
C33 S 20,
(2 +C))
and using that cos is strictly decreasing for v < we can therefore guarantee o' < 0 as long as S cos- 2/20/81. Writing cos"l // //20/81, we estimate numerically 0.90 2 2 0.89, so that this bound is nonvacuous. For v 2 c, we apply again concavity of cos to develop the lower bound
COSV 2
cos C,
E [c, x/2].
2 -
Using this to estimate the cos term our upper bound for o' we obtain bound
( 20
4/2
202 + +C,
22 +
22'
TCA
C1) v Cs. csvST/2.
7/2
π
2
We define D = 20/(m/22 c)., A = 202/T π + C4, B 203/ + xC4/2 + C₁ and C C3. so that the RHS can be written as - D(T v)a to zero results the quadratic equation
Av + By C. Differentiating once and equating
2A 3D
B 3D
3D
D2
2//4
M
which has roots M/2 M 4N. Numerically estimating the constants, we get that the two
roots lie in [0.99, 1] and [3.3, 3.4]. so that we need only consider the smaller root. Differentiating once more to determine the class of the critical point, we find for the second derivative at M/2 VM 4N
-3DV M 4N <0.
so that M/2 1/2 V 4N is a maximizer for our cubic bound, and the bound is increasing for arguments less than this point and decreasing for arguments greater than it; conclude that the
zero in [3.3,3, 3.4] is a minimizer, so that our bound can be ascertained negative by checking its value at M/2 - V2 4N. We find using a numerical estimate
4/2 (M/2 L/M² 4N)
20
T/2
202
+ C.
(M/2 -/ - 2
M2
π
+
2C'
+
CC1
+ +.)
(M/2 - ///// 122 4N) - - -1.7 <0
π
2
230

---

Published as a conference paper at ICLR 2021
which proves that q' < 0 on (c, π /2]. This shows that our lower bound on 1//((1) ho(v) in (E.76)
nonincreasing on [0, 2/]]. so that we can assert
(v) + 2 (22 (6T² + 128)) sin((1/2) 27 sin diiin(z7)) (x/2)
+ (189 (== + 31m) cOs(TT/2)
5n 101.
remains to bound (²h2(u) (3n cos v 11 sin On [0, n 1/2 cos is decreasing and sin increasing, so 3π cos / 11 sin v is decreasing here; it is positive at v = 0 and negative at -/2.
so that by continuity has a unique zero in (0, x/2). Denote this zero as vo: then using that v² with zeros in the interior, can write
h22(()) 0, OSuS0o
inf
and
sup v² x²)
inf ( ( bz(()
inf SoSUST/2
hh2(1)2 2
VOSUSA/2
2 (m/2) cos( 11 sin( 11x²
which gives the bound 12/2(1) 2 - 11x /4 on [0, 12). Putting it all together, we have
11
(0)
+ 5m 101 - x² /8 - 83,
4
where the last inequality follows from a numerical estimate of the constants.
Lemma E.48 (Uniformization). Let (2, be a complete probability space. For some t E R, 2 O, S Rdd and event & E F, suppose that f S - E. is second-argument measurable and satisfies
all E S, P[[(,, ) 11121- b;;
2. For all g E, f(.,g) L-Lipschitz;
3. There is M > such that supaes ||0||2 S M.
Then g + supees (x, g) measurable, and for every 0, one has
sup f(z, )) st LE 2 1 6, (1+ 2M d LESS
PP[]].
(E.78)
Proof. Because S is subset of the separable metric space (IRd) ||. |12) and all sample trajectories f(.,g) are assumed (Lipschitz) continuous, the supremum the definition of g + sup f(x, g) can be taken countable subset of S, and the resulting function of g is measurable (e.g.. (Ledoux & Talagrand 1991 $2.2 p. 45)). By (Vershynin) 2018, Proposition 4.2.12) and boundedness of S, for every E 0 there exists an E-net of S having cardinality at most (1 + 2M/s) denote these nets as Ng. Since each N₂ finite, we may also define for each x E S a point æ; such that ||æ: l2 I E; then for every g E, we have f(x f(x,, g)l < Le. We define collection
of events & by
(g E Ω Vx No. f(x g) St.
(E.79)
The triangle inequality then implies that if g E EE,,, then for all x E S, one has f(x, g) < + Ls. Consequently, several union bounds yield
P sup f(x, )) > 1 + L7 wES
< P sup
f(a, g) S + P[8]
LØEN
(E.80)
2M
1 1
P[E].
claimed.
231

---

Published as a conference paper at ICLR 2021
E.4 DEFERRED PROOFS
Proof of Lemma E.5 The function cos is C on (-1, 1), and because [(v) :- cos (f()) ) is smooth
and satisfies (V) (m-ty- 1) sin < 0 if v with s(0) = 1 and ((x) O, we see that 4ي is
-oe on (0, by the chain rule. This also shows (00) cos" (1) = and (TT) = cos cos-(()) (0) = 7/2.
Direct calculation gives
(π 2/2 sin '
(0()) V x² ((T v) cosv + sin
(E.81)
and
(((V) (n²² [(tT v) cos v + sin wi
v) cosv sin
(T² [[ - 2 cos + sin v/ 12)3/22
(E.82)
v) v) cos v sin // sin
(7²2 [(T 1/) cos + sin //3}3//
Calculating endpoint limits using these expressions will suffice to show the derivatives are continu- ous on [0, T] and give the claimed values there. We have
( v)2 sin" v
lim (b(v))) = lim
~ x² - ((T v) cos 1/ + sin
2(T v) sin 1) cosv sin
lim
w'00 (-2) - v) cos + sin sin v/ - (a - 1) (i)ssin V/ - cos v/
lim (x u) 008 sin 1, 1,
~/0 (πt 1 cos sin v
by L'Hôpital's rule, whereas direct evaluation gives
lim ,
(+(v))
0.
Continuity of the square root function gives the claimed results for gb. Again by direct calculation, we find
lim (f(x)) = 3 0.
Since :2² is meromorphic neighborhood of with, as we have shown, removable singularity at 0, it is actually analytic, and we can calculate further derivatives at 0 by expanding it locally at O.
We use the expansions sin v = V - v²/6 + O(v²) and cos v = 1- 1-112/2 v2/22 + 1A 12/24- + O(25) near to calculate
(1
sin 2 v²
(1
2
T² - 1/ + +0(03)
-V
π
π
3
and
(1
sin
2
v² (
2
_/2 + 0(0²))
1
COS v +
1
-
V
π
3
3
from which it follows
(+(v))²=
2
#² 3-33 2
0(v²) (1
12 + 0(00))
π
3n²
3
3
By the geometric series, we then obtain
(v(x))² -
+ +
3m 9x
Taking the square root of this expression and applying the binomial series, we thus have from which we read off
2
(Qv)) = -
22 O(v3),
3π
2
1
lim ((v) =
lim ((v)
3x¹
3
232

---

Published as a conference paper at ICLR 2021
is clear from the analytical expression for ف and the mean value theorem that y is strictly increasing on [0, x|, since (T - 2/) sin v > i < V < æ. To prove strict concavity for v E (0, T), we start by simplifying notation. Consider the function (v) -p(TT 2),, which satisfies by the chain rule (5(()) = p - v). Because yp is strictly concave if and only if yp is strictly concave, it suffices to prove that ((T 2) O. We note
(( <0-
( |v cos v - sin (- sin v y cos v) < <v² vissnn v V - V/ cos v/).
Multiplying both sides of the latter inequality by sin v, dividing through by (v cos sin 1/) (which is positive on r),, since it equals cos² yp composed with a reversal about т), and distributing and moving terms to the RHS gives the equivalent condition
2² cos v sin" V
< v² - -sinn V,
(M v sin u)2
and canceling once more gives equivalently
cos + sin
122 sin?
(E.83)
p V sin
x²2
Using v cosv- sin v < 0, which follows from its derivative -v sin being negative on r). and
writing g(v) = 2 (v² sin² v), we have equivalently v cosv sinv > g()) cos V- sin v). and
rearranging gives the inequality
(1 g(v)) cos + g(v) sin sin V.
(E.84)
follows after squaring; so the LHS is a convex combination of cos and sin which in particular satisfies [(1 g(v)) cos v g(v) sin vl S max{]sin v/,, al cos v/}. As argued before, we have sin v v cos > 0 if v E (0, x); moreover, because v > O we have v cos > 0 if v E (0, π /2) and V I < 0 if y E (T/2,, x). We
Strict concavity of sin on (0, π) gives sin v < V, and < g(v)
can numerically determine sin(55/ /8) + (5m /8) cos (5 (5x/8) > O, and given that 5T /8 2 1.95 > T/2, it follows
cost:/ [[1 g cos + gv)) sin // < |sin 0 KV S 1.95,
which implies E.84 when 0 < S 1.95. Recalling that we are arguing for Φr in this setting, we
translate our results back to o and conclude that (2v) < 0 if π - 1.95 < v < म. To address the case where 0- < V
to conclude & < 0 provided 0 < V < 7/2, and a numerical estimate gives that म - 1.95 < T/22, so that we have S < 0 for all
1.95, we employ Lemma E.47 it allows
0 < v æ. Taking limits in o gives concavity at the endpoints (0, πT) well.
To bound @ away from zero on [0, π /2]. we apply Lemma
E.47 to assert
3
83 244
((v) - 3x
24
0 D<vST/2
(1 - cos2
(3)))//2
The numerator in the last expression is nonpositive if 0 S U S 7/2, and using the lower bound in Lemma E. [0, 1/2], have
1> 0.
cos2 plv)) max? (1
2 22200
1
From nonpositivity of the numerator, it follows
1/ + 2473 2/4
*(v) S (1- 1i-max2(1 1/12 0}) 3/2"
0 0<D
(E.85)
0))3322
We have 1 - !u² > 0 as long as 0 s v < 22: so after removing the max, distributing, and
cancelling, we have
2 83
((v) < 3x
v
0 0< V2.
(1 122) 3/2
The denominator of this last expression is nonnegative and has singularities +22 and clearly even symmetric; so it is maximized on S /2 at 22, and we have
83
((v) S v8
3x + 24
0<0SV2
233

---

Published as a conference paper at ICLR 2021
Taking limits 0, assert this bound on [0, v2], and the bound is clearly increasing function of v, from which follows
2
83v
sup velovv
( S v8
S
-0.15,
3π
24
where the last inequality follows from numerical estimate of the constants. On the other hand, when v2 < S 2/2, we have from (E.85) that
83 24m
V2 S T/2.
3π
If we differentiate the degree four polynomial we find a double critical point at V 0 and a critical point at V = 12m²² /83; a numerical estimate confirms that this critical point lies in the interior of [y π /2]. The second derivative of the RHS is -4//x)u 83/ (3/(2r))'² and plugging in 12 2 /83 gives value of -484/83 + 144 144x/83, which is positive; hence the RHS is maximized on the boundary, i.e.
the RHS of this bound and solve for critical points,
83
max {
25/2
83
n²
83
((v) S
33 +
52
3T
24
3x
6
12
384
numerical estimate shows that the RHS of the last inequality is no larger than -0. 0..1.. Since the intervals we have proved bound over cover (0, 1/2,, this proves the claim with 00.14.
The bound و < 1 on (0, 77) follows from the fact that yp is strictly concave on (0. .T and the mean value theorem; have already shown 3 0 in proving strict increasingness of yp. Similarly, the proof of strict concavity in the interior has already established S < 0. To obtain the lower bound on G, that ş is continuous on [0, ø| and the Weierstrass theorem assert that there is C such that c 2 -C on [0, n|; because (0) * O, we actually have C' > 0.
For the quadratic model, we use our previous results and Taylor expand yp about O; we get immedi- ately
p(v) 2 v 2² inf vej.=]] ((V) -- (C/2))²
2
For the upper bound, we can assert immediately on [0, x/2 bound
v(v) 1リー-va cu?
where c = 0.07 suffices. To extend the bound to E [m/2, TT. we employ bootstrapping argument; because yp concave, we have bound
(p)) ((T/2) (TT/2) 7/2)
T/2
cos π
(v x12),
V
where the second line plugs into the formulas for yp and هb. We will show that the graph of v - cv² lies entirely above the graph of the RHS of this inequality. This condition is equivalent to
-ov² (1-
(T/2) - cos" π x-1) 2 0:
v
72
the LHS of this inequality is concave quadratic with maximizer = 1/(2c) ) and numerical estimation of the constants gives v4 2 m. Since 24 outside /n/2, πt] and the quadratic concave, we conclude that the bound is tightest at the boundary point π 2/2,, and one checks numeri- cally
7/2
(1
1
(m/2))
cT /4+
(1--
7/2+
cos --
2 0.15 > 0,
-1
V
which establishes that the bound ppv) 5 az2 actually holds on all of [0, T). This completes the proof of all of the claims.
234

---

Published as a conference paper at ICLR 2021
F CONTROLLING CHANGES DURING TRAINING
F.1 PRELIMINARIES
We now consider the changes in the integral operator O₂ during gradient descent. In this section we restore the iteration subscript (that is dropped in other sections to lighten notation) to various quantities. O2 changes during training as result of both smooth changes in the features at all layers and non-smooth changes in the backward features {3) (a: due to the non-smoothness of the derivative of the ReLU function.
Because of the difficulty of reasoning precisely about the changes in O, we will bound these rather naively by controlling θ over all possible support patterns of the features given a bound on the norm change of the pre-activations.
We now define trajectory in parameter space that interpolates between the iterates of gradient descent, given for any k' E (k and [0, 1] by
0 O
TS T N (0
(F.1)
(with the formal derivative V defined in Appendix A.I.p We will henceforth use k'' to denote an integer indexing the iteration number and to denote a continuous parameter taking values in [0, k] (such that k' [ 8 l [t] )- Quantities indexed by / are ones where the parameters take the value O, To lighten notation, we will drop the superscript when referring to time-indexed quantities (aside from G and O ) but all such quantities depend on the parameters defined by
Instead of considering the change in the features { directly, it will be more convenient to work in terms of the pre- pre-activations, which are given at layer é by
p. Wi Ple...
W P1e22(()) W
(h..(r) W.
We define maximal allowable change in the pre-activation norm by
C T,1,3/24
(F22
Vn
for q and constant C, to be specified later, where the scaling is chosen with foresight. We can then define a maximal number of iterations such that the pre-activation norms at all layers along the trajectory (F.T) change by no more than v.. This number ki, must satisfy
sup 1E )
[ (x) <1
(F3)
for 1) given by F2
Our goal will be to show that we can in fact train for long enough so as to reduce the fitting error without exceeding k. iterations.
F.2 CHANGES IN FEATURE SUPPORTS DURING TRAINING
Recalling the definition of the feature supports at layer E, time and x E by It,(3)) =
(x) > 0) C [n]. we denote by [xxx) (Zz, I.,((:))) the collection of these support patterns at all layers. We would next like to relate the smooth changes in the pre-activation norms to the non-smooth changes in the supports collection of support patterns with J; [n]. We now consider sets of support patterns that not too different from those at initialization, as defined by
the features. We denote by J = (JI,...,JJ JL)
B(y {supp (y + > 0) ||lll2 - n
(F.4)
J X B(p((x),m) m)) EE[L]
(F.5)
J (M Jz
(F.6)
aEM
Note that (x)) .7) is simply the set of supports of the positive entries of p (x)) + v for every possible perturbation e of norm most 7). We consider all possible perturbations due to the complex
235

---

Published as a conference paper at ICLR 2021
nature of the training dynamics. As result of this worst-casing, the scaling will require of the depth and width of the network in order to guarantee that the changes during training are sufficiently small is expected to be suboptimal.
For a given general support pattern J. we define generalized backward features and transfer matrices
B'J.
(W PJ W.N W.,,, WN P.
(F.7)
W.P,, W. Ps.. W
where the weights are given by F.1 (and thus (()z) - (a) By controlling these objects for every possible set of supports I that can be encountered during training, we can control the smooth changes in the features themselves. A first step towards this end is understanding how many such support patterns we expect to see given the constraint in F.2
In order to bound the number supports that can be encountered during training, we need to control the diameter of (( ()) This can be done by defining
(z) max supp (x) 0) e supp (p)(2)) + +0>000 >2>0 0)
(F.8)
||
where è denotes the symmetric difference. Since the pre-activation at given layer are Gaussian variables conditioned all the previous layer weights, bounding the size of 6, (x)())) can be reduced to showing concentration of a certain function of Gaussian order statistics. This is achieved in the following lemma:
Lemma F.1. For given by F22 if L, d satisfy the requirements of lemma F6 and n > d for some constant K, then for a vector g E R", g RR, mid NO, ( we have
IP (,(g)) > Cmnj2/3 s C'd
for some constants C, C, C"
Proof. Let
(gl()) S (9 S... lgl())
denote the order statistics of the magnitudes of the elements of g. We will show that bounding (,( can be reduced to understanding what is the smallest & such that
gl + +
1 (kk 2 n7²
We denote this value of by knp Define indices je by IgJ, = (g(()) (and breaking ties arbitrarily in case several order statistics are equal). To see that
kn S (g) < kn
(F.9)
k
suffices to note that since
9 (k)
< n one can choose E > 0 small enough such that
kn 1
== Σ
(1+ + 1eegee,, E BE(g.").
which will give vg kn 8, kn To prove the second inequality, consider by g E 9.. e.. Clearly for any y' such that dg y') = S, we have ||g -112 S g|g y y'll2-
Since there exists at least one such y' E BB 11) it follows that ||g - 2ll S lgg - y ll2 < and
hence the smallest à such that Σ |atal 2 2 1/² must obey k 2 6g.
For y defined in F22 satisfy the requirement on in lemma F.6 by requiring > d for some K. Applying this lemma, we find that there is constant K" such that for k 2/3 2/37 we have
k
P
L
N
21 Ccd
236

---

Published as a conference paper at ICLR 2021
from which it follows immediately that
P
kn S
2/3 K K'nn22/
2 Cee
Choosing some constant C' such that K K''nn22 2/3 (g) with the same probability.
I Cnn2/3 and using F.9) allows us to bound
With this result in hand, we can control the objects in (F.7) for all the supports in (M,)..
Lemma F.2. Assume d, L, n satisfy the assumptions lemmas D.2
D.8 D.14 and additionally
n 2 max KdL K' (log (logn)"/2 3/22 C2 6424
for some constants K, K', Co. where is the constant in F.2
Then
f)for To I (2)) defined in
F.2
F.5
on an event of probability at least ecdd simultaneously
sup
sup
sup (E[L]
||
S SC2
JET
| S
sup sup
sup LE[L]
M ,,T,, ist'st
| 3.0
C2VL.
sup sup =EM TTT,((a)
sup fE(r]
ii) for T,, defined FF on an event of probability at least ood
| - 11100)0011 SCO2/S (CC// (1)
sup aE.MIE(O.T,,,eEj
for some constants e, C.
Proof. Deferred
HS
F.3 CHANGES IN FEATURES DURING TRAINING
We can now bound the smooth changes during training:
Lemma F33 (Smooth changes during training). Set the step size T and a bound on the maximal number of iterations kmax such that
L
kmax
for some constant Assume n, L, d satisfy the requirements of lemmas F22 and in particular n 2 K 9424 for some K. Assume also that given some kimx for all k' E t0....,k. k}.
| llee, SCVd.
(F.10)
Then event of probability at least - e cd one has simultaneously
[1 - pb(2))12 $ CC
sup E LE[L], # EE(O.....b+1))
(118 (x)
(z))1 (z) -10))(0)))8(d()))
sup #EM, BE[L], E k+1)
for some constants c, C, C".
237

---

Published as a conference paper at ICLR 2021
Proof. We will bound the smooth changes in the network features during gradient descent with respect to either the population measure ut or the finite sample measure μ We denote a measure that can be one of these two by μ..
For any collection of supports J E (M), define generalized backward features and transfer matrices at by B'ge Nt. These are obtained by setting the network parameters to be 0, according to (F.T), but setting all the support patterns to be those in J.
We then define for any t [0, k + 1).
pl =
sup (EM)
(1(((x) Pb(2))11+ + sup
sup 32 (E10),)E7,(,,),,((((..
118'r 8300122 + (E(().JET,(MM)
11,30012
sup
(F.11)
r
3r -1 +
sup
sup (3TE[()JJET,MM)
30
(
We have for all N k
-b(cz))12 S Pk. Po;
(F.12)
while
(x) - (b(())
B
Enl(as) - B 12.
=
B B (a)))) - B (T) 11 + 111
(F.13)
<3 30 + +118/ 32o(m).0
It follows that we can control the difference norms of the pre-activations and backward features by controlling the magnitudes of ph. B3 In order to control these, we also note that for any e [0, k+ 1].
||a( s ||((()))l2 s [|(((a)) pb(x))l2 + | S pp.
(F.14)
and similarly
||
S3'
2
(F.15)
Neg"" Jt
<T
In particular, the above bounds hold when J = I, (x)) We would now like to understand how the quantities (ph- Fg, F% evolve under gradient descent. Towards this end, for any k' E (O....,k)
and s [0, 1] we compute at any point differentiability
8 Os (xx)
Öp (xz)
7. (0
00
0 1
0pt +s (x)
J. (0N)
00
1e-1
Op (x) Oc (ON)
OW
aW
(at-1(x) (x), a (x')) peit1 Ph 1 (20) (2e (a')dx² (x').
ECEM
238

---

Published as a conference paper at ICLR 2021
Using (.14) and (F.15)
then gives
Pw Pw +s (x)
Stl. ST (pk+4o)
P Po 104tTi kst W++
/ (xe)) du
Us
Z'EM
S-l. (p%++t)
P F k'+s
|1 1112
S CV (pk+ P
where we used Jensen's inequality in the second line and our assumption that the error up to iteration has bounded 1nN norm, and we additionally assumed pi Br, r 2 Arguing as in the proof of Lemma B.8 for absolute continuity, it follows that
[|)((x)) 2b(2))12
1 pz) + ||(((0) ple, (20)
6-1e]
+ /
kk Pk (x)ds
p
Pi(+o(2)
0s
Os
(F.16)
[t--1
t-(
Os
Pk (x)
ds +
Os
11|ts
(x)
ds
# ==
SCVdrtL (p!) BTT
Since the above holds for all choices of x, simultaneously, conclude that
p - p < (
(F.17)
An analogous caleulation for the other quantities in F.1I gives the following of coupled differ- ence inequalities:
p
3
3
(F.18)
F"
Instead of solving F.18). we obtain sufficient control by defining s.t.
Vt E k0
2
3
23
r $ 21
(F.19)
For any namely
[0, k*), we obtain a sufficient condition for satisfying the above constraint using (F.18).
p d'v (7))
<2p8
(F:20)
+ rt
C'VIITE
for some constant C". Using the bounds for 3. - 3 and 1 -T E.18) to satisfy the second and
third condition (F.9 gives an identical constraint on rt.
In order to control these quantities at / = 0 we define an event
(
s C2}
3
S Cavn
G =
(F.21)
REEM
Fer 3.0
< C2VL
(w),
15e ((E
the probability of which can be controlled using lemma F2
On G. the upper bound on rt in E.20
at least
(F.22)
C' V'LL"
C" /d
239

---

Published as a conference paper at ICLR 2021
for some C
We would now like to pick Thmax. and ensure that any t E [0, k satisfies the constraint above k + 1 S kmax. The analysis also assumes that Tkmax Tn for which (F3) holds. We will then pick
the scaling factor C, for the pre-activation norm bound in (.2) in order to satisfy that constraint as well. We choose
L
Tkmax
(F.23)
In order to ensure that kmax S k" holds, we use (.20 and F22 and require
L
n
C"
which is satisfied by demanding we have
3 for some constant K. Using F.17 and (F222 on g
sup MEB
[ pb(2))12
SPi P
s SCTIVVaL (T8) BoT
5 v d/2//2 vn.
In order to ensure that kmax kn therefore require
Th VdI3 Vn Sn.
which using
F.2 and F.23 is equivalent to
O VdI.+3/2 Vn
Lo+3/2
S C
Vn
and thus the constraint can be satisfied by choosing C, CHI Vd. Note that the constant C2 in F.21 (which enters C") is set in lemma F22 which takes C, as input (despite this, C2 is independent of Cj)). This lemma holds as long as n 2 C.C 16+24 which we can guarantee by demanding Co (Vāc") T,6424.
2
We have thus ensured that our choice of kimax satisfies kinax min [k* k. We then obtain from the constraints in (F11),, the inequalities in (F.8) and the definition G. that on this event
C" Vd13/2+9
Th - Po < Tkma V k'e k+1)
sup
3%. - 3/ < Tkmax 3OL3/2 S C" VdL31220
sup (0((.....k+1)
Then using F.12 and F.13 we obtain on an event of probability at least 1 - e-d simultaneously
(x) p6(2))l1 C²1,312+4
sup .E.,, (E(r.) E' (011...+11)
(110 (xx)
(x))(=) 1 12/16(00) - -32 (2)) 1) V'V
sup mE., RE(L), (o....,
The combination of the last two lemmas allows us to control the changes all the forward and backward features uniformly:
240

---

Published as a conference paper at ICLR 2021
Lemma F.4. Assume n, L, d. k satisfy the requirements of lemmas F.2 and F33
and additionally
n 2 K L 8 for some K. Then one has simultnnously on an event of probability at least
1 e
||a(()) a1(xx)112 = C1 n
sup LEM, EEIL]
| -a) S Cloo 8/4 (L))0(/43+2003,
sup
(EM, FE(T.)
||);( SC.
sup
rE,,, EE
sup 1EM,, (a) S SCvn.
(m))l
for some constants C.
Proof. Combine the results of lemmas F.2 and F33 and take a union bound, using the triangle in- equality to obtain the second two bounds. The assumption 2 KI,30+8a is required in showing | (2))l2 S CVn.
F.4 CHANGES IN OF DURING TRAINING
With these results in hand, control of the changes in 6 during training is straightforward.
Lemma F5 (Uniform control of changes in & during training). Denoting the gradient descent step size by T, choose some kmax such that
L n
kimaxT
for some constant 9. Assume also that given some io S kimax - 1, for all W' k).
|| 11,, S Vd.
(F:24)
Define
( (ab (x), at (x')) + Σ af(x)) (((())),((x'))
AN
O (z. x') =(((m,=)))
sup (a JMMMM ke((O....k)
Assume 2 36+6+84 d, d > K'do log (mm)CM) for constants K, K', Then on an event of probability least ed
A Clogg// (/log3/(((L))) (L) 11/12
for some constants c, C.
Proof. Recall that
0fo(z)
0fo(z)
O (x,
ds
d0
80
==0
8
ΣΣ
( (m), of())) (x) ds
t= ==0
with the convention (x) for all t, and the parameters 0: given by
F.I We thus have
( (w), at (x())) (3f (x), 3 (x'))
I (x. - - O x')l
ds.
a6 (I)) (2()) (2')) (z) (()))))))((())))))
E=0
241

---

Published as a conference paper at ICLR 2021
We consider single summand in the above expression. On the event defined in lemma F4 for all 2, E &M,& {0, Lo....,L}.
(,) - (ab(a),, (((())
ab(x')) (oti))))
-
(()) (((')))
-(( (a') - a=(20)\>>
SC
+vm tv ( vn ([ (x(z')1
<C' (,3/2+9 log 3/4 (T.) 11/12)
<C log (T)
for some constants. Summing this bound over gives the desired result.
F.5 AUXILIARY LEMMAS AND PROOFS
Lemma F6. Consider a collection of ii.d variables X; = g. ~ NN h)) and denote the order statistics by (()) (so that X(1) S X(2)...) For any d 2 K' K' logn, n 2 K" d3 > d and integer Knnp//3 < k S where K, K' K" appropriately chosen absolute constants, have
33/4
k P
X
2 n2
21 -le-d
where c, C, C' are absolute constants.
Proof. We will relate sums of order statistics of X; to functions of uniform order statistics and show that these concentrate, We denote the CDF the X, and its inverse by F and F F-1 respectively. We use
(Xa)...,, ((x) (F- (U (U()))
where U() are order statisties with respect to Unif(0, 1) David 2011). Since X, 1y, Y, xi we have
a
F(x) = erf(
n
2
2 F-)(()= (erf-'(t))
CD
n
n
where the inequality we used the series representation erf This gives
h
k
(U
60
U2 ())
The joint PDF of the first order statistics for any distribution admitting a density is given by
n!!
k
(E....))(1,..... xx) (n k))! (1 F(xx)) nー I f(x)
242

---

Published as a conference paper at ICLR 2021
where 21 - 22 ... ze (David 2011
Applying this to the uniform order statistics, we
compute the mean of the summands (U/)
U2
MK
n!.
ux)= dug...du
nli(i (n k)!(k + 1)!
uk (1 ux) mm)
duy
(n k:)!
i(i+1) (n + 2) 1)
k[A + 1) [ 2) 3(n 2) + 1) n²
E
U
In order show concentration, appeal the Rényi representation of order statistics (Boucheron
& al. 2012). This allows us write EU (R)
Lipschitz function of independent exponential
random varables, and we can then apply standard concentration results for such functions (Tala- grand 1995). This representation is due useful property unique to the exponential distribution whereby the differences between order statistics are independent exponentially distributed variables themselves when properly normalized.
If we define by Ei,... En collection of independent standard exponential variables, the Rényi representation of the uniform order statistics gives
(-Ei E1 (-
E, -Σ exp - n n - +1
(U()),... U(x) _ 11--ey exp
n
We now truncate the (E1, Es), so that w.p. P ke K we have Vi E, [0, K]. and
b U is equal in distribution to convex
denote this event by Eg. Using K < Φ. it is evident that function of (Ei,..., Ex) after truncation (which can be seen by calculating second derivatives). The 4 Lipschitz constant of this function is bounded by n- = A.
we define rescaled variables E, XE,, then with the same probability they take values in [0, KX].
Σ U2 written terms of E, now 1-I.ipschitz and convex, and we can apply Talagrand's concen-
tration inequality TTlagrand
1995)
obtain
Ex
U/
EIg
EK
U2 (1)
tKA
S SCexp (-ct3).
Setting t = 2n
k
(n (n-k) 8
if we now assume
2/ 2/3
S k c'n
for some < 1 we obtain
&
k
c² n²nK2
c² nnn2118/3
Σ 1ex U8
1s
U2
S C exp
Cxxp
<
exp
2n²
K2
We would also like to ensure that the truncation does not cause large deviation in the mean. We have
UG - 1xx
2
(1)
)10) 1=1
i=1
)
2
))
E
Ej 1
1- -xexp exp
Σ
1
exp
i
i i+11
243

---

Published as a conference paper at ICLR 2021
))
2
Ej
E 1 Em
exp
-E
S k
E [B.}
1
Em K = 122 E 1E,SK
(Ei}
j=1
e --e K
Since would like this to be small compared E
E U3
>
cika n²
can require K
4n2 log
k Evg E Lex
which gives
<
c in
We can then choose the constant 0o such that with
(Uo 1=
==1
probability P2 1 - exp (log k K) Ceex
cn"
8/3 8/8
K²
(1)
CD
1x
Cu
E1 E1sk Σ
U
k-
71
72
(1)
2n²
k
CO
E i=1
C1
E1
0
E
U
2
11
2n
(1)
1=1
CuC k.
2
4n²
The upper bound on & can then be removed since the inequality then applies to all larger l: automat- ically. If we now set according to equation (F2),, and choose K K' log n and n satisfying n 2 K"d for appropriate constants K", K" we obtain
CC 1,484/3
( 2 17
21 1
exp (log k- d) Cep
21 21-C'
d²
and due to our choice of 1), this result holds for all k Cmnz2/3 > CC 2/3 2424/33
2/3
Proof lemma F2 i) We begin by controlling the pre-activation norms. Considering a point V e
N n an 11/2, where N 1n-3 the net defined in Appendix D.3.1 rotational invariance of the
Gaussian distribution gives
|| (x) W6ah - (x) d
|a (
| (wo)
where (Wo) (s.1)
is the first column of W. Bernstein's 's inequality then gives
IP [x) C a (x) [1]] 2 C'e
for appropriate constants. As discussed in Lemma D.8. if we choose d to satisfy the requirements of this lemma then N - < ddd for some constant. We can then uniformize over the net using a union bound, obtaining
Va E N ano 1/2
| s la [ 2 21-C'0"d > 21 -1e
for some c', assuming n 2 kd. We now need to control the feature norms and pre-activation norms off of the net. From (.62 and lemma G.10 obtain that for d satisfying the requirements of lemma D.9
Vx CREM,EE [L] Ex N n 1/2 N. n -9 1/2
8.1.
2 1 cod
{l)p((=) - pf(=)) | S On
5/2
< 2) { a (z) - 1| 1/15//
By taking a union bound over the above two results, we obtain
P P[ E M,E [T]
||
2 - -ed
(F.25)
244

---

Published as a conference paper at ICLR 2021
for constants
We next turn to controlling the generalized backward features and transfer matrices. Our first task is to bound the number of support patterns that can be encountered, namely [ In order to do this it will be convenient introduce set that contains J (M) and is easier reason about. We define a metric between supports by
dsppp(S,S") S
and denote by (S, C. P([zl) a ball defined with respect to this metric, where ò E {0) [n] and P(A) is the power set of set A. For (1) and B(y. 17) defined in (F8) and (F.4) respectively, it is clear that
Bly, n) (supp(y 0), 8,,))
and consequently
L @ (suup((((((2)) > 0), 8, (x)()))).
J,
U 2E,M
(F.26)
We will aim to control the volume of this set, which we will achieve by controlling first on a net. This will require transferring control between different nearby points.
For any S' E [m] and E {0} U [m], the triangle incquality implics
(5, S (S" 6 d,(S,
For some p Be(grr), also have
8, (p)
max dg (p. y) S de (p.g) + max dg (g. y) vEig(p.))
yERg(p.))
d, (g.y) <d. (p.g) + max
(F:27)
yCBr(g.tt>)
=d (p.g) + (g)
where we used Be(p 1) S B r). It follows that (supp(p > 0), (p))
(sup( (,) ()) +d.(gg))
(F.28)
(supp g > 0), ( 2d.
From D.62
and lemma G.10 obtain that for d satisfying the requirements of lemma D.8
Va E M, [Z] E E n 1/2 ON (11)((()())))
P
s.t.
2 1 6e d/2
(F.29)
pp(z)) 1 < Cn-5n/55 5/2 n (d. (T( <d]
since under the assumptions of the lemma de (p)(()) (z))
<
Σ Rg (z, On Cn-*)/, with
Rg (z,CC On- denoting the number of risky features as defined section D.3.1 event by Ep.
We denote this
On Ep, we can transfer control of the ball feature supports from a point on the any point on the manifold. For some é, x we denote by E the point on the net that satisfies the above condition.
Considering F.28), we choose = (T(=), = p,(x), r Cn 552 and obtaining
C, L3,33/24 1/2
(s( 0), 0),b,(P(())))
C B. (Ssppppppp (z) & s/2 5/2 (p(())) 2d.(p)()))
(F:30)
S B. (supp( p [ 0), S2n (p/ (z) + 2d),
where we assumed 3/2 C.
We next turn to controlling 52n (z)), which is now random variable, for all & E [L],
N ng 1/2. From lemma F.1 we have for vector with gi ~ia N/0, 1))
IP (2)(g) 2 2/1 < C'e -ed
245

---

Published as a conference paper at ICLR 2021
Considering vector (z) for some & [L]. N n-"nn 1/22 have
pb(T) a 1
Lemma
D.2 then gives
P
V2 0
2-1 (z) (3)
<1
& Cte od S Ce -'d
for some constants, assuming > K log L for some K. Since on the complement of this event we have 22n 2-1 (xx)l22 21,, lemma R.1 and a rescaling gives
P 62n (x)) 2 Comn2//3
IP | - (g) Canpars
S PP (2n Conn 2/3 P [22 (x) <1
(F31)
Ce
Ce od + C'e -d'd C'e-dà I cdddd
for some constants. Taking a union bound over N n-3n 1/2 and [T.] we obtain
P
3e N -- 1/, l [L] s.t. 62 (2n(p((())
Camp2/5 N
LCe S C'ee-d'd
(F.32)
under same assumptions on d as in lemma D&8 and additionally assuming d 2 log for some K.
Since n. d, y satisfy the assumptions of lemma C" and hence
we have m//2/3/ 2 C'n 2 C'd for some
P
E E N anp 1 & E [L] s.l. (p)()) + 2d 2 Cinu)2/8 < Ce Ce-ed
2/3
(F.33)
for some constants e, C, Ci. Denoting the complement of above event by Eg we find that on Ep n for every we can find æ E N an, O N n n-3ng (a)( such that
(.(supp(p((x)) (p(()()))))) C IB, (supp ( 0), (2)((( + 2d)
S B. (supp
( 0). 2/3)
where we used F.30p. On Ep neg we can thus bound the size of the set that contains J,, denoting its size by
S, Vol 2 B((sign(p((()))
#EM t=1
We first note that for any p.
[cayp/37
VolB, Cinn 2/3)
Σ
S SC Cincin2/
n / < C'n C"
for appropriate constants, assuming 2//2/3 K 1og(n?2/33 for some K. It follows that on &, NEN,
S, Vol U
B, (supp(p6(#)) (x))) Vol U & (s( 2/3)2/3)
m=1
aeM
3EM
(sup(ss))()(( (2mpp))
1=1VEN 2-30-112
< C² N n- Cdinn?13 < C² C"
for appropriate constants, since map 2/3 C and d satisfies the assumptions of lemma D.8
Since
after worsening constants we have E N I C'e--dd we obtain
P Sn > C' dd
< C'''-
(F.34)
246

---

Published as a conference paper at ICLR 2021
for some constants.
We would next like to employ lemma D.14 in order to control the quantities of interest for a single J I a (M). and then take union bound utilizing the upper bound above on [J,]- This will require controlling the event EsK in the lemma statement with an appropriate choice of the constants 5g, g. As in other sections, we use the convention r I for any E [L].
At a given collection of supports I E (2)) for some a E M, we choose a as the anchor point in lemma D.14
From F227
we have, for any E E n-3n 1/2,
(x)(()) < de (x)(2), p (x)) + +0++|lo((0 (Pb(T)).
Then using F.29)
we obtain bound the two terms in the RHS gives
P Vo E M, [T]:
Tr E 3no 1/2 O N Pnp 1/2 (x) s.l. 6n (p)(())) s d 82n (p)(T))
6c 6e d/2.
where we used 17 = C,,33/2tn 11/2 and d satisfies the requirements of lemma D.8 Using F.33 to bound + 62, (P(()) uniformly on N n nn
1/2 and l, and combining the failure probabilities of
these events by union bound, we obtain
P [ EM s.l. (p(p((=)) Cmnn2// I 6/- + Ce <
C'e-c'a
for some constants. Since (p(((r)) 2 Je e [=) for any J. E E (x),,, implies directly that
P V M, E J E (x) J/ e [.,0(>)] S C 2 C'e
(F.35)
In the notation lemma D.14 we denote this event by Es, and choose d₂ = C2n12/13 From the definition of J n, for every æ E . and Jr that is an element of J E J (n)()),
Jt = supp (x)(x) + v 2>>00 0)
for some such that ||0||2 S 17. We now consider the vector
w = (P,, - (()))) pt(x))
Note that for any element of w; that is non-zero, must have |vd| 2 ( (since the perturbation must change the sign of this element), in which case we have [ p6(=); (z))). Denoting the set of indices of these non-zero elements by Q, we have
|| ||00|22 = Σ n
Σ (pb())))² tEQ
1 ア
FEQ
IEQ
This holds for all E E [4]. Thus if we set K2 for Ka, the event Ex in lemma D.14 holds with probability We therefore choose
ESK- Eg
with &s defined in F35 In order apply D.14 must also ensure
8, 2
2/3
< L'
K, = 17 3//2
2/2
Setting 1) C2
Vn
as per
F2
we can satisfy these requirements by demanding n 2
We are now in a position to apply lemma D.14 to control the objects of interest. We use rotational invariance of the Gaussian distribution repeatedly to obtain
1xx 1|33yoo1,,
- -1
|| [ L 1 Plecall2 < 11
||
J0
a.a.
d EsK
||, d Lxx
|
L+ W
2
247

---

Published as a conference paper at ICLR 2021
Recalling that W (0, 1), Bernstein's 's inequality to obtain C"e and another application of lemma D. 14 gives [
P W Cyn f
|
el
> C
s C c c' for
some constants. Hence after worsening constant
[LExx |183
CVn sC'e
(F:36)
We also obtain
1
Feee 3,0
= 1ExM
1 J,0
Psp < LEw W Ty
P
LEax
F&ge 3.0
CVL S SCe en + e
- S e cc
where we used an e-net argument bound Will and lemma D.14 bound EsK T
1:' We
-||
now combine this result with (F36)). It remains to uniformize this result over the choice of l and J. Combining F.26 and (F.44 gives
[M(M) o" I CCe'
(F.37)
C213/249 vn
Denoting the complement of this event by Ey, and setting
on this event have
(EE
Fele 3.0
S CVT..
P VJ E (MM),,
<EE
&g
n LESK ||3y,0112
- Cvn
2 C²e dL +
Cn of o2 11.21 2/3
C'e
assuming 2 d for some constant K. Taking union bound over the probabilities of E or EKó not holding, we finally obtain
/ S CVT.] |3 S svn]
]
VJ E J (M), <LE
2 1 - C'e C'e-c' - P [89] - P
d'
2 1 -- cºe Cm
2- 61-
for appropriate constants, where used (H:35 bound TP [E and in the last inequality used
n 2 KLd for some K. Combining this with (F255 and taking a union bound gives the desired result.
ii) We will control (Lo(2)).0 B (00)0
using lemma D.21
For 1 (0, Ty) note that by
definition of T, and J, (œ).
I(x)) E (z)).
As noted in the previous section. if we set d to satisfy lemma D.8 and n 2 Kd for some K, then the requirements lemma D.14 are satisfied with
6, 2/3
K. 1/.
and
P|esk] = P[E)]] 2 1 -ed
where the last bound uses the definition of E, in F.35 From the definition of d in lemma D.21 on the event εgK we have
|
1 8, < 2/3
248

---

Published as a conference paper at ICLR 2021
Thus for some fixed t [0, 1,). if we denote d; [1t,((x) ((0z) for [L],
apply the
second result lemma D.21 choosing
du loo L,
K Kd -1/3
db K' d'd 2 n1
(F.38)
8; K" d 2424/3
max max(1,d) (1,41] di}
for some appropriately chosen constants K,K', K Assuming n n 1/12 / L3424/3 2 K for some constant K to simplify the result, we obtain
12
dL2 and
5/12 Ess (2 - Pz (2, > K=" / //4
S CCe Co" 2200/3,
+ Ch
The constants K, K' are chosen such that this result can be uniformized over the set of possible supports J (M) and [L]. Since on the event E defined in FF37) the size of this set is bounded, we have
1 EAN B
B 11000
Vn E M,ME (., [M]] P
Eg 21 C²e -do 2.232/222
CK"C2/3 5/12 3 d.
for some constant e, C, C", assuming 2 K 1 for some constant K"" Taking union bound over the complements of &g and Σ9K using FF37 and F.35). we have
x E M,, V t E [0.
P
g -
Io(x)) I | K" CK"C//nn>// log 3/4 (L)L 83/4
8E [L]
21 L'e-odgL
222 c'dd
21 Cn cdd'
for appropriate constants, assuming log ( for some constant K.
G AUXILIARY RESULTS
Lemma G.I (Hoeffding's Inequality (Vershynin, 2018 Theorem 2.2.6).. Let X1,..., NN be in- dependent random variables. Assume that X; E [m, Mi] for every i. Then for any t > 0, we have
N
21
P
Σ
XX,
E(X.)) 2 S exp
EN (M, (M,m,))3
Lemma G.2 (Bernstein's inequality (Vershynin, 2018 pendent mean-zero subexponential random variables. Then, for every 2 0, one has
Theorem 2.8.1)). Let XI,..., XN be inde-
E 1x max; X }):
t
Σ
x 2t
<
2cxp (-omin
where is absolute constant, and 11v1 inf 0 Biel 1/9 < 2) is the subexponential norm.
Lemma G.3 (Bernstein's 's inequality for bounded RVs
Vershyyin
2018 Thm. 2.8.4). For
X1,..., Xn independent, zero mean random variables such that Vi X K, and every 2 0, we have
2xxp (-
2/2 (- Kt/3)
P
X
2t
where x² =>=EXX² EX? 1=1
249

---

Published as a conference paper at ICLR 2021
Theorem 6.2.1)). Let g be vector of Lemma G.4 (Hanson-Wol Wright Inequality (Vershynin 2018 i..dd, mean zero, Lub-Gaussian variables and A be an n. x n matrix. Then for any > 0, we have
t²
P "lg" Ag Eg" Ag 2 1)
2exp
-cmin
K*
KA
K
||l
where max 11:illp2 I K (with Il-11 denoting the sub-Gaussian norm).
Lemma G.5 (Freedman's Inequality (Freedman of martingale differences, with
1975
Theorem 166). Let (Ai,F) be a sequence
E
E
0, S
Fi
and suppose that
A
a.s.
Define the quadratic variation
1 VL
Σ
I8 (A'²
zi-1].
1=1
Then
f2/2
P
31
L
A
and V s 0²
S eexp
(T 02 Rt/3/
i=
Lemma G.6 (Moment control Freedman's de la Peña 1999)). Let (Ai Fi) be sequence of mar- tingale differences, with
E A
Fi
0,
and suppose that
IE
(A')
F4-1]
I
k!
E (Ai)²
Fir1|
R
Vk, a.s.
2
Set
V Σ (A)²
Fi-1]
Then
f/22
P
3i
1... s.t.
> and V v²: I 2exp
(- 02 + Rt
Lemma G.7 (Martingales with subgaussian increments). Let (Ai, Fi) be sequence of martingale differences, and suppose that
12V2
E Exexp (AA') | [T < exp
AA, a.s.
Then
L
82
IP
S 2 exp
2LV2
Proof. By assumption, E conditional expectation
| = for each E [L].
We calculate using standard properties of the
E a El A
E E El
A'
FL-1
E
e D
A
F1-11]
I 2v² /2
e
A
Moreover, one has Es
[T = Ese < @VV22/2
An induction therefore implies
e E 47
s LV /2
I
and the result follows from standard equivalence propertics of subgaussian random variables (Ver- shynin 2018, Proposition 2.5.2).
250

---

Published as a conference paper at ICLR 2021
Lemma G.8 (Azuma- Hoeffddig Inequality (Azuma gale differences. and suppose that
1967 ) Let (Ai, Fi) be sequence of martin-
A
R₁ a.s..
Then
-12
A
S 2 exp
1 2 R i=1
Lemma G.9 (Chi and Inverse-Chi Expectations). Let X x(1) be chi random variable with degrees of freedom, equal to the square rool of the sum of n independent and identically distributed squared (0, 1) random variables. Then
[| = V2 ((((+
T(Zn)
and, if 2,
r ( 1))
----1==
V2 n()
Proof. We use the fact that the density X is given by
p(x) l2220(z) 2n/2 IT (n)
12/2
which can be proved easily using the Gaussian law and a transformation to spherical polar coordi-
nates (Muirhead) 1982 Theorem 2.1.3). The expectation of X then results from simple sequence of calculations using the change of variables formula:
E E
00
x =² dr
2
5
x/2 x: 11122 11/221122 1/22 dr
23
V2
(n122+1/2) dr
r(fn)
[ (( +1)
m()
Now study X-1 By the change of variables formula, its density is given by
(xx 1 x20 (x)
" 1/(2²))
2n/2 TT
similar sequence of calculations then yields
E
2
x e- -1/(2x²) dr
2n/21(
1
x. (n+1) 1) -1/(2x) dx
2n/2T (
1
1 (n 1) te- _ dx
2n/2T(])n)
1 (n-1)-1 1)-1 1--10- dx
V2T())) Jo
x22
1 r (n 1))
V2 n(
provided 1.
251

---

Published as a conference paper at ICLR 2021
Lemma G.10 (Equivalence of &e Norms), Let 1 S ppsq to0. Then for every R one has
| S | S n 1//1111911 ||=|||
Lemma G.11 (Gaussian Moments). Let p 1, and let NO0 1) be standard normal random variable. Then
E|gg|"] =2 r (p+1)
22/2
B[lg]]] 2 =l1
1
2
where [x]]+ max 0}. In particular [[[g]]] yp12, so that g is subgaussian and g² is subexpo- nential.
252


Reviews:
[{"review_id": "ta16eFzAadg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper under review studies the question of whether gradient descent can solve the problem of calibrating a deep neural network for separating two submanifolds of the sphere. The problem studied in the paper is very interesting and as been the subject of recent increasing interest in the machine learning community. The contribution is restricted to a simple set up and addresses the question in the finite sample regime. The framework of the analysis hinges on the Neural Tangent Kernel approximation of Jacot et al. \n\nThis extremely technical paper is indimidating and I wonder how many readers will actually read the proofs. Moreover, one may wonder if the NTK approach precludes a deeper understanding of the actual performance of the network under consideration.  \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "lPCutTimFAL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the conditions for a deep fully-connected network to separate low-dimensional data classes. A binary classification setting is considered, where the two classes are modelled as two different manifolds. The manifolds are assumed to be one-dimensional for the ease of analysis. It is shown that the network depth should be sufficiently large so as to adapt to the geometrical properties of data (e.g. the manifold curvature); the network width should increase polynomially with the network depth; and the number of data samples should also scale at a polynomial rate with the network depth. The authors show that if these conditions are met, with high probability a randomly initialized network converges to a classifier that separates the two class manifolds. The proof technique relies on conditioning the network parameters of the l-th layer on the parameters of the previous layers using a Martingale model, which gives sharp concentration guarantees. \n\nThe paper is rigorous and well-written, and very detailed proofs have been provided for the presented results. The authors may still find it useful to address the following issues:\n\n1. The separation Delta between the two manifolds seems to be a critical parameter in the proposed performance bounds. However, the definition of Delta in Section 3.1 is not very clear. In particular, what does the notation \\angle(x, x') mean? Does this mean Euclidean distance, or angle as the notation suggests? If this means an angle, please explain whether this refers to the angle of points on the sphere, or the angle measured on the tangent space of the Riemannian manifold, etc.\n\n2. Although I like very much the analysis proposed by the authors, some of the main assumptions regarding the studied setting may be too restrictive in practice: \n\na. The proposed bounds apply only when the separation parameter Delta is positive. In a practical application, the manifolds representing different classes may be (asymptotically) intersecting due to the degeneracy of image transformations/articulations in some manifold regions - e.g. extreme illumination conditions, scale changes, etc.\n\nb. The assumption that the data samples lie exactly on a Riemannian manifold also seems to be restrictive. A probabilistic model for the data samples, e.g. involving a noise model capturing the deviations of the points from the manifold, would have been more realistic. Would the theoretical analysis be completely intractable under such extensions, or could that be a potential future direction of the study? \n\nIt would be nice if the authors could comment on such issues. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_qhIZwuRi5B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyzes deep networks in terms of multiple manifold problems. However, it needs easier explanations for a wider audience.  Moreover, I think the article was written as a book rather than a conference paper, considering 245 pages. \n\nWhat would be the actual benefits of the analysis? \n\nThe paper talks about one-dimensional cases, and I was wondering if the multiple manifold problem in one-dimension is enough. If I understand correctly, this one-dimension manifold is supposed to represent one class in the last layer. \n\nSome notations are hard to follow, especially in Section 3.1. Actually, some of them are used without the description.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "6XjRdhzEYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider a binary classification task. As a model the authors use a deep fully-connected neural network and train it to separate the submanifolds, representing different classes. They assume that sub-manifolds belong to the unit sphere. Also, the authors restrict their analysis to a one-dimensional case. The main claim is that by increasing depth we can improve model generalization of a network, trained by SGD.\n\nThe proved result seems to be imporant for understanding the generalization ability of deep neural networks. While proving it the authors used various imporant tools from martingale concentration approach.\n\nHowever, the paper is not well suited for such venue as ICLR. Moreover, I am sure that the proof itself contains some parts that are of separate interest for the community. I would propose to divide the paper into several parts, so that some of them contain general results about approaches to prove concentration inequalities, which are of general interest and can be used somewhere else. Also, I would expect more comments on which tools, used for proving the main results, are completely new and how they can be used to establish similar results for other network architectures. \nAlso, I would expect some comments how in principle the restriction that the submanifolds belong to the unit sphere can be removed. Some more discussion how the proposed setup is related to the setup, considered in the paper of Goldt (2019), are needed.\n\n========\n\nAfter reading authors comments.\n\nIn principle, I tried to understand the main steps of the proof, they looks OK. Although, I can not verify the details of the proof.\nI still think that ICLR venue is not OK for such long submissions. \nIn principle, I am OK to increase the grade for one point.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the multiple manifold problem - in a simple setting there are two data manifolds representing the positive and negative samples, and the goal is to train a neural network (or any predictor) that separates these two manifolds. The paper showed that this is possible to do with a deep neural network under certain assumptions - notably on the shape of the manifold and also on the ability of the neural network to represent certain functions (which is harder to verify, and only verified for a 1-d case in the paper). The optimization of neural network falls in the NTK regime but requires new techniques. Overall the question seems very natural and the results are reasonable first steps. There are some concerns about clarity that the authors should address in the paper.
Decision: Accept (Poster)