A Mean Field Theory of Batch Normalization

1 INTRODUCTION Deep neural networks have been enormously successful across a broad range of disciplines . These successes are often driven by architectural innovations . For example , the combination of convolutions ( LeCun et al. , 1990 ) , residual connections ( He et al. , 2015 ) , and batch normalization ( Ioffe & Szegedy , 2015 ) has allowed for the training of very deep networks and these components have become essential parts of models in vision ( Zoph et al . ) , language ( Chen & Wu , 2017 ) , and reinforcement learning ( Silver et al. , 2017 ) . However , a fundamental problem that has accompanied this rapid progress is a lack of theoretical clarity . An important consequence of this gap between theory and experiment is that two important issues become conflated . In particular , it is generally unclear whether novel neural network components improve generalization or whether they merely increase the fraction of hyperparameter configurations where good generalization can be achieved . Resolving this confusion has the promise of allowing researchers to more effectively and deliberately design neural networks . Recently , progress has been made ( Poole et al. , 2016 ; Schoenholz et al. , 2016 ; Daniely et al. , 2016 ; Pennington et al. , 2017 ; Hanin & Rolnick , 2018 ; Yang , 2019 ) in this direction by considering neural networks at initialization , before any training has occurred . In this case , the parameters of the network are random variables which induces a distribution of the activations of the network as well as the gradients . Studying these distributions is equivalent to understanding the prior over functions that these random neural networks compute . Picking hyperparameters that correspond to well-conditioned priors ensures that the neural network will be trainable and this fact has been extensively verified experimentally . However , to fulfill its promise of making neural network design less of a black box , these techniques must be applied to neural network architectures that are used in practice . Over the past year , this gap has closed significantly and theory for networks with skip connections ( Yang & Schoenholz , 2017 ; 2018 ) , convolutional networks ( Xiao et al. , 2018 ) , and gated ∗Please see https : //arxiv.org/abs/1902.08129 for the full and most current version of this paper recurrent networks ( Chen et al. , 2018 ; Gilboa et al. , 2019 ) have been developed . More recently , Yang ( 2019 ) devised a formalism that extends this approach to include an even wider range of architectures . Before state-of-the-art models can be analyzed in this framework , a slowly-decreasing number of architectural innovations must be studied . One particularly important component that has thus-far remained elusive is batch normalization . Our Contributions . In this paper , we develop a theory of fully-connected networks with batch normalization whose weights and biases are randomly distributed . A significant complication in the case of batch normalization ( compared to e.g . layer normalization or weight normalization ) is that the statistics of the network depend non-locally on the entire batch . Thus , our first main result is to recast the theory for random fully-connected networks so that it can be applied to batches of data . We then extend the theory to include batch normalization explicitly and validate this theory against Monte-Carlo simulations . We show that as in previous cases we can leverage our theory to predict valid hyperparameter configurations . In the process of our investigation , we identify a number of previously unknown properties of batch normalization that make training unstable . In particular , for most nonlinearities used in practice , batchnorm in a deep , randomly initialized network induces high degree of symmetry in the embeddings of the samples in the batch ( Thm 3.4 ) . Whenever this symmetry takes hold , we show that for any choice of nonlinearity , gradients of fully-connected networks with batch normalization explode exponentially in the depth of the network ( Thm 3.9 ) . This imposes strong limits on the maximum trainable depth of batch normalized networks . This limit can be lifted partially but not completely by pushing activation functions to be more linear at initialization . It might seem that such gradient explosion ought to lead to learning dynamics that are unfavorable . However , we show that networks with batch normalization causes the scale of the gradients to naturally equilibrate after a single step of gradient descent ( provided the initial gradients are not so large as to cause numerical instabilities ) . For shallower networks , this equilibrating effect is sufficient to allow adequate training . Finally , we note that there is a related vein of research that has emerged that leverages the prior over functions induced by random networks to perform exact Bayesian inference ( Lee et al. , 2017 ; de G. Matthews et al. , 2018 ; Novak et al. , 2019 ; Garriga-Alonso et al. , 2019 ; Yang , 2019 ) . One of the natural consequences of this work is that the prior for networks with batch normalization can be computed exactly in the wide network limit . As such , it is now possible to perform exact Bayesian inference in the case of wide neural networks with batch normalization . 2 RELATED WORK Batch normalization has rapidly become an essential part of the deep learning toolkit . Since then , a number of similar modifications have been proposed including layer normalization ( Ba et al. , 2016 ) and weight normalization ( Salimans & Kingma , 2016 ) . Comparisons of performance between these different schemes have been challenging and inconclusive ( Gitman & Ginsburg , 2017 ) . The original introduction of batchnorm in Ioffe & Szegedy ( 2015 ) proposed that batchnorm prevents “ internal covariate shift ” as an explanation for its effectiveness . Since then , several papers have approached batchnorm from a theoretical angle , especially following Ali Rahimi ’ s catalyzing call to action at NIPS 2017 . Balduzzi et al . ( 2017 ) found that batchnorm in resnets allow deep gradient signal propagation in contrast to the case without batchnorm . Santurkar et al . ( 2018 ) found that batchnorm does not help covariate shift but helps by smoothing loss landscape . Bjorck et al . ( 2018 ) reached the opposite conclusion as our paper for residual networks with batchnorm , that batchnorm works in this setting because it induces beneficial gradient dynamics and thus allows a much bigger learning rate . Luo et al . ( 2018 ) explores similar ideas that batchnorm allows large learning rates and likewise uses random matrix theory to support their claims . Kohler et al . ( 2018 ) identified situations in which batchnorm can provably induce acceleration in training . Of the above that mathematically analyze batchnorm , all but Santurkar et al . ( 2018 ) make simplifying assumptions on the form of batchnorm and typically do not have gradients flowing through the batch variance . Even Santurkar et al . ( 2018 ) only analyzes a vanilla network which gets added a single batchnorm at a single moment in training . Our analysis here on the other hand works for networks with arbitrarily many batchnorm layers with very general activation functions , and indeed , this deep stacking of batchnorm is precisely what leads to gradient explosion . It is an initialization time analysis for networks of infinite width , but we show experimentally that the resulting insights predict both training and test time behavior . We remark that Philipp & Carbonell ( 2018 ) has also empirically noted gradient explosion happens in deep batchnorm networks with various nonlinearities . In this work , in contrast , we develop a precise theoretical characterization of gradient statistics and , as a result , we are able to make significantly stronger conclusions . 3 THEORY We begin with a brief recapitulation of mean field theory in the fully-connected setting . In addition to recounting earlier results , we rephrase the formalism developed previously to compute statistics of neural networks over a batch of data . Later , we will extend the theory to include batch normalization . We consider a fully-connected network of depth L whose layers have width Nl , activation function1 φ , weights W l ∈ RNl−1×Nl , and biases bl ∈ RNl . Given a batch of B inputs2 { xi : xi ∈ RN0 } i=1 , ··· , B , the pre-activations of the network are defined by the recurrence relation , h1i = W 1xi + b 1 and hli = W lφ ( hl−1i ) + b l ∀ l > 1 . ( 1 ) At initialization , we choose the weights and biases to be i.i.d . as W lαβ ∼ N ( 0 , σ2w/Nl−1 ) and blα ∼ N ( 0 , σ2b ) . In the following , we use α , β , . . . for the neuron index , and i , j , . . . for the batch index . We will be concerned with understanding the statistics of the pre-activations and the gradients induced by the randomness in the weights and biases . For ease of exposition we will typically take the network to have constant width Nl = N . In the mean field approximation , we iteratively replace the pre-activations in Eq . ( 2 ) by Gaussian random variables with matching first and second moments . In the infinite width limit this approximation becomes exact ( Lee et al. , 2017 ; de G. Matthews et al. , 2018 ; Yang , 2019 ) . Since the weights are i.i.d . with zero mean it follows that the mean of each pre-activation is zero and the covariance between distinct neurons are zero . The pre-activation statistics are therefore given by ( hlα11 , · · · , hlαBB ) Nl−1→∞−−−−−−→ N ( 0 , Σlδα1···αB ) where Σl are B × B covariance matrices and δα1 , ···αB is the Kronecker-δ that is one if α1 = α2 = · · · = αB and zero otherwise . Definition 3.1 . Let V be the operator on functions φ such that Vφ ( Σ ) = E [ φ ( h ) φ ( h ) T : h ∼ N ( 0 , Σ ) ] computes the matrix of uncentered second moments of φ ( h ) for h ∼ N ( 0 , Σ ) . Using the above notation , we can express the covariance matrices by the recurrence relation , Σl = σ2wVφ ( Σ l−1 ) + σ2b11 T ( 2 ) At first Eq . ( 2 ) may seem challenging since the expectation involves a Gaussian integral in RB . However , each term in the expectation of Vφ involves at most a pair of pre-activations and so the expectation may be reduced to the evaluation of O ( B2 ) two-dimensional integrals . These integrals can either be performed analytically ( Cho & Saul , 2009 ; Williams , 1997 ) or efficiently approximated numerically ( Lee et al. , 2017 ) , and so Eq . ( 2 ) defines a computationally efficient method for computing the statistics of neural networks after random initialization . This theme of dimensionality reduction will play a prominent role in the forthcoming discussion on batch normalization . Eq . ( 2 ) defines a dynamical system over the space of covariance matrices . Studying the statistics of random feed-forward networks therefore amounts to investigating this dynamical system and is an enormous simplification compared with studying the pre-activations of the network directly . As is common in the dynamical systems literature , a significant amount of insight can be gained by investigating the behavior of Eq . ( 2 ) in the vicinity of its fixed points . For most common activation functions , Eq . ( 2 ) has a fixed point at some Σ∗ . Moreover , when the inputs are non-degenerate , this fixed point generally has a simple structure with Σ∗ = q∗ [ ( 1− c∗ ) I + c∗11T ] owing to permutation symmetry among elements of the batch . We refer to fixed points with such symmetry as BSB1 ( Batch Symmetry Breaking 1 or 1 Block Symmetry Breaking ) fixed points . As we will discuss later , in the context of batch normalization other fixed points with fewer symmetries ( “ BSBk fixed points ” ) 1The activation function may be layer dependent , but for ease of exposition we assume that it is not . 2Throughout the text , we assume that all elements of the batch are unique . may become preferred . In the fully-connected setting fixed points may efficiently be computed by solving the fixed point equation induced by Eq . ( 2 ) in the special case B = 2 . The structure of this fixed point implies that in asymptotically deep feed-forward neural networks all inputs yield pre-activations of identical norm with identical angle between them . Neural networks that are deep enough so that their pre-activation statistics lie in this regime have been shown to be untrainable ( Schoenholz et al. , 2016 ) . Notation As we often talk about matrices and also linear operators over matrices , we write T { Σ } for an operator T applied to a matrix Σ , and matrix multiplication is still written as juxtaposition . Composition of matrix operators are denoted with T1 ◦ T2 . Local Convergence to BSB1 Fixed Point . To understand the behavior of Eq . ( 2 ) near its BSB1 fixed point we can consider the Taylor series in the deviation from the fixed point , ∆Σl = Σl −Σ∗ . To lowest order we generically find , ∆Σl = J { ∆Σl−1 } ( 3 ) where J = dVφdΣ ∣∣ Σ=Σ∗ is the B2 ×B2 Jacobian of Vφ . In most prior work where φ was a pointwise non-linearity , Eq . ( 3 ) reduces to the case B = 2 which naturally gave rise to linearized dynamics in ql = E [ ( hli ) 2 ] and cl = E [ hlizlj ] /ql . However , in the case of batch normalization we will see that one must consider the evolution of Eq . ( 3 ) as a whole . This is qualitatively reminiscent of the case of convolutional networks studied in Xiao et al . ( 2018 ) where the evolution of the entire pixel × pixel covariance matrix had to be evaluated . The dynamics induced by Eq . ( 3 ) will be controlled by the eigenvalues of J : Suppose J has eigenvalues λi — ordered such that λ1 ≥ λ2 ≥ · · · ≥ λB2 — with associated eigen “ vectors ” ei ( note that the ei will themselves be B×B matrices ) . It follows that if ∆Σ0 = ∑ i ciei for some choice of constants ci then ∆Σl = ∑ i ciλ l iei . Thus , if λi < 1 for all i , ∆Σ l will approach zero exponentially and the fixed-point will be stable . The number of layers over which Σ will approach Σ∗ will be given by −1/ log ( λ1 ) . By contrast if λi > 1 for any i then the fixed point will be unstable . In this case , there is typically a different , stable , fixed point that must be identified . It follows that if the eigenvalues of J can be computed then the dynamics will follow immediately . While J may appear to be a complicated object at first , a moment ’ s thought shows that each diagonal element J { ∆Σ } ii is only affected by the corresponding diagonal element ∆Σii , and each off-diagonal element J { ∆Σ } ij is only affected by ∆Σii , ∆Σjj , and ∆Σij . Such a Diagonal-Offdiagonal Semidirect ( DOS ) operator has a simple eigendecomposition with two eigenspaces corresponding to changes in the off-diagonal and changes in the diagonal ( Thm E.72 ) . The associated eigenvalues are precisely those calculated by Schoenholz et al . ( 2016 ) in a simplified analysis . DOS operators are a particularly simple form of a more general operator possessing an abundance of symmetries called ultrasymmetric operators , which will play a prominent role in the analysis of batchnorm below . Gradient Dynamics . Similar arguments allow us to develop a theory for the statistics of gradients . The backpropogation algorithm gives an efficient method of propagating gradients from the end of the network to the earlier layers as , dL dW l = ∑ i δliφ ( h l−1 i ) T δlαi = φ ′ ( hlαi ) ∑ β W l+1βα δ l+1 βi . ( 4 ) Here L is the loss function and δli = ∇hliL are Nl-dimensional vectors that describe the error signal from neurons in the l ’ th layer due to the i ’ th element of the batch . The preceding discussion gave a precise characterization of the statistics of the hli that we can leverage to understand the statistics of δli . Assuming gradient independence , that is , that an iid set of weights are used during backpropagation ( see Appendix B for more discussions ) , it is easy to see that E [ δlαi ] = 0 and E [ δlαiδlβj ] = Πlij if α = β and 0 otherwise , where Πl is a covariance matrix and we may once again drop the neuron index . We can construct a recurrence relation to compute Πl , Πl = σ2wVφ′ ( Σ l ) Πl+1 . ( 5 ) Typically , we will be interested in understanding the dynamics of Πl when Σl has converged exponentially towards its fixed point . Thus , we study the approximation , Πl ≈ σ2wVφ′ ( Σ∗ ) Πl+1 . ( 6 ) Since these dynamics are linear ( and in fact componentwise ) , explosion and vanishing of gradients will be controlled by Vφ′ ( Σ∗ ) . 3.1 BATCH NORMALIZATION We now extend the mean field formalism to include batch normalization . Here , the definition for the neural network is modified to be the coupled equations , hli = W lφ ( h̃l−1i ) + b l h̃lαi = γα hlαi − µα σα + βα ( 7 ) where γα and βα are parameters , and µα = 1Nl ∑ i hαi and σ 2 α = √ 1 Nl ∑ i ( hαi − µα ) 2 + are the per-neuron batch statistics . In practice ≈ 10−5 or so to prevent division by zero , but in this paper , unless stated otherwise ( in the last few sections ) , is assumed to be 0 . Unlike in the case of vanilla fully-connected networks , here the pre-activations are invariant to σ2w and σ 2 b . Without a loss of generality , we therefore set σ2w = 1 and σ 2 b = 0 for the remainder of the text . In principal , batch normalization additionally yields a pair of hyperparameters γ and β which are set to be constants . However , these may be incorporated into the nonlinearity and so without a loss of generality we set γ = 1 and β = 0 . In order to avoid degenerate results , we assume B ≥ 4 unless stated otherwise ; we shall discuss the small B regime in Appendix J . If one treats batchnorm as a “ batchwise nonlinearity ” , then the arguments from the previous section can proceed identically and we conclude that as the width of the network grows , the pre-activations will be jointly Gaussian with identically distributed neurons . Thus , we arrive at an analogous expression to Eq . ( 2 ) , Σl = VBφ ( Σ l−1 ) where Bφ : RB → RB , Bφ ( h ) = φ ( √ BGh ||Gh|| ) . ( 8 ) Here we have introduced the projection operator G = I − 1B11T which is defined such that Gx = x − µ1 with µ = ∑i xi/B . Unlike φ , Bφ does not act component-wise on h. It is therefore not obvious whether VBφ can be evaluated without performing a B-dimensional Gaussian integral . Theoretical tools . In this paper , we present several ways to analyze high dimensional integrals like the above : 1. the Laplace method 2. the Fourier method 3. spherical integration 4. and the Gegenbauer method . The former two use the Laplace and Fourier transforms to simplify expressions like the above , where the Laplace method requires that φ be positive homogeneous . Often the Laplace method will give clean , closed form answers for such φ . Because batchnorm can be thought of as a linear projection ( G ) followed by projection to the sphere of radius √ B , spherical integration techniques are often very useful and in fact is typically the most straightforward way of numerically evaluating quantities . Lastly , the Gegenbauer method expresses objects in terms of the Gegenbauer coefficients of φ . Briefly , Gegenbauer polynomials { C ( α ) l ( x ) } ∞l=0 are orthogonal polynomials with respect to the measure ( 1 − x2 ) α− 12 on [ −1 , 1 ] . They are intimately related to spherical harmonics ( a natural basis for functions on a sphere ) , which explains their appearance in this context . The Gegenbauer method is the most illuminating amongst them all , and is what allows us to conclude that gradient explosion happens regardless of nonlinearity under general conditions . See Appendix D for a more in-depth discussion of these techniques . In what follows , we will mostly present results by the Laplace method for φ = relu and by the Gegenbauer method for general φ , but give pointers to the appendix for others . Back to the topic of Eq . ( 8 ) , we present a pair of results that expresses Eq . ( 8 ) in a more manageable form . From previous work ( Poole et al. , 2016 ) , Vφ can be expressed in terms of a two-dimensional Gaussian integrals independent of B . When φ is degree-α positive homogeneous ( e.g . rectified linear activations ) we can relate Vφ and VBφ by the Laplace transform . ( see Thm E.5 ) . Theorem 3.2 . Suppose φ : R→ R is degree-α positive homogeneous . For any positive semi-definite matrix Σ define the projection ΣG = GΣG . Then VBφ ( Σ ) = Bα Γ ( α ) ∫ ∞ 0 ds sα−1 Vφ ( Σ G ( I + 2sΣG ) −1 ) √ det ( I + 2sΣG ) . ( 9 ) whenever the integral exists . Using this parameterization , when Vφ has a closed form solution ( like ReLU ) , VBφ involves only a single integral . A similar expression can be derived for general φ by applying Fourier transform ; see Appendix E.2 . Next we express Eq . ( 8 ) as a spherical integral ( see Proposition E.34 ) Theorem 3.3 . Let e ∈ RB×B−1 have as columns a set of orthonormal basis vectors of { x ∈ RB : Gx = x } . Then with SB−2 ⊆ RB−1 denoting the ( B − 2 ) -dimensional sphere , VBφ ( Σ ) = E v∼SB−2 φ ( √ Bev ) ⊗2√ det eTΣe ( vT ( eTΣe ) −1v ) B−1 2 . ( 10 ) Together these theorems provide analytic recurrence relations for random neural networks with batch normalization over a wide range of activation functions . By analogy to the fully-connected case we would like to study the dynamical system over covariance matrices induced by these equations . We begin by investigating the fixed point structure of Eq . ( 8 ) . As in the case of feed-forward networks , permutation symmetry implies that there exist BSB1 fixed points Σ∗ = q∗ [ ( 1−c∗ ) I+c∗11T ] . We will see that this fixed point is in fact unique , and a clean expression of q∗ and c∗ can be obtained in terms of Gegenbauer basis ( see Thm F.13 ) . Theorem 3.4 ( Gegenbauer expansions of BSB1 fixed point ) . If φ ( √ B − 1x ) has Gegenbauer ex- pansion ∑∞ l=0 al 1 cB−1 , l C ( B−32 ) l ( x ) where cB−1 , l = B−3 B−3+2l , then q∗ = ∞∑ l=0 a2l 1 cB−1 , l C ( B−32 ) l ( 1 ) , q ∗c∗ = ∞∑ l=0 a2l 1 cB−1 , l C ( B−32 ) l ( −1 B − 1 ) . Thus the entries of the BSB1 fixed point are diagonal quadratic forms of the Gegenbauer coefficients of φ ( √ B − 1x ) . Even more concise closed forms are available when the activation functions are degree α positive homogeneous ( see Thm F.8 ) . In particular , for ReLU we arrive at the following Theorem 3.5 ( BSB1 fixed point for ReLU ) . When φ = relu , then q∗ = 1 2 , c∗ = J1 ( −1 B − 1 ) = 1 π − 1 2 ( B − 1 ) +O ( 1 ( B − 1 ) 2 ) ( 11 ) where J1 ( c ) = 1π ( √ 1− c2 + ( π − arccos ( c ) ) c ) is the arccosine kernel ( Cho & Saul , 2009 ) . In the appendix , Thm F.5 also describes a trigonometric integral formula for general nonlinearities . In the presence of batch normalization , when the activation function grows quickly , a winner-takeall phenomenon can occur where a subset of samples in the batch have much bigger activations than others . This causes the covariance matrix to form blocks of differing magnitude , breaking the BSB1 symmetry . One notices this , for example , as the degree α of α-ReLU ( i.e . αth power of ReLU ) increases past a point αtransition ( B ) depending on the batch size B ( see Fig . 1 ) . However , we observe , through simulations , that by far most of the nonlinearities used in practice , like ReLU , leaky ReLU , tanh , sigmoid , etc , all lead to BSB1 fixed points , and we can prove this rigorously for φ = id ( see Corollary F.3 ) . We discuss symmetry-broken fixed points ( BSB2 fixed points ) in Appendices J and K , but in the main text , from here on , Assumption 1 . Unless stated otherwise , we assume that any nonlinearity φ mentioned induces Σl to converge to a BSB1 fixed point under the dynamics of Eq . ( 8 ) . 3.1.1 LINEARIZED DYNAMICS With the fixed point structure for batch normalized networks having been described , we now investigate the linearized dynamics of Eq . ( 8 ) in the vicinity of these fixed points . To determine the eigenvalues of dVBφ dΣ ∣∣ Σ=Σ∗ it is helpful to consider the action of batch normalization in more detail . In particular , we notice that Bφ can be decomposed into the composition of three separate operations , Bφ = φ ◦ n ◦G . As discussed above , Gh subtracts the mean from h and we introduce the new function n ( h ) = √ Bh/||h|| which normalizes a centered h by its standard deviation . Applying the chain rule , we can rewrite the Jacobian as , dVBφ ( Σ ) dΣ = dV [ φ◦n ] ( Σ G ) dΣG ◦G⊗2 ( 12 ) where ◦ denotes composition andG⊗2 is the natural extension ofG to act on matrices asG⊗2 { Σ } = GΣG = ΣG . It ends up being advantageous to study G⊗2 ◦ dV [ φ◦n ] dΣ ∣∣∣ Σ=Σ∗ ◦G⊗2 = : G⊗2 ◦ Ĵ ◦G⊗2 and to note that the nonzero eigenvalues of this object are identical to the nonzero eigenvalues of the Jacobian ( see Lemma F.17 ) . At face value , this is a complicated object since it simultaneously has large dimension and possesses an intricate block structure . However , the permutation symmetry of the BSB1 Σ∗ induces strong symmetries in Ĵ that significantly simplify the analysis ( see Appendix F.3 ) . In particular while Ĵijkl is a four-index object , we have Ĵijkl = Ĵπ ( i ) π ( j ) π ( k ) π ( l ) for all permutations π on B and Ĵijkl = Ĵjilk . We call linear operators possessing such symmetries ultrasymmetric ( Defn E.53 ) and show that all ultrasymmetric operators conjugated by G⊗2 admit an eigendecomposition that contains three distinct eigenspaces with associated eigenvalues ( see Thm E.62 ) . Theorem 3.6 . Let T be an ultrasymmetric matrix operator . Then on the space of symmetric matrices , G⊗2 ◦ T ◦G⊗2 has the following orthogonal ( under trace inner product ) eigendecomposition , 1. an eigenspace { Σ : ΣG = 0 } with eigenvalue 0 . 2. a 1-dimensional eigenspace RG with eigenvalue λG , TG . 3. a ( B − 1 ) -dimensional eigenspace L : = { DG : D diagonal , trD = 0 } , with eigenvalue λG , TL . 4. a B ( B−3 ) 2 -dimensional eigenspace M : = { Σ : ΣG = Σ , DiagΣ = 0 } , with eigenvalue λG , TM . The specific forms of the eigenvalues can be obtained as linear functions of the entries of T ; see Thm E.62 for details . Note that , as the eigenspaces are orthogonal , this implies that G⊗2 ◦ T ◦G⊗2 is self-adjoint ( even when T is not ) . In our context with T = Ĵ , the eigenspaces can be roughly interpreted as follows : The deviation ∆Σ = Σ − Σ∗ from the fixed point decomposes as a linear combination of components in each of the eigenspaces . The RG-component captures the average norm of elements of the batch ( the trace of ∆Σ ) , the L-component captures the fluctuation of such norms , and the M-component captures the covariances between elements of the batch . Because of the explicit normalization of batchnorm , one sees immediately that the RG-component goes to 0 after 1 step . For positive homogeneous φ , we can use the Laplace method to obtain closed form expressions for the other eigenvalues ( see Thm F.33 ) . The below theorem shows that , as the batch size becomes larger , a deep ReLU-batchnorm network takes more layers to converge to a BSB1 fixed point . Theorem 3.7 . Let φ = relu and B > 3 . The eigenvalues of G⊗2 ◦ Ĵ ◦G⊗2 for L and M are λ↑L = 1 2 ( B + 1 ) µ∗ ( ( B − 2 ) [ 1− J1 ( −1 B − 1 ) ] + B B − 1J ′ 1 ( −1 B − 1 ) ) ↗ 1 ( 13 ) λ↑M = B 2 ( B + 1 ) µ∗ J′1 ( −1 B − 1 ) ↗ π 2 ( π − 1 ) ≈ 0.733 ( 14 ) where µ∗ = q∗ ( 1− c∗ ) = 12 ( 1− J1 ( −1 B−1 ) ) , and↗ denotes increasing limit as B →∞ . More generally , we can evaluate them for general nonlinearity using spherical integration ( Appendix F.3.1 ) and , more enlightening , using the Gegenbauer method , is the following Theorem 3.8 . If φ ( √ B − 1x ) has Gegenbauer expansion∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) , then λ↑L = ∑∞ l=0 a 2 lwB−1 , l + alal+2uB−1 , l∑∞ l=0 a 2 l vB−1 , l , λ↑M = ∑∞ l=0 a 2 l w̃B−1 , l + alal+2ũB−1 , l∑∞ l=0 a 2 l vB−1 , l ( 15 ) where the coefficients wB−1 , l , uB−1 , l , w̃B−1 , l , ũB−1 , l , vB−1 , l are given in Thms F.22 and F.24 . A BSB1 fixed point is not locally attracting if λ↑L > 1 or λ ↑ M > 1 . Thus Thm 3.8 yields insight on the stability of the BSB1 fixed point , which we can interpret heuristically as follows . The specific forms of the coefficients wB−1 , l , uB−1 , l , w̃B−1 , l , ũB−1 , l , vB−1 , l show that λ ↑ M is typically much smaller than λ↑L ( but there are exceptions like φ = sin ) , and wB−1,1 < vB−1,1 but wB−1 , l ≥ vB−1 , l for all l ≥ 2 . Thus one expects that , the larger a1 is , i.e . the “ more linear ” and less explosive φ is , the smaller λ↑L is and the more likely that Eq . ( 8 ) converges to a BSB1 fixed point . This is consistent with the “ winner-take-all ” intuition for the emergence of BSB2 fixed point explained above . See Appendix F.3.2 for more discussion . 3.1.2 GRADIENT BACKPROPAGATION With a mean field theory of the pre-activations of feed-forward networks with batch normalization having been developed , we turn our attention to the backpropagation of gradients . In contrast to the case of networks without batch normalization , we will see that exploding gradients at initialization are a severe problem here . To this end , one of the main results from this section will be to show that fully-connected networks with batch normalization feature exploding gradients for any choice of nonlinearity such that Σl converges to a BSB1 fixed point . Below , by rate of gradient explosion we mean the β such that the gradient norm squared grows as βL+o ( L ) with depth L. As before , all computations below assumes gradient independence ( see Appendix B for a discussion ) . As a starting point we seek an analog of Eq . ( 6 ) in the case of batch normalization . However , because the activation functions no longer act point-wise on the pre-activations , the backpropagation equation becomes , δlαi = ∑ βj ∂Bφ ( hlα ) j ∂hlαi W l+1βα δ l+1 βj ( 16 ) where hlα = ( h l α1 , . . . , h l αB ) and we observe the additional sum over the batch . Computing the resulting covariance matrix Πl , we arrive at the recurrence relation , Πl = E [ ( dBφ ( h ) dh ) T Πl+1 dBφ ( h ) dh : h ∼ N ( 0 , Σl ) ] = : VB′φ ( Σ l ) † { Πl+1 } ( 17 ) where we have defined the linear operator VF ( Σ ) † { · } such that VF ( Σ ) † { Π } = E [ FTh ΠFh : h ∼ N ( 0 , Σ ) ] for any vector-indexed linear operator Fh . As in the case of vanilla feed-forward networks , here we will be concerned with the behavior of gradients when Σl is close to its fixed point . We therefore study the asymptotic approximation to Eq . ( 17 ) given by Πl = VB′φ ( Σ ∗ ) † { Πl+1 } . In this case the dynamics of Π are linear and are therefore naturally determined by the eigenvalues of VB′φ ( Σ ∗ ) † . As in the forward case , batch normalization is the composition of three operations Bφ = φ ◦ n ◦G . Applying the chain rule , Eq . ( 17 ) can be rewritten as , VB′φ ( Σ ) † = G⊗2 ◦ E ( d ( φ ◦ n ) ( z ) dz ∣∣∣∣T z=Gh ) ⊗2 : h ∼ N ( 0 , Σ ) = : G⊗2 ◦ F ( Σ ) ( 18 ) with F ( Σ ) appropriately defined . Note that since G⊗2 is an idempotent operator , ( VB′φ ( Σ ) † ) n = ( G⊗2 ◦ F ( Σ ) ) n = ( G⊗2 ◦ F ( Σ ) ◦G⊗2 ) n−1 ◦G⊗2 ◦ F ( Σ ) , so that it suffices to study the eigendecomposition of G⊗2 ◦ F ( Σ∗ ) ◦G⊗2 . Due to the symmetry of Σ∗ , F ( Σ∗ ) is ultrasymmetric , so that G⊗2 ◦ F ( Σ∗ ) ◦ G⊗2 has eigenspaces RG , L , M and we can compute its eigenvalues via Thm 3.6 . More illuminating , however , is the Gegenbauer expansion ( see Thm G.5 ) . It requires a new identity Thm E.47 involving Gegenbauer polynomials integrated over a sphere , which may be of independent interest . Theorem 3.9 ( Batchnorm causes gradient explosion ) . Suppose φ ( √ B − 1x ) , φ′ ( √ B − 1x ) ∈ L2 ( ( 1− x2 ) B−32 ) . If φ ( √ B − 1x ) has Gegenbauer expansion∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) , then gradients explode at the rate of λ↓G = ∑∞ l=0 ( l+B−3 B−3 · l ) a2l rl∑∞ l=0 a 2 l rl where rl = c−1B−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) > 0 . Consequently , for any non-constant φ ( i.e . there is a j > 0 such that aj 6= 0 ) , λ↓G > 1 ; φ minimizes λ↓G iff it is linear ( i.e . ai = 0 , ∀i ≥ 2 ) , in which case gradients explode at the rate of B−2B−3 . This contrasts starkly with the case of non-normalized fully-connected networks , which can use the weight and bias variances to control its mean field network dynamics ( Poole et al. , 2016 ; Schoenholz et al. , 2016 ) . As a corollary , we disprove the conjecture of the original batchnorm paper ( Ioffe & Szegedy , 2015 ) that “ Batch Normalization may lead the layer Jacobians to have singular values close to 1 ” in the initialization setting , and in fact prove the exact opposite , that batchnorm forces the layer Jacobian singular values away from 1 . Appendix G.1 discusses the numerical evaluation of all eigenvalues , and as usual , the Laplace method yields closed forms for positive homogeneous φ ( Thm G.12 ) . We highlight the result for ReLU . Theorem 3.10 . In a ReLU-batchnorm network , the gradient norm explodes exponentially at the rate of 1 B − 3 B − 1 + J′1 ( −1 B−1 ) 1− J1 ( −1 B−1 ) − 1 ( 19 ) which decreases to ππ−1 ≈ 1.467 as B → ∞ . In contrast , for a linear batchnorm network , the gradient norm explodes exponentially at the rate of B−2B−3 , which goes to 1 as B →∞ . Fig . 1 shows theory and simulation for ReLU gradient dynamics . Weight Gradient While all of the above only study the gradient with respect to the hidden preactivations , Appendix L shows that the weight gradient norms at layer l is just 〈Πl , µ∗G〉 = µ∗ tr Πl , and thus by Thm 3.9 , the weight gradients explode as well at the same rate λ↓G . Effect of as a hyperparameter In practice , is usually treated as small constant and is not regarded as a hyperparameter to be tuned . Nevertheless , we can investigate its effect on gradient explosion . A straightforward generalization of the analysis presented above to the case of > 0 suggests somewhat larger values than typically used can ameliorate ( but not eliminate ) gradient explosion problems . See Fig . 5 ( c , d ) . 3.2 CROSS-BATCH DYNAMICS Forward In addition to analyzing the correlation between preactivations of samples in a batch , we also study the correlation between those of different batches . The dynamics Eq . ( 8 ) can be generalized to simultaneous propagation of k batches ( see Eq . ( 54 ) and Appendix H ) . Σ̃l = VB⊕kφ ( Σ̃l−1 ) = E ( h1 , ... , hk ) ∼N ( 0 , Σ̃l−1 ) ( Bφ ( h1 ) , . . . , Bφ ( hk ) ) ⊗2 , with Σ̃l ∈ RkB×kB . ( 20 ) Here the domain of the dynamics is the space of block matrices , with diagonal blocks and offdiagonal blocks resp . representing within-batch and cross-batch covariance . We observe empirically Published as a conference paper at ICLR 2019 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1.6 3.1 4.6 6.1 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 0.12 0.18 0.25 0.32 0.38 0.45 0.51 0 20 40 60 80 100 layer 0 10 20 30 40 50 60 lo gn or m batch 8 16 32 64 128 20 40 60 80 100 batchsLze 0.8 1.0 1.2 1.4 1.6 1.8 eL ge nv al ue s G L 0 Var ( aa ) Var ( ab ) transition 0.5 1.0 1.5 2.0 10-4 0.001 0.010 0.100 1 10a b c d 0.1 0.5 1.0 6.0 Figure 1 : Numerical confirmation of theoretical predictions . ( a , b ) Comparison between theoretical prediction ( dashed lines ) and Monte Carlo simulations ( solid lines ) for the eigenvalues of the backwards Jacobian ( see Thms 3.10 and G.12 ) as a function of batch size and the magnitude of gradients as a function of depth respectively for rectified linear networks . In each case Monte Carlo simulations are averaged over 200 sample networks of width 1000 and shaded regions denote 1 standard deviation . Dashed lines are shifted slightly for easier comparison . ( c , d ) Demonstration of the existence of a BSB1 to BSB2 symmetry breaking transition as a function of α for α-ReLU ( i.e . the αth power of ReLU ) activations . In ( c ) we plot the empirical variance of the diagonal and off-diagonal entries of the covariance matrix which clearly shows a jump at the transition . In ( d ) we plot representative covariance matrices for the two phases ( BSB1 bottom , BSB2 top ) .Published as a conference paper at ICLR 2019 ( a ) ( b ) Figure A.1 : Batch norm leads to a chaotic input-output map with increasing depth . A linear network with batch norm is shown acting on two minibatches of size 64 after random orthogonal initialization . The datapoints in the minibatch are chosen to form a 2d circle in input space , except for one datapoint that is perturbed separately in each minibatch ( leftmost datapoint at input layer 0 ) . Because the network is linear , for a given minibatch it performs an affine transformation on its inputs – a circle in input space remains an ellipse throughout the network . However , due to batch norm the coefficients of that affine transformation change nonlinearly as the datapoints in the minibatch are changed . ( a ) Each pane shows a scatterplot of activations at a given layer for all datapoints in the minibatch , projected onto the top two PCA directions . PCA directions are computed using the concatenation of the two minibatches . Due to the batch norm nonlinearity , minibatches that are nearly identical in input space grow increasingly dissimilar with depth . Intuitively , this chaotic input-output map can be understood as the source of exploding gradients when batch norm is applied to very deep networks , since very small changes in an input correspond to very large movements in network outputs . ( b ) The correlation between the two minibatches , as a function of layer , for the same network . Despite having a correlation near one at the input layer , the two minibatches rapidly decorrelate with depth . See ? ? for a theoretical treatment . A.1 VGG19 WITH BATCHNORM ON CIFAR100 Even though at initialization time batchnorm causes gradient explosion , after the first few epochs , the relative gradient norms kr✓Lk/k✓k for weight parameters ✓ = W or BN scale parameter ✓ = , equilibrate to about the same magnitude . See Fig . A.2 . Published as a conference paper at ICLR 2019 ( a ) ( b ) Figure A.1 : Batch norm leads to a chaotic input-output map with increasing depth . A linear network with batch norm is shown acting on two minibatches of size 64 after random orthogonal initialization . The datapoints in the minibatch are chosen to form a 2d circle in input space , except for one datapoint that is perturbed separately in each minibatch ( leftmost datapoint at input layer 0 ) . Because the network is linear , for a given minibatch it performs an affine transformation on its inputs – a circle in input space remains an ellipse throughout the network . However , due to batch norm the coefficients of that affine transformation change nonlinearly as the datapoints in the minibatch are changed . ( a ) Each pane shows a scatterplot of activations at a given layer for all datapoints in the minibatch , projected onto the top two PCA directions . PCA directions are computed using the concatenation of the two minibatches . Due to the batch norm nonlinearity , minibatches that are nearly identical in input space grow increasingly dissimilar with depth . Intuitively , this chaotic input-output map can be understood as the source of exploding gradients when batch norm is applied to very deep networks , since very small changes in an input correspond to very large movements in network outputs . ( b ) The correlation between the two minibatches , as a function of layer , for the same network . Despite having a correlation near one at the input layer , the two minibatches rapidly decorrelate with depth . See ? ? for a theoretical treatment . A.1 VGG19 WITH BATCHNORM ON CIFAR100 Even though at initialization time batchnorm causes gradient explosion , after the first few epochs , the relative gradient norms kr✓Lk/k✓k for weight parameters ✓ = W or BN scale parameter ✓ = , equilibrate to about the same magnitude . See Fig . A.2 . 14 ( a ) ( b ) Figure A.1 : Batch norm leads to a chaotic input-output map with increasing depth . A linear network with batch norm is shown acting on two minibatches of size 64 after random orthogonal initialization . The datapoints in the minibatch are chosen to form a 2d circle in input space , except for one datapoint that is perturbed separately in each minibatch ( leftmost datapoint at input layer 0 ) . Because the network is linear , for a given minibatch it performs an affine transformation on its inputs – a circle in input space remains an ellipse throughout the network . However , due to batch norm the coefficients of that affine transformation change nonlinearly as the datapoints in the minibatch are changed . ( a ) Each pane shows a scatterplot of activations at a given layer for all datapoints in the minibatch , projected onto the top two PCA directions . PCA directions are computed using the conc tenation of the two minibatches . Due to the batch norm nonlinearity , minibatches that are nearly identical in input space grow increasingly dissimilar with depth . Intuitively , this chaotic input-output map can be understood as the source of exploding gradients when batch norm is applied to very deep networks , since very small changes in an input correspond to very large movements in network outputs . ( b ) The correlation between the two minibatches , as a function of layer , for the same network . Despite havi g a correlation near one at the input layer , the two minibatches rapidly decorrelate with depth . See ? ? for a theoretical treatment . A.1 VGG19 WITH BATCHNORM ON CIFAR100 Even though at initialization time batchnorm causes gradient explosion , after the first few epochs , the relative gradient norms kr✓Lk/k✓k for weight parameters ✓ = W or BN scale parameter ✓ = , equilibrate to about the same magnitude . See Fig . A.2 . 14 a b that for most nonlinearities used in practice like ReLU , or even for the identity function , ie no pointwise nonlinearity , the global fixed point of this dynamics is cross-batch BSB1 , with diagonal BSB1 blocks , and off-diagon l e tr es ll equal to the same constant c∗CB : Σ̃∗ = Σ∗ c∗CB c ∗ CB · · · c∗CB Σ ∗ c∗CB · · · c∗CB c ∗ CB Σ ∗ · · · ... ... ... . . . . ( 21 ) 10 To interpret this phenomenon , note that , after mean centering of the preactivations , this covariance matrix becomes a multiple of identity . Thus , deep embedding of batches loses the mutual information between them in the input space . Qualitatively , this implies that two batches that are similar at the input to the network will become increasingly dissimilar — i.e . chaotic — as the signal propagates deep into the network . This loss in fact happens exponentially fast , as illustrated in Fig . 2 , and as shown theoretically by Thm H.13 , at a rate of λ↑ M̃ < 1 . Formally , the linearized dynamics of Eq . ( 20 ) has an eigenspace M̃ given by all block matrices with zero diagonal blocks . The associated eigenvalue is λ↑ M̃ . We shall return to λ↓ M̃ shortly and give its Gegenbauer expansion ( Thm 3.11 ) . Backward The gradients of two batches of input are correlated throughout the course of backpropagation . We hence also study the generalization of Eq . ( 17 ) to the k-batch setting ( see Appendix I ) : Π̃l = VB⊕2φ ′ ( Σ̃l ) † { Π̃l+1 } ( 22 ) VB⊕2φ ′ ( Σ̃∗ ) † { ( Π1 Ξ ΞT Π2 ) } = E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ ( B′φ ( x ) TΠ1B′φ ( x ) B′φ ( x ) TΞB′φ ( y ) B′φ ( y ) TΞTB′φ ( x ) B′φ ( y ) TΠ2B′φ ( y ) ) ] ( 23 ) where for simplicity of exposition and without loss of generality , we take k = 2 . Like in Eq . ( 17 ) , we have assumed that Σ̃l has converged to its limit Σ̃∗ . It is not hard to see that each block of Π̃l evolves independently , with the diagonal blocks in particular evolving according to Eq . ( 17 ) . Analyzing the off-diagonal blocks might seem unwieldy at first , but it turns out that its dynamics is given simply by scalar multiplication by a constant λ↓ M̃ ( Thm I.4 ) . Even more surprisingly , Theorem 3.11 ( Batchnorm causes information loss ) . The convergence rate of cross-batch preactivation covariance λ↑ M̃ is equal to the decay rate of the cross-batch gradient covariance λ↓ M̃ . If φ ( √ B − 1x ) has Gegenbauer expansion∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) , then both are given by λ↑ M̃ = λ↓ M̃ = 1 2πB ( B − 1 ) a21 Beta ( B 2 , 1 2 ) 2∑∞ l=0 a 2 l 1 cB−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) . ( 24 ) This quantity is always < 1 , and for any fixed B , is maximized by φ = id . Furthermore , for φ = id , λ↑ M̃ = λ↓ M̃ = B−12 P ( B 2 , 1 2 ) −2 and increase to 1 from below as B →∞ . Thus , a deep batchnorm network loses correlation information between two input batches , exponentially fast in depth , no matter what nonlinearity ( that induces fixed point of the form given by Eq . ( 21 ) ) . This again contrasts with the case for vanilla networks which can control this rate of information loss by tweaking the initialization variances for weights and biases Poole et al . ( 2016 ) ; Schoenholz et al . ( 2016 ) . The absence of coordinatewise nonlinearity , i.e . φ = id , maximally suppresses both this loss of information as well as the gradient explosion , and in the ideal , infinite batch scenario , can cure both problems . 4 EXPERIMENTS Having developed a theory for neural networks with batch normalization at initialization , we now explore the relationship between the properties of these random networks and their learning dynamics . We will see that the trainability of networks with batch normalization is controlled by gradient explosion . We quantify the depth scale over which gradients explode by ξ = 1/ log λ↓G where , as above , λ↓G is the largest eigenvalue of the jacobian . Across many different experiments we will see strong agreement between ξ and the maximum trainable depth . We first investigate the relationship between trainability and initialization for rectified linear networks as a function of batch size . The results of these experiments are shown in Fig . 3 where in each case we plot the test accuracy after training as a function of the depth and the batch size and overlay 16ξ in white dashed lines . In Fig . 3 ( a ) we consider networks trained using SGD on MNIST where we observe that networks deeper than about 50 layers are untrainable regardless of batch size . In ( b ) we compare standard batch normalization with a modified version in which the batch size is a b c d 0.1 1.0 0.1 0.55 Figure 3 : Batch normalization strongly limits the maximum trainable depth . Colors show test accuracy for rectified linear networks with batch normalization and γ = 1 , β = 0 , = 10−3 , N = 384 , and η = 10−5B . ( a ) trained on MNIST for 10 epochs ( b ) trained with fixed batch size 1000 and batch statistics computed over sub batches of size B . ( c ) trained using RMSProp . ( d ) Trained on CIFAR10 for 50 epochs . In each case , White dashed line indicates a theoretical prediction of trainable depth , as discussed in the text . t < latexit sha1_base64= '' USoNv0fCIvi1lLEl3uP/p3RZN/c= '' > AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w= < /latexit > < latexit sha1_base64= '' USoNv0fCIvi1lLEl3uP/p3RZN/c= '' > AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w= < /latexit > < latexit sha1_base64= '' USoNv0fCIvi1lLEl3uP/p3RZN/c= '' > AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w= < /latexit > < latexit sha1_base64= '' USoNv0fCIvi1lLEl3uP/p3RZN/c= '' > AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w= < /latexit > a b c 0 2 4 6 10 Figure 4 : Gradients in networks with batch normalization quickly achieve dynamical equilibrium . Plots of the relative magnitudes of ( a ) the weights ( b ) the gradients of the loss with respect to the pre-activations and ( c ) the gradients of the loss with respect to the weights for rectified linear networks of varying depths during the first 10 steps of training . Colors show step number from 0 ( black ) to 10 ( green ) . held fixed but batch statistics are computed over subsets of size B . This removes subtle gradient fluctuation effects noted in Smith & Le ( 2018 ) . In ( c ) we do the same experiment with RMSProp and in ( d ) we train the networks on CIFAR10 . In all cases we observe a nearly identical trainable region . It is counter intuitive that training can occur at intermediate depths , from 10 to 50 layers , where there is significant gradient explosion . To gain insight into the behavior of the network during learning we record the magnitudes of the weights , the gradients with respect to the pre-activations , and the gradients with respect to the weights for the first 10 steps of training for networks of different depths . The result of this experiment is shown in Fig . 4 . Here we see that before learning , as expected , the norm of the weights is constant and independent of layer while the gradients feature exponential explosion . However , we observe that two related phenomena occur after a single step of learning : the weights grow exponentially in the depth and the magnitude of the gradients are stable up to some threshold after which they vanish exponentially in the depth . This is as the result of the scaling property of batchnorm , where B′φ ( αh ) = α−1B′φ ( h ) : The first-step gradients dominate the weights due to gradient explosion , hence the exponential growth in weight norms , and thereafter , the gradients are scaled down commensurately . Thus , it seems that although the gradients of batch normalized networks at initialization are ill-conditioned , the gradients appear to quickly reach a stable dynamical equilibrium . While this appears to be beneficial for shallower networks , in deeper ones , the relative gradient vanishing can in fact be so severe as to cause lower layers to mostly stay constant during training . Aside from numerical issues , this seems to be the primary mechanism through which gradient explosion causes training problems for networks deeper than 50 layers . As discussed in the theoretical exposition above , batch normalization necessarily features exploding gradients for any nonlinearity that converges to a BSB1 fixed point . We performed a number of experiments exploring different ways of ameliorating this gradient explosion . These experiments a b c d 0.1 1.0 Figure 5 : Three techniques for counteracting gradient explosion . Test accuracy on MNIST as a function of different hyperparameters along with theoretical predictions ( white dashed line ) for the maximum trainable depth . ( a ) tanh network changing the overall scale of the pre-activations , here γ → 0 corresponds to the linear regime . ( b ) Rectified linear network changing the mean of the preactivations , here β → ∞ corresponds to the linear regime . ( c , d ) tanh and rectified linear networks respectively as a function of , here we observe a well defined phase transition near ∼ 1 . Note that in the case of rectified linear activations we use β = 2 so that the function is locally linear about 0 . We also find initializing β and/or setting > 0 having positive effect on VGG19 with batchnorm . See Figs . 8 and 9. are shown in Fig . 5 with theoretical predictions for the maximum trainable depth overlaid ; in all cases we see exceptional agreement . In Fig . 5 ( a , b ) we explore two different ways of tuning the degree to which activation functions in a network are nonlinear . In Fig . 5 ( a ) we tune γ ∈ [ 0 , 2 ] for networks with tanh-activations and note that in the γ → 0 limit the function is linear . In Fig . 5 ( b ) we tune β ∈ [ 0 , 2 ] for networks with rectified linear activations and we note , similarly , that in the β → ∞ limit the function is linear . As expected , we see the maximum trainable depth increase significantly with decreasing γ and increasing β . In Fig . 5 ( c , d ) we vary for tanh and rectified linear networks respectively . In both cases , we observe a critical point at large where gradients do not explode and very deep networks are trainable . 5 CONCLUSION In this work we have presented a theory for neural networks with batch normalization at initialization . In the process of doing so , we have uncovered a number of counterintuitive aspects of batch normalization and – in particular – the fact that at initialization it unavoidably causes gradients to explode with depth . We have introduced several methods to reduce the degree of gradient explosion , enabling the training of significantly deeper networks in the presence of batch normalization . Finally , this work paves the way for future work on more advanced , state-of-the-art , network architectures and topologies . REFERENCES Jimmy Lei Ba , Jamie Ryan Kiros , and Geoffrey E. Hinton . Layer Normalization . arXiv:1607.06450 [ cs , stat ] , July 2016 . URL http : //arxiv.org/abs/1607.06450 . David Balduzzi , Marcus Frean , Lennox Leary , J. P. Lewis , Kurt Wan-Duo Ma , and Brian McWilliams . The Shattered Gradients Problem : If resnets are the answer , then what is the question ? In PMLR , pp . 342–350 , July 2017 . URL http : //proceedings.mlr.press/v70/ balduzzi17b.html . Johan Bjorck , Carla Gomes , and Bart Selman . Understanding Batch Normalization . June 2018 . URL https : //arxiv.org/abs/1806.02375 . Minmin Chen , Jeffrey Pennington , and Samuel Schoenholz . Dynamical isometry and a mean field theory of RNNs : Gating enables signal propagation in recurrent neural networks . In Jennifer Dy and Andreas Krause ( eds . ) , Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp . 873–882 , Stockholmsmssan , Stockholm Sweden , 10–15 Jul 2018 . PMLR . URL http : //proceedings.mlr.press/ v80/chen18i.html . Q. Chen and R. Wu . CNN Is All You Need . ArXiv e-prints , December 2017 . Youngmin Cho and Lawrence K. Saul . Kernel methods for deep learning . In Advances in neural information processing systems , pp . 342–350 , 2009 . URL http : //papers.nips.cc/ paper/3628-kernel-methods-for-deep-learning . Amit Daniely , Roy Frostig , and Yoram Singer . Toward Deeper Understanding of Neural Networks : The Power of Initialization and a Dual View on Expressivity . arXiv:1602.05897 [ cs , stat ] , February 2016 . URL http : //arxiv.org/abs/1602.05897 . Alexander G. de G. Matthews , Jiri Hron , Mark Rowland , Richard E. Turner , and Zoubin Ghahramani . Gaussian process behaviour in wide deep neural networks . In International Conference on Learning Representations , 2018 . URL https : //openreview.net/forum ? id= H1-nGgWC- . Adrià Garriga-Alonso , Laurence Aitchison , and Carl Edward Rasmussen . Deep convolutional networks as shallow gaussian processes . International Conference on Learning Representations , 2019 . Dar Gilboa , Bo Chang , Minmin Chen , Greg Yang , Samuel S. Schoenholz , Ed H. Chi , and Jeffrey Pennington . Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs . arXiv e-prints , art . arXiv:1901.08987 , Jan 2019 . Igor Gitman and Boris Ginsburg . Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification . arXiv:1709.08145 [ cs ] , September 2017 . URL http : //arxiv.org/abs/1709.08145 . Boris Hanin and David Rolnick . How to start training : The effect of initialization and architecture . arXiv preprint arXiv:1803.01719 , 2018 . Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep Residual Learning for Image Recognition . arXiv:1512.03385 [ cs ] , December 2015 . URL http : //arxiv.org/abs/ 1512.03385 . Sergey Ioffe and Christian Szegedy . Batch Normalization : Accelerating Deep Network Training by Reducing Internal Covariate Shift . arXiv:1502.03167 [ cs ] , February 2015 . URL http : //arxiv.org/abs/1502.03167 . Jonas Kohler , Hadi Daneshmand , Aurelien Lucchi , Ming Zhou , Klaus Neymeyr , and Thomas Hofmann . Towards a Theoretical Understanding of Batch Normalization . arXiv:1805.10694 [ cs , stat ] , May 2018 . URL http : //arxiv.org/abs/1805.10694 . Yann LeCun , Bernhard E Boser , John S Denker , Donnie Henderson , Richard E Howard , Wayne E Hubbard , and Lawrence D Jackel . Handwritten digit recognition with a back-propagation network . In Advances in neural information processing systems , pp . 396–404 , 1990 . Jaehoon Lee , Yasaman Bahri , Roman Novak , Samuel S Schoenholz , Jeffrey Pennington , and Jascha Sohl-Dickstein . Deep neural networks as gaussian processes . arXiv preprint arXiv:1711.00165 , 2017 . Ping Luo , Xinjiang Wang , Wenqi Shao , and Zhanglin Peng . Understanding Regularization in Batch Normalization . arXiv:1809.00846 [ cs , stat ] , September 2018 . URL http : //arxiv.org/ abs/1809.00846 . Roman Novak , Lechao Xiao , Jaehoon Lee , Yasaman Bahri , Greg Yang , Jiri Hron , Daniel A. Abolafia , Jeffrey Pennington , and Jascha Sohl-Dickstein . Bayesian deep convolutional networks with many channels are gaussian processes . International Conference of Learning Representations , 2019 . URL https : //openreview.net/forum ? id=B1g30j0qF7 . Jeffrey Pennington , Samuel Schoenholz , and Surya Ganguli . Resurrecting the sigmoid in deep learning through dynamical isometry : theory and practice . In Advances in neural information processing systems , pp . 4785–4795 , 2017 . George Philipp and Jaime G. Carbonell . The Nonlinearity Coefficient - Predicting Overfitting in Deep Neural Networks . arXiv:1806.00179 [ cs , stat ] , May 2018 . URL http : //arxiv.org/ abs/1806.00179 . Ben Poole , Subhaneil Lahiri , Maithra Raghu , Jascha Sohl-Dickstein , and Surya Ganguli . Exponential expressivity in deep neural networks through transient chaos . arXiv:1606.05340 [ cond-mat , stat ] , June 2016 . URL http : //arxiv.org/abs/1606.05340 . Tim Salimans and Diederik P. Kingma . Weight Normalization : A Simple Reparameterization to Accelerate Training of Deep Neural Networks . February 2016 . URL https : //arxiv.org/ abs/1602.07868 . Shibani Santurkar , Dimitris Tsipras , Andrew Ilyas , and Aleksander Madry . How Does Batch Normalization Help Optimization ? ( No , It Is Not About Internal Covariate Shift ) . arXiv:1805.11604 [ cs , stat ] , May 2018 . URL http : //arxiv.org/abs/1805.11604 . Samuel S. Schoenholz , Justin Gilmer , Surya Ganguli , and Jascha Sohl-Dickstein . Deep Information Propagation . arXiv:1611.01232 [ cs , stat ] , November 2016 . URL http : //arxiv.org/abs/ 1611.01232 . David Silver , Julian Schrittwieser , Karen Simonyan , Ioannis Antonoglou , Aja Huang , Arthur Guez , Thomas Hubert , Lucas Baker , Matthew Lai , Adrian Bolton , et al . Mastering the game of go without human knowledge . Nature , 550 ( 7676 ) :354 , 2017 . Samuel L Smith and Quoc V Le . A bayesian perspective on generalization and stochastic gradient descent . 2018 . P.K . Suetin . Ultraspherical polynomials - Encyclopedia of Mathematics . URL https : //www . encyclopediaofmath.org/index.php/Ultraspherical_polynomials . Eric W. Weisstein . Gegenbauer Polynomial . URL http : //mathworld.wolfram.com/ GegenbauerPolynomial.html . Christopher KI Williams . Computing with infinite networks . In Advances in neural information processing systems , pp . 295–301 , 1997 . Lechao Xiao , Yasaman Bahri , Jascha Sohl-Dickstein , Samuel Schoenholz , and Jeffrey Pennington . Dynamical isometry and a mean field theory of CNNs : How to train 10,000-layer vanilla convolutional neural networks . In Jennifer Dy and Andreas Krause ( eds . ) , Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp . 5393–5402 , Stockholmsmssan , Stockholm Sweden , 10–15 Jul 2018 . PMLR . URL http : //proceedings.mlr.press/v80/xiao18a.html . Greg Yang . Scaling limits of wide neural networks with weight sharing : Gaussian process behavior , gradient independence , and neural tangent kernel derivation . arXiv preprint arXiv:1902.04760 , 2019 . Greg Yang and Samuel S. Schoenholz . Meanfield Residual Network : On the Edge of Chaos . In Advances in neural information processing systems , 2017 . Greg Yang and Samuel S Schoenholz . Deep mean field theory : Layerwise variance and width variation as methods to control gradient explosion . 2018 . Barret Zoph , Vijay Vasudevan , Jonathon Shlens , and Quoc V Le . Learning transferable architectures for scalable image recognition . A VGG19 WITH BATCHNORM ON CIFAR100 Even though at initialization time batchnorm causes gradient explosion , after the first few epochs , the relative gradient norms ‖∇θL‖/‖θ‖ for weight parameters θ = W or BN scale parameter θ = γ , equilibrate to about the same magnitude . See Fig . 7 . B GRADIENT INDEPENDENCE ASSUMPTION Following prior literature Schoenholz et al . ( 2016 ) ; Yang & Schoenholz ( 2017 ) ; Xiao et al . ( 2018 ) , in this paper , in regards to computations involving backprop , we assume Assumption 2 . During backpropagation , whenever we multiply by WT for some weight matrix W , we multiply by an iid copy instead . As in these previous works , we find excellent agreement between computations made under this assumption and the simulations ( see Fig . 1 ) . Yang ( 2019 ) in fact recently rigorously justified this assumption as used in the computation of moments , for a wide variety of architectures , like multilayer perceptron ( Schoenholz et al. , 2016 ) , residual networks ( Yang & Schoenholz , 2017 ) , and convolutional networks ( Xiao et al. , 2018 ) studied previously , but without batchnorm . The reason that their argument does not extend to batchnorm is because of the singularity in its Jacobian at 0 . However , as observed in our experiments , we expect that a proof can be found to extend Yang & Schoenholz ( 2017 ) ’ s work to batchnorm . C NOTATIONS Definition C.1 . Let SB be the space of PSD matrices of size B × B . Given a measurable function Φ : RB → RA , define the integral transform VΦ : SB → SA by VΦ ( Σ ) = E [ Φ ( h ) ⊗2 : h ∼ N ( 0 , Σ ) ] .3 When φ : R → R and B is clear from context , we also write Vφ for V applied to the function acting coordinatewise by φ . Definition C.2 . For any φ : R → R , let Bφ : RB → RB be batchnorm ( applied to a batch of neuronal activations ) followed by coordinatewise applications of φ , Bφ ( x ) j = φ ( ( xj −Avg x ) / √ 1 B ∑B i=1 ( xi −Avg x ) 2 ) ( here Avg x = 1B ∑B i=1 xi ) . When φ = id we will also write B = Bid . Definition C.3 . Define the matrix GB = I − 1B11T . Let SGB be the space of PSD matrices of size B × B with zero mean across rows and columns , SGB : = { Σ ∈ SB : GBΣGB = Σ } = { Σ ∈ SB : Σ1 = 0 } . 3This definition of V absorbs the previous definitions of V and W in Yang & Schoenholz ( 2017 ) for the scalar case When B is clear from context , we will suppress the subscript/superscript B . In short , for h ∈ RB , Gh zeros the sample mean of h. G is a projection matrix to the subspace of vectors h ∈ RB of zero coordinate sum . With the above definitions , we then have Bφ ( h ) = φ ( √ BGh/‖Gh‖ ) . We use Γ to denote the Gamma function , P ( a , b ) : = Γ ( a + b ) /Γ ( a ) to denote the Pochhammer symbol , and Beta ( a , b ) = Γ ( a ) Γ ( b ) Γ ( a+b ) to denote the Beta function . We use 〈· , ·〉 to denote the dot product for vectors and trace inner product for matrices . Sometimes when it makes for simpler notation , we will also use · for dot product . We adopt the matrix convention that , for a multivariate function f : Rn → Rm , the Jacobian is df dx = ∂f1 ∂x1 ∂f1 ∂x2 · · · ∂f1∂xn ∂f2 ∂x1 ∂f2 ∂x2 · · · ∂f2∂xn ... ... . . . ... ∂fm ∂x1 ∂fm ∂x2 · · · ∂fm∂xn , so that f ( x + ∆x ) = dfdx∆x + o ( ∆x ) , where x and ∆x are treated as column vectors . In what follows we further abbreviate B′φ ( x ) = dBφ ( z ) dz ∣∣ z=x , B′φ ( x ) ∆x = dBφ ( z ) dz ∣∣ z=x ∆x . D A GUIDE TO THE REST OF THE APPENDIX As discussed in the main text , we are interested in several closely related dynamics induced by batchnorm in a fully-connected network with random weights . One is the forward propagation equation Eq . ( 43 ) Σl = VBφ ( Σ l−1 ) studied in Appendix F. Another is the backward propagation equation Eq . ( 51 ) Πl = VB′φ ( Σ ∗ ) † { Πl+1 } studied in Appendix G. We will also study their generalizations to the simultaneous propagation of multiple batches in Appendices H and I . In general , we discover several ways to go about such analyses , each with their own benefits and drawbacks : • The Laplace Method ( Appendix E.1 ) : If the nonlinearity φ is positive homogeneous , we can use Schwinger ’ s parametrization to turn VBφ and VB′φ into nicer forms involving only 1 or 2 integrals , as mentioned in the main text . All relevant quantities , such as eigenvalues of the backward dynamics , can be further simplified to closed forms . Lemma E.2 gives the master equation for the Laplace method . Because the Laplace method requires φ to be positive homogeneous to apply , we review relevant results of such functions in Appendices E.1.1 and E.1.2 . • The Fourier Method ( Appendix E.2 ) : For polynomially bounded and continuous nonlinearity φ ( for the most general set of conditions , see Thm E.25 ) , we can obtain similar simplifications using the Fourier expansion of the delta function , with the penalty of an additional complex integral . • Spherical Integration ( Appendix E.3 ) : Batchnorm can naturally be interpreted as a linear projection followed by projection to a sphere of radius √ B . Assuming that the forward dynamics converges to a BSB1 fixed point , one can express VBφ and VB′φ as spherical integrals , for all measurable φ . One can reduce this ( B−1 ) -dimensional integral , in spherical angular coordinates , into 1 or 2 dimensions by noting that the integrand depending on φ only depends on 1 or 2 angles . Thus this method is very suitable for numerical evaluation of quantities of interest for general φ . • The Gegenbauer Method ( Appendix E.4 ) : Gegenbauer polynomials are orthogonal polynomials under the weight ( 1 − x2 ) α for some α . They correspond to a special type of spherical harmonics called zonal harmonics ( Defn E.44 ) that has the very useful reproducing property ( Fact E.46 ) . By expressing the nonlinearity φ in this basis , we can see easily that the relevant eigenvalues of the forward and backward dynamics are ratios of two quadratic forms of φ ’ s Gegenbauer coefficients . In particular , for the largest eigenvalue of the backward dynamics , the two quadratic forms are both diagonal , and in a way that makes apparent the necessity of gradient explosion ( under the BSB1 fixed point assumption ) . In regards to numerical computations , the Gegenbauer method has the benefit of only requiring 1-dimensional integrals to obtain the coefficients , but if φ has slowlydecaying coefficients , then a large number of such integrals may be required to get an accurate answer . In each of the following sections , we will attempt to conduct an analysis with each method when it is feasible . When studying the convergence rate of the forward dynamics and the gradient explosion of the backward dynamics , we encounter linear operators on matrices that satisfy an abundance of symmetries , called ultrasymmetric operators . We study these operators in Appendix E.5 and contribute several structural results Thms E.61 , E.62 , E.72 and E.72 on their eigendecomposition that are crucial to deriving the asymptotics of the dynamics . All of the above will be explained in full in the technique section Appendix E. We then proceed to study the forward and backward dynamics in detail ( Appendices F to I ) , now armed with tools we need . Up to this point , we only consider the dynamics assuming Eq . ( 43 ) converges to a BSB1 fixed point , and B ≥ 4 . In Appendix J we discuss what happens outside this regime , and in Appendix K we show our current understanding of BSB2 fixed point dynamics . Note that the backward dynamics studied in these sections is that of the gradient with respect to the hidden preactivations . Nevertheless , we can compute the moments of the weight gradients from this , which is of course what is eventually used in gradient descent . We do so in the final section Appendix L. Main Technical Results Corollary F.3 establishes the global convergence of Eq . ( 43 ) to BSB1 fixed point for φ = id ( as long as the initial covariance is nondegenerate ) , but we are not able to prove such a result for more general φ . We thus resort to studying the local convergence properties of BSB1 fixed points . First , Thms F.5 , F.8 and F.13 respectively compute the BSB1 fixed point from the perspectives of spherical integration , the Laplace method , and the Gegenbauer method . Thms F.22 and F.24 give the Gegenbauer expansion of the BSB1 local convergence rate , and Thm F.33 gives a more succinct form of the same for positive homogeneous φ . Appendix F.3.1 discusses how to do this computation via spherical integration . Next , we study the gradient dynamics , and Thms G.5 and G.12 yield the gradient explosion rates with the Gegenbauer method and the Laplace method ( for positive homogeneous φ ) , respectively . Appendix G.1.1 discusses how to compute this for general φ using spherical integration . We then turn to the dynamics of cross batch covariances . Thm H.3 gives the form of the cross batch fixed point via spherical integration and the Gegenbauer method , and Thm H.6 yields a more specific form for positive homogeneous φ via the Laplace method . The local convergence rate to this fixed point is given by Thms H.13 and H.14 , respectively using the Gegenbauer method and the Laplace method . Finally , the correlation between gradients of the two batches is shown to decrease exponentially in Corollary I.6 using the Gegenbauer method , and the decay rate is given succinctly for positive homogeneous φ in Thm I.3 . E TECHNIQUES We introduce the key techniques and tools in our analysis in this section . E.1 LAPLACE METHOD As discussed above , the Laplace method is useful for deriving closed form expressions for positive homogeneous φ . The key insight here is to apply Schwinger parametrization to deal with normalization . Lemma E.1 ( Schwinger parametrization ) . For z > 0 and c > 0 , c−z = Γ ( z ) −1 ∫ ∞ 0 xz−1e−cx dx The following is the key lemma in the Laplace method . Lemma E.2 ( The Laplace Method Master Equation ) . For A , A′ ∈ N , let f : RA → RA′ and let k ≥ 0 . Suppose ‖f ( y ) ‖ ≤ h ( ‖y‖ ) for some nondecreasing function h : R≥0 → R≥0 such that E [ h ( r‖y‖ ) : y ∈ N ( 0 , IA ) ] exists for every r ≥ 0 . Define ℘ ( Σ ) : = E [ ‖y‖−2kf ( y ) : y ∼ N ( 0 , Σ ) ] . Then on { Σ ∈ SA : rank Σ > 2k } , ℘ ( Σ ) is well-defined and continuous , and furthermore satisfies ℘ ( Σ ) = Γ ( k ) −1 ∫ ∞ 0 ds sk−1 det ( I + 2sΣ ) −1/2 E [ f ( y ) : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] ( 25 ) Proof . ℘ ( Σ ) is well-defined for full rank Σ because the ‖y‖−2k singularity at y = 0 is Lebesgueintegrable in a neighborhood of 0 in dimension A > 2k . We prove Eq . ( 25 ) in the case when Σ is full rank and then apply a continuity argument . Proof of Eq . ( 25 ) for full rank Σ . First , we will show that we can exchange the order of integration∫ ∞ 0 ds ∫ RA dy sk−1f ( y ) e− 1 2y T ( Σ−1+2sI ) y = ∫ RA dy ∫ ∞ 0 ds sk−1f ( y ) e− 1 2y T ( Σ−1+2sI ) y by Fubini-Tonelli ’ s theorem . Observe that ( 2π ) −A/2 det Σ−1/2 ∫ ∞ 0 ds sk−1 ∫ RA dy ‖f ( y ) ‖e− 12yT ( Σ−1+2sI ) y = ∫ ∞ 0 ds sk−1 det ( Σ ( Σ−1 + 2sI ) ) −1/2 E [ ‖f ( y ) ‖ : y ∼ N ( 0 , ( Σ−1 + 2sI ) −1 ) ] = ∫ ∞ 0 ds sk−1 det ( I + 2sΣ ) −1/2 E [ ‖f ( y ) ‖ : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] For λ = ‖Σ‖2 , ‖f ( √ Σ ( I + 2sΣ ) −1y ) ‖ ≤ h ( ‖ √ Σ ( I + 2sΣ ) −1y‖ ≤ h ( ∥∥∥∥∥ √ λ 1 + 2sλ y ∥∥∥∥∥ ) ≤ h ( √ λ‖y‖ ) Because E [ h ( √ λ‖y‖ ) : y ∈ N ( 0 , I ) ] exists by assumption , E [ ‖f ( y ) ‖ : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] = E [ ‖f ( √ Σ ( I + 2sΣ ) −1y ) ‖ : y ∼ N ( 0 , I ) ] → E [ ‖f ( 0 ) ‖ : y ∼ N ( 0 , I ) ] = ‖f ( 0 ) ‖ as s → ∞ , by dominated convergence with dominating function h ( √ λ‖y‖ ) e− 12‖y‖2 ( 2π ) −A/2 . By the same reasoning , the function s 7→ E [ ‖f ( y ) ‖ : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] is continuous . In particular this implies that sup0≤s≤∞ E [ ‖f ( y ) ‖ : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] < ∞ . Combined with the fact that det ( I + 2sΣ ) −1/2 = Θ ( s−A/2 ) as s→∞ , ∫ ∞ 0 ds sk−1 det ( I + 2sΣ ) −1/2 E [ ‖f ( y ) ‖ : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] ≤ ∫ ∞ 0 ds Θ ( sk−1−A/2 ) which is bounded by our assumption that A/2 > k. This shows that we can apply Fubini-Tonelli ’ s theorem to allow exchanging order of integration . Thus , E [ ‖y‖−2kf ( y ) : y ∼ N ( 0 , Σ ) ] = E [ f ( y ) ∫ ∞ 0 ds Γ ( k ) −1sk−1e−‖y‖ 2s : y ∼ N ( 0 , Σ ) ] = ( 2π ) −A/2 det Σ−1/2 ∫ dy e− 1 2y TΣ−1yf ( y ) ∫ ∞ 0 ds Γ ( k ) −1sk−1e−‖y‖ 2s = ( 2π ) −A/2Γ ( k ) −1 det Σ−1/2 ∫ ∞ 0 ds sk−1 ∫ dy f ( y ) e− 1 2y T ( Σ−1+2sI ) y ( by Fubini-Tonelli ) = Γ ( k ) −1 ∫ ∞ 0 ds sk−1 det ( Σ ( Σ−1 + 2sI ) ) −1/2 E [ f ( y ) : y ∼ N ( 0 , ( Σ−1 + 2sI ) −1 ) ] = Γ ( k ) −1 ∫ ∞ 0 ds sk−1 det ( I + 2sΣ ) −1/2 E [ f ( y ) : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] Domain and continuity of ℘ ( Σ ) . The LHS of Eq . ( 25 ) , ℘ ( Σ ) , is defined and continuous on rank Σ/2 > k. Indeed , if Σ = MICMT , where M is a full rank A × C matrix with rank Σ = C ≤ A , then E [ ‖y‖−2k‖f ( y ) ‖ : y ∼ N ( 0 , Σ ) ] = E [ ‖Mz‖−2k‖f ( Mz ) ‖ : z ∼ N ( 0 , IC ) ] . This is integrable in a neighborhood of 0 iff C > 2k , while it ’ s always integrable outside a ball around 0 because ‖f‖ by itself already is . So ℘ ( Σ ) is defined whenever rank Σ > 2k . Its continuity can be established by dominated convergence . Proof of Eq . ( 25 ) for rank Σ > 2k . Observe that det ( I + 2sΣ ) −1/2 is continuous in Σ and , by an application of dominated convergence as in the above , E [ f ( y ) : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] is continuous in Σ . So the RHS of Eq . ( 25 ) is continuous in Σ whenever the integral exists . By the reasoning above , E [ f ( y ) : y ∼ N ( 0 , Σ ( I + 2sΣ ) −1 ) ] is bounded in s and det ( I + 2sΣ ) −1/2 = Θ ( s− rank Σ/2 ) , so that the integral exists iff rank Σ/2 > k. To summarize , we have proved that both sides of Eq . ( 25 ) are defined and continous for rank Σ > 2k . Because the full rank matrices are dense in this set , by continuity Eq . ( 25 ) holds for all rank Σ > 2k . If φ is degree-α positive homogeneous , i.e . φ ( ru ) = rαφ ( u ) for any u ∈ R , r ∈ R+ , we can apply Lemma E.2 with k = α , VBφ ( Σ ) = E [ φ ( √ BGh/‖Gh‖ ) ⊗2 : h ∼ N ( 0 , Σ ) ] = Bα E [ φ ( Gh ) ⊗2/‖Gh‖2α : h ∼ N ( 0 , Σ ) ] = Bα E [ φ ( z ) ⊗2/‖z‖2α : z ∼ N ( 0 , GΣG ) ] = BαΓ ( α ) −1 ∫ ∞ 0 ds sα−1 det ( I + 2sΣG ) −1/2Vφ ( G ( I + 2sΣG ) −1ΣG ) . If φ = relu , then Vφ ( Σ ) ij = { 1 2Σii if i = j 1 2J1 ( Σij/ √ ΣiiΣjj ) √ ΣiiΣjj otherwise and , more succinctly , Vφ ( Σ ) = D1/2Vφ ( D−1/2ΣD−1/2 ) D1/2 where D = Diag ( Σ ) . Here J1 ( c ) : = 1 π ( √ 1− c2 + ( π − arccos ( c ) ) c ) ( Cho & Saul , 2009 ) . Matrix simplification . We can simplify the expression G ( I + 2sΣG ) −1ΣG , leveraging the fact that G is a projection matrix . Definition E.3 . Let e be an B × ( B − 1 ) matrix whose columns form an orthonormal basis of imG : = { Gv : v ∈ RB } = { w ∈ RB : ∑i wi = 0 } . Then the B × B matrix ẽ = ( e|B−1/21 ) is an orthogonal matrix . For much of this paper e can be any such basis , but at certain sections we will consider specific realizations of e for explicit computation . From easy computations it can be seen that G = ẽ ( IB−1 0 0 0 ) ẽ T . Suppose Σ = ẽ ( Σ v vT a ) ẽ T where Σ is ( B − 1 ) × ( B − 1 ) , v is a column vector and a is a scalar . Then Σ = eTΣe and ΣG = ẽ ( Σ 0 vT 0 ) ẽ T is block lower triangular , and ( I + 2sΣG ) −1 = ẽ ( ( I + 2sΣ ) −1 0 ∗ 1 ) ẽ T ( I + 2sΣG ) −1ΣG = ẽ ( ( I + 2sΣ ) −1Σ 0 ∗ 0 ) ẽ T G ( I + 2sΣG ) −1ΣG = ẽ ( ( I + 2sΣ ) −1Σ 0 0 0 ) ẽ T = e ( I + 2sΣ ) −1Σ e T = GΣG ( I + 2sGΣG ) −1 = : ΣG ( I + 2sΣG ) −1 where Definition E.4 . For any matrix Σ , write ΣG : = GΣG . Similarly , det ( IB + 2sΣG ) = det ( IB−1 + 2sΣ ) = det ( IB + 2sΣG ) . So , altogether , we have Theorem E.5 . Suppose φ : R → R is degree-α positive homogeneous . Then for any B × ( B − 1 ) matrix e whose columns form an orthonormal basis of imG : = { Gv : v ∈ RB } = { w ∈ RB : ∑ i wi = 0 } , with Σ = eTΣe , VBφ ( Σ ) = B αΓ ( α ) −1 ∫ ∞ 0 ds sα−1 det ( I + 2sΣ ) −1/2Vφ ( e ( I + 2sΣ ) −1Σ ) −1 e T ) ( 26 ) = BαΓ ( α ) −1 ∫ ∞ 0 ds sα−1 det ( I + 2sΣG ) −1/2Vφ ( ( I + 2sΣ G ) −1ΣG ) −1 ) ( 27 ) Now , in order to use Laplace ’ s method , we require φ to be positive homogeneous . What do these functions look like ? The most familiar example to a machine learning audience is most likely ReLU . It turns out that 1-dimensional positive homogeneous functions can always be described as a linear combination of powers of ReLU and its reflection across the y-axis . Below , we review known facts about these functions and their integral transforms , starting with the powers of ReLU in Appendix E.1.1 and then onto the general case in Appendix E.1.2 . E.1.1 α-RELU Recall that α-ReLUs ( Yang & Schoenholz , 2017 ) are , roughly speaking , the αth power of ReLU . Definition E.6 . The α-ReLU function ρα : R → R sends x 7→ xα when x > 0 and x 7→ 0 otherwise . This is a continuous function for α > 0 but discontinuous at 0 for all other α . We briefly review what is currently known about the V and W transforms of ρα ( Cho & Saul , 2009 ; Yang & Schoenholz , 2017 ) . Definition E.7 . For any α > − 12 , define cα = 1√π2α−1Γ ( α+ 1 2 ) . When considering only 1-dimensional Gaussians , Vρα is very simple . Proposition E.8 . If α > − 12 , then for any q ∈ S1 = R≥0 , Vρα ( q ) = cαqα To express results of Vρα on SB for higher B , we first need the following Definition E.9 . Define Jα ( θ ) : = 1 2πcα ( sin θ ) 2α+1Γ ( α+ 1 ) ∫ π/2 0 dη cosα η ( 1− cos θ cos η ) 1+α and Jα ( c ) = Jα ( arccos c ) for α > −1/2 . Then Proposition E.10 . For any Σ ∈ SB , let D be the diagonal matrix with the same diagonal as Σ . Then Vρα ( Σ ) = cαD α/2Jα ( D −1/2ΣD−1/2 ) Dα/2 where Jα is applied entrywise . For example , Jα and Jα for the first few integral α are J0 ( θ ) = π − θ π J1 ( θ ) = sin θ + ( π − θ ) cos θ π J2 ( θ ) = 3 sin θ cos θ + ( π − θ ) ( 1 + 2 cos2 θ ) 3π J0 ( c ) = π − arccos c π J1 ( c ) = √ 1− c2 + ( π − arccos c ) c π J2 ( c ) = 3c √ 1− c2 + ( π − arccos c ) ( 1 + 2c2 ) 3π One can observe very easily that ( Daniely et al. , 2016 ; Yang & Schoenholz , 2017 ) Proposition E.11 . For each α > −1/2 , Jα ( c ) is an increasing and convex function on c ∈ [ 0 , 1 ] , and is continous on c ∈ [ 0 , 1 ] and smooth on c ∈ ( 0 , 1 ) . Jα ( 1 ) = 1 , Jα ( 0 ) = 12√π Γ ( α2 + 1 2 ) 2 Γ ( α+ 12 ) , and Jα ( −1 ) = 0 . Yang & Schoenholz ( 2017 ) also showed the following fixed point structure Theorem E.12 . For α ∈ [ 1/2 , 1 ) , Jα ( c ) = c has two solutions : an unstable solution at 1 ( ” unstable ” meaning J′α ( 1 ) > 1 ) and a stable solution in c ∗ ∈ ( 0 , 1 ) ( ” stable ” meaning J′α ( c∗ ) < 1 ) . The α-ReLUs satisfy very interesting relations amongst themselves . For example , Lemma E.13 . Suppose α > 1 . Then Jα ( θ ) = cos θJα−1 ( θ ) + ( α− 1 ) 2 ( 2α− 1 ) −1 ( 2α− 3 ) −1 sin2 θJα−2 ( θ ) Jα ( c ) = cJα−1 ( c ) + ( α− 1 ) 2 ( 2α− 1 ) −1 ( 2α− 3 ) −1 ( 1− c2 ) Jα−2 ( c ) In additional , surprisingly , one can use differentiation to go from α to α+ 1 and from α to α− 1 ! Proposition E.14 ( Yang & Schoenholz ( 2017 ) ) . Suppose α > 1/2 . Then J ′α ( θ ) = −α2 ( 2α− 1 ) −1Jα−1 ( θ ) sin θ J′α ( c ) = α 2 ( 2α− 1 ) −1Jα−1 ( c ) so that Jα−n ( c ) = α∏ β=α−n+1 β−2 ( 2β − 1 ) ( ∂/∂c ) nJα ( c ) We have the following from Cho & Saul ( 2009 ) Proposition E.15 ( Cho & Saul ( 2009 ) ) . For all α ≥ 0 and integer n ≥ 1 Jn+α ( c ) = cα cn+α ( 1− c2 ) n+α+ 12 ( ∂/∂c ) n ( Jα ( c ) / ( 1− c2 ) α+ 1 2 ) = α+n−1∏ β=α ( 2β + 1 ) −1 ( 1− c2 ) n+α+ 12 ( ∂/∂c ) n ( Jα ( c ) / ( 1− c2 ) α+ 12 ) This implies in particular that we can obtain J′α from Jα and Jα+1 . Proposition E.16 . For all α ≥ 0 , J′α ( c ) = ( 2α+ 1 ) ( 1− c2 ) −1 ( J1+α ( c ) − cJα ( c ) ) ) Proof . J1+α ( c ) = cα c1+α ( 1− c2 ) 1+α+ 12 ( ∂/∂c ) ( Jα ( c ) / ( 1− c2 ) α+ 1 2 ) = ( 2α+ 1 ) −1 ( 1− c2 ) α+3/2 ( J′α ( c ) / ( 1− c2 ) α+1/2 + 2c ( α+ 1/2 ) Jα ( c ) / ( 1− c2 ) α+3/2 ) = ( 2α+ 1 ) −1J′α ( c ) ( 1− c2 ) + cJα ( c ) J′α ( c ) = ( 2α+ 1 ) ( 1− c2 ) −1 ( J1+α ( c ) − cJα ( c ) ) Note that we can also obtain this via Lemma E.13 and Proposition E.14 . E.1.2 POSITIVE-HOMOGENEOUS FUNCTIONS IN 1 DIMENSION Suppose for some α ∈ R , φ : R → R is degree α positive-homogeneous , i.e . φ ( rx ) = rαφ ( x ) for any x ∈ R , r > 0 . The following simple lemma says that we can always express φ as linear combination of powers of α-ReLUs . Proposition E.17 . Any degree α positive-homogeneous function φ : R → R with φ ( 0 ) = 0 can be written as x 7→ aρα ( x ) − bρα ( −x ) . Proof . Take a = φ ( 1 ) and b = φ ( −1 ) . Then positive-homogeneity determines the value of φ on R \ { 0 } and it coincides with x 7→ aρα ( x ) − bρα ( −x ) . As a result we can express the V and W transforms of any positive-homogeneous function in terms of those of α-ReLUs . Proposition E.18 . Suppose φ : R → R is degree α positive-homogeneous . By Proposition E.17 , φ restricted to R \ { 0 } can be written as x 7→ aρα ( x ) − bρα ( −x ) for some a and b . Then for any PSD 2× 2 matrix M , Vφ ( M ) 11 = ( a 2 + b2 ) Vρα ( M ) 11 = cα ( a 2 + b2 ) Mα11 Vφ ( M ) 22 = ( a 2 + b2 ) Vρα ( M ) 22 = cα ( a 2 + b2 ) Mα22 Vφ ( M ) 12 = Vφ ( M ) 21 = ( a 2 + b2 ) Vρα ( M ) 12 − 2abVρα ( M ′ ) 12 = cα ( M11M22 ) α/2 ( ( a2 + b2 ) Jα ( M12/ √ M11M22 ) − 2abJα ( −M12/ √ M11M22 ) ) where M ′ : = ( 1 0 0 −1 ) M ( 1 0 0 −1 ) . Proof . We directly compute , using the expansion of φ into ραs : Vφ ( M ) 11 = E [ φ ( x ) 2 : x ∼ N ( 0 , M11 ) ] = E [ a2ρα ( x ) 2 + b2ρα ( −x ) 2 + 2abρα ( x ) ρα ( −x ) ] = a2 E [ ρα ( x ) 2 ] + b2 E [ ρα ( −x ) 2 ] = ( a2 + b2 ) E [ ρα ( x ) 2 : x ∼ N ( 0 , M11 ) ] ( 28 ) = cα ( a 2 + b2 ) Mα11 where in Eq . ( 28 ) we used negation symmetry of centered Gaussians . The case of Vφ ( M ) 22 is similar . Vφ ( M ) 12 = E [ φ ( x ) φ ( y ) : ( x , y ) ∼ N ( 0 , M ) ] = E [ a2ρα ( x ) ρα ( y ) + b2ρα ( −x ) ρα ( −y ) − abρα ( x ) ρα ( −y ) − abρα ( −x ) ρα ( y ) ] = ( a2 + b2 ) Vρα ( M ) 12 − 2abVρα ( M ′ ) 12 = cα ( M11M22 ) α/2 ( ( a2 + b2 ) Jα ( M12/ √ M11M22 ) − 2abJα ( −M12/ √ M11M22 ) ) where in the last equation we have applied Proposition E.10 . This then easily generalizes to PSD matrices of arbitrary dimension : Corollary E.19 . Suppose φ : R → R is degree α positive-homogeneous . By Proposition E.17 , φ restricted to R\ { 0 } can be written as x 7→ aρα ( x ) − bρα ( −x ) for some a and b . Let Σ ∈ SB . Then Vφ ( Σ ) = cαD α/2Jφ ( D −1/2ΣD−1/2 ) Dα/2 where Jφ is defined below and is applied entrywise . Explicitly , this means that for all i , Vφ ( Σ ) ii = cαΣ α iiJφ ( 1 ) Vφ ( Σ ) ij = cαJφ ( Σij/ √ ΣiiΣjj ) Σ α/2 ii Σ α/2 jj Definition E.20 . Suppose φ : R → R is degree α positive-homogeneous . By Proposition E.17 , φ restricted to R \ { 0 } can be written as x 7→ aρα ( x ) − bρα ( −x ) for some a and b . Define Jφ ( c ) : = ( a 2 + b2 ) Jα ( c ) − 2abJα ( −c ) . Let us immediately make the following easy but important observations . Proposition E.21 . Jφ ( 1 ) = a2 + b2 and Jφ ( 0 ) = ( a2 + b2 − 2ab ) Jα ( 0 ) = ( a− b ) 2 12√π Γ ( α2 + 1 2 ) 2 Γ ( α+ 12 ) , Proof . Use the fact that Jα ( −1 ) = 0 and Jα ( 0 ) = 12√π Γ ( α2 + 1 2 ) 2 Γ ( α+ 12 ) by Proposition E.11 . As a sanity check , we can easily compute that Jid ( c ) = 2J1 ( c ) − 2J1 ( −c ) = 2c because id ( x ) = relu ( x ) −relu ( −x ) . By Corollary E.19 and c1 = 12 , this recovers the obvious fact that V ( id ) ( Σ ) = Σ . We record the partial derivatives of Vφ . Proposition E.22 . Let φ be positive homogeneous of degree α . Then for all i with Σii 6= 0 , ∂Vφ ( Σ ) ii ∂Σii = cααΣ α−1 ii Jφ ( 1 ) For all i 6= j with Σii , Σjj 6= 0 , ∂Vφ ( Σ ) ii ∂Σij = 0 ∂Vφ ( Σ ) ij ∂Σij = cαΣ ( α−1 ) /2 ii Σ ( α−1 ) /2 jj J ′ φ ( cij ) ∂Vφ ( Σ ) ij ∂Σii = 1 2 cαΣ 1 2 ( α−2 ) ii Σ 1 2α jj ( αJφ ( cij ) − cijJ′φ ( cij ) ) where cij = Σij/ √ ΣiiΣjj and J′φ denotes its derivative . Proposition E.23 . If φ ( c ) = aρα ( c ) − bρα ( −c ) with α > 1/2 on R \ { 0 } , then J′φ ( c ) = ( 2α− 1 ) −1Jφ′ ( c ) = ( 2α+ 1 ) ( 1− c2 ) −1 ( ( a2 + b2 ) Jα+1 ( c ) + 2abJα+1 ( −c ) ) − cJφ ( c ) J′φ ( 0 ) = ( a+ b ) 2 1√ π Γ ( α2 + 1 ) 2 Γ ( α+ 12 ) Proof . We have φ′ ( c ) = aαρα−1 ( c ) + bαρα−1 ( −c ) . On the other hand , J′φ = ( a2 + b2 ) J′α ( c ) + 2abJ′α ( −c ) which by Proposition E.14 is α2 ( 2α − 1 ) −1 ( ( a2 + b2 ) Jα−1 ( c ) + 2abJα−1 ( −c ) ) = ( 2α− 1 ) −1Jφ′ ( c ) . This proves the first equation . With Proposition E.16 , J′φ ( c ) = ( a 2 + b2 ) J′α ( c ) + 2abJ ′ α ( −c ) = ( a2 + b2 ) ( 2α+ 1 ) ( 1− c2 ) −1 ( Jα+1 ( c ) − cJα ( c ) ) + 2ab ( 2α+ 1 ) ( 1− c2 ) −1 ( Jα+1 ( −c ) + cJα ( −c ) ) = ( 2α+ 1 ) ( 1− c2 ) −1 ( ( a2 + b2 ) Jα+1 ( c ) + 2abJα+1 ( −c ) ) − cJφ ( c ) This gives the second equation . Expanding Jα+1 ( 0 ) With Proposition E.11 , we get J′φ ( 0 ) = ( 2α+ 1 ) ( ( a 2 + b2 ) Jα+1 ( 0 ) + 2abJα+1 ( 0 ) ) = ( 2α+ 1 ) ( a+ b ) 2 1 2 √ π Γ ( α2 + 1 ) 2 Γ ( α+ 32 ) Unpacking the definition of Γ ( α+ 32 ) then yields the third equation . In general , we can factor diagonal matrices out of Vφ . Proposition E.24 . For any Σ ∈ SB , D any diagonal matrix , and φ positive-homogeneous with degree α , Vφ ( DΣD ) = D αVφ ( Σ ) D α Proof . For any i , Vφ ( DΣD ) ii = E [ φ ( x ) 2 : x ∼ N ( 0 , D2iiΣii ) ] = E [ φ ( Diix ) 2 : x ∼ N ( 0 , Σii ) ] = E [ D2αii φ ( x ) 2 : x ∼ N ( 0 , Σii ) ] = D2αii Vφ ( Σ ) ii For any i 6= j , Vφ ( DΣD ) ij = E [ φ ( x ) φ ( y ) : ( x , y ) ∼ N ( 0 , ( D2iiΣii DiiDjjΣij DiiDjjΣji D 2 jjΣjj ) ) = E [ φ ( Diix ) φ ( Djjy ) : ( x , y ) ∼ N ( 0 , ( Σii Σij Σji Σjj ) ) ] = DαiiD α jj E [ φ ( x ) φ ( y ) : ( x , y ) ∼ N ( 0 , ( Σii Σij Σji Σjj ) ) ] = DαiiD α jjVφ ( Σ ) ij E.2 FOURIER METHOD The Laplace method crucially used the fact that we can pull out the norm factor ‖Gh‖ out of φ , so that we can apply Schwinger parametrization . For general φ this is not possible , but we can apply some wishful thinking and proceed as follows VBφ ( Σ ) = E [ φ ( √ BGh/‖Gh‖ ) ⊗2 : h ∼ N ( 0 , Σ ) ] = E [ φ ( Gh/r ) ⊗2 ∫ ∞ 0 d ( r2 ) δ ( r2 − ‖Gh‖2/B ) : h ∼ N ( 0 , Σ ) ] = E [ φ ( Gh/r ) ⊗2 ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π eiλ ( r 2−‖Gh‖2/B ) : h ∼ N ( 0 , Σ ) ] “ = ” ∫ RB dh√ det ( 2πΣ ) e −1 2 h TΣ−1hφ ( Gh/r ) ⊗2 ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π eiλ ( r 2−‖Gh‖2/B ) “ = ” ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π ∫ RB dh√ det ( 2πΣ ) e −1 2 h TΣ−1hφ ( Gh/r ) ⊗2eiλ ( r 2−‖Gh‖2/B ) ( 29 ) = 1√ det ( 2πΣ ) ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π eiλr 2 ∫ RB dhφ ( Gh/r ) ⊗2e −1 2 ( h TΣ−1h+2iλ‖Gh‖2/B ) ( 30 ) = 1√ det ( 2πΣ ) ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π eiλr 2 ∫ RB dhφ ( Gh/r ) ⊗2e −1 2 ( h T ( Σ−1+2iλG/B ) h ) “ = ” 1√ det ( 2πΣ ) ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π eiλr 2 √ det ( 2π ( Σ−1 + 2iλG/B ) −1 ) E [ φ ( Gh/r ) ⊗2 : h ∼ N ( 0 , Σ−1 + 2iλG/B ) ] = ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π eiλr 2 det ( I + 2iλGΣ/B ) −1/2 E h∼N ( 0 , G ( Σ−1+2iλG/B ) Gr−2 ) ) φ ( h ) ⊗2 ( 31 ) = ∫ ∞ 0 d ( r2 ) ∫ ∞ −∞ dλ 2π eiλr 2 det ( I + 2iλGΣ/B ) −1/2Vφ ( G ( Σ −1 + 2iλG/B ) Gr−2 ) Here all steps are legal except for possibly Eq . ( 29 ) , Eq . ( 30 ) , and Eq . ( 31 ) . In Eq . ( 29 ) , we “ Fourier expanded ” the delta function — this could possibly be justified as a principal value integral , but this can not work in combination with Eq . ( 30 ) where we have switched the order of integration , integrating over h first . Fubini ’ s theorem would not apply here because eiλ ( r 2−‖Gh‖2/B ) has norm 1 and is not integrable . Finally , in Eq . ( 31 ) , we need to extend the definition of Gaussian to complex covariance matrices , via complex Gaussian integration . This last point is no problem , and we will just define E h∼N ( 0 , Σ ) f ( h ) : = det ( 2πΣ ) −1/2 ∫ RB dh f ( h ) e− 1 2h TΣ−1h for general complex Σ , whenever Σ is nonsingular and this integral is exists , and E h∼N ( 0 , GΣG ) f ( h ) : = det ( 2π ( eTΣe ) ) −1/2 ∫ RB−1 dz f ( ez ) e− 1 2 z T ( eTΣe ) −1z if eTΣe is nonsingular ( as in the case above ) . The definition of Vφ stays the same given the above extension of the definition of Gaussian expectation . Nevertheless , the above derivation is not correct mathematically due to the other points . However , its end result can be justified rigorously by carefully expressing the delta function as the limit of mollifiers . Theorem E.25 . Let Σ be a positive-definite D × D matrix . Let φ : R → R. Suppose for any b > a > 0 , 1 . ∫ b a d ( r2 ) |φ ( za/r ) φ ( zb/r ) | exists and is finite for any za , zb ∈ R ( i.e . φ ( za/r ) φ ( zb/r ) is locally integrable in r ) . 2. there exists an > 0 such that any γ ∈ [ − , ] and for each a , b ∈ [ D ] , ∫ ‖z‖2∈ [ a , b ] dz e− 1 2 z TΣ−1z|φ ( za/ √ ‖z‖2 + γ ) φ ( zb/ √ ‖z‖2 + γ ) | exists and is uniformly bounded by some number possibly depending on a and b . 3. for each a , b ∈ [ D ] , ∫ RD dz ∫ b a d ( r2 ) e− 1 2 z TΣ−1z|φ ( za/r ) φ ( zb/r ) | exists and is finite . 4 . ∫∞ −∞ dλ ∣∣det ( I + 2iλΣ ) −1/2Vφ ( ( Σ−1 + 2iλI ) −1r−2 ) ∣∣ exists and is finite for each r > 0 . If e− 1 2 z TΣ−1zφ ( z/‖z‖ ) ⊗2 is integrable over z ∈ RD , then ( 2π ) −D/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zφ ( z/‖z‖ ) ⊗2 = 1 2π lim a↘0 , b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det ( I + 2iλΣ ) −1/2Vφ ( ( Σ −1 + 2iλI ) −1s−1 ) . Similarly , ( 2π ) −D/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zφ ( √ Dz/‖z‖ ) ⊗2 = 1 2π lim a↘0 , b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det ( I + 2iλΣ ) −1/2Vφ ( ( Σ −1 + 2iλI/D ) −1s−1 ) . If G is the mean-centering projection matrix and ΣG = GΣG , then by the same reasoning as in Thm E.5 , VBφ ( Σ ) = ( 2π ) −D/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zφ ( √ DGz/‖Gz‖ ) ⊗2 = 1 2π lim a↘0 , b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det ( I + 2iλΣG/D ) −1/2Vφ ( Σ G ( I + 2iλΣG/D ) −1s−1 ) . ( 32 ) Note that assumption ( 1 ) is satisfied if φ is continuous ; assumption ( 4 ) is satisfied if D ≥ 3 and for all Π , ‖Vφ ( Π ) ‖ ≤ ‖Π‖α for some α ≥ 0 ; this latter condition , as well as assumptions ( 2 ) and ( 3 ) , will be satisfied if φ is polynomially bounded . Thus the common coordinatewise nonlinearities ReLU , identity , tanh , etc all satisfy these assumptions . Warning : in general , we can not swap the order of integration as ∫∞ −∞ dλ ∫∞ −∞ ds . For example , if φ = id , then ( 2π ) −D/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zφ ( z/‖z‖ ) ⊗2 = 1 2π lim a↘0 , b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det ( I + 2iλΣ ) −1/2 ( Σ−1 + 2iλI ) −1s−1 6= 1 2π ∫ ∞ −∞ dλ ∫ ∞ 0 ds eiλs det ( I + 2iλΣ ) −1/2 ( Σ−1 + 2iλI ) −1s−1 because the s-integral in the latter diverges ( in a neighborhood of 0 ) . Proof . We will only prove the first equation ; the others follow similarly . By dominated convergence , ∫ dze− 1 2 z TΣ−1zφ ( z/‖z‖ ) ⊗2 = lim a↘0 , b↗∞ ∫ dze− 1 2 z TΣ−1zφ ( z/‖z‖ ) ⊗2I ( ‖z‖2 ∈ [ a , b ] ) . Let η : R→ R be a nonnegative bump function ( i.e . compactly supported and smooth ) with support [ −1 , 1 ] and integral 1 such that η′ ( 0 ) = η′′ ( 0 ) = 0 . Then its Fourier transform η̂ ( t ) decays like O ( t−2 ) . Furthermore , η ( x ) : = −1η ( x/ ) is a mollifier , i.e . for all f ∈ L1 ( R ) , f ∗ η → f in L1 and pointwise almost everywhere . Now , we will show that∫ dze− 1 2 z TΣ−1zφ ( za/r ) φ ( zb/r ) I ( ‖z‖2 ∈ [ a , b ] ) = lim ↘0 ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) η ( ‖z‖2 − r2 ) by dominated convergence . Pointwise convergence is immediate , because ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) η ( r 2 − ‖z‖2 ) = ∫ ∞ −∞ d ( r2 ) φ ( za/r ) φ ( zb/r ) I ( r2 ∈ [ a , b ] ) η ( ‖z‖2 − r2 ) = [ ( r2 7→ φ ( za/r ) φ ( zb/r ) I ( r2 ∈ [ a , b ] ) ) ∗ η ] ( ‖z‖2 ) → φ ( za/‖z‖ ) φ ( zb/‖z‖ ) I ( ‖z‖2 ∈ [ a , b ] ) as → 0 ( where we used assumption ( 1 ) that φ ( za/r ) φ ( zb/r ) I ( r2 ∈ [ a , b ] ) is L1 ) . Finally , we construct a dominating integrable function . Observe ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) |φ ( za/r ) φ ( zb/r ) |η ( ‖z‖2 − r2 ) = ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) |φ ( za/r ) φ ( zb/r ) |η ( ‖z‖2 − r2 ) = ∫ dze− 1 2 z TΣ−1z ∫ b a d ( ‖z‖2 + γ ) |φ ( za/ √ ‖z‖2 + γ ) φ ( zb/ √ ‖z‖2 + γ ) |η ( −γ ) = ∫ dze− 1 2 z TΣ−1z ∫ b−‖z‖2 a−‖z‖2 dγ |φ ( za/ √ ‖z‖2 + γ ) φ ( zb/ √ ‖z‖2 + γ ) |η ( −γ ) = ∫ ‖z‖2∈ [ a− , b+ ] dze− 1 2 z TΣ−1z ∫ b−‖z‖2 a−‖z‖2 dγ |φ ( za/ √ ‖z‖2 + γ ) φ ( zb/ √ ‖z‖2 + γ ) |η ( −γ ) since supp η = [ − , ] ≤ ∫ ‖z‖2∈ [ a− , b+ ] dze− 1 2 z TΣ−1z ∫ − dγ |φ ( za/ √ ‖z‖2 + γ ) φ ( zb/ √ ‖z‖2 + γ ) |η ( −γ ) = ∫ − dγ η ( −γ ) ∫ ‖z‖2∈ [ a− , b+ ] dze− 1 2 z TΣ−1z|φ ( za/ √ ‖z‖2 + γ ) φ ( zb/ √ ‖z‖2 + γ ) | For small enough then , this is integrable by assumption ( 2 ) , and yields a dominating integrable function for our application of dominated convergence . In summary , we have just proven that ∫ dze− 1 2 z TΣ−1zφ ( z/‖z‖ ) ⊗2 = lim a↘0 , b↗∞ lim ↘0 ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) η ( ‖z‖2 − r2 ) Now , ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) η ( ‖z‖2 − r2 ) = ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) 1 2π ∫ ∞ −∞ dλ η̂ ( λ ) e −iλ ( ‖z‖2−r2 ) = 1 2π ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) ∫ ∞ −∞ dλ η̂ ( λ ) e−iλ ( ‖z‖ 2−r2 ) Note that the absolute value of the integral is bounded above by 1 2π ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) |φ ( za/r ) φ ( zb/r ) | ∫ ∞ −∞ dλ |η̂ ( λ ) e−iλ ( ‖z‖2−r2 ) | = 1 2π ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) |φ ( za/r ) φ ( zb/r ) | ∫ ∞ −∞ dλ |η̂ ( λ ) | ≤ C ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) |φ ( za/r ) φ ( zb/r ) | for some C , by our construction of η that η̂ ( t ) = O ( t−2 ) for large |t| . By assumption ( 3 ) , this integral exists . Therefore we can apply the Fubini-Tonelli theorem and swap order of integration . 1 2π ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) ∫ ∞ −∞ dλ η̂ ( λ ) e−iλ ( ‖z‖ 2−r2 ) = 1 2π ∫ b a d ( r2 ) ∫ ∞ −∞ dλ η̂ ( λ ) e−iλ ( ‖z‖ 2−r2 ) ∫ dzφ ( za/r ) φ ( zb/r ) e − 12 z TΣ−1z = 1 2π ∫ b a d ( r2 ) ∫ ∞ −∞ dλ η̂ ( λ ) eiλr 2 ∫ dzφ ( za/r ) φ ( zb/r ) e − 12 z T ( Σ−1+2iλI ) z = 1 2π ∫ b a d ( r2 ) rD ∫ ∞ −∞ dλ η̂ ( λ ) eiλr 2 ∫ dzφ ( za ) φ ( zb ) e − 12 r 2zT ( Σ−1+2iλI ) z = ( 2π ) D/2−1 det Σ1/2 ∫ b a d ( r2 ) ∫ ∞ −∞ dλ η̂ ( λ ) eiλr 2 det ( I + 2iλΣ ) −1/2Vφ ( ( Σ −1 + 2iλI ) −1r−2 ) By assumption ( 4 ) , ∫ b a d ( r2 ) ∫ ∞ −∞ dλ ∣∣∣η̂ ( λ ) eiλr2 det ( I + 2iλΣ ) −1/2Vφ ( ( Σ−1 + 2iλI ) −1r−2 ) ∣∣∣ ≤ C ∫ b a d ( r2 ) ∫ ∞ −∞ dλ ∣∣∣det ( I + 2iλΣ ) −1/2Vφ ( ( Σ−1 + 2iλI ) −1r−2 ) ∣∣∣ is integrable ( where we used the fact that η̂ is bounded — which is true for all η ∈ C∞0 ( R ) ) , so by dominated convergence ( applied to ↘ 0 ) , ∫ dze− 1 2 z TΣ−1zφ ( z/‖z‖ ) ⊗2 = lim a↘0 , b↗∞ lim ↘0 ∫ dze− 1 2 z TΣ−1z ∫ b a d ( r2 ) φ ( za/r ) φ ( zb/r ) η ( ‖z‖2 − r2 ) = ( 2π ) D/2−1 det Σ1/2 lim a↘0 , b↗∞ ∫ b a d ( r2 ) ∫ ∞ −∞ dλ eiλr 2 det ( I + 2iλΣ ) −1/2Vφ ( ( Σ −1 + 2iλI ) −1r−2 ) where we used the fact that η̂ ( 0 ) = ∫ R η ( x ) dx = 1 by construction . This gives us the desired result after putting back in some constants and changing r2 7→ s. E.3 SPHERICAL INTEGRATION As mentioned above , the scale invariance of batchnorm combined with the BSB1 fixed point assumption will naturally lead us to consider spherical integration . Here we review several basic facts . Definition E.26 . We define the spherical angular coordinates in RB , ( r , θ1 , . . . , θB−2 ) ∈ [ 0 , ∞ ) × A = : [ 0 , π ] B−3 × [ 0 , 2π ] , by x1 = r cos θ1 x2 = r sin θ1 cos θ2 x3 = r sin θ1 sin θ2 cos θ3 ... xB−2 = r sin θ1 · · · sin θB−3 cos θB−2 xB−1 = r sin θ1 · · · sin θB−3 sin θB−2 . Then we set v : = cos θ1 sin θ1 cos θ2 sin θ1 sin θ2 cos θ3 ... sin θ1 · · · sin θB−3 cos θB−2 sin θ1 · · · sin θB−3 sin θB−2 as the unit norm version of x . The integration element in this coordinate satisfies dx1 . . . dxB−1 = r B−2 sinB−3 θ1 sin B−4 θ2 · · · sin θB−3 dr dθ1 · · · dθB−2 . Fact E.27 . If F : SB−2 → R , then its expectation on the sphere can be written as E v∼SB−2 F ( v ) = 1 2 π 1−B 2 Γ ( B − 1 2 ) ∫ A F ( v ) sinB−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 where A = [ 0 , π ] B−3 × [ 0 , 2π ] and v are as in Defn E.26 . The following integrals will be helpful for simplifying many expressions involving spherical integration using angular coordinates . Lemma E.28 . For j , k ≥ 0 and 0 ≤ s ≤ t ≤ π/2 , ∫ t s sinj θ cosk θ dθ = 1 2 Beta ( cos2 t , cos2 s ; k + 1 2 , j + 1 2 ) . By antisymmetry of cos with respect to θ 7→ π − θ , if π/2 ≤ s ≤ t ≤ π , ∫ t s sinj θ cosk θ dθ = 1 2 ( −1 ) k Beta ( cos2 s , cos2 t ; k + 1 2 , j + 1 2 ) . Proof . Set x : = cos2 θ =⇒ dx = −2 cos θ sin θ dθ . So the integral in question is −1 2 ∫ cos2 t cos2 s ( 1− x ) j−12 x k−12 dx = 1 2 ( Beta ( cos2 s ; k + 1 2 , j + 1 2 ) − Beta ( cos2 t ; k + 1 2 , j + 1 2 ) ) As consequences , Lemma E.29 . For j , k ≥ 0 , ∫ π 0 sinj θ cosk θ dθ = 1 + ( −1 ) k 2 Beta ( j + 1 2 , k + 1 2 ) = 1 + ( −1 ) k 2 Γ ( 1+j2 ) Γ ( 1+k 2 ) Γ ( 2+j+k2 ) . Lemma E.30 . ∫ π 0 sin θ1 dθ1 ∫ π 0 sin2 θ2 dθ2 · · · ∫ π 0 sink θk dθk = π k/2/Γ ( k + 2 2 ) . Proof . By Lemma E.29 , ∫ π 0 sinr θr dθr = Beta ( r+1 2 , 1 2 ) = Γ ( r+12 ) √ π Γ ( r+22 ) . Thus this product of inte- grals is equal to k∏ r=1 √ π Γ ( r+12 ) Γ ( r+22 ) = πk/2Γ ( 1 ) /Γ ( k + 2 2 ) = πk/2/Γ ( k + 2 2 ) . Lemma E.31 . For B > 1 , µ > 0 , ∫∞ 0 rBe−r 2/2µ dr = 2 B−1 2 µ B+1 2 Γ ( B+12 ) Proof . Apply change of coordinates z = r2/2µ . Definition E.32 . Let K ( v ; Σ , β ) : = ( det eTΣe ) −1/2 ( vT ( eTΣe ) −1v ) −B−1+β2 and K ( v ; Σ ) : = K ( v ; Σ , 0 ) . The following proposition allows us to express integrals involving batchnorm as expectations over the uniform spherical measure . Proposition E.33 . If f : RB−1 → RB′ , B > 1− β , and Σ ∈ R ( B−1 ) × ( B−1 ) is nonsingular , then E h∼N ( 0 , Σ ) ‖h‖βf ( h/‖h‖ ) = 2β/2P ( B − 1 2 , β 2 ) E v∼SB−2 f ( v ) ( det Σ ) −1/2 ( vTΣ−1v ) − B−1+β 2 . If f : RB → RB′ and Σ ∈ RB×B is such that eTΣe is nonsingular ( where e is as in Defn E.3 ) and GΣG = Σ , E h∼N ( 0 , Σ ) ‖Gh‖βf ( Gh/‖Gh‖ ) = 2β/2P ( B − 1 2 , β 2 ) E v∼SB−2 f ( ev ) K ( v ; Σ , β ) . In particular , E [ f ( B ( h ) ) : h ∼ N ( 0 , Σ ) ] = E v∼SB−2 f ( ev ) K ( v ; Σ ) Proof . We will prove the second statement . The others follow trivially . E [ ‖Gh‖βf ( B ( h ) ) : h ∼ N ( 0 , Σ ) ] = E [ ‖x‖βf ( ex/‖x‖ ) : x ∼ N ( 0 , eTΣe ) ] = ( 2π ) 1−B 2 ( det eTΣe ) −1/2 ∫ RB−1 ‖x‖βf ( ex/‖x‖ ) e− 12xT ( eTΣe ) −1x dx = ( 2π ) 1−B 2 ( det eTΣe ) −1/2 ∫ A f ( ev ) sinB−3 θ1 · · · sin θB−3 dr dθ1 · · · dθB−2×∫ ∞ 0 rB−2+βe− 1 2 r 2·vT ( eTΣe ) −1v ( spherical coordinates Defn E.26 ) = ( 2π ) 1−B 2 ( det eTΣe ) −1/2 × 2B−32 Γ ( B − 1 2 ) × 2 B−3+β 2 Γ ( B−1+β2 ) 2 B−3 2 Γ ( B−1 2 ) × ∫ A f ( ev ) sinB−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 ( vT ( eTΣe ) −1v ) B−1+β 2 = 1 2 π 1−B 2 Γ ( B − 1 2 ) × 2β/2Γ ( B − 1 + β 2 ) Γ ( B − 1 2 ) −1 × ∫ A f ( ev ) sinB−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 ( vT ( eTΣe ) −1v ) B−1+β 2 = 2β/2Γ ( B − 1 + β 2 ) Γ ( B − 1 2 ) −1 E v∼SB−2 f ( ev ) ( det eTΣe ) 1/2 ( vT ( eTΣe ) −1v ) B−1+β 2 = 2β/2Γ ( B − 1 + β 2 ) Γ ( B − 1 2 ) −1 E v∼SB−2 f ( ev ) K ( v ; Σ , β ) As a straightforward consequence , Proposition E.34 . If eTΣe is nonsingular , VBφ ( Σ ) = E v∼SB−2 φ ( √ Bev ) ⊗2K ( v ; Σ ) For numerical calculations , it is useful to realize e as the matrix whose columns are eB−m : = ( m ( m+ 1 ) ) −1/2 ( 0 , . . . , 0 , −m , m︷ ︸︸ ︷ 1 , 1 , . . . , 1 ) T , for each m = 1 , . . . , B − 1. e T = 1√ B ( B−1 ) 1√ ( B−1 ) ( B−2 ) . . . 6−1/2 2−1/2 × − ( B − 1 ) 1 1 · · · 1 1 1 0 − ( B − 2 ) 1 · · · 1 1 1 ... ... ... . . . ... ... ... 0 0 0 · · · −2 1 1 0 0 0 · · · 0 −1 1 ( 33 ) We have ( ev ) 1 = − √ B − 1 B cos θ1 ( ev ) 2 = cos θ1 − √ B − 2 B − 1 sin θ1 cos θ2 ( ev ) 3 = cos θ1 + sin θ1 cos θ2 − √ B − 3 B − 2 sin θ1 sin θ2 cos θ3 ( ev ) 4 = cos θ1 + sin θ1 cos θ2 + sin θ1 sin θ2 cos θ3 − √ B − 4 B − 3 sin θ1 sin θ2 sin θ3 cos θ4 ( 34 ) so in particular they only depend on θ1 , . . . , θ4 . Angular coordinates In many situations , the sparsity of this realization of e combined with sufficient symmetry allows us to simplify the high-dimensional spherical integral into only a few dimensions , using one of the following lemmas Lemma E.35 . Let f : R4 → RA for some A ∈ N. Suppose B ≥ 6 . Then for any k < B − 1 E [ r−kf ( v1 , . . . , v4 ) : x ∼ N ( 0 , IB−1 ) ] = Γ ( ( B − 1− k ) /2 ) Γ ( ( B − 5 ) /2 ) 2 −k/2π−2 ∫ π 0 dθ1 · · · ∫ π 0 dθ4 f ( v θ 1 , . . . , v θ 4 ) sin B−3 θ1 · · · sinB−6 θ4 where vi = xi/‖x‖ and r = ‖x‖ , and vθ1 = cos θ1 vθ2 = sin θ1 cos θ2 vθ3 = sin θ1 sin θ2 cos θ3 vθ4 = sin θ1 sin θ2 sin θ3 cos θ4 . Lemma E.36 . Let f : R2 → RA for some A ∈ N. Suppose B ≥ 4 . Then for any k < B − 1 E [ r−kf ( v1 , v2 ) : x ∼ N ( 0 , IB−1 ) ] = Γ ( ( B − 1− k ) /2 ) Γ ( ( B − 3 ) /2 ) 2 −k/2π−1 ∫ π 0 dθ1 ∫ π 0 dθ2 f ( v θ 1 , v θ 2 ) sin B−3 θ1 sin B−4 θ2 where vi = xi/‖x‖ and r = ‖x‖ , and vθ1 = cos θ1 vθ2 = sin θ1 cos θ2 Lemma E.37 . Let f : R→ RA for some A ∈ N. Suppose B ≥ 3 . Then for any k < B − 1 E [ r−kf ( v ) : x ∼ N ( 0 , IB−1 ) ] = Γ ( ( B − 1− k ) /2 ) Γ ( ( B − 2 ) /2 ) 2 −k/2π−1/2 ∫ π 0 dθ f ( cos θ ) sinB−3 θ where vi = xi/‖x‖ and r = ‖x‖ . We will prove Lemma E.35 ; those of Lemma E.36 and Lemma E.37 are similar . Proof of Lemma E.35 . E [ r−kf ( v1 , . . . , v4 ) : x ∼ N ( 0 , IB−1 ) ] = ( 2π ) − ( B−1 ) /2 ∫ dx r−kf ( v1 , . . . , v4 ) e −r2/2 = ( 2π ) − ( B−1 ) /2 ∫ ( rB−2 sinB−3 θ1 · · · sin θB−3 dr dθ1 · · · dθB−2 ) r−kf ( vθ1 , . . . , v θ 4 ) e −r2/2 = ( 2π ) − ( B−1 ) /2 ∫ f ( vθ1 , . . . , v θ 4 ) sin B−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 ∫ ∞ 0 rB−2−ke−r 2/2 dr = ( 2π ) − ( B−1 ) /2 ∫ f ( vθ1 , . . . , v θ 4 ) sin B−3 θ1 · · · sin θB−3 dθ1 · · · dθB−22 ( B−3−k ) /2Γ ( ( B − 1− k ) /2 ) ( change of coordinate r ← r2 and definition of Γ function ) = Γ ( ( B − 1− k ) /2 ) 2− ( 2+k ) /2π− ( B−1 ) /2 ∫ f ( vθ1 , . . . , v θ 4 ) sin B−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 Because vθ1 , . . . , v θ 4 only depends on θ1 , . . . , θ4 , we can integrate out θ5 , . . . , θB−2 . By applying Lemma E.30 to θ5 , . . . θB−3 and noting that ∫ 2π 0 dθB−2 = 2π , we get E [ r−kf ( vθ1 , . . . , vθ4 ) : x ∼ N ( 0 , IB−1 ) ] = Γ ( ( B − 1− k ) /2 ) 2− ( 2+k ) /2π− ( B−1 ) /2 ∫ f ( vθ1 , . . . , v θ 4 ) sin B−3 θ1 · · · sinB−6 θ4π ( B−7 ) /2Γ ( ( B − 5 ) /2 ) −1 ( 2π ) = Γ ( ( B − 1− k ) /2 ) Γ ( ( B − 5 ) /2 ) 2 −k/2π−2 ∫ f ( vθ1 , . . . , v θ 4 ) sin B−3 θ1 · · · sinB−6 θ4 Cartesian coordinates . We can often also simplify the high dimensional spherical integrals without trigonometry , as the next two lemmas show . Both are proved easily using change of coordinates . Lemma E.38 . For any u ∈ Sn , E v∼Sn f ( u · v ) = ωn−1 ωn ∫ 1 −1 dh ( 1− h2 ) n−22 f ( h ) where ωn−1 is hypersurface of the ( n− 1 ) -dimensional unit hypersphere . Lemma E.39 . For any u1 , u2 ∈ Sn , E v∼Sn f ( u1 · v , u2 · v ) = ωn−2 ωn ∫ ‖v‖≤1 v∈span ( u1 , u2 ) dv ( 1− ‖v‖2 ) n−32 f ( u1 · v , u2 · v ) where ωn−1 is hypersurface of the ( n− 1 ) -dimensional unit hypersphere . E.4 GEGENBAUER EXPANSION Definition E.40 . The Gegenbauer polynomials { C ( α ) l ( x ) } ∞l=0 , with degC ( α ) l ( x ) = l , are the set of orthogonal polynomials with respect to the weight function ( 1 − x2 ) α−1/2 ( i.e . ∫ 1 −1 ( 1 − x2 ) α−1/2C ( α ) l ( x ) C ( α ) m ( x ) = 0 if l 6= m ) . By convention , they are normalized so that∫ 1 −1 [ C ( α ) n ( x ) ] 2 ( 1− x2 ) α− 12 dx = π2 1−2αΓ ( n+ 2α ) n ! ( n+ α ) Γ ( α ) 2 . Here are several examples of low degree Gegenbauer polynomials C ( α ) 0 ( x ) = 1 C ( α ) 1 ( x ) = 2αx C ( α ) 2 ( x ) = −α+ 2α ( 1 + α ) x2 C ( α ) 3 ( x ) = −2α ( 1 + α ) x+ 4 3 α ( 1 + α ) ( 2 + α ) x3 . They satisfy a few identities summarized below Fact E.41 ( Suetin ) . C ( α ) n ( ±1 ) = ( ±1 ) n ( n+2α−1 n ) Fact E.42 ( Weisstein ) . C ( λ ) n ′ ( x ) = 2λC ( λ+1 ) n−1 ( x ) ( 35 ) xC ( λ ) n ′ ( x ) = nC ( λ ) n ( x ) + C ( λ ) n−1 ′ ( x ) ( 36 ) C ( λ ) n+1 ′ ( x ) = ( n+ 2λ ) C ( λ ) n ( x ) + xC ( λ ) n ′ ( x ) ( 37 ) nC ( λ ) n ( x ) = 2 ( n− 1 + λ ) xC ( λ ) n−1 ( x ) − ( n− 2 + 2λ ) C ( λ ) n−2 ( x ) , ∀n ≥ 2 ( 38 ) By repeated applications of Eq . ( 38 ) , Proposition E.43 . xC ( λ ) n ( x ) = 1 2 ( n+ λ ) ( ( n+ 1 ) C ( λ ) n+1 ( x ) + ( n− 1 + 2λ ) C ( λ ) n−1 ( x ) ) , ∀n ≥ 1 x2C ( λ ) n ( x ) = κ2 ( n , λ ) C ( λ ) n+2 ( x ) + κ0 ( n , λ ) C ( λ ) n ( x ) + κ−2 ( n , λ ) C ( λ ) n−2 ( x ) , ∀n ≥ 2 xC ( λ ) 0 ( x ) = x = 1 2λ C ( λ ) 1 ( x ) x2C ( λ ) 0 ( x ) = 1 4λ ( 1 + λ ) ( 2C ( λ ) 2 ( x ) + 2λC ( λ ) 0 ( x ) ) = κ2 ( 0 , λ ) C ( λ ) 2 ( x ) + κ0 ( 0 , λ ) C ( λ ) 0 ( x ) x2C ( λ ) 1 ( x ) = κ2 ( 1 , λ ) C ( λ ) 3 ( x ) + κ0 ( 1 , λ ) C ( λ ) 1 ( x ) where κ2 ( n , λ ) : = ( n+1 ) ( n+2 ) 4 ( n+λ ) ( n+1+λ ) , κ0 ( n , λ ) : = ( n+1 ) ( n+2λ ) 4 ( n+λ ) ( n+1+λ ) + ( n−1+2λ ) n 4 ( n+λ ) ( n−1+λ ) , κ−2 ( n , λ ) : = ( n−1+2λ ) ( n−2+2λ ) 4 ( n+λ ) ( n−1+λ ) . This proposition is useful for the Gegenbauer expansion of the local convergence rate of Eq . ( 43 ) . Zonal harmonics . The Gegenbauer polynomials are closely related to a special type of spherical harmonics called zonal harmonics , which are intuitively those harmonics that only depend on the “ height ” of the point along some fixed axis . Definition E.44 . Let ZN−1 , ( l ) u denote the degree l zonal spherical harmonics on the ( N − 1 ) - dimensional sphere with axis u ∈ SN−1 , defined by ZN−1 , ( l ) u ( v ) = c−1N , lC ( N−22 ) l ( u · v ) for any u , v ∈ SN−1 , where cN , l = N−2N+2l−2 . The first few zonal spherical harmonics are ZN−1 , ( 0 ) u ( v ) = 1 , Z N−1 , ( 1 ) u ( v ) = Nu · v , . . . . We note a trivial lemma that is useful for simplying many expressions in Gegenbauer expansions . Lemma E.45 . cB+1 , l−1cB−1 , l = B−1 B−3 . Let 〈− , −〉SN denote the inner product induced by the uniform measure on SN . One of the most important properties of zonal harmonics is their Fact E.46 ( Reproducing property ( Suetin ) ) . 〈ZN−1 , ( l ) u , ZN−1 , ( l ) v 〉SN−1 = ZN−1 , ( l ) u ( v ) = c−1N , lC ( N−22 ) l ( u · v ) 〈ZN−1 , ( l ) u , ZN−1 , ( m ) v 〉SN−1 = 0 , if l 6= m. In our applications of the reproducing property , u and v will always be êa , : : = ea , :/‖ea , :‖ = ea , : √ B B−1 , where ea , : is the ath row of e. Next , we present a new ( to the best of our knowledge ) identity showing that a certain quadratic form of φ , reminiscent of Dirichlet forms , depending only on the derivative φ′ through a spherical integral , is diagonalized in the Gegenbauer basis . This will be crucial to proving the necessity of gradient explosion under the BSB1 fixed point assumption . Theorem E.47 ( Dirichlet-like form of φ diagonalizes over Gegenbauer basis ) . Let u1 , u2 ∈ SB−2 . Suppose φ ( √ B − 1x ) , φ′ ( √ B − 1x ) ∈ L2 ( ( 1 − x2 ) B−32 ) and φ ( √ B − 1x ) has Gegenbauer ex- pansion ∑∞ l=0 al 1 cB−1 , l C ( B−32 ) l ( x ) . Then E v∈SB−2 ( u1 · u2 − ( u1 · v ) ( u2 · v ) ) φ′ ( √ B − 1u1 · v ) φ′ ( √ B − 1u2 · v ) = ∞∑ l=0 a2l ( l +B − 3 ) lc−1B−1 , lC ( B−32 ) l ( u1 · u2 ) This is proved using the following lemmas . Lemma E.48 . xC ( α ) l ′ ( x ) = lC ( α ) l ( x ) + ∑ j∈ [ l−2 ] 2 2 ( j + α ) C ( α ) j ( x ) C ( α ) l ′ ( x ) = ∑ j∈ [ l−1 ] 2 2 ( j + α ) C ( α ) j ( x ) where [ n ] 2 is the sequence { ( n % 2 ) , ( n % 2 ) + 1 , . . . , n− 2 , n } and n % 2 is the remainder of n/2 . Proof . Apply Eq . ( 36 ) and Eq . ( 37 ) alternatingly . Lemma E.49 . lC ( α ) l ( t ) + ∑ j∈ [ l−2 ] 2 2 ( j + α ) C ( α ) j ( t ) − t ∑ j∈ [ l−1 ] 2 2 ( j + α ) C ( α ) j ( t ) = 0 Proof . Apply Eq . ( 38 ) to lC ( α ) l ( t ) , ( l − 2 ) C ( α ) l−2 ( t ) , . . .. Proof of Thm E.47 . We have φ′ ( √ B − 1x ) = 1√ B − 1 ∞∑ l=0 al 1 cB−1 , l C ( B−32 ) l ′ ( x ) = 1√ B − 1 ∞∑ l=0 al 1 cB−1 , l ∑ j∈ [ l−1 ] 2 ( 2j +B − 3 ) C ( B−3 2 ) j ( x ) = 1√ B − 1 ∞∑ l=0 ∑ j∈ [ l−1 ] 2 al cB−1 , j cB−1 , l ( 2j +B − 3 ) c−1B−1 , jC ( B−32 ) j ( x ) = 1√ B − 1 ∞∑ j=0 c−1B−1 , jC ( B−32 ) j ( x ) ∑ l∈ [ j+1 , ∞ ] 2 al cB−1 , j cB−1 , l ( 2j +B − 3 ) where [ n , m ] 2 is the sequence n , n+ 2 , . . . , m. So for any u1 , u2 ∈ SB−2 , E v∈SB−2 φ′ ( √ B − 1u1 · v ) φ′ ( √ B − 1u2 · v ) = 1 B − 1 ∞∑ j=0 c−1B−1 , jC ( B−32 ) j ( u1 · u2 ) ∑ l∈ [ j+1 , ∞ ] 2 al cB−1 , j cB−1 , l ( 2j +B − 3 ) 2 = 1 B − 1 ∞∑ j=0 cB−1 , j ( 2j +B − 3 ) 2C ( B−3 2 ) j ( u1 · u2 ) ∑ l , m∈ [ j+1 , ∞ ] 2 al cB−1 , l am cB−1 , m . For l < m and m− l even , the coefficient of alam is 2c−1B−1 , lc −1 B−1 , m B − 1 ∑ j∈ [ l−1 ] 2 cB−1 , j ( 2j +B − 3 ) 2C ( B−3 2 ) j ( u1 · u2 ) = 2c−1B−1 , lc −1 B−1 , m B − 1 ∑ j∈ [ l−1 ] 2 ( B − 3 ) ( 2j +B − 3 ) C ( B−3 2 ) j ( u1 · u2 ) . ( 39 ) For each l , the coefficient of a2l is c−2B−1 , l B − 1 ∑ j∈ [ l−1 ] 2 cB−1 , j ( 2j +B − 3 ) 2C ( B−3 2 ) j ( u1 · u2 ) = c−2B−1 , l B − 1 ∑ j∈ [ l−1 ] 2 ( B − 3 ) ( 2j +B − 3 ) C ( B−3 2 ) j ( u1 · u2 ) . ( 40 ) Similarly , if φ′ ( x ) ∈ L2 ( ( 1− x2 ) B−32 ) , we have xφ′ ( √ B − 1x ) = 1√ B − 1 ∞∑ l=0 al 1 cB−1 , l xC ( B−32 ) l ′ ( x ) = 1√ B − 1 ∞∑ l=0 al 1 cB−1 , l lC ( B−32 ) l ( x ) + ∑ j∈ [ l−2 ] 2 ( 2j +B − 3 ) C ( B−3 2 ) j ( x ) = 1√ B − 1 ∞∑ j=0 c−1B−1 , jC ( B−32 ) j ( x ) ajj + ∑ l∈ [ j+2 , ∞ ] 2 al cB−1 , j cB−1 , l ( 2j +B − 3 ) Then E v∈SB−2 ( u1 · v ) ( u2 · v ) φ′ ( √ B − 1u1 · v ) φ′ ( √ B − 1u2 · v ) = 1 B − 1 ∞∑ j=0 c−1B−1 , jC ( B−32 ) j ( u1 · u2 ) ajj + ∑ l∈ [ j+2 , ∞ ] 2 al cB−1 , j cB−1 , l ( 2j +B − 3 ) 2 For l < m and m− l even , the coefficient of alam is 1 B − 1 ( 2l ( 2l +B − 3 ) cB−1 , l cB−1 , m c−1B−1 , lC ( B−32 ) l ( u1 · u2 ) +2 ∑ j∈ [ l−2 ] 2 cB−1 , j cB−1 , l cB−1 , j cB−1 , m ( 2j +B − 3 ) 2c−1B−1 , jC ( B−32 ) j ( u1 · u2 ) = 2c−1B−1 , lc −1 B−1 , m B − 3 B − 1 lC ( B−32 ) l ( u1 · u2 ) + ∑ j∈ [ l−2 ] 2 ( 2j +B − 3 ) C ( B−3 2 ) j ( u1 · u2 ) ( 41 ) For each l , the coefficient of a2l is 1 B − 1 l2c−1B−1 , lC ( B−32 ) l ( u1 · u2 ) + ∑ j∈ [ l−2 ] 2 ( cB−1 , j cB−1 , l ) 2 ( 2j +B − 3 ) 2c−1B−1 , jC ( B−32 ) j ( u1 · u2 ) = c−2B−1 , l B − 3 B − 1 l2 2l +B − 3C ( B−32 ) l ( u1 · u2 ) + ∑ j∈ [ l−2 ] 2 ( 2j +B − 3 ) C ( B−3 2 ) j ( u1 · u2 ) ( 42 ) By Lemma E.49 and Eqs . ( 39 ) and ( 41 ) , the coefficient of alam withm > l in Ev∈SB−2 ( u1·u2− ( u1· v ) ( u2 ·v ) ) φ′ ( √ B − 1u1 ·v ) φ′ ( √ B − 1u2 ·v ) is 0 . Similarly , by Lemma E.49 and Eqs . ( 40 ) and ( 42 ) , the coefficient of a2l is c −2 B−1 , l B−3 B−1 l+B−3 2l+B−3 lC ( B−32 ) l ( u1 · u2 ) = c−1B−1 , l l+B−3B−1 lC ( B−32 ) l ( u1 · u2 ) . E.5 SYMMETRY AND ULTRASYMMETRY As remarked before , all commonly used nonlinearities ReLU , tanh , and so on induce the dynamics Eq . ( 43 ) to converge to BSB1 fixed points , which we formally define as follows . Definition E.50 . We say a matrix Σ ∈ SB is BSB1 ( short for “ 1-Block Symmetry Breaking ” ) if Σ has one common entry on the diagonal and one common entry on the off-diagonal , i.e . a b b · · · b a b · · · b b a · · · ... ... ... . . . We will denote such a matrix as BSB1 ( a , b ) . Note that BSB1 ( a , b ) can be written as ( a−b ) I+b11T . Its spectrum is given below . Lemma E.51 . AB×B matrix of the form µI+ν11T has two eigenvalues µ andBν+µ , each with eigenspaces { x : ∑i xi = 0 } and { x : x1 = · · · = xB } . Equivalently , if it has a on the diagonal and b on the off-diagonal , then the eigenvalues are a− b and ( B − 1 ) b+ a . The following simple lemma will also be very useful Lemma E.52 . Let Σ : = BSB1B ( a , b ) . Then GΣG = ( a− b ) G. Proof . G and BSB1B ( a , b ) can be simultaneously diagonalized by Lemma E.51 . Note that G zeros out the eigenspace R1 , and is identity on its orthogonal complement . The result then follows from easy computations . The symmetry of BSB1 covariance matrices , especially the fact that they are isotropic in a codimension 1 subspace , is crucial to much of our analysis . Ultrasymmetry . Indeed , as mentioned before , when investigating the asymptotics of the forward or backward dynamics , we encounter 4-tensor objects , such as dVBφ dΣ or VB′φ , that would seem daunting to analyze at first glance . However , under the BSB1 fixed point assumption , these 4- tensors contain considerable symmetry that somehow makes this possible . We formalize the notion of symmetry arising from such scenarios : Definition E.53 . Let T : RB×B → RB×B be a linear operator . Let δi ∈ RB be the vector with 0 everywhere except 1 in coordinate i ; then δiδTj is the matrix with 0 everywhere except 1 in position ( i , j ) . Write [ kl|ij ] : = T ( δiδTj ) kl . Suppose T has the property that for all i , j , k , l ∈ [ B ] = { 1 , . . . , B } • [ kl|ij ] = [ π ( k ) π ( l ) |π ( i ) π ( j ) ] for all permutation π on [ B ] , and • [ ij|kl ] = [ ji|lk ] . Then we say T is ultrasymmetric . Remark E.54 . In what follows , we will often “ normalize ” the representation “ [ ij|kl ] ” to the unique “ [ i′j′|k′l′ ] ” that is in the same equivalence class according to Defn E.53 and such that i′ , j′ , k′ , l′ ∈ [ 4 ] and i′ ≤ j′ , k′ ≤ l′ unless i′ = l′ , j′ = k′ , in which case the normalization is [ 12|21 ] . Explicitly , we have the following equivalence classes and their normalized representations We will study the eigendecomposition of an ultrasymmetric T as well as the projection G⊗2 ◦ T ◦ G⊗2 : Σ 7→ G ( T { GΣG } ) G , respectively with the following domains Definition E.55 . Denote by HB the space of symmetric matrices of dimension B . Also write HGB for the space of symmetric matrices Σ of dimension B such that GΣG = Σ ( which is equivalent to saying rows of Σ sum up to 0 ) . As in the case of S , we omit subscript B when it ’ s clear from context . The following subspaces ofHB andHGB will turn out to comprise the eigenspaces ofG⊗2◦T ◦G⊗2 . class repr . i = j = k = l [ 11|11 ] i = j = k or i = j = l [ 11|12 ] i = j and k = l [ 11|22 ] i = j [ 11|23 ] i = k = l or j = k = l [ 12|11 ] i = k and j = l [ 12|12 ] i = k or i = l [ 12|13 ] i = l and j = k [ 12|21 ] k = l [ 12|33 ] all different [ 12|34 ] Definition E.56 . Let LB : = { GDG : D diagonal , trD = 0 } ⊆ HGB and MB : = { Σ ∈ HGB : DiagΣ = 0 } . Note that dimLB = B − 1 , dimMB = B ( B−3 ) 2 and RGB ⊕ LB ⊕MB = HGB is an orthogonal decomposition w.r.t Frobenius inner product . For the eigenspaces of T , we also need to define Definition E.57 . For any nonzero a , b ∈ R , set LB ( a , b ) : = a 0 −b −b · · · 0 −a b b · · · −b b 0 0 · · · −b b 0 0 · · · ... ... ... ... . . . ∈ HB . In general , we say a matrix M is LB ( a , b ) -shaped if M = PLB ( a , b ) PT for some permutation matrix P . Note that Proposition E.58 . L ( B − 2 , 1 ) -shaped matrices span LB . Proposition E.59 . L ( B − 2 , 1 ) = 1B [ ( −B + 1 , 1 , 1 , . . . , 1 ) ⊗2 − ( 1 , −B + 1 , 1 , . . . , 1 ) ⊗2 ] = BG [ ( 1 , 0 , 0 , . . . , 0 ) ⊗2 − ( 0 , 1 , 0 , . . . , 0 ) ⊗2 ] G Proof . Straightforward computation . Lemma E.60 . Let L = LB ( a , b ) . Then G⊗2 { L } = GLG = a+2bB LB ( B − 2 , 1 ) . Proof . GLG can be written as the sum of outer products GLG = aG1 ⊗G1 − aG2 ⊗G2 + B∑ i=3 −bG1 ⊗Gi + bG2 ⊗Gi − bGi ⊗G1 + bGi ⊗G2 = aLB ( ( B − 1 B ) 2 − 1 B2 , B B2 ) + b B∑ i=3 ( −δ1 + δ2 ) ⊗Gi +Gi ⊗ ( −δ1 + δ2 ) = a B LB ( B − 2 , 1 ) + b ( ( −δ1 + δ2 ) ⊗ v + v ⊗ ( −δ1 + δ2 ) ) with v = ( −B − 2 B , −B − 2 B , 2 B , 2 B , . . . ) = a B LB ( B − 2 , 1 ) + 2b B LB ( B − 2 , 1 ) = a+ 2b B LB ( B − 2 , 1 ) We are now ready to discuss the eigendecomposition of ultrasymmetric operators . Theorem E.61 ( Eigendecomposition of an ultrasymmetric operator ) . Let T : RB×B → RB×B be an ultrasymmetric linear operator . Then T has the following eigendecomposition . 1 . Two 1-dimensional eigenspace R ·BSB1 ( λTBSB1 , i − α22 , α21 ) with eigenvalue λTBSB1 , i for i = 1 , 2 , where α11 = [ 11|11 ] + [ 11|22 ] ( B − 1 ) α12 = 2 ( B − 1 ) [ 11|12 ] + ( B − 2 ) ( B − 1 ) [ 11|23 ] α21 = 2 [ 12|11 ] + ( B − 2 ) [ 12|33 ] α22 = [ 12|12 ] + 4 ( B − 2 ) [ 12|13 ] + [ 12|21 ] + ( B − 2 ) ( B − 3 ) [ 12|34 ] and λTBSB1,1 and λ T BSB1,2 are the roots to the quadratic x2 − ( α11 + α22 ) x+ α11α22 − α12α21 . 2 . Two ( B − 1 ) -dimensional eigenspaces SB · L ( λTL , i − β22 , β21 ) with eigenvalue λ T L , i for i = 1 , 2 . Here SB ·W denotes the linear span of the orbit of matrixW under simultaneous permutation of its column and rows ( by the same permutation ) , and β11 = [ 11|11 ] − [ 11|22 ] β12 = 2 ( B − 2 ) ( [ 11|23 ] − [ 11|12 ] ) β21 = − [ 12|11 ] + [ 12|33 ] β22 = [ 12|21 ] + [ 12|12 ] + 2 ( B − 4 ) [ 12|13 ] − 2 ( B − 3 ) [ 12|34 ] . and λTL,1 and λ T L,2 are the roots to the quadratic x2 − ( β11 + β22 ) x+ β11β22 − β12β21 . 3 . Eigenspace M ( dimension B ( B − 3 ) /2 ) with eigenvalue λTM : = [ 12|12 ] + [ 12|21 ] − 4 [ 12|13 ] + 2 [ 12|34 ] . The proof is by careful , but ultimately straightforward , computation . Proof . We will use the bracket notation of Defn E.53 to denote entries of T , and implicitly simplify it according to Remark E.54 . Item 1 . Let U ∈ RB×B be the BSB1 matrix . By ultrasymmetry of T and BSB1 symmetry of A , T { U } is also BSB1 . So we proceed to calculate the diagonal and off-diagonal entries of T { G } . We have T { BSB1 ( a , b ) } 11 = [ 11|11 ] a+ 2 ( B − 1 ) [ 11|12 ] b+ [ 11|22 ] ( B − 1 ) a+ [ 11|23 ] ( B − 2 ) ( B − 1 ) b = ( [ 11|11 ] + [ 11|22 ] ( B − 1 ) ) a+ ( 2 ( B − 1 ) [ 11|12 ] + ( B − 2 ) ( B − 1 ) [ 11|23 ] ) b T { BSB1 ( a , b ) } 12 = [ 12|12 ] b+ 2 [ 12|11 ] a+ ( B − 2 ) [ 12|33 ] a+ 2 ( B − 2 ) [ 12|13 ] b + [ 12|21 ] b+ 2 ( B − 2 ) [ 12|23 ] b+ ( B − 2 ) ( B − 3 ) [ 12|34 ] b = ( 2 [ 12|11 ] + ( B − 2 ) [ 12|33 ] ) a + ( [ 12|12 ] + 2 ( B − 2 ) [ 12|13 ] + [ 12|21 ] + 2 ( B − 2 ) [ 12|13 ] + ( B − 2 ) ( B − 3 ) [ 12|34 ] ) b = ( 2 [ 12|11 ] + ( B − 2 ) [ 12|33 ] ) a + ( [ 12|12 ] + 4 ( B − 2 ) [ 12|13 ] + [ 12|21 ] + ( B − 2 ) ( B − 3 ) [ 12|34 ] ) b Thus BSB1 ( ω1 , γ1 ) and BSB1 ( ω2 , γ2 ) are the eigenmatrices of T , where ( ω1 , γ1 ) and ( ω2 , γ2 ) are the eigenvectors of the matrix ( α11 α12 α21 α22 ) with α11 = [ 11|11 ] + [ 11|22 ] ( B − 1 ) α12 = 2 ( B − 1 ) [ 11|12 ] + ( B − 2 ) ( B − 1 ) [ 11|23 ] α21 = 2 [ 12|11 ] + ( B − 2 ) [ 12|33 ] α22 = [ 12|12 ] + 4 ( B − 2 ) [ 12|13 ] + [ 12|21 ] + ( B − 2 ) ( B − 3 ) [ 12|34 ] The eigenvalues are the two roots λTBSB1,1 , λ T BSB1,2 to the quadratic x2 − ( α11 + α22 ) x+ α11α22 − α12α21 and the corresponding eigenvectors are ( ω1 , γ1 ) = ( λ1 − α22 , α21 ) ( ω2 , γ2 ) = ( λ2 − α22 , α21 ) . Item 2 . We will study the image of LB ( a , b ) ( Defn E.57 ) under T . We have T { L ( a , b ) } 11 = −T { L ( a , b ) } 22 = [ 11|11 ] a+ 2 ( B − 2 ) [ 11|12 ] ( −b ) + [ 11|22 ] ( −a ) + 2 ( B − 2 ) [ 11|23 ] b = ( [ 11|11 ] − [ 11|22 ] ) a+ 2 ( B − 2 ) ( [ 11|23 ] − [ 11|12 ] ) b T { L ( a , b ) } 12 = T { L ( a , b ) } 21 = [ 12|12 ] 0 + [ 12|21 ] 0 + [ 12|11 ] ( a− a ) + [ 12|13 ] ( b− b ) + [ 12|31 ] ( b− b ) + [ 12|33 ] 0 + [ 12|34 ] 0 = 0 T { L ( a , b ) } 33 = T { L ( a , b ) } ii , ∀i ≥ 3 = [ 11|11 ] 0 + [ 11|12 ] ( 2b− 2b ) + [ 11|22 ] ( a− a ) + [ 11|23 ] ( 2 ( B − 3 ) b− 2 ( B − 3 ) b ) = 0 T { L ( a , b ) } 34 = T { L ( a , b ) } ij , ∀i 6= j & i , j ≥ 3 = [ 12|12 ] 0 + [ 12|11 ] 0 + [ 12|13 ] ( 2b− 2b ) + [ 12|21 ] 0 + [ 12|31 ] ( 2b− 2b ) + [ 12|33 ] ( a− a ) + [ 12|34 ] ( 2 ( B − 4 ) b− 2 ( B − 4 ) b ) = 0 T { L ( a , b ) } 13 = T { L ( a , b ) } 1j , ∀j ≥ 3 = T { L ( a , b ) } j1 , ∀j ≥ 3 = [ 12|12 ] ( −b ) + [ 12|13 ] ( B − 3− 1 ) ( −b ) + [ 12|11 ] a+ [ 12|21 ] ( −b ) + [ 12|31 ] ( B − 3− 1 ) ( −b ) + [ 12|33 ] ( −a ) + [ 12|34 ] 2 ( B − 3 ) b = ( [ 12|11 ] − [ 12|33 ] ) a+ ( 2 ( B − 3 ) [ 12|34 ] − 2 ( B − 4 ) [ 12|13 ] − [ 12|12 ] − [ 12|21 ] ) b Thus L ( a , b ) transforms under T by the matrix L ( a , b ) 7→ L ( ( β11 β12 β21 β22 ) ( a b ) ) with β11 = [ 11|11 ] − [ 11|22 ] β12 = 2 ( B − 2 ) ( [ 11|23 ] − [ 11|12 ] ) β21 = − [ 12|11 ] + [ 12|33 ] β22 = [ 12|21 ] + [ 12|12 ] + 2 ( B − 4 ) [ 12|13 ] − 2 ( B − 3 ) [ 12|34 ] . So if λTL,1 and λ T L,2 are the roots of the equation x2 − ( β11 + β22 ) x+ β11β22 − β12β21 then T { L ( λTL,1 − β22 , β21 ) } = λ T L,1L ( λ T L,1 − β22 , β21 ) T { L ( λTL,2 − β22 , β21 ) } = λ T L,2L ( λ T L,2 − β22 , β21 ) Similarly , any image of these eigenvectors under simultaneous permutation of rows and columns remains eigenvectors with the same eigenvalue . This derives Item 2 . Item 3 . Let M ∈M . We first show that T { M } has zero diagonal . We have T { M } 11 = [ 11|12 ] ( B∑ i=2 M1i +Mi1 ) + [ 11|23 ] ( B∑ i , j=1 Mij − ( B∑ i=2 M1i +Mi1 ) ) = 0 + 0 = 0 which follows from M1 = 0 by definition of M. Similarly T { M } ii = 0 for all i . Now we show that M is an eigenmatrix . T { M } 12 = [ 12|12 ] M12 + [ 12|11 ] 0 + [ 12|33 ] 0 + [ 12|13 ] ( B∑ i=3 M1i + B∑ i=1 Mi2 ) + [ 12|21 ] M21 + [ 12|31 ] ( B∑ i=3 Mi1 + B∑ i=1 M2i ) + [ 12|34 ] ∑ i≥3 , j≥3 Mij = [ 12|12 ] M12 − 2M12 [ 12|13 ] +M21 [ 12|21 ] + [ 12|31 ] ( −2M12 ) + [ 12|34 ] ( B∑ i=3 −Mi1 −M1i ) = M12 ( [ 12|12 ] − 2 [ 12|13 ] + [ 12|21 ] − 2 [ 12|31 ] + 2 [ 12|34 ] ) = M12 ( [ 12|12 ] + [ 12|21 ] − 4 [ 12|13 ] + 2 [ 12|34 ] ) = λTMM12 Similarly T { M } ij = λTMMij for all i 6= j . Note that the eigenspaces described above are in general not orthogonal under trace inner product , so T is not self-adjoint ( relative to the trace inner product ) typically . However , as we see next , after projection by G⊗2 , it is self-adjoint . Theorem E.62 ( Eigendecomposition of a projected ultrasymmetric operator ) . Let T : RB×B → RB×B be an ultrasymmetric linear operator . We write G⊗2 ◦ T HGB for the operator HGB → HGB , Σ 7→ T { Σ } 7→ G ( T { Σ } ) G , restricted to Σ ∈ HGB . Then G⊗2 ◦ T HGB has the following eigendecomposition . 1 . Eigenspace RG with eigenvalue λG , TG : = B−1 ( ( B − 1 ) ( α11 − α21 ) − ( α12 − α22 ) ) , where as in Thm E.61 , α11 = [ 11|11 ] + [ 11|22 ] ( B − 1 ) α12 = 2 ( B − 1 ) [ 11|12 ] + ( B − 2 ) ( B − 1 ) [ 11|23 ] α21 = 2 [ 12|11 ] + ( B − 2 ) [ 12|33 ] α22 = [ 12|12 ] + 4 ( B − 2 ) [ 12|13 ] + [ 12|21 ] + ( B − 2 ) ( B − 3 ) [ 12|34 ] 2 . Eigenspace L with eigenvalue λG , TL : = B−1 ( ( B − 2 ) β11 + β12 + 2 ( B − 2 ) β21 + 2β22 ) , where as in Thm E.61 , β11 = [ 11|11 ] − [ 11|22 ] β12 = 2 ( B − 2 ) ( [ 11|23 ] − [ 11|12 ] ) β21 = − [ 12|11 ] + [ 12|33 ] β22 = [ 12|21 ] + [ 12|12 ] + 2 ( B − 4 ) [ 12|13 ] − 2 ( B − 3 ) [ 12|34 ] . 3 . Eigenspace M with eigenvalue λG , TM : = λTM = [ 12|12 ] + [ 12|21 ] − 4 [ 12|13 ] + 2 [ 12|34 ] . Proof . Item 1 As in the proof of Thm E.61 , we find T { BSB1 ( a , b ) } = BSB1 ( ( α11 α12 α21 α22 ) ( a b ) ) where α11 = [ 11|11 ] + [ 11|22 ] ( B − 1 ) α12 = 2 ( B − 1 ) [ 11|12 ] + ( B − 2 ) ( B − 1 ) [ 11|23 ] α21 = 2 [ 12|11 ] + ( B − 2 ) [ 12|33 ] α22 = [ 12|12 ] + 4 ( B − 2 ) [ 12|13 ] + [ 12|21 ] + ( B − 2 ) ( B − 3 ) [ 12|34 ] . For a = B − 1 , b = −1 so that BSB1 ( B − 1 , −1 ) = BG , we get T { BSB1 ( B − 1 , −1 ) } = BSB1 ( ( B − 1 ) α11 − α12 , ( B − 1 ) α21 − α22 ) G⊗2 ◦ T { BSB1 ( B − 1 , −1 ) } = G BSB1 ( ( B − 1 ) α11 − α12 , ( B − 1 ) α21 − α22 ) G = ( ( B − 1 ) ( α11 − α21 ) − ( α12 − α22 ) ) G = B−1 ( ( B − 1 ) ( α11 − α21 ) − ( α12 − α22 ) ) BSB1 ( B − 1 , −1 ) = λG , TG BSB1 ( B − 1 , −1 ) by Lemma E.52 . Item 2 . It suffices to show that L ( B − 2 , 1 ) is an eigenmatrix with the eigenvalue λG , TL . As in the proof of Thm E.61 , we find T { L ( a , b ) } = L ( ( β11 β12 β21 β22 ) ( a b ) ) where β11 = [ 11|11 ] − [ 11|22 ] β12 = 2 ( B − 2 ) ( [ 11|23 ] − [ 11|12 ] ) β21 = − [ 12|11 ] + [ 12|33 ] β22 = [ 12|21 ] + [ 12|12 ] + 2 ( B − 4 ) [ 12|13 ] − 2 ( B − 3 ) [ 12|34 ] . So with a = B − 2 , b = 1 , we have T { L ( B − 2 , 1 ) } = L ( ( B − 2 ) β11 + β12 , ( B − 2 ) β21 + β22 ) G⊗2 ◦ T { L ( B − 2 , 1 ) } = G L ( ( B − 2 ) β11 + β12 , ( B − 2 ) β21 + β22 ) G = B−1 ( ( B − 2 ) β11 + β12 + 2 ( ( B − 2 ) β21 + β22 ) ) L ( B − 2 , 1 ) = λG , TL L ( B − 2 , 1 ) by Lemma E.60 . Item 3 . The proof is exactly the same as in that of Thm E.61 . Noting that the eigenspaces of Thm E.62 are orthogonal , we have Proposition E.63 . G⊗2 ◦ T HG for any ultrasymmetric operator T is self-adjoint . DOS Operators . In some cases , such as the original study of vanilla tanh networks by Schoenholz et al . ( 2016 ) , the ultrasymmetric operator involved has a much simpler form : Definition E.64 . Let T : HB → HB be such that for any Σ ∈ HB and any i 6= j ∈ [ B ] , T { Σ } ii = uΣii T { Σ } ij = vΣii + vΣjj + wΣij Then we say that T is diagonal-off-diagonal semidirect , or DOS for short . We write more specifically T = DOSB ( u , v , w ) . Thm E.62 and Thm E.61 still hold for DOS operators , but we can simplify the results and reason about them in a more direct way . Lemma E.65 . Let T : = DOSB ( u , v , w ) . Then T { LB ( a , b ) } = LB ( ua , wb− va ) . Proof . Let L : = LB ( a , b ) . It suffices to verify the following , each of which is a direct computation . 1 . T { L } 3 : B,3 : B = 0 . 2 . T { L } 1,2 = T { L } 2,1 = 0 3 . T { L } 1,1 = −T { L } 2,2 = ua 4 . T { L } 1 , i = −T { L } 2 , i = T { L } i,1 = −T { L } i,2 = va− wb . Lemma E.66 . Let T : = DOSB ( u , v , w ) . Then LB ( w − u , v ) and LB ( 0 , 1 ) are its eigenvectors : T { LB ( w − u , v ) } = uLB ( w − u , v ) T { LB ( 0 , 1 ) } = wLB ( 0 , 1 ) Proof . The map ( a , b ) 7→ ( ua , wb − va ) has eigenvalues u and w with corresponding eigenvectors ( w − u , v ) and ( 0 , 1 ) . By Lemma E.65 , this implies the desired results . Lemma E.67 . Let T : = DOSB ( u , v , w ) . Then for any L ∈ LB , G⊗2◦T { L } = ( u−2v ) ( B−2 ) +2wB L. Proof . By Lemma E.60 and Lemma E.65 , G⊗2 ◦ T { L ( B− 2 , 1 ) } = G⊗2 { L ( u ( B− 1 ) , w− v ( B− 1 ) ) } = ( u−2v ) ( B−2 ) +2wB LB ( B − 2 , 1 ) . By permutation symmetry , we also have the general result for any LB ( B − 2 , 1 ) -shaped matrix L. Since they span LB , this gives the conclusion we want . Lemma E.68 . Let T : = DOSB ( u , v , w ) . Then T { BSB1B ( a , b ) } = BSB1B ( ua , wb+ 2va ) . Proof . Direct computation . Lemma E.69 . Let T : = DOSB ( u , v , w ) . Then BSB1B ( w − u , wv ) and BSB1B ( 0 , 1 ) are eigenvectors of T . T { BSB1B ( u− w , 2v ) } = uBSB1B ( u− w , 2v ) T { BSB1B ( 0 , 1 ) } = wBSB1B ( 0 , 1 ) Proof . The linear map ( a , b ) 7→ ( ua , wb+ 2va ) has eigenvalues u and w with corresponding eigenvectors ( u− w , 2v ) and ( 0 , 1 ) . The result then immediately follows from Lemma E.68 . Lemma E.70 . Let T : = DOSB ( u , v , w ) . Then G⊗2 ◦ T { G } = ( B−1 ) ( u−2v ) +wB G. Proof . Direct computation with Lemma E.68 and Lemma E.52 Definition E.71 . Define MB : = { Σ ∈ SB : DiagΣ = 0 } ( so compared to M , matrices in M do not need to have zero row sums ) . In addition , for any a , b ∈ R , set LB ( a , b ) : = a −b −b · · · −b 0 0 · · · −b 0 0 · · · ... ... ... . . . ∈ HB . Define LB ( a , b ) : = span ( PT1iLB ( a , b ) P1i : i ∈ [ B ] ) where P1i is the permutation matrix that swap the first entry with the ith entry , i.e . LB ( a , b ) is the span of the orbit of LB ( a , b ) under permuting rows and columns simultaneously . Note that LB ( a , b ) = LB ( a , b ) − PT12LB ( a , b ) P12 BSB1B ( a , −2b ) = B∑ i=1 PT1iLB ( a , b ) P1i So LB ( a , b ) , BSB1B ( a , −2b ) ∈ LB ( a , b ) . Theorem E.72 . Let T : = DOSB ( u , v , w ) . Suppose w 6= u . Then T HB has the following eigendecomposition : • MB has eigenvalue w ( dimMB = B ( B − 1 ) /2 ) • LB ( w − u , v ) has eigenvalue of u ( dimLB ( w − u , v ) = B ) If w = u , then LB ( w − u , v ) ⊆MB . Proof . The case of MB is obvious . We will show that LB ( w−u , v ) is an eigenspace with eigenvalue u . Then by dimensionality consideration they are all of the eigenspaces of T . Let L : = LB ( w − u , v ) . Then it ’ s not hard to see T { L } = LB ( a , b ) for some a , b ∈ R. It follows that a has to be u ( w−u ) and b has to be − ( v ( w−u ) −wv ) = uv , which yields what we want . Theorem E.73 . Let T : = DOSB ( u , v , w ) . Then G⊗2 ◦ T HGB : HGB → HGB has the following eigendecomposition : • RG has eigenvalue ( B−1 ) ( u−2v ) +wB ( dimRG = 1 ) • MB has eigenvalue w ( dimMB = B ( B − 3 ) /2 ) • LB has eigenvalue ( B−2 ) ( u−2v ) +2wB ( dimLB = B − 1 ) Proof . The case of MB is obvious . The case for RG follows from Lemma E.70 . The case for LB follows from Lemma E.67 . By dimensionality considerations these are all of the eigenspaces . F FORWARD DYNAMICS In this section we will be interested in studying the dynamics on PSD matrices of the form Σl = VBφ ( Σ l−1 ) = E [ φ ( √ BGh/‖Gh‖ ) ⊗2 : h ∼ N ( 0 , Σl−1 ) ] ( 43 ) where Σl ∈ SB and φ : R→ R. F.1 GLOBAL CONVERGENCE Basic questions regarding the dynamics Eq . ( 43 ) are 1 ) does it converge ? 2 ) What are the limit points ? 3 ) How fast does it converge ? Here we answer these questions definitively when φ = id . The following is the key lemma in our analysis . Lemma F.1 . Consider the dynamics Σl = E [ ( h/‖h‖ ) ⊗2 : h ∼ N ( 0 , Σl−1 ) ] on Σl ∈ SA . Suppose Σ0 is full rank . Then 1. liml→∞ Σl = 1AI . 2 . This convergence is exponential in the sense that , for any full rank Σ0 , there is a constant K < 1 such that λ1 ( Σl ) − λA ( Σl ) < K ( λ1 ( Σl−1 ) − λA ( Σl−1 ) ) for all l ≥ 2 . Here λ1 ( resp . λA ) denotes that largest ( resp . smallest ) eigenvalue . 3 . Asymptotically , λ1 ( Σl ) − λA ( Σl ) = O ( ( 1− 2A+2 ) l ) . Proof . Let λ1l ≥ λ2l ≥ · · · ≥ λAl be the eigenvalues of Σl . It ’ s easy to see that ∑ i λi l = 1 for all l ≥ 1 . So WLOG we assume l ≥ 1 and this equality holds . We will show the “ exponential convergence ” statement ; that liml→∞Σl = 1AI then follows from the trace condition above . Proof of Item 2 . For brevity , we write suppress the superscript index l , and use · to denote ·l . We will now compute the eigenvalues λ1 ≥ · · · ≥ λA of Σ . First , notice that Σ and Σ can be simultaneously diagonalized , and by induction all of { Σl } l≥0 can be simultaneous diagonalized . Thus we will WLOG assume that Σ is diagonal , Σ = Diag ( λ1 , . . . , λA ) , so that Σ = Diag ( γ1 , . . . , γA ) for some { γi } i . These { γi } i form the eigenvalues of Σ but a priori we don ’ t know whether they fall in decreasing order ; in fact we will soon see that they do , and γi = λi . We have γi = ( π 2 ) −A/2 ∫ ∞A 0A λix 2 i∑ j λjx 2 j e−‖x‖ 2/2 dx = ( π 2 ) −A/2 ∫ ∞A 0A λix 2 i e −‖x‖2/2 dx ∫ ∞ 0 e−s ∑ j λjx 2 j ds = ( π 2 ) −A/2 ∫ ∞A+1 0A+1 λix 2 i e − 12 ∑ j ( 1+2sλj ) x 2 j dxds = ( π 2 ) −A/2 ∫ ∞2 02 λix 2 i e − 12 ( 1+2sλi ) x 2 i · ( π 2 ) A−1 2 ∏ j 6=i ( 1 + 2sλj ) −1/2 dxi ds = ( π 2 ) −1/2 ∫ ∞2 02 λix 2 i e − 12 ( 1+2sλi ) x 2 i dxi ∏ j 6=i ( 1 + 2sλj ) −1/2 ds = ∫ ∞ 0 λi ( 1 + 2sλi ) −3/2 ∏ j 6=i ( 1 + 2sλj ) −1/2 ds = ∫ ∞ 0 A∏ j=1 ( 1 + 2sλj ) −1/2λi ( 1 + 2sλi ) −1 ds ( 44 ) Therefore , γi − γk = ∫ ∞ 0 A∏ j=1 ( 1 + 2sλj ) −1/2 ( λi ( 1 + 2sλi ) −1 − λk ( 1 + 2sλk ) −1 ) ds = ( λi − λk ) ∫ ∞ 0 A∏ j=1 ( 1 + 2sλj ) −1/2 ( 1 + 2sλi ) −1 ( 1 + 2sλk ) −1 ds Since the RHS integral is always positive , λi ≥ λk =⇒ γi ≥ γk and thus γi = λi for each i . Define T ( λ ) : = ∫∞ 0 ∏A j=1 ( 1 + 2sλ1 ) −1/2 ( 1 + 2sλ1 ) −1 ( 1 + 2sλA ) −1 ds , so that λ1 − λA = ( λ1 − λA ) T ( λ ) . Note first that ∏A j=1 ( 1 + 2sλj ) −1/2 ( 1 + 2sλi ) −1 ( 1 + 2sλk ) −1 is ( strictly ) log-convex and hence ( strictly ) convex in λ . Furthermore , T ( λ ) is ( strictly ) convex because it is an integral of ( strictly ) convex functions . Thus T is maximized over any convex region by its extremal points , and only by its extremal points because of strict convexity . The convex region we are interested in is given by A : = { ( λi ) i : λ1 ≥ λ2 ≥ · · · ≥ λA ≥ 0 & ∑ i λi = 1 } . The unit sum condition follows from the normalization h/‖h‖ of the iteration map . The extremal points of A are { ωk : = ( k times︷ ︸︸ ︷ 1/k , 1/k , · · · , 1/k , 0 , 0 , · · · , 0 ) } Ak=1 . For k = 1 , . . . , A− 1 , we have T ( ωk ) = ∫ ∞ 0 ( 1 + 2s/k ) −k/2−1 ds = 2/k −k/2 ( 1 + 2s/k ) −k/2 ∣∣∣∣∞ 0 = 1 . But for k = A , T ( ωA ) = ∫ ∞ 0 ( 1 + 2s/A ) −A/2−2 ds = 2/A −A/2− 1 ( 1 + 2s/A ) −k/2 ∣∣∣∣∞ 0 = A A+ 2 This shows that T ( λ ) ≤ 1 , with equality iff λ = ωk for k = 1 , . . . , A − 1 . In fact , because every point λ ∈ A is a convex combination of ωk for k = 1 , . . . , A , λ = ∑Ak=1 akωk , by convexity of T , we must have T ( λ ) ≤ A∑ k=1 akT ( ω k ) = 1− aA 2 A+ 2 = 1− λA 2 A+ 2 ( 45 ) where the last line follows because ωA is the only point with last coordinate nonzero so that aA = λA . We now show that the gap λ1l − λAl → 0 as l → ∞ . There are two cases : if λAl is bounded away from 0 infinitely often , then T ( λ ) < 1 − infinitely often for a fixed > 0 so that the gap indeed vanishes with l. Now suppose otherwise , that λAl converges to 0 ; we will show this leads to a contradiction . Notice that λA = ∫ ∞ 0 A∏ j=1 ( 1 + 2sλj ) −1/2λA ( 1 + 2sλA ) −1 ds λA/λA ≥ ∫ ∞ 0 A∏ j=1 ( 1 + 2s ( 1− λA ) / ( A− 1 ) ) − ( A−1 ) /2 ( 1 + 2sλA ) −3/2 ds where the first lines is Eq . ( 44 ) and the 2nd line follows from the convexity of ∏A−1 j=1 ( 1+2sλj ) −1/2 as a function of ( λ1 , . . . , λA−1 ) . By a simple application of dominated convergence , as λA → 0 , this integral converges to a particular simple form , λA/λA ≥ ∫ ∞ 0 A∏ j=1 ( 1 + 2s/ ( A− 1 ) ) − ( A−1 ) /2 ds = 1 + 2 A− 3 Thus for large enough l , λAl+1/λAl is at least 1 + for some > 0 , but this contradicts the convergence of λA to 0 . Altogether , this proves that λ1l − λAl → 0 and therefore λl → ωA as l → ∞ . Consequently , λAl is bounded from below , say by K ′ ( where K ′ > 0 because by Eq . ( 44 ) , λAl is never 0 ) , for all l , and by Eq . ( 45 ) , we prove Item 2 by taking K = 1 −K ′ 2A+2 . In addition , asymptotically , the gap decreases exponentially as T ( ωA ) l = ( A A+2 ) l , proving Item 3 . Theorem F.2 . Consider the dynamics Σl = E [ ( h/‖h‖ ) ⊗2 : h ∼ N ( 0 , Σl−1 ) ] on Σl ∈ SA . Suppose Σ0 = MTDM where M is orthogonal and D is a diagonal matrix . If D has rank C and Dii 6= 0 , ∀1 ≤ i ≤ C , then 1. liml→∞ Σl = 1CM TD′M where D′ is the diagonal matrix with D′ii = I ( Dii 6= 0 ) . 2 . This convergence is exponential in the sense that , for any Σ0 of rank C , there is a constant K < 1 such that λ1 ( Σl ) − λC ( Σl ) < K ( λ1 ( Σl−1 ) − λC ( Σl−1 ) ) for all l ≥ 2 . Here λi denotes the ith largest eigenvalue . 3 . Asymptotically , λ1 ( Σl ) − λC ( Σl ) = O ( ( 1− 2C+2 ) l ) . Proof . Note that Σl can always be simultaneously diagonalized with Σ0 , so that Σl = MTDlM for some diagonal Dl which has Dlii = 0 , ∀0 ≤ i ≤ C. Then we have ( Dl ) ′ = E [ ( h/‖h‖ ) ⊗2 : h ∼ N ( 0 , ( Dl−1 ) ′ ) ] , where ( Dl ) ′ means the diagonal matrix obtained from Dl by deleting the dimensions with zero eigenvalues . The proof then finishes by Lemma F.1 . From this it easily follows the following characterization of the convergence behavior . Corollary F.3 . Consider the dynamics of Eq . ( 43 ) for φ = id : Σl = B E [ ( Gh/‖Gh‖ ) ⊗2 : h ∼ N ( 0 , Σl−1 ) ] on Σl ∈ SB . Suppose GΣG has rank C < B and factors as êDêT where D ∈ RC×C is a diagonal matrix with no zero diagonal entries and ê is an B × C matrix whose columns form an orthonormal basis of a subspace of imG . Then 1. liml→∞ Σl = BC êIC ê T . 2 . This convergence is exponential in the sense that , for any GΣ0G of rank C , there is a constant K < 1 such that λ1 ( Σl ) − λC ( Σl ) < K ( λ1 ( Σl−1 ) − λC ( Σl−1 ) ) for all l ≥ 2 . Here λi denotes the ith largest eigenvalue . 3 . Asymptotically , λ1 ( Σl ) − λC ( Σl ) = O ( ( 1− 2C+2 ) l ) . General nonlinearity . We don ’ t ( currently ) have a proof of any characterization of the basin of attraction for Eq . ( 43 ) for general φ . Thus we are forced to resort to finding its fixed points manually and characterize their local convergence properties . F.2 LIMIT POINTS Batchnorm Bφ is permutation-equivariant , in the sense that Bφ ( πh ) = πBφ ( h ) for any permtuation matrix π . Along with the case of φ = id studied above , this suggests that we look into BSB1 fixed points Σ∗ . What are the BSB1 fixed points of Eq . ( 43 ) ? F.2.1 SPHERICAL INTEGRATION The main result ( Thm F.5 ) of this section is an expression of the BSB1 fixed point diagonal and offdiagonal entries in terms of 1- and 2-dimensional integrals . This allows one to numerically compute such fixed points . By a simple symmetry argument , we have the following Lemma F.4 . Suppose X is a random vector in RB , symmetric in the sense that for any permutation matrix π and any subset U of RB measurable with respect to the distribution of X , P ( X ∈ U ) = P ( X ∈ π ( U ) ) . Let Φ : RB → RB be a symmetric function in the sense that Φ ( πx ) = πΦ ( x ) for any permutation matrix π . Then EΦ ( X ) Φ ( X ) T = µI + ν11T for some µ and ν . This lemma implies that VBφ ( Σ ) is BSB1 whenever Σ is BSB1 . In the following , We in fact show that for any BSB1 Σ , VBφ ( Σ ) is the same , and this is the unique BSB1 fixed point of Eq . ( 43 ) . Theorem F.5 . The BSB1 fixed point Σ∗ = µ∗I + ν∗11T to Eq . ( 43 ) is unique and satisfies µ∗ + ν∗ = Γ ( B−12 ) Γ ( B−22 ) √ π ∫ π 0 dθ1 sin B−3 θ1φ ( √ Bζ1 ( θ1 ) ) 2 ν∗ = { B−3 2π ∫ π 0 dθ1 ∫ π 0 dθ2 sin B−3 θ1 sin B−4 θ2φ ( √ Bζ1 ( θ1 ) ) φ ( √ Bζ2 ( θ1 , θ2 ) ) , if B ≥ 4 1 2π ∫ 2π 0 dθφ ( − √ 2 sin θ ) φ ( sin θ/ √ 2− cos θ √ 3/ √ 2 ) , if B = 3 where ζ1 ( θ ) = − √ B − 1 B cos θ ζ2 ( θ1 , θ2 ) = 1√ B ( B − 1 ) cos θ1 − √ B − 2 B − 1 sin θ1 cos θ2 Proof . For any BSB1 Σ , eTΣe = µIB−1 for some µ . Thus we can apply Proposition E.33 toGΣG . We observe that K ( v ; GΣG ) = 1 , so that VBφ ( Σ ) = VBφ ( GΣG ) = E v∼SB−2 φ ( √ Bev ) ⊗2 . Note that this is now independent of the specific values of Σ . Now , using the specific form of e given by Eq . ( 33 ) , we see that VBφ ( Σ ) 11 only depends on θ1 and VBφ ( Σ ) 12 only depends on θ1 and θ2 . Then applying Proposition E.33 followed by Lemma E.36 and Lemma E.37 yield the desired result . F.2.2 LAPLACE METHOD In the case that φ is positive-homogeneous , we can apply Laplace ’ s method Appendix E.1 and obtain the BSB1 fixed point in a much nicer form . Definition F.6 . For any α ≥ 0 , define Kα , B : = cαP ( B−1 2 , α ) −1 ( B−1 2 ) α where P ( a , b ) : = Γ ( a+ b ) /Γ ( a ) is the Pochhammer symbol . Note that Proposition F.7 . limB→∞Kα , B = cα . We give a closed form description of the BSB1 fixed point for a positive homogeneous φ in terms of its J function . Theorem F.8 . Suppose φ : R → R is degree α positive-homogeneous . For any BSB1 Σ ∈ SB , VBφ ( Σ ) is BSB1 . The diagonal entries are Kα , BJφ ( 1 ) and the off-diagonal entries are Kα , BJφ ( −1 B−1 ) . Here cα is as defined in Defn E.7 and Jφ is as defined in Defn E.20 . Thus a BSB1 fixed point of Eq . ( 43 ) exists and is unique . Proof . Let e be anB× ( B−1 ) matrix whose columns form an orthonormal basis of imG : = { Gv : v ∈ RB } = { w ∈ RB : ∑i wi = 0 } . Let BSB1 ( a , b ) denote a matrix with a on the diagonal and b on the off-diagnals . Note that Vφ is positive homogeneous of degree α , so for any µ , Vφ ( eµIB−1 ( IB−1 + 2sµIB−1 ) −1 e T ) = Vφ ( µ 1 + 2sµ ee T ) = ( µ 1 + 2sµ ) α Vφ ( ee T ) by Proposition E.24 = ( µ 1 + 2sµ ) α Vφ ( G ) = ( µ 1 + 2sµ B − 1 B ) α Vφ ( BSB1 ( 1 , −1 B − 1 ) ) = ( µ 1 + 2sµ B − 1 B ) α cαJφ ( BSB1 ( 1 , −1 B − 1 ) ) by Corollary E.19 So by Eq . ( 26 ) , VBφ ( eµIe T ) = BαΓ ( α ) −1 ∫ ∞ 0 ds sα−1 ( 1 + 2sµ ) − ( B−1 ) /2 ( µ 1 + 2sµ B − 1 B ) α cαJφ ( BSB1 ( 1 , −1 B − 1 ) ) = Γ ( α ) −1 ( B − 1 ) αcαJφ ( BSB1 ( 1 , −1 B − 1 ) ) ∫ ∞ 0 ds sα−1 ( 1 + 2sµ ) − ( B−1 ) /2−αµα = Γ ( α ) −1 ( B − 1 ) αcαJφ ( BSB1 ( 1 , −1 B − 1 ) ) Beta ( α , ( B − 1 ) /2 ) 2−α = cα Γ ( ( B − 1 ) /2 ) Γ ( α+ ( B − 1 ) /2 ) ( B − 1 2 ) α Jφ ( BSB1 ( 1 , −1 B − 1 ) ) Corollary F.9 . Suppose φ : R→ R is degree α positive-homogeneous . If Σ∗ is the BSB1 fixed point of Eq . ( 43 ) as given by Thm F.8 , then GΣ∗G = µ∗IB−1 where µ∗ = Kα , B ( Jφ ( 1 ) − Jφ ( −1/ ( B − 1 ) ) ) By setting α = 1 and φ = relu , we get Corollary F.10 . For any BSB1 Σ ∈ SB , VBrelu ( Σ ) is BSB1 with diagonal entries 12 and off-diagonal entries 12J1 ( −1 B−1 ) , so that G⊗2 { VBrelu ( Σ ) } = G ( VBrelu ( Σ ) ) G = ( 1 2 − 12J1 ( −1 B−1 ) ) G. By setting α = 1 and φ = id = x 7→ relu ( x ) − relu ( −x ) , we get Corollary F.11 . For any BSB1 Σ ∈ SB , VBid ( Σ ) is BSB1 with diagonal entries 1 and off-diagonal entries −1B−1 , so that G ⊗2 { VBid ( Σ ) } = B/ ( B − 1 ) G. Remark F.12 . One might hope to tweak the Laplace method for computing the fixed point to work for the Fourier method , but because there is no nice relation between Vφ ( cΣ ) and Vφ ( Σ ) in general , we can not simplify Eq . ( 32 ) as we can Eq . ( 26 ) and Eq . ( 27 ) . F.2.3 GEGENBAUER EXPANSION It turns out that the BSB1 fixed point can be described very cleanly using the Gegenbauer coefficients of φ . When Σ is BSB1 , K ( v , Σ ) = 1 for all v ∈ SB−2 , so that by Proposition E.34 , Σ∗ = E v∼SB−2 φ ( √ Bev ) ⊗2 Σ∗ab = E v∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) for any a , b ∈ [ B ] , independent of the actual values of Σ . Here ea , : is the ath row of e. Theorem F.13 . If φ ( √ B − 1x ) ∈ L2 ( ( 1 − x2 ) B−32 ) has Gegenbauer expansion∑∞ l=0 al 1 cB−1 , l C ( B−32 ) l ( x ) , then Σ ∗ = µ∗I + ν∗11T with µ∗ = ∞∑ l=0 a2l 1 cB−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B − 1 ) ) = ∞∑ l=0 a2l 1 cB−1 , l ( ( B − 4 + l l ) − C ( B−3 2 ) l ( −1 B − 1 ) ) ν∗ = ∞∑ l=0 a2l 1 cB−1 , l C ( B−32 ) l ( −1 B − 1 ) Proof . By the reproducing property Fact E.46 , we have Σ∗ab = 〈φ ( √ Bea , :· ) , φ ( √ Beb , :· ) 〉SB−2 = 〈φ ( √ B − 1êa , :· ) , φ ( √ B − 1êb , :· ) 〉SB−2 = 〈 ∞∑ l=0 al 1 cB−1 , l C ( B−32 ) l ( êa , :· ) , ∞∑ l=0 al 1 cB−1 , l C ( B−32 ) l ( êb , :· ) 〉 SB−2 = 〈 ∞∑ l=0 alZ B−2 , ( l ) êa , : , ∞∑ l=0 alZ B−2 , ( l ) êb , : 〉 SB−2 = ∞∑ l=0 a2lZ B−2 , ( l ) êa , : ( êb , : ) = ∞∑ l=0 a2l 1 cB−1 , l C ( B−32 ) l ( 〈êa , : , êb , :〉 ) = ∞∑ l=0 a2l 1 cB−1 , l C ( B−32 ) l ( { 1 if a = b −1/ ( B − 1 ) else ) This gives the third equation . Since µ∗ = Σ∗11 − Σ∗12 , straightforward arithmetic yields the first equation . The second follows from Fact E.41 . We will see later that the rate of gradient explosion also decomposes nicely in terms of Gegenbauer coefficients ( Thm G.5 ) . However , this is not quite true for the eigenvalues of the forward convergence ( Thms F.22 and F.24 ) . F.3 LOCAL CONVERGENCE In this section we consider linearization of the dynamics given in Eq . ( 43 ) . Thus we must consider linear operators on the space of PSD linear operators SB . To avoid confusion , we use the following notation : If T : SB → SB ( for example the Jacobian of VBφ ) and Σ ∈ SB , then write T { Σ } for the image of Σ under T . A priori , the Jacobian of VBφ at its BSB1 fixed point may seem like a very daunting object . But a moment of thought shows that Proposition F.14 . The Jacobian dVBφ dΣ ∣∣ Σ : HB → HB is ultrasymmetric for any BSB1 Σ . Now we prepare several helper reults in order to make progress understanding batchnorm . Definition F.15 . Define n ( x ) = √ Bx/‖x‖ , i.e . division by sample standard deviation . Batchnorm Bφ can be decomposed as the composition of three steps , φ ◦ n ◦ G , where G is meancentering , n is division by standard deviation , and φ is coordinate-wise application of nonlinearity . We have , as operatorsH → H , dVBφ ( Σ ) dΣ = dE [ Bφ ( z ) ⊗2 : z ∈ N ( 0 , Σ ) ] dΣ = dE [ ( φ ◦ n ) ( x ) ⊗2 : x ∈ N ( 0 , GΣG ) ] dΣ = dV ( φ◦n ) ( Σ G ) dΣG ◦ dΣ G dΣ = dV ( φ◦n ) ( Σ G ) dΣG ◦G⊗2 Definition F.16 . With Σ∗ being the unique BSB1 fixed point , write U : = G⊗2 ◦ dV ( φ◦n ) ( Σ G ) dΣG ∣∣ ΣG=GΣ∗G : HG → HG . It turns out to be advantageous to study U first and relate its eigendecomposition back to that of dVBφ ( Σ ) dΣ ∣∣ Σ=Σ∗ , where Σ∗ is the BSB1 fixed point , by applying Lemma F.17 . Lemma F.17 . Let X and Y be two vector spaces . Consider linear operators A : X → Y , B : Y → X . Then 1. rankAB = rankBA 2 . If v ∈ Y is an eigenvector of AB with nonzero eigenvalue , then X 3 Bv 6= 0 and Bv is an eigenvector of BA of the same eigenvalue 3 . Suppose AB has k = rankAB linearly independent eigenvectors { vi } ki=1 of nonzero eigenvalues { λi } ki=1 . Then BA has k linearly independent eigenvectors { Bvi } ki=1 with the same eigenvalues { λi } ki=1 which are all eigenvectors of BA with nonzero eigenvalues , up to linear combinations within eigenvectors with the same eigenvalue . WithA = G⊗2 andB = dV ( φ◦n ) ( Σ G ) dΣG , Thm E.61 implies thatAB andBA can both be diagonalized , and this lemma implies that all nonzero eigenvalues of dVBφ ( Σ ) dΣ can be recovered from those of U . Proof . ( Item 1 ) Observe rankAB = rankABAB ≤ rankBA . By symmetry the two sides are in fact equal . ( Item 2 ) Bv can not be zero or otherwise ABv = A0 = 0 , contradicting the fact that v is an eigenvector with nonzero eigenvalue . Suppose λ is the eigenvalue associated to v. ThenBA ( Bv ) = B ( ABv ) = B ( λv ) = λBv , so Bv is an eigenvector of BA with the same eigenvalue . ( Item 3 ) Item 2 shows that { Bvi } ki=1 are eigenvectors BA with the same eigenvalues { λi } ki=1 . The eigenspaces with different eigenvalues are linearly independent , so it suffices to show that if { Bvij } j are eigenvectors of the same eigenvalue λs , then they are linearly independent . But ∑ j ajBvij = 0 =⇒ ∑j ajvij = 0 because B is injective on eigenvectors by Item 2 , so that aj = 0 identically . Hence { Bvj } j is linearly independent . Since rankBA = k , these are all of the eigenvectors with nonzero eigenvalues of BA up to linearly combinations . Lemma F.18 . Let f : RB → RA be measurable , and Σ ∈ SB be invertible . Then for any Λ ∈ RB×B , with 〈· , ·〉 denoting trace inner product , d dΣ E [ f ( z ) : z ∼ N ( 0 , Σ ) ] { Λ } = 1 2 E [ f ( z ) 〈 Σ−1zzTΣ−1 − Σ−1 , Λ 〉 : z ∼ N ( 0 , Σ ) ] If f is in addition twice-differentiable , then d dΣ E [ f ( z ) : z ∼ N ( 0 , Σ ) ] = 1 2 E [ d2f ( z ) dz2 : z ∼ N ( 0 , Σ ) ] whenever both sides exist . Proof . Let Σt , t ∈ ( − , ) be a smooth path in SB , with Σ0 = Σ . Write ddtΣt = Σ̇t . Then d dt E [ f ( z ) : z ∼ N ( 0 , Σt ) ] = d dt ( 2π ) −B/2 det Σ −1/2 t ∫ dz e− 1 2 z TΣ−1t zf ( z ) ∣∣∣∣ t=0 = ( 2π ) −B/2 −1 2 det Σ −1/2 0 tr ( Σ−10 Σ̇0 ) ∫ dz e− 1 2 z TΣ−10 z/2f ( z ) + ( 2π ) −B/2 det Σ −1/2 0 ∫ dz e− 1 2 z TΣ−10 z −1 2 zT ( −Σ−10 Σ̇0Σ−10 ) zf ( z ) = −1 2 ( 2π ) −B/2 det Σ −1/2 0 ∫ dz e− 1 2 z TΣ−10 zf ( z ) ( tr Σ−10 Σ̇0 − zTΣ−10 Σ̇0Σ−10 z ) = 1 2 ( 2π ) −B/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zf ( z ) 〈 Σ−1zzTΣ−1 − Σ−1 , Σ̇0 〉 = −1 2 E [ f ( z ) 〈 Σ−1 − Σ−1zzTΣ−1 , Σ̇0 〉 : z ∼ N ( 0 , Σ ) ] Note that vT d dz e −1 2 z TΣ−1z = −vTΣ−1ze−12 zTΣ−1z wT ( d2 dz2 e −1 2 z TΣ−1z ) v = ( wTΣ−1zzTΣ−1v − wTΣ−1v ) e −1 2 z TΣ−1z = 〈 Σ−1zzTΣ−1 − Σ−1 , vwT 〉 e −1 2 z TΣ−1z so that as a cotensor , d2 dz2 e −1 2 z TΣ−1z { Λ } = 〈 Σ−1zzTΣ−1 − Σ−1 , Λ 〉 e −1 2 z TΣ−1z for any Λ ∈ RB×B . Therefore , dE [ f ( z ) : z ∼ N ( 0 , Σ ) ] dΣ { Λ } = 1 2 ( 2π ) −B/2 det Σ−1/2 ∫ dz f ( z ) d2e− 1 2 z TΣ−1z d2z { Λ } = 1 2 ( 2π ) −B/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1z d 2f ( z ) d2z { Λ } ( by integration by parts ) = 1 2 E [ d2f ( z ) d2z : z ∼ N ( 0 , Σ ) ] { Λ } Note that for any Λ ∈ HB with ‖Λ‖op < µ∗ , E [ φ ◦ n ( z ) ⊗2 : z ∈ N ( 0 , µ∗G+ Λ ) ] = E [ φ ◦ n ( Gz ) ⊗2 : z ∈ N ( 0 , µ∗I + Λ ) ] = E [ Bφ ( z ) ⊗2 : z ∈ N ( 0 , µ∗I + Λ ) ] so that we have , for any Λ ∈ HB , U { Λ } = dVBφ ( Σ ) dΣ ∣∣∣∣ Σ=µ∗I { Λ } = 1 2 E [ Bφ ( z ) ⊗2〈µ∗−1zzT − µ∗−1I , Λ〉 : z ∼ N ( 0 , µ∗I ) ] ( by Lemma F.18 ) = 1 2 µ∗−2 E [ Bφ ( z ) ⊗2〈zzT , Λ〉 : z ∼ N ( 0 , µ∗I ) ] − 1 2 µ∗−1Σ∗〈I , Λ〉 ( by Thm F.8 ) = ( 2µ∗ ) −1 ( E [ Bφ ( z ) ⊗2〈zzT , Λ〉 : z ∼ N ( 0 , I ) ] − Σ∗〈I , Λ〉 ) ( Bφ is scale-invariant ) = ( 2µ∗ ) −1 ( E [ Bφ ( z ) ⊗2〈GzzTG , Λ〉 : z ∼ N ( 0 , I ) ] − Σ∗〈I , Λ〉 ) ( Λ = GΛG ) Let ’ s extend to all matrices by this formula : Definition F.19 . Define Ũ : RB×B → SB , Λ 7→ ( 2µ∗ ) −1 ( E [ Bφ ( z ) ⊗2〈GzzTG , Λ〉 : z ∼ N ( 0 , I ) ] − Σ∗〈I , Λ〉 ) F.3.1 SPHERICAL INTEGRATION So Ũ HB = U HB . Ultimately we will apply Thm E.61 to G⊗2 ◦ Ũ HB = G⊗2 ◦ U Definition F.20 . Write τij = 12 ( δiδ T j + δjδ T i ) . Then Ũ { τij } kl = ( 2µ∗ ) −1 ( E [ Bφ ( z ) kBφ ( z ) l ( Gz ) i ( Gz ) j : z ∼ N ( 0 , I ) ] − Σ∗klI ( i = j ) ) = ( 2µ∗ ) −1 ( E [ φ ( n ( y ) ) kφ ( n ( y ) ) lyiyj : y ∼ N ( 0 , G ) ] − Σ∗klI ( i = j ) ) = ( 2µ∗ ) −1 ( E [ φ ( n ( y ) ) kφ ( n ( y ) ) lyiyj : y ∼ N ( 0 , eeT ) ] − Σ∗klI ( i = j ) ) ( See Defn E.3 for defn of e ) = ( 2µ∗ ) −1 ( E [ φ ( n ( ex ) ) kφ ( n ( ex ) ) l ( ex ) i ( ex ) j : x ∼ N ( 0 , IB−1 ) ] − Σ∗klI ( i = j ) ) Here we will realize e as the matrix in Eq . ( 33 ) . Definition F.21 . Define Wij|kl : = E [ φ ( n ( ex ) ) kφ ( n ( ex ) ) l ( ex ) i ( ex ) j : x ∼ N ( 0 , IB−1 ) ] . Then Ũ { τij } kl = ( 2µ∗ ) −1 ( Wij|kl − Σ∗klI ( i = j ) ) . If we can evaluate Wij|kl then we can use Thm E.61 to compute the eigenvalues of G⊗2 ◦ U . It ’ s easy to see that Wij|kl is ultrasymmetric Thus WLOG we can take i , j , k , l from { 1 , 2 , 3 , 4 } . By Lemma E.35 , and the fact that x 7→ ex is an isometry , Wij|kl = E [ r2φ ( √ Bev ) kφ ( √ Bev ) l ( ev ) i ( ev ) j : x ∼ N ( 0 , IB−1 ) ] = ( B − 5 ) ( B − 3 ) ( B − 1 ) ( 2π ) −2×∫ π 0 dθ1 · · · ∫ π 0 dθ4 φ ( √ Bev ) kφ ( √ Bev ) l ( ev ) i ( ev ) j sin B−3 θ1 · · · sinB−6 θ4 If WLOG we further assume that k , l ∈ { 1 , 2 } ( by ultrasymmetry ) , then there is no dependence on θ3 and θ4 inside φ . So we can expand ( ev ) i and ( ev ) j in trigonometric expressions as in Eq . ( 34 ) and integrate out θ3 and θ4 via Lemma E.29 . We will not write out this integral explicitly but instead focus on other techniques for evaluating the eigenvalues . F.3.2 GEGENBAUER EXPANSION Now let ’ s compute the local convergence rate via Gegenbauer expansion . By differentiating Proposition E.34 through a path Σt ∈ SB , t ∈ ( − , ) , we get d dt VBφ ( Σt ) = E v∼SB−2 φ ( √ Bev ) ⊗2 d dt K ( v ; Σt ) = E v∼SB−2 φ ( √ Bev ) ⊗2K ( v ; Σt ) ( B − 1 2 ( vTΣ −1v ) −1vTΣ −1 dΣ dt Σ −1v − 1 2 tr ( Σ −1 dΣ dt ) ) where Σ = Σ t = eTΣte ∈ SB−1 ( as introduced below Defn E.3 ) . At Σ0 = Σ∗ , the BSB1 fixed point , we have Σ 0 = µ∗I , and d dt VBφ ( Σt ) ∣∣∣∣ t=0 = E v∼SB−2 φ ( √ Bev ) ⊗2K ( v ; Σ∗ ) µ∗−1 ( B − 1 2 vT dΣ dt v − 1 2 tr ( dΣ dt ) ) = µ∗−1 E v∼SB−2 φ ( √ Bev ) ⊗2 ( B − 1 2 vT dΣ dt v − 1 2 tr ( dΣ dt ) ) ( 46 ) If ddtΣ|t=0 = G , then the term in the parenthesis vanishes . This shows that dVBφ dΣ ∣∣∣ Σ=Σ∗ { G } = 0 . Eigenvalue for L. If ddtΣ|t=0 = L ( B− 2 , 1 ) = BG ( δ⊗21 − δ⊗22 ) G where δ1 = ( 1 , 0 , . . . , 0 ) , δ2 = ( 0 , 1 , 0 , . . . , 0 ) , both in RB ( Proposition E.59 ) , then we know by Thm E.62 that G⊗2 ◦ d dtVBφ ( Σt ) ∣∣ t=0 is a multiple of L ( B − 2 , 1 ) . We compute ( d dt VBφ ( Σt ) ∣∣∣∣ t=0 ) ab = B B − 1 2 µ∗−1 E v∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) ( ( e1 , :v ) 2 − ( e2 , :v ) 2 ) . Clearly , ddtVBφ ( Σt ) ∣∣ t=0 is L-shaped . Now define the quantity A ( a , b ; c ) : = Ev∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) ( ec , :v ) 2 , so that ( d dtVBφ ( Σt ) ∣∣ t=0 ) ab = B ( B−1 ) 2 µ ∗−1 ( A ( a , b ; 1 ) − A ( a , b ; 2 ) ) . Because e is an isometry , ∑ b ( eb , :v ) 2 = ‖v‖2 = 1 for all v ∈ SB−2 . Thus B∑ c=1 A ( a , b ; c ) = E v∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) = Σ ∗ ab . ( 47 ) By symmetry , A ( a , b ; a ) = A ( a , b ; b ) and for any c , c′ 6∈ { a , b } , A ( a , b ; c ) = A ( a , b ; c′ ) . So we have , for any a , b , c not equal , A ( a , a ; a ) + ( B − 1 ) A ( a , a ; c ) = Σ∗aa 2A ( a , b ; b ) + ( B − 2 ) A ( a , b ; c ) = Σ∗ab So the eigenvalue associated to L ( B − 2 , 1 ) is , So by Lemma E.60 , G⊗2 ◦ U { L ( B − 2 , 1 ) } = λ↑LL ( B − 2 , 1 ) , where λ↑L = B B − 1 2 µ∗−1 ( d dtVBφ ( Σt ) ∣∣ t=0 ) 11 − 2 ( d dtVBφ ( Σt ) ∣∣ t=0 ) 13 B = B B − 1 2 µ∗−1 A ( 1 , 1 ; 1 ) −A ( 1 , 1 ; 2 ) − 2 ( A ( 1 , 3 ; 1 ) −A ( 1 , 3 ; 2 ) ) B = B − 1 2 µ∗−1 ( A ( 1 , 1 ; 1 ) − 1 B − 1 ( Σ ∗ aa −A ( 1 , 1 ; 1 ) ) − 2 ( A ( 1 , 3 ; 1 ) − 1 B − 2 ( Σ ∗ ab − 2A ( 1 , 3 ; 1 ) ) ) ) = B − 1 2 µ∗−1 ( B B − 1A ( 1 , 1 ; 1 ) − 1 B − 1Σ ∗ aa − 2 B B − 2A ( 1 , 3 ; 1 ) + 2 B − 2Σ ∗ ab ) ( 48 ) Thus , if φ ( √ B − 1x ) ∈ L2 ( ( 1 − x2 ) B−32 ) has Gegenbauer expansion ∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) , then by Proposition E.43 , x2φ ( √ B − 1x ) = ∞∑ l=0 al 1 cB−1 , l x2C ( B−32 ) l ( x ) = ∞∑ l=0 al 1 cB−1 , l ( κ2 ( l , B − 3 2 ) C ( B−32 ) l+2 ( x ) + κ0 ( l , B − 3 2 ) C ( B−32 ) l ( x ) + κ−2 ( l , B − 3 2 ) C ( B−32 ) l−2 ( x ) ) where C ( B−3 2 ) l ( x ) is understood to be 0 for l < 0 = ∞∑ l=0 C ( B−32 ) l ( x ) ( al−2 cB−1 , l−2 κ2 ( l − 2 , B − 3 2 ) + al cB−1 , l κ0 ( l , B − 3 2 ) + al+2 cB−1 , l+2 κ−2 ( l + 2 , B − 3 2 ) ) where κi ( l , B − 3 2 ) is understood to be 0 for l < 0 = ∞∑ l=0 c−1B−1 , lC ( B−32 ) l ( x ) ( al−2cB−1 , l cB−1 , l−2 κ2 ( l − 2 , B − 3 2 ) + alκ0 ( l , B − 3 2 ) + al+2cB−1 , l cB−1 , l+2 κ−2 ( l + 2 , B − 3 2 ) ) We can evaluate , for any a , b ( possibly equal ) , A ( a , b ; b ) = E v∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) ( eb , :v ) 2 = E v∼SB−2 φ ( √ B − 1êa , :v ) φ ( √ B − 1êb , :v ) ( êb , :v ) 2 B − 1 B = B − 1 B ∞∑ l=0 c−1B−1 , lC ( B−32 ) l ( 〈êa , : , êb , :〉 ) al ( al−2cB−1 , l cB−1 , l−2 κ2 ( l − 2 , B − 3 2 ) + alκ0 ( l , B − 3 2 ) + al+2cB−1 , l cB−1 , l+2 κ−2 ( l + 2 , B − 3 2 ) ) ( 49 ) By Eq . ( 48 ) , the eigenvalue associated with L ( B − 2 , 1 ) is B − 1 2 µ∗−1 ( B B − 1A ( 1 , 1 ; 1 ) − 1 B − 1Σ ∗ aa − 2 B B − 2A ( 1 , 3 ; 1 ) + 2 B − 2Σ ∗ ab ) = B − 1 2 µ∗−1 ∞∑ l=0 c−1B−1 , l ( γlC ( B−32 ) l ( 1 ) − τlC ( B−32 ) l ( −1 B − 1 ) ) where γl : = al ( al−2cB−1 , l cB−1 , l−2 κ2 ( l − 2 , B − 3 2 ) + alκ0 ( l , B − 3 2 ) + al+2cB−1 , l cB−1 , l+2 κ−2 ( l + 2 , B − 3 2 ) ) − 1 B − 1a 2 l τl : = 2 ( B − 1 ) B − 2 γl . Thus , Theorem F.22 . Suppose φ ( √ B − 1x ) has Gegenbauer expansion∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) . Then the eigenvalue of U with respect to the eigenspace L is a ratio of quadratic forms λ↑L = ∑∞ l=0 a 2 lwB−1 , l + alal+2uB−1 , l∑∞ l=0 a 2 l vB−1 , l where vB−1 , l : = c −1 B−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B − 1 ) ) wB−1 , l : = B − 1 2 ( κ0 ( l , B − 3 2 ) − 1 B − 1 ) c−1B−1 , l ( C ( B−32 ) l ( 1 ) − 2 ( B − 1 ) B − 2 C ( B−32 ) l ( −1 B − 1 ) ) = l ( B − 3 + l ) ( B − 3 ) ( B − 5 + 2l ) ( B − 1 + 2l ) c −1 B−1 , l ( C ( B−32 ) l ( 1 ) − 2 ( B − 1 ) B − 2 C ( B−32 ) l ( −1 B − 1 ) ) = l ( B − 3 + l ) ( B − 3 + 2l ) ( B − 5 + 2l ) ( B − 1 + 2l ) ( C ( B−32 ) l ( 1 ) − 2 ( B − 1 ) B − 2 C ( B−32 ) l ( −1 B − 1 ) ) uB−1 , l : = B − 1 2 ( c−1B−1 , l+2κ−2 ( l + 2 , B − 3 2 ) ( C ( B−32 ) l ( 1 ) − 2 ( B − 1 ) B − 2 C ( B−32 ) l ( −1 B − 1 ) ) + c−1B−1 , lκ2 ( l , B − 3 2 ) ( C ( B−32 ) l+2 ( 1 ) − 2 ( B − 1 ) B − 2 C ( B−32 ) l+2 ( −1 B − 1 ) ) ) = B − 1 2 ( ( B − 3 + l ) ( B − 2 + l ) ( B − 3 ) ( B − 1 + 2l ) ( C ( B−32 ) l ( 1 ) − 2 ( B − 1 ) B − 2 C ( B−32 ) l ( −1 B − 1 ) ) + ( l + 1 ) ( l + 2 ) ( B − 3 ) ( B − 1 + 2l ) ( C ( B−32 ) l+2 ( 1 ) − 2 ( B − 1 ) B − 2 C ( B−32 ) l+2 ( −1 B − 1 ) ) ) . Note that vB−1,0 = wB−1,0 = uB−1,0 = 0 , so that there is in fact no dependence on a0 , as expected since batchnorm is invariant under additive shifts of φ . We see that λ↑L ≥ 1 iff 0 ≤ ∞∑ l=1 a2l ( wB−1 , l − vB−1 , l ) + alal+2uB−1 , l . This is a quadratic form on the coefficients { al } l. We now analyze it heuristically and argue that the eigenvalue is ≥ 1 typically when φ ( √ B − 1x ) explodes sufficiently as x→ 1 or x→ −1 ; in other words , the more explosive φ is , the less likely it is to induce Eq . ( 43 ) to converge to a BSB1 fixed point . Heuristically , C ( B−3 2 ) l+2 ( −1 B−1 ) is negligible compared to C ( B−32 ) l+2 ( 1 ) for sufficiently large B and l , so that wB−1 , l − vB−1 , l ≈ ( l ( B−3+l ) ( B−3 ) ( B−5+2l ) ( B−1+2l ) − 1 ) c−1B−1 , lC ( B−32 ) l+2 ( 1 ) = ( B−7 4 +O ( B −2 ) +O ( l−2 ) ) c−1B−1 , lC ( B−32 ) l+2 ( 1 ) and is positive . For small l , we can calculate wB−1,1 − vB−1,1 = −2B 1 +B < 0 wB−1,2 − vB−1,2 = B ( B + 1 ) ( B2 − 9B + 12 ) 2 ( B − 1 ) ( B + 3 ) ≥ 0 , ∀B ≥ 10 wB−1,3 − vB−1,3 = ( B − 3 ) ( B + 3 ) B2 ( 2B3 − 19B2 + 16B + 13 ) 6 ( B − 1 ) 2 ( B + 1 ) ( B + 5 ) ≥ 0 , ∀B ≥ 10 . In fact , plottingwB−1 , l−vB−1 , l for various values of l suggests that forB ≥ 10 , wB−1 , l−vB−1 , l ≥ 0 for all l ≥ 1 ( Fig . 10 ) . Thus , the more “ linear ” φ is , the larger a21 is compared to the rest of { al } l , and the more likely that the eigenvalue is < 1 . In the case that alal+2 = 0 ∀l , then indeed the eigenvalue is < 1 precisely when a21 is sufficiently large . Because higher degree Gegenbauer polynomials explodes more violently as x→ ±1 , this is consistent with our claim . Eigenvalue for M We use a similar technique shows the Gegenbauer expansion of the eigenvalue for M. Define Ãab ( c , d ) : = Ev∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) ( ec , :v ) ( ed , :v ) . Then Ãab ( c , c ) = A ( a , b ; c ) . Note the ultrasymmetries Ãab ( c , d ) = Ãπ ( a ) π ( b ) ( π ( c ) , π ( d ) ) for any permutation π , and Ãab ( c , d ) = Ãab ( d , c ) = Ãba ( c , d ) = Ãba ( d , c ) . By Eq . ( 46 ) , we have Ũ { δcδTd } ab = µ∗−1 E v∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) ( B − 1 2 ( ec , :v ) ( ed , :v ) − 1 2 ( I ( c = d ) − 1 B ) ) = µ∗−1 B − 1 2 Ã ( a , b ; c , d ) − µ∗−1 1 2 Σ∗ab ( I ( c = d ) − 1 B ) . By Thm E.62 , G⊗2 ◦ U has eigenspace M with eigenvalue λ↑M = Ũ { δ1δT2 } 12 + Ũ { δ2δT1 } 12 − 4Ũ { δ1δT3 } 12 + 2Ũ { δ3δT4 } 12 = 2 ( Ũ { δ1δT2 } 12 − 2Ũ { δ1δT3 } 12 + Ũ { δ3δT4 } 12 ) = B − 1 µ∗ ( Ã12 ( 12 ) − Ã12 ( 13 ) + Ã12 ( 34 ) ) . Thus we need to evaluate Ã12 ( 12 ) , Ã12 ( 13 ) , Ã12 ( 34 ) . We will do so by exploiting linear dependences between different values of Ãab ( c , d ) , the computed values of Ãab ( c , c ) = A ( a , b ; c ) , and the value of Ãab ( a , b ) computed below in Lemma F.23 . Indeed , we have B∑ d=1 Ãab ( c , d ) = E v∼SB−2 φ ( √ Bea , :v ) φ ( √ Beb , :v ) ( ec , :v ) B∑ d=1 ( ed , :v ) = 0 . Leveraging the symmmetries of Ã , we get ( B − 3 ) Ã12 ( 34 ) + 2Ã12 ( 13 ) + Ã12 ( 33 ) = 0 ( B − 2 ) Ã12 ( 13 ) + Ã12 ( 11 ) + Ã12 ( 12 ) = 0 . Expressing in terms of Ã12 ( 11 ) and Ã12 ( 12 ) , we get Ã12 ( 13 ) = −1 B − 2 ( Ã12 ( 11 ) + Ã12 ( 12 ) ) Ã12 ( 34 ) = −1 B − 3 ( 2Ã12 ( 13 ) + Ã12 ( 33 ) ) = −1 B − 3 ( −2 B − 2 ( Ã12 ( 11 ) + Ã12 ( 12 ) ) + 1 B − 2 ( Σ∗12 − 2Ã12 ( 22 ) ) ) By Eq . ( 47 ) = −1 ( B − 3 ) ( B − 2 ) ( Σ∗12 − 4Ã12 ( 11 ) − 2Ã12 ( 12 ) ) . These relations allow us to simplify λ↑M = B − 1 ( B − 3 ) µ∗ ( −1 B − 2Σ ∗ 12 + 2 ( B − 1 ) B − 2 Ã12 ( 11 ) + ( B − 1 ) Ã12 ( 12 ) ) . By Eq . ( 49 ) , we know the Gegenbauer expansion of Ã12 ( 11 ) . The following shows the Gegenbauer expansion of Ã12 ( 12 ) . Lemma F.23 . Suppose φ ( √ B − 1x ) , xφ ( √ B − 1x ) ∈ L2 ( ( 1 − x2 ) B−32 ) , and φ ( √ B − 1x ) has Gegenbauer expansion ∑∞ l=0 al 1 cB−1 , l C ( B−32 ) l ( x ) . Then Ã12 ( 12 ) = E v∼SB−2 φ ( √ Be1 , :v ) φ ( √ Be2 , :v ) ( e1 , :v ) ( e2 , :v ) = ∞∑ l=0 αla 2 l + βlalal+2 where αl = B − 1 B × ( l+1 B−1+2l ) 2 c−1B−1 , l+1C ( B−32 ) l+1 ( −1 B−1 ) + ( B+l−4 B+2l−5 ) 2 c−1B−1 , l−1C ( B−32 ) l−1 ( −1 B−1 ) if l ≥ 1 ( 1 B−1 ) 2 c−1B−1,1C ( B−32 ) 1 ( −1 B−1 ) otherwise βl = B − 1 B × 2 ( l + 1 ) ( B + l − 2 ) ( B + 2l − 1 ) 2 c −1 B−1 , l+1C ( B−32 ) l+1 ( −1 B − 1 ) . Proof . Let ψ ( x ) = xφ ( √ B − 1x ) . Then by Proposition E.43 , we have ψ ( x ) = ∞∑ l=0 alc −1 B−1 , l 1 B − 3 + 2l ( ( l + 1 ) C ( B−32 ) l+1 ( x ) + ( B + l − 4 ) C ( B−32 ) l−1 ( x ) ) = ∞∑ l=0 ( al−1 l B + 2l − 3 + al+1 B + l − 3 B + 2l − 3 ) c−1B−1 , lC ( B−32 ) l ( x ) . Then Ã12 ( 12 ) = B − 1 B E v∼SB−2 ψ ( ê1 , :v ) ψ ( ê2 , :v ) = B − 1 B ∞∑ l=0 ( al−1 l B + 2l − 3 + al+1 B + l − 3 B + 2l − 3 ) 2 c−1B−1 , lC ( B−32 ) l ( −1 B − 1 ) . Rearranging the sum in terms of al gives the result . Combining all of the above , we obtain the following Theorem F.24 . Suppose φ ( √ B − 1x ) has Gegenbauer expansion∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) . Then the eigenvalue of U with respect to the eigenspace M is a ratio of quadratic forms λ↑M = ∑∞ l=0 a 2 l w̃B−1 , l + alal+2ũB−1 , l∑∞ l=0 a 2 l vB−1 , l where ũB−1 , l : = 2 ( B − 1 ) 3 ( B − 3 ) 2 ( B − 2 ) B ( ( B − 3 + l ) ( B − 2 + l ) B − 1 + 2l C ( B−32 ) l ( −1 B − 1 ) + ( B − 2 ) ( l + 1 ) ( B − 2 + l ) B − 1 + 2l C ( B−32 ) l+1 ( −1 B − 1 ) + ( l + 3 ) ( l + 4 ) ( B − 3 + 2l ) ( B + 2l + 1 ) ( B + 2l + 3 ) C ( B−32 ) l+2 ( −1 B − 1 ) ) w̃B−1 , l : = B − 1 ( B − 3 ) 2 ( B − 2 ) [ ( 2 ( B − 1 ) 2 ( B + 2l − 3 ) ( 2Bl +B + 2l2 − 6l − 5 ) B ( B + 2l − 5 ) ( B + 2l − 1 ) − ( B − 3 + 2l ) ) C ( B−32 ) l ( −1 B − 1 ) + ( B − 2 ) ( B − 1 ) 2 ( l + 1 ) 2 B ( B + 2l − 1 ) C ( B−32 ) l+1 ( −1 B − 1 ) + ( B − 2 ) ( B − 1 ) 2 ( B − 4 + l ) 2 B ( B + 2l − 5 ) C ( B−32 ) l−1 ( −1 B − 1 ) ] where C ( B−3 2 ) −1 ( x ) = 0 by convention . Note that ũB−1 , l and w̃B−1 , l depends only on C ( B−32 ) • ( −1 B−1 ) which is much smaller than C ( B−32 ) • ( 1 ) for degree larger than 0 . Thus the eigenvalue for M is typically much smaller than the eigenvalue for L ( though there are counterexamples , like sin ) . F.3.3 LAPLACE METHOD Differentiating Eq . ( 26 ) In what follows , let φ be a positive-homogeneous function of degree α . We begin by studying dV ( φ◦n ) ( Σ G ) dΣG . We will differentiate Eq . ( 26 ) directly at G⊗2 { Σ∗ } = GΣ∗G where Σ∗ is the BSB1 fixed point given by Thm F.8 . To that end , consider a smooth path Σt ∈ SGB , t ∈ ( − , ) for some > 0 , with Σ0 = GΣ∗G . Set Σ t = eTΣte ∈ SB−1 , so that Σt = eΣ te T and Σ 0 = µ∗IB−1 where µ∗ = Kα , B ( Jφ ( 1 ) − Jφ ( −1B−1 ) ) as in Thm F.8 . If we write Σ̇ and Σ̇ for the time derivatives , we have B−αΓ ( α ) d dt V ( φ◦n ) ( eΣ te T ) ∣∣∣∣ t=0 = B−αΓ ( α ) d dt VBφ ( eΣ te T ) ∣∣∣∣ t=0 = ∫ ∞ 0 ds sα−1 [ d dt det ( I + 2sΣ t ) −1/2 ∣∣∣∣ t=0 ×Vφ ( eΣ 0 ( I + 2sΣ 0 ) −1 e T ) + det ( I + 2sΣ 0 ) −1/2 d dt Vφ ( eΣ t ( I + 2sΣ t ) −1 e T ) ∣∣∣∣ t=0 ] = ∫ ∞ 0 ds sα−1 [ − s 1 + 2sµ∗ det ( I + 2sΣ 0 ) −1/2 tr ( Σ̇ 0 ) × ( µ∗ 1 + 2sµ∗ ) α Vφ ( G ) + det ( I + 2sΣ 0 ) −1/2 dVφ ( Σ ) dΣ ∣∣∣∣ Σ=eΣ 0 ( I+2sΣ 0 ) −1 e T { e ( I + 2sΣ 0 ) −1Σ̇ 0 ( I + 2sΣ 0 ) −1 e T } ] ( apply Lemma F.26 and Lemma F.27 ) = ∫ ∞ 0 ds sα−1 [ − s 1 + 2sµ∗ ( 1 + 2sµ∗ ) − ( B−1 ) /2 tr ( Σ̇ 0 ) × ( µ∗ 1 + 2sµ∗ ) α Vφ ( G ) + ( 1 + 2sµ∗ ) − ( B−1 ) /2 ( µ∗ 1 + 2sµ∗ ) α−1 dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { ( 1 + 2sµ∗ ) −2eΣ̇ 0e T } ] ( using fact that dVφ/dΣ is degree ( α− 1 ) -positive homogeneous ) = − ( ∫ ∞ 0 ( 1 + 2sµ∗ ) − B−1 2 −1−αµ∗αsα ) tr ( Σ̇ 0 ) Vφ ( G ) + ( ∫ ∞ 0 ( 1 + 2sµ∗ ) − B−1 2 −1−αµ∗α−1sα−1 ) dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { eΣ̇ 0e T } = −µ∗−12−1−α Beta ( B − 1 2 , α+ 1 ) tr ( Σ̇ 0 ) Vφ ( G ) + µ∗−12−α Beta ( B + 1 2 , α ) dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { Σ̇0 } ( apply Lemma F.28 ) if B−12 + 2 > α ( precondition for Lemma F.28 ) . With some trivial simplifications , we obtain the following Lemma F.25 . Let φ be positive-homogeneous of degree α . Consider a smooth path Σt ∈ SGB with Σ0 = µ ∗G . If B−12 + 2 > α , then d dt V ( φ◦n ) ( Σt ) ∣∣∣∣ t=0 = dV ( φ◦n ) ( Σ ) dΣ ∣∣∣∣ Σ=µ∗G { Σ̇0 } = −αBαµ∗−12−1−αP ( B − 1 2 , α+ 1 ) −1 tr ( Σ̇ 0 ) Vφ ( G ) +Bαµ∗−12−αP ( B + 1 2 , α ) −1 dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { Σ̇0 } ( 50 ) where P ( a , b ) = Γ ( a+ b ) /Γ ( a ) is the Pochhammer symbol . Lemma F.26 . For any s ∈ R , ddt det ( I + 2sΣ ) −1/2 = −sdet ( I + 2sΣ ) −1/2 tr ( ( I + 2sΣ ) −1dΣ /dt ) . For Σ 0 = µI , this is also equal to − s1+2sµ det ( I + 2sΣ ) −1/2 tr ( ddtΣ ) at t = 0 . Proof . Straightforward computation . Lemma F.27 . For any s ∈ R , ddtΣ ( I + 2sΣ ) −1 = ( I + 2sΣ ) dΣ dt ( I + 2sΣ ) −1 . Proof . Straightforward computation . Lemma F.28 . For a > b+ 1 , ∫∞ 0 ( 1 + 2sµ ) −asb ds = ( 2µ ) −1−b Beta ( b+ 1 , a− 1− b ) . Proof . Apply change of variables x = 2µs1+2µs . This immediately gives the following consequence . Theorem F.29 . Let φ : R→ R be any function with finite first and second Gaussian moments . Then G⊗2 ◦ dVφdΣ ∣∣ Σ=BSB1 ( a , b ) : HGB → HGB has the following eigendecomposition : • RG has eigenvalue ( B−1 ) ( u−2v ) +wB ( dimRG = 1 ) • MB has eigenvalue w ( dimMB = B ( B − 3 ) /2 ) • LB has eigenvalue ( B−2 ) ( u−2v ) +2wB ( dimLB = B − 1 ) where u = ∂Vφ ( Σ ) 11 ∂Σ11 ∣∣∣∣ Σ=BSB1 ( a , b ) , v = ∂Vφ ( Σ ) 12 ∂Σ11 ∣∣∣∣ Σ=BSB1 ( a , b ) , w = ∂Vφ ( Σ ) 12 ∂Σ12 ∣∣∣∣ Σ=BSB1 ( a , b ) . Theorem F.30 . Let φ : R → R be positive-homogeneous of degree α . Then for any p 6= 0 , c ∈ R , G⊗2 ◦ dVφdΣ ∣∣ Σ=BSB1 ( p , cp ) : HGB → HGB has the following eigendecomposition : • MB has eigenvalue cαpα−1J′φ ( c ) • LB has eigenvalue cαpα−1 1B ( ( B − 2 ) α [ Jφ ( 1 ) − Jφ ( c ) ] + ( 2 + c ( B − 2 ) ) J′φ ( c ) ) • RG has eigenvalue cαpα−1 1B ( ( B − 1 ) α [ Jφ ( 1 ) − Jφ ( c ) ] + ( 1 + c ( B − 1 ) ) J′φ ( c ) ) Proof . By Proposition E.22 , dVφdΣ ∣∣ Σ=BSB1 ( p , cp ) is DOS ( u , v , w ) with u = ∂Vφ ( Σ ) 11 ∂Σ11 ∣∣∣∣ Σ=BSB1 ( p , cp ) = cααp α−1Jφ ( 1 ) v = ∂Vφ ( Σ ) 12 ∂Σ11 ∣∣∣∣ Σ=BSB1 ( p , cp ) = 1 2 cαΣ 1 2 ( α−2 ) ii Σ 1 2α jj ( αJφ ( cij ) − cijJ′φ ( cij ) ) = 1 2 cαp α−1 ( αJφ ( c ) − cJ′φ ( c ) ) w = ∂Vφ ( Σ ) 12 ∂Σ12 ∣∣∣∣ Σ=BSB1 ( p , cp ) = cαΣ ( α−1 ) /2 ii Σ ( α−1 ) /2 jj J ′ φ ( cij ) = cαp α−1J′φ ( c ) With Thm E.73 , we can do the computation : • MB has eigenvalue w = cαpα−1J′φ ( c ) • LB has eigenvalue ( u− 2v ) ( B − 2 ) + 2w B = 1 B ( ( B − 2 ) ( cααp α−1Jφ ( 1 ) − cαpα−1 ( αJφ ( c ) − cJ′φ ( c ) ) ) + 2cαp α−1J′φ ( c ) ) = cαp α−1 1 B ( ( B − 2 ) α [ Jφ ( 1 ) − Jφ ( c ) ] + ( 2 + c ( B − 2 ) ) J′φ ( c ) ) • RG has eigenvalue ( B − 1 ) ( u− 2v ) + w B = 1 B ( ( B − 1 ) ( cααp α−1Jφ ( 1 ) − cαpα−1 ( αJφ ( c ) − cJ′φ ( c ) ) ) + cαp α−1J′φ ( c ) ) = cαp α−1 1 B ( ( B − 1 ) α [ Jφ ( 1 ) − Jφ ( c ) ] + ( 1 + c ( B − 1 ) ) J′φ ( c ) ) We record the following consequence which will be used frequently in the sequel . Theorem F.31 . Let φ : R → R be positive-homogeneous of degree α . Then G⊗2 ◦ dVφdΣ ∣∣ Σ=G : HGB → HGB has the following eigendecomposition : • MB has eigenvalue λG , φM ( B , α ) : = cα ( B−1 B ) α−1 J′φ ( −1 B−1 ) • LB has eigenvalue λG , φL ( B , α ) : = cα ( B−1 B ) α−1 1 B ( ( B − 2 ) α [ Jφ ( 1 ) − Jφ ( −1 B−1 ) ] + BB−1J ′ φ ( −1 B−1 ) ) • RG has eigenvalue λG , φG ( B , α ) : = cα ( B−1 B ) α α [ Jφ ( 1 ) − Jφ ( −1 B−1 ) ] Proof . Plug in p = B−1B and c = −1/ ( B − 1 ) for Thm F.30 . Theorem F.32 . Let φ : R → R be a degree α positive-homogeneous function . Then for p 6= 0 , c ∈ R , dVφdΣ ∣∣ Σ=BSB1 ( p , cp ) : HB → HB has the following eigendecomposition : • MB has eigenvalue cαpα−1J′φ ( c ) • LB ( a , b ) has eigenvalue cααpα−1Jφ ( 1 ) where a = 2 ( J′φ ( c ) − αJφ ( 1 ) ) and b = αJφ ( c ) − cJ′φ ( c ) Proof . Use Thm E.72 with the computations from the proof of Thm F.30 as well as the following computation w − u = cαpα−1J′φ ( c ) − cααpα−1Jφ ( 1 ) = cαp α−1 ( J′φ ( c ) − αJφ ( 1 ) ) v = 1 2 cαp α−1 ( αJφ ( c ) − cJ′φ ( c ) ) Theorem F.33 . Let φ be positive-homogeneous with degree α . Assume B−12 > α . The operator U = G⊗2 ◦ dV ( φ◦n ) ( Σ G ) dΣG ∣∣ ΣG=µ∗G : HGB → HGB has 3 distinct eigenvalues . They are as follows : 1. λ↑G ( B , α ) : = 0 with dimension 1 eigenspace RG . 2. λ↑M ( B , α ) : = B αµ∗−12−αP ( B + 1 2 , α ) −1 λG , φM ( B , α ) = cαB ( B − 1 ) α−1µ∗−12−αP ( B + 1 2 , α ) −1 J′φ ( −1 B − 1 ) with dimension B ( B−3 ) 2 eigenspace M. 3. λ↑L ( B , α ) : = B αµ∗−12−αP ( B + 1 2 , α ) −1 λG , φL ( B , α ) = cα ( B − 1 ) α−1µ∗−12−αP ( B + 1 2 , α ) −1 ( ( B − 2 ) α [ Jφ ( 1 ) − Jφ ( −1 B − 1 ) ] + B B − 1J ′ φ ( −1 B − 1 ) ) with dimension B − 1 eigenspace L. Proof . Item 1 . The case of λ↑G ( B , α ) . Since ( φ ◦ n ) ( µ∗G ) = ( φ ◦ n ) ( µ∗G + ωG ) for any ω , the operator dV ( φ◦n ) ( Σ G ) dΣG ∣∣ ΣG=µ∗G sends G to 0 . Item 2 . The case of λ↑M ( B , α ) . Assume Σ̇0 ∈ M , Thm F.31 gives ( with denoting Hadamard product , i.e . entrywise multiplication ) dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { Σ̇0 } = λG , φM ( B , α ) Σ̇0 = cα ( B − 1 B ) α−1 J′φ ( −1 B − 1 ) Σ̇0 Since tr ( Σ̇0 ) = 0 , Eq . ( 50 ) gives G⊗2 ◦ dV ( φ◦n ) ( Σ G ) dΣG ∣∣∣∣ ΣG=µ∗G { Σ̇0 } = G⊗2 ◦ d dt V ( φ◦n ) ( Σt ) ∣∣∣∣ t=0 { Σ̇0 } = Bαµ∗−12−αP ( B + 1 2 , α ) −1 G⊗2 ◦ dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { Σ̇0 } = Bαµ∗−12−αP ( B + 1 2 , α ) −1 cα ( B − 1 B ) α−1 J′φ ( −1 B − 1 ) G⊗2 { Σ̇0 } = cαB ( B − 1 ) α−1µ∗−12−αP ( B + 1 2 , α ) −1 J′φ ( −1 B − 1 ) Σ̇0 So the eigenvalue for M is λ↑L = cαB ( B − 1 ) α−1µ∗−12−αP ( B+1 2 , α ) −1 J′φ ( −1 B−1 ) . Item 3 . The case of λ↑L ( B , α ) . Let Σ̇0 = L ( B − 2 , 1 ) . Thm F.31 gives G⊗2 ◦ dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { Σ̇0 } = λG , φL ( B , α ) Σ̇0 where λG , φL ( B , α ) : = 1 B cα ( B − 1 B ) α−1 ( ( B − 2 ) α [ Jφ ( 1 ) − Jφ ( −1 B − 1 ) ] + B B − 1J ′ φ ( −1 B − 1 ) ) Since tr ( Σ̇0 ) = 0 , Eq . ( 50 ) gives G⊗2 ◦ dV ( φ◦n ) ( Σ G ) dΣG ∣∣∣∣ ΣG=µ∗G { Σ̇0 } = G⊗2 ◦ d dt V ( φ◦n ) ( Σt ) ∣∣∣∣ t=0 = Bαµ∗−12−αP ( B + 1 2 , α ) −1 G⊗2 ◦ dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { Σ̇0 } = Bαµ∗−12−αP ( B + 1 2 , α ) −1 λ̃Σ̇0 = ( B − 1 ) α−1cαµ∗−12−αP ( B + 1 2 , α ) −1 × ( ( B − 2 ) α [ Jφ ( 1 ) − Jφ ( −1 B − 1 ) ] + B B − 1J ′ φ ( −1 B − 1 ) ) × Σ̇0 So λ↑L = ( B−1 ) α−1cαµ∗−12−αP ( B+1 2 , α ) −1 ( ( B − 2 ) α [ Jφ ( 1 ) − Jφ ( −1 B−1 ) ] + BB−1J ′ φ ( −1 B−1 ) ) With some routine computation , we obtain Proposition F.34 . With φ and α as above , as B →∞ , λ↑L ( B , α ) ∼ α+B−1 ( 2α3 − 4α2 − α+ J′φ ( 0 ) Jφ ( 1 ) − Jφ ( 0 ) ) +O ( B−2 ) λ↑M ( B , α ) ∼ J′φ ( 0 ) Jφ ( 1 ) − Jφ ( 0 ) +B−1 ( ( 2α2 − 4α+ 1 ) J′φ ( 0 ) Jφ ( 1 ) − Jφ ( 0 ) − J′′φ ( 0 ) Jφ ( 1 ) − Jφ ( 0 ) − ( J′φ ( 0 ) Jφ ( 1 ) − Jφ ( 0 ) ) 2 ) +O ( B−2 ) We can compute , by Proposition E.23 , J′φ ( 0 ) Jφ ( 1 ) − Jφ ( 0 ) = ( a+ b ) 2 1√ π Γ ( α2 +1 ) 2 Γ ( α+ 12 ) ( a2 + b2 ) − ( a− b ) 2 1 2 √ π Γ ( α2 + 1 2 ) 2 Γ ( α+ 12 ) ≤ 2Γ ( α 2 + 1 ) 2 √ πΓ ( α+ 12 ) where the last part is obtained by optimizing over a and b . On α ∈ ( −1/2 , ∞ ) , this is greater than α iff α < 1 . Thus for α ≥ 1 , the maximum eigenvalue of U is always achieved by eigenspace L for large enough B. G BACKWARD DYNAMICS Let ’ s extend the definition of the V operator : Definition G.1 . Suppose Tx : Rn → Rm is a linear operator parametrized by x ∈ Rk . Then for a PSD matrix Σ ∈ Sk , define VT ( Σ ) : = E [ Tx⊗ Tx : x ∈ N ( 0 , Σ ) ] : Rn×n → Rm×m , which acts on n× n matrices Π by VT ( Σ ) { Π } = E [ TxΠTTx : x ∈ N ( 0 , Σ ) ] ∈ Rm×m . Under this definition , VB′φ ( Σ ) is a linear operator R B×B → RB×B . Recall the notion of adjoint : Definition G.2 . If V is a ( real ) vector space , then V † , its dual space , is defined as the space of linear functionals f : V → R on V . If T : V → W is a linear operator , then T † , its adjoint , is a linear operator W † → V † , defined by T † ( f ) = v 7→ f ( T ( v ) ) . If a linear operator is represented by a matrix , with function application represented by matrixvector multiplication ( matrix on the left , vector on the right ) , then the adjoint is represented by matrix transpose . The backward equation . In this section we are interested in the backward dynamics , given by the following equation Πl = VB′φ ( Σ l ) † { Πl+1 } where Σl is given by the forward dynamics . Particularly , we are interested in the specific case when we have exponential convergence of Σl to a BSB1 fixed point . Thus we will study the asymptotic approximation of the above , namely the linear system . Πl = VB′φ ( Σ ∗ ) † { Πl+1 } ( 51 ) where Σ∗ is the BSB1 fixed point . Note that after one step of backprop , ΠL−1 = VB′φ ( Σ ∗ ) † { ΠL } is inHGB . Thus the large L dynamics of Eq . ( 51 ) is given by the eigendecomposition of VB′φ ( Σ ∗ ) † HGB : HGB → HGB . It turns out to be much more convenient to study its adjoint G⊗2 ◦ VB′φ ( Σ ∗ ) , which has the same eigenvalues ( in fact , it will turn out that it is self-adjoint ) . G.1 EIGENDECOMPOSITION OF G⊗2 ◦VB′φ ( Σ ∗ ) . By chain rule , B′φ ( x ) = dφ◦n ( z ) dz ∣∣ z=Gx dGx dx = dφ◦n ( z ) dz ∣∣ z=Gx G , B′φ ( x ) ∆x = dφ◦n ( z ) dz ∣∣ z=Gx G∆x Thus VB′φ ( Σ ) = E [ B′φ ( x ) ⊗2 : x ∼ N ( 0 , Σ ) ] = E [ ( dφ ◦ n ( z ) dz ∣∣∣∣ z=Gx G ) ⊗2 : x ∼ N ( 0 , Σ ) ] = E [ dφ ◦ n ( z ) dz ∣∣∣∣⊗2 z=Gx ◦G⊗2 : x ∼ N ( 0 , Σ ) ] = E [ dφ ◦ n ( z ) dz ∣∣∣∣⊗2 z=Gx : x ∼ N ( 0 , Σ ) ] ◦G⊗2 = V [ x 7→ dφ◦n ( z ) dz |z=Gx ] ( Σ ) ◦G ⊗2 If we let F ( Σ ) : = V [ x7→ dφ◦n ( z ) dz |z=Gx ] ( Σ ) , then VB ′ φ ( Σ ) = F ( Σ ) ◦ G⊗2 . As discussed above , we will seek the eigendecomposition of G⊗2 ◦ F ( Σ ) HGB : HGB → HGB . We first make some basic calculations . Proposition G.3 . dn ( z ) dz ∣∣∣∣ z=y = √ Br−1 ( I − vvT ) , where r = ‖y‖ , v = y/‖y‖ = n ( y ) / √ B . Proof . We have dn ( z ) dz ∣∣∣ z=y = √ B dz/‖z‖dz ∣∣∣ z=y , and ∂ ( yi/‖y‖ ) ∂yj = δij‖y‖ − ( ∂‖y‖/∂yj ) yi ‖y‖2 = δij ‖y‖ − yi ‖y‖2 ∂ √ y · y ∂yj = δij ‖y‖ − yi ‖y‖2 yj ‖y‖ = δij ‖y‖ − yiyj ‖y‖3 so that dz/‖z‖ dz ∣∣∣∣ z=y = r−1 ( I − vvT ) . By chain rule , this easily gives Proposition G.4 . dφ ◦ n ( z ) dz ∣∣∣∣ z=y = √ BDr−1 ( I − vvT ) , where D = Diag ( φ′ ( n ( y ) ) ) , r = ‖y‖ , v = y/‖y‖ = n ( y ) / √ B . With v = y/‖y‖ , r = ‖y‖ , and D̃ = Diag ( φ′ ( n ( y ) ) ) , we then have F ( Σ ) = E [ D̃⊗2 ◦ ( √ B dz/‖z‖ dz ∣∣∣∣ z=Gx ) ⊗2 : x ∼ N ( 0 , Σ ) ] = E D̃⊗2 ◦ ( √B dz/‖z‖ dz ∣∣∣∣ z=y ) ⊗2 : y ∼ N ( 0 , GΣG ) = B E [ r−2 ( D̃⊗2 + ( D̃vvT ) ⊗2 − D̃ ⊗ ( D̃vvT ) − ( D̃vvT ) ⊗ D̃ ) : y ∼ N ( 0 , GΣG ) ] ( 52 ) F ( Σ ) { Λ } = B E [ r−2 ( D̃ΛD̃ + D̃vvTΛvvT D̃ − D̃ΛvvT D̃ − D̃vvTΛD̃ ) : y ∼ N ( 0 , GΣG ) ] G.1.1 SPHERICAL INTEGRATION When Σ = Σ∗ is the BSB1 fixed point , one can again easily see that F ( Σ∗ ) is ultrasymmetric . With τij = 1 2 ( δiδ T j + δjδ T i ) ( Defn F.20 ) , we have F ( Σ∗ ) { Λ } = B E [ r−2 ( D̃ΛD̃ + D̃vvTΛvvT D̃ − D̃ΛvvT D̃ − D̃vvTΛD̃ ) : y ∼ N ( 0 , µ∗G ) ] = µ∗−1B E [ r−2 ( D̃ΛD̃ + D̃vvTΛvvT D̃ − D̃ΛvvT D̃ − D̃vvTΛD̃ ) : y ∼ N ( 0 , G ) ] F ( Σ∗ ) { τij } kl = µ∗−1B E rv∼N ( 0 , G ) [ r−2 ( φ′ ( √ Bvk ) φ ′ ( √ Bvl ) I ( k = i & l = j ) + I ( k = j & l = i ) 2 + vivjvkφ ′ ( √ Bvk ) vlφ ′ ( √ Bvl ) − 1 2 ( I ( i = k ) φ′ ( √ Bvk ) vjvlφ ′ ( √ Bvl ) + I ( j = k ) φ′ ( √ Bvk ) vivlφ ′ ( √ Bvl ) + I ( i = l ) φ′ ( √ Bvl ) vjvkφ ′ ( √ Bvk ) + I ( j = l ) φ′ ( √ Bvl ) vivkφ ′ ( √ Bvk ) ) ) ] = µ∗−1B Γ ( ( B − 3 ) /2 ) Γ ( ( B − 5 ) /2 ) 2 −2/2π−2 ∫ π 0 dθ1 · · · ∫ π 0 dθ4 f ( v θ 1 , . . . , v θ 4 ) sin B−3 θ1 · · · sinB−6 θ4 = µ∗−1B ( B − 5 ) ( 2π ) −2 ∫ π 0 dθ1 · · · ∫ π 0 dθ4 f ( v θ 1 , . . . , v θ 4 ) sin B−3 θ1 · · · sinB−6 θ4 by Lemma E.35 , where we assume , WLOG by ultrasymmetry , k , l ∈ { 1 , 2 } ; i , j ∈ { 1 , . . . , 4 } , and f ( v1 , . . . , v4 ) : = 1 2 φ′ ( √ Bev ) kφ ′ ( √ Bev ) l ( I ( k = i & l = j ) + I ( k = j & l = i ) + 2 ( ev ) i ( ev ) j ( ev ) k ( ev ) l − ( I ( i = k ) ( ev ) j ( ev ) l + I ( j = k ) ( ev ) i ( ev ) l + I ( i = l ) ( ev ) j ( ev ) k + I ( j = l ) ( ev ) i ( ev ) k ) ) vθ1 = cos θ1 vθ2 = sin θ1 cos θ2 vθ3 = sin θ1 sin θ2 cos θ3 vθ4 = sin θ1 sin θ2 sin θ3 cos θ4 . and e as in Eq . ( 33 ) . We can integrate out θ3 and θ4 symbolically by Lemma E.29 since their dependence only appear outside of φ′ . This reduces each entry of F ( Σ∗ ) { τij } kl to 2-dimensional integrals to evaluate numerically . The eigenvalues of G⊗2 ◦ F ( Σ∗ ) can then be obtained from Thm E.62 . We omit details here and instead focus on results from other methods . G.1.2 GEGENBAUER EXPANSION Notice that when Λ = G , the expression for F ( Σ∗ ) { Λ } significantly simplifies because G acts as identity on v : F ( Σ∗ ) { G } = µ∗−1B E [ r−2 ( D̃ΛD̃ + D̃vvTGvvT D̃ − D̃GvvT D̃ − D̃vvTGD̃ ) : y ∼ N ( 0 , G ) ] = µ∗−1B E [ r−2 ( D̃GD̃ − D̃vvT D̃ ) : y ∼ N ( 0 , G ) ] Then the diagonal of the image satisfies F ( Σ∗ ) { G } aa = µ∗−1B E [ r−2 ( B − 1 B φ′ ( √ Bva ) 2 − v2aφ′ ( √ Bva ) 2 ) : rv ∼ N ( 0 , G ) ] = µ∗−1B ( B − 3 ) −1 E v∼SB−2 ( B − 1 B φ′ ( √ Bea , :v ) 2 − ( ea , :v ) 2φ′ ( √ Bea , :v ) 2 ) K ( v ; G , 2 ) ( by Proposition E.33 ) = µ∗−1B ( B − 3 ) −1 E v∼SB−2 ( 1− ( êa , :v ) 2 ) B − 1 B φ′ ( √ B − 1êa , :v ) 2 ( K ( v ; G , 2 ) = 1 ) = µ∗−1 ( B − 1 ) ( B − 3 ) −1 E v∼SB−2 ( 1− ( êa , :v ) 2 ) φ′ ( √ B − 1êa , :v ) 2 And the off-diagonal entries satisfy F ( Σ∗ ) { G } ab = µ∗−1B E [ r−2 ( −1 B φ′ ( √ Bea , :v ) φ ′ ( √ Beb , :v ) − ( ea , :v ) ( eb , :v ) φ′ ( √ Bea , :v ) φ ′ ( ea , :v ) ] : rv ∼ N ( 0 , G ) ] = µ∗−1B ( B − 3 ) −1 E v∼SB−2 ( −1 B φ′ ( √ B − 1êa , :v ) φ′ ( √ B − 1êb , :v ) −B − 1 B ( êa , :v ) ( êb , :v ) φ ′ ( √ B − 1êa , :v ) φ′ ( √ B − 1êb , :v ) ) K ( v ; G , 2 ) ( by Proposition E.33 ) = µ∗−1B ( B − 3 ) −1 E v∼SB−2 −1 B ( 1 + ( B − 1 ) ( êa , :v ) ( êb , :v ) ) φ′ ( √ B − 1êa , :v ) φ′ ( √ B − 1êb , :v ) ( K ( v ; G , 2 ) = 1 ) = −µ∗−1 ( B − 3 ) −1 E v∼SB−2 ( 1 + ( B − 1 ) ( êa , :v ) ( êb , :v ) ) φ′ ( √ B − 1êa , :v ) φ′ ( √ B − 1êb , :v ) Lifting Spherical Expectation . We first show a way of expressing F ( Σ∗ ) aa in terms of Gegenbauer basis by lifting the spherical integral over SB−2 to spherical integral over SB . While this technique can not be extended to F ( Σ∗ ) ab , a 6= b , we believe it could be of use to future related problems . If φ ( √ B − 1x ) has Gegenbauer expansion ∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) , then φ′ ( √B − 1x ) = 1√ B−1 ∑∞ l=0 al 1 cB−1 , l C ( B−32 ) l ′ ( x ) = 1√ B−1 ∑∞ l=0 al B−3 cB−1 , l C ( B−12 ) l−1 ( x ) , so that φ ′ ( √ Bea , :v ) = 1√ B−1 ∑∞ l=0 al B−3 cB−1 , l C ( B−12 ) l−1 ( êa , :v ) = 1√ B−1 ∑∞ l=0 al ( B − 3 ) cB+1 , l−1 cB−1 , l Z B , ( l−1 ) êa , : ( v ) . Here êa , : and v are treated as points on the B-dimensional sphere with last 2 coordinates 0 . Thus by Lemmas E.38 and E.39 , F ( Σ∗ ) { G } aa = µ∗−1 ωB−3 ωB−2 ( B − 1 ) ( B − 3 ) −1 ∫ 1 −1 dh ( 1− h2 ) B−42 ( 1− h2 ) φ′ ( √ B − 1h ) 2 = µ∗−1 ωB−3 ωB−2 ( B − 1 ) ( B − 3 ) −1 ωB ωB−1 ωB−1 ωB ∫ 1 −1 dh ( 1− h2 ) B−22 φ′ ( √ B − 1h ) 2 = µ∗−1 ωB−3ωB ωB−2ωB−1 ( B − 1 ) ( B − 3 ) −1 E w∼SB φ′ ( √ B − 1˜̂ea , :w ) 2 = µ∗−1 ωB−3ωB ωB−2ωB−1 ( B − 1 ) ( B − 3 ) −1 E w∼SB ( 1√ B − 1 ∞∑ l=0 al ( B − 3 ) cB+1 , l−1 cB−1 , l Z B , ( l−1 ) ˜̂ ea , : ( w ) ) 2 = µ∗−1 ωB−3ωB ωB−2ωB−1 ( B − 3 ) ∞∑ l=0 a2l ( cB+1 , l−1 cB−1 , l ) 2 c−1B+1 , l−1C ( B−12 ) l−1 ( 1 ) = µ∗−1 ωB−3ωB ωB−2ωB−1 ( B − 1 ) ∞∑ l=0 a2l 1 cB−1 , l C ( B−12 ) l−1 ( 1 ) by Lemma E.45 = µ∗−1 ( B − 2 ) ∞∑ l=0 a2l 1 cB−1 , l ( l +B − 3 l − 1 ) = µ∗−1 ( B − 2 ) ∞∑ l=0 a2l 1 cB−1 , l ( l +B − 3 B − 2 ) = µ∗−1 ∞∑ l=0 a2l l +B − 3 cB−1 , l ( l +B − 4 B − 3 ) = µ∗−1 ∞∑ l=0 a2l l +B − 3 cB−1 , l l B − 3 ( l +B − 4 B − 4 ) Dirichlet-like form of φ . For F ( Σ∗ ) { G } ab , there ’ s no obvious way of lifting the spherical expectation to a higher dimension with the weight ( 1 + ( B − 1 ) ( êa , :v ) ( êb , :v ) ) . We will instead leverage Thm E.47 . In Thm E.47 , setting u1 = êa , : = u2 , we get F ( Σ∗ ) { G } aa = µ∗−1 B − 1 B − 3 Ev∼SB−2 ( ( êa , : · êa , : ) − ( êa , :v ) 2 ) φ′ ( √ B − 1êa , :v ) 2 = µ∗−1 ∞∑ l=0 a2l l +B − 3 B − 3 lc −1 B−1 , lC ( B−32 ) l ( 1 ) . Likewise , setting u1 = êa , : , u2 = êb , : , we get F ( Σ∗ ) { G } ab = µ∗−1 B − 1 B − 3 Ev∼SB−2 ( ( êa , : · êb , : ) − ( êa , :v ) ( êb , :v ) ) φ ′ ( √ B − 1êa , :v ) φ′ ( √ B − 1êb , :v ) = µ∗−1 ∞∑ l=0 a2l l +B − 3 B − 3 lc −1 B−1 , lC ( B−32 ) l ( −1 B − 1 ) Thus , Theorem G.5 . Suppose φ ( √ B − 1x ) , φ′ ( √ B − 1x ) ∈ L2 ( ( 1 − x2 ) B−32 ) and φ ( √ B − 1x ) has Gegenbauer expansion ∑∞ l=0 al 1 cB−1 , l C ( B−32 ) l ( x ) . Then for Σ ∗ being the unique BSB1 fixed point of Eq . ( 43 ) , the eigenvalue of G⊗2 ◦ F ( Σ∗ ) corresponding to eigenspace RG is λ↓G = µ ∗−1 ∞∑ l=0 a2l l +B − 3 B − 3 lc −1 B−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B − 1 ) ) = ∑∞ l=0 a 2 l l+B−3 B−3 lc −1 B−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) ∑∞ l=0 a 2 l c −1 B−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) > 1 This is minimized ( over choices of φ ) iff φ is linear , in which case λ↓G = B−2 B−3 . G.2 LAPLACE METHOD In this section suppose that φ is degree α positive-homogeneous . Set D = Diag ( φ′ ( y ) ) ( and recall v = y/‖y‖ , r = ‖y‖ ) . Then D is degree α − 1 positive-homogeneous in x ( because φ′ is ) . Consequently we can rewrite Eq . ( 52 ) as follows , F ( Σ∗ ) = Bα E y∼N ( 0 , GΣ∗G ) [ r−2 ( r−2 ( α−1 ) D⊗2 + r−2 ( α+1 ) ( DvvT ) ⊗2 − r−2αD ⊗ ( DvvT ) − r−2α ( DvvT ) ⊗D ) ] = Bα E y∼N ( 0 , µ∗G ) [ r−2αD⊗2 + r−2 ( α+2 ) ( DyyT ) ⊗2 − r−2 ( α+1 ) ( D ⊗ ( DyyT ) + ( DyyT ) ⊗D ) ] = Bα ( A + B− C ) ( 53 ) where Σ∗ is the BSB1 fixed point of Eq . ( 43 ) , GΣ∗G = µ∗G , and A = E [ r−2αD⊗2 : y ∼ N ( 0 , µ∗G ) ] B = E [ r−2 ( α+2 ) ( DyyT ) ⊗2 : y ∼ N ( 0 , µ∗G ) ] C = E [ r−2 ( α+1 ) ( D ⊗ ( DyyT ) + ( DyyT ) ⊗D ) : y ∼ N ( 0 , µ∗G ) ] Each term in the sum above is ultrasymmetric , so has the same eigenspaces RG , M , L ( this will also become apparent in the computations below without resorting to Thm E.61 ) . We can thus compute the eigenvalues for each of them in order . In all computation below , we apply Lemma E.2 first to relate the quantity in question to Vφ . Computing A . For two matrices Σ , Λ , write Σ Λ for entrywise multiplication . We have by Lemma E.2 A { Λ } = E [ r−2αDΛD : y ∼ N ( 0 , µ∗G ) ] = E [ r−2αΛ φ′ ( y ) φ′ ( y ) T : y ∼ N ( 0 , µ∗G ) ] = Γ ( α ) −1 ∫ ∞ 0 ds sα−1 det ( I + 2sµ∗G ) −1/2Λ Vφ′ ( µ∗ 1 + 2sµ∗ G ) = Γ ( α ) −1 ∫ ∞ 0 ds sα−1 ( 1 + 2sµ∗ ) − ( B−1 ) /2Λ ( µ∗ 1 + 2sµ∗ ) α−1 Vφ′ ( G ) = Γ ( α ) −1Λ Vφ′ ( G ) ∫ ∞ 0 ds ( µ∗s ) α−1 ( 1 + 2sµ∗ ) − ( B−1 ) /2−α+1 = Γ ( α ) −1Λ Vφ′ ( G ) Beta ( B − 3 2 , α ) µ∗−12−α = Λ cα−1 ( B − 1 B ) α−1 BSB1 ( Jφ′ ( 1 ) , Jφ′ ( −1 B − 1 ) ) P ( B − 3 2 , α ) −1 µ∗−12−α Then Thm E.73 gives the eigendecomposition for BαG⊗2 ◦ A HGB Theorem G.6 . LetRB , α : = ( 2α−1 ) cα−1B ( B−1 ) α−1P ( B−3 2 , α ) −1 2−α . ThenBαG⊗2◦A HGB has the following eigendecomposition . λAG = RB , αµ ∗−1 ( B − 1 B J′φ ( 1 ) + 1 B J′φ ( −1 B − 1 ) ) = ( 2α+B − 3 ) ( α′2 ( B − 1 ) Jφ ( 1 ) + ( 2α− 1 ) Jφ ( 1 1−B ) ) ( 2α− 1 ) ( B − 3 ) ( B − 1 ) ( J ( 1 ) − J ( 1 1−B ) ) λAL = RB , αµ ∗−1 ( B − 2 B J′φ ( 1 ) + 2 B J′φ ( −1 B − 1 ) ) = ( 2α+B − 3 ) ( α2 ( B − 2 ) Jφ ( 1 ) + 2 ( 2α− 1 ) J′φ ( 1 1−B ) ) ( 2α− 1 ) ( B − 3 ) ( B − 1 ) ( J ( 1 ) − J ( 1 1−B ) ) λAM = RB , αµ ∗−1J′φ ( −1 B − 1 ) = B ( 2α+B − 3 ) J′φ ( 1 1−B ) ( B − 3 ) ( B − 1 ) ( Jφ ( 1 ) − Jφ ( 1 1−B ) ) Proof . We have BαA = ( 2α − 1 ) −1RB , αµ∗−1DOS ( Jφ′ ( 1 ) , 0 , Jφ′ ( −1 B−1 ) ) . This allows us to apply Thm E.73 . We make a further simplification Jφ′ ( c ) = ( 2α−1 ) J′φ ( c ) via Proposition E.22 . Computing B . We simplify B { Λ } = E [ r−2 ( α+2 ) DyyTΛyyTD : y ∼ N ( 0 , µ∗G ) ] = E [ r−2 ( α+2 ) ψ ( y ) yTΛyψ ( y ) T : y ∼ N ( 0 , µ∗G ) ] where ψ ( y ) : = yφ′ ( y ) , which , in this case of φ being degree α positive-homogeneous , is also equal to αφ ( y ) . By Lemma E.2 , B { Λ } equals Γ ( α+ 2 ) −1 ∫ ∞ 0 ds sα+1 det ( I + 2sµ∗G ) −1/2 E [ ψ ( y ) yTΛyψ ( y ) T : y ∼ N ( 0 , µ ∗ 1 + 2sµ∗ G ) ] This expression naturally leads us to consider the following definition . Definition G.7 . Let ψ : R→ R be measurable and Σ ∈ SB . Define V ( 4 ) ψ ( Σ ) : HB → HB by V ( 4 ) ψ ( Σ ) { Λ } = E [ ψ ( y ) yTΛyψ ( y ) T : y ∼ N ( 0 , Σ ) ] For two matrices Σ , Λ , write 〈Σ , Λ〉 : = tr ( ΣTΛ ) . We have the following identity . Proposition G.8 . V ( 4 ) φ ( Σ ) { Λ } = Vφ ( Σ ) 〈Σ , Λ〉+ 2 dVφ ( Σ ) dΣ { ΣΛΣ } . Proof . We have dVφ ( Σ ) dΣ { Λ } = ( 2π ) −B/2 ∫ dz φ ( z ) ⊗2 ( d dΣ det ( Σ ) −1/2e− 1 2 z TΣ−1z ) { Λ } = ( 2π ) −B/2 ∫ dz φ ( z ) ⊗2 〈 [ −1 2 det Σ−1/2Σ−1 + det Σ−1/2 1 2 Σ−1zzTΣ−1 ] e− 1 2 z TΣ−1z , Λ 〉 = 1 2 ( 2π ) −B/2 det Σ−1/2 ∫ dz φ ( z ) ⊗2e− 1 2 z TΣ−1z [ 〈zzT , Σ−1ΛΣ−1〉 − 〈Σ−1 , Λ〉 ] = 1 2 V ( 4 ) φ ( Σ ) { Σ−1ΛΣ−1 } − 1 2 Vφ ( Σ ) 〈Σ−1 , Λ〉 . Making the substitution Λ→ ΣΛΣ , we get the desired result . If ϕ is degree α positive-homogeneous , then V ( 4 ) ϕ is degree α+ 1 positive-homogeneous . Thus , B { Λ } = Γ ( α+ 2 ) −1 ∫ ∞ 0 ds sα+1 det ( I + 2sµ∗G ) −1/2V ( 4 ) ψ ( µ∗ 1 + 2sµ∗ G ) { Λ } = Γ ( α+ 2 ) −1 ∫ ∞ 0 ds sα+1 ( 1 + 2sµ∗ ) − ( B−1 ) /2 ( µ∗ 1 + 2sµ∗ ) α+1 V ( 4 ) ψ ( G ) { Λ } = Γ ( α+ 2 ) −1V ( 4 ) ψ ( G ) { Λ } ∫ ∞ 0 ds ( µ∗s ) α+1 ( 1 + 2sµ∗ ) − ( B−1 ) /2−α−1 = Γ ( α+ 2 ) −1V ( 4 ) ψ ( G ) { Λ } Beta ( B − 3 2 , α+ 2 ) 2−α−2µ∗−1 = P ( B − 3 2 , α+ 2 ) −1 2−α−2µ∗−1V ( 4 ) ψ ( G ) { Λ } . By Proposition G.8 , V ( 4 ) ψ ( G ) { Λ } = Vψ ( G ) tr Λ + 2 dVψ ( Σ ) dΣ { Λ } if Λ ∈ HGB . Thus Theorem G.9 . BαG⊗2 ◦ B HGB has the following eigendecomposition ( note that here µ∗ is still with respect to φ , not ψ = αφ ) 1 . Eigenspace RG with eigenvalue λBG : = α2 B − 3 2 . Eigenspace L with eigenvalue λBL : = B αP ( B − 3 2 , α+ 2 ) −1 2−α−2µ∗−12λG , ψL ( B , α ) = 2α3 ( B − 2 ) ( B − 3 ) ( B − 1 ) ( B − 1 + 2α ) + 2α2BJ′φ ( −1 B−1 ) ( B − 3 ) ( B − 1 ) 2 ( B − 1 + 2α ) ( Jφ ( 1 ) − Jφ ( −1 B−1 ) ) 3 . Eigenspace M with eigenvalue λBM : = B αP ( B − 3 2 , α+ 2 ) −1 2−α−2µ∗−12λG , ψM ( B , α ) = 2α2BJ′φ ( −1 B−1 ) ( B − 3 ) ( B − 1 ) ( B − 1 + 2α ) ( Jφ ( 1 ) − Jφ ( −1 B−1 ) ) Proof . The only thing to justify is the value of λBG . We have λBG = B αP ( B − 3 2 , α+ 2 ) −1 2−α−2µ∗−1 × ( ( B − 1 ) cα ( B − 1 B ) α ( Jψ ( 1 ) − Jψ ( −1 B − 1 ) ) + 2λG , ψM ( B , α ) ) = BαP ( B − 3 2 , α+ 2 ) −1 2−α−2µ∗−1 ( B − 1 + 2α ) cα ( B − 1 B ) α ( Jψ ( 1 ) − Jψ ( −1 B − 1 ) ) = cα ( B − 1 ) α ( B − 1 + 2α ) P ( B − 3 2 , α+ 2 ) −1 2−α−2µ∗−1 ( Jψ ( 1 ) − Jψ ( −1 B − 1 ) ) = cα ( B − 1 ) αP ( B − 3 2 , α+ 1 ) −1 2−α−1µ∗−1 ( Jψ ( 1 ) − Jψ ( −1 B − 1 ) ) = cα ( B − 1 ) αP ( B − 3 2 , α+ 1 ) −1 2−α−1K−1α , Bα 2 ( since Jψ = α2Jφ ) = α2cα ( B − 1 ) αP ( B−3 2 , α+ 1 ) −1 2−α−1 cαP ( B−1 2 , α ) −1 ( B−1 2 ) α ( Defn F.6 ) = α22−1 ( B − 3 ) /2 = α2 B − 3 Computing C. By Lemma E.2 , C { Λ } = E [ r−2 ( α+1 ) ( DΛyyTD +DyyTΛD ) : y ∼ N ( 0 , µ∗G ) ] = Γ ( α+ 1 ) −1 ∫ ds sα det ( I + 2sµ∗G ) −1/2 E [ DΛyyTD +DyyTΛD : y ∼ N ( 0 , µ ∗ 1 + 2sµ∗ G ) ] = Γ ( α+ 1 ) −1 ∫ ds sα ( 1 + 2sµ∗ ) − ( B−1 ) /2 ( µ∗ 1 + 2sµ∗ ) α E [ DΛyyTD +DyyTΛD : y ∼ N ( 0 , G ) ] = Γ ( α+ 1 ) −1 ∫ ds ( µ∗s ) α ( 1 + 2sµ∗ ) − ( B−1 ) /2−α E [ DΛyyTD +DyyTΛD : y ∼ N ( 0 , G ) ] = Γ ( α+ 1 ) −1 Beta ( B − 3 2 , α+ 1 ) 2−α−1µ∗−1 E [ DΛyyTD +DyyTΛD : y ∼ N ( 0 , G ) ] = P ( B − 3 2 , α+ 1 ) −1 2−α−1µ∗−1 E [ DΛyyTD +DyyTΛD : y ∼ N ( 0 , G ) ] Lemma G.10 . Suppose φ is degree α positive-homogeneous . Then for Λ ∈ HB , E [ D ⊗ ( DyyT ) +DyyT ⊗D : y ∼ N ( 0 , Σ ) ] = α dVφ ( ΠΣΠ ) dΠ ∣∣∣∣ Π=I E [ DΛyyTD +DyyTΛD : y ∼ N ( 0 , Σ ) ] = αdVφ ( Σ ) dΣ { ΣΛ + ΛΣ } where D = Diag ( φ′ ( y ) ) . Proof . Let Πt , t ∈ ( − , ) be a smooth path in HB with Π0 = I . Write Dt = Diag ( φ′ ( Πty ) ) , so that D0 = D. Then , using Π̇t to denote t derivative , d dt Vφ ( ΠtΣΠt ) = d dt E [ φ ( Πty ) φ ( Πty ) T : y ∼ N ( 0 , Σ ) ] = E [ DtΠ̇tyφ ( Πty ) T + φ ( Πty ) yT Π̇tDt : y ∼ N ( 0 , Σ ) ] d dt Vφ ( ΠtΣΠt ) ∣∣∣∣ t=0 = E [ DΠ̇0yφ ( y ) T + φ ( y ) yT Π̇0D : y ∼ N ( 0 , Σ ) ] Because for x ∈ R , αφ ( x ) = xφ′ ( x ) , for y ∈ RB we can write φ ( y ) = α−1Diag ( φ′ ( y ) ) y . Then α dVφ ( Σ ) dΣ { ΣΠ̇0 + Π̇0Σ } = α dVφ ( Σ ) dΣ { dΠtΣΠt dt } = α d dt Vφ ( ΠtΣΠt ) ∣∣∣∣ t=0 = E [ DΠ̇0yyTD +DyyT Π̇0D : y ∼ N ( 0 , Σ ) ] Therefore , for Λ ∈ HGB , C { Λ } = P ( B − 3 2 , α+ 1 ) −1 2−α−1µ∗−1α dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { GΛ + ΛG } = 2αP ( B − 3 2 , α+ 1 ) −1 2−α−1µ∗−1 dVφ ( Σ ) dΣ ∣∣∣∣ Σ=G { Λ } So Thm F.31 gives Theorem G.11 . BαG⊗2 ◦ C has the following eigendecomposition 1. eigenspace RG with eigenvalue λCG : = 2B ααP ( B − 3 2 , α+ 1 ) −1 2−α−1µ∗−1λG , φG ( B , α ) = 2α2 B − 3 2. eigenspace L with eigenvalue λCL : = 2B ααP ( B − 3 2 , α+ 1 ) −1 2−α−1µ∗−1λG , φL ( B , α ) = 2α2 ( B − 2 ) ( B − 3 ) ( B − 1 ) + 2αBJ′φ ( −1 B−1 ) ( B − 3 ) ( B − 1 ) 2 ( Jφ ( 1 ) − Jφ ( −1 B−1 ) ) 3. eigenspace M with eigenvalue λCM : = 2B ααP ( B − 3 2 , α+ 1 ) −1 2−α−1µ∗−1λG , φM ( B , α ) = 2αBJ′φ ( −1 B−1 ) ( B − 3 ) ( B − 1 ) ( Jφ ( 1 ) − Jφ ( −1 B−1 ) Altogether , by Eq . ( 53 ) and Thms G.6 , G.9 and G.11 , this implies Theorem G.12 . G⊗2 ◦ F ( Σ∗ ) : HGB → HGB has eigenspaces RG , M , L respectively with the following eigenvalues λ↓G = ( B − 3 + 2α ) ( ( 2α− 1 ) J′φ ( −1 B−1 ) + α2 ( B − 1 ) Jφ ( 1 ) ) ( 2α− 1 ) ( B − 3 ) ( B − 1 ) ( Jφ ( 1 ) − Jφ ( −1 B−1 ) ) − α 2 B − 3 = α2 ( B − 3 + 2α ) ( Jφ ( 1 ) + 1 B−1Jα−1φ′ ( −1 B−1 ) ) ( 2α− 1 ) ( B − 3 ) ( Jφ ( 1 ) − Jφ ( −1 B−1 ) ) − α 2 B − 3 = α2 ( ( B − 2 ) Jφ ( 1 ) + ( B − 3 + 2α ) 1B−1Jα−1φ′ ( −1 B−1 ) + ( 2α− 1 ) ( Jφ ( 1 ) − Jφ ( −1 B−1 ) ) ( 2α− 1 ) ( B − 3 ) ( Jφ ( 1 ) − Jφ ( −1 B−1 ) ) λ↓L = − 2α2 ( B − 2 ) ( B − 1 + α ) ( B − 3 ) ( B − 1 ) ( B − 1 + 2α ) + 2 α2 ( 3B − 4 ) + α ( 3B2 − 11B + 8 ) + ( B − 3 ) ( B − 1 ) 2 ( B − 3 ) ( B − 1 ) 2 ( B − 1 + 2α ) J′φ ( −1 B−1 ) Jφ ( 1 ) − Jφ ( −1 B−1 ) + α2 ( B − 2 ) ( B − 3 + 2α ) ( 2α− 1 ) ( B − 3 ) ( B − 1 ) Jφ ( 1 ) Jφ ( 1 ) − Jφ ( −1 B−1 ) λ↓M = B ( B2 + 2 ( α− 2 ) B + 2 ( α− 3 ) α+ 3 ) ( B − 3 ) ( B − 1 ) ( B − 1 + 2α ) J′φ ( −1 B−1 ) Jφ ( 1 ) − Jφ ( −1 B−1 ) . H CROSS BATCH : FORWARD DYNAMICS In this section , we study the generalization of Eq . ( 43 ) to multiple batches . Definition H.1 . For linear operators Ti : Xi → Yi , i = 1 , . . . , k , we write ⊕ i Ti : ⊕ i Xi → ⊕ i Yi for the operator ( x1 , . . . , xk ) 7→ ( T1 ( x1 ) , . . . , Tk ( xk ) ) . We also write T ⊕n1 : = ⊕n j=1 T1 for the direct sum of n copies of T1 . For k ≥ 2 , now consider the extended ( “ k-batch ” ) dynamics on Σ̃ ∈ SkB defined by Σ̃l = VB⊕kφ ( Σ̃l−1 ) ( 54 ) = E [ ( Bφ ( h1 : B ) , Bφ ( hB+1:2B ) , . . . , Bφ ( h ( k−1 ) B+1 : kB ) ) ⊗2 : h ∼ N ( 0 , Σ̃l−1 ) ] If we restrict the dynamics to just the upper left B × B submatrix ( as well as any of the diagonal B ×B blocks ) of Σ̃l , then we recover Eq . ( 43 ) . H.1 LIMIT POINTS In general , like in the case of Eq . ( 43 ) , it is difficult to prove global convergence behavior . Thus we manually look for fixed points and the local convergence behaviors around them . A very natural extension of the notion of BSB1 matrices is the following Definition H.2 . We say a matrix Σ̃ ∈ SkB is CBSB1 ( short for “ 1-Step Cross-Batch Symmetry Breaking ” ) if Σ̃ in block form ( k × k blocks , each of size B ×B ) has one common BSB1 block on the diagonal and one common constant block on the off-diagonal , i.e . BSB1 ( a , b ) c11T c11T · · · c11T BSB1 ( a , b ) c11T · · · c11T c11T BSB1 ( a , b ) · · · ... ... ... . . . We will study the fixed point Σ̃∗ to Eq . ( 54 ) of CBSB1 form . H.1.1 SPHERICAL INTEGRATION AND GEGENBAUER EXPANSION Theorem H.3 . Let Σ̃ ∈ SkB . If Σ̃ is CBSB1 then VB⊕kφ ( Σ̃ ) is CBSB1 and equals Σ̃ ∗ : = ( Σ∗ c∗ c∗ Σ∗ ) where Σ∗ is the BSB1 fixed point of Thm F.5 and c∗ ≥ 0 with √ c∗ = Γ ( ( B−1 ) /2 ) Γ ( ( B−2 ) /2 ) π −1/2 ∫ π 0 dθ φ ( − √ B − 1 cos θ ) sinB−3 θ . If φ ( √ B − 1x ) has Gegenbauer expansion∑∞ l=0 al 1 cB−1 , l C ( B−32 ) l ( x ) , then c ∗ = a20 . Proof . We will prove for k = 2 . The general k cases follow from the same reasoning . Let Σ̃ = ( Σ c11T c11T Σ ) where Σ = BSB1 ( a , b ) . As remarked below Eq . ( 54 ) , restricting to any diagonal blocks just recovers the dynamics of Eq . ( 43 ) , which gives the claim about the diagonal blocks being Σ∗ through Thm F.5 . We now look at the off-diagonal blocks . VB⊕2φ ( Σ̃ ) 1 : B , B+1:2B = E [ Bφ ( z1 : B ) ⊗ Bφ ( zB+1:2B ) : z ∼ N ( 0 , Σ̃ ) ] = E [ φ ◦ n ( y1 : B ) ⊗ φ ◦ n ( yB+1:2B ) : y ∼ N ( 0 , Σ̃G ) ] where Σ̃G : = G⊕2Σ̃G⊕2 = ( G 0 0 G ) Σ̃ ( G 0 0 G ) = ( GΣG cG11TG cG11TG GΣG ) = ( ( a− b ) G 0 0 ( a− b ) G ) ( the last step follows from Lemma E.52 ) . Thus y1 : B is independent from yB+1:2B , and VB⊕2φ ( Σ̃ ) 1 : B , B+1:2B = E [ φ ◦ n ( x ) : x ∼ N ( 0 , ( a− b ) G ) ] ⊗2 By symmetry , E [ φ ◦ n ( x ) : x ∼ N ( 0 , ( a− b ) G ) ] = √ c∗1 E [ φ ◦ n ( x ) : x ∼ N ( 0 , ( a− b ) G ) ] ⊗2 = c∗11T where √ c∗ : = E [ φ ( n ( x ) 1 ) : x ∼ N ( 0 , ( a− b ) G ) ] . We can compute√ c∗ = E [ φ ( n ( x ) 1 ) : x ∼ N ( 0 , G ) ] because n is scale-invariant = E [ φ ( n ( ex ) 1 ) : x ∼ N ( 0 , IB−1 ) ] where e is as in Eq . ( 33 ) = E [ φ ( ( en ( x ) ) 1 ) : x ∼ N ( 0 , IB−1 ) ] because e is an isometry = E [ φ ( √ B ( ev ) 1 ) : x ∼ N ( 0 , IB−1 ) ] with v = x/‖x‖ = Γ ( ( B − 1 ) /2 ) Γ ( ( B − 2 ) /2 ) π −1/2 ∫ π 0 dθ φ ( − √ B − 1 cos θ ) sinB−3 θ by Lemma E.37 . At the same time , by Proposition E.33 , we can obtain the Gegenbauer expansion√ c∗ = E v∼SB−2 φ ( √ B − 1ê1 , :v ) = E v∼SB−2 ∞∑ l=0 al 1 cB−1 , l C ( B−32 ) l ( e1 , :v ) = 〈 ∞∑ l=0 alZ B−2 , ( l ) ê1 , : , 1 〉 = a0Z B−2 , ( 0 ) ê1 , : ( ê1 , : ) = a0 . Corollary H.4 . With the notation as in Thm H.3 , if φ is positive-homogeneous of degree α and φ ( x ) = aρα ( x ) − bρα ( −x ) , then √ c∗ = ( a− b ) 1 2 √ π ( B − 1 ) α/2 Γ ( ( B−1 ) /2 ) Γ ( ( α+1 ) /2 ) Γ ( ( α+B−1 ) /2 ) . Proof . We compute ∫ π/2 0 dθ ( cos θ ) α sinB−3 θ = 1 2 Beta ( 0 , 1 ; α+ 1 2 , B − 2 2 ) by Lemma E.28 = 1 2 Beta ( α+ 1 2 , B − 2 2 ) ∫ π π/2 dθ ( − cos θ ) α sinB−3 θ = ∫ π/2 0 dθ ( cos θ ) α sinB−3 θ = 1 2 Beta ( α+ 1 2 , B − 2 2 ) So for a positive homogeneous function φ ( x ) = aρα ( x ) − bρα ( −x ) , √ c∗ = Γ ( ( B − 1 ) /2 ) Γ ( ( B − 2 ) /2 ) π −1/2 ( a ∫ π π/2 dθ ( − √ B − 1 cos θ ) α sinB−3 θ − b ∫ π/2 0 dθ ( √ B − 1 cos θ ) α sinB−3 θ ) = ( a− b ) Γ ( ( B − 1 ) /2 ) Γ ( ( B − 2 ) /2 ) π −1/2 ( B − 1 ) α/2 1 2 Beta ( α+ 1 2 , B − 2 2 ) = ( a− b ) 1 2 √ π ( B − 1 ) α/2 Γ ( ( B − 1 ) /2 ) Γ ( ( α+ 1 ) /2 ) Γ ( ( α+B − 1 ) /2 ) Expanding the beta function and combining with Thm H.3 gives the desired result . H.1.2 LAPLACE METHOD While we don ’ t need to use the Laplace method to compute c∗ for positive homogeneous functions ( since it is already given by Corollary H.4 ) , going through the computation is instructive for the machinery for computing the eigenvalues in a later section . Lemma H.5 ( The Laplace Method Cross Batch Master Equation ) . For A , B , C ∈ N , let f : RA+B → RC and let a , b ≥ 0 . Suppose for any y ∈ RA , z ∈ RB , ‖f ( y , z ) ‖ ≤ h ( √ ‖y‖2 + ‖z‖2 ) for some nondecreasing function h : R≥0 → R≥0 such that E [ h ( r‖z‖ ) : z ∈ N ( 0 , IA+B ) ] exists for every r ≥ 0 . Define ℘ ( Σ ) : = E [ ‖y‖−2a‖z‖−2bf ( y , z ) : ( y , z ) ∼ N ( 0 , Σ ) ] . Then on { Σ ∈ SA+B : rank Σ > 2 ( a+ b ) } , ℘ ( Σ ) is well-defined and continuous , and furthermore satisfies ℘ ( Σ ) = Γ ( a ) −1Γ ( b ) −1 ∫ ∞ 0 ds ∫ ∞ 0 dt sa−1tb−1 det ( IA+B + 2Ω ) −1/2 E ( y , z ) ∼N ( 0 , Π ) f ( y , z ) ( 55 ) where D = ( √ sIA 0 0 √ tIB ) , Ω = DΣD , and Π = D−1Ω ( I + 2Ω ) −1D−1 . Proof . If Σ is full rank , we can show Fubini-Tonelli theorem is valid in the following computation by the same arguments of the proof of Lemma E.2 . E [ ‖y‖−2a‖z‖−2bf ( y , z ) : ( y , z ) ∼ N ( 0 , Σ ) ] = E [ f ( y , z ) ∫ ∞ 0 ds Γ ( a ) −1sa−1e−‖y‖ 2s ∫ ∞ 0 dt Γ ( b ) −1tb−1e−‖y‖ 2t : ( y , z ) ∼ N ( 0 , Σ ) ] = ( 2π ) − A+B 2 Γ ( a ) −1Γ ( b ) −1 det Σ−1/2 × ∫ ∞ 0 ds ∫ ∞ 0 dt sa−1tb−1 ∫ RA+B dy dz f ( y , z ) e− 1 2 ( y , z ) ( Σ −1+2D2 ) ( y , z ) T ( Fubini-Tonelli ) = Γ ( a ) −1Γ ( b ) −1 ∫ ∞ 0 ds ∫ ∞ 0 dt sa−1tb−1 det ( Σ ( Σ−1 + 2D2 ) ) −1/2 E f ( y , z ) where in the last line , ( y , z ) ∼ N ( 0 , ( Σ−1 + 2D2 ) −1 ) . We recover the equation in question with the following simplifications . ( Σ−1 + 2D2 ) −1 = Σ ( IA+B + 2D 2Σ ) −1 = Σ ( D−1 + 2DΣ ) −1D−1 = ΣD ( IA+B + 2DΣD ) −1D−1 = D−1Ω ( IA+B + 2Ω ) −1D−1 det ( Σ ( Σ−1 + 2D2 ) ) = det ( IA+B + 2ΣD 2 ) = det ( IA+B + 2DΣD ) = det ( IA+B + 2Ω ) The case of general Σ with rank Σ > 2 ( a + b ) is given by the same continuity arguments as in Lemma E.2 . Let Σ̃ ∈ S2B , Σ̃ = ( Σ Ξ ΞT Σ′ ) where Σ , Σ′ ∈ SB and Ξ ∈ RB×B . Consider the off-diagonal block of VB⊕2φ ( Σ̃ ) . E [ Bφ ( z ) ⊗ Bφ ( z′ ) : ( z , z′ ) ∼ N ( 0 , Σ̃ ) ] = E [ φ ( n ( y ) ) ⊗ φ ( n ( y′ ) ) : ( y , y′ ) ∼ N ( 0 , Σ̃G ) ] = Bα E [ ‖y‖−α‖y′‖−αφ ( y ) ⊗ φ ( y′ ) : ( y , y′ ) ∼ N ( 0 , Σ̃G ) ] = BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 det ( I2B + 2Ω ) −1/2 E ( y , y′ ) ∼N ( 0 , Π ) φ ( y ) ⊗ φ ( y′ ) ( by Lemma H.5 ) = BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 det ( I2B + 2Ω ) −1/2Vφ ( Π ) 1 : B , B+1:2B ( 56 ) where Ω = ( sΣG √ stΞG√ st ( ΞG ) T tΣ′G ) and Π = D−1Ω ( I + 2Ω ) −1D−1 with D = √ sIB ⊕ √ tIB = ( √ sIB 0 0 √ tIB ) . Theorem H.6 ( Rephrasing and adding to Corollary H.4 ) . Let Σ̃ ∈ SkB and φ be positive homogeneous of degree α . If Σ̃ is CBSB1 then VB⊕kφ ( Σ̃ ) is CBSB1 and equals Σ̃ ∗ = ( Σ∗ c∗11T c∗11T Σ∗ ) where Σ∗ is the BSB1 fixed point of Thm F.5 and c∗ = cα ( B − 1 2 ) α P ( B − 1 2 , α 2 ) −2 Jφ ( 0 ) = ( a− b ) 2 1 4π ( B − 1 ) α ( Γ ( ( B − 1 ) /2 ) Γ ( ( α+ 1 ) /2 ) Γ ( ( α+B − 1 ) /2 ) ) 2 Proof . Like in the proof of Corollary H.4 , we only need to compute the cross-batch block , which is given above by Eq . ( 56 ) . Recall that Σ̃ = ( Σ c11T c11T Σ ) where Σ = BSB1 ( a , b ) . Set µ : = a− b . Note that because the off-diagonal block Ξ = c11T by assumption , ΞG = 0 , so that Ω = sΣG ⊕ tΣG = sµG⊕ tµG Ω ( I + 2Ω ) −1 = sµ 1 + 2sµ G⊕ tµ 1 + 2tµ G Π = µ 1 + 2sµ G⊕ µ 1 + 2tµ G Therefore , det ( I + 2Ω ) = ( 1 + 2sµ ) B−1 ( 1 + 2tµ ) B−1 Vφ ( Π ) = µ α ( B − 1 B ) α ∆Vφ ( BSB1 ( 1 , −1B−1 ) 0 0 BSB1 ( 1 , −1B−1 ) ) ∆ where ∆ = ( ( 1 + 2sµ ) −α/2IB 0 0 ( 1 + 2tµ ) −α/2IB ) . In particular , the off-diagonal block is con- stant with entry cαµα ( B−1 B ) α ( 1 + 2sµ ) −α/2 ( 1 + 2tµ ) −α/2Jφ ( 0 ) . Finally , by Eq . ( 56 ) , E [ Bφ ( z ) ⊗ Bφ ( z′ ) : ( z , z′ ) ∼ N ( 0 , Σ̃ ) ] = BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 det ( I2B + 2Ω ) −1/2Vφ ( Π ) 1 : B , B+1:2B = BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 [ ( 1 + 2sµ ) ( 1 + 2tµ ) ] − B−1 2 × cαµα ( B − 1 B ) α [ ( 1 + 2sµ ) ( 1 + 2tµ ) ] −α/2Jφ ( 0 ) 11 T = cα ( B − 1 ) αΓ ( α/2 ) −2Jφ ( 0 ) 11T ∫ ∞ 0 dσ ∫ ∞ 0 dτ ( στ ) α/2−1 ( 1 + 2σ ) − B−1+α 2 ( 1 + 2τ ) − B−1+α 2 ( with σ = µs , τ = µt ) = cα ( B − 1 ) αΓ ( α/2 ) −2Jφ ( 0 ) 11T ( 2−α/2 Beta ( B − 1 2 , α 2 ) ) 2 = cα ( B − 1 2 ) α P ( B − 1 2 , α 2 ) −2 Jφ ( 0 ) 11 T Simplification with Defn E.7 and Proposition E.21 gives the result . H.2 LOCAL CONVERGENCE We now look at the linearized dynamics around CBSB1 fixed points of Eq . ( 54 ) , and investigate the eigen-properties of dV B⊕2 φ dΣ̃ ∣∣∣∣ Σ̃=Σ̃∗ . By Lemma F.17 , its nonzero eigenvalues are exactly those of F̃ : = ( G⊕2 ) ⊗2 ◦ dV ( φ◦n ) ⊕2 ( Σ̃ G ) dΣ̃G ∣∣∣∣ Σ̃G=µ∗G⊕2 , F̃ { Λ } = G⊕2 ( dV ( φ◦n ) ⊕2 dΣ̃G ∣∣∣ µ∗G⊕2 { Λ } ) G⊕2 . Here Σ̃G = G⊕2Σ̃G⊕2 . Thus it suffices to obtain the eigendecomposition of F̃ . First , notice that F̃ acts on the diagonal blocks as G⊗2 ◦ dVφ◦n dΣG , and the off-diagonal components of Λ has no effect on the diagonal blocks of F̃ { Λ } . We formalize this notion as follows Definition H.7 . Let T : HkB → HkB be a linear operator such that for any block matrix Σ̃ ∈ HkB with k × k blocks Σ̃ij , i , j ∈ [ k ] , of size B , we have T { Σ̃ } ii = U { Σ̃ii } T { Σ̃ } ij = V { Σ̃ii } + V { Σ̃jj } T +W { Σ̃ij } , ∀i 6= j , where U : HB → HB , V : HB → RB×B , and W : RB×B → RB×B are linear operators , and T { Σ̃ } ij denotes the ( i , j ) th block of T { Σ̃ } . We say that T is blockwise diagonal-off-diagonal semidirect , or BDOS for short . We write more specifically T = BDOS ( U , V , W ) . We have F̃ = BDOS ( U , V , W ) where U = G⊗2 ◦ dVφ◦n dΣG ∣∣∣∣ ΣG=µ∗G V = G⊗2 ◦ d d ( Σ̃G ) 11 E ( h , h′ ) ∼N ( 0 , Σ̃G ) φ ( n ( h ) ) ⊗ φ ( n ( h′ ) ) ∣∣∣∣∣ Σ̃G=µ∗G⊕2 W = 2G⊗2 ◦ d d ( Σ̃G ) 12 E ( h , h′ ) ∼N ( 0 , Σ̃G ) φ ( n ( h ) ) ⊗ φ ( n ( h′ ) ) ∣∣∣∣∣ Σ̃G=µ∗G⊕2 . ( 57 ) Note that the factor 2 in W is due to the contribution of ( Σ̃G ) 21 by symmetry . We will first show thatW is multiplication by a constant . H.2.1 SPHERICAL INTEGRATION AND GEGENBAUER EXPANSION Theorem H.8 . Suppose φ ( √ B − 1x ) has Gegenbauer expansion ∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) . Then for any Λ ∈ RB×B satisfying GΛG = Λ , W { Λ } = 2a21µ∗−1P ( B − 1 2 , 1 2 ) 2 B B − 1Λ = a21 1 2P ( B 2 , 1 2 ) −2 B ( B − 1 ) ∑∞ l=0 a 2 l 1 cB−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) Λ Proof . Let J = d d ( Σ̃G ) 12 E ( h , h′ ) ∼N ( 0 , Σ̃G ) φ ( n ( h ) ) ⊗ φ ( n ( h′ ) ) ∣∣∣ Σ̃G=µ∗G⊕2 . Combining Lemma F.18 with projection by e , we get J { Λ } = 1 2 E y , y′∼N ( 0 , µ∗IB−1 ) φ ( n ( ey ) ) ⊗ φ ( n ( ey′ ) ) 〈µ∗−2yy′T − µ∗−1IB−1 , eTΛe〉 = 1 2 µ∗−1 E y , y′∼N ( 0 , IB−1 ) φ ( n ( ey ) ) ⊗ φ ( n ( ey′ ) ) ( y′TeTΛey − tr Λ ) = 1 2 µ∗−1 ( ( √ 2P ( B − 1 2 , 1 2 ) ) 2 E v , v′∼SB−2 φ ( √ Bev ) ⊗ φ ( √ Bev′ ) ( v′TeTΛev ) − ( tr Λ ) E v , v′∼SB−2 φ ( √ Bev ) ⊗ φ ( √ Bev′ ) ) by Proposition E.33 . Note that Ev , v′∼SB−2 φ ( √ Bev ) ⊗φ ( √ Bev′ ) = ( Ev∼SB−2 φ ( √ Bev ) ) ⊗2 is a constant matrix , because of the independence of v from v′ and the spherical symmetry of the integrand . Thus this term drops out in G⊗2 ◦ J { Λ } . Next , we analyze H { Λ } = Ev , v′∼SB−2 φ ( √ Bev ) ⊗ φ ( √ Bev′ ) ( v′TeTΛev ) . This is a linear function in Λ with coefficients Hab|cd = B − 1 B ( E v∼SB−2 φ ( √ B − 1êb , :v ) ( êc , :v ) ) ( E v∼SB−2 φ ( √ B − 1êa , :v ) ( êd , :v ) ) . By the usual zonal spherical harmonics argument , E v∼SB−2 φ ( √ B − 1êb , :v ) ( êc , :v ) = 〈 ∞∑ l=0 al 1 cB−1 , l C ( B−32 ) l ( 〈êb , : , ·〉 ) , 〈êc , : , ·〉 〉 SB−2 = 〈 ∞∑ l=0 alZ B−2 , ( l ) êb , : , 1 B − 1Z B−2 , ( 1 ) êc , : 〉 SB−2 = a1 B − 1Z B−2 , ( 1 ) êc , : ( êb , : ) = a1 B − 1 1 cB−1,1 C ( B−32 ) 1 ( 〈êb , : , êc , :〉 ) = { a1 if b = c −a1 B−1 otherwise . Thus , H { Λ } ab = B − 1 B ( ( −a1 B − 1 ) 2 ∑ c , d∈ [ B ] Λcd + ( a1 − −a1 B − 1 ) ( −a1 B − 1 ) ∑ c=b XOR d=a Λcd + ( a21 − a21 ( B − 1 ) 2 ) Λab ) = B − 1 B ( −2 ( a1 − −a1 B − 1 ) ( −a1 B − 1 ) Λab + ( a21 − a21 ( B − 1 ) 2 ) Λab ) because rows and columns of Λ sum to 0 = a21 B B − 1Λab . Therefore , W { Λ } = 2G⊗2 ◦ J { Λ } = 2a21µ ∗−1P ( B − 1 2 , 1 2 ) 2 B B − 1Λ = a21 1 2P ( B 2 , 1 2 ) −2 B ( B − 1 ) ∑∞ l=0 a 2 l 1 cB−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) Λ where we used the identity P ( B−1 2 , 1 2 ) ( 2 B−1 ) = P ( B 2 , 1 2 ) −1 . Next , we step back a bit to give a full treatment of the eigen-properties of BDOS operators . Then we specialize back to our case and give the result for F̃ . Definition H.9 . Let HGkB : = { Σ̃ ∈ HkB : G⊕kΣ̃G⊕k = Σ̃ } be the subspace of HkB stable under conjugation by G⊕k . Definition H.10 . Let M̃kB ∈ HGkB be the space of block matrices Σ̃ ∈ HGkB where the diagonal blocks Σ̃ii are all zero . Likewise , let M̃kB ∈ HkB be the space of block matrices Σ̃ ∈ HkB where the diagonal blocks Σ̃ii are all zero . We suppress the subscript kB when it is clear from the context . Definition H.11 . Define the block L-shaped matrix LkB ( Σ , Λ ) : = Σ −Λ · · ·−ΛT 0 · · · ... ... . . . BDOS operators of the form BDOS ( U , V , w ) , where w stands for multiplication by a constant w , like F̃ , have simple eigendecomposition , just like DOS operators . Theorem H.12 . If V −wIB is not singular , then BDOS ( V , U , w ) : HkB → HkB has the following eigenspaces and eigenvectors 1 . M̃ with associated eigenvalue w. 2 . For each eigenvalue λ and eigenvector Σ ∈ HB of U , the block matrix LkB ( Σ , − ( λ− w ) −1V { Σ } ) = Σ ( λ− w ) −1V { Σ } · · · ( λ− w ) −1V { Σ } T 0 · · · ... ... . . . along with any matrix obtained from permutating its block columns and rows simultaneously . The associated eigenvalue is λ . If BDOS ( V , U , w ) also restricts to HGkB → HGkB , then with respect to this signature , the eigendecomposition is obtained by replacingHkB withHGkB in the above : 1 . Block diagonal matrices Σ̃ ∈ HGkB where the diagonal blocks Σ̃ii are all zero . The associated eigenvalue is w. 2 . For each eigenvalue λ and eigenvector Σ ∈ HGB of U , the matrix LkB ( Σ , − ( λ− w ) −1V { Σ } ) along with any matrix obtained from permutating its block columns and rows simultaneously . The associated eigenvalue is λ . Proof . The proofs are similar in both cases : One can easily verify that these are eigenvectors and eigenvalues . Then by dimensionality considerations , these must be all of them . By Thms H.8 and H.12 , we easily obtain Theorem H.13 . Suppose φ ( √ B − 1x ) has Gegenbauer expansion ∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) . The eigendecomposition of F̃ : HG2B → HG2B is given below , as long as λ↑M̃ 6= λ ↑ L , λ ↑ M. 1 . M̃ with associated eigenvalue λ↑ M̃ : = 2a21µ ∗−1P ( B − 1 2 , 1 2 ) 2 B B − 1 = a21 1 2P ( B 2 , 1 2 ) −2 B ( B − 1 ) ∑∞ l=0 a 2 l 1 cB−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) . 2 . The space generated by L2B ( G , − ( λ↑L − λ ↑ M̃ ) −1V { G } ) along with any matrix obtained from permutating its block columns and rows simultaneously . The associated eigenvalue is 0 . 3 . The space generated by L2B ( L , − ( λ↑L − λ ↑ M̃ ) −1V { L } ) , where L = LB ( B − 2 , 1 ) , along with any matrix obtained from permutating its block columns and rows simultaneously . The associated eigenvalue is λ↑L . 4 . The space generated by L2B ( M , − ( λ↑M − λ ↑ M̃ ) −1V { M } ) , for any M ∈ M , along with any matrix obtained from permutating its block columns and rows simultaneously . The associated eigenvalue is λ↑M . Here , V is as in Eq . ( 57 ) . The local convergence dynamics of Eq . ( 54 ) to the CBSB1 fixed point Σ̃∗ has the same eigenvalues λ↑ M̃ , λ↑L , λ ↑ M , and 0 . Note that , to reason about F̃ extended over k batches , one can replace 2B with kB everywhere in Thm H.13 and the theorem will still hold . As we will see below , λ↑ M̃ is the same as the eigenvalue governing the dynamics of the cross batch correlations in the gradient propagation . H.2.2 LAPLACE METHOD As with the single batch case , with positive homogeneous φ , we can obtain a more direct answer of the new eigenvalue λ↑ M̃ in terms of the J function of φ. Theorem H.14 . Let φ be degree α positive homogeneous . Then the eigenvalue of F̃ on M̃ is λ↑ M̃ = B 2 ( B − 1 2 ) α−1 P ( B 2 , α 2 ) −2 µ∗−1cαJ ′ φ ( 0 ) = B B − 1 P ( B−1 2 , α ) P ( B 2 , α 2 ) 2 J′φ ( 0 ) Jφ ( 1 ) − Jφ ( −1 B−1 ) Proof . Let Σ̃∗ = ( Σ∗ c∗11T c∗11T Σ∗ ) be the CBSB1 fixed point in Thm H.6 . Take a smooth path Σ̃τ = ( Στ Ξτ ΞTτ Σ ′ τ ) ∈ S2B , τ ∈ ( − , ) such that Σ̃0 = Σ̃∗ , ˙̃Σ0 : = ddτ Σ̃ = ( 0 Υ ΥT 0 ) for some Υ ∈ RB×B . Then with Ωτ = ( sΣGτ √ stΞGτ√ st ( ΞGτ ) T tΣ′τ G ) and Πτ = D−1Ωτ ( I + 2Ωτ ) −1D−1 with D = √ sIB ⊕ √ tIB = ( √ sIB 0 0 √ tIB ) , d dτ E [ Bφ ( z ) ⊗ Bφ ( z′ ) : ( z , z′ ) ∼ N ( 0 , Σ̃ ) ] ∣∣∣∣ τ=0 = d dτ BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 det ( I2B + 2Ω ) −1/2Vφ ( Π ) 1 : B , B+1:2B ∣∣∣∣ τ=0 = BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 ( d dτ det ( I2B + 2Ω ) −1/2 ∣∣∣∣ τ=0 Vφ ( Π ) 1 : B , B+1:2B + det ( I2B + 2Ω ) −1/2 d dτ Vφ ( Π ) 1 : B , B+1:2B ∣∣∣∣ τ=0 ) = BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 ( − det ( I + 2Ω0 ) −1/2 tr ( ( I + 2Ω0 ) −1Ω̇0 ) Vφ ( Π ) + det ( I + 2Ω0 ) −1/2 ( D−α ) ⊗2 ◦ J { d dτ Ω ( I + 2Ω ) −1 ∣∣∣∣ τ=0 } ) 1 : B , B+1:2B ( 58 ) by chain rule and Lemma F.26 , where J = dVφdΣ ∣∣∣∣ Σ=Ω0 ( I+2Ω0 ) −1 . We compute Ω0 = µ ∗ ( sG⊕ tG ) Ω̇0 = √ st ( 0 ΥG ( ΥG ) T 0 ) ( I + 2Ω0 ) −1 = ( ( 1 + 2sµ∗ ) −1G+ 1 B 11T ) ⊕ ( ( 1 + 2tµ∗ ) −1G+ 1 B 11T ) det ( I + 2Ω0 ) = ( 1 + 2sµ ∗ ) B−1 ( 1 + 2tµ∗ ) B−1 Ω0 ( I + 2Ω0 ) −1 = sµ∗ 1 + 2sµ∗ G⊕ tµ ∗ 1 + 2tµ∗ G. So then J = ( ( sµ∗ 1 + 2sµ∗ ) ( α−1 ) /2 IB ⊕ ( tµ∗ 1 + 2tµ∗ ) ( α−1 ) /2 IB ) ⊗2 ◦ ( B − 1 B ) α−1 dVφ dΣ ∣∣∣∣ Σ=BSB1 ( 1 , −1B−1 ) ⊕2 With d dτ Ω ( I + 2Ω ) −1 ∣∣∣∣ τ=0 = ( I + 2Ω0 ) −1Ω̇0 ( I + 2Ω0 ) −1 ( by Lemma F.27 ) = √ st ( 1 + 2sµ∗ ) ( 1 + 2tµ∗ ) ( 0 ΥG ( ΥG ) T 0 ) we have J { d dτ Ω ( I + 2Ω ) −1 ∣∣∣∣ τ=0 } = ( B − 1 B ) α−1 √ st ( 1 + 2sµ∗ ) ( 1 + 2tµ∗ ) cαJ ′ φ ( 0 ) × ( sµ∗ 1 + 2sµ∗ ) ( α−1 ) /2 ( tµ∗ 1 + 2tµ∗ ) ( α−1 ) /2 ( 0 ΥG ( ΥG ) T 0 ) = ( B − 1 B ) α−1 ( st ) α/2µ∗α−1cαJ′φ ( 0 ) ( 1 + 2sµ∗ ) ( α+1 ) /2 ( 1 + 2tµ∗ ) ( α+1 ) /2 ( 0 ΥG ( ΥG ) T 0 ) . Thus the product ( I + 2Ω0 ) −1Ω̇0 has zero diagonal blocks so that its trace is 0 . Therefore , only the second term in the sum in Eq . ( 58 ) above survives , and we have d dτ E [ Bφ ( z ) ⊗ Bφ ( z′ ) : ( z , z′ ) ∼ N ( 0 , Σ̃ ) ] ∣∣∣∣ τ=0 = BαΓ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 ( 1 + 2sµ∗ ) − B−1 2 ( 1 + 2tµ∗ ) − B−1 2 × ( st ) −α/2 ( B − 1 B ) α−1 ( st ) α/2µ∗α−1cαJ′φ ( 0 ) ( 1 + 2sµ∗ ) ( α+1 ) /2 ( 1 + 2tµ∗ ) ( α+1 ) /2 ΥG = B ( B − 1 ) α−1Γ ( α/2 ) −2 ∫ ∞ 0 ds ∫ ∞ 0 dt ( st ) α/2−1 µ∗α−1cαJ ′ φ ( 0 ) ( 1 + 2sµ∗ ) ( B+α ) /2 ( 1 + 2tµ∗ ) ( B+α ) /2 ΥG = B ( B − 1 ) α−1Γ ( α/2 ) −2µ∗−1cαJ′φ ( 0 ) ( ∫ ∞ 0 d ( µ∗s ) ( µ∗s ) α/2−1 ( 1 + 2sµ∗ ) ( B+α ) /2 ) 2 ΥG = B ( B − 1 ) α−1Γ ( α/2 ) −2µ∗−1cαJ′φ ( 0 ) ( 2−α/2 Beta ( B 2 , α 2 ) ) 2 ΥG = B 2 ( B − 1 2 ) α−1 P ( B 2 , α 2 ) −2 µ∗−1cαJ ′ φ ( 0 ) Υ G I CROSS BATCH : BACKWARD DYNAMICS For k ≥ 2 , we wish to study the following generalization of Eq . ( 51 ) , Π̃l = VB⊕kφ ′ ( Σ̃l ) † { Π̃l+1 } . As in the single batch case , we approximate it by taking Σ̃l to its CBSB1 limit Σ̃∗ , so that we analyze Π̃l = VB⊕kφ ′ ( Σ̃∗ ) † { Π̃l+1 } ( 59 ) Its dynamics is then given by the eigendecomposition of VB⊕kφ ′ ( Σ̃∗ ) , in parallel to the single batch scenario . WLOG , we only need to consider the case k = 2 . VB⊕2φ ′ ( Σ̃∗ ) = E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ ( B′φ ( x ) ⊕ B′φ ( y ) ) ⊗2 ] VB⊕2φ ′ ( Σ̃∗ ) { ( Σ Ξ ΞT Λ ) } = E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ ( B′φ ( x ) ΣB′φ ( x ) T B′φ ( x ) ΞB′φ ( y ) T B′φ ( y ) ΞTB′φ ( x ) T B′φ ( y ) ΛB′φ ( y ) T ) ] From this one sees that V [ ( B⊕2φ ) ′ ] acts independently on each block , and consequently so does its adjoint . The diagonal blocks of Eq . ( 59 ) evolves according to Eq . ( 51 ) which we studied in Appendix G. In this section we will study the evolution of the off-diagonal blocks , which is given by Ξl = E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ B′φ ( x ) ⊗ B′φ ( y ) ] † { Ξl+1 } = E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ B′φ ( y ) T ⊗ B′φ ( x ) T ] { Ξl+1 } ( 60 ) = E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ B′φ ( y ) TΞl+1B′φ ( x ) T ] Definition I.1 . Let MB : = RB×B be the set of all B × B real matrices , and let MGB : = { Ξ ∈ MB : GΞG = Ξ } . Define T : = E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ B′φ ( x ) ⊗B′φ ( y ) ] . As in the single batch case , after one step of backprop , ΞL−1 = V† { ΞL } is inMGB . Thus it suffices to study the eigendecomposition of G⊗2 ◦ T † MGB : MGB →MGB . Below , we will show that its adjoint T ◦G⊗2 acting onMGB is in fact multiplication by a constant , and thus so is it . We first compute T = E ( ξ , η ) ∼N ( 0 , G⊕2Σ̃∗G⊕2 ) [ dφ ◦ n ( z ) dz ∣∣∣∣ z=ξ ⊗ dφ ◦ n ( z ) dz ∣∣∣∣ z=η ] ◦G⊗2 = E ( ξ , η ) ∼N ( 0 , µ∗ ( G 00 G ) ) [ dφ ◦ n ( z ) dz ∣∣∣∣ z=ξ ⊗ dφ ◦ n ( z ) dz ∣∣∣∣ z=η ] ◦G⊗2 since Σ̃∗ is CBSB1 with diagonal blocks Σ∗ ( Corollary H.4 ) . Then ξ is independent from η , so this is just E ( x , y ) ∼N ( 0 , Σ̃∗ ) [ B′φ ( x ) ⊗ B′φ ( y ) ] = E ξ∼N ( 0 , µ∗G ) [ dφ ◦ n ( z ) dz ∣∣∣∣ z=ξ ] ⊗2 ◦G⊗2 = E ξ∼N ( 0 , µ∗G ) [ √ BDr−1 ( I − vvT ) ] ⊗2 ◦G⊗2 ( by Proposition G.4 ) where D = Diag ( φ′ ( √ Bv ) ) , r = ‖ξ‖ , v = ξ/‖ξ‖ . But notice that T : = Eξ∼N ( 0 , µ∗G ) [ Dr−1 ( I − vvT ) ] is actually BSB1 , with diagonal E rv∼N ( 0 , µ∗G ) r−1 ( 1− v2i ) φ′ ( √ Bvi ) = E rv∼N ( 0 , µ∗G ) r−1 ( 1− v21 ) φ′ ( √ Bv1 ) , ∀i ∈ [ B ] and off-diagonal E rv∼N ( 0 , µ∗G ) r−1 ( −vivj ) φ′ ( √ Bvi ) = E rv∼N ( 0 , µ∗G ) r−1 ( −v1v2 ) φ′ ( √ Bv1 ) , ∀i 6= j ∈ [ B ] . Thus by Lemma E.51 , T has two eigenspaces , { x : Gx = x } and R1 , with respective eigenvalues T11 − T12 and T11 + ( B − 1 ) T12 . However , because of the composition with G⊗2 , only the former eigenspace survives with nonzero eigenvalue in G⊗2 ◦ T . Theorem I.2 . T on MGB is just multiplication by λ↓M̃ : = 1 2πBµ ∗−1 ( ∫ π 0 dθ sinB−1 θφ′ ( − √ B − 1 cos θ ) ) 2 . Proof . By the above reasoning , the sole eigenspace with nonzero eigenvalue is { x : Gx = x } ⊗2 = MGB . It has eigenvalue B ( E [ r−1 ( 1− v21 + v1v2 ) φ′ ( √ Bv1 ) : rv ∼ N ( 0 , µ∗G ) ] ) 2 = Bµ∗−1 ( E [ r−1 ( 1− v21 + v1v2 ) φ′ ( √ Bv1 ) : rv ∼ N ( 0 , G ) ] ) 2 = Bµ∗−1 ( E [ r−1 ( 1− ( ew ) 21 + ( ew ) 1 ( ew ) 2 ) φ′ ( √ B ( ew ) 1 ) : rw ∼ N ( 0 , IB−1 ) ] ) 2 = Bµ∗−1 ( Γ ( ( B − 2 ) /2 ) Γ ( ( B − 3 ) /2 ) 2 −1/2π−1∫ π 0 dθ1 ∫ π 0 dθ2 ( 1− ζ ( θ1 ) 2 + ζ ( θ1 ) ω ( θ1 , θ2 ) ) φ′ ( √ Bζ ( θ1 ) ) sin B−3 θ1 sin B−4 θ2 ) 2 where we applied Lemma E.36 with ζ ( θ ) = − √ B−1 B cos θ and ω ( θ1 , θ2 ) = 1√ B ( B−1 ) cos θ1 −√ B−2 B−1 sin θ1 cos θ2 . We can further simplify∫ π 0 dθ1 ∫ π 0 dθ2 ( 1− ζ ( θ1 ) 2 + ζ ( θ1 ) ω ( θ1 , θ2 ) ) φ′ ( √ Bζ ( θ1 ) ) sin B−3 θ1 sin B−4 θ2 = ∫ π 0 dθ1 sin B−3 θ1φ ′ ( √ Bζ ( θ1 ) ) ( ( 1− ζ ( θ1 ) 2 + ζ ( θ1 ) 1√ B ( B − 1 ) cos θ1 ) ∫ π 0 dθ2 sin B−4 θ2 − √ B − 2 B − 1 sin θ1ζ ( θ1 ) ∫ π 0 dθ2 sin B−4 θ2 cos θ2 ) = ∫ π 0 dθ1 sin B−3 θ1φ ′ ( √ Bζ ( θ1 ) ) ( ( 1− ζ ( θ1 ) 2 + ζ ( θ1 ) 1√ B ( B − 1 ) cos θ1 ) Beta ( B − 3 2 , 1 2 ) ) ( by Lemma E.29 ) = ∫ π 0 dθ1 sin B−3 θ1φ ′ ( √ Bζ ( θ1 ) ) ( sin2 θ1 Beta ( B − 3 2 , 1 2 ) ) = Beta ( B − 3 2 , 1 2 ) ∫ π 0 dθ1 sin B−1 θ1φ ′ ( − √ B − 1 cos θ1 ) so the cross-batch backward off-diagonal eigenvalue is Bµ∗−1 ( 1√ 2π ∫ π 0 dθ1 sin B−1 θ1φ ′ ( − √ B − 1 cos θ1 ) ) 2 = 1 2π Bµ∗−1 ( ∫ π 0 dθ1 sin B−1 θ1φ ′ ( − √ B − 1 cos θ1 ) ) 2 For positive homogeneous functions we can evaluate the eigenvalues explicitly . Theorem I.3 . If φ is positive-homogeneous of degree α with φ ( c ) = aρα ( c ) − bρα ( −c ) , then T : MGB →MGB is just multiplication by λ↓ M̃ = 1 8π B ( B − 1 ) α−1µ∗−1α2 ( a+ b ) 2 Beta ( α 2 , B 2 ) 2 . Proof . As in the proof of Corollary H.4 , ∫ π/2 0 dθ ( cos θ ) α−1 sinB−1 θ = ∫ π π/2 dθ ( − cos θ ) α−1 sinB−1 θ = 1 2 Beta ( α 2 , B 2 ) So 1 2π Bµ∗−1 ( ∫ π 0 dθ1 sin B−1 θ1φ ′ ( − √ B − 1 cos θ1 ) ) 2 = 1 2π Bµ∗−1 ( B − 1 ) α−1α2 ( a ∫ π π/2 dθ ( − cos θ ) α−1 sinB−1 θ + b ∫ π/2 0 dθ ( cos θ ) α−1 sinB−1 θ ) 2 = 1 2π Bµ∗−1 ( B − 1 ) α−1α2 ( a+ b ) 22−2 Beta ( α 2 , B 2 ) 2 = 1 8π B ( B − 1 ) α−1µ∗−1α2 ( a+ b ) 2 Beta ( α 2 , B 2 ) 2 It is also straightforward to obtain an expression of the eigenvalue in terms of Gegenbauer coefficients . Theorem I.4 . If φ ( √ B − 1x ) has Gegenbauer expansion ∑∞l=0 al 1cB−1 , lC ( B−32 ) l ( x ) , then T : MGB →MGB is just multiplication by λ↓ M̃ = 1 2π B ( B − 1 ) µ∗−1a21 Beta ( B 2 , 1 2 ) 2 = 1 2πB ( B − 1 ) a21 Beta ( B 2 , 1 2 ) 2∑∞ l=0 a 2 l 1 cB−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) = λ↑ M̃ . Proof . It suffices to express the integral in Thm I.2 in Gegenbauer coefficients . Changing coordinates x = − cos θ , so that dx = sin θ dθ , we get∫ π 0 dθ sinB−1 θφ′ ( − √ B − 1 cos θ ) = ∫ 1 −1 dx ( 1− x2 ) B−22 φ′ ( √ B − 1x ) . By Eq . ( 35 ) , φ′ ( √ B − 1x ) = ( B − 1 ) −1/2 d dx φ ( √ B − 1x ) = ( B − 1 ) −1/2 ∞∑ l=0 al 1 cB−1 , l C ( B−32 ) l ′ ( x ) = ( B − 1 ) −1/2 ∞∑ l=1 al B − 3 cB−1 , l C ( B−12 ) l−1 ( x ) . Then by the orthogonality relations among { C ( B−1 2 ) l−1 } l≥1 , ∫ π 0 dθ sinB−1 θφ′ ( − √ B − 1 cos θ ) = ∫ 1 −1 dx ( 1− x2 ) B−22 φ′ ( √ B − 1x ) = ( B − 1 ) −1/2a1 B − 3 cB−1,1 ∫ 1 −1 [ C ( B−12 ) 0 ( x ) ] 2 ( 1− x2 ) B−22 dx = ( B − 1 ) −1/2a1 B − 3 cB−1,1 √ πΓ ( B 2 ) Γ ( B+1 2 ) = a1 √ B − 1 Beta ( B 2 , 1 2 ) . To summarize , Theorem I.5 . The backward cross batch dynamics Eq . ( 59 ) over HGkB is separable over each individual B × B block . The diagonal blocks evolve as in Eq . ( 51 ) , and this linear dynamics has eigenvalues λ↓G , λ ↓ L , λ ↓ M as described by Appendix G. The off-diagonal blocks dynamics is multiplication by λ↓ M̃ = λ↑ M̃ . Note that λ↓ M̃ = λ↑ M̃ = 1 2πB ( B − 1 ) a21 Beta ( B 2 , 1 2 ) 2∑∞ l=0 a 2 l 1 cB−1 , l ( C ( B−32 ) l ( 1 ) − C ( B−32 ) l ( −1 B−1 ) ) ≤ 1 2πB ( B − 1 ) Beta ( B 2 , 1 2 ) 2 1 cB−1,1 ( C ( B−32 ) 1 ( 1 ) − C ( B−32 ) 1 ( −1 B−1 ) ) = B − 1 2 P ( B 2 , 1 2 ) −2 which increases to 1 from below as B →∞ . Thus , Corollary I.6 . For any φ inducing Eq . ( 54 ) to converge to a CBSB1 fixed point , λ↓ M̃ = λ↑ M̃ < 1 . For any fixed batch size B , λ↓ M̃ is maximized by φ = id . Furthermore , for φ = id , λ↓ M̃ = B−1 2 P ( B 2 , 1 2 ) −2 and limB→∞ λ ↓ M̃ = 1 , increasing to the limit from below . This corollary indicates that stacking batchnorm in a deep neural network will always cause chaotic behavior , in the sense that cross batch correlation , both forward and backward , decreases to 0 exponentially with depth . The φ that can maximally ameliorate the exponential loss of information is linear . J UNCOMMON REGIMES In the above exposition of the mean field theory of batchnorm , we have assumed B ≥ 4 and that φ induces BSB1 fixed points in Eq . ( 8 ) . Small Batch Size What happens when B < 4 ? It is clear that for B = 1 , batchnorm is not well-defined . For B = 2 , Bφ ( h ) = ( ±1 , ∓1 ) depending on the signs of h. Thus , the gradient of a batchnorm network with B = 2 is 0 . Therefore , we see an abrupt phase transition from the immediate gradient vanishing of B = 2 to the gradient explosion of B ≥ 4 . We empirically see that B = 3 suffers similar gradient explosion as B = 4 and conjecture that a generalization of Thm 3.9 holds for B = 3 . Batch Symmetry Breaking What about other nonlinearities ? Empirically , we observe that if the fixed point is not BSB1 , then it is BSB2 , like in Fig . 1 , where a submatrix of Σ ( the dominant block ) is much larger in magnitude than everything else ( see Defn K.1 ) . If the initial Σ0 is permutationinvariant , then convergence to this fixed point requires spontaneous symmetry breaking , as the dominant block can appear in any part of Σ along the diagonal . This symmetry breaking is lost when we take the mean field limit , but in real networks , the symmetry is broken by the network weight randomness . Because small fluctuation in the input can also direct the dynamics toward one BSB1 fixed point against others , causing large change in the output , the gradient is intuitively large as well . Additionally , at the BSB2 fixed point , we expect the dominant block goes through a similar dynamics as if it were a BSB1 fixed point for a smaller B , thus suffering from similar gradient explosion . Appendix K discusses several results on our current understanding of BSB2 fixed points . A specific form of BSB2 fixed point with a 1 × 1 dominant block can be analyzed much further , and this is done in Appendix K.1 . Finite width effect For certain nonlinearities , the favored fixed point can be different between the large width limit and small width . For example , for φ = C ( B−3 2 ) 5 ( x/ √ B − 1 ) −C ( B−3 2 ) 7 ( x/ √ B − 1 ) where B = 10 , one can observe that for width 100 , Eq . ( 43 ) favors BSB2 fixed points , but for width 1000 and more , Eq . ( 43 ) favors BSB1 fixed points . K TOWARD UNDERSTANDING BSB2 FIXED POINTS Definition K.1 . For B1 , B2 ≥ 2 , a BSB2 matrix is one of the form BSB2B1 , B2 ( d1 , f1 , d2 , f2 , c ) : = ( BSB1B1 ( d1 , f1 ) c c BSB1B2 ( d2 , f2 ) ) up to simultaneous permutation of rows and columns . A B̂SB2 matrix is a specific kind of BSB2 matrix B̂SB2 B′ B ( d , f , c , b ) = BSB2B′ , B−B′ ( d , f , c , b , b ) , where B ′ ∈ [ 2 , B − 1 ] ( where for B′ = B−1 the lower right hand block is a scalar b ) . We call the upper left hand block of B̂SB2 B′ B ( d , f , c , b ) its main block . When φ grows quickly , Eq . ( 43 ) can converge to BSB2 fixed points of the form B̂SB2 B′ B ( d , f , c , b ) , up to simultaneous permutation of rows and columns . In this section we present several results on BSB2 fixed points that sheds light on their structure . We leave the study of the forward Eq . ( 43 ) and backward dynamics Eq . ( 51 ) to future work . First , we give the eigendecomposition of BSB2 matrices . Theorem K.2 . Let Σ = BSB2B1 , B2 ( d1 , f1 , d2 , f2 , c ) , where B1 , B2 ≥ 2 . Then G⊗2 { Σ } = GΣG = ΣG has the following eigenspaces and eigenvalues . 1 . ( B1 − 1 ) -dimensional eigenspace Z1 : = { ( x1 , . . . , xB1 , 0 , . . . , 0 ) : ∑B1 i=1 xi = 0 } , with eigenvalue d1 − f1 . 2 . ( B2 − 1 ) -dimensional eigenspace Z2 : = { ( 0 , . . . , 0 , y1 , . . . , yB2 ) : ∑B2 j=1 yj = 0 } , with eigenvalue d2 − f2 . 3 . 1-dimensional Rq with eigenvalue ( d2−f2 ) B1+ ( d1−f1 ) B2+ ( f1+f2−2c ) B1B2B1+B2 , where q = ( B1︷ ︸︸ ︷ B−11 , . . . , B −1 1 , B2︷ ︸︸ ︷ −B−12 , . . . , −B−12 ) . 4 . 1-dimensional R1 with eigenvalue 0 . Proof . We can verify all eigenspaces and their eigenvalues in a straightforward manner . These then must be all of them by a dimensionality argument . Specializing to B̂SB2 matrices , we get Corollary K.3 . Let Σ = B̂SB2 B′ B ( d , f , c , b ) with B ′ ≥ 2 . Then G⊗2 { Σ } = GΣG = ΣG has the following eigenspaces and eigenvalues • ( B′ − 1 ) -dimensional eigenspace Z1 : = { ( x1 , . . . , xB′ , 0 , . . . , 0 ) : ∑B′ i=1 xi = 0 } , with eigenvalue d− f . • 1-dimensional Rq with eigenvalue ( d−f ) ( B−B ′ ) + ( f−c ) B ( B−B′ ) B , where q = ( B′︷ ︸︸ ︷ B′−1 , . . . , B′−1 , B2︷ ︸︸ ︷ −B−12 , . . . , −B−12 ) where B2 = B −B′ . • ( B−B′ ) -dimensional eigenspace { ( B′︷ ︸︸ ︷ −µ , . . . , −µ , y1 , . . . , yB−B′ : µ = 1B−B′ ∑B−B′ j=1 yj } with eigenvalue 0 . Note also that when B′ = B , then Z1 and Rq become the usual eigenspaces of BSB1 ( d , f ) . For fixed B′ < B , specializing the forward dynamics Eq . ( 43 ) to B̂SB2 B′ B fixed points yields a 2-dimensional dynamics on the eigenvalues for the eigenspaces Z1 and Rq . This dynamics in general is not degenerate , so that the fixed point seems difficult to obtain analytically . Moreoever , the Gegenbauer method , essential for proving that gradient explosion is unavoidable when Eq . ( 43 ) has a BSB1 fixed point , does not immediately generalize to the BSB2 case , since Z1 and Rq in general do not have the same eigenvalues so that we can not reduce the integrals to that on a sphere . For this reason , at present we do not have any rigorous result on the BSB2 case . However , we expect that the main block of the B̂SB2 fixed point should undergo a similar dynamics to that of a BSB1 fixed point , leading to similar gradient explosion . K.1 B̂SB2 1 B FIXED POINTS When φ grows extremely rapidly , for example φ ( x ) = relu ( x ) 30 , and the width of the network is not too large , then we sometimes observe B̂SB2 1 B fixed points B̂SB2 1 B ( d , c , b ) = ( d c1T c1 b11T ) where 1 ∈ RB−1 is the all 1s column vector . In these situations we can in fact see pathological gradient vanishing , in a way reminiscent of the gradient vanishing for B = 2 batchnorm , as we shall see shortly . We can extend Corollary K.3 to the B′ = 1 case . Theorem K.4 . B̂SB2 1 B ( d , c , b ) G is rank 1 and equal to λq̂⊗2 , where λ = B−1B ( d − 2c + b ) and q̂ = ( 1 , −1B−1 , . . . , −1 B−1 ) √ B−1 B . Proof . Straightforward verification . Because B̂SB2 1 B under G-projection is rank 1 , such a fixed point of Eq . ( 43 ) can be determined ( unlike the general B̂SB2 B′ B case ) . Theorem K.5 . There is a unique B̂SB2 1 B fixed point of Eq . ( 43 ) , B̂SB2 1 B ( d ∗ , c∗ , b∗ ) where d∗ = 1 2 ( φ ( √ B − 1 ) 2 + φ ( − √ B − 1 ) 2 ) c∗ = 1 2 ( φ ( √ B − 1 ) φ ( −1√ B − 1 ) + φ ( − √ B − 1 ) φ ( 1√ B − 1 ) ) b∗ = 1 2 ( φ ( −1√ B − 1 ) 2 + φ ( 1√ B − 1 ) 2 ) . Additionally , B̂SB2 1 B ( d ∗ , c∗ , b∗ ) G = λ∗q̂⊗2 , where λ∗ = B − 1 2B [ ( φ ( √ B − 1 ) − φ ( −1√ B − 1 ) ) 2 + ( φ ( − √ B − 1 ) − φ ( 1√ B − 1 ) ) 2 ] Proof . If Σ = B̂SB2 1 B ( d , c , b ) , then VBφ ( Σ ) = E y∼N ( 0 , GΣG ) φ ( n ( y ) ) ⊗2 = 1 2 [ φ ( √ Bq̂ ) ⊗2 + φ ( − √ Bq̂ ) ⊗2 ] since GΣG = λq̂⊗2 for some λ by Thm K.4 . The rest then follows from straightforward computation and another application of Thm K.4 . The rank-1 nature of the B̂SB2 1 B fixed point under G-projection is reminiscent of the case of B = 2 batchnorm . As we will see , gradients of a certain form will vanish similarly . Let δ be a gradient vector . Then by Proposition G.4 , for Σ∗ being the B̂SB2 1 B fixed point , one step of backpropagation yields dBφ dΣ ∣∣∣∣† Σ=Σ∗ δ = √ Br−1G ( I − vvT ) Dδ where D = Diag ( φ′ ( n ( y ) ) ) , r = ‖y‖ , v = y/‖y‖ , and y ∼ N ( 0 , GΣ∗G ) . Because GΣ∗G = λ∗q̂⊗2 by Thm K.5 , n ( y ) = √ Bv = ± √ Bq̂ with equal probability and r ∼ |N ( 0 , λ∗ ) | ( the absolute value of a Gaussian ) . Furthermore , G ( I − q̂⊗2 ) = G− q̂⊗2 because Gq̂ = q̂ . Thus dBφ dΣ ∣∣∣∣† Σ=Σ∗ δ = √ B|r|−1 ( G− q̂⊗2 ) ( φ′ ( √ B sgn ( r ) q̂ ) δ ) , r ∼ N ( 0 , λ∗ ) Now notice the following Lemma K.6 . GB − q̂⊗2 = ( 0 0 0 GB−1 ) = 0 0 0 · · · 0 0 1− 1B−1 −1B−1 · · · −1B−1 0 −1B−1 1− 1B−1 · · · −1B−1 0 −1B−1 −1 B−1 · · · 1− 1B−1 . Proof . Straightforward computation . Thus if δ = ( δ1 , δ2 ) T ∈ R1 × RB−1 , then dBφ dΣ ∣∣∣∣† Σ=Σ∗ δ = √ Bφ′ ( − sgn ( r ) √ B − 1 ) |r|−1 ( 0 , GB−1δ2 ) T , r ∼ N ( 0 , λ∗ ) If δ2 = 0 , then this is deterministically 0 . Otherwise , this random variable does not have finite mean because of the |r|−1 term , and this is solely due to Σ∗ being rank 1 . Thus as long as the gradient δ is not constant in δ2 , then we expect severe gradient explosion ; otherwise we expect severe gradient vanishing . Remark K.7 . We would like to note that , in pytorch , with either Float Tensors or Double Tensors , the gradient empirically vanishes rapidly with generic gradient δ , when φ is rapidly increasing and leads to a B̂SB2 1 B fixed point , e.g . φ ( x ) = relu ( x ) 30 . This is due to numerical issues rather than mathematical issues , where the computation of ( G− q̂⊗2 ) δ in backprop does not kill δ1 completely . L WEIGHT GRADIENTS Recall weight gradient and the backpropagation equation Eq . ( 16 ) of the main text . ∂L ∂W lαβ = ∑ i δlαix l−1 βi , δ l αi = ∑ βj ∂xlαj ∂hlαi W l+1βα δ l+1 βj where we have used xlβi to denote Bφ ( hlβ ) i , δli = ∇hliL , and subscript i denotes slice over sample i in the batch , and subscript α denotes slice over neuron α in the layer . Then E ( ∂L ∂W lαβ ) 2 = E ( ∑ i δlαix l−1 βi ) 2 = E ∑ i , j δlαix l−1 βi δ l αjx l−1 βj . By gradient independence assumption ( Appendix B ) , E δlαix l−1 βi δ l αjx l−1 βj = E [ δ l αiδ l αj ] E [ x l−1 βi x l−1 βj ] = ΠlijΣ l−1 ij . Thus E ( ∂L ∂W lαβ ) 2 = ∑ i , j ΠlijΣ l−1 ij = 〈Πl , Σl−1〉 . If l is large and φ is well-behaved , then we expect Σl−1 ≈ Σ∗ , the unique BSB1 fixed point of Eq . ( 43 ) . Assuming equality , we then have E ( ∂L ∂W lαβ ) 2 = 〈Πl , Σ∗〉 = 〈G⊗2 { Πl } , Σ∗〉 = 〈Πl , G⊗2 { Σ∗ } 〉 = 〈Πl , µ∗G〉 = µ∗ tr Πl . Thus in general , we expect the magnitude of weight gradients to follow the magnitude of hidden unit gradient δli .

Summary: This paper develops a mean field theory for batch normalization (BN) in fully-connected networks with randomly initialized weights. There are a number of interesting predictions made in this paper on the basis of this analysis. The main technical results of the paper are Theorems 5-8 which compute the statistics of the covariance of the activations and the gradients.