A MEAN FIELD THEORY OF BATCH NORMALIZATION
Year: 2019

Abstract:
We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.

1 INTRODUCTION
Deep neural networks have been enormously successful across a broad range of disciplines. These successes are often driven by architectural innovations. For example, the combination of convolutions (LeCun et al., 1990), residual connections (He et al., 2015), and batch normalization (Ioffe & Szegedy, 2015) has allowed for the training of very deep networks and these components have become essential parts of models in vision (Zoph et al.), language (Chen & Wu, 2017), and reinforcement learning (Silver et al., 2017). However, a fundamental problem that has accompanied this rapid progress is a lack of theoretical clarity. An important consequence of this gap between theory and experiment is that two important issues become conflated. In particular, it is generally unclear whether novel neural network components improve generalization or whether they merely increase the fraction of hyperparameter configurations where good generalization can be achieved. Resolving this confusion has the promise of allowing researchers to more effectively and deliberately design neural networks.
Recently, progress has been made (Poole et al., 2016; Schoenholz et al., 2016; Daniely et al., 2016; Pennington et al., 2017; Hanin & Rolnick, 2018; Yang, 2019) in this direction by considering neural networks at initialization, before any training has occurred. In this case, the parameters of the network are random variables which induces a distribution of the activations of the network as well as the gradients. Studying these distributions is equivalent to understanding the prior over functions that these random neural networks compute. Picking hyperparameters that correspond to well-conditioned priors ensures that the neural network will be trainable and this fact has been extensively verified experimentally. However, to fulfill its promise of making neural network design less of a black box, these techniques must be applied to neural network architectures that are used in practice. Over the past year, this gap has closed significantly and theory for networks with skip connections (Yang & Schoenholz, 2017; 2018), convolutional networks (Xiao et al., 2018), and gated
∗Please see https://arxiv.org/abs/1902.08129 for the full and most current version of this paper
recurrent networks (Chen et al., 2018; Gilboa et al., 2019) have been developed. More recently, Yang (2019) devised a formalism that extends this approach to include an even wider range of architectures.
Before state-of-the-art models can be analyzed in this framework, a slowly-decreasing number of architectural innovations must be studied. One particularly important component that has thus-far remained elusive is batch normalization.
Our Contributions. In this paper, we develop a theory of fully-connected networks with batch normalization whose weights and biases are randomly distributed. A significant complication in the case of batch normalization (compared to e.g. layer normalization or weight normalization) is that the statistics of the network depend non-locally on the entire batch. Thus, our first main result is to recast the theory for random fully-connected networks so that it can be applied to batches of data. We then extend the theory to include batch normalization explicitly and validate this theory against Monte-Carlo simulations. We show that as in previous cases we can leverage our theory to predict valid hyperparameter configurations.
In the process of our investigation, we identify a number of previously unknown properties of batch normalization that make training unstable. In particular, for most nonlinearities used in practice, batchnorm in a deep, randomly initialized network induces high degree of symmetry in the embeddings of the samples in the batch (Thm 3.4). Whenever this symmetry takes hold, we show that for any choice of nonlinearity, gradients of fully-connected networks with batch normalization explode exponentially in the depth of the network (Thm 3.9). This imposes strong limits on the maximum trainable depth of batch normalized networks. This limit can be lifted partially but not completely by pushing activation functions to be more linear at initialization. It might seem that such gradient explosion ought to lead to learning dynamics that are unfavorable. However, we show that networks with batch normalization causes the scale of the gradients to naturally equilibrate after a single step of gradient descent (provided the initial gradients are not so large as to cause numerical instabilities). For shallower networks, this equilibrating effect is sufficient to allow adequate training.
Finally, we note that there is a related vein of research that has emerged that leverages the prior over functions induced by random networks to perform exact Bayesian inference (Lee et al., 2017; de G. Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Yang, 2019). One of the natural consequences of this work is that the prior for networks with batch normalization can be computed exactly in the wide network limit. As such, it is now possible to perform exact Bayesian inference in the case of wide neural networks with batch normalization.
2 RELATED WORK
Batch normalization has rapidly become an essential part of the deep learning toolkit. Since then, a number of similar modifications have been proposed including layer normalization (Ba et al., 2016) and weight normalization (Salimans & Kingma, 2016). Comparisons of performance between these different schemes have been challenging and inconclusive (Gitman & Ginsburg, 2017). The original introduction of batchnorm in Ioffe & Szegedy (2015) proposed that batchnorm prevents “internal covariate shift” as an explanation for its effectiveness. Since then, several papers have approached batchnorm from a theoretical angle, especially following Ali Rahimi’s catalyzing call to action at NIPS 2017. Balduzzi et al. (2017) found that batchnorm in resnets allow deep gradient signal propagation in contrast to the case without batchnorm. Santurkar et al. (2018) found that batchnorm does not help covariate shift but helps by smoothing loss landscape. Bjorck et al. (2018) reached the opposite conclusion as our paper for residual networks with batchnorm, that batchnorm works in this setting because it induces beneficial gradient dynamics and thus allows a much bigger learning rate. Luo et al. (2018) explores similar ideas that batchnorm allows large learning rates and likewise uses random matrix theory to support their claims. Kohler et al. (2018) identified situations in which batchnorm can provably induce acceleration in training. Of the above that mathematically analyze batchnorm, all but Santurkar et al. (2018) make simplifying assumptions on the form of batchnorm and typically do not have gradients flowing through the batch variance. Even Santurkar et al. (2018) only analyzes a vanilla network which gets added a single batchnorm at a single moment in training. Our analysis here on the other hand works for networks with arbitrarily many batchnorm layers with very general activation functions, and indeed, this deep stacking of batchnorm is precisely what
leads to gradient explosion. It is an initialization time analysis for networks of infinite width, but we show experimentally that the resulting insights predict both training and test time behavior.
We remark that Philipp & Carbonell (2018) has also empirically noted gradient explosion happens in deep batchnorm networks with various nonlinearities. In this work, in contrast, we develop a precise theoretical characterization of gradient statistics and, as a result, we are able to make significantly stronger conclusions.
3 THEORY
We begin with a brief recapitulation of mean field theory in the fully-connected setting. In addition to recounting earlier results, we rephrase the formalism developed previously to compute statistics of neural networks over a batch of data. Later, we will extend the theory to include batch normalization. We consider a fully-connected network of depth L whose layers have width Nl, activation function1 φ, weights W l ∈ RNl−1×Nl , and biases bl ∈ RNl . Given a batch of B inputs2 {xi : xi ∈ RN0}i=1,··· ,B , the pre-activations of the network are defined by the recurrence relation,
h1i = W 1xi + b 1 and hli = W lφ(hl−1i ) + b l ∀ l > 1. (1) At initialization, we choose the weights and biases to be i.i.d. as W lαβ ∼ N (0, σ2w/Nl−1) and blα ∼ N (0, σ2b ). In the following, we use α, β, . . . for the neuron index, and i, j, . . . for the batch index. We will be concerned with understanding the statistics of the pre-activations and the gradients induced by the randomness in the weights and biases. For ease of exposition we will typically take the network to have constant width Nl = N .
In the mean field approximation, we iteratively replace the pre-activations in Eq. (2) by Gaussian random variables with matching first and second moments. In the infinite width limit this approximation becomes exact (Lee et al., 2017; de G. Matthews et al., 2018; Yang, 2019). Since the weights are i.i.d. with zero mean it follows that the mean of each pre-activation is zero and the covariance between distinct neurons are zero. The pre-activation statistics are therefore given by (hlα11, · · · , hlαBB) Nl−1→∞−−−−−−→ N (0,Σlδα1···αB ) where Σl are B × B covariance matrices and δα1,···αB is the Kronecker-δ that is one if α1 = α2 = · · · = αB and zero otherwise. Definition 3.1. Let V be the operator on functions φ such that Vφ(Σ) = E[φ(h)φ(h)T : h ∼ N (0,Σ)] computes the matrix of uncentered second moments of φ(h) for h ∼ N (0,Σ).
Using the above notation, we can express the covariance matrices by the recurrence relation,
Σl = σ2wVφ(Σ l−1) + σ2b11 T (2)
At first Eq. (2) may seem challenging since the expectation involves a Gaussian integral in RB . However, each term in the expectation of Vφ involves at most a pair of pre-activations and so the expectation may be reduced to the evaluation of O(B2) two-dimensional integrals. These integrals can either be performed analytically (Cho & Saul, 2009; Williams, 1997) or efficiently approximated numerically (Lee et al., 2017), and so Eq. (2) defines a computationally efficient method for computing the statistics of neural networks after random initialization. This theme of dimensionality reduction will play a prominent role in the forthcoming discussion on batch normalization.
Eq. (2) defines a dynamical system over the space of covariance matrices. Studying the statistics of random feed-forward networks therefore amounts to investigating this dynamical system and is an enormous simplification compared with studying the pre-activations of the network directly. As is common in the dynamical systems literature, a significant amount of insight can be gained by investigating the behavior of Eq. (2) in the vicinity of its fixed points. For most common activation functions, Eq. (2) has a fixed point at some Σ∗. Moreover, when the inputs are non-degenerate, this fixed point generally has a simple structure with Σ∗ = q∗[(1− c∗)I + c∗11T ] owing to permutation symmetry among elements of the batch. We refer to fixed points with such symmetry as BSB1 (Batch Symmetry Breaking 1 or 1 Block Symmetry Breaking) fixed points. As we will discuss later, in the context of batch normalization other fixed points with fewer symmetries (“BSBk fixed points”)
1The activation function may be layer dependent, but for ease of exposition we assume that it is not. 2Throughout the text, we assume that all elements of the batch are unique.
may become preferred. In the fully-connected setting fixed points may efficiently be computed by solving the fixed point equation induced by Eq. (2) in the special case B = 2. The structure of this fixed point implies that in asymptotically deep feed-forward neural networks all inputs yield pre-activations of identical norm with identical angle between them. Neural networks that are deep enough so that their pre-activation statistics lie in this regime have been shown to be untrainable (Schoenholz et al., 2016).
Notation As we often talk about matrices and also linear operators over matrices, we write T {Σ} for an operator T applied to a matrix Σ, and matrix multiplication is still written as juxtaposition. Composition of matrix operators are denoted with T1 ◦ T2.
Local Convergence to BSB1 Fixed Point. To understand the behavior of Eq. (2) near its BSB1 fixed point we can consider the Taylor series in the deviation from the fixed point, ∆Σl = Σl −Σ∗. To lowest order we generically find,
∆Σl = J{∆Σl−1} (3)
where J = dVφdΣ ∣∣ Σ=Σ∗
is the B2 ×B2 Jacobian of Vφ. In most prior work where φ was a pointwise non-linearity, Eq. (3) reduces to the case B = 2 which naturally gave rise to linearized dynamics in ql = E[(hli)2] and cl = E[hlizlj ]/ql. However, in the case of batch normalization we will see that one must consider the evolution of Eq. (3) as a whole. This is qualitatively reminiscent of the case of convolutional networks studied in Xiao et al. (2018) where the evolution of the entire pixel × pixel covariance matrix had to be evaluated.
The dynamics induced by Eq. (3) will be controlled by the eigenvalues of J : Suppose J has eigenvalues λi — ordered such that λ1 ≥ λ2 ≥ · · · ≥ λB2 — with associated eigen“vectors” ei (note that the ei will themselves be B×B matrices). It follows that if ∆Σ0 = ∑ i ciei for some choice of
constants ci then ∆Σl = ∑ i ciλ l iei. Thus, if λi < 1 for all i, ∆Σ
l will approach zero exponentially and the fixed-point will be stable. The number of layers over which Σ will approach Σ∗ will be given by −1/ log(λ1). By contrast if λi > 1 for any i then the fixed point will be unstable. In this case, there is typically a different, stable, fixed point that must be identified. It follows that if the eigenvalues of J can be computed then the dynamics will follow immediately.
While J may appear to be a complicated object at first, a moment’s thought shows that each diagonal element J{∆Σ}ii is only affected by the corresponding diagonal element ∆Σii, and each off-diagonal element J{∆Σ}ij is only affected by ∆Σii,∆Σjj , and ∆Σij . Such a Diagonal-Offdiagonal Semidirect (DOS) operator has a simple eigendecomposition with two eigenspaces corresponding to changes in the off-diagonal and changes in the diagonal (Thm E.72). The associated eigenvalues are precisely those calculated by Schoenholz et al. (2016) in a simplified analysis. DOS operators are a particularly simple form of a more general operator possessing an abundance of symmetries called ultrasymmetric operators, which will play a prominent role in the analysis of batchnorm below.
Gradient Dynamics. Similar arguments allow us to develop a theory for the statistics of gradients. The backpropogation algorithm gives an efficient method of propagating gradients from the end of the network to the earlier layers as,
dL dW l = ∑ i δliφ(h l−1 i ) T δlαi = φ ′(hlαi) ∑ β W l+1βα δ l+1 βi . (4)
Here L is the loss function and δli = ∇hliL are Nl-dimensional vectors that describe the error signal from neurons in the l’th layer due to the i’th element of the batch. The preceding discussion gave a precise characterization of the statistics of the hli that we can leverage to understand the statistics of δli. Assuming gradient independence, that is, that an iid set of weights are used during backpropagation (see Appendix B for more discussions), it is easy to see that E[δlαi] = 0 and E[δlαiδlβj ] = Πlij if α = β and 0 otherwise, where Πl is a covariance matrix and we may once again drop the neuron index. We can construct a recurrence relation to compute Πl,
Πl = σ2wVφ′(Σ l) Πl+1. (5)
Typically, we will be interested in understanding the dynamics of Πl when Σl has converged exponentially towards its fixed point. Thus, we study the approximation,
Πl ≈ σ2wVφ′(Σ∗) Πl+1. (6) Since these dynamics are linear (and in fact componentwise), explosion and vanishing of gradients will be controlled by Vφ′(Σ∗).
3.1 BATCH NORMALIZATION
We now extend the mean field formalism to include batch normalization. Here, the definition for the neural network is modified to be the coupled equations,
hli = W lφ(h̃l−1i ) + b l h̃lαi = γα hlαi − µα
σα + βα (7)
where γα and βα are parameters, and µα = 1Nl ∑ i hαi and σ 2 α = √ 1 Nl ∑ i(hαi − µα)2 + are the per-neuron batch statistics. In practice ≈ 10−5 or so to prevent division by zero, but in this paper, unless stated otherwise (in the last few sections), is assumed to be 0. Unlike in the case of vanilla fully-connected networks, here the pre-activations are invariant to σ2w and σ 2 b . Without a loss of generality, we therefore set σ2w = 1 and σ 2 b = 0 for the remainder of the text. In principal, batch normalization additionally yields a pair of hyperparameters γ and β which are set to be constants. However, these may be incorporated into the nonlinearity and so without a loss of generality we set γ = 1 and β = 0. In order to avoid degenerate results, we assume B ≥ 4 unless stated otherwise; we shall discuss the small B regime in Appendix J.
If one treats batchnorm as a “batchwise nonlinearity”, then the arguments from the previous section can proceed identically and we conclude that as the width of the network grows, the pre-activations will be jointly Gaussian with identically distributed neurons. Thus, we arrive at an analogous expression to Eq. (2),
Σl = VBφ(Σ l−1) where Bφ : RB → RB , Bφ(h) = φ
(√ BGh
||Gh||
) . (8)
Here we have introduced the projection operator G = I − 1B11T which is defined such that Gx = x − µ1 with µ = ∑i xi/B. Unlike φ, Bφ does not act component-wise on h. It is therefore not obvious whether VBφ can be evaluated without performing a B-dimensional Gaussian integral.
Theoretical tools. In this paper, we present several ways to analyze high dimensional integrals like the above: 1. the Laplace method 2. the Fourier method 3. spherical integration 4. and the Gegenbauer method. The former two use the Laplace and Fourier transforms to simplify expressions like the above, where the Laplace method requires that φ be positive homogeneous. Often the Laplace method will give clean, closed form answers for such φ. Because batchnorm can be thought of as a linear projection (G) followed by projection to the sphere of radius √ B, spherical integration techniques are often very useful and in fact is typically the most straightforward way of numerically evaluating quantities. Lastly, the Gegenbauer method expresses objects in terms of the Gegenbauer coefficients of φ. Briefly, Gegenbauer polynomials {C(α)l (x)}∞l=0 are orthogonal polynomials with respect to the measure (1 − x2)α− 12 on [−1, 1]. They are intimately related to spherical harmonics (a natural basis for functions on a sphere), which explains their appearance in this context. The Gegenbauer method is the most illuminating amongst them all, and is what allows us to conclude that gradient explosion happens regardless of nonlinearity under general conditions. See Appendix D for a more in-depth discussion of these techniques. In what follows, we will mostly present results by the Laplace method for φ = relu and by the Gegenbauer method for general φ, but give pointers to the appendix for others.
Back to the topic of Eq. (8), we present a pair of results that expresses Eq. (8) in a more manageable form. From previous work (Poole et al., 2016), Vφ can be expressed in terms of a two-dimensional Gaussian integrals independent of B. When φ is degree-α positive homogeneous (e.g. rectified linear activations) we can relate Vφ and VBφ by the Laplace transform. (see Thm E.5).
Theorem 3.2. Suppose φ : R→ R is degree-α positive homogeneous. For any positive semi-definite matrix Σ define the projection ΣG = GΣG. Then
VBφ(Σ) = Bα
Γ(α) ∫ ∞ 0 ds sα−1 Vφ(Σ G(I + 2sΣG)−1)√ det(I + 2sΣG) . (9)
whenever the integral exists.
Using this parameterization, when Vφ has a closed form solution (like ReLU), VBφ involves only a single integral. A similar expression can be derived for general φ by applying Fourier transform; see Appendix E.2. Next we express Eq. (8) as a spherical integral (see Proposition E.34) Theorem 3.3. Let e ∈ RB×B−1 have as columns a set of orthonormal basis vectors of {x ∈ RB : Gx = x}. Then with SB−2 ⊆ RB−1 denoting the (B − 2)-dimensional sphere,
VBφ(Σ) = E v∼SB−2
φ( √ Bev)⊗2√
det eTΣe(vT (eTΣe)−1v) B−1 2
. (10)
Together these theorems provide analytic recurrence relations for random neural networks with batch normalization over a wide range of activation functions. By analogy to the fully-connected case we would like to study the dynamical system over covariance matrices induced by these equations.
We begin by investigating the fixed point structure of Eq. (8). As in the case of feed-forward networks, permutation symmetry implies that there exist BSB1 fixed points Σ∗ = q∗[(1−c∗)I+c∗11T ]. We will see that this fixed point is in fact unique, and a clean expression of q∗ and c∗ can be obtained in terms of Gegenbauer basis (see Thm F.13). Theorem 3.4 (Gegenbauer expansions of BSB1 fixed point). If φ( √ B − 1x) has Gegenbauer ex-
pansion ∑∞ l=0 al 1 cB−1,l C (B−32 ) l (x) where cB−1,l = B−3 B−3+2l , then
q∗ = ∞∑ l=0 a2l 1 cB−1,l C (B−32 ) l (1), q ∗c∗ = ∞∑ l=0 a2l 1 cB−1,l C (B−32 ) l ( −1 B − 1 ) .
Thus the entries of the BSB1 fixed point are diagonal quadratic forms of the Gegenbauer coefficients of φ( √ B − 1x). Even more concise closed forms are available when the activation functions are degree α positive homogeneous (see Thm F.8). In particular, for ReLU we arrive at the following Theorem 3.5 (BSB1 fixed point for ReLU). When φ = relu, then
q∗ = 1
2 , c∗ = J1 ( −1 B − 1 ) = 1 π − 1 2(B − 1) +O (
1 (B − 1)2 )
(11)
where J1(c) = 1π ( √ 1− c2 + (π − arccos(c))c) is the arccosine kernel (Cho & Saul, 2009).
In the appendix, Thm F.5 also describes a trigonometric integral formula for general nonlinearities.
In the presence of batch normalization, when the activation function grows quickly, a winner-takeall phenomenon can occur where a subset of samples in the batch have much bigger activations than others. This causes the covariance matrix to form blocks of differing magnitude, breaking the BSB1 symmetry. One notices this, for example, as the degree α of α-ReLU (i.e. αth power of ReLU) increases past a point αtransition(B) depending on the batch size B (see Fig. 1). However, we observe, through simulations, that by far most of the nonlinearities used in practice, like ReLU, leaky ReLU, tanh, sigmoid, etc, all lead to BSB1 fixed points, and we can prove this rigorously for φ = id (see Corollary F.3). We discuss symmetry-broken fixed points (BSB2 fixed points) in Appendices J and K, but in the main text, from here on, Assumption 1. Unless stated otherwise, we assume that any nonlinearity φ mentioned induces Σl to converge to a BSB1 fixed point under the dynamics of Eq. (8).
3.1.1 LINEARIZED DYNAMICS
With the fixed point structure for batch normalized networks having been described, we now investigate the linearized dynamics of Eq. (8) in the vicinity of these fixed points.
To determine the eigenvalues of dVBφ
dΣ ∣∣ Σ=Σ∗
it is helpful to consider the action of batch normalization in more detail. In particular, we notice that Bφ can be decomposed into the composition of three separate operations, Bφ = φ ◦ n ◦G. As discussed above, Gh subtracts the mean from h and we introduce the new function n(h) = √ Bh/||h|| which normalizes a centered h by its standard deviation. Applying the chain rule, we can rewrite the Jacobian as,
dVBφ(Σ)
dΣ =
dV[φ◦n](Σ G)
dΣG ◦G⊗2 (12)
where ◦ denotes composition andG⊗2 is the natural extension ofG to act on matrices asG⊗2{Σ} = GΣG = ΣG. It ends up being advantageous to study G⊗2 ◦ dV[φ◦n]dΣ ∣∣∣ Σ=Σ∗
◦G⊗2 =: G⊗2 ◦ Ĵ ◦G⊗2 and to note that the nonzero eigenvalues of this object are identical to the nonzero eigenvalues of the Jacobian (see Lemma F.17).
At face value, this is a complicated object since it simultaneously has large dimension and possesses an intricate block structure. However, the permutation symmetry of the BSB1 Σ∗ induces strong symmetries in Ĵ that significantly simplify the analysis (see Appendix F.3). In particular while Ĵijkl is a four-index object, we have Ĵijkl = Ĵπ(i)π(j)π(k)π(l) for all permutations π on B and Ĵijkl = Ĵjilk. We call linear operators possessing such symmetries ultrasymmetric (Defn E.53) and show that all ultrasymmetric operators conjugated by G⊗2 admit an eigendecomposition that contains three distinct eigenspaces with associated eigenvalues (see Thm E.62).
Theorem 3.6. Let T be an ultrasymmetric matrix operator. Then on the space of symmetric matrices, G⊗2 ◦ T ◦G⊗2 has the following orthogonal (under trace inner product) eigendecomposition,
1. an eigenspace {Σ : ΣG = 0} with eigenvalue 0.
2. a 1-dimensional eigenspace RG with eigenvalue λG,TG .
3. a (B − 1)-dimensional eigenspace L := {DG : D diagonal, trD = 0}, with eigenvalue λG,TL .
4. a B(B−3)2 -dimensional eigenspace M := {Σ : ΣG = Σ,DiagΣ = 0}, with eigenvalue λG,TM .
The specific forms of the eigenvalues can be obtained as linear functions of the entries of T ; see Thm E.62 for details. Note that, as the eigenspaces are orthogonal, this implies that G⊗2 ◦ T ◦G⊗2 is self-adjoint (even when T is not). In our context with T = Ĵ , the eigenspaces can be roughly interpreted as follows: The deviation ∆Σ = Σ − Σ∗ from the fixed point decomposes as a linear combination of components in each of the eigenspaces. The RG-component captures the average norm of elements of the batch (the trace of ∆Σ), the L-component captures the fluctuation of such norms, and the M-component captures the covariances between elements of the batch.
Because of the explicit normalization of batchnorm, one sees immediately that the RG-component goes to 0 after 1 step. For positive homogeneous φ, we can use the Laplace method to obtain closed form expressions for the other eigenvalues (see Thm F.33). The below theorem shows that, as the batch size becomes larger, a deep ReLU-batchnorm network takes more layers to converge to a BSB1 fixed point.
Theorem 3.7. Let φ = relu and B > 3. The eigenvalues of G⊗2 ◦ Ĵ ◦G⊗2 for L and M are
λ↑L = 1
2(B + 1)µ∗
( (B − 2) [ 1− J1 ( −1 B − 1 )] + B B − 1J ′ 1 ( −1 B − 1 )) ↗ 1 (13)
λ↑M = B 2(B + 1)µ∗ J′1 ( −1 B − 1 ) ↗ π 2(π − 1) ≈ 0.733 (14)
where µ∗ = q∗(1− c∗) = 12 ( 1− J1 ( −1 B−1 )) , and↗ denotes increasing limit as B →∞.
More generally, we can evaluate them for general nonlinearity using spherical integration (Appendix F.3.1) and, more enlightening, using the Gegenbauer method, is the following
Theorem 3.8. If φ( √ B − 1x) has Gegenbauer expansion∑∞l=0 al 1cB−1,lC(B−32 )l (x), then
λ↑L =
∑∞ l=0 a
2 lwB−1,l + alal+2uB−1,l∑∞
l=0 a 2 l vB−1,l
, λ↑M =
∑∞ l=0 a
2 l w̃B−1,l + alal+2ũB−1,l∑∞
l=0 a 2 l vB−1,l
(15)
where the coefficients wB−1,l, uB−1,l, w̃B−1,l, ũB−1,l, vB−1,l are given in Thms F.22 and F.24.
A BSB1 fixed point is not locally attracting if λ↑L > 1 or λ ↑ M > 1. Thus Thm 3.8 yields insight on the stability of the BSB1 fixed point, which we can interpret heuristically as follows. The specific forms of the coefficients wB−1,l, uB−1,l, w̃B−1,l, ũB−1,l, vB−1,l show that λ ↑ M is typically much smaller than λ↑L (but there are exceptions like φ = sin), and wB−1,1 < vB−1,1 but wB−1,l ≥ vB−1,l for all l ≥ 2. Thus one expects that, the larger a1 is, i.e. the “more linear” and less explosive φ is, the smaller λ↑L is and the more likely that Eq. (8) converges to a BSB1 fixed point. This is consistent with the “winner-take-all” intuition for the emergence of BSB2 fixed point explained above. See Appendix F.3.2 for more discussion.
3.1.2 GRADIENT BACKPROPAGATION
With a mean field theory of the pre-activations of feed-forward networks with batch normalization having been developed, we turn our attention to the backpropagation of gradients. In contrast to the case of networks without batch normalization, we will see that exploding gradients at initialization are a severe problem here. To this end, one of the main results from this section will be to show that fully-connected networks with batch normalization feature exploding gradients for any choice of nonlinearity such that Σl converges to a BSB1 fixed point. Below, by rate of gradient explosion we mean the β such that the gradient norm squared grows as βL+o(L) with depth L. As before, all computations below assumes gradient independence (see Appendix B for a discussion).
As a starting point we seek an analog of Eq. (6) in the case of batch normalization. However, because the activation functions no longer act point-wise on the pre-activations, the backpropagation equation becomes,
δlαi = ∑ βj ∂Bφ(hlα)j ∂hlαi W l+1βα δ l+1 βj (16)
where hlα = (h l α1, . . . , h l αB) and we observe the additional sum over the batch. Computing the resulting covariance matrix Πl, we arrive at the recurrence relation,
Πl = E
[( dBφ(h)
dh
)T Πl+1
dBφ(h) dh
: h ∼ N (0,Σl) ]
=: VB′φ(Σ l)†{Πl+1} (17)
where we have defined the linear operator VF (Σ)†{·} such that VF (Σ)†{Π} = E[FTh ΠFh : h ∼ N (0,Σ)] for any vector-indexed linear operator Fh. As in the case of vanilla feed-forward networks, here we will be concerned with the behavior of gradients when Σl is close to its fixed point. We therefore study the asymptotic approximation to Eq. (17) given by Πl = VB′φ(Σ
∗)†{Πl+1}. In this case the dynamics of Π are linear and are therefore naturally determined by the eigenvalues of VB′φ(Σ ∗)†.
As in the forward case, batch normalization is the composition of three operations Bφ = φ ◦ n ◦G. Applying the chain rule, Eq. (17) can be rewritten as,
VB′φ(Σ) † = G⊗2 ◦ E (d(φ ◦ n)(z) dz ∣∣∣∣T z=Gh )⊗2 : h ∼ N (0,Σ)  =: G⊗2 ◦ F (Σ) (18) with F (Σ) appropriately defined. Note that since G⊗2 is an idempotent operator, (VB′φ(Σ) †)n = (G⊗2 ◦ F (Σ))n = (G⊗2 ◦ F (Σ) ◦G⊗2)n−1 ◦G⊗2 ◦ F (Σ), so that it suffices to study the eigendecomposition of G⊗2 ◦ F (Σ∗) ◦G⊗2. Due to the symmetry of Σ∗, F (Σ∗) is ultrasymmetric, so that
G⊗2 ◦ F (Σ∗) ◦ G⊗2 has eigenspaces RG,L,M and we can compute its eigenvalues via Thm 3.6. More illuminating, however, is the Gegenbauer expansion (see Thm G.5). It requires a new identity Thm E.47 involving Gegenbauer polynomials integrated over a sphere, which may be of independent interest. Theorem 3.9 (Batchnorm causes gradient explosion). Suppose φ( √ B − 1x), φ′( √ B − 1x) ∈
L2((1− x2)B−32 ). If φ( √ B − 1x) has Gegenbauer expansion∑∞l=0 al 1cB−1,lC(B−32 )l (x), then gradients explode at the rate of
λ↓G =
∑∞ l=0 ( l+B−3 B−3 · l ) a2l rl∑∞
l=0 a 2 l rl where rl = c−1B−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1 )) > 0. Consequently, for any non-constant φ (i.e. there is a j > 0 such that aj 6= 0), λ↓G > 1; φ minimizes λ↓G iff it is linear (i.e. ai = 0,∀i ≥ 2), in which case gradients explode at the rate of B−2B−3 .
This contrasts starkly with the case of non-normalized fully-connected networks, which can use the weight and bias variances to control its mean field network dynamics (Poole et al., 2016; Schoenholz et al., 2016). As a corollary, we disprove the conjecture of the original batchnorm paper (Ioffe & Szegedy, 2015) that “Batch Normalization may lead the layer Jacobians to have singular values close to 1” in the initialization setting, and in fact prove the exact opposite, that batchnorm forces the layer Jacobian singular values away from 1.
Appendix G.1 discusses the numerical evaluation of all eigenvalues, and as usual, the Laplace method yields closed forms for positive homogeneous φ (Thm G.12). We highlight the result for ReLU. Theorem 3.10. In a ReLU-batchnorm network, the gradient norm explodes exponentially at the rate of
1
B − 3
B − 1 + J′1 ( −1 B−1 ) 1− J1 ( −1 B−1 ) − 1  (19)
which decreases to ππ−1 ≈ 1.467 as B → ∞. In contrast, for a linear batchnorm network, the gradient norm explodes exponentially at the rate of B−2B−3 , which goes to 1 as B →∞.
Fig. 1 shows theory and simulation for ReLU gradient dynamics.
Weight Gradient While all of the above only study the gradient with respect to the hidden preactivations, Appendix L shows that the weight gradient norms at layer l is just 〈Πl, µ∗G〉 = µ∗ tr Πl, and thus by Thm 3.9, the weight gradients explode as well at the same rate λ↓G.
Effect of as a hyperparameter In practice, is usually treated as small constant and is not regarded as a hyperparameter to be tuned. Nevertheless, we can investigate its effect on gradient explosion. A straightforward generalization of the analysis presented above to the case of > 0 suggests somewhat larger values than typically used can ameliorate (but not eliminate) gradient explosion problems. See Fig. 5(c,d).
3.2 CROSS-BATCH DYNAMICS
Forward In addition to analyzing the correlation between preactivations of samples in a batch, we also study the correlation between those of different batches. The dynamics Eq. (8) can be generalized to simultaneous propagation of k batches (see Eq. (54) and Appendix H).
Σ̃l = VB⊕kφ (Σ̃l−1) = E (h1,...,hk)∼N (0,Σ̃l−1) (Bφ(h1), . . . ,Bφ(hk))⊗2, with Σ̃l ∈ RkB×kB . (20)
Here the domain of the dynamics is the space of block matrices, with diagonal blocks and offdiagonal blocks resp. representing within-batch and cross-batch covariance. We observe empirically
Published as a conference paper at ICLR 2019
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10 1.6 3.1 4.6
6.1
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10 0.12 0.18 0.25 0.32 0.38 0.45
0.51
0 20 40 60 80 100 layer
0
10
20
30
40
50
60
lo gn or m
batch 8 16 32 64 128
20 40 60 80 100 batchsLze
0.8
1.0
1.2
1.4
1.6
1.8
eL ge nv al ue s
G L 0 Var( aa)
Var( ab)
transition
0.5 1.0 1.5 2.0
10-4
0.001
0.010
0.100
1
10a b c d
0.1
0.5 1.0
6.0
Figure 1: Numerical confirmation of theoretical predictions. (a,b) Comparison between theoretical prediction (dashed lines) and Monte Carlo simulations (solid lines) for the eigenvalues of the backwards Jacobian (see Thms 3.10 and G.12) as a function of batch size and the magnitude of gradients as a function of depth respectively for rectified linear networks. In each case Monte Carlo simulations are averaged over 200 sample networks of width 1000 and shaded regions denote 1 standard deviation. Dashed lines are shifted slightly for easier comparison. (c,d) Demonstration of the existence of a BSB1 to BSB2 symmetry breaking transition as a function of α for α-ReLU (i.e. the αth power of ReLU) activations. In (c) we plot the empirical variance of the diagonal and off-diagonal entries of the covariance matrix which clearly shows a jump at the transition. In (d) we
plot representative covariance matrices for the two phases (BSB1 bottom, BSB2 top).Published as a conference paper at ICLR 2019
(a)
(b)
Figure A.1: Batch norm leads to a chaotic input-output map with increasing depth. A linear network with batch norm is shown acting on two minibatches of size 64 after random orthogonal initialization. The datapoints in the minibatch are chosen to form a 2d circle in input space, except for one datapoint that is perturbed separately in each minibatch (leftmost datapoint at input layer 0). Because the network is linear, for a given minibatch it performs an affine transformation on its inputs – a circle in input space remains an ellipse throughout the network. However, due to batch norm the coefficients of that affine transformation change nonlinearly as the datapoints in the minibatch are changed. (a) Each pane shows a scatterplot of activations at a given layer for all datapoints in the minibatch, projected onto the top two PCA directions. PCA directions are computed using the concatenation of the two minibatches. Due to the batch norm nonlinearity, minibatches that are nearly identical in input space grow increasingly dissimilar with depth. Intuitively, this chaotic input-output map can be understood as the source of exploding gradients when batch norm is applied to very deep networks, since very small changes in an input correspond to very large movements in network outputs. (b) The correlation between the two minibatches, as a function of layer, for the same network. Despite having a correlation near one at the input layer, the two minibatches rapidly decorrelate with depth. See ?? for a theoretical treatment.
A.1 VGG19 WITH BATCHNORM ON CIFAR100
Even though at initialization time batchnorm causes gradient explosion, after the first few epochs, the relative gradient norms kr✓Lk/k✓k for weight parameters ✓ = W or BN scale parameter ✓ = , equilibrate to about the same magnitude. See Fig. A.2.
Published as a conference paper at ICLR 2019
(a)
(b)
Figure A.1: Batch norm leads to a chaotic input-output map with increasing depth. A linear
network with batch norm is shown acting on two minibatches of size 64 after random orthogonal
initialization. The datapoints in the minibatch are chosen to form a 2d circle in input space, except
for one datapoint that is perturbed separately in each minibatch (leftmost datapoint at input layer 0).
Because the network is linear, for a given minibatch it performs an affine transformation on its inputs
– a circle in input space remains an ellipse throughout the network. However, due to batch norm
the coefficients of that affine transformation change nonlinearly as the datapoints in the minibatch
are changed. (a) Each pane shows a scatterplot of activations at a given layer for all datapoints
in the minibatch, projected onto the top two PCA directions. PCA directions are computed using
the concatenation of the two minibatches. Due to the batch norm nonlinearity, minibatches that
are nearly identical in input space grow increasingly dissimilar with depth. Intuitively, this chaotic
input-output map can be understood as the source of exploding gradients when batch norm is applied
to very deep networks, since very small changes in an input correspond to very large movements in
network outputs. (b) The correlation between the two minibatches, as a function of layer, for the
same network. Despite having a correlation near one at the input layer, the two minibatches rapidly decorrelate with depth. See ?? for a theoretical treatment.
A.1 VGG19 WITH BATCHNORM ON CIFAR100
Even though at initialization time batchnorm causes gradient explosion, after the first few epochs, the relative gradient norms kr✓Lk/k✓k for weight parameters ✓ = W or BN scale parameter ✓ = , equilibrate to about the same magnitude. See Fig. A.2.
14
(a)
(b)
Figure A.1: Batch norm leads to a chaotic input-output map with increasing depth. A linear network with batch norm is shown acting on two minibatches of size 64 after random orthogonal initialization. The datapoints in the minibatch are chosen to form a 2d circle in input space, except for one datapoint that is perturbed separately in each minibatch (leftmost datapoint at input layer 0). Because the network is linear, for a given minibatch it performs an affine transformation on its inputs – a circle in input space remains an ellipse throughout the network. However, due to batch norm the coefficients of that affine transformation change nonlinearly as the datapoints in the minibatch are changed. (a) Each pane shows a scatterplot of activations at a given layer for all datapoints in the minibatch, projected onto the top two PCA directions. PCA directions are computed using the conc tenation of the two minibatches. Due to the batch norm nonlinearity, minibatches that are nearly identical in input space grow increasingly dissimilar with depth. Intuitively, this chaotic input-output map can be understood as the source of exploding gradients when batch norm is applied to very deep networks, since very small changes in an input correspond to very large movements in network outputs. (b) The correlation between the two minibatches, as a function of layer, for the same network. Despite havi g a correlation near one at the input layer, the two minibatches rapidly decorrelate with depth. See ?? for a theoretical treatment.
A.1 VGG19 WITH BATCHNORM ON CIFAR100
Even though at initialization time batchnorm causes gradient explosion, after the first few epochs, the relative gradient norms kr✓Lk/k✓k for weight parameters ✓ = W or BN scale parameter ✓ = , equilibrate to about the same magnitude. See Fig. A.2.
14
a b
that for most nonlinearities used in practice like ReLU, or even for the identity function, ie no pointwise nonlinearity, the global fixed point of this dynamics is cross-batch BSB1, with diagonal BSB1 blocks, and off-diagon l e tr es ll equal to the same constant c∗CB:
Σ̃∗ =  Σ∗ c∗CB c ∗ CB · · · c∗CB Σ ∗ c∗CB · · · c∗CB c ∗ CB Σ
∗ · · · ... ... ... . . .
 . (21)
10
To interpret this phenomenon, note that, after mean centering of the preactivations, this covariance matrix becomes a multiple of identity. Thus, deep embedding of batches loses the mutual information between them in the input space. Qualitatively, this implies that two batches that are similar at the input to the network will become increasingly dissimilar — i.e. chaotic — as the signal propagates deep into the network. This loss in fact happens exponentially fast, as illustrated in Fig. 2, and as shown theoretically by Thm H.13, at a rate of λ↑
M̃ < 1. Formally, the linearized dynamics of
Eq. (20) has an eigenspace M̃ given by all block matrices with zero diagonal blocks. The associated eigenvalue is λ↑
M̃ . We shall return to λ↓ M̃ shortly and give its Gegenbauer expansion (Thm 3.11).
Backward The gradients of two batches of input are correlated throughout the course of backpropagation. We hence also study the generalization of Eq. (17) to the k-batch setting (see Appendix I):
Π̃l = VB⊕2φ ′(Σ̃l)†{Π̃l+1} (22)
VB⊕2φ ′(Σ̃∗)† {( Π1 Ξ ΞT Π2 )} = E
(x,y)∼N (0,Σ̃∗) [(B′φ(x)TΠ1B′φ(x) B′φ(x)TΞB′φ(y) B′φ(y)TΞTB′φ(x) B′φ(y)TΠ2B′φ(y) )] (23)
where for simplicity of exposition and without loss of generality, we take k = 2. Like in Eq. (17), we have assumed that Σ̃l has converged to its limit Σ̃∗. It is not hard to see that each block of Π̃l evolves independently, with the diagonal blocks in particular evolving according to Eq. (17). Analyzing the off-diagonal blocks might seem unwieldy at first, but it turns out that its dynamics is given simply by scalar multiplication by a constant λ↓
M̃ (Thm I.4). Even more surprisingly,
Theorem 3.11 (Batchnorm causes information loss). The convergence rate of cross-batch preactivation covariance λ↑
M̃ is equal to the decay rate of the cross-batch gradient covariance λ↓ M̃ . If φ( √ B − 1x) has Gegenbauer expansion∑∞l=0 al 1cB−1,lC(B−32 )l (x), then both are given by
λ↑ M̃ = λ↓ M̃
= 1
2πB(B − 1)a21 Beta ( B 2 , 1 2 )2∑∞ l=0 a 2 l 1 cB−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1
)) . (24) This quantity is always < 1, and for any fixed B, is maximized by φ = id. Furthermore, for φ = id, λ↑ M̃ = λ↓ M̃ = B−12 P ( B 2 , 1 2 )−2 and increase to 1 from below as B →∞.
Thus, a deep batchnorm network loses correlation information between two input batches, exponentially fast in depth, no matter what nonlinearity (that induces fixed point of the form given by Eq. (21)). This again contrasts with the case for vanilla networks which can control this rate of information loss by tweaking the initialization variances for weights and biases Poole et al. (2016); Schoenholz et al. (2016). The absence of coordinatewise nonlinearity, i.e. φ = id, maximally suppresses both this loss of information as well as the gradient explosion, and in the ideal, infinite batch scenario, can cure both problems.
4 EXPERIMENTS
Having developed a theory for neural networks with batch normalization at initialization, we now explore the relationship between the properties of these random networks and their learning dynamics. We will see that the trainability of networks with batch normalization is controlled by gradient explosion. We quantify the depth scale over which gradients explode by ξ = 1/ log λ↓G where, as above, λ↓G is the largest eigenvalue of the jacobian. Across many different experiments we will see strong agreement between ξ and the maximum trainable depth.
We first investigate the relationship between trainability and initialization for rectified linear networks as a function of batch size. The results of these experiments are shown in Fig. 3 where in each case we plot the test accuracy after training as a function of the depth and the batch size and overlay 16ξ in white dashed lines. In Fig. 3 (a) we consider networks trained using SGD on MNIST where we observe that networks deeper than about 50 layers are untrainable regardless of batch size. In (b) we compare standard batch normalization with a modified version in which the batch size is
a b c d 0.1 1.0 0.1 0.55
Figure 3: Batch normalization strongly limits the maximum trainable depth. Colors show test accuracy for rectified linear networks with batch normalization and γ = 1, β = 0, = 10−3, N = 384, and η = 10−5B. (a) trained on MNIST for 10 epochs (b) trained with fixed batch size 1000 and batch statistics computed over sub batches of size B. (c) trained using RMSProp. (d) Trained on CIFAR10 for 50 epochs. In each case, White dashed line indicates a theoretical prediction of trainable depth, as discussed in the text.
t <latexit sha1_base64="USoNv0fCIvi1lLEl3uP/p3RZN/c=">AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w=</latexit><latexit sha1_base64="USoNv0fCIvi1lLEl3uP/p3RZN/c=">AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w=</latexit><latexit sha1_base64="USoNv0fCIvi1lLEl3uP/p3RZN/c=">AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w=</latexit><latexit sha1_base64="USoNv0fCIvi1lLEl3uP/p3RZN/c=">AAAB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbQM2FgmYD4gOcLeZi9Zsrt37M4JMeQX2GpvJ7b+G1t/iZvkCk18YeDhnRlmeKNUCou+/+UVNja3tneKu6W9/YPDo/LxScsmmWG8yRKZmE5ELZdC8yYKlLyTGk5VJHk7Gt/N++1HbqxI9ANOUh4qOtQiFoyisxrYL1f8qr8QWYcghwrkqvfL371BwjLFNTJJre0GforhlBoUTPJZqZdZnlI2pkPedaip4jacLh6dkQvnDEicGFcaycL9vTGlytqJitykojiyq725+W8vUiuXMb4Np0KnGXLNlofjTBJMyDwFMhCGM5QTB5QZ4X4nbEQNZeiyKrlQgtUI1qF1VQ0cN64rNT+PpwhncA6XEMAN1OAe6tAEBhye4QVevSfvzXv3PpajBS/fOYU/8j5/ABK4k4w=</latexit>
a b c
0 2 4 6 10
Figure 4: Gradients in networks with batch normalization quickly achieve dynamical equilibrium. Plots of the relative magnitudes of (a) the weights (b) the gradients of the loss with respect to the pre-activations and (c) the gradients of the loss with respect to the weights for rectified linear networks of varying depths during the first 10 steps of training. Colors show step number from 0 (black) to 10 (green).
held fixed but batch statistics are computed over subsets of size B. This removes subtle gradient fluctuation effects noted in Smith & Le (2018). In (c) we do the same experiment with RMSProp and in (d) we train the networks on CIFAR10. In all cases we observe a nearly identical trainable region.
It is counter intuitive that training can occur at intermediate depths, from 10 to 50 layers, where there is significant gradient explosion. To gain insight into the behavior of the network during learning we record the magnitudes of the weights, the gradients with respect to the pre-activations, and the gradients with respect to the weights for the first 10 steps of training for networks of different depths. The result of this experiment is shown in Fig. 4. Here we see that before learning, as expected, the norm of the weights is constant and independent of layer while the gradients feature exponential explosion. However, we observe that two related phenomena occur after a single step of learning: the weights grow exponentially in the depth and the magnitude of the gradients are stable up to some threshold after which they vanish exponentially in the depth. This is as the result of the scaling property of batchnorm, where B′φ(αh) = α−1B′φ(h): The first-step gradients dominate the weights due to gradient explosion, hence the exponential growth in weight norms, and thereafter, the gradients are scaled down commensurately. Thus, it seems that although the gradients of batch normalized networks at initialization are ill-conditioned, the gradients appear to quickly reach a stable dynamical equilibrium. While this appears to be beneficial for shallower networks, in deeper ones, the relative gradient vanishing can in fact be so severe as to cause lower layers to mostly stay constant during training. Aside from numerical issues, this seems to be the primary mechanism through which gradient explosion causes training problems for networks deeper than 50 layers.
As discussed in the theoretical exposition above, batch normalization necessarily features exploding gradients for any nonlinearity that converges to a BSB1 fixed point. We performed a number of experiments exploring different ways of ameliorating this gradient explosion. These experiments
a b c d 0.1 1.0
Figure 5: Three techniques for counteracting gradient explosion. Test accuracy on MNIST as a function of different hyperparameters along with theoretical predictions (white dashed line) for the maximum trainable depth. (a) tanh network changing the overall scale of the pre-activations, here γ → 0 corresponds to the linear regime. (b) Rectified linear network changing the mean of the preactivations, here β → ∞ corresponds to the linear regime. (c,d) tanh and rectified linear networks respectively as a function of , here we observe a well defined phase transition near ∼ 1. Note that in the case of rectified linear activations we use β = 2 so that the function is locally linear about 0. We also find initializing β and/or setting > 0 having positive effect on VGG19 with batchnorm. See Figs. 8 and 9.
are shown in Fig. 5 with theoretical predictions for the maximum trainable depth overlaid; in all cases we see exceptional agreement. In Fig. 5 (a,b) we explore two different ways of tuning the degree to which activation functions in a network are nonlinear. In Fig. 5 (a) we tune γ ∈ [0, 2] for networks with tanh-activations and note that in the γ → 0 limit the function is linear. In Fig. 5 (b) we tune β ∈ [0, 2] for networks with rectified linear activations and we note, similarly, that in the β → ∞ limit the function is linear. As expected, we see the maximum trainable depth increase significantly with decreasing γ and increasing β. In Fig. 5 (c,d) we vary for tanh and rectified linear networks respectively. In both cases, we observe a critical point at large where gradients do not explode and very deep networks are trainable.
5 CONCLUSION
In this work we have presented a theory for neural networks with batch normalization at initialization. In the process of doing so, we have uncovered a number of counterintuitive aspects of batch normalization and – in particular – the fact that at initialization it unavoidably causes gradients to explode with depth. We have introduced several methods to reduce the degree of gradient explosion, enabling the training of significantly deeper networks in the presence of batch normalization. Finally, this work paves the way for future work on more advanced, state-of-the-art, network architectures and topologies.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv:1607.06450 [cs, stat], July 2016. URL http://arxiv.org/abs/1607.06450.
David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The Shattered Gradients Problem: If resnets are the answer, then what is the question? In PMLR, pp. 342–350, July 2017. URL http://proceedings.mlr.press/v70/ balduzzi17b.html.
Johan Bjorck, Carla Gomes, and Bart Selman. Understanding Batch Normalization. June 2018. URL https://arxiv.org/abs/1806.02375.
Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 873–882, Stockholmsmssan,
Stockholm Sweden, 10–15 Jul 2018. PMLR. URL http://proceedings.mlr.press/ v80/chen18i.html.
Q. Chen and R. Wu. CNN Is All You Need. ArXiv e-prints, December 2017.
Youngmin Cho and Lawrence K. Saul. Kernel methods for deep learning. In Advances in neural information processing systems, pp. 342–350, 2009. URL http://papers.nips.cc/ paper/3628-kernel-methods-for-deep-learning.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity. arXiv:1602.05897 [cs, stat], February 2016. URL http://arxiv.org/abs/1602.05897.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id= H1-nGgWC-.
Adrià Garriga-Alonso, Laurence Aitchison, and Carl Edward Rasmussen. Deep convolutional networks as shallow gaussian processes. International Conference on Learning Representations, 2019.
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, and Jeffrey Pennington. Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs. arXiv e-prints, art. arXiv:1901.08987, Jan 2019.
Igor Gitman and Boris Ginsburg. Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification. arXiv:1709.08145 [cs], September 2017. URL http://arxiv.org/abs/1709.08145.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. arXiv preprint arXiv:1803.01719, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs], December 2015. URL http://arxiv.org/abs/ 1512.03385.
Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167 [cs], February 2015. URL http: //arxiv.org/abs/1502.03167.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Ming Zhou, Klaus Neymeyr, and Thomas Hofmann. Towards a Theoretical Understanding of Batch Normalization. arXiv:1805.10694 [cs, stat], May 2018. URL http://arxiv.org/abs/1805.10694.
Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, pp. 396–404, 1990.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165, 2017.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Understanding Regularization in Batch Normalization. arXiv:1809.00846 [cs, stat], September 2018. URL http://arxiv.org/ abs/1809.00846.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. International Conference of Learning Representations, 2019. URL https://openreview.net/forum?id=B1g30j0qF7.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Advances in neural information processing systems, pp. 4785–4795, 2017.
George Philipp and Jaime G. Carbonell. The Nonlinearity Coefficient - Predicting Overfitting in Deep Neural Networks. arXiv:1806.00179 [cs, stat], May 2018. URL http://arxiv.org/ abs/1806.00179.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. arXiv:1606.05340 [cond-mat, stat], June 2016. URL http://arxiv.org/abs/1606.05340.
Tim Salimans and Diederik P. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. February 2016. URL https://arxiv.org/ abs/1602.07868.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift). arXiv:1805.11604 [cs, stat], May 2018. URL http://arxiv.org/abs/1805.11604.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation. arXiv:1611.01232 [cs, stat], November 2016. URL http://arxiv.org/abs/ 1611.01232.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient descent. 2018.
P.K. Suetin. Ultraspherical polynomials - Encyclopedia of Mathematics. URL https://www. encyclopediaofmath.org/index.php/Ultraspherical_polynomials.
Eric W. Weisstein. Gegenbauer Polynomial. URL http://mathworld.wolfram.com/ GegenbauerPolynomial.html.
Christopher KI Williams. Computing with infinite networks. In Advances in neural information processing systems, pp. 295–301, 1997.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5393–5402, Stockholmsmssan, Stockholm Sweden, 10–15 Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/xiao18a.html.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760, 2019.
Greg Yang and Samuel S. Schoenholz. Meanfield Residual Network: On the Edge of Chaos. In Advances in neural information processing systems, 2017.
Greg Yang and Samuel S Schoenholz. Deep mean field theory: Layerwise variance and width variation as methods to control gradient explosion. 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition.
A VGG19 WITH BATCHNORM ON CIFAR100
Even though at initialization time batchnorm causes gradient explosion, after the first few epochs, the relative gradient norms ‖∇θL‖/‖θ‖ for weight parameters θ = W or BN scale parameter θ = γ, equilibrate to about the same magnitude. See Fig. 7.
B GRADIENT INDEPENDENCE ASSUMPTION
Following prior literature Schoenholz et al. (2016); Yang & Schoenholz (2017); Xiao et al. (2018), in this paper, in regards to computations involving backprop, we assume Assumption 2. During backpropagation, whenever we multiply by WT for some weight matrix W , we multiply by an iid copy instead.
As in these previous works, we find excellent agreement between computations made under this assumption and the simulations (see Fig. 1). Yang (2019) in fact recently rigorously justified this assumption as used in the computation of moments, for a wide variety of architectures, like multilayer perceptron (Schoenholz et al., 2016), residual networks (Yang & Schoenholz, 2017), and convolutional networks (Xiao et al., 2018) studied previously, but without batchnorm. The reason that their argument does not extend to batchnorm is because of the singularity in its Jacobian at 0. However, as observed in our experiments, we expect that a proof can be found to extend Yang & Schoenholz (2017)’s work to batchnorm.
C NOTATIONS
Definition C.1. Let SB be the space of PSD matrices of size B × B. Given a measurable function Φ : RB → RA, define the integral transform VΦ : SB → SA by VΦ(Σ) = E[Φ(h)⊗2 : h ∼ N (0,Σ)].3 When φ : R → R and B is clear from context, we also write Vφ for V applied to the function acting coordinatewise by φ.
Definition C.2. For any φ : R → R, let Bφ : RB → RB be batchnorm (applied to a batch of neuronal activations) followed by coordinatewise applications of φ, Bφ(x)j = φ ( (xj −Avg x) / √ 1 B ∑B i=1 (xi −Avg x) 2 ) (here Avg x = 1B ∑B i=1 xi). When φ = id we will
also write B = Bid. Definition C.3. Define the matrix GB = I − 1B11T . Let SGB be the space of PSD matrices of size B × B with zero mean across rows and columns, SGB := {Σ ∈ SB : GBΣGB = Σ} = {Σ ∈ SB : Σ1 = 0}.
3This definition of V absorbs the previous definitions of V and W in Yang & Schoenholz (2017) for the scalar case
When B is clear from context, we will suppress the subscript/superscript B. In short, for h ∈ RB , Gh zeros the sample mean of h. G is a projection matrix to the subspace of vectors h ∈ RB of zero coordinate sum. With the above definitions, we then have Bφ(h) = φ( √ BGh/‖Gh‖).
We use Γ to denote the Gamma function, P (a, b) := Γ(a + b)/Γ(a) to denote the Pochhammer symbol, and Beta(a, b) = Γ(a)Γ(b)Γ(a+b) to denote the Beta function. We use 〈·, ·〉 to denote the dot product for vectors and trace inner product for matrices. Sometimes when it makes for simpler notation, we will also use · for dot product. We adopt the matrix convention that, for a multivariate function f : Rn → Rm, the Jacobian is
df dx =  ∂f1 ∂x1 ∂f1 ∂x2 · · · ∂f1∂xn ∂f2 ∂x1 ∂f2 ∂x2 · · · ∂f2∂xn ... ... . . .
... ∂fm ∂x1 ∂fm ∂x2 · · · ∂fm∂xn  , so that f(x + ∆x) = dfdx∆x + o(∆x), where x and ∆x are treated as column vectors. In what follows we further abbreviate B′φ(x) = dBφ(z) dz ∣∣ z=x ,B′φ(x)∆x = dBφ(z) dz ∣∣ z=x ∆x.
D A GUIDE TO THE REST OF THE APPENDIX
As discussed in the main text, we are interested in several closely related dynamics induced by batchnorm in a fully-connected network with random weights. One is the forward propagation equation Eq. (43)
Σl = VBφ(Σ l−1)
studied in Appendix F. Another is the backward propagation equation Eq. (51)
Πl = VB′φ(Σ ∗)†{Πl+1}
studied in Appendix G. We will also study their generalizations to the simultaneous propagation of multiple batches in Appendices H and I.
In general, we discover several ways to go about such analyses, each with their own benefits and drawbacks:
• The Laplace Method (Appendix E.1): If the nonlinearity φ is positive homogeneous, we can use Schwinger’s parametrization to turn VBφ and VB′φ into nicer forms involving only 1 or 2 integrals, as mentioned in the main text. All relevant quantities, such as eigenvalues of the backward dynamics, can be further simplified to closed forms. Lemma E.2 gives the master equation for the Laplace method. Because the Laplace method requires φ to be positive homogeneous to apply, we review relevant results of such functions in Appendices E.1.1 and E.1.2.
• The Fourier Method (Appendix E.2): For polynomially bounded and continuous nonlinearity φ (for the most general set of conditions, see Thm E.25), we can obtain similar simplifications using the Fourier expansion of the delta function, with the penalty of an additional complex integral.
• Spherical Integration (Appendix E.3): Batchnorm can naturally be interpreted as a linear projection followed by projection to a sphere of radius √ B. Assuming that the forward
dynamics converges to a BSB1 fixed point, one can express VBφ and VB′φ as spherical integrals, for all measurable φ. One can reduce this (B−1)-dimensional integral, in spherical angular coordinates, into 1 or 2 dimensions by noting that the integrand depending on φ only depends on 1 or 2 angles. Thus this method is very suitable for numerical evaluation of quantities of interest for general φ.
• The Gegenbauer Method (Appendix E.4): Gegenbauer polynomials are orthogonal polynomials under the weight (1 − x2)α for some α. They correspond to a special type of
spherical harmonics called zonal harmonics (Defn E.44) that has the very useful reproducing property (Fact E.46). By expressing the nonlinearity φ in this basis, we can see easily that the relevant eigenvalues of the forward and backward dynamics are ratios of two quadratic forms of φ’s Gegenbauer coefficients. In particular, for the largest eigenvalue of the backward dynamics, the two quadratic forms are both diagonal, and in a way that makes apparent the necessity of gradient explosion (under the BSB1 fixed point assumption). In regards to numerical computations, the Gegenbauer method has the benefit of only requiring 1-dimensional integrals to obtain the coefficients, but if φ has slowlydecaying coefficients, then a large number of such integrals may be required to get an accurate answer.
In each of the following sections, we will attempt to conduct an analysis with each method when it is feasible.
When studying the convergence rate of the forward dynamics and the gradient explosion of the backward dynamics, we encounter linear operators on matrices that satisfy an abundance of symmetries, called ultrasymmetric operators. We study these operators in Appendix E.5 and contribute several structural results Thms E.61, E.62, E.72 and E.72 on their eigendecomposition that are crucial to deriving the asymptotics of the dynamics.
All of the above will be explained in full in the technique section Appendix E. We then proceed to study the forward and backward dynamics in detail (Appendices F to I), now armed with tools we need. Up to this point, we only consider the dynamics assuming Eq. (43) converges to a BSB1 fixed point, and B ≥ 4. In Appendix J we discuss what happens outside this regime, and in Appendix K we show our current understanding of BSB2 fixed point dynamics.
Note that the backward dynamics studied in these sections is that of the gradient with respect to the hidden preactivations. Nevertheless, we can compute the moments of the weight gradients from this, which is of course what is eventually used in gradient descent. We do so in the final section Appendix L.
Main Technical Results Corollary F.3 establishes the global convergence of Eq. (43) to BSB1 fixed point for φ = id (as long as the initial covariance is nondegenerate), but we are not able to prove such a result for more general φ. We thus resort to studying the local convergence properties of BSB1 fixed points.
First, Thms F.5, F.8 and F.13 respectively compute the BSB1 fixed point from the perspectives of spherical integration, the Laplace method, and the Gegenbauer method. Thms F.22 and F.24 give the Gegenbauer expansion of the BSB1 local convergence rate, and Thm F.33 gives a more succinct form of the same for positive homogeneous φ. Appendix F.3.1 discusses how to do this computation via spherical integration.
Next, we study the gradient dynamics, and Thms G.5 and G.12 yield the gradient explosion rates with the Gegenbauer method and the Laplace method (for positive homogeneous φ), respectively. Appendix G.1.1 discusses how to compute this for general φ using spherical integration.
We then turn to the dynamics of cross batch covariances. Thm H.3 gives the form of the cross batch fixed point via spherical integration and the Gegenbauer method, and Thm H.6 yields a more specific form for positive homogeneous φ via the Laplace method. The local convergence rate to this fixed point is given by Thms H.13 and H.14, respectively using the Gegenbauer method and the Laplace method. Finally, the correlation between gradients of the two batches is shown to decrease exponentially in Corollary I.6 using the Gegenbauer method, and the decay rate is given succinctly for positive homogeneous φ in Thm I.3.
E TECHNIQUES
We introduce the key techniques and tools in our analysis in this section.
E.1 LAPLACE METHOD
As discussed above, the Laplace method is useful for deriving closed form expressions for positive homogeneous φ. The key insight here is to apply Schwinger parametrization to deal with normalization. Lemma E.1 (Schwinger parametrization). For z > 0 and c > 0,
c−z = Γ(z)−1 ∫ ∞
0
xz−1e−cx dx
The following is the key lemma in the Laplace method.
Lemma E.2 (The Laplace Method Master Equation). For A,A′ ∈ N, let f : RA → RA′ and let k ≥ 0. Suppose ‖f(y)‖ ≤ h(‖y‖) for some nondecreasing function h : R≥0 → R≥0 such that E[h(r‖y‖) : y ∈ N (0, IA)] exists for every r ≥ 0. Define ℘(Σ) := E[‖y‖−2kf(y) : y ∼ N (0,Σ)]. Then on {Σ ∈ SA : rank Σ > 2k}, ℘(Σ) is well-defined and continuous, and furthermore satisfies
℘(Σ) = Γ(k)−1 ∫ ∞
0
ds sk−1 det(I + 2sΣ)−1/2 E[f(y) : y ∼ N (0,Σ(I + 2sΣ)−1)] (25)
Proof. ℘(Σ) is well-defined for full rank Σ because the ‖y‖−2k singularity at y = 0 is Lebesgueintegrable in a neighborhood of 0 in dimension A > 2k.
We prove Eq. (25) in the case when Σ is full rank and then apply a continuity argument.
Proof of Eq. (25) for full rank Σ. First, we will show that we can exchange the order of integration∫ ∞ 0 ds ∫ RA dy sk−1f(y)e− 1 2y T (Σ−1+2sI)y = ∫ RA dy ∫ ∞ 0 ds sk−1f(y)e− 1 2y T (Σ−1+2sI)y
by Fubini-Tonelli’s theorem. Observe that (2π)−A/2 det Σ−1/2 ∫ ∞
0 ds sk−1 ∫ RA dy ‖f(y)‖e− 12yT (Σ−1+2sI)y
= ∫ ∞ 0 ds sk−1 det(Σ(Σ−1 + 2sI))−1/2 E[‖f(y)‖ : y ∼ N (0, (Σ−1 + 2sI)−1)]
= ∫ ∞ 0 ds sk−1 det(I + 2sΣ)−1/2 E[‖f(y)‖ : y ∼ N (0,Σ(I + 2sΣ)−1)]
For λ = ‖Σ‖2,
‖f( √ Σ(I + 2sΣ)−1y)‖ ≤ h(‖ √ Σ(I + 2sΣ)−1y‖ ≤ h (∥∥∥∥∥ √ λ 1 + 2sλ y ∥∥∥∥∥ ) ≤ h( √ λ‖y‖)
Because E[h( √ λ‖y‖) : y ∈ N (0, I)] exists by assumption,
E[‖f(y)‖ : y ∼ N (0,Σ(I + 2sΣ)−1)] = E[‖f( √ Σ(I + 2sΣ)−1y)‖ : y ∼ N (0, I)]
→ E[‖f(0)‖ : y ∼ N (0, I)] = ‖f(0)‖
as s → ∞, by dominated convergence with dominating function h( √ λ‖y‖)e− 12‖y‖2(2π)−A/2. By the same reasoning, the function s 7→ E[‖f(y)‖ : y ∼ N (0,Σ(I + 2sΣ)−1)] is continuous. In particular this implies that sup0≤s≤∞ E[‖f(y)‖ : y ∼ N (0,Σ(I + 2sΣ)−1)] <∞. Combined with the fact that det(I + 2sΣ)−1/2 = Θ(s−A/2) as s→∞,∫ ∞
0
ds sk−1 det(I + 2sΣ)−1/2 E[‖f(y)‖ : y ∼ N (0,Σ(I + 2sΣ)−1)]
≤ ∫ ∞
0
ds Θ(sk−1−A/2)
which is bounded by our assumption that A/2 > k. This shows that we can apply Fubini-Tonelli’s theorem to allow exchanging order of integration.
Thus,
E[‖y‖−2kf(y) : y ∼ N (0,Σ)] = E[f(y) ∫ ∞
0
ds Γ(k)−1sk−1e−‖y‖ 2s : y ∼ N (0,Σ)]
= (2π)−A/2 det Σ−1/2 ∫ dy e− 1 2y TΣ−1yf(y) ∫ ∞ 0 ds Γ(k)−1sk−1e−‖y‖ 2s
= (2π)−A/2Γ(k)−1 det Σ−1/2 ∫ ∞
0
ds sk−1 ∫ dy f(y)e− 1 2y T (Σ−1+2sI)y
(by Fubini-Tonelli) = Γ(k)−1 ∫ ∞
0
ds sk−1 det(Σ(Σ−1 + 2sI))−1/2 E[f(y) : y ∼ N (0, (Σ−1 + 2sI)−1)]
= Γ(k)−1 ∫ ∞
0
ds sk−1 det(I + 2sΣ)−1/2 E[f(y) : y ∼ N (0,Σ(I + 2sΣ)−1)]
Domain and continuity of ℘(Σ). The LHS of Eq. (25), ℘(Σ), is defined and continuous on rank Σ/2 > k. Indeed, if Σ = MICMT , where M is a full rank A × C matrix with rank Σ = C ≤ A, then
E[‖y‖−2k‖f(y)‖ : y ∼ N (0,Σ)] = E[‖Mz‖−2k‖f(Mz)‖ : z ∼ N (0, IC)].
This is integrable in a neighborhood of 0 iff C > 2k, while it’s always integrable outside a ball around 0 because ‖f‖ by itself already is. So ℘(Σ) is defined whenever rank Σ > 2k. Its continuity can be established by dominated convergence.
Proof of Eq. (25) for rank Σ > 2k. Observe that det(I + 2sΣ)−1/2 is continuous in Σ and, by an application of dominated convergence as in the above, E[f(y) : y ∼ N (0,Σ(I + 2sΣ)−1)] is continuous in Σ. So the RHS of Eq. (25) is continuous in Σ whenever the integral exists. By the reasoning above, E[f(y) : y ∼ N (0,Σ(I + 2sΣ)−1)] is bounded in s and det(I + 2sΣ)−1/2 = Θ(s− rank Σ/2), so that the integral exists iff rank Σ/2 > k.
To summarize, we have proved that both sides of Eq. (25) are defined and continous for rank Σ > 2k. Because the full rank matrices are dense in this set, by continuity Eq. (25) holds for all rank Σ > 2k.
If φ is degree-α positive homogeneous, i.e. φ(ru) = rαφ(u) for any u ∈ R, r ∈ R+, we can apply Lemma E.2 with k = α,
VBφ(Σ) = E[φ( √ BGh/‖Gh‖)⊗2 : h ∼ N (0,Σ)] = Bα E[φ(Gh)⊗2/‖Gh‖2α : h ∼ N (0,Σ)]
= Bα E[φ(z)⊗2/‖z‖2α : z ∼ N (0, GΣG)] = BαΓ(α)−1 ∫ ∞
0
ds sα−1 det(I + 2sΣG)−1/2Vφ(G(I + 2sΣG) −1ΣG).
If φ = relu, then
Vφ(Σ)ij = { 1 2Σii if i = j 1 2J1(Σij/ √ ΣiiΣjj) √ ΣiiΣjj otherwise
and, more succinctly, Vφ(Σ) = D1/2Vφ(D−1/2ΣD−1/2)D1/2 where D = Diag(Σ). Here J1(c) := 1 π ( √ 1− c2 + (π − arccos(c))c) (Cho & Saul, 2009).
Matrix simplification. We can simplify the expression G(I + 2sΣG)−1ΣG, leveraging the fact that G is a projection matrix. Definition E.3. Let e be an B × (B − 1) matrix whose columns form an orthonormal basis of imG := {Gv : v ∈ RB} = {w ∈ RB : ∑i wi = 0}. Then the B × B matrix ẽ = (e|B−1/21) is an orthogonal matrix. For much of this paper e can be any such basis, but at certain sections we will consider specific realizations of e for explicit computation.
From easy computations it can be seen that G = ẽ ( IB−1 0
0 0
) ẽ T . Suppose Σ = ẽ ( Σ v vT a ) ẽ T
where Σ is (B − 1) × (B − 1), v is a column vector and a is a scalar. Then Σ = eTΣe and ΣG = ẽ ( Σ 0 vT 0 ) ẽ T is block lower triangular, and
(I + 2sΣG)−1 = ẽ
( (I + 2sΣ )
−1 0 ∗ 1
) ẽ T
(I + 2sΣG)−1ΣG = ẽ
( (I + 2sΣ )
−1Σ 0 ∗ 0
) ẽ T
G(I + 2sΣG)−1ΣG = ẽ
( (I + 2sΣ )
−1Σ 0 0 0
) ẽ T
= e(I + 2sΣ ) −1Σ e T
= GΣG(I + 2sGΣG)−1
=: ΣG(I + 2sΣG)−1
where Definition E.4. For any matrix Σ, write ΣG := GΣG.
Similarly, det(IB + 2sΣG) = det(IB−1 + 2sΣ ) = det(IB + 2sΣG). So, altogether, we have Theorem E.5. Suppose φ : R → R is degree-α positive homogeneous. Then for any B × (B − 1) matrix e whose columns form an orthonormal basis of imG := {Gv : v ∈ RB} = {w ∈ RB :∑ i wi = 0}, with Σ = eTΣe,
VBφ(Σ) = B αΓ(α)−1 ∫ ∞ 0 ds sα−1 det(I + 2sΣ ) −1/2Vφ(e(I + 2sΣ ) −1Σ ) −1 e T ) (26)
= BαΓ(α)−1 ∫ ∞
0
ds sα−1 det(I + 2sΣG)−1/2Vφ((I + 2sΣ G)−1ΣG)−1) (27)
Now, in order to use Laplace’s method, we require φ to be positive homogeneous. What do these functions look like? The most familiar example to a machine learning audience is most likely ReLU. It turns out that 1-dimensional positive homogeneous functions can always be described as a linear combination of powers of ReLU and its reflection across the y-axis. Below, we review known facts about these functions and their integral transforms, starting with the powers of ReLU in Appendix E.1.1 and then onto the general case in Appendix E.1.2.
E.1.1 α-RELU
Recall that α-ReLUs (Yang & Schoenholz, 2017) are, roughly speaking, the αth power of ReLU. Definition E.6. The α-ReLU function ρα : R → R sends x 7→ xα when x > 0 and x 7→ 0 otherwise.
This is a continuous function for α > 0 but discontinuous at 0 for all other α.
We briefly review what is currently known about the V and W transforms of ρα (Cho & Saul, 2009; Yang & Schoenholz, 2017). Definition E.7. For any α > − 12 , define cα = 1√π2α−1Γ(α+ 1 2 ).
When considering only 1-dimensional Gaussians, Vρα is very simple. Proposition E.8. If α > − 12 , then for any q ∈ S1 = R≥0, Vρα(q) = cαqα
To express results of Vρα on SB for higher B, we first need the following Definition E.9. Define
Jα(θ) := 1
2πcα (sin θ)2α+1Γ(α+ 1) ∫ π/2 0
dη cosα η
(1− cos θ cos η)1+α and Jα(c) = Jα(arccos c) for α > −1/2.
Then Proposition E.10. For any Σ ∈ SB , let D be the diagonal matrix with the same diagonal as Σ. Then
Vρα(Σ) = cαD α/2Jα(D −1/2ΣD−1/2)Dα/2
where Jα is applied entrywise.
For example, Jα and Jα for the first few integral α are
J0(θ) = π − θ π J1(θ) = sin θ + (π − θ) cos θ
π
J2(θ) = 3 sin θ cos θ + (π − θ)(1 + 2 cos2 θ)
3π
J0(c) = π − arccos c
π
J1(c) =
√ 1− c2 + (π − arccos c)c
π
J2(c) = 3c √
1− c2 + (π − arccos c)(1 + 2c2) 3π
One can observe very easily that (Daniely et al., 2016; Yang & Schoenholz, 2017) Proposition E.11. For each α > −1/2, Jα(c) is an increasing and convex function on c ∈ [0, 1], and is continous on c ∈ [0, 1] and smooth on c ∈ (0, 1). Jα(1) = 1, Jα(0) = 12√π Γ(α2 + 1 2 ) 2
Γ(α+ 12 ) , and
Jα(−1) = 0.
Yang & Schoenholz (2017) also showed the following fixed point structure Theorem E.12. For α ∈ [1/2, 1), Jα(c) = c has two solutions: an unstable solution at 1 (”unstable” meaning J′α(1) > 1) and a stable solution in c ∗ ∈ (0, 1) (”stable” meaning J′α(c∗) < 1).
The α-ReLUs satisfy very interesting relations amongst themselves. For example, Lemma E.13. Suppose α > 1. Then
Jα(θ) = cos θJα−1(θ) + (α− 1)2(2α− 1)−1(2α− 3)−1 sin2 θJα−2(θ) Jα(c) = cJα−1(c) + (α− 1)2(2α− 1)−1(2α− 3)−1(1− c2)Jα−2(c)
In additional, surprisingly, one can use differentiation to go from α to α+ 1 and from α to α− 1! Proposition E.14 (Yang & Schoenholz (2017)). Suppose α > 1/2. Then
J ′α(θ) = −α2(2α− 1)−1Jα−1(θ) sin θ J′α(c) = α
2(2α− 1)−1Jα−1(c) so that
Jα−n(c) =  α∏ β=α−n+1 β−2(2β − 1)  (∂/∂c)nJα(c)
We have the following from Cho & Saul (2009) Proposition E.15 (Cho & Saul (2009)). For all α ≥ 0 and integer n ≥ 1
Jn+α(c) = cα cn+α (1− c2)n+α+ 12 (∂/∂c)n(Jα(c)/(1− c2)α+ 1 2 )
= α+n−1∏ β=α (2β + 1) −1 (1− c2)n+α+ 12 (∂/∂c)n(Jα(c)/(1− c2)α+ 12 ) This implies in particular that we can obtain J′α from Jα and Jα+1. Proposition E.16. For all α ≥ 0,
J′α(c) = (2α+ 1)(1− c2)−1(J1+α(c)− cJα(c)))
Proof.
J1+α(c) = cα c1+α (1− c2)1+α+ 12 (∂/∂c)(Jα(c)/(1− c2)α+ 1 2 )
= (2α+ 1)−1(1− c2)α+3/2(J′α(c)/(1− c2)α+1/2 + 2c(α+ 1/2)Jα(c)/(1− c2)α+3/2) = (2α+ 1)−1J′α(c)(1− c2) + cJα(c)
J′α(c) = (2α+ 1)(1− c2)−1(J1+α(c)− cJα(c))
Note that we can also obtain this via Lemma E.13 and Proposition E.14.
E.1.2 POSITIVE-HOMOGENEOUS FUNCTIONS IN 1 DIMENSION
Suppose for some α ∈ R, φ : R → R is degree α positive-homogeneous, i.e. φ(rx) = rαφ(x) for any x ∈ R, r > 0. The following simple lemma says that we can always express φ as linear combination of powers of α-ReLUs. Proposition E.17. Any degree α positive-homogeneous function φ : R → R with φ(0) = 0 can be written as x 7→ aρα(x)− bρα(−x).
Proof. Take a = φ(1) and b = φ(−1). Then positive-homogeneity determines the value of φ on R \ {0} and it coincides with x 7→ aρα(x)− bρα(−x).
As a result we can express the V and W transforms of any positive-homogeneous function in terms of those of α-ReLUs. Proposition E.18. Suppose φ : R → R is degree α positive-homogeneous. By Proposition E.17, φ restricted to R \ {0} can be written as x 7→ aρα(x)− bρα(−x) for some a and b. Then for any PSD 2× 2 matrix M ,
Vφ(M)11 = (a 2 + b2)Vρα(M)11 = cα(a 2 + b2)Mα11
Vφ(M)22 = (a 2 + b2)Vρα(M)22 = cα(a 2 + b2)Mα22
Vφ(M)12 = Vφ(M)21 = (a 2 + b2)Vρα(M)12 − 2abVρα(M ′)12
= cα(M11M22) α/2((a2 + b2)Jα(M12/ √ M11M22)− 2abJα(−M12/ √ M11M22))
where M ′ := (
1 0 0 −1
) M ( 1 0 0 −1 ) .
Proof. We directly compute, using the expansion of φ into ραs: Vφ(M)11 = E[φ(x)2 : x ∼ N (0,M11)]
= E[a2ρα(x)2 + b2ρα(−x)2 + 2abρα(x)ρα(−x)] = a2 E[ρα(x)2] + b2 E[ρα(−x)2] = (a2 + b2)E[ρα(x)2 : x ∼ N (0,M11)] (28) = cα(a 2 + b2)Mα11
where in Eq. (28) we used negation symmetry of centered Gaussians. The case of Vφ(M)22 is similar.
Vφ(M)12 = E[φ(x)φ(y) : (x, y) ∼ N (0,M)] = E[a2ρα(x)ρα(y) + b2ρα(−x)ρα(−y)− abρα(x)ρα(−y)− abρα(−x)ρα(y)] = (a2 + b2)Vρα(M)12 − 2abVρα(M ′)12 = cα(M11M22) α/2((a2 + b2)Jα(M12/ √ M11M22)− 2abJα(−M12/ √ M11M22))
where in the last equation we have applied Proposition E.10.
This then easily generalizes to PSD matrices of arbitrary dimension: Corollary E.19. Suppose φ : R → R is degree α positive-homogeneous. By Proposition E.17, φ restricted to R\{0} can be written as x 7→ aρα(x)− bρα(−x) for some a and b. Let Σ ∈ SB . Then
Vφ(Σ) = cαD α/2Jφ(D −1/2ΣD−1/2)Dα/2
where Jφ is defined below and is applied entrywise. Explicitly, this means that for all i,
Vφ(Σ)ii = cαΣ α iiJφ(1) Vφ(Σ)ij = cαJφ(Σij/ √ ΣiiΣjj)Σ α/2 ii Σ α/2 jj
Definition E.20. Suppose φ : R → R is degree α positive-homogeneous. By Proposition E.17, φ restricted to R \ {0} can be written as x 7→ aρα(x) − bρα(−x) for some a and b. Define Jφ(c) := (a 2 + b2)Jα(c)− 2abJα(−c).
Let us immediately make the following easy but important observations.
Proposition E.21. Jφ(1) = a2 + b2 and Jφ(0) = (a2 + b2 − 2ab)Jα(0) = (a− b)2 12√π Γ(α2 + 1 2 ) 2
Γ(α+ 12 ) ,
Proof. Use the fact that Jα(−1) = 0 and Jα(0) = 12√π Γ(α2 + 1 2 ) 2
Γ(α+ 12 ) by Proposition E.11.
As a sanity check, we can easily compute that Jid(c) = 2J1(c) − 2J1(−c) = 2c because id(x) = relu(x)−relu(−x). By Corollary E.19 and c1 = 12 , this recovers the obvious fact that V(id)(Σ) = Σ. We record the partial derivatives of Vφ. Proposition E.22. Let φ be positive homogeneous of degree α. Then for all i with Σii 6= 0,
∂Vφ(Σ)ii ∂Σii = cααΣ α−1 ii Jφ(1)
For all i 6= j with Σii,Σjj 6= 0, ∂Vφ(Σ)ii ∂Σij = 0
∂Vφ(Σ)ij ∂Σij = cαΣ (α−1)/2 ii Σ (α−1)/2 jj J ′ φ(cij)
∂Vφ(Σ)ij ∂Σii = 1 2 cαΣ 1 2 (α−2) ii Σ 1 2α jj (αJφ(cij)− cijJ′φ(cij))
where cij = Σij/ √ ΣiiΣjj and J′φ denotes its derivative.
Proposition E.23. If φ(c) = aρα(c)− bρα(−c) with α > 1/2 on R \ {0}, then J′φ(c) = (2α− 1)−1Jφ′(c)
= (2α+ 1)(1− c2)−1((a2 + b2)Jα+1(c) + 2abJα+1(−c))− cJφ(c)
J′φ(0) = (a+ b) 2 1√
π
Γ(α2 + 1) 2
Γ(α+ 12 )
Proof. We have φ′(c) = aαρα−1(c) + bαρα−1(−c). On the other hand, J′φ = (a2 + b2)J′α(c) + 2abJ′α(−c) which by Proposition E.14 is α2(2α − 1)−1((a2 + b2)Jα−1(c) + 2abJα−1(−c)) = (2α− 1)−1Jφ′(c). This proves the first equation. With Proposition E.16,
J′φ(c) = (a 2 + b2)J′α(c) + 2abJ ′ α(−c)
= (a2 + b2)(2α+ 1)(1− c2)−1(Jα+1(c)− cJα(c)) + 2ab(2α+ 1)(1− c2)−1(Jα+1(−c) + cJα(−c))
= (2α+ 1)(1− c2)−1((a2 + b2)Jα+1(c) + 2abJα+1(−c))− cJφ(c)
This gives the second equation.
Expanding Jα+1(0) With Proposition E.11, we get
J′φ(0) = (2α+ 1)((a 2 + b2)Jα+1(0) + 2abJα+1(0))
= (2α+ 1)(a+ b)2 1
2 √ π
Γ(α2 + 1) 2
Γ(α+ 32 )
Unpacking the definition of Γ(α+ 32 ) then yields the third equation.
In general, we can factor diagonal matrices out of Vφ.
Proposition E.24. For any Σ ∈ SB , D any diagonal matrix, and φ positive-homogeneous with degree α,
Vφ(DΣD) = D αVφ(Σ)D α
Proof. For any i,
Vφ(DΣD)ii = E[φ(x)2 : x ∼ N (0, D2iiΣii)] = E[φ(Diix)2 : x ∼ N (0,Σii)] = E[D2αii φ(x)2 : x ∼ N (0,Σii)] = D2αii Vφ(Σ)ii
For any i 6= j,
Vφ(DΣD)ij = E[φ(x)φ(y) : (x, y) ∼ N (0, (
D2iiΣii DiiDjjΣij DiiDjjΣji D 2 jjΣjj
) )
= E[φ(Diix)φ(Djjy) : (x, y) ∼ N (0, (
Σii Σij Σji Σjj
) )]
= DαiiD α jj E[φ(x)φ(y) : (x, y) ∼ N (0, ( Σii Σij Σji Σjj ) )]
= DαiiD α jjVφ(Σ)ij
E.2 FOURIER METHOD
The Laplace method crucially used the fact that we can pull out the norm factor ‖Gh‖ out of φ, so that we can apply Schwinger parametrization. For general φ this is not possible, but we can apply
some wishful thinking and proceed as follows VBφ(Σ) = E[φ( √ BGh/‖Gh‖)⊗2 : h ∼ N (0,Σ)]
= E[φ(Gh/r)⊗2 ∫ ∞
0
d(r2)δ(r2 − ‖Gh‖2/B) : h ∼ N (0,Σ)]
= E[φ(Gh/r)⊗2 ∫ ∞
0
d(r2) ∫ ∞ −∞ dλ 2π eiλ(r 2−‖Gh‖2/B) : h ∼ N (0,Σ)]
“ = ” ∫ RB dh√ det(2πΣ) e −1 2 h TΣ−1hφ(Gh/r)⊗2 ∫ ∞ 0 d(r2) ∫ ∞ −∞ dλ 2π eiλ(r 2−‖Gh‖2/B)
“ = ” ∫ ∞ 0 d(r2) ∫ ∞ −∞ dλ 2π ∫ RB dh√ det(2πΣ) e −1 2 h TΣ−1hφ(Gh/r)⊗2eiλ(r 2−‖Gh‖2/B)
(29)
= 1√
det(2πΣ) ∫ ∞ 0 d(r2) ∫ ∞ −∞ dλ 2π eiλr 2 ∫ RB dhφ(Gh/r)⊗2e −1 2 (h TΣ−1h+2iλ‖Gh‖2/B)
(30)
= 1√
det(2πΣ) ∫ ∞ 0 d(r2) ∫ ∞ −∞ dλ 2π eiλr 2 ∫ RB dhφ(Gh/r)⊗2e −1 2 (h T (Σ−1+2iλG/B)h)
“ = ” 1√
det(2πΣ) ∫ ∞ 0 d(r2) ∫ ∞ −∞ dλ 2π eiλr 2
√ det(2π(Σ−1 + 2iλG/B)−1)E[φ(Gh/r)⊗2 : h ∼ N (0,Σ−1 + 2iλG/B)]
= ∫ ∞ 0 d(r2) ∫ ∞ −∞ dλ 2π eiλr 2 det(I + 2iλGΣ/B)−1/2 E h∼N (0,G(Σ−1+2iλG/B)Gr−2)) φ(h)⊗2
(31)
= ∫ ∞ 0 d(r2) ∫ ∞ −∞ dλ 2π eiλr 2 det(I + 2iλGΣ/B)−1/2Vφ(G(Σ −1 + 2iλG/B)Gr−2)
Here all steps are legal except for possibly Eq. (29), Eq. (30), and Eq. (31). In Eq. (29), we “Fourier expanded” the delta function — this could possibly be justified as a principal value integral, but this cannot work in combination with Eq. (30) where we have switched the order of integration, integrating over h first. Fubini’s theorem would not apply here because eiλ(r
2−‖Gh‖2/B) has norm 1 and is not integrable. Finally, in Eq. (31), we need to extend the definition of Gaussian to complex covariance matrices, via complex Gaussian integration. This last point is no problem, and we will just define
E h∼N (0,Σ) f(h) := det(2πΣ)−1/2 ∫ RB dh f(h)e− 1 2h TΣ−1h
for general complex Σ, whenever Σ is nonsingular and this integral is exists, and
E h∼N (0,GΣG) f(h) := det(2π(eTΣe))−1/2 ∫ RB−1 dz f(ez)e− 1 2 z T (eTΣe)−1z
if eTΣe is nonsingular (as in the case above). The definition of Vφ stays the same given the above extension of the definition of Gaussian expectation.
Nevertheless, the above derivation is not correct mathematically due to the other points. However, its end result can be justified rigorously by carefully expressing the delta function as the limit of mollifiers. Theorem E.25. Let Σ be a positive-definite D × D matrix. Let φ : R → R. Suppose for any b > a > 0,
1. ∫ b a
d(r2)|φ(za/r)φ(zb/r)| exists and is finite for any za, zb ∈ R (i.e. φ(za/r)φ(zb/r) is locally integrable in r).
2. there exists an > 0 such that any γ ∈ [− , ] and for each a, b ∈ [D],∫ ‖z‖2∈[a,b] dz e− 1 2 z TΣ−1z|φ(za/ √ ‖z‖2 + γ)φ(zb/ √ ‖z‖2 + γ)|
exists and is uniformly bounded by some number possibly depending on a and b. 3. for each a, b ∈ [D], ∫ RD dz ∫ b a d(r2)e− 1 2 z TΣ−1z|φ(za/r)φ(zb/r)| exists and is finite.
4. ∫∞ −∞ dλ ∣∣det(I + 2iλΣ)−1/2Vφ((Σ−1 + 2iλI)−1r−2)∣∣ exists and is finite for each r > 0. If e− 1 2 z TΣ−1zφ(z/‖z‖)⊗2 is integrable over z ∈ RD, then
(2π)−D/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zφ(z/‖z‖)⊗2
= 1
2π lim a↘0,b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det(I + 2iλΣ)−1/2Vφ((Σ −1 + 2iλI)−1s−1).
Similarly, (2π)−D/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zφ( √ Dz/‖z‖)⊗2
= 1
2π lim a↘0,b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det(I + 2iλΣ)−1/2Vφ((Σ −1 + 2iλI/D)−1s−1).
If G is the mean-centering projection matrix and ΣG = GΣG, then by the same reasoning as in Thm E.5,
VBφ(Σ) = (2π) −D/2 det Σ−1/2
∫ dz e− 1 2 z TΣ−1zφ( √ DGz/‖Gz‖)⊗2
= 1
2π lim a↘0,b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det(I + 2iλΣG/D)−1/2Vφ(Σ G(I + 2iλΣG/D)−1s−1).
(32)
Note that assumption (1) is satisfied if φ is continuous; assumption (4) is satisfied if D ≥ 3 and for all Π, ‖Vφ(Π)‖ ≤ ‖Π‖α for some α ≥ 0; this latter condition, as well as assumptions (2) and (3), will be satisfied if φ is polynomially bounded. Thus the common coordinatewise nonlinearities ReLU, identity, tanh, etc all satisfy these assumptions.
Warning: in general, we cannot swap the order of integration as ∫∞ −∞ dλ ∫∞ −∞ ds. For example, if φ = id, then
(2π)−D/2 det Σ−1/2 ∫ dz e− 1 2 z TΣ−1zφ(z/‖z‖)⊗2
= 1
2π lim a↘0,b↗∞ ∫ b a ds ∫ ∞ −∞ dλ eiλs det(I + 2iλΣ)−1/2(Σ−1 + 2iλI)−1s−1
6= 1 2π ∫ ∞ −∞ dλ ∫ ∞ 0 ds eiλs det(I + 2iλΣ)−1/2(Σ−1 + 2iλI)−1s−1
because the s-integral in the latter diverges (in a neighborhood of 0).
Proof. We will only prove the first equation; the others follow similarly.
By dominated convergence,∫ dze− 1 2 z TΣ−1zφ(z/‖z‖)⊗2
= lim a↘0,b↗∞
∫ dze− 1 2 z TΣ−1zφ(z/‖z‖)⊗2I(‖z‖2 ∈ [a, b]).
Let η : R→ R be a nonnegative bump function (i.e. compactly supported and smooth) with support [−1, 1] and integral 1 such that η′(0) = η′′(0) = 0. Then its Fourier transform η̂(t) decays like O(t−2). Furthermore, η (x) := −1η(x/ ) is a mollifier, i.e. for all f ∈ L1(R), f ∗ η → f in L1 and pointwise almost everywhere.
Now, we will show that∫ dze− 1 2 z TΣ−1zφ(za/r)φ(zb/r)I(‖z‖2 ∈ [a, b])
= lim ↘0
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)φ(za/r)φ(zb/r)η (‖z‖2 − r2)
by dominated convergence. Pointwise convergence is immediate, because ∫ b a d(r2)φ(za/r)φ(zb/r)η (r 2 − ‖z‖2) = ∫ ∞ −∞ d(r2)φ(za/r)φ(zb/r)I(r2 ∈ [a, b])η (‖z‖2 − r2)
= [(r2 7→ φ(za/r)φ(zb/r)I(r2 ∈ [a, b])) ∗ η ](‖z‖2) → φ(za/‖z‖)φ(zb/‖z‖)I(‖z‖2 ∈ [a, b])
as → 0 (where we used assumption (1) that φ(za/r)φ(zb/r)I(r2 ∈ [a, b]) is L1). Finally, we construct a dominating integrable function. Observe
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)|φ(za/r)φ(zb/r)|η (‖z‖2 − r2)
= ∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)|φ(za/r)φ(zb/r)|η (‖z‖2 − r2)
= ∫ dze− 1 2 z TΣ−1z ∫ b a d(‖z‖2 + γ)|φ(za/ √ ‖z‖2 + γ)φ(zb/ √ ‖z‖2 + γ)|η (−γ)
= ∫ dze− 1 2 z TΣ−1z ∫ b−‖z‖2 a−‖z‖2 dγ |φ(za/ √ ‖z‖2 + γ)φ(zb/ √ ‖z‖2 + γ)|η (−γ)
= ∫ ‖z‖2∈[a− ,b+ ] dze− 1 2 z TΣ−1z ∫ b−‖z‖2 a−‖z‖2 dγ |φ(za/ √ ‖z‖2 + γ)φ(zb/ √ ‖z‖2 + γ)|η (−γ)
since supp η = [− , ] ≤ ∫ ‖z‖2∈[a− ,b+ ] dze− 1 2 z TΣ−1z ∫ − dγ |φ(za/ √ ‖z‖2 + γ)φ(zb/ √ ‖z‖2 + γ)|η (−γ)
= ∫ − dγ η (−γ) ∫ ‖z‖2∈[a− ,b+ ] dze− 1 2 z TΣ−1z|φ(za/ √ ‖z‖2 + γ)φ(zb/ √ ‖z‖2 + γ)|
For small enough then, this is integrable by assumption (2), and yields a dominating integrable function for our application of dominated convergence.
In summary, we have just proven that
∫ dze− 1 2 z TΣ−1zφ(z/‖z‖)⊗2
= lim a↘0,b↗∞ lim ↘0
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)φ(za/r)φ(zb/r)η (‖z‖2 − r2)
Now, ∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)φ(za/r)φ(zb/r)η (‖z‖2 − r2)
= ∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)φ(za/r)φ(zb/r) 1 2π ∫ ∞ −∞ dλ η̂ (λ)e −iλ(‖z‖2−r2)
= 1
2π
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)φ(za/r)φ(zb/r) ∫ ∞ −∞ dλ η̂( λ)e−iλ(‖z‖ 2−r2)
Note that the absolute value of the integral is bounded above by
1
2π
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)|φ(za/r)φ(zb/r)| ∫ ∞ −∞ dλ |η̂( λ)e−iλ(‖z‖2−r2)|
= 1
2π
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)|φ(za/r)φ(zb/r)| ∫ ∞ −∞ dλ |η̂( λ)|
≤ C ∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)|φ(za/r)φ(zb/r)|
for some C, by our construction of η that η̂(t) = O(t−2) for large |t|. By assumption (3), this integral exists. Therefore we can apply the Fubini-Tonelli theorem and swap order of integration.
1
2π
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)φ(za/r)φ(zb/r) ∫ ∞ −∞ dλ η̂( λ)e−iλ(‖z‖ 2−r2)
= 1
2π ∫ b a d(r2) ∫ ∞ −∞ dλ η̂( λ)e−iλ(‖z‖ 2−r2) ∫ dzφ(za/r)φ(zb/r)e − 12 z TΣ−1z
= 1
2π ∫ b a d(r2) ∫ ∞ −∞ dλ η̂( λ)eiλr 2 ∫ dzφ(za/r)φ(zb/r)e − 12 z T (Σ−1+2iλI)z
= 1
2π ∫ b a d(r2)rD ∫ ∞ −∞ dλ η̂( λ)eiλr 2 ∫ dzφ(za)φ(zb)e − 12 r 2zT (Σ−1+2iλI)z
= (2π)D/2−1 det Σ1/2 ∫ b a d(r2) ∫ ∞ −∞ dλ η̂( λ)eiλr 2 det(I + 2iλΣ)−1/2Vφ((Σ −1 + 2iλI)−1r−2)
By assumption (4),∫ b a d(r2) ∫ ∞ −∞ dλ ∣∣∣η̂( λ)eiλr2 det(I + 2iλΣ)−1/2Vφ((Σ−1 + 2iλI)−1r−2)∣∣∣
≤ C ∫ b a d(r2) ∫ ∞ −∞ dλ ∣∣∣det(I + 2iλΣ)−1/2Vφ((Σ−1 + 2iλI)−1r−2)∣∣∣
is integrable (where we used the fact that η̂ is bounded — which is true for all η ∈ C∞0 (R)), so by dominated convergence (applied to ↘ 0),∫
dze− 1 2 z TΣ−1zφ(z/‖z‖)⊗2
= lim a↘0,b↗∞ lim ↘0
∫ dze− 1 2 z TΣ−1z ∫ b a d(r2)φ(za/r)φ(zb/r)η (‖z‖2 − r2)
= (2π)D/2−1 det Σ1/2 lim a↘0,b↗∞ ∫ b a d(r2) ∫ ∞ −∞ dλ eiλr 2 det(I + 2iλΣ)−1/2Vφ((Σ −1 + 2iλI)−1r−2)
where we used the fact that η̂(0) = ∫ R η(x) dx = 1 by construction. This gives us the desired result after putting back in some constants and changing r2 7→ s.
E.3 SPHERICAL INTEGRATION
As mentioned above, the scale invariance of batchnorm combined with the BSB1 fixed point assumption will naturally lead us to consider spherical integration. Here we review several basic facts. Definition E.26. We define the spherical angular coordinates in RB , (r, θ1, . . . , θB−2) ∈ [0,∞)× A =: [0, π]B−3 × [0, 2π], by
x1 = r cos θ1
x2 = r sin θ1 cos θ2
x3 = r sin θ1 sin θ2 cos θ3
... xB−2 = r sin θ1 · · · sin θB−3 cos θB−2 xB−1 = r sin θ1 · · · sin θB−3 sin θB−2.
Then we set
v := 
cos θ1 sin θ1 cos θ2
sin θ1 sin θ2 cos θ3 ...
sin θ1 · · · sin θB−3 cos θB−2 sin θ1 · · · sin θB−3 sin θB−2  as the unit norm version of x. The integration element in this coordinate satisfies
dx1 . . . dxB−1 = r B−2 sinB−3 θ1 sin B−4 θ2 · · · sin θB−3 dr dθ1 · · · dθB−2. Fact E.27. If F : SB−2 → R, then its expectation on the sphere can be written as
E v∼SB−2
F (v) = 1
2 π
1−B 2 Γ
( B − 1
2 )∫ A F (v) sinB−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2
where A = [0, π]B−3 × [0, 2π] and v are as in Defn E.26.
The following integrals will be helpful for simplifying many expressions involving spherical integration using angular coordinates. Lemma E.28. For j, k ≥ 0 and 0 ≤ s ≤ t ≤ π/2,∫ t
s
sinj θ cosk θ dθ = 1
2 Beta
( cos2 t, cos2 s; k + 1
2 , j + 1 2
) .
By antisymmetry of cos with respect to θ 7→ π − θ, if π/2 ≤ s ≤ t ≤ π,∫ t s sinj θ cosk θ dθ = 1 2 (−1)k Beta ( cos2 s, cos2 t; k + 1 2 , j + 1 2 ) .
Proof. Set x := cos2 θ =⇒ dx = −2 cos θ sin θ dθ. So the integral in question is
−1 2 ∫ cos2 t cos2 s (1− x) j−12 x k−12 dx
= 1
2 (Beta(cos2 s;
k + 1 2 , j + 1 2 )− Beta(cos2 t; k + 1 2 , j + 1 2 ))
As consequences, Lemma E.29. For j, k ≥ 0,∫ π
0
sinj θ cosk θ dθ = 1 + (−1)k
2 Beta(
j + 1 2 , k + 1 2 ) = 1 + (−1)k 2 Γ( 1+j2 )Γ( 1+k 2 )
Γ( 2+j+k2 ) .
Lemma E.30. ∫ π 0 sin θ1 dθ1 ∫ π 0 sin2 θ2 dθ2 · · · ∫ π 0 sink θk dθk = π k/2/Γ( k + 2 2 ).
Proof. By Lemma E.29, ∫ π
0 sinr θr dθr = Beta( r+1 2 , 1 2 ) =
Γ( r+12 ) √ π
Γ( r+22 ) . Thus this product of inte-
grals is equal to
k∏ r=1 √ π Γ( r+12 ) Γ( r+22 ) = πk/2Γ(1)/Γ( k + 2 2 ) = πk/2/Γ( k + 2 2 ).
Lemma E.31. For B > 1, µ > 0, ∫∞
0 rBe−r 2/2µ dr = 2 B−1 2 µ B+1 2 Γ(B+12 )
Proof. Apply change of coordinates z = r2/2µ.
Definition E.32. Let K(v; Σ, β) := (det eTΣe)−1/2(vT (eTΣe)−1v)−B−1+β2 and K(v; Σ) := K(v; Σ, 0).
The following proposition allows us to express integrals involving batchnorm as expectations over the uniform spherical measure.
Proposition E.33. If f : RB−1 → RB′ , B > 1− β, and Σ ∈ R(B−1)×(B−1) is nonsingular, then
E h∼N (0,Σ)
‖h‖βf(h/‖h‖) = 2β/2P ( B − 1
2 , β 2
) E
v∼SB−2 f(v)(det Σ)−1/2(vTΣ−1v)− B−1+β 2 .
If f : RB → RB′ and Σ ∈ RB×B is such that eTΣe is nonsingular (where e is as in Defn E.3) and GΣG = Σ,
E h∼N (0,Σ)
‖Gh‖βf(Gh/‖Gh‖) = 2β/2P ( B − 1
2 , β 2
) E
v∼SB−2 f(ev)K(v; Σ, β).
In particular,
E[f(B(h)) : h ∼ N (0,Σ)] = E v∼SB−2 f(ev)K(v; Σ)
Proof. We will prove the second statement. The others follow trivially.
E[‖Gh‖βf(B(h)) : h ∼ N (0,Σ)] = E[‖x‖βf(ex/‖x‖) : x ∼ N (0, eTΣe)]
= (2π) 1−B 2 (det eTΣe)−1/2 ∫ RB−1 ‖x‖βf(ex/‖x‖)e− 12xT (eTΣe)−1x dx
= (2π) 1−B 2 (det eTΣe)−1/2 ∫ A
f(ev) sinB−3 θ1 · · · sin θB−3 dr dθ1 · · · dθB−2×∫ ∞ 0 rB−2+βe− 1 2 r 2·vT (eTΣe)−1v
(spherical coordinates Defn E.26)
= (2π) 1−B 2 (det eTΣe)−1/2 × 2B−32 Γ ( B − 1
2
) × 2 B−3+β 2 Γ(B−1+β2 )
2 B−3 2 Γ ( B−1
2 ) × ∫ A f(ev) sinB−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 (vT (eTΣe)−1v) B−1+β 2
= 1
2 π
1−B 2 Γ
( B − 1
2
) × 2β/2Γ ( B − 1 + β
2
) Γ ( B − 1
2 )−1 × ∫ A f(ev) sinB−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 (vT (eTΣe)−1v) B−1+β 2
= 2β/2Γ
( B − 1 + β
2
) Γ ( B − 1
2
)−1 E
v∼SB−2 f(ev) (det eTΣe)1/2(vT (eTΣe)−1v) B−1+β 2
= 2β/2Γ
( B − 1 + β
2
) Γ ( B − 1
2
)−1 E
v∼SB−2 f(ev)K(v; Σ, β)
As a straightforward consequence,
Proposition E.34. If eTΣe is nonsingular,
VBφ(Σ) = E v∼SB−2
φ( √ Bev)⊗2K(v; Σ)
For numerical calculations, it is useful to realize e as the matrix whose columns are eB−m := (m(m+ 1))−1/2(0, . . . , 0,−m, m︷ ︸︸ ︷ 1, 1, . . . , 1)T , for each m = 1, . . . , B − 1.
e T =  1√ B(B−1) 1√ (B−1)(B−2) . . . 6−1/2
2−1/2

×  −(B − 1) 1 1 · · · 1 1 1 0 −(B − 2) 1 · · · 1 1 1 ... ... ... . . . ... ... ...
0 0 0 · · · −2 1 1 0 0 0 · · · 0 −1 1
 (33)
We have
(ev)1 = − √ B − 1 B cos θ1
(ev)2 = cos θ1 − √ B − 2 B − 1 sin θ1 cos θ2
(ev)3 = cos θ1 + sin θ1 cos θ2 − √ B − 3 B − 2 sin θ1 sin θ2 cos θ3
(ev)4 = cos θ1 + sin θ1 cos θ2 + sin θ1 sin θ2 cos θ3 − √ B − 4 B − 3 sin θ1 sin θ2 sin θ3 cos θ4 (34)
so in particular they only depend on θ1, . . . , θ4.
Angular coordinates In many situations, the sparsity of this realization of e combined with sufficient symmetry allows us to simplify the high-dimensional spherical integral into only a few dimensions, using one of the following lemmas
Lemma E.35. Let f : R4 → RA for some A ∈ N. Suppose B ≥ 6. Then for any k < B − 1
E[r−kf(v1, . . . , v4) : x ∼ N (0, IB−1)]
= Γ((B − 1− k)/2)
Γ((B − 5)/2) 2 −k/2π−2 ∫ π 0 dθ1 · · · ∫ π 0 dθ4 f(v θ 1 , . . . , v θ 4) sin B−3 θ1 · · · sinB−6 θ4
where vi = xi/‖x‖ and r = ‖x‖, and
vθ1 = cos θ1
vθ2 = sin θ1 cos θ2
vθ3 = sin θ1 sin θ2 cos θ3
vθ4 = sin θ1 sin θ2 sin θ3 cos θ4.
Lemma E.36. Let f : R2 → RA for some A ∈ N. Suppose B ≥ 4. Then for any k < B − 1
E[r−kf(v1, v2) : x ∼ N (0, IB−1)]
= Γ((B − 1− k)/2)
Γ((B − 3)/2) 2 −k/2π−1 ∫ π 0 dθ1 ∫ π 0 dθ2 f(v θ 1 , v θ 2) sin B−3 θ1 sin B−4 θ2
where vi = xi/‖x‖ and r = ‖x‖, and
vθ1 = cos θ1
vθ2 = sin θ1 cos θ2
Lemma E.37. Let f : R→ RA for some A ∈ N. Suppose B ≥ 3. Then for any k < B − 1
E[r−kf(v) : x ∼ N (0, IB−1)]
= Γ((B − 1− k)/2)
Γ((B − 2)/2) 2 −k/2π−1/2 ∫ π 0 dθ f(cos θ) sinB−3 θ
where vi = xi/‖x‖ and r = ‖x‖.
We will prove Lemma E.35; those of Lemma E.36 and Lemma E.37 are similar.
Proof of Lemma E.35. E[r−kf(v1, . . . , v4) : x ∼ N (0, IB−1)]
= (2π)−(B−1)/2 ∫ dx r−kf(v1, . . . , v4)e −r2/2
= (2π)−(B−1)/2 ∫ ( rB−2 sinB−3 θ1 · · · sin θB−3 dr dθ1 · · · dθB−2 ) r−kf(vθ1 , . . . , v θ 4)e −r2/2
= (2π)−(B−1)/2 ∫ f(vθ1 , . . . , v θ 4) sin B−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2 ∫ ∞
0
rB−2−ke−r 2/2 dr
= (2π)−(B−1)/2 ∫ f(vθ1 , . . . , v θ 4) sin B−3 θ1 · · · sin θB−3 dθ1 · · · dθB−22(B−3−k)/2Γ((B − 1− k)/2)
(change of coordinate r ← r2 and definition of Γ function) = Γ((B − 1− k)/2)2−(2+k)/2π−(B−1)/2 ∫ f(vθ1 , . . . , v θ 4) sin B−3 θ1 · · · sin θB−3 dθ1 · · · dθB−2
Because vθ1 , . . . , v θ 4 only depends on θ1, . . . , θ4, we can integrate out θ5, . . . , θB−2. By applying Lemma E.30 to θ5, . . . θB−3 and noting that ∫ 2π
0 dθB−2 = 2π, we get
E[r−kf(vθ1 , . . . , vθ4) : x ∼ N (0, IB−1)] = Γ((B − 1− k)/2)2−(2+k)/2π−(B−1)/2 ∫ f(vθ1 , . . . , v θ 4) sin B−3 θ1 · · · sinB−6 θ4π(B−7)/2Γ((B − 5)/2)−1(2π)
= Γ((B − 1− k)/2)
Γ((B − 5)/2) 2 −k/2π−2
∫ f(vθ1 , . . . , v θ 4) sin B−3 θ1 · · · sinB−6 θ4
Cartesian coordinates. We can often also simplify the high dimensional spherical integrals without trigonometry, as the next two lemmas show. Both are proved easily using change of coordinates. Lemma E.38. For any u ∈ Sn,
E v∼Sn f(u · v) = ωn−1 ωn ∫ 1 −1 dh (1− h2)n−22 f(h)
where ωn−1 is hypersurface of the (n− 1)-dimensional unit hypersphere. Lemma E.39. For any u1, u2 ∈ Sn,
E v∼Sn f(u1 · v, u2 · v) = ωn−2 ωn ∫ ‖v‖≤1
v∈span(u1,u2)
dv (1− ‖v‖2)n−32 f(u1 · v, u2 · v)
where ωn−1 is hypersurface of the (n− 1)-dimensional unit hypersphere.
E.4 GEGENBAUER EXPANSION
Definition E.40. The Gegenbauer polynomials {C(α)l (x)}∞l=0, with degC (α) l (x) = l, are the set of orthogonal polynomials with respect to the weight function (1 − x2)α−1/2 (i.e. ∫ 1 −1(1 − x2)α−1/2C (α) l (x)C
(α) m (x) = 0 if l 6= m). By convention, they are normalized so that∫ 1 −1 [ C(α)n (x) ]2 (1− x2)α− 12 dx = π2 1−2αΓ(n+ 2α) n!(n+ α)Γ(α)2 .
Here are several examples of low degree Gegenbauer polynomials
C (α) 0 (x) = 1
C (α) 1 (x) = 2αx
C (α) 2 (x) = −α+ 2α(1 + α)x2
C (α) 3 (x) = −2α(1 + α)x+
4 3 α(1 + α)(2 + α)x3.
They satisfy a few identities summarized below Fact E.41 (Suetin). C(α)n (±1) = (±1)n ( n+2α−1
n ) Fact E.42 (Weisstein).
C(λ)n ′(x) = 2λC (λ+1) n−1 (x) (35)
xC(λ)n ′(x) = nC(λ)n (x) + C (λ) n−1 ′(x) (36)
C (λ) n+1 ′(x) = (n+ 2λ)C(λ)n (x) + xC (λ) n ′(x) (37)
nC(λ)n (x) = 2(n− 1 + λ)xC(λ)n−1(x)− (n− 2 + 2λ)C (λ) n−2(x),∀n ≥ 2 (38)
By repeated applications of Eq. (38), Proposition E.43.
xC(λ)n (x) = 1
2(n+ λ)
( (n+ 1)C
(λ) n+1(x) + (n− 1 + 2λ)C (λ) n−1(x)
) ,∀n ≥ 1
x2C(λ)n (x) = κ2(n, λ)C (λ) n+2(x) + κ0(n, λ)C (λ) n (x) + κ−2 (n, λ)C (λ) n−2(x),∀n ≥ 2
xC (λ) 0 (x) = x =
1
2λ C
(λ) 1 (x)
x2C (λ) 0 (x) =
1
4λ(1 + λ)
( 2C
(λ) 2 (x) + 2λC (λ) 0 (x) ) = κ2(0, λ)C (λ) 2 (x) + κ0(0, λ)C (λ) 0 (x)
x2C (λ) 1 (x) = κ2(1, λ)C (λ) 3 (x) + κ0(1, λ)C (λ) 1 (x)
where κ2(n, λ) := (n+1)(n+2) 4(n+λ)(n+1+λ) , κ0(n, λ) := (n+1)(n+2λ) 4(n+λ)(n+1+λ) + (n−1+2λ)n
4(n+λ)(n−1+λ) , κ−2(n, λ) := (n−1+2λ)(n−2+2λ)
4(n+λ)(n−1+λ) .
This proposition is useful for the Gegenbauer expansion of the local convergence rate of Eq. (43).
Zonal harmonics. The Gegenbauer polynomials are closely related to a special type of spherical harmonics called zonal harmonics, which are intuitively those harmonics that only depend on the “height” of the point along some fixed axis.
Definition E.44. Let ZN−1,(l)u denote the degree l zonal spherical harmonics on the (N − 1)- dimensional sphere with axis u ∈ SN−1, defined by ZN−1,(l)u (v) = c−1N,lC (N−22 )
l (u · v) for any u, v ∈ SN−1, where cN,l = N−2N+2l−2 .
The first few zonal spherical harmonics are ZN−1,(0)u (v) = 1, Z N−1,(1) u (v) = Nu · v, . . . . We note a trivial lemma that is useful for simplying many expressions in Gegenbauer expansions. Lemma E.45. cB+1,l−1cB−1,l = B−1 B−3 .
Let 〈−,−〉SN denote the inner product induced by the uniform measure on SN . One of the most important properties of zonal harmonics is their Fact E.46 (Reproducing property (Suetin)).
〈ZN−1,(l)u , ZN−1,(l)v 〉SN−1 = ZN−1,(l)u (v) = c−1N,lC (N−22 )
l (u · v) 〈ZN−1,(l)u , ZN−1,(m)v 〉SN−1 = 0, if l 6= m.
In our applications of the reproducing property, u and v will always be êa,: := ea,:/‖ea,:‖ = ea,: √ B B−1 , where ea,: is the ath row of e.
Next, we present a new (to the best of our knowledge) identity showing that a certain quadratic form of φ, reminiscent of Dirichlet forms, depending only on the derivative φ′ through a spherical integral, is diagonalized in the Gegenbauer basis. This will be crucial to proving the necessity of gradient explosion under the BSB1 fixed point assumption.
Theorem E.47 (Dirichlet-like form of φ diagonalizes over Gegenbauer basis). Let u1, u2 ∈ SB−2. Suppose φ( √ B − 1x), φ′( √ B − 1x) ∈ L2((1 − x2)B−32 ) and φ( √ B − 1x) has Gegenbauer ex-
pansion ∑∞ l=0 al 1 cB−1,l C (B−32 ) l (x). Then
E v∈SB−2
(u1 · u2 − (u1 · v)(u2 · v))φ′( √ B − 1u1 · v)φ′( √ B − 1u2 · v)
= ∞∑ l=0 a2l (l +B − 3)lc−1B−1,lC (B−32 ) l (u1 · u2)
This is proved using the following lemmas.
Lemma E.48.
xC (α) l ′(x) = lC (α) l (x) + ∑ j∈[l−2]2 2(j + α)C (α) j (x)
C (α) l ′(x) = ∑ j∈[l−1]2 2(j + α)C (α) j (x)
where [n]2 is the sequence {(n%2), (n%2) + 1, . . . , n− 2, n} and n%2 is the remainder of n/2.
Proof. Apply Eq. (36) and Eq. (37) alternatingly.
Lemma E.49.
lC (α) l (t) + ∑ j∈[l−2]2 2(j + α)C (α) j (t)− t ∑ j∈[l−1]2 2(j + α)C (α) j (t) = 0
Proof. Apply Eq. (38) to lC(α)l (t), (l − 2)C (α) l−2(t), . . ..
Proof of Thm E.47. We have
φ′( √ B − 1x) = 1√
B − 1 ∞∑ l=0 al 1 cB−1,l C (B−32 ) l ′(x)
= 1√ B − 1 ∞∑ l=0 al 1 cB−1,l ∑ j∈[l−1]2 (2j +B − 3)C( B−3 2 ) j (x)
= 1√ B − 1 ∞∑ l=0 ∑ j∈[l−1]2 al cB−1,j cB−1,l (2j +B − 3)c−1B−1,jC (B−32 ) j (x)
= 1√ B − 1 ∞∑ j=0 c−1B−1,jC (B−32 ) j (x) ∑ l∈[j+1,∞]2 al cB−1,j cB−1,l (2j +B − 3)
where [n,m]2 is the sequence n, n+ 2, . . . ,m. So for any u1, u2 ∈ SB−2,
E v∈SB−2
φ′( √ B − 1u1 · v)φ′( √ B − 1u2 · v)
= 1 B − 1 ∞∑ j=0 c−1B−1,jC (B−32 ) j (u1 · u2)  ∑ l∈[j+1,∞]2 al cB−1,j cB−1,l (2j +B − 3) 2
= 1 B − 1 ∞∑ j=0 cB−1,j(2j +B − 3)2C( B−3 2 ) j (u1 · u2)  ∑ l,m∈[j+1,∞]2 al cB−1,l am cB−1,m  .
For l < m and m− l even, the coefficient of alam is 2c−1B−1,lc −1 B−1,m B − 1 ∑
j∈[l−1]2
cB−1,j(2j +B − 3)2C( B−3 2 ) j (u1 · u2)
= 2c−1B−1,lc −1 B−1,m B − 1 ∑
j∈[l−1]2
(B − 3)(2j +B − 3)C( B−3
2 ) j (u1 · u2). (39)
For each l, the coefficient of a2l is
c−2B−1,l B − 1 ∑ j∈[l−1]2 cB−1,j(2j +B − 3)2C( B−3 2 ) j (u1 · u2)
= c−2B−1,l B − 1 ∑ j∈[l−1]2 (B − 3)(2j +B − 3)C( B−3 2 ) j (u1 · u2). (40)
Similarly, if φ′(x) ∈ L2((1− x2)B−32 ), we have
xφ′( √ B − 1x) = 1√
B − 1 ∞∑ l=0 al 1 cB−1,l xC (B−32 ) l ′(x)
= 1√ B − 1 ∞∑ l=0 al 1 cB−1,l lC(B−32 )l (x) + ∑ j∈[l−2]2 (2j +B − 3)C( B−3 2 ) j (x)  =
1√ B − 1 ∞∑ j=0 c−1B−1,jC (B−32 ) j (x) ajj + ∑ l∈[j+2,∞]2 al cB−1,j cB−1,l (2j +B − 3)  Then
E v∈SB−2
(u1 · v)(u2 · v)φ′( √ B − 1u1 · v)φ′( √ B − 1u2 · v)
= 1 B − 1 ∞∑ j=0 c−1B−1,jC (B−32 ) j (u1 · u2) ajj + ∑ l∈[j+2,∞]2 al cB−1,j cB−1,l (2j +B − 3) 2 For l < m and m− l even, the coefficient of alam is
1
B − 1
( 2l(2l +B − 3) cB−1,l
cB−1,m c−1B−1,lC
(B−32 )
l (u1 · u2)
+2 ∑
j∈[l−2]2
cB−1,j cB−1,l cB−1,j cB−1,m (2j +B − 3)2c−1B−1,jC (B−32 ) j (u1 · u2)  = 2c−1B−1,lc −1 B−1,m B − 3 B − 1 lC(B−32 )l (u1 · u2) + ∑ j∈[l−2]2 (2j +B − 3)C( B−3 2 ) j (u1 · u2)
 (41) For each l, the coefficient of a2l is
1
B − 1 l2c−1B−1,lC(B−32 )l (u1 · u2) + ∑ j∈[l−2]2 ( cB−1,j cB−1,l )2 (2j +B − 3)2c−1B−1,jC (B−32 ) j (u1 · u2)  = c−2B−1,l B − 3 B − 1  l2 2l +B − 3C (B−32 ) l (u1 · u2) + ∑
j∈[l−2]2
(2j +B − 3)C( B−3
2 ) j (u1 · u2)  (42) By Lemma E.49 and Eqs. (39) and (41), the coefficient of alam withm > l in Ev∈SB−2(u1·u2−(u1· v)(u2 ·v))φ′( √ B − 1u1 ·v)φ′( √ B − 1u2 ·v) is 0. Similarly, by Lemma E.49 and Eqs. (40) and (42), the coefficient of a2l is c −2 B−1,l B−3 B−1 l+B−3 2l+B−3 lC (B−32 ) l (u1 · u2) = c−1B−1,l l+B−3B−1 lC (B−32 ) l (u1 · u2).
E.5 SYMMETRY AND ULTRASYMMETRY
As remarked before, all commonly used nonlinearities ReLU, tanh, and so on induce the dynamics Eq. (43) to converge to BSB1 fixed points, which we formally define as follows. Definition E.50. We say a matrix Σ ∈ SB is BSB1 (short for “1-Block Symmetry Breaking”) if Σ has one common entry on the diagonal and one common entry on the off-diagonal, i.e.
a b b · · · b a b · · · b b a · · · ... ... ... . . .  We will denote such a matrix as BSB1(a, b). Note that BSB1(a, b) can be written as (a−b)I+b11T . Its spectrum is given below. Lemma E.51. AB×B matrix of the form µI+ν11T has two eigenvalues µ andBν+µ, each with eigenspaces {x : ∑i xi = 0} and {x : x1 = · · · = xB}. Equivalently, if it has a on the diagonal and b on the off-diagonal, then the eigenvalues are a− b and (B − 1)b+ a.
The following simple lemma will also be very useful Lemma E.52. Let Σ := BSB1B(a, b). Then GΣG = (a− b)G.
Proof. G and BSB1B(a, b) can be simultaneously diagonalized by Lemma E.51. Note that G zeros out the eigenspace R1, and is identity on its orthogonal complement. The result then follows from easy computations.
The symmetry of BSB1 covariance matrices, especially the fact that they are isotropic in a codimension 1 subspace, is crucial to much of our analysis.
Ultrasymmetry. Indeed, as mentioned before, when investigating the asymptotics of the forward or backward dynamics, we encounter 4-tensor objects, such as
dVBφ dΣ or VB′φ , that would seem
daunting to analyze at first glance. However, under the BSB1 fixed point assumption, these 4- tensors contain considerable symmetry that somehow makes this possible. We formalize the notion of symmetry arising from such scenarios: Definition E.53. Let T : RB×B → RB×B be a linear operator. Let δi ∈ RB be the vector with 0 everywhere except 1 in coordinate i; then δiδTj is the matrix with 0 everywhere except 1 in position (i, j). Write [kl|ij] := T (δiδTj )kl. Suppose T has the property that for all i, j, k, l ∈ [B] = {1, . . . , B}
• [kl|ij] = [π(k)π(l)|π(i)π(j)] for all permutation π on [B], and • [ij|kl] = [ji|lk].
Then we say T is ultrasymmetric. Remark E.54. In what follows, we will often “normalize” the representation “[ij|kl]” to the unique “[i′j′|k′l′]” that is in the same equivalence class according to Defn E.53 and such that i′, j′, k′, l′ ∈ [4] and i′ ≤ j′, k′ ≤ l′ unless i′ = l′, j′ = k′, in which case the normalization is [12|21]. Explicitly, we have the following equivalence classes and their normalized representations
We will study the eigendecomposition of an ultrasymmetric T as well as the projection G⊗2 ◦ T ◦ G⊗2 : Σ 7→ G(T {GΣG})G, respectively with the following domains Definition E.55. Denote by HB the space of symmetric matrices of dimension B. Also write HGB for the space of symmetric matrices Σ of dimension B such that GΣG = Σ (which is equivalent to saying rows of Σ sum up to 0).
As in the case of S, we omit subscript B when it’s clear from context. The following subspaces ofHB andHGB will turn out to comprise the eigenspaces ofG⊗2◦T ◦G⊗2.
class repr. i = j = k = l [11|11] i = j = k or i = j = l [11|12] i = j and k = l [11|22] i = j [11|23] i = k = l or j = k = l [12|11] i = k and j = l [12|12] i = k or i = l [12|13] i = l and j = k [12|21] k = l [12|33] all different [12|34]
Definition E.56. Let LB := {GDG : D diagonal, trD = 0} ⊆ HGB and MB := {Σ ∈ HGB : DiagΣ = 0}. Note that dimLB = B − 1,dimMB = B(B−3)2 and RGB ⊕ LB ⊕MB = HGB is an orthogonal decomposition w.r.t Frobenius inner product.
For the eigenspaces of T , we also need to define Definition E.57. For any nonzero a, b ∈ R, set
LB(a, b) :=  a 0 −b −b · · · 0 −a b b · · · −b b 0 0 · · · −b b 0 0 · · · ... ... ... ... . . .  ∈ HB . In general, we say a matrix M is LB(a, b)-shaped if M = PLB(a, b)PT for some permutation matrix P .
Note that Proposition E.58. L(B − 2, 1)-shaped matrices span LB . Proposition E.59. L(B − 2, 1) = 1B [(−B + 1, 1, 1, . . . , 1)⊗2 − (1,−B + 1, 1, . . . , 1)⊗2] = BG[(1, 0, 0, . . . , 0)⊗2 − (0, 1, 0, . . . , 0)⊗2]G
Proof. Straightforward computation.
Lemma E.60. Let L = LB(a, b). Then G⊗2{L} = GLG = a+2bB LB(B − 2, 1).
Proof. GLG can be written as the sum of outer products
GLG = aG1 ⊗G1 − aG2 ⊗G2 + B∑ i=3 −bG1 ⊗Gi + bG2 ⊗Gi − bGi ⊗G1 + bGi ⊗G2
= aLB (( B − 1 B )2 − 1 B2 , B B2 ) + b B∑ i=3 (−δ1 + δ2)⊗Gi +Gi ⊗ (−δ1 + δ2) = a
B LB(B − 2, 1) + b((−δ1 + δ2)⊗ v + v ⊗ (−δ1 + δ2)) with v = ( −B − 2
B ,−B − 2 B , 2 B , 2 B , . . . ) = a
B LB(B − 2, 1) +
2b B LB(B − 2, 1)
= a+ 2b
B LB(B − 2, 1)
We are now ready to discuss the eigendecomposition of ultrasymmetric operators.
Theorem E.61 (Eigendecomposition of an ultrasymmetric operator). Let T : RB×B → RB×B be an ultrasymmetric linear operator. Then T has the following eigendecomposition.
1. Two 1-dimensional eigenspace R ·BSB1(λTBSB1,i − α22, α21) with eigenvalue λTBSB1,i for i = 1, 2, where
α11 = [11|11] + [11|22](B − 1) α12 = 2(B − 1)[11|12] + (B − 2)(B − 1)[11|23] α21 = 2[12|11] + (B − 2)[12|33] α22 = [12|12] + 4(B − 2)[12|13] + [12|21] + (B − 2)(B − 3)[12|34]
and λTBSB1,1 and λ T BSB1,2 are the roots to the quadratic
x2 − (α11 + α22)x+ α11α22 − α12α21.
2. Two (B − 1)-dimensional eigenspaces SB · L(λTL,i − β22, β21) with eigenvalue λ T L,i for
i = 1, 2. Here SB ·W denotes the linear span of the orbit of matrixW under simultaneous permutation of its column and rows (by the same permutation), and
β11 = [11|11]− [11|22] β12 = 2(B − 2)([11|23]− [11|12]) β21 = −[12|11] + [12|33] β22 = [12|21] + [12|12] + 2(B − 4)[12|13]− 2(B − 3)[12|34].
and λTL,1 and λ T L,2 are the roots to the quadratic
x2 − (β11 + β22)x+ β11β22 − β12β21.
3. Eigenspace M (dimension B(B − 3)/2) with eigenvalue λTM := [12|12] + [12|21] − 4[12|13] + 2[12|34].
The proof is by careful, but ultimately straightforward, computation.
Proof. We will use the bracket notation of Defn E.53 to denote entries of T , and implicitly simplify it according to Remark E.54.
Item 1. Let U ∈ RB×B be the BSB1 matrix. By ultrasymmetry of T and BSB1 symmetry of A, T {U} is also BSB1. So we proceed to calculate the diagonal and off-diagonal entries of T {G}. We have
T {BSB1(a, b)}11 = [11|11]a+ 2(B − 1)[11|12]b+ [11|22](B − 1)a+ [11|23](B − 2)(B − 1)b = ([11|11] + [11|22](B − 1))a+ (2(B − 1)[11|12] + (B − 2)(B − 1)[11|23])b
T {BSB1(a, b)}12 = [12|12]b+ 2[12|11]a+ (B − 2)[12|33]a+ 2(B − 2)[12|13]b + [12|21]b+ 2(B − 2)[12|23]b+ (B − 2)(B − 3)[12|34]b
= (2[12|11] + (B − 2)[12|33])a + ([12|12] + 2(B − 2)[12|13] + [12|21] + 2(B − 2)[12|13] + (B − 2)(B − 3)[12|34])b
= (2[12|11] + (B − 2)[12|33])a + ([12|12] + 4(B − 2)[12|13] + [12|21] + (B − 2)(B − 3)[12|34])b
Thus BSB1(ω1, γ1) and BSB1(ω2, γ2) are the eigenmatrices of T , where (ω1, γ1) and (ω2, γ2) are the eigenvectors of the matrix (
α11 α12 α21 α22
)
with
α11 = [11|11] + [11|22](B − 1) α12 = 2(B − 1)[11|12] + (B − 2)(B − 1)[11|23] α21 = 2[12|11] + (B − 2)[12|33] α22 = [12|12] + 4(B − 2)[12|13] + [12|21] + (B − 2)(B − 3)[12|34]
The eigenvalues are the two roots λTBSB1,1, λ T BSB1,2 to the quadratic
x2 − (α11 + α22)x+ α11α22 − α12α21 and the corresponding eigenvectors are
(ω1, γ1) = (λ1 − α22, α21) (ω2, γ2) = (λ2 − α22, α21).
Item 2. We will study the image of LB(a, b) (Defn E.57) under T . We have T {L(a, b)}11 = −T {L(a, b)}22
= [11|11]a+ 2(B − 2)[11|12](−b) + [11|22](−a) + 2(B − 2)[11|23]b = ([11|11]− [11|22])a+ 2(B − 2)([11|23]− [11|12])b
T {L(a, b)}12 = T {L(a, b)}21 = [12|12]0 + [12|21]0 + [12|11](a− a) + [12|13](b− b) + [12|31](b− b) + [12|33]0 + [12|34]0 = 0
T {L(a, b)}33 = T {L(a, b)}ii,∀i ≥ 3 = [11|11]0 + [11|12](2b− 2b) + [11|22](a− a) + [11|23](2(B − 3)b− 2(B − 3)b) = 0
T {L(a, b)}34 = T {L(a, b)}ij ,∀i 6= j & i, j ≥ 3 = [12|12]0 + [12|11]0 + [12|13](2b− 2b) + [12|21]0
+ [12|31](2b− 2b) + [12|33](a− a) + [12|34](2(B − 4)b− 2(B − 4)b) = 0
T {L(a, b)}13 = T {L(a, b)}1j ,∀j ≥ 3 = T {L(a, b)}j1,∀j ≥ 3 = [12|12](−b) + [12|13](B − 3− 1)(−b) + [12|11]a+ [12|21](−b)
+ [12|31](B − 3− 1)(−b) + [12|33](−a) + [12|34]2(B − 3)b = ( [12|11]− [12|33] ) a+ ( 2(B − 3)[12|34]− 2(B − 4)[12|13]− [12|12]− [12|21] ) b
Thus L(a, b) transforms under T by the matrix L(a, b) 7→ L( ( β11 β12 β21 β22 )( a b ) )
with
β11 = [11|11]− [11|22] β12 = 2(B − 2)([11|23]− [11|12]) β21 = −[12|11] + [12|33] β22 = [12|21] + [12|12] + 2(B − 4)[12|13]− 2(B − 3)[12|34].
So if λTL,1 and λ T L,2 are the roots of the equation
x2 − (β11 + β22)x+ β11β22 − β12β21 then
T {L(λTL,1 − β22, β21)} = λ T L,1L(λ T L,1 − β22, β21) T {L(λTL,2 − β22, β21)} = λ T L,2L(λ T L,2 − β22, β21)
Similarly, any image of these eigenvectors under simultaneous permutation of rows and columns remains eigenvectors with the same eigenvalue. This derives Item 2.
Item 3. Let M ∈M. We first show that T {M} has zero diagonal. We have
T {M}11 = [11|12]( B∑ i=2 M1i +Mi1) + [11|23]( B∑ i,j=1
Mij − (
B∑ i=2 M1i +Mi1
) )
= 0 + 0 = 0
which follows from M1 = 0 by definition of M. Similarly T {M}ii = 0 for all i. Now we show that M is an eigenmatrix.
T {M}12 = [12|12]M12 + [12|11]0 + [12|33]0 + [12|13] (
B∑ i=3 M1i + B∑ i=1 Mi2
)
+ [12|21]M21 + [12|31] (
B∑ i=3 Mi1 + B∑ i=1 M2i
) + [12|34]
∑ i≥3,j≥3 Mij
= [12|12]M12 − 2M12[12|13] +M21[12|21] + [12|31](−2M12) + [12|34]( B∑ i=3 −Mi1 −M1i) = M12([12|12]− 2[12|13] + [12|21]− 2[12|31] + 2[12|34]) = M12([12|12] + [12|21]− 4[12|13] + 2[12|34]) = λTMM12
Similarly T {M}ij = λTMMij for all i 6= j.
Note that the eigenspaces described above are in general not orthogonal under trace inner product, so T is not self-adjoint (relative to the trace inner product) typically. However, as we see next, after projection by G⊗2, it is self-adjoint.
Theorem E.62 (Eigendecomposition of a projected ultrasymmetric operator). Let T : RB×B → RB×B be an ultrasymmetric linear operator. We write G⊗2 ◦ T HGB for the operator HGB → HGB ,Σ 7→ T {Σ} 7→ G(T {Σ})G, restricted to Σ ∈ HGB . Then G⊗2 ◦ T HGB has the following eigendecomposition.
1. Eigenspace RG with eigenvalue λG,TG := B−1 ((B − 1)(α11 − α21)− (α12 − α22)), where as in Thm E.61,
α11 = [11|11] + [11|22](B − 1) α12 = 2(B − 1)[11|12] + (B − 2)(B − 1)[11|23] α21 = 2[12|11] + (B − 2)[12|33] α22 = [12|12] + 4(B − 2)[12|13] + [12|21] + (B − 2)(B − 3)[12|34]
2. Eigenspace L with eigenvalue λG,TL := B−1((B − 2)β11 + β12 + 2(B − 2)β21 + 2β22), where as in Thm E.61,
β11 = [11|11]− [11|22] β12 = 2(B − 2)([11|23]− [11|12]) β21 = −[12|11] + [12|33] β22 = [12|21] + [12|12] + 2(B − 4)[12|13]− 2(B − 3)[12|34].
3. Eigenspace M with eigenvalue λG,TM := λTM = [12|12] + [12|21]− 4[12|13] + 2[12|34].
Proof. Item 1 As in the proof of Thm E.61, we find T {BSB1(a, b)} = BSB1 ((
α11 α12 α21 α22 )( a b )) where
α11 = [11|11] + [11|22](B − 1) α12 = 2(B − 1)[11|12] + (B − 2)(B − 1)[11|23] α21 = 2[12|11] + (B − 2)[12|33] α22 = [12|12] + 4(B − 2)[12|13] + [12|21] + (B − 2)(B − 3)[12|34].
For a = B − 1, b = −1 so that BSB1(B − 1,−1) = BG, we get T {BSB1(B − 1,−1)} = BSB1 ((B − 1)α11 − α12, (B − 1)α21 − α22)
G⊗2 ◦ T {BSB1(B − 1,−1)} = G BSB1 ((B − 1)α11 − α12, (B − 1)α21 − α22) G = ((B − 1)(α11 − α21)− (α12 − α22))G = B−1 ((B − 1)(α11 − α21)− (α12 − α22)) BSB1(B − 1,−1) = λG,TG BSB1(B − 1,−1)
by Lemma E.52.
Item 2. It suffices to show that L(B − 2, 1) is an eigenmatrix with the eigenvalue λG,TL . As in the proof of Thm E.61, we find
T {L(a, b)} = L ((
β11 β12 β21 β22 )( a b )) where
β11 = [11|11]− [11|22] β12 = 2(B − 2)([11|23]− [11|12]) β21 = −[12|11] + [12|33] β22 = [12|21] + [12|12] + 2(B − 4)[12|13]− 2(B − 3)[12|34].
So with a = B − 2, b = 1, we have T {L(B − 2, 1)} = L((B − 2)β11 + β12, (B − 2)β21 + β22)
G⊗2 ◦ T {L(B − 2, 1)} = G L((B − 2)β11 + β12, (B − 2)β21 + β22) G = B−1((B − 2)β11 + β12 + 2((B − 2)β21 + β22))L(B − 2, 1) = λG,TL L(B − 2, 1)
by Lemma E.60.
Item 3. The proof is exactly the same as in that of Thm E.61.
Noting that the eigenspaces of Thm E.62 are orthogonal, we have Proposition E.63. G⊗2 ◦ T HG for any ultrasymmetric operator T is self-adjoint.
DOS Operators. In some cases, such as the original study of vanilla tanh networks by Schoenholz et al. (2016), the ultrasymmetric operator involved has a much simpler form: Definition E.64. Let T : HB → HB be such that for any Σ ∈ HB and any i 6= j ∈ [B],
T {Σ}ii = uΣii T {Σ}ij = vΣii + vΣjj + wΣij
Then we say that T is diagonal-off-diagonal semidirect, or DOS for short. We write more specifically T = DOSB(u, v, w).
Thm E.62 and Thm E.61 still hold for DOS operators, but we can simplify the results and reason about them in a more direct way. Lemma E.65. Let T := DOSB(u, v, w). Then T {LB(a, b)} = LB(ua,wb− va).
Proof. Let L := LB(a, b). It suffices to verify the following, each of which is a direct computation.
1. T {L}3:B,3:B = 0. 2. T {L}1,2 = T {L}2,1 = 0 3. T {L}1,1 = −T {L}2,2 = ua 4. T {L}1,i = −T {L}2,i = T {L}i,1 = −T {L}i,2 = va− wb.
Lemma E.66. Let T := DOSB(u, v, w). Then LB(w − u, v) and LB(0, 1) are its eigenvectors: T {LB(w − u, v)} = uLB(w − u, v)
T {LB(0, 1)} = wLB(0, 1)
Proof. The map (a, b) 7→ (ua,wb − va) has eigenvalues u and w with corresponding eigenvectors (w − u, v) and (0, 1). By Lemma E.65, this implies the desired results.
Lemma E.67. Let T := DOSB(u, v, w). Then for any L ∈ LB ,G⊗2◦T {L} = (u−2v)(B−2)+2wB L.
Proof. By Lemma E.60 and Lemma E.65, G⊗2 ◦ T {L(B− 2, 1)} = G⊗2{L(u(B− 1), w− v(B− 1))} = (u−2v)(B−2)+2wB LB(B − 2, 1). By permutation symmetry, we also have the general result for any LB(B − 2, 1)-shaped matrix L. Since they span LB , this gives the conclusion we want. Lemma E.68. Let T := DOSB(u, v, w). Then T {BSB1B(a, b)} = BSB1B(ua,wb+ 2va).
Proof. Direct computation.
Lemma E.69. Let T := DOSB(u, v, w). Then BSB1B(w − u,wv) and BSB1B(0, 1) are eigenvectors of T .
T {BSB1B(u− w, 2v)} = uBSB1B(u− w, 2v) T {BSB1B(0, 1)} = wBSB1B(0, 1)
Proof. The linear map (a, b) 7→ (ua,wb+ 2va) has eigenvalues u and w with corresponding eigenvectors (u− w, 2v) and (0, 1). The result then immediately follows from Lemma E.68.
Lemma E.70. Let T := DOSB(u, v, w). Then G⊗2 ◦ T {G} = (B−1)(u−2v)+wB G.
Proof. Direct computation with Lemma E.68 and Lemma E.52
Definition E.71. Define MB := {Σ ∈ SB : DiagΣ = 0} (so compared to M, matrices in M do not need to have zero row sums). In addition, for any a, b ∈ R, set
LB(a, b) :=  a −b −b · · · −b 0 0 · · · −b 0 0 · · · ... ... ... . . .  ∈ HB . Define LB(a, b) := span(PT1iLB(a, b)P1i : i ∈ [B]) where P1i is the permutation matrix that swap the first entry with the ith entry, i.e. LB(a, b) is the span of the orbit of LB(a, b) under permuting rows and columns simultaneously.
Note that
LB(a, b) = LB(a, b)− PT12LB(a, b)P12
BSB1B(a,−2b) = B∑ i=1 PT1iLB(a, b)P1i
So LB(a, b),BSB1B(a,−2b) ∈ LB(a, b). Theorem E.72. Let T := DOSB(u, v, w). Suppose w 6= u. Then T HB has the following eigendecomposition:
• MB has eigenvalue w (dimMB = B(B − 1)/2)
• LB(w − u, v) has eigenvalue of u (dimLB(w − u, v) = B)
If w = u, then LB(w − u, v) ⊆MB .
Proof. The case of MB is obvious. We will show that LB(w−u, v) is an eigenspace with eigenvalue u. Then by dimensionality consideration they are all of the eigenspaces of T . Let L := LB(w − u, v). Then it’s not hard to see T {L} = LB(a, b) for some a, b ∈ R. It follows that a has to be u(w−u) and b has to be −(v(w−u)−wv) = uv, which yields what we want. Theorem E.73. Let T := DOSB(u, v, w). Then G⊗2 ◦ T HGB : HGB → HGB has the following eigendecomposition:
• RG has eigenvalue (B−1)(u−2v)+wB (dimRG = 1) • MB has eigenvalue w (dimMB = B(B − 3)/2)
• LB has eigenvalue (B−2)(u−2v)+2wB (dimLB = B − 1)
Proof. The case of MB is obvious. The case for RG follows from Lemma E.70. The case for LB follows from Lemma E.67. By dimensionality considerations these are all of the eigenspaces.
F FORWARD DYNAMICS
In this section we will be interested in studying the dynamics on PSD matrices of the form
Σl = VBφ(Σ l−1) = E[φ(
√ BGh/‖Gh‖)⊗2 : h ∼ N (0,Σl−1)] (43)
where Σl ∈ SB and φ : R→ R.
F.1 GLOBAL CONVERGENCE
Basic questions regarding the dynamics Eq. (43) are 1) does it converge? 2) What are the limit points? 3) How fast does it converge? Here we answer these questions definitively when φ = id.
The following is the key lemma in our analysis. Lemma F.1. Consider the dynamics Σl = E[(h/‖h‖)⊗2 : h ∼ N (0,Σl−1)] on Σl ∈ SA. Suppose Σ0 is full rank. Then
1. liml→∞ Σl = 1AI .
2. This convergence is exponential in the sense that, for any full rank Σ0, there is a constant K < 1 such that λ1(Σl) − λA(Σl) < K(λ1(Σl−1) − λA(Σl−1)) for all l ≥ 2. Here λ1 (resp. λA) denotes that largest (resp. smallest) eigenvalue.
3. Asymptotically, λ1(Σl)− λA(Σl) = O((1− 2A+2 )l).
Proof. Let λ1l ≥ λ2l ≥ · · · ≥ λAl be the eigenvalues of Σl. It’s easy to see that ∑ i λi
l = 1 for all l ≥ 1. So WLOG we assume l ≥ 1 and this equality holds. We will show the “exponential convergence” statement; that liml→∞Σl = 1AI then follows from the trace condition above.
Proof of Item 2. For brevity, we write suppress the superscript index l, and use · to denote ·l. We will now compute the eigenvalues λ1 ≥ · · · ≥ λA of Σ. First, notice that Σ and Σ can be simultaneously diagonalized, and by induction all of {Σl}l≥0 can be simultaneous diagonalized. Thus we will WLOG assume that Σ is diagonal, Σ = Diag(λ1, . . . , λA), so that Σ = Diag(γ1, . . . , γA) for some {γi}i. These {γi}i form the eigenvalues of Σ but a priori we don’t know whether they fall in decreasing order; in fact we will soon see that they do, and γi = λi.
We have γi = (π
2 )−A/2 ∫ ∞A 0A λix 2 i∑ j λjx 2 j e−‖x‖ 2/2 dx
= (π
2 )−A/2 ∫ ∞A 0A λix 2 i e −‖x‖2/2 dx ∫ ∞ 0 e−s ∑ j λjx 2 j ds
= (π
2 )−A/2 ∫ ∞A+1 0A+1 λix 2 i e − 12 ∑ j(1+2sλj)x 2 j dxds
= (π
2 )−A/2 ∫ ∞2 02 λix 2 i e − 12 (1+2sλi)x 2 i · (π 2 )A−1 2 ∏ j 6=i (1 + 2sλj) −1/2 dxi ds
= (π
2 )−1/2 ∫ ∞2 02 λix 2 i e − 12 (1+2sλi)x 2 i dxi ∏ j 6=i (1 + 2sλj) −1/2 ds
= ∫ ∞ 0 λi(1 + 2sλi) −3/2 ∏ j 6=i (1 + 2sλj) −1/2 ds
= ∫ ∞ 0 A∏ j=1 (1 + 2sλj) −1/2λi(1 + 2sλi) −1 ds (44)
Therefore, γi − γk = ∫ ∞
0 A∏ j=1 (1 + 2sλj) −1/2(λi(1 + 2sλi) −1 − λk(1 + 2sλk)−1) ds
= (λi − λk) ∫ ∞
0 A∏ j=1 (1 + 2sλj) −1/2(1 + 2sλi) −1(1 + 2sλk) −1 ds
Since the RHS integral is always positive, λi ≥ λk =⇒ γi ≥ γk and thus γi = λi for each i. Define T (λ) := ∫∞ 0 ∏A j=1(1 + 2sλ1) −1/2(1 + 2sλ1) −1(1 + 2sλA)
−1 ds, so that λ1 − λA = (λ1 − λA)T (λ). Note first that ∏A j=1(1 + 2sλj) −1/2(1 + 2sλi) −1(1 + 2sλk)
−1 is (strictly) log-convex and hence (strictly) convex in λ. Furthermore, T (λ) is (strictly) convex because it is an integral of (strictly) convex functions. Thus T is maximized over any convex region by its extremal points, and only by its extremal points because of strict convexity.
The convex region we are interested in is given by A := {(λi)i : λ1 ≥ λ2 ≥ · · · ≥ λA ≥ 0 & ∑ i λi = 1}.
The unit sum condition follows from the normalization h/‖h‖ of the iteration map. The extremal points of A are {ωk := ( k times︷ ︸︸ ︷
1/k, 1/k, · · · , 1/k, 0, 0, · · · , 0)}Ak=1. For k = 1, . . . , A− 1, we have
T (ωk) = ∫ ∞ 0 (1 + 2s/k)−k/2−1 ds
= 2/k
−k/2(1 + 2s/k) −k/2 ∣∣∣∣∞ 0
= 1.
But for k = A,
T (ωA) = ∫ ∞ 0 (1 + 2s/A)−A/2−2 ds
= 2/A
−A/2− 1(1 + 2s/A) −k/2 ∣∣∣∣∞ 0
= A
A+ 2
This shows that T (λ) ≤ 1, with equality iff λ = ωk for k = 1, . . . , A − 1. In fact, because every point λ ∈ A is a convex combination of ωk for k = 1, . . . , A, λ = ∑Ak=1 akωk, by convexity of T , we must have
T (λ) ≤ A∑ k=1 akT (ω k)
= 1− aA 2
A+ 2
= 1− λA 2
A+ 2 (45)
where the last line follows because ωA is the only point with last coordinate nonzero so that aA = λA.
We now show that the gap λ1l − λAl → 0 as l → ∞. There are two cases: if λAl is bounded away from 0 infinitely often, then T (λ) < 1 − infinitely often for a fixed > 0 so that the gap indeed vanishes with l. Now suppose otherwise, that λAl converges to 0; we will show this leads to a contradiction. Notice that
λA = ∫ ∞ 0 A∏ j=1 (1 + 2sλj) −1/2λA(1 + 2sλA) −1 ds
λA/λA ≥ ∫ ∞
0 A∏ j=1 (1 + 2s(1− λA)/(A− 1))−(A−1)/2(1 + 2sλA)−3/2 ds
where the first lines is Eq. (44) and the 2nd line follows from the convexity of ∏A−1 j=1 (1+2sλj) −1/2 as a function of (λ1, . . . , λA−1). By a simple application of dominated convergence, as λA → 0, this integral converges to a particular simple form,
λA/λA ≥ ∫ ∞
0 A∏ j=1 (1 + 2s/(A− 1))−(A−1)/2 ds
= 1 + 2
A− 3 Thus for large enough l, λAl+1/λAl is at least 1 + for some > 0, but this contradicts the convergence of λA to 0.
Altogether, this proves that λ1l − λAl → 0 and therefore λl → ωA as l → ∞. Consequently, λAl is bounded from below, say by K ′ (where K ′ > 0 because by Eq. (44), λAl is never 0), for all l,
and by Eq. (45), we prove Item 2 by taking K = 1 −K ′ 2A+2 . In addition, asymptotically, the gap decreases exponentially as T (ωA)l = ( A A+2 )l , proving Item 3.
Theorem F.2. Consider the dynamics Σl = E[(h/‖h‖)⊗2 : h ∼ N (0,Σl−1)] on Σl ∈ SA. Suppose Σ0 = MTDM where M is orthogonal and D is a diagonal matrix. If D has rank C and Dii 6= 0,∀1 ≤ i ≤ C, then
1. liml→∞ Σl = 1CM TD′M where D′ is the diagonal matrix with D′ii = I(Dii 6= 0).
2. This convergence is exponential in the sense that, for any Σ0 of rank C, there is a constant K < 1 such that λ1(Σl) − λC(Σl) < K(λ1(Σl−1) − λC(Σl−1)) for all l ≥ 2. Here λi denotes the ith largest eigenvalue.
3. Asymptotically, λ1(Σl)− λC(Σl) = O((1− 2C+2 )l).
Proof. Note that Σl can always be simultaneously diagonalized with Σ0, so that Σl = MTDlM for some diagonal Dl which has Dlii = 0,∀0 ≤ i ≤ C. Then we have (Dl)′ = E[(h/‖h‖)⊗2 : h ∼ N (0, (Dl−1)′)], where (Dl)′ means the diagonal matrix obtained from Dl by deleting the dimensions with zero eigenvalues. The proof then finishes by Lemma F.1.
From this it easily follows the following characterization of the convergence behavior. Corollary F.3. Consider the dynamics of Eq. (43) for φ = id: Σl = B E[(Gh/‖Gh‖)⊗2 : h ∼ N (0,Σl−1)] on Σl ∈ SB . Suppose GΣG has rank C < B and factors as êDêT where D ∈ RC×C is a diagonal matrix with no zero diagonal entries and ê is an B × C matrix whose columns form an orthonormal basis of a subspace of imG. Then
1. liml→∞ Σl = BC êIC ê T .
2. This convergence is exponential in the sense that, for any GΣ0G of rank C, there is a constant K < 1 such that λ1(Σl) − λC(Σl) < K(λ1(Σl−1) − λC(Σl−1)) for all l ≥ 2. Here λi denotes the ith largest eigenvalue.
3. Asymptotically, λ1(Σl)− λC(Σl) = O((1− 2C+2 )l).
General nonlinearity. We don’t (currently) have a proof of any characterization of the basin of attraction for Eq. (43) for general φ. Thus we are forced to resort to finding its fixed points manually and characterize their local convergence properties.
F.2 LIMIT POINTS
Batchnorm Bφ is permutation-equivariant, in the sense that Bφ(πh) = πBφ(h) for any permtuation matrix π. Along with the case of φ = id studied above, this suggests that we look into BSB1 fixed points Σ∗. What are the BSB1 fixed points of Eq. (43)?
F.2.1 SPHERICAL INTEGRATION
The main result (Thm F.5) of this section is an expression of the BSB1 fixed point diagonal and offdiagonal entries in terms of 1- and 2-dimensional integrals. This allows one to numerically compute such fixed points.
By a simple symmetry argument, we have the following Lemma F.4. Suppose X is a random vector in RB , symmetric in the sense that for any permutation matrix π and any subset U of RB measurable with respect to the distribution of X , P (X ∈ U) = P (X ∈ π(U)). Let Φ : RB → RB be a symmetric function in the sense that Φ(πx) = πΦ(x) for any permutation matrix π. Then EΦ(X)Φ(X)T = µI + ν11T for some µ and ν.
This lemma implies that VBφ(Σ) is BSB1 whenever Σ is BSB1. In the following, We in fact show that for any BSB1 Σ, VBφ(Σ) is the same, and this is the unique BSB1 fixed point of Eq. (43).
Theorem F.5. The BSB1 fixed point Σ∗ = µ∗I + ν∗11T to Eq. (43) is unique and satisfies
µ∗ + ν∗ = Γ(B−12 )
Γ(B−22 ) √ π ∫ π 0 dθ1 sin B−3 θ1φ( √ Bζ1(θ1)) 2
ν∗ =
{ B−3 2π ∫ π 0 dθ1 ∫ π 0 dθ2 sin B−3 θ1 sin B−4 θ2φ( √ Bζ1(θ1))φ( √ Bζ2(θ1, θ2)), if B ≥ 4
1 2π ∫ 2π 0 dθφ(− √ 2 sin θ)φ(sin θ/ √ 2− cos θ √ 3/ √ 2), if B = 3
where
ζ1(θ) = − √ B − 1 B cos θ
ζ2(θ1, θ2) = 1√
B(B − 1) cos θ1 − √ B − 2 B − 1 sin θ1 cos θ2
Proof. For any BSB1 Σ, eTΣe = µIB−1 for some µ. Thus we can apply Proposition E.33 toGΣG. We observe that K(v;GΣG) = 1, so that
VBφ(Σ) = VBφ(GΣG) = E v∼SB−2
φ( √ Bev)⊗2.
Note that this is now independent of the specific values of Σ. Now, using the specific form of e given by Eq. (33), we see that VBφ(Σ)11 only depends on θ1 and VBφ(Σ)12 only depends on θ1 and θ2. Then applying Proposition E.33 followed by Lemma E.36 and Lemma E.37 yield the desired result.
F.2.2 LAPLACE METHOD
In the case that φ is positive-homogeneous, we can apply Laplace’s method Appendix E.1 and obtain the BSB1 fixed point in a much nicer form.
Definition F.6. For any α ≥ 0, define Kα,B := cαP ( B−1 2 , α )−1 (B−1 2 )α where P (a, b) := Γ(a+ b)/Γ(a) is the Pochhammer symbol.
Note that Proposition F.7. limB→∞Kα,B = cα.
We give a closed form description of the BSB1 fixed point for a positive homogeneous φ in terms of its J function. Theorem F.8. Suppose φ : R → R is degree α positive-homogeneous. For any BSB1 Σ ∈ SB , VBφ(Σ) is BSB1. The diagonal entries are Kα,BJφ(1) and the off-diagonal entries are Kα,BJφ ( −1 B−1 ) . Here cα is as defined in Defn E.7 and Jφ is as defined in Defn E.20. Thus a BSB1 fixed point of Eq. (43) exists and is unique.
Proof. Let e be anB×(B−1) matrix whose columns form an orthonormal basis of imG := {Gv : v ∈ RB} = {w ∈ RB : ∑i wi = 0}. Let BSB1(a, b) denote a matrix with a on the diagonal and b on the off-diagnals. Note that Vφ is positive homogeneous of degree α, so for any µ,
Vφ(eµIB−1(IB−1 + 2sµIB−1) −1 e T )
= Vφ( µ
1 + 2sµ ee
T )
=
( µ
1 + 2sµ
)α Vφ(ee T ) by Proposition E.24
=
( µ
1 + 2sµ
)α Vφ(G)
=
( µ
1 + 2sµ B − 1 B
)α Vφ ( BSB1 ( 1, −1 B − 1 )) = ( µ
1 + 2sµ B − 1 B
)α cαJφ ( BSB1 ( 1, −1 B − 1 )) by Corollary E.19
So by Eq. (26),
VBφ(eµIe T ) = BαΓ(α)−1 ∫ ∞ 0 ds sα−1(1 + 2sµ)−(B−1)/2 ( µ 1 + 2sµ B − 1 B )α cαJφ ( BSB1 ( 1, −1 B − 1 )) = Γ(α)−1(B − 1)αcαJφ ( BSB1 ( 1, −1 B − 1 ))∫ ∞ 0 ds sα−1(1 + 2sµ)−(B−1)/2−αµα
= Γ(α)−1(B − 1)αcαJφ ( BSB1 ( 1, −1 B − 1 )) Beta(α, (B − 1)/2)2−α
= cα Γ((B − 1)/2)
Γ(α+ (B − 1)/2)
( B − 1
2
)α Jφ ( BSB1 ( 1, −1 B − 1 ))
Corollary F.9. Suppose φ : R→ R is degree α positive-homogeneous. If Σ∗ is the BSB1 fixed point of Eq. (43) as given by Thm F.8, then GΣ∗G = µ∗IB−1 where µ∗ = Kα,B ( Jφ(1) − Jφ(−1/(B −
1)) )
By setting α = 1 and φ = relu, we get
Corollary F.10. For any BSB1 Σ ∈ SB , VBrelu(Σ) is BSB1 with diagonal entries 12 and off-diagonal entries 12J1 ( −1 B−1 ) , so that G⊗2{VBrelu(Σ)} = G(VBrelu(Σ))G = ( 1 2 − 12J1 ( −1 B−1 )) G.
By setting α = 1 and φ = id = x 7→ relu(x)− relu(−x), we get Corollary F.11. For any BSB1 Σ ∈ SB , VBid(Σ) is BSB1 with diagonal entries 1 and off-diagonal entries −1B−1 , so that G
⊗2{VBid(Σ)} = B/(B − 1)G. Remark F.12. One might hope to tweak the Laplace method for computing the fixed point to work for the Fourier method, but because there is no nice relation between Vφ(cΣ) and Vφ(Σ) in general, we cannot simplify Eq. (32) as we can Eq. (26) and Eq. (27).
F.2.3 GEGENBAUER EXPANSION
It turns out that the BSB1 fixed point can be described very cleanly using the Gegenbauer coefficients of φ.
When Σ is BSB1, K(v,Σ) = 1 for all v ∈ SB−2, so that by Proposition E.34,
Σ∗ = E v∼SB−2
φ( √ Bev)⊗2
Σ∗ab = E v∼SB−2
φ( √ Bea,:v)φ( √ Beb,:v)
for any a, b ∈ [B], independent of the actual values of Σ. Here ea,: is the ath row of e. Theorem F.13. If φ(
√ B − 1x) ∈ L2((1 − x2)B−32 ) has Gegenbauer expansion∑∞
l=0 al 1 cB−1,l C
(B−32 )
l (x), then Σ ∗ = µ∗I + ν∗11T with
µ∗ = ∞∑ l=0 a2l 1 cB−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B − 1 ))
= ∞∑ l=0 a2l 1 cB−1,l (( B − 4 + l l ) − C( B−3 2 ) l ( −1 B − 1 ))
ν∗ = ∞∑ l=0 a2l 1 cB−1,l C (B−32 ) l ( −1 B − 1 )
Proof. By the reproducing property Fact E.46, we have
Σ∗ab = 〈φ( √ Bea,:·), φ( √ Beb,:·)〉SB−2
= 〈φ( √ B − 1êa,:·), φ( √ B − 1êb,:·)〉SB−2
= 〈 ∞∑ l=0 al 1 cB−1,l C (B−32 ) l (êa,:·), ∞∑ l=0 al 1 cB−1,l C (B−32 ) l (êb,:·) 〉 SB−2
= 〈 ∞∑ l=0 alZ B−2,(l) êa,: , ∞∑ l=0 alZ B−2,(l) êb,: 〉 SB−2
= ∞∑ l=0 a2lZ B−2,(l) êa,: (êb,:)
= ∞∑ l=0 a2l 1 cB−1,l C (B−32 ) l (〈êa,:, êb,:〉)
= ∞∑ l=0 a2l 1 cB−1,l C (B−32 ) l ({ 1 if a = b −1/(B − 1) else ) This gives the third equation. Since µ∗ = Σ∗11 − Σ∗12, straightforward arithmetic yields the first equation. The second follows from Fact E.41.
We will see later that the rate of gradient explosion also decomposes nicely in terms of Gegenbauer coefficients (Thm G.5). However, this is not quite true for the eigenvalues of the forward convergence (Thms F.22 and F.24).
F.3 LOCAL CONVERGENCE
In this section we consider linearization of the dynamics given in Eq. (43). Thus we must consider linear operators on the space of PSD linear operators SB . To avoid confusion, we use the following notation: If T : SB → SB (for example the Jacobian of VBφ ) and Σ ∈ SB , then write T {Σ} for the image of Σ under T . A priori, the Jacobian of VBφ at its BSB1 fixed point may seem like a very daunting object. But a moment of thought shows that
Proposition F.14. The Jacobian dVBφ
dΣ ∣∣ Σ : HB → HB is ultrasymmetric for any BSB1 Σ.
Now we prepare several helper reults in order to make progress understanding batchnorm. Definition F.15. Define n(x) = √ Bx/‖x‖, i.e. division by sample standard deviation.
Batchnorm Bφ can be decomposed as the composition of three steps, φ ◦ n ◦ G, where G is meancentering, n is division by standard deviation, and φ is coordinate-wise application of nonlinearity. We have, as operatorsH → H,
dVBφ(Σ)
dΣ = dE[Bφ(z)⊗2 : z ∈ N (0,Σ)] dΣ
= dE[(φ ◦ n)(x)⊗2 : x ∈ N (0, GΣG)]
dΣ
= dV(φ◦n)(Σ
G)
dΣG ◦ dΣ
G
dΣ
= dV(φ◦n)(Σ
G)
dΣG ◦G⊗2
Definition F.16. With Σ∗ being the unique BSB1 fixed point, write U := G⊗2 ◦ dV(φ◦n)(Σ G)
dΣG
∣∣ ΣG=GΣ∗G : HG → HG.
It turns out to be advantageous to study U first and relate its eigendecomposition back to that of dVBφ (Σ)
dΣ ∣∣ Σ=Σ∗ , where Σ∗ is the BSB1 fixed point, by applying Lemma F.17.
Lemma F.17. Let X and Y be two vector spaces. Consider linear operators A : X → Y,B : Y → X . Then
1. rankAB = rankBA
2. If v ∈ Y is an eigenvector of AB with nonzero eigenvalue, then X 3 Bv 6= 0 and Bv is an eigenvector of BA of the same eigenvalue
3. Suppose AB has k = rankAB linearly independent eigenvectors {vi}ki=1 of nonzero eigenvalues {λi}ki=1. Then BA has k linearly independent eigenvectors {Bvi}ki=1 with the same eigenvalues {λi}ki=1 which are all eigenvectors of BA with nonzero eigenvalues, up to linear combinations within eigenvectors with the same eigenvalue.
WithA = G⊗2 andB = dV(φ◦n)(Σ G)
dΣG , Thm E.61 implies thatAB andBA can both be diagonalized,
and this lemma implies that all nonzero eigenvalues of dVBφ (Σ)
dΣ can be recovered from those of U .
Proof. (Item 1) Observe rankAB = rankABAB ≤ rankBA. By symmetry the two sides are in fact equal.
(Item 2) Bv cannot be zero or otherwise ABv = A0 = 0, contradicting the fact that v is an eigenvector with nonzero eigenvalue. Suppose λ is the eigenvalue associated to v. ThenBA(Bv) = B(ABv) = B(λv) = λBv, so Bv is an eigenvector of BA with the same eigenvalue.
(Item 3) Item 2 shows that {Bvi}ki=1 are eigenvectors BA with the same eigenvalues {λi}ki=1. The eigenspaces with different eigenvalues are linearly independent, so it suffices to show that if {Bvij}j are eigenvectors of the same eigenvalue λs, then they are linearly independent. But ∑ j ajBvij =
0 =⇒ ∑j ajvij = 0 because B is injective on eigenvectors by Item 2, so that aj = 0 identically. Hence {Bvj}j is linearly independent. Since rankBA = k, these are all of the eigenvectors with nonzero eigenvalues of BA up to linearly combinations.
Lemma F.18. Let f : RB → RA be measurable, and Σ ∈ SB be invertible. Then for any Λ ∈ RB×B , with 〈·, ·〉 denoting trace inner product,
d dΣ E[f(z) : z ∼ N (0,Σ)]{Λ} = 1 2 E[f(z)
〈 Σ−1zzTΣ−1 − Σ−1,Λ 〉 : z ∼ N (0,Σ)]
If f is in addition twice-differentiable, then
d dΣ E[f(z) : z ∼ N (0,Σ)] = 1 2 E [ d2f(z) dz2 : z ∼ N (0,Σ) ]
whenever both sides exist.
Proof. Let Σt, t ∈ (− , ) be a smooth path in SB , with Σ0 = Σ. Write ddtΣt = Σ̇t. Then
d dt E[f(z) : z ∼ N (0,Σt)]
= d
dt (2π)−B/2 det Σ −1/2 t
∫ dz e− 1 2 z TΣ−1t zf(z) ∣∣∣∣ t=0
= (2π)−B/2 −1 2 det Σ −1/2 0 tr ( Σ−10 Σ̇0 )∫ dz e− 1 2 z TΣ−10 z/2f(z)
+ (2π)−B/2 det Σ −1/2 0
∫ dz e− 1 2 z TΣ−10 z
−1 2 zT (−Σ−10 Σ̇0Σ−10 )zf(z)
= −1 2 (2π)−B/2 det Σ −1/2 0
∫ dz e− 1 2 z TΣ−10 zf(z)(tr Σ−10 Σ̇0 − zTΣ−10 Σ̇0Σ−10 z)
= 1
2 (2π)−B/2 det Σ−1/2
∫ dz e− 1 2 z TΣ−1zf(z) 〈 Σ−1zzTΣ−1 − Σ−1, Σ̇0 〉 = −1 2 E[f(z) 〈 Σ−1 − Σ−1zzTΣ−1, Σ̇0 〉 : z ∼ N (0,Σ)]
Note that
vT d dz e −1 2 z TΣ−1z = −vTΣ−1ze−12 zTΣ−1z
wT ( d2
dz2 e −1 2 z TΣ−1z
) v = ( wTΣ−1zzTΣ−1v − wTΣ−1v ) e −1 2 z TΣ−1z
= 〈 Σ−1zzTΣ−1 − Σ−1, vwT 〉 e −1 2 z TΣ−1z
so that as a cotensor,
d2
dz2 e −1 2 z
TΣ−1z{Λ} = 〈 Σ−1zzTΣ−1 − Σ−1,Λ 〉 e −1 2 z TΣ−1z
for any Λ ∈ RB×B . Therefore,
dE[f(z) : z ∼ N (0,Σ)] dΣ {Λ} = 1 2
(2π)−B/2 det Σ−1/2 ∫ dz f(z) d2e− 1 2 z TΣ−1z
d2z {Λ}
= 1
2 (2π)−B/2 det Σ−1/2
∫ dz e− 1 2 z TΣ−1z d 2f(z)
d2z {Λ}
(by integration by parts)
= 1 2 E [ d2f(z) d2z : z ∼ N (0,Σ) ] {Λ}
Note that for any Λ ∈ HB with ‖Λ‖op < µ∗,
E[φ ◦ n(z)⊗2 : z ∈ N (0, µ∗G+ Λ)] = E[φ ◦ n(Gz)⊗2 : z ∈ N (0, µ∗I + Λ)] = E[Bφ(z)⊗2 : z ∈ N (0, µ∗I + Λ)]
so that we have, for any Λ ∈ HB ,
U{Λ} = dVBφ(Σ) dΣ ∣∣∣∣ Σ=µ∗I {Λ}
= 1
2 E[Bφ(z)⊗2〈µ∗−1zzT − µ∗−1I,Λ〉 : z ∼ N (0, µ∗I)]
(by Lemma F.18)
= 1
2 µ∗−2 E[Bφ(z)⊗2〈zzT ,Λ〉 : z ∼ N (0, µ∗I)]−
1 2 µ∗−1Σ∗〈I,Λ〉 (by Thm F.8)
= (2µ∗)−1 ( E[Bφ(z)⊗2〈zzT ,Λ〉 : z ∼ N (0, I)]− Σ∗〈I,Λ〉 ) (Bφ is scale-invariant)
= (2µ∗)−1 ( E[Bφ(z)⊗2〈GzzTG,Λ〉 : z ∼ N (0, I)]− Σ∗〈I,Λ〉 ) (Λ = GΛG)
Let’s extend to all matrices by this formula: Definition F.19. Define
Ũ : RB×B → SB , Λ 7→ (2µ∗)−1 ( E[Bφ(z)⊗2〈GzzTG,Λ〉 : z ∼ N (0, I)]− Σ∗〈I,Λ〉 )
F.3.1 SPHERICAL INTEGRATION
So Ũ HB = U HB . Ultimately we will apply Thm E.61 to G⊗2 ◦ Ũ HB = G⊗2 ◦ U Definition F.20. Write τij = 12 ( δiδ T j + δjδ T i ) .
Then
Ũ{τij}kl = (2µ∗)−1 (E[Bφ(z)kBφ(z)l(Gz)i(Gz)j : z ∼ N (0, I)]− Σ∗klI(i = j)) = (2µ∗)−1 (E[φ(n(y))kφ(n(y))lyiyj : y ∼ N (0, G)]− Σ∗klI(i = j)) = (2µ∗)−1 ( E[φ(n(y))kφ(n(y))lyiyj : y ∼ N (0, eeT )]− Σ∗klI(i = j)
) (See Defn E.3 for defn of e)
= (2µ∗)−1 (E[φ(n(ex))kφ(n(ex))l(ex)i(ex)j : x ∼ N (0, IB−1)]− Σ∗klI(i = j))
Here we will realize e as the matrix in Eq. (33). Definition F.21. Define Wij|kl := E[φ(n(ex))kφ(n(ex))l(ex)i(ex)j : x ∼ N (0, IB−1)].
Then Ũ{τij}kl = (2µ∗)−1(Wij|kl − Σ∗klI(i = j)). If we can evaluate Wij|kl then we can use Thm E.61 to compute the eigenvalues of G⊗2 ◦ U . It’s easy to see that Wij|kl is ultrasymmetric Thus WLOG we can take i, j, k, l from {1, 2, 3, 4}. By Lemma E.35, and the fact that x 7→ ex is an isometry,
Wij|kl = E[r2φ( √ Bev)kφ( √ Bev)l(ev)i(ev)j : x ∼ N (0, IB−1)]
= (B − 5)(B − 3)(B − 1)(2π)−2×∫ π 0 dθ1 · · · ∫ π 0 dθ4 φ( √ Bev)kφ( √ Bev)l(ev)i(ev)j sin B−3 θ1 · · · sinB−6 θ4
If WLOG we further assume that k, l ∈ {1, 2} (by ultrasymmetry), then there is no dependence on θ3 and θ4 inside φ. So we can expand (ev)i and (ev)j in trigonometric expressions as in Eq. (34) and integrate out θ3 and θ4 via Lemma E.29. We will not write out this integral explicitly but instead focus on other techniques for evaluating the eigenvalues.
F.3.2 GEGENBAUER EXPANSION
Now let’s compute the local convergence rate via Gegenbauer expansion. By differentiating Proposition E.34 through a path Σt ∈ SB , t ∈ (− , ), we get
d dt VBφ(Σt) = E v∼SB−2 φ( √ Bev)⊗2 d dt K(v; Σt)
= E v∼SB−2
φ( √ Bev)⊗2K(v; Σt)
( B − 1
2 (vTΣ
−1v)−1vTΣ −1 dΣ dt Σ −1v − 1 2 tr(Σ −1 dΣ dt ) ) where Σ = Σ t = eTΣte ∈ SB−1 (as introduced below Defn E.3). At Σ0 = Σ∗, the BSB1 fixed point, we have Σ 0 = µ∗I , and
d dt VBφ(Σt) ∣∣∣∣ t=0 = E v∼SB−2 φ( √ Bev)⊗2K(v; Σ∗)µ∗−1 ( B − 1 2 vT dΣ dt v − 1 2 tr( dΣ dt ) ) = µ∗−1 E
v∼SB−2 φ( √ Bev)⊗2
( B − 1
2 vT dΣ dt v − 1 2 tr( dΣ dt )
) (46)
If ddtΣ|t=0 = G, then the term in the parenthesis vanishes. This shows that dVBφ dΣ ∣∣∣ Σ=Σ∗ {G} = 0.
Eigenvalue for L. If ddtΣ|t=0 = L(B− 2, 1) = BG(δ⊗21 − δ⊗22 )G where δ1 = (1, 0, . . . , 0), δ2 = (0, 1, 0, . . . , 0), both in RB (Proposition E.59), then we know by Thm E.62 that G⊗2 ◦ d dtVBφ(Σt) ∣∣ t=0
is a multiple of L(B − 2, 1). We compute( d
dt VBφ(Σt) ∣∣∣∣ t=0 ) ab = B B − 1 2 µ∗−1 E v∼SB−2 φ( √ Bea,:v)φ( √ Beb,:v)((e1,:v) 2 − (e2,:v)2).
Clearly, ddtVBφ(Σt) ∣∣ t=0 is L-shaped. Now define the quantity A(a, b; c) := Ev∼SB−2 φ( √ Bea,:v)φ( √ Beb,:v)(ec,:v)
2, so that( d dtVBφ(Σt) ∣∣ t=0 ) ab = B(B−1)2 µ ∗−1(A(a, b; 1) − A(a, b; 2)). Because e is an isometry,∑
b(eb,:v) 2 = ‖v‖2 = 1 for all v ∈ SB−2. Thus
B∑ c=1 A(a, b; c) = E v∼SB−2 φ( √ Bea,:v)φ( √ Beb,:v) = Σ ∗ ab. (47)
By symmetry, A(a, b; a) = A(a, b; b) and for any c, c′ 6∈ {a, b}, A(a, b; c) = A(a, b; c′). So we have, for any a, b, c not equal,
A(a, a; a) + (B − 1)A(a, a; c) = Σ∗aa 2A(a, b; b) + (B − 2)A(a, b; c) = Σ∗ab
So the eigenvalue associated to L(B − 2, 1) is, So by Lemma E.60, G⊗2 ◦ U{L(B − 2, 1)} = λ↑LL(B − 2, 1), where
λ↑L = B B − 1
2 µ∗−1
( d dtVBφ(Σt) ∣∣ t=0 ) 11 − 2 ( d dtVBφ(Σt) ∣∣ t=0 ) 13
B
= B B − 1
2 µ∗−1 A(1, 1; 1)−A(1, 1; 2)− 2(A(1, 3; 1)−A(1, 3; 2)) B
= B − 1 2 µ∗−1(A(1, 1; 1)− 1 B − 1(Σ ∗ aa −A(1, 1; 1))− 2(A(1, 3; 1)− 1 B − 2(Σ ∗ ab − 2A(1, 3; 1)))) = B − 1
2 µ∗−1(
B B − 1A(1, 1; 1)− 1 B − 1Σ ∗ aa − 2 B B − 2A(1, 3; 1) + 2 B − 2Σ ∗ ab) (48)
Thus, if φ( √ B − 1x) ∈ L2((1 − x2)B−32 ) has Gegenbauer expansion ∑∞l=0 al 1cB−1,lC(B−32 )l (x), then by Proposition E.43,
x2φ( √ B − 1x)
= ∞∑ l=0 al 1 cB−1,l x2C (B−32 ) l (x)
= ∞∑ l=0 al 1 cB−1,l ( κ2(l, B − 3 2 )C (B−32 ) l+2 (x)
+ κ0(l, B − 3
2 )C
(B−32 ) l (x) + κ−2(l, B − 3
2 )C
(B−32 ) l−2 (x) ) where C( B−3 2 )
l (x) is understood to be 0 for l < 0
= ∞∑ l=0 C (B−32 )
l (x)( al−2
cB−1,l−2 κ2(l − 2, B − 3 2 ) + al cB−1,l κ0(l, B − 3 2 ) + al+2 cB−1,l+2 κ−2(l + 2, B − 3 2 ) ) where κi(l,
B − 3 2 ) is understood to be 0 for l < 0
= ∞∑ l=0 c−1B−1,lC (B−32 )
l (x)( al−2cB−1,l cB−1,l−2 κ2(l − 2, B − 3 2 ) + alκ0(l, B − 3 2 ) + al+2cB−1,l cB−1,l+2 κ−2(l + 2, B − 3 2 ) )
We can evaluate, for any a, b (possibly equal),
A(a, b; b) = E v∼SB−2
φ( √ Bea,:v)φ( √ Beb,:v)(eb,:v) 2
= E v∼SB−2
φ( √ B − 1êa,:v)φ( √ B − 1êb,:v)(êb,:v)2 B − 1 B
= B − 1 B ∞∑ l=0 c−1B−1,lC (B−32 ) l (〈êa,:, êb,:〉)
al ( al−2cB−1,l cB−1,l−2 κ2(l − 2, B − 3 2 ) + alκ0(l, B − 3 2 ) + al+2cB−1,l cB−1,l+2 κ−2(l + 2, B − 3 2 ) ) (49)
By Eq. (48), the eigenvalue associated with L(B − 2, 1) is B − 1
2 µ∗−1(
B B − 1A(1, 1; 1)− 1 B − 1Σ ∗ aa − 2 B B − 2A(1, 3; 1) + 2 B − 2Σ ∗ ab)
= B − 1
2 µ∗−1 ∞∑ l=0 c−1B−1,l(γlC (B−32 ) l (1)− τlC (B−32 ) l ( −1 B − 1))
where
γl := al ( al−2cB−1,l cB−1,l−2 κ2(l − 2, B − 3 2 ) + alκ0(l, B − 3 2 ) + al+2cB−1,l cB−1,l+2 κ−2(l + 2, B − 3 2 ) ) − 1 B − 1a 2 l τl := 2(B − 1) B − 2 γl.
Thus,
Theorem F.22. Suppose φ( √ B − 1x) has Gegenbauer expansion∑∞l=0 al 1cB−1,lC(B−32 )l (x). Then the eigenvalue of U with respect to the eigenspace L is a ratio of quadratic forms
λ↑L =
∑∞ l=0 a
2 lwB−1,l + alal+2uB−1,l∑∞
l=0 a 2 l vB−1,l
where
vB−1,l := c −1 B−1,l
( C (B−32 ) l (1)− C (B−32 ) l ( −1 B − 1) ) wB−1,l :=
B − 1 2
( κ0 ( l, B − 3
2 ) − 1 B − 1 ) c−1B−1,l ( C (B−32 ) l (1)− 2(B − 1) B − 2 C (B−32 ) l ( −1 B − 1) ) =
l(B − 3 + l)(B − 3) (B − 5 + 2l)(B − 1 + 2l)c −1 B−1,l
( C (B−32 )
l (1)− 2(B − 1) B − 2 C (B−32 ) l ( −1 B − 1) ) = l(B − 3 + l)(B − 3 + 2l) (B − 5 + 2l)(B − 1 + 2l) ( C (B−32 ) l (1)− 2(B − 1) B − 2 C (B−32 ) l ( −1 B − 1)
) uB−1,l :=
B − 1 2 ( c−1B−1,l+2κ−2 ( l + 2, B − 3 2 )( C (B−32 ) l (1)− 2(B − 1) B − 2 C (B−32 ) l ( −1 B − 1) ) + c−1B−1,lκ2 ( l, B − 3
2
)( C (B−32 )
l+2 (1)− 2(B − 1) B − 2 C (B−32 ) l+2 ( −1 B − 1) )) = B − 1
2 ( (B − 3 + l)(B − 2 + l) (B − 3)(B − 1 + 2l) ( C (B−32 ) l (1)− 2(B − 1) B − 2 C (B−32 ) l ( −1 B − 1) ) + (l + 1)(l + 2)
(B − 3)(B − 1 + 2l)
( C (B−32 )
l+2 (1)− 2(B − 1) B − 2 C (B−32 ) l+2 ( −1 B − 1)
)) .
Note that vB−1,0 = wB−1,0 = uB−1,0 = 0, so that there is in fact no dependence on a0, as expected since batchnorm is invariant under additive shifts of φ.
We see that λ↑L ≥ 1 iff
0 ≤ ∞∑ l=1 a2l (wB−1,l − vB−1,l) + alal+2uB−1,l.
This is a quadratic form on the coefficients {al}l. We now analyze it heuristically and argue that the eigenvalue is ≥ 1 typically when φ( √ B − 1x) explodes sufficiently as x→ 1 or x→ −1; in other words, the more explosive φ is, the less likely it is to induce Eq. (43) to converge to a BSB1 fixed point.
Heuristically, C( B−3 2 )
l+2 ( −1 B−1 ) is negligible compared to C
(B−32 ) l+2 (1) for sufficiently large B and l, so that wB−1,l − vB−1,l ≈ ( l(B−3+l)(B−3) (B−5+2l)(B−1+2l) − 1 ) c−1B−1,lC (B−32 )
l+2 (1) =( B−7
4 +O(B −2) +O(l−2) ) c−1B−1,lC (B−32 ) l+2 (1) and is positive. For small l, we can calculate
wB−1,1 − vB−1,1 = −2B 1 +B < 0 wB−1,2 − vB−1,2 = B(B + 1)(B2 − 9B + 12)
2(B − 1)(B + 3) ≥ 0,∀B ≥ 10
wB−1,3 − vB−1,3 = (B − 3)(B + 3)B2(2B3 − 19B2 + 16B + 13)
6(B − 1)2(B + 1)(B + 5) ≥ 0,∀B ≥ 10.
In fact, plottingwB−1,l−vB−1,l for various values of l suggests that forB ≥ 10, wB−1,l−vB−1,l ≥ 0 for all l ≥ 1 (Fig. 10). Thus, the more “linear” φ is, the larger a21 is compared to the rest of {al}l, and the more likely that the eigenvalue is < 1. In the case that alal+2 = 0 ∀l, then indeed the eigenvalue is < 1 precisely when a21 is sufficiently large. Because higher degree Gegenbauer polynomials explodes more violently as x→ ±1, this is consistent with our claim.
Eigenvalue for M We use a similar technique shows the Gegenbauer expansion of the eigenvalue for M. Define Ãab(c, d) := Ev∼SB−2 φ( √ Bea,:v)φ( √ Beb,:v)(ec,:v)(ed,:v). Then Ãab(c, c) = A(a, b; c). Note the ultrasymmetries Ãab(c, d) = Ãπ(a)π(b)(π(c), π(d)) for any permutation π, and Ãab(c, d) = Ãab(d, c) = Ãba(c, d) = Ãba(d, c). By Eq. (46), we have
Ũ{δcδTd }ab
= µ∗−1 E v∼SB−2
φ( √ Bea,:v)φ( √ Beb,:v)
( B − 1
2 (ec,:v)(ed,:v)−
1
2
( I(c = d)− 1
B )) = µ∗−1
B − 1 2 Ã(a, b; c, d)− µ∗−1 1 2 Σ∗ab
( I(c = d)− 1
B
) .
By Thm E.62, G⊗2 ◦ U has eigenspace M with eigenvalue λ↑M = Ũ{δ1δT2 }12 + Ũ{δ2δT1 }12 − 4Ũ{δ1δT3 }12 + 2Ũ{δ3δT4 }12
= 2(Ũ{δ1δT2 }12 − 2Ũ{δ1δT3 }12 + Ũ{δ3δT4 }12) = B − 1 µ∗ (Ã12(12)− Ã12(13) + Ã12(34)).
Thus we need to evaluate Ã12(12), Ã12(13), Ã12(34). We will do so by exploiting linear dependences between different values of Ãab(c, d), the computed values of Ãab(c, c) = A(a, b; c), and the value of Ãab(a, b) computed below in Lemma F.23. Indeed, we have
B∑ d=1 Ãab(c, d) = E v∼SB−2 φ( √ Bea,:v)φ( √ Beb,:v)(ec,:v) B∑ d=1 (ed,:v) = 0.
Leveraging the symmmetries of Ã, we get (B − 3)Ã12(34) + 2Ã12(13) + Ã12(33) = 0 (B − 2)Ã12(13) + Ã12(11) + Ã12(12) = 0.
Expressing in terms of Ã12(11) and Ã12(12), we get
Ã12(13) = −1 B − 2
( Ã12(11) + Ã12(12) ) Ã12(34) =
−1 B − 3
( 2Ã12(13) + Ã12(33) ) = −1 B − 3 ( −2 B − 2 ( Ã12(11) + Ã12(12) ) + 1 B − 2 ( Σ∗12 − 2Ã12(22) ))
By Eq. (47)
= −1 (B − 3)(B − 2) ( Σ∗12 − 4Ã12(11)− 2Ã12(12) ) .
These relations allow us to simplify
λ↑M = B − 1 (B − 3)µ∗ ( −1 B − 2Σ ∗ 12 + 2(B − 1) B − 2 Ã12(11) + (B − 1)Ã12(12) ) .
By Eq. (49), we know the Gegenbauer expansion of Ã12(11). The following shows the Gegenbauer expansion of Ã12(12). Lemma F.23. Suppose φ( √ B − 1x), xφ( √ B − 1x) ∈ L2((1 − x2)B−32 ), and φ( √ B − 1x) has
Gegenbauer expansion ∑∞ l=0 al 1 cB−1,l C (B−32 ) l (x). Then
Ã12(12) = E v∼SB−2
φ( √ Be1,:v)φ( √ Be2,:v)(e1,:v)(e2,:v)
= ∞∑ l=0 αla 2 l + βlalal+2
where
αl = B − 1 B ×  ( l+1 B−1+2l )2 c−1B−1,l+1C (B−32 ) l+1 ( −1 B−1 ) + ( B+l−4 B+2l−5 )2 c−1B−1,l−1C (B−32 ) l−1 ( −1 B−1 ) if l ≥ 1(
1 B−1 )2 c−1B−1,1C (B−32 ) 1 ( −1 B−1 ) otherwise
βl = B − 1 B × 2(l + 1)(B + l − 2) (B + 2l − 1)2 c −1 B−1,l+1C (B−32 ) l+1 ( −1 B − 1 ) .
Proof. Let ψ(x) = xφ( √ B − 1x). Then by Proposition E.43, we have
ψ(x) = ∞∑ l=0 alc −1 B−1,l
1 B − 3 + 2l ( (l + 1)C (B−32 ) l+1 (x) + (B + l − 4)C (B−32 ) l−1 (x) )
= ∞∑ l=0 ( al−1
l B + 2l − 3 + al+1 B + l − 3 B + 2l − 3 ) c−1B−1,lC (B−32 ) l (x).
Then
Ã12(12) = B − 1 B E v∼SB−2 ψ(ê1,:v)ψ(ê2,:v)
= B − 1 B ∞∑ l=0 ( al−1
l B + 2l − 3 + al+1 B + l − 3 B + 2l − 3 )2 c−1B−1,lC (B−32 ) l ( −1 B − 1 ) .
Rearranging the sum in terms of al gives the result.
Combining all of the above, we obtain the following Theorem F.24. Suppose φ( √ B − 1x) has Gegenbauer expansion∑∞l=0 al 1cB−1,lC(B−32 )l (x). Then the eigenvalue of U with respect to the eigenspace M is a ratio of quadratic forms
λ↑M =
∑∞ l=0 a
2 l w̃B−1,l + alal+2ũB−1,l∑∞
l=0 a 2 l vB−1,l
where
ũB−1,l := 2(B − 1)3
(B − 3)2(B − 2)B
( (B − 3 + l)(B − 2 + l)
B − 1 + 2l C (B−32 ) l ( −1 B − 1 ) +
(B − 2)(l + 1)(B − 2 + l) B − 1 + 2l C (B−32 ) l+1 ( −1 B − 1 ) +
(l + 3)(l + 4)(B − 3 + 2l) (B + 2l + 1)(B + 2l + 3) C (B−32 ) l+2 ( −1 B − 1 )) w̃B−1,l :=
B − 1 (B − 3)2(B − 2) [ (
2(B − 1)2(B + 2l − 3) ( 2Bl +B + 2l2 − 6l − 5 ) B(B + 2l − 5)(B + 2l − 1) − (B − 3 + 2l) ) C (B−32 ) l ( −1 B − 1 ) +
(B − 2)(B − 1)2(l + 1)2 B(B + 2l − 1) C (B−32 ) l+1 ( −1 B − 1 ) +
(B − 2)(B − 1)2(B − 4 + l)2 B(B + 2l − 5) C (B−32 ) l−1 ( −1 B − 1 )]
where C( B−3
2 ) −1 (x) = 0 by convention.
Note that ũB−1,l and w̃B−1,l depends only on C (B−32 ) • ( −1 B−1 ) which is much smaller than C (B−32 ) • (1) for degree larger than 0. Thus the eigenvalue for M is typically much smaller than the eigenvalue for L (though there are counterexamples, like sin).
F.3.3 LAPLACE METHOD
Differentiating Eq. (26) In what follows, let φ be a positive-homogeneous function of degree α. We begin by studying dV(φ◦n)(Σ G)
dΣG . We will differentiate Eq. (26) directly at G⊗2{Σ∗} = GΣ∗G
where Σ∗ is the BSB1 fixed point given by Thm F.8. To that end, consider a smooth path Σt ∈ SGB , t ∈ (− , ) for some > 0, with Σ0 = GΣ∗G. Set Σ t = eTΣte ∈ SB−1, so that Σt = eΣ te
T and Σ 0 = µ∗IB−1 where µ∗ = Kα,B(Jφ(1)− Jφ( −1B−1 )) as in Thm F.8. If we write Σ̇ and Σ̇ for the time derivatives, we have
B−αΓ(α) d
dt V(φ◦n)(eΣ te
T ) ∣∣∣∣ t=0
= B−αΓ(α) d
dt VBφ(eΣ te
T ) ∣∣∣∣ t=0
= ∫ ∞ 0 ds sα−1 [ d dt det(I + 2sΣ t) −1/2 ∣∣∣∣ t=0 ×Vφ ( eΣ 0(I + 2sΣ 0) −1 e T )
+ det(I + 2sΣ 0) −1/2 d
dt Vφ(eΣ t(I + 2sΣ t)
−1 e T ) ∣∣∣∣ t=0 ] =
∫ ∞ 0 ds sα−1 [ − s 1 + 2sµ∗ det(I + 2sΣ 0) −1/2 tr ( Σ̇ 0 ) × ( µ∗ 1 + 2sµ∗ )α Vφ(G)
+ det(I + 2sΣ 0) −1/2 dVφ(Σ)
dΣ ∣∣∣∣ Σ=eΣ 0(I+2sΣ 0)
−1 e T{
e(I + 2sΣ 0) −1Σ̇ 0(I + 2sΣ 0) −1 e T }]
(apply Lemma F.26 and Lemma F.27)
= ∫ ∞ 0 ds sα−1 [ − s 1 + 2sµ∗ (1 + 2sµ∗)−(B−1)/2 tr ( Σ̇ 0 ) × ( µ∗ 1 + 2sµ∗ )α Vφ(G)
+ (1 + 2sµ∗)−(B−1)/2 ( µ∗
1 + 2sµ∗
)α−1 dVφ(Σ)
dΣ ∣∣∣∣ Σ=G { (1 + 2sµ∗)−2eΣ̇ 0e T }]
(using fact that dVφ/dΣ is degree (α− 1)-positive homogeneous) = − (∫ ∞
0
(1 + 2sµ∗)− B−1 2 −1−αµ∗αsα ) tr (
Σ̇ 0 ) Vφ(G)
+ (∫ ∞ 0 (1 + 2sµ∗)− B−1 2 −1−αµ∗α−1sα−1 ) dVφ(Σ) dΣ ∣∣∣∣ Σ=G { eΣ̇ 0e T }
= −µ∗−12−1−α Beta ( B − 1
2 , α+ 1
) tr (
Σ̇ 0 ) Vφ(G)
+ µ∗−12−α Beta
( B + 1
2 , α
) dVφ(Σ)
dΣ ∣∣∣∣ Σ=G { Σ̇0 } (apply Lemma F.28)
if B−12 + 2 > α (precondition for Lemma F.28).
With some trivial simplifications, we obtain the following Lemma F.25. Let φ be positive-homogeneous of degree α. Consider a smooth path Σt ∈ SGB with Σ0 = µ ∗G. If B−12 + 2 > α, then
d dt V(φ◦n)(Σt) ∣∣∣∣ t=0
= dV(φ◦n)(Σ)
dΣ
∣∣∣∣ Σ=µ∗G {Σ̇0}
= −αBαµ∗−12−1−αP ( B − 1
2 , α+ 1
)−1 tr (
Σ̇ 0 ) Vφ(G)
+Bαµ∗−12−αP
( B + 1
2 , α
)−1 dVφ(Σ)
dΣ ∣∣∣∣ Σ=G { Σ̇0 } (50)
where P (a, b) = Γ(a+ b)/Γ(a) is the Pochhammer symbol.
Lemma F.26. For any s ∈ R, ddt det(I + 2sΣ )−1/2 = −sdet(I + 2sΣ )−1/2 tr((I + 2sΣ ) −1dΣ /dt).
For Σ 0 = µI , this is also equal to − s1+2sµ det(I + 2sΣ )−1/2 tr( ddtΣ ) at t = 0.
Proof. Straightforward computation.
Lemma F.27. For any s ∈ R, ddtΣ (I + 2sΣ )−1 = (I + 2sΣ ) dΣ dt (I + 2sΣ ) −1.
Proof. Straightforward computation. Lemma F.28. For a > b+ 1, ∫∞
0 (1 + 2sµ)−asb ds = (2µ)−1−b Beta(b+ 1, a− 1− b).
Proof. Apply change of variables x = 2µs1+2µs .
This immediately gives the following consequence. Theorem F.29. Let φ : R→ R be any function with finite first and second Gaussian moments. Then G⊗2 ◦ dVφdΣ ∣∣ Σ=BSB1(a,b) : HGB → HGB has the following eigendecomposition:
• RG has eigenvalue (B−1)(u−2v)+wB (dimRG = 1) • MB has eigenvalue w (dimMB = B(B − 3)/2)
• LB has eigenvalue (B−2)(u−2v)+2wB (dimLB = B − 1)
where
u = ∂Vφ(Σ)11 ∂Σ11 ∣∣∣∣ Σ=BSB1(a,b) , v = ∂Vφ(Σ)12 ∂Σ11 ∣∣∣∣ Σ=BSB1(a,b) , w = ∂Vφ(Σ)12 ∂Σ12 ∣∣∣∣ Σ=BSB1(a,b) .
Theorem F.30. Let φ : R → R be positive-homogeneous of degree α. Then for any p 6= 0, c ∈ R, G⊗2 ◦ dVφdΣ ∣∣ Σ=BSB1(p,cp) : HGB → HGB has the following eigendecomposition:
• MB has eigenvalue cαpα−1J′φ (c)
• LB has eigenvalue cαpα−1 1B ( (B − 2)α [Jφ(1)− Jφ (c)] + (2 + c(B − 2)) J′φ (c) )
• RG has eigenvalue cαpα−1 1B ( (B − 1)α [Jφ(1)− Jφ (c)] + (1 + c(B − 1)) J′φ (c) )
Proof. By Proposition E.22, dVφdΣ ∣∣ Σ=BSB1(p,cp) is DOS(u, v, w) with
u = ∂Vφ(Σ)11 ∂Σ11 ∣∣∣∣ Σ=BSB1(p,cp)
= cααp α−1Jφ(1)
v = ∂Vφ(Σ)12 ∂Σ11 ∣∣∣∣ Σ=BSB1(p,cp)
= 1
2 cαΣ
1 2 (α−2) ii Σ 1 2α jj (αJφ(cij)− cijJ′φ(cij))
= 1
2 cαp α−1 (αJφ (c)− cJ′φ (c)) w =
∂Vφ(Σ)12 ∂Σ12 ∣∣∣∣ Σ=BSB1(p,cp)
= cαΣ (α−1)/2 ii Σ (α−1)/2 jj J ′ φ(cij) = cαp α−1J′φ (c)
With Thm E.73, we can do the computation:
• MB has eigenvalue w = cαpα−1J′φ (c)
• LB has eigenvalue (u− 2v)(B − 2) + 2w
B
= 1
B
( (B − 2) ( cααp α−1Jφ(1)− cαpα−1 ( αJφ (c)− cJ′φ (c) )) + 2cαp α−1J′φ (c) )
= cαp α−1 1
B
( (B − 2)α [Jφ(1)− Jφ (c)] + (2 + c(B − 2)) J′φ (c) )
• RG has eigenvalue (B − 1)(u− 2v) + w
B
= 1
B
( (B − 1) ( cααp α−1Jφ(1)− cαpα−1 ( αJφ (c)− cJ′φ (c) )) + cαp α−1J′φ (c) )
= cαp α−1 1
B
( (B − 1)α [Jφ(1)− Jφ (c)] + (1 + c(B − 1)) J′φ (c) )
We record the following consequence which will be used frequently in the sequel. Theorem F.31. Let φ : R → R be positive-homogeneous of degree α. Then G⊗2 ◦ dVφdΣ ∣∣ Σ=G : HGB → HGB has the following eigendecomposition:
• MB has eigenvalue λG,φM (B,α) := cα ( B−1 B )α−1 J′φ ( −1 B−1 ) • LB has eigenvalue λG,φL (B,α) := cα ( B−1 B )α−1 1 B ( (B − 2)α [ Jφ(1)− Jφ ( −1 B−1 )] + BB−1J ′ φ ( −1 B−1
)) • RG has eigenvalue λG,φG (B,α) := cα ( B−1 B )α α [ Jφ(1)− Jφ ( −1 B−1
)] Proof. Plug in p = B−1B and c = −1/(B − 1) for Thm F.30. Theorem F.32. Let φ : R → R be a degree α positive-homogeneous function. Then for p 6= 0, c ∈ R, dVφdΣ ∣∣ Σ=BSB1(p,cp) : HB → HB has the following eigendecomposition:
• MB has eigenvalue cαpα−1J′φ (c)
• LB(a, b) has eigenvalue cααpα−1Jφ(1) where a = 2 ( J′φ (c)− αJφ(1) ) and b =
αJφ (c)− cJ′φ (c)
Proof. Use Thm E.72 with the computations from the proof of Thm F.30 as well as the following computation
w − u = cαpα−1J′φ (c)− cααpα−1Jφ(1) = cαp α−1 (J′φ (c)− αJφ(1)) v = 1
2 cαp
α−1 (αJφ (c)− cJ′φ (c))
Theorem F.33. Let φ be positive-homogeneous with degree α. Assume B−12 > α. The operator U = G⊗2 ◦ dV(φ◦n)(Σ G)
dΣG
∣∣ ΣG=µ∗G : HGB → HGB has 3 distinct eigenvalues. They are as follows:
1. λ↑G(B,α) := 0 with dimension 1 eigenspace RG.
2.
λ↑M(B,α) := B αµ∗−12−αP
( B + 1
2 , α
)−1 λG,φM (B,α)
= cαB(B − 1)α−1µ∗−12−αP ( B + 1
2 , α
)−1 J′φ ( −1 B − 1 ) with dimension B(B−3)2 eigenspace M.
3.
λ↑L(B,α) := B αµ∗−12−αP
( B + 1
2 , α
)−1 λG,φL (B,α)
= cα(B − 1)α−1µ∗−12−αP ( B + 1
2 , α )−1 (
(B − 2)α [ Jφ(1)− Jφ ( −1 B − 1 )] + B B − 1J ′ φ ( −1 B − 1 )) with dimension B − 1 eigenspace L.
Proof. Item 1. The case of λ↑G(B,α). Since (φ ◦ n)(µ∗G) = (φ ◦ n)(µ∗G + ωG) for any ω, the operator dV(φ◦n)(Σ G)
dΣG
∣∣ ΣG=µ∗G sends G to 0.
Item 2. The case of λ↑M(B,α). Assume Σ̇0 ∈ M, Thm F.31 gives (with denoting Hadamard product, i.e. entrywise multiplication)
dVφ(Σ)
dΣ ∣∣∣∣ Σ=G {Σ̇0} = λG,φM (B,α)Σ̇0
= cα ( B − 1 B )α−1 J′φ ( −1 B − 1 ) Σ̇0
Since tr(Σ̇0) = 0, Eq. (50) gives
G⊗2 ◦ dV(φ◦n)(Σ G)
dΣG
∣∣∣∣ ΣG=µ∗G {Σ̇0}
= G⊗2 ◦ d dt V(φ◦n)(Σt) ∣∣∣∣ t=0 {Σ̇0}
= Bαµ∗−12−αP
( B + 1
2 , α
)−1 G⊗2 ◦ dVφ(Σ)
dΣ ∣∣∣∣ Σ=G { Σ̇0 } = Bαµ∗−12−αP ( B + 1
2 , α
)−1 cα ( B − 1 B )α−1 J′φ ( −1 B − 1 ) G⊗2{Σ̇0}
= cαB(B − 1)α−1µ∗−12−αP ( B + 1
2 , α
)−1 J′φ ( −1 B − 1 ) Σ̇0
So the eigenvalue for M is λ↑L = cαB(B − 1)α−1µ∗−12−αP ( B+1 2 , α )−1 J′φ ( −1 B−1 ) .
Item 3. The case of λ↑L(B,α). Let Σ̇0 = L(B − 2, 1). Thm F.31 gives
G⊗2 ◦ dVφ(Σ) dΣ ∣∣∣∣ Σ=G {Σ̇0} = λG,φL (B,α)Σ̇0
where
λG,φL (B,α) := 1
B cα ( B − 1 B )α−1( (B − 2)α [ Jφ(1)− Jφ ( −1 B − 1 )] + B B − 1J ′ φ ( −1 B − 1 )) Since tr(Σ̇0) = 0, Eq. (50) gives
G⊗2 ◦ dV(φ◦n)(Σ G)
dΣG
∣∣∣∣ ΣG=µ∗G {Σ̇0}
= G⊗2 ◦ d dt V(φ◦n)(Σt) ∣∣∣∣ t=0
= Bαµ∗−12−αP
( B + 1
2 , α
)−1 G⊗2 ◦ dVφ(Σ)
dΣ ∣∣∣∣ Σ=G { Σ̇0 } = Bαµ∗−12−αP ( B + 1
2 , α
)−1 λ̃Σ̇0
= (B − 1)α−1cαµ∗−12−αP ( B + 1
2 , α )−1 × ( (B − 2)α [ Jφ(1)− Jφ ( −1 B − 1 )] + B B − 1J ′ φ ( −1 B − 1
)) × Σ̇0
So λ↑L = (B−1)α−1cαµ∗−12−αP ( B+1 2 , α )−1 ( (B − 2)α [ Jφ(1)− Jφ ( −1 B−1 )] + BB−1J ′ φ ( −1 B−1 ))
With some routine computation, we obtain Proposition F.34. With φ and α as above, as B →∞,
λ↑L(B,α) ∼ α+B−1(2α3 − 4α2 − α+ J′φ(0)
Jφ(1)− Jφ(0) ) +O(B−2)
λ↑M(B,α) ∼ J′φ(0)
Jφ(1)− Jφ(0)
+B−1 (( 2α2 − 4α+ 1
) J′φ(0) Jφ(1)− Jφ(0) − J′′φ(0) Jφ(1)− Jφ(0) − (
J′φ(0)
Jφ(1)− Jφ(0) )2) +O ( B−2
) We can compute, by Proposition E.23,
J′φ(0)
Jφ(1)− Jφ(0) =
(a+ b)2 1√ π
Γ(α2 +1) 2 Γ(α+ 12 )
(a2 + b2)− (a− b)2 1 2 √ π
Γ(α2 + 1 2 ) 2
Γ(α+ 12 ) ≤ 2Γ ( α 2 + 1 )2 √ πΓ ( α+ 12
) where the last part is obtained by optimizing over a and b. On α ∈ (−1/2,∞), this is greater than α iff α < 1. Thus for α ≥ 1, the maximum eigenvalue of U is always achieved by eigenspace L for large enough B.
G BACKWARD DYNAMICS
Let’s extend the definition of the V operator:
Definition G.1. Suppose Tx : Rn → Rm is a linear operator parametrized by x ∈ Rk. Then for a PSD matrix Σ ∈ Sk, define VT (Σ) := E[Tx⊗ Tx : x ∈ N (0,Σ)] : Rn×n → Rm×m, which acts on n× n matrices Π by
VT (Σ){Π} = E[TxΠTTx : x ∈ N (0,Σ)] ∈ Rm×m.
Under this definition, VB′φ(Σ) is a linear operator R B×B → RB×B . Recall the notion of adjoint:
Definition G.2. If V is a (real) vector space, then V †, its dual space, is defined as the space of linear functionals f : V → R on V . If T : V → W is a linear operator, then T †, its adjoint, is a linear operator W † → V †, defined by T †(f) = v 7→ f(T (v)).
If a linear operator is represented by a matrix, with function application represented by matrixvector multiplication (matrix on the left, vector on the right), then the adjoint is represented by matrix transpose.
The backward equation. In this section we are interested in the backward dynamics, given by the following equation
Πl = VB′φ(Σ l)†{Πl+1}
where Σl is given by the forward dynamics. Particularly, we are interested in the specific case when we have exponential convergence of Σl to a BSB1 fixed point. Thus we will study the asymptotic approximation of the above, namely the linear system.
Πl = VB′φ(Σ ∗)†{Πl+1} (51)
where Σ∗ is the BSB1 fixed point. Note that after one step of backprop, ΠL−1 = VB′φ(Σ ∗)†{ΠL} is inHGB . Thus the large L dynamics of Eq. (51) is given by the eigendecomposition of VB′φ(Σ ∗)† HGB : HGB → HGB . It turns out to be much more convenient to study its adjoint G⊗2 ◦ VB′φ(Σ ∗), which has the same eigenvalues (in fact, it will turn out that it is self-adjoint).
G.1 EIGENDECOMPOSITION OF G⊗2 ◦VB′φ(Σ ∗).
By chain rule, B′φ(x) = dφ◦n(z)dz ∣∣ z=Gx dGx dx = dφ◦n(z) dz ∣∣ z=Gx G,B′φ(x)∆x = dφ◦n(z)dz ∣∣ z=Gx
G∆x Thus
VB′φ(Σ) = E [ B′φ(x)⊗2 : x ∼ N (0,Σ) ] = E [( dφ ◦ n(z)
dz
∣∣∣∣ z=Gx G )⊗2 : x ∼ N (0,Σ) ]
= E
[ dφ ◦ n(z)
dz
∣∣∣∣⊗2 z=Gx ◦G⊗2 : x ∼ N (0,Σ) ]
= E
[ dφ ◦ n(z)
dz
∣∣∣∣⊗2 z=Gx : x ∼ N (0,Σ) ] ◦G⊗2
= V[x 7→ dφ◦n(z)dz |z=Gx](Σ) ◦G ⊗2
If we let F(Σ) := V[x7→ dφ◦n(z)dz |z=Gx](Σ), then VB ′ φ (Σ) = F(Σ) ◦ G⊗2. As discussed above, we will seek the eigendecomposition of G⊗2 ◦ F(Σ) HGB : HGB → HGB . We first make some basic calculations. Proposition G.3.
dn(z)
dz ∣∣∣∣ z=y = √ Br−1(I − vvT ),
where r = ‖y‖, v = y/‖y‖ = n(y)/ √ B.
Proof. We have dn(z)dz ∣∣∣ z=y = √ B dz/‖z‖dz ∣∣∣ z=y , and
∂(yi/‖y‖) ∂yj = δij‖y‖ − (∂‖y‖/∂yj)yi ‖y‖2
= δij ‖y‖ − yi ‖y‖2 ∂ √ y · y ∂yj
= δij ‖y‖ − yi ‖y‖2 yj ‖y‖
= δij ‖y‖ − yiyj ‖y‖3
so that dz/‖z‖
dz ∣∣∣∣ z=y = r−1(I − vvT ).
By chain rule, this easily gives Proposition G.4.
dφ ◦ n(z) dz ∣∣∣∣ z=y = √ BDr−1(I − vvT ),
where D = Diag(φ′(n(y))), r = ‖y‖, v = y/‖y‖ = n(y)/ √ B.
With v = y/‖y‖, r = ‖y‖, and D̃ = Diag(φ′(n(y))), we then have F(Σ) = E [ D̃⊗2 ◦ (√ B
dz/‖z‖ dz ∣∣∣∣ z=Gx )⊗2 : x ∼ N (0,Σ) ]
= E D̃⊗2 ◦(√B dz/‖z‖ dz ∣∣∣∣ z=y )⊗2 : y ∼ N (0, GΣG)  = B E [ r−2 ( D̃⊗2 + (D̃vvT )⊗2 − D̃ ⊗ (D̃vvT )− (D̃vvT )⊗ D̃ ) : y ∼ N (0, GΣG)
] (52)
F(Σ){Λ} = B E [ r−2 ( D̃ΛD̃ + D̃vvTΛvvT D̃ − D̃ΛvvT D̃ − D̃vvTΛD̃ ) : y ∼ N (0, GΣG) ] G.1.1 SPHERICAL INTEGRATION
When Σ = Σ∗ is the BSB1 fixed point, one can again easily see that F(Σ∗) is ultrasymmetric. With τij = 1 2 ( δiδ T j + δjδ T i ) (Defn F.20), we have
F(Σ∗){Λ} = B E [ r−2 ( D̃ΛD̃ + D̃vvTΛvvT D̃ − D̃ΛvvT D̃ − D̃vvTΛD̃ ) : y ∼ N (0, µ∗G) ] = µ∗−1B E [ r−2 ( D̃ΛD̃ + D̃vvTΛvvT D̃ − D̃ΛvvT D̃ − D̃vvTΛD̃ ) : y ∼ N (0, G)
] F(Σ∗){τij}kl = µ∗−1B E
rv∼N (0,G)
[ r−2 ( φ′( √ Bvk)φ ′( √ Bvl)
I(k = i & l = j) + I(k = j & l = i) 2
+ vivjvkφ ′( √ Bvk)vlφ ′( √ Bvl)
− 1 2
( I(i = k)φ′( √ Bvk)vjvlφ ′( √ Bvl) + I(j = k)φ′( √ Bvk)vivlφ ′( √ Bvl)
+ I(i = l)φ′( √ Bvl)vjvkφ ′( √ Bvk) + I(j = l)φ′( √ Bvl)vivkφ ′( √ Bvk) ))] = µ∗−1B
Γ((B − 3)/2) Γ((B − 5)/2)2
−2/2π−2 ∫ π
0
dθ1 · · · ∫ π
0
dθ4 f(v θ 1 , . . . , v θ 4) sin B−3 θ1 · · · sinB−6 θ4
= µ∗−1B(B − 5)(2π)−2 ∫ π
0
dθ1 · · · ∫ π
0
dθ4 f(v θ 1 , . . . , v θ 4) sin B−3 θ1 · · · sinB−6 θ4
by Lemma E.35, where we assume, WLOG by ultrasymmetry, k, l ∈ {1, 2}; i, j ∈ {1, . . . , 4}, and
f(v1, . . . , v4) := 1 2 φ′( √ Bev)kφ ′( √ Bev)l ( I(k = i & l = j) + I(k = j & l = i) + 2(ev)i(ev)j(ev)k(ev)l − (I(i = k)(ev)j(ev)l + I(j = k)(ev)i(ev)l + I(i = l)(ev)j(ev)k + I(j = l)(ev)i(ev)k))
vθ1 = cos θ1
vθ2 = sin θ1 cos θ2
vθ3 = sin θ1 sin θ2 cos θ3
vθ4 = sin θ1 sin θ2 sin θ3 cos θ4.
and e as in Eq. (33).
We can integrate out θ3 and θ4 symbolically by Lemma E.29 since their dependence only appear outside of φ′. This reduces each entry of F(Σ∗){τij}kl to 2-dimensional integrals to evaluate numerically. The eigenvalues of G⊗2 ◦ F(Σ∗) can then be obtained from Thm E.62. We omit details here and instead focus on results from other methods.
G.1.2 GEGENBAUER EXPANSION
Notice that when Λ = G, the expression for F(Σ∗){Λ} significantly simplifies because G acts as identity on v:
F(Σ∗){G} = µ∗−1B E [ r−2 ( D̃ΛD̃ + D̃vvTGvvT D̃ − D̃GvvT D̃ − D̃vvTGD̃ ) : y ∼ N (0, G) ] = µ∗−1B E [ r−2 ( D̃GD̃ − D̃vvT D̃ ) : y ∼ N (0, G) ]
Then the diagonal of the image satisfies
F(Σ∗){G}aa = µ∗−1B E [ r−2 ( B − 1 B φ′( √ Bva) 2 − v2aφ′( √ Bva) 2 ) : rv ∼ N (0, G) ] = µ∗−1B(B − 3)−1 E
v∼SB−2 ( B − 1 B φ′( √ Bea,:v) 2 − (ea,:v)2φ′( √ Bea,:v) 2 ) K(v;G, 2)
(by Proposition E.33)
= µ∗−1B(B − 3)−1 E v∼SB−2
( 1− (êa,:v)2 ) B − 1 B φ′( √ B − 1êa,:v)2
(K(v;G, 2) = 1) = µ∗−1(B − 1)(B − 3)−1 E
v∼SB−2
( 1− (êa,:v)2 ) φ′( √ B − 1êa,:v)2
And the off-diagonal entries satisfy
F(Σ∗){G}ab = µ∗−1B E [ r−2 (−1 B φ′( √ Bea,:v)φ ′( √ Beb,:v)
−(ea,:v)(eb,:v)φ′( √ Bea,:v)φ ′(ea,:v) ] : rv ∼ N (0, G) ]
= µ∗−1B(B − 3)−1 E v∼SB−2 (−1 B φ′( √ B − 1êa,:v)φ′( √ B − 1êb,:v)
−B − 1 B (êa,:v)(êb,:v)φ ′( √ B − 1êa,:v)φ′( √ B − 1êb,:v)
) K(v;G, 2)
(by Proposition E.33)
= µ∗−1B(B − 3)−1 E v∼SB−2 −1 B (1 + (B − 1)(êa,:v)(êb,:v))
φ′( √ B − 1êa,:v)φ′( √ B − 1êb,:v)
(K(v;G, 2) = 1) = −µ∗−1(B − 3)−1 E
v∼SB−2 (1 + (B − 1)(êa,:v)(êb,:v))φ′(
√ B − 1êa,:v)φ′( √ B − 1êb,:v)
Lifting Spherical Expectation. We first show a way of expressing F(Σ∗)aa in terms of Gegenbauer basis by lifting the spherical integral over SB−2 to spherical integral over SB . While this technique cannot be extended to F(Σ∗)ab, a 6= b, we believe it could be of use to future related problems.
If φ( √ B − 1x) has Gegenbauer expansion ∑∞l=0 al 1cB−1,lC(B−32 )l (x), then φ′(√B − 1x) =
1√ B−1 ∑∞ l=0 al 1 cB−1,l C (B−32 ) l ′(x) = 1√ B−1 ∑∞ l=0 al B−3 cB−1,l C (B−12 ) l−1 (x), so that φ ′( √ Bea,:v) =
1√ B−1 ∑∞ l=0 al B−3 cB−1,l C (B−12 ) l−1 (êa,:v) = 1√ B−1 ∑∞ l=0 al(B − 3) cB+1,l−1 cB−1,l Z B,(l−1) êa,:
(v). Here êa,: and v are treated as points on the B-dimensional sphere with last 2 coordinates 0.
Thus by Lemmas E.38 and E.39,
F(Σ∗){G}aa
= µ∗−1 ωB−3 ωB−2 (B − 1)(B − 3)−1 ∫ 1 −1 dh (1− h2)B−42 (1− h2)φ′( √ B − 1h)2
= µ∗−1 ωB−3 ωB−2 (B − 1)(B − 3)−1 ωB ωB−1 ωB−1 ωB ∫ 1 −1 dh (1− h2)B−22 φ′( √ B − 1h)2 = µ∗−1 ωB−3ωB ωB−2ωB−1 (B − 1)(B − 3)−1 E w∼SB φ′( √ B − 1˜̂ea,:w)2
= µ∗−1 ωB−3ωB ωB−2ωB−1 (B − 1)(B − 3)−1 E w∼SB ( 1√ B − 1 ∞∑ l=0 al(B − 3) cB+1,l−1 cB−1,l Z B,(l−1)˜̂ ea,: (w) )2
= µ∗−1 ωB−3ωB ωB−2ωB−1 (B − 3) ∞∑ l=0 a2l ( cB+1,l−1 cB−1,l )2 c−1B+1,l−1C (B−12 ) l−1 (1)
= µ∗−1 ωB−3ωB ωB−2ωB−1 (B − 1) ∞∑ l=0 a2l 1 cB−1,l C (B−12 ) l−1 (1)
by Lemma E.45
= µ∗−1(B − 2) ∞∑ l=0 a2l 1 cB−1,l ( l +B − 3 l − 1 )
= µ∗−1(B − 2) ∞∑ l=0 a2l 1 cB−1,l ( l +B − 3 B − 2 )
= µ∗−1 ∞∑ l=0 a2l l +B − 3 cB−1,l ( l +B − 4 B − 3 )
= µ∗−1 ∞∑ l=0 a2l l +B − 3 cB−1,l l B − 3 ( l +B − 4 B − 4 )
Dirichlet-like form of φ. For F(Σ∗){G}ab, there’s no obvious way of lifting the spherical expectation to a higher dimension with the weight (1 + (B − 1)(êa,:v)(êb,:v)). We will instead leverage Thm E.47. In Thm E.47, setting u1 = êa,: = u2, we get
F(Σ∗){G}aa = µ∗−1 B − 1 B − 3 Ev∼SB−2((êa,: · êa,:)− (êa,:v) 2)φ′( √ B − 1êa,:v)2
= µ∗−1 ∞∑ l=0 a2l l +B − 3 B − 3 lc −1 B−1,lC (B−32 ) l (1) .
Likewise, setting u1 = êa,:, u2 = êb,:, we get
F(Σ∗){G}ab = µ∗−1 B − 1 B − 3 Ev∼SB−2((êa,: · êb,:)− (êa,:v)(êb,:v))φ ′( √ B − 1êa,:v)φ′( √ B − 1êb,:v)
= µ∗−1 ∞∑ l=0 a2l l +B − 3 B − 3 lc −1 B−1,lC (B−32 ) l ( −1 B − 1 )
Thus, Theorem G.5. Suppose φ( √ B − 1x), φ′( √ B − 1x) ∈ L2((1 − x2)B−32 ) and φ( √ B − 1x) has
Gegenbauer expansion ∑∞ l=0 al 1 cB−1,l C (B−32 ) l (x). Then for Σ ∗ being the unique BSB1 fixed point
of Eq. (43), the eigenvalue of G⊗2 ◦ F(Σ∗) corresponding to eigenspace RG is
λ↓G = µ ∗−1 ∞∑ l=0 a2l l +B − 3 B − 3 lc −1 B−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B − 1 ))
=
∑∞ l=0 a 2 l l+B−3 B−3 lc −1 B−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1 )) ∑∞ l=0 a 2 l c −1 B−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1
)) > 1
This is minimized (over choices of φ) iff φ is linear, in which case λ↓G = B−2 B−3 .
G.2 LAPLACE METHOD
In this section suppose that φ is degree α positive-homogeneous. Set D = Diag(φ′(y)) (and recall v = y/‖y‖, r = ‖y‖). Then D is degree α − 1 positive-homogeneous in x (because φ′ is). Consequently we can rewrite Eq. (52) as follows, F(Σ∗)
= Bα E y∼N (0,GΣ∗G)
[ r−2 ( r−2(α−1)D⊗2 + r−2(α+1)(DvvT )⊗2
− r−2αD ⊗ (DvvT )− r−2α(DvvT )⊗D )]
= Bα E y∼N (0,µ∗G)
[ r−2αD⊗2 + r−2(α+2)(DyyT )⊗2 − r−2(α+1) ( D ⊗ (DyyT ) + (DyyT )⊗D )] = Bα(A + B− C) (53)
where Σ∗ is the BSB1 fixed point of Eq. (43), GΣ∗G = µ∗G, and A = E [ r−2αD⊗2 : y ∼ N (0, µ∗G) ] B = E [ r−2(α+2)(DyyT )⊗2 : y ∼ N (0, µ∗G)
] C = E [ r−2(α+1) ( D ⊗ (DyyT ) + (DyyT )⊗D ) : y ∼ N (0, µ∗G)
] Each term in the sum above is ultrasymmetric, so has the same eigenspaces RG,M,L (this will also become apparent in the computations below without resorting to Thm E.61). We can thus compute the eigenvalues for each of them in order.
In all computation below, we apply Lemma E.2 first to relate the quantity in question to Vφ.
Computing A. For two matrices Σ,Λ, write Σ Λ for entrywise multiplication. We have by Lemma E.2 A{Λ} = E [ r−2αDΛD : y ∼ N (0, µ∗G) ] = E [ r−2αΛ φ′(y)φ′(y)T : y ∼ N (0, µ∗G)
] = Γ(α)−1
∫ ∞ 0 ds sα−1 det(I + 2sµ∗G)−1/2Λ Vφ′ ( µ∗ 1 + 2sµ∗ G ) = Γ(α)−1
∫ ∞ 0 ds sα−1(1 + 2sµ∗)−(B−1)/2Λ ( µ∗ 1 + 2sµ∗ )α−1 Vφ′(G)
= Γ(α)−1Λ Vφ′(G) ∫ ∞
0
ds (µ∗s)α−1(1 + 2sµ∗)−(B−1)/2−α+1
= Γ(α)−1Λ Vφ′(G) Beta ( B − 3
2 , α
) µ∗−12−α
= Λ cα−1 ( B − 1 B )α−1 BSB1 ( Jφ′(1), Jφ′ ( −1 B − 1 )) P ( B − 3 2 , α )−1 µ∗−12−α
Then Thm E.73 gives the eigendecomposition for BαG⊗2 ◦ A HGB
Theorem G.6. LetRB,α := (2α−1)cα−1B(B−1)α−1P ( B−3 2 , α )−1
2−α. ThenBαG⊗2◦A HGB has the following eigendecomposition.
λAG = RB,αµ ∗−1 ( B − 1 B J′φ(1) + 1 B J′φ ( −1 B − 1 ))
= (2α+B − 3)
( α′2(B − 1)Jφ (1) + (2α− 1)Jφ ( 1
1−B )) (2α− 1)(B − 3)(B − 1) ( J(1)− J ( 1
1−B )) λAL = RB,αµ ∗−1 ( B − 2 B J′φ(1) + 2 B J′φ ( −1 B − 1 ))
= (2α+B − 3)
( α2(B − 2)Jφ (1) + 2(2α− 1)J′φ ( 1 1−B )) (2α− 1)(B − 3)(B − 1) ( J(1)− J ( 1
1−B )) λAM = RB,αµ ∗−1J′φ ( −1 B − 1 )
= B(2α+B − 3)J′φ
( 1
1−B ) (B − 3)(B − 1) ( Jφ(1)− Jφ ( 1
1−B )) Proof. We have BαA = (2α − 1)−1RB,αµ∗−1DOS ( Jφ′(1), 0, Jφ′ ( −1 B−1 )) . This allows us to apply Thm E.73. We make a further simplification Jφ′(c) = (2α−1)J′φ(c) via Proposition E.22.
Computing B. We simplify B{Λ} = E [ r−2(α+2)DyyTΛyyTD : y ∼ N (0, µ∗G) ] = E [ r−2(α+2)ψ(y)yTΛyψ(y)T : y ∼ N (0, µ∗G)
] where ψ(y) := yφ′(y), which, in this case of φ being degree α positive-homogeneous, is also equal to αφ(y).
By Lemma E.2, B{Λ} equals Γ(α+ 2)−1 ∫ ∞
0
ds sα+1 det(I + 2sµ∗G)−1/2 E[ψ(y)yTΛyψ(y)T : y ∼ N (0, µ ∗
1 + 2sµ∗ G)]
This expression naturally leads us to consider the following definition.
Definition G.7. Let ψ : R→ R be measurable and Σ ∈ SB . Define V(4)ψ (Σ) : HB → HB by V
(4) ψ (Σ){Λ} = E[ψ(y)yTΛyψ(y)T : y ∼ N (0,Σ)]
For two matrices Σ,Λ, write 〈Σ,Λ〉 := tr(ΣTΛ). We have the following identity. Proposition G.8. V(4)φ (Σ){Λ} = Vφ(Σ)〈Σ,Λ〉+ 2 dVφ(Σ) dΣ {ΣΛΣ}.
Proof. We have dVφ(Σ)
dΣ {Λ} = (2π)−B/2
∫ dz φ(z)⊗2 ( d
dΣ det(Σ)−1/2e− 1 2 z TΣ−1z
) {Λ}
= (2π)−B/2 ∫ dz φ(z)⊗2 〈[−1
2 det Σ−1/2Σ−1 + det Σ−1/2
1 2 Σ−1zzTΣ−1
] e− 1 2 z TΣ−1z,Λ 〉 = 1
2 (2π)−B/2 det Σ−1/2
∫ dz φ(z)⊗2e− 1 2 z TΣ−1z [ 〈zzT ,Σ−1ΛΣ−1〉 − 〈Σ−1,Λ〉 ] = 1
2 V
(4) φ (Σ){Σ−1ΛΣ−1} −
1 2 Vφ(Σ)〈Σ−1,Λ〉.
Making the substitution Λ→ ΣΛΣ, we get the desired result.
If ϕ is degree α positive-homogeneous, then V(4)ϕ is degree α+ 1 positive-homogeneous. Thus,
B{Λ} = Γ(α+ 2)−1 ∫ ∞
0
ds sα+1 det(I + 2sµ∗G)−1/2V (4) ψ
( µ∗
1 + 2sµ∗ G
) {Λ}
= Γ(α+ 2)−1 ∫ ∞
0
ds sα+1(1 + 2sµ∗)−(B−1)/2 ( µ∗
1 + 2sµ∗
)α+1 V
(4) ψ (G){Λ}
= Γ(α+ 2)−1V (4) ψ (G){Λ} ∫ ∞ 0 ds (µ∗s)α+1(1 + 2sµ∗)−(B−1)/2−α−1
= Γ(α+ 2)−1V (4) ψ (G){Λ}Beta( B − 3 2 , α+ 2)2−α−2µ∗−1
= P
( B − 3
2 , α+ 2
)−1 2−α−2µ∗−1V
(4) ψ (G){Λ}.
By Proposition G.8, V(4)ψ (G){Λ} = Vψ(G) tr Λ + 2 dVψ(Σ) dΣ {Λ} if Λ ∈ HGB . Thus
Theorem G.9. BαG⊗2 ◦ B HGB has the following eigendecomposition (note that here µ∗ is still with respect to φ, not ψ = αφ)
1. Eigenspace RG with eigenvalue
λBG := α2
B − 3
2. Eigenspace L with eigenvalue
λBL := B αP
( B − 3
2 , α+ 2
)−1 2−α−2µ∗−12λG,ψL (B,α)
= 2α3(B − 2) (B − 3)(B − 1)(B − 1 + 2α) + 2α2BJ′φ
( −1 B−1 ) (B − 3)(B − 1)2(B − 1 + 2α)(Jφ(1)− Jφ ( −1 B−1 ) )
3. Eigenspace M with eigenvalue
λBM := B αP
( B − 3
2 , α+ 2
)−1 2−α−2µ∗−12λG,ψM (B,α)
= 2α2BJ′φ
( −1 B−1 ) (B − 3)(B − 1)(B − 1 + 2α)(Jφ(1)− Jφ ( −1 B−1 ) )
Proof. The only thing to justify is the value of λBG. We have
λBG = B αP
( B − 3
2 , α+ 2
)−1 2−α−2µ∗−1
× ( (B − 1)cα ( B − 1 B )α( Jψ(1)− Jψ ( −1 B − 1 )) + 2λG,ψM (B,α) ) = BαP ( B − 3
2 , α+ 2
)−1 2−α−2µ∗−1(B − 1 + 2α)cα ( B − 1 B )α( Jψ(1)− Jψ ( −1 B − 1 )) = cα(B − 1)α(B − 1 + 2α)P ( B − 3
2 , α+ 2
)−1 2−α−2µ∗−1 ( Jψ(1)− Jψ ( −1 B − 1 )) = cα(B − 1)αP ( B − 3
2 , α+ 1
)−1 2−α−1µ∗−1 ( Jψ(1)− Jψ ( −1 B − 1 )) = cα(B − 1)αP ( B − 3
2 , α+ 1
)−1 2−α−1K−1α,Bα 2
(since Jψ = α2Jφ)
= α2cα(B − 1)αP
( B−3 2 , α+ 1 )−1 2−α−1
cαP ( B−1 2 , α )−1 (B−1 2 )α (Defn F.6)
= α22−1
(B − 3)/2
= α2
B − 3
Computing C. By Lemma E.2,
C{Λ} = E [ r−2(α+1) ( DΛyyTD +DyyTΛD ) : y ∼ N (0, µ∗G) ] = Γ(α+ 1)−1 ∫ ds sα det(I + 2sµ∗G)−1/2 E[DΛyyTD +DyyTΛD : y ∼ N (0, µ ∗
1 + 2sµ∗ G)] = Γ(α+ 1)−1 ∫ ds sα(1 + 2sµ∗)−(B−1)/2 ( µ∗
1 + 2sµ∗
)α E[DΛyyTD +DyyTΛD : y ∼ N (0, G)]
= Γ(α+ 1)−1 ∫ ds (µ∗s)α(1 + 2sµ∗)−(B−1)/2−α E[DΛyyTD +DyyTΛD : y ∼ N (0, G)]
= Γ(α+ 1)−1 Beta
( B − 3
2 , α+ 1
) 2−α−1µ∗−1 E[DΛyyTD +DyyTΛD : y ∼ N (0, G)]
= P
( B − 3
2 , α+ 1
)−1 2−α−1µ∗−1 E[DΛyyTD +DyyTΛD : y ∼ N (0, G)]
Lemma G.10. Suppose φ is degree α positive-homogeneous. Then for Λ ∈ HB ,
E[D ⊗ (DyyT ) +DyyT ⊗D : y ∼ N (0,Σ)] = α dVφ(ΠΣΠ) dΠ ∣∣∣∣ Π=I
E[DΛyyTD +DyyTΛD : y ∼ N (0,Σ)] = αdVφ(Σ) dΣ {ΣΛ + ΛΣ}
where D = Diag(φ′(y)).
Proof. Let Πt, t ∈ (− , ) be a smooth path in HB with Π0 = I . Write Dt = Diag(φ′(Πty)), so that D0 = D. Then, using Π̇t to denote t derivative,
d dt Vφ(ΠtΣΠt) = d dt E[φ(Πty)φ(Πty)T : y ∼ N (0,Σ)]
= E[DtΠ̇tyφ(Πty)T + φ(Πty)yT Π̇tDt : y ∼ N (0,Σ)] d dt Vφ(ΠtΣΠt) ∣∣∣∣ t=0 = E[DΠ̇0yφ(y)T + φ(y)yT Π̇0D : y ∼ N (0,Σ)]
Because for x ∈ R, αφ(x) = xφ′(x), for y ∈ RB we can write φ(y) = α−1Diag(φ′(y))y. Then
α dVφ(Σ)
dΣ {ΣΠ̇0 + Π̇0Σ} = α
dVφ(Σ)
dΣ { dΠtΣΠt dt } = α d
dt Vφ(ΠtΣΠt) ∣∣∣∣ t=0
= E[DΠ̇0yyTD +DyyT Π̇0D : y ∼ N (0,Σ)]
Therefore, for Λ ∈ HGB , C{Λ} = P ( B − 3
2 , α+ 1
)−1 2−α−1µ∗−1α dVφ(Σ)
dΣ ∣∣∣∣ Σ=G {GΛ + ΛG}
= 2αP
( B − 3
2 , α+ 1
)−1 2−α−1µ∗−1 dVφ(Σ)
dΣ ∣∣∣∣ Σ=G {Λ}
So Thm F.31 gives Theorem G.11. BαG⊗2 ◦ C has the following eigendecomposition
1. eigenspace RG with eigenvalue
λCG := 2B ααP
( B − 3
2 , α+ 1
)−1 2−α−1µ∗−1λG,φG (B,α)
= 2α2
B − 3
2. eigenspace L with eigenvalue
λCL := 2B ααP
( B − 3
2 , α+ 1
)−1 2−α−1µ∗−1λG,φL (B,α)
= 2α2(B − 2) (B − 3)(B − 1) + 2αBJ′φ
( −1 B−1 ) (B − 3)(B − 1)2(Jφ(1)− Jφ ( −1 B−1 ) )
3. eigenspace M with eigenvalue
λCM := 2B ααP
( B − 3
2 , α+ 1
)−1 2−α−1µ∗−1λG,φM (B,α)
= 2αBJ′φ
( −1 B−1 ) (B − 3)(B − 1)(Jφ(1)− Jφ ( −1 B−1
) Altogether, by Eq. (53) and Thms G.6, G.9 and G.11, this implies
Theorem G.12. G⊗2 ◦ F(Σ∗) : HGB → HGB has eigenspaces RG,M,L respectively with the following eigenvalues
λ↓G = (B − 3 + 2α)
( (2α− 1)J′φ ( −1 B−1 ) + α2(B − 1)Jφ(1) ) (2α− 1)(B − 3)(B − 1)(Jφ(1)− Jφ ( −1 B−1 ) ) − α 2 B − 3
= α2(B − 3 + 2α)
( Jφ(1) + 1 B−1Jα−1φ′ ( −1 B−1 )) (2α− 1)(B − 3)(Jφ(1)− Jφ ( −1 B−1 ) ) − α 2 B − 3
= α2 ( (B − 2)Jφ(1) + (B − 3 + 2α) 1B−1Jα−1φ′ ( −1 B−1 ) + (2α− 1)(Jφ(1)− Jφ ( −1 B−1 )) (2α− 1)(B − 3)(Jφ(1)− Jφ ( −1 B−1 ) )
λ↓L = − 2α2(B − 2)(B − 1 + α)
(B − 3)(B − 1)(B − 1 + 2α)
+ 2 α2(3B − 4) + α
( 3B2 − 11B + 8 ) + (B − 3)(B − 1)2
(B − 3)(B − 1)2(B − 1 + 2α) J′φ
( −1 B−1 ) Jφ(1)− Jφ ( −1 B−1
) + α2(B − 2)(B − 3 + 2α) (2α− 1)(B − 3)(B − 1) Jφ(1)
Jφ(1)− Jφ ( −1 B−1 ) λ↓M = B(B2 + 2(α− 2)B + 2(α− 3)α+ 3) (B − 3)(B − 1)(B − 1 + 2α) J′φ ( −1 B−1 ) Jφ(1)− Jφ ( −1 B−1
) . H CROSS BATCH: FORWARD DYNAMICS
In this section, we study the generalization of Eq. (43) to multiple batches. Definition H.1. For linear operators Ti : Xi → Yi, i = 1, . . . , k, we write ⊕ i Ti : ⊕ i Xi → ⊕ i Yi
for the operator (x1, . . . , xk) 7→ (T1(x1), . . . , Tk(xk)). We also write T ⊕n1 := ⊕n
j=1 T1 for the direct sum of n copies of T1.
For k ≥ 2, now consider the extended (“k-batch”) dynamics on Σ̃ ∈ SkB defined by Σ̃l = VB⊕kφ (Σ̃l−1) (54)
= E[(Bφ(h1:B),Bφ(hB+1:2B), . . . ,Bφ(h(k−1)B+1:kB))⊗2 : h ∼ N (0, Σ̃l−1)] If we restrict the dynamics to just the upper left B × B submatrix (as well as any of the diagonal B ×B blocks) of Σ̃l, then we recover Eq. (43).
H.1 LIMIT POINTS
In general, like in the case of Eq. (43), it is difficult to prove global convergence behavior. Thus we manually look for fixed points and the local convergence behaviors around them. A very natural extension of the notion of BSB1 matrices is the following Definition H.2. We say a matrix Σ̃ ∈ SkB is CBSB1 (short for “1-Step Cross-Batch Symmetry Breaking”) if Σ̃ in block form (k × k blocks, each of size B ×B) has one common BSB1 block on the diagonal and one common constant block on the off-diagonal, i.e. BSB1(a, b) c11T c11T · · · c11T BSB1(a, b) c11T · · · c11T c11T BSB1(a, b) · · ·
... ...
... . . .  We will study the fixed point Σ̃∗ to Eq. (54) of CBSB1 form.
H.1.1 SPHERICAL INTEGRATION AND GEGENBAUER EXPANSION
Theorem H.3. Let Σ̃ ∈ SkB . If Σ̃ is CBSB1 then VB⊕kφ (Σ̃) is CBSB1 and equals Σ̃ ∗ :=(
Σ∗ c∗ c∗ Σ∗
) where Σ∗ is the BSB1 fixed point of Thm F.5 and c∗ ≥ 0 with √ c∗ =
Γ((B−1)/2) Γ((B−2)/2)π −1/2 ∫ π 0 dθ φ(− √ B − 1 cos θ) sinB−3 θ. If φ(
√ B − 1x) has Gegenbauer expansion∑∞
l=0 al 1 cB−1,l C
(B−32 )
l (x), then c ∗ = a20.
Proof. We will prove for k = 2. The general k cases follow from the same reasoning. Let Σ̃ = ( Σ c11T
c11T Σ
) where Σ = BSB1(a, b). As remarked below Eq. (54), restricting to any
diagonal blocks just recovers the dynamics of Eq. (43), which gives the claim about the diagonal blocks being Σ∗ through Thm F.5.
We now look at the off-diagonal blocks.
VB⊕2φ (Σ̃)1:B,B+1:2B = E[Bφ(z1:B)⊗ Bφ(zB+1:2B) : z ∼ N (0, Σ̃)]
= E[φ ◦ n(y1:B)⊗ φ ◦ n(yB+1:2B) : y ∼ N (0, Σ̃G)] where Σ̃G := G⊕2Σ̃G⊕2 = ( G 0 0 G ) Σ̃ ( G 0 0 G ) = ( GΣG cG11TG cG11TG GΣG ) =(
(a− b)G 0 0 (a− b)G ) (the last step follows from Lemma E.52). Thus y1:B is independent from
yB+1:2B , and
VB⊕2φ (Σ̃)1:B,B+1:2B = E[φ ◦ n(x) : x ∼ N (0, (a− b)G)]⊗2
By symmetry,
E[φ ◦ n(x) : x ∼ N (0, (a− b)G)] = √ c∗1
E[φ ◦ n(x) : x ∼ N (0, (a− b)G)]⊗2 = c∗11T
where √ c∗ := E[φ(n(x)1) : x ∼ N (0, (a− b)G)]. We can compute√
c∗ = E[φ(n(x)1) : x ∼ N (0, G)] because n is scale-invariant
= E[φ(n(ex)1) : x ∼ N (0, IB−1)] where e is as in Eq. (33)
= E[φ((en(x))1) : x ∼ N (0, IB−1)] because e is an isometry = E[φ( √ B(ev)1) : x ∼ N (0, IB−1)]
with v = x/‖x‖
= Γ((B − 1)/2) Γ((B − 2)/2)π
−1/2 ∫ π
0
dθ φ(− √ B − 1 cos θ) sinB−3 θ
by Lemma E.37. At the same time, by Proposition E.33, we can obtain the Gegenbauer expansion√ c∗ = E
v∼SB−2 φ( √ B − 1ê1,:v)
= E v∼SB−2 ∞∑ l=0 al 1 cB−1,l C (B−32 ) l (e1,:v)
= 〈 ∞∑ l=0 alZ B−2,(l) ê1,: , 1 〉 = a0Z B−2,(0) ê1,: (ê1,:)
= a0.
Corollary H.4. With the notation as in Thm H.3, if φ is positive-homogeneous of degree α and φ(x) = aρα(x)− bρα(−x), then √ c∗ = (a− b) 1
2 √ π (B − 1)α/2 Γ((B−1)/2)Γ((α+1)/2)Γ((α+B−1)/2) .
Proof. We compute
∫ π/2 0 dθ (cos θ)α sinB−3 θ = 1 2 Beta ( 0, 1; α+ 1 2 , B − 2 2 ) by Lemma E.28
= 1
2 Beta
( α+ 1
2 , B − 2 2 ) ∫ π π/2 dθ (− cos θ)α sinB−3 θ = ∫ π/2 0 dθ (cos θ)α sinB−3 θ
= 1
2 Beta
( α+ 1
2 , B − 2 2 ) So for a positive homogeneous function φ(x) = aρα(x)− bρα(−x),
√ c∗ = Γ((B − 1)/2) Γ((B − 2)/2)π
−1/2 ( a ∫ π π/2 dθ (− √ B − 1 cos θ)α sinB−3 θ
− b ∫ π/2
0
dθ ( √ B − 1 cos θ)α sinB−3 θ ) = (a− b)Γ((B − 1)/2)
Γ((B − 2)/2)π −1/2(B − 1)α/2 1 2 Beta
( α+ 1
2 , B − 2 2 ) = (a− b) 1
2 √ π (B − 1)α/2 Γ((B − 1)/2)Γ((α+ 1)/2) Γ((α+B − 1)/2)
Expanding the beta function and combining with Thm H.3 gives the desired result.
H.1.2 LAPLACE METHOD
While we don’t need to use the Laplace method to compute c∗ for positive homogeneous functions (since it is already given by Corollary H.4), going through the computation is instructive for the machinery for computing the eigenvalues in a later section.
Lemma H.5 (The Laplace Method Cross Batch Master Equation). For A,B,C ∈ N, let f : RA+B → RC and let a, b ≥ 0. Suppose for any y ∈ RA, z ∈ RB , ‖f(y, z)‖ ≤ h( √ ‖y‖2 + ‖z‖2) for some nondecreasing function h : R≥0 → R≥0 such that E[h(r‖z‖) : z ∈ N (0, IA+B)] exists for every r ≥ 0. Define ℘(Σ) := E[‖y‖−2a‖z‖−2bf(y, z) : (y, z) ∼ N (0,Σ)]. Then on {Σ ∈ SA+B : rank Σ > 2(a+ b)}, ℘(Σ) is well-defined and continuous, and furthermore satisfies
℘(Σ) = Γ(a)−1Γ(b)−1 ∫ ∞
0
ds ∫ ∞ 0 dt sa−1tb−1 det(IA+B + 2Ω) −1/2 E (y,z)∼N (0,Π) f(y, z) (55)
where D = (√
sIA 0 0 √ tIB
) , Ω = DΣD, and Π = D−1Ω(I + 2Ω)−1D−1.
Proof. If Σ is full rank, we can show Fubini-Tonelli theorem is valid in the following computation by the same arguments of the proof of Lemma E.2.
E[‖y‖−2a‖z‖−2bf(y, z) : (y, z) ∼ N (0,Σ)] = E[f(y, z) ∫ ∞
0
ds Γ(a)−1sa−1e−‖y‖ 2s ∫ ∞ 0 dt Γ(b)−1tb−1e−‖y‖ 2t : (y, z) ∼ N (0,Σ)]
= (2π)− A+B 2 Γ(a)−1Γ(b)−1 det Σ−1/2 × ∫ ∞
0
ds ∫ ∞ 0 dt sa−1tb−1 ∫ RA+B dy dz f(y, z)e− 1 2 (y,z)(Σ −1+2D2)(y,z)T
(Fubini-Tonelli) = Γ(a)−1Γ(b)−1 ∫ ∞
0
ds ∫ ∞ 0 dt sa−1tb−1 det(Σ(Σ−1 + 2D2))−1/2 E f(y, z)
where in the last line, (y, z) ∼ N (0, (Σ−1 + 2D2)−1). We recover the equation in question with the following simplifications.
(Σ−1 + 2D2)−1 = Σ(IA+B + 2D 2Σ)−1
= Σ(D−1 + 2DΣ)−1D−1
= ΣD(IA+B + 2DΣD) −1D−1
= D−1Ω(IA+B + 2Ω) −1D−1
det(Σ(Σ−1 + 2D2)) = det(IA+B + 2ΣD 2)
= det(IA+B + 2DΣD)
= det(IA+B + 2Ω)
The case of general Σ with rank Σ > 2(a + b) is given by the same continuity arguments as in Lemma E.2.
Let Σ̃ ∈ S2B , Σ̃ = (
Σ Ξ ΞT Σ′
) where Σ,Σ′ ∈ SB and Ξ ∈ RB×B . Consider the off-diagonal block
of VB⊕2φ (Σ̃).
E[Bφ(z)⊗ Bφ(z′) : (z, z′) ∼ N (0, Σ̃)] = E[φ(n(y))⊗ φ(n(y′)) : (y, y′) ∼ N (0, Σ̃G)] = Bα E[‖y‖−α‖y′‖−αφ(y)⊗ φ(y′) : (y, y′) ∼ N (0, Σ̃G)]
= BαΓ(α/2)−2 ∫ ∞
0
ds ∫ ∞ 0 dt (st)α/2−1 det(I2B + 2Ω) −1/2 E (y,y′)∼N (0,Π) φ(y)⊗ φ(y′)
(by Lemma H.5) = BαΓ(α/2)−2 ∫ ∞
0
ds ∫ ∞ 0 dt (st)α/2−1 det(I2B + 2Ω) −1/2Vφ(Π)1:B,B+1:2B (56)
where Ω = ( sΣG √ stΞG√
st(ΞG)T tΣ′G
) and Π = D−1Ω(I + 2Ω)−1D−1 with D = √ sIB ⊕
√ tIB =(√
sIB 0 0 √ tIB
) .
Theorem H.6 (Rephrasing and adding to Corollary H.4). Let Σ̃ ∈ SkB and φ be positive homogeneous of degree α. If Σ̃ is CBSB1 then VB⊕kφ (Σ̃) is CBSB1 and equals Σ̃ ∗ =
( Σ∗ c∗11T
c∗11T Σ∗ ) where Σ∗ is the BSB1 fixed point of Thm F.5 and
c∗ = cα
( B − 1
2
)α P ( B − 1
2 , α 2
)−2 Jφ(0)
= (a− b)2 1 4π
(B − 1)α (
Γ((B − 1)/2)Γ((α+ 1)/2) Γ((α+B − 1)/2)
)2
Proof. Like in the proof of Corollary H.4, we only need to compute the cross-batch block, which is given above by Eq. (56). Recall that Σ̃ = ( Σ c11T
c11T Σ
) where Σ = BSB1(a, b). Set µ := a− b.
Note that because the off-diagonal block Ξ = c11T by assumption, ΞG = 0, so that
Ω = sΣG ⊕ tΣG = sµG⊕ tµG
Ω(I + 2Ω)−1 = sµ 1 + 2sµ G⊕ tµ 1 + 2tµ G
Π = µ 1 + 2sµ G⊕ µ 1 + 2tµ G
Therefore,
det(I + 2Ω) = (1 + 2sµ)B−1(1 + 2tµ)B−1
Vφ(Π) = µ α ( B − 1 B )α ∆Vφ ( BSB1(1, −1B−1 ) 0
0 BSB1(1, −1B−1 )
) ∆
where ∆ = (
(1 + 2sµ)−α/2IB 0 0 (1 + 2tµ)−α/2IB
) . In particular, the off-diagonal block is con-
stant with entry cαµα ( B−1 B )α (1 + 2sµ)−α/2(1 + 2tµ)−α/2Jφ(0).
Finally, by Eq. (56),
E[Bφ(z)⊗ Bφ(z′) : (z, z′) ∼ N (0, Σ̃)] = BαΓ(α/2)−2 ∫ ∞
0
ds ∫ ∞ 0 dt (st)α/2−1 det(I2B + 2Ω) −1/2Vφ(Π)1:B,B+1:2B
= BαΓ(α/2)−2 ∫ ∞
0
ds ∫ ∞ 0 dt (st)α/2−1[(1 + 2sµ)(1 + 2tµ)]− B−1 2
× cαµα ( B − 1 B )α [(1 + 2sµ)(1 + 2tµ)]−α/2Jφ(0)11 T
= cα(B − 1)αΓ(α/2)−2Jφ(0)11T ∫ ∞
0
dσ ∫ ∞ 0 dτ (στ)α/2−1(1 + 2σ)− B−1+α 2 (1 + 2τ)− B−1+α 2
(with σ = µs, τ = µt) = cα(B − 1)αΓ(α/2)−2Jφ(0)11T ( 2−α/2 Beta ( B − 1
2 , α 2 ))2 = cα ( B − 1
2
)α P ( B − 1
2 , α 2
)−2 Jφ(0)11 T
Simplification with Defn E.7 and Proposition E.21 gives the result.
H.2 LOCAL CONVERGENCE
We now look at the linearized dynamics around CBSB1 fixed points of Eq. (54), and investigate the eigen-properties of dV B⊕2 φ
dΣ̃ ∣∣∣∣ Σ̃=Σ̃∗ . By Lemma F.17, its nonzero eigenvalues are exactly those
of F̃ := (G⊕2)⊗2 ◦ dV(φ◦n)⊕2 (Σ̃ G)
dΣ̃G
∣∣∣∣ Σ̃G=µ∗G⊕2 , F̃{Λ} = G⊕2 ( dV(φ◦n)⊕2 dΣ̃G ∣∣∣ µ∗G⊕2 {Λ} ) G⊕2. Here
Σ̃G = G⊕2Σ̃G⊕2. Thus it suffices to obtain the eigendecomposition of F̃ . First, notice that F̃ acts on the diagonal blocks as G⊗2 ◦ dVφ◦n
dΣG , and the off-diagonal components of
Λ has no effect on the diagonal blocks of F̃{Λ}. We formalize this notion as follows
Definition H.7. Let T : HkB → HkB be a linear operator such that for any block matrix Σ̃ ∈ HkB with k × k blocks Σ̃ij , i, j ∈ [k], of size B, we have
T {Σ̃}ii = U{Σ̃ii} T {Σ̃}ij = V{Σ̃ii}+ V{Σ̃jj}T +W{Σ̃ij},∀i 6= j,
where U : HB → HB ,V : HB → RB×B , and W : RB×B → RB×B are linear operators, and T {Σ̃}ij denotes the (i, j)th block of T {Σ̃}. We say that T is blockwise diagonal-off-diagonal semidirect, or BDOS for short. We write more specifically T = BDOS(U ,V,W).
We have F̃ = BDOS(U ,V,W) where
U = G⊗2 ◦ dVφ◦n dΣG ∣∣∣∣ ΣG=µ∗G
V = G⊗2 ◦ d d(Σ̃G)11 E (h,h′)∼N (0,Σ̃G) φ(n(h))⊗ φ(n(h′)) ∣∣∣∣∣ Σ̃G=µ∗G⊕2
W = 2G⊗2 ◦ d d(Σ̃G)12 E (h,h′)∼N (0,Σ̃G) φ(n(h))⊗ φ(n(h′)) ∣∣∣∣∣ Σ̃G=µ∗G⊕2 . (57)
Note that the factor 2 in W is due to the contribution of (Σ̃G)21 by symmetry. We will first show thatW is multiplication by a constant.
H.2.1 SPHERICAL INTEGRATION AND GEGENBAUER EXPANSION
Theorem H.8. Suppose φ( √ B − 1x) has Gegenbauer expansion ∑∞l=0 al 1cB−1,lC(B−32 )l (x). Then for any Λ ∈ RB×B satisfying GΛG = Λ,
W{Λ} = 2a21µ∗−1P ( B − 1
2 ,
1
2
)2 B
B − 1Λ
= a21
1 2P ( B 2 , 1 2 )−2 B(B − 1)∑∞
l=0 a 2 l 1 cB−1,l
( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1 ))Λ Proof. Let J = d
d(Σ̃G)12 E(h,h′)∼N (0,Σ̃G) φ(n(h))⊗ φ(n(h′)) ∣∣∣ Σ̃G=µ∗G⊕2 . Combining Lemma F.18
with projection by e, we get
J{Λ} = 1 2 E y,y′∼N (0,µ∗IB−1) φ (n(ey))⊗ φ (n(ey′)) 〈µ∗−2yy′T − µ∗−1IB−1, eTΛe〉
= 1
2 µ∗−1 E y,y′∼N (0,IB−1) φ (n(ey))⊗ φ (n(ey′)) (y′TeTΛey − tr Λ)
= 1 2 µ∗−1
((√ 2P ( B − 1
2 ,
1
2
))2 E v,v′∼SB−2 φ (√ Bev ) ⊗ φ (√ Bev′ ) ( v′TeTΛev ) − (tr Λ) E v,v′∼SB−2 φ (√ Bev ) ⊗ φ (√ Bev′
)) by Proposition E.33.
Note that Ev,v′∼SB−2 φ (√ Bev ) ⊗φ (√ Bev′ ) = ( Ev∼SB−2 φ (√ Bev ))⊗2 is a constant matrix, because of the independence of v from v′ and the spherical symmetry of the integrand. Thus this term drops out in G⊗2 ◦ J{Λ}.
Next, we analyze H{Λ} = Ev,v′∼SB−2 φ (√ Bev ) ⊗ φ (√ Bev′ ) ( v′TeTΛev ) . This is a linear
function in Λ with coefficients
Hab|cd = B − 1 B
( E
v∼SB−2 φ( √ B − 1êb,:v)(êc,:v)
)( E
v∼SB−2 φ( √ B − 1êa,:v)(êd,:v)
) .
By the usual zonal spherical harmonics argument,
E v∼SB−2
φ( √ B − 1êb,:v)(êc,:v)
= 〈 ∞∑ l=0 al 1 cB−1,l C (B−32 ) l (〈êb,:, ·〉), 〈êc,:, ·〉 〉 SB−2
= 〈 ∞∑ l=0 alZ B−2,(l) êb,: , 1 B − 1Z B−2,(1) êc,: 〉 SB−2 = a1
B − 1Z B−2,(1) êc,: (êb,:)
= a1 B − 1 1 cB−1,1 C (B−32 ) 1 (〈êb,:, êc,:〉)
= { a1 if b = c −a1 B−1 otherwise.
Thus,
H{Λ}ab = B − 1 B (( −a1 B − 1 )2 ∑ c,d∈[B] Λcd
+ ( a1 −
−a1 B − 1 )( −a1 B − 1 ) ∑ c=b XOR d=a Λcd
+ ( a21 −
a21 (B − 1)2
) Λab ) = B − 1 B ( −2 ( a1 − −a1 B − 1 )( −a1 B − 1 ) Λab + ( a21 − a21 (B − 1)2 ) Λab
) because rows and columns of Λ sum to 0
= a21 B
B − 1Λab. Therefore,
W{Λ} = 2G⊗2 ◦ J{Λ}
= 2a21µ ∗−1P
( B − 1
2 ,
1
2
)2 B
B − 1Λ
= a21
1 2P ( B 2 , 1 2 )−2 B(B − 1)∑∞
l=0 a 2 l 1 cB−1,l
( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1 ))Λ where we used the identity P ( B−1
2 , 1 2
) ( 2
B−1
) = P ( B 2 , 1 2 )−1 .
Next, we step back a bit to give a full treatment of the eigen-properties of BDOS operators. Then we specialize back to our case and give the result for F̃ . Definition H.9. Let HGkB := {Σ̃ ∈ HkB : G⊕kΣ̃G⊕k = Σ̃} be the subspace of HkB stable under conjugation by G⊕k.
Definition H.10. Let M̃kB ∈ HGkB be the space of block matrices Σ̃ ∈ HGkB where the diagonal blocks Σ̃ii are all zero. Likewise, let M̃kB ∈ HkB be the space of block matrices Σ̃ ∈ HkB where the diagonal blocks Σ̃ii are all zero. We suppress the subscript kB when it is clear from the context. Definition H.11. Define the block L-shaped matrix
LkB(Σ,Λ) :=  Σ −Λ · · ·−ΛT 0 · · ·... ... . . . 
BDOS operators of the form BDOS(U ,V, w), where w stands for multiplication by a constant w, like F̃ , have simple eigendecomposition, just like DOS operators. Theorem H.12. If V −wIB is not singular, then BDOS(V,U , w) : HkB → HkB has the following eigenspaces and eigenvectors
1. M̃ with associated eigenvalue w.
2. For each eigenvalue λ and eigenvector Σ ∈ HB of U , the block matrix
LkB(Σ,−(λ− w)−1V{Σ}) =  Σ (λ− w) −1V{Σ} · · ·
(λ− w)−1V{Σ}T 0 · · · ... ... . . .  along with any matrix obtained from permutating its block columns and rows simultaneously. The associated eigenvalue is λ.
If BDOS(V,U , w) also restricts to HGkB → HGkB , then with respect to this signature, the eigendecomposition is obtained by replacingHkB withHGkB in the above:
1. Block diagonal matrices Σ̃ ∈ HGkB where the diagonal blocks Σ̃ii are all zero. The associated eigenvalue is w.
2. For each eigenvalue λ and eigenvector Σ ∈ HGB of U , the matrix LkB(Σ,−(λ− w)−1V{Σ}) along with any matrix obtained from permutating its block columns and rows simultaneously. The associated eigenvalue is λ.
Proof. The proofs are similar in both cases: One can easily verify that these are eigenvectors and eigenvalues. Then by dimensionality considerations, these must be all of them.
By Thms H.8 and H.12, we easily obtain Theorem H.13. Suppose φ( √ B − 1x) has Gegenbauer expansion ∑∞l=0 al 1cB−1,lC(B−32 )l (x). The eigendecomposition of F̃ : HG2B → HG2B is given below, as long as λ↑M̃ 6= λ ↑ L, λ ↑ M.
1. M̃ with associated eigenvalue
λ↑ M̃ := 2a21µ ∗−1P
( B − 1
2 ,
1
2
)2 B
B − 1
= a21
1 2P ( B 2 , 1 2 )−2 B(B − 1)∑∞
l=0 a 2 l 1 cB−1,l
( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1 )) . 2. The space generated by L2B(G,−(λ↑L − λ ↑ M̃ ) −1V{G}) along with any matrix obtained
from permutating its block columns and rows simultaneously. The associated eigenvalue is 0.
3. The space generated by L2B(L,−(λ↑L − λ ↑ M̃ ) −1V{L}), where L = LB(B − 2, 1), along
with any matrix obtained from permutating its block columns and rows simultaneously. The associated eigenvalue is λ↑L.
4. The space generated by L2B(M,−(λ↑M − λ ↑ M̃ ) −1V{M}), for any M ∈ M, along with
any matrix obtained from permutating its block columns and rows simultaneously. The associated eigenvalue is λ↑M.
Here, V is as in Eq. (57). The local convergence dynamics of Eq. (54) to the CBSB1 fixed point Σ̃∗ has the same eigenvalues λ↑
M̃ , λ↑L, λ ↑ M, and 0.
Note that, to reason about F̃ extended over k batches, one can replace 2B with kB everywhere in Thm H.13 and the theorem will still hold. As we will see below, λ↑
M̃ is the same as the eigenvalue
governing the dynamics of the cross batch correlations in the gradient propagation.
H.2.2 LAPLACE METHOD
As with the single batch case, with positive homogeneous φ, we can obtain a more direct answer of the new eigenvalue λ↑
M̃ in terms of the J function of φ.
Theorem H.14. Let φ be degree α positive homogeneous. Then the eigenvalue of F̃ on M̃ is
λ↑ M̃ = B
2
( B − 1
2
)α−1 P ( B
2 , α 2
)−2 µ∗−1cαJ ′ φ(0)
= B B − 1 P ( B−1 2 , α )
P ( B 2 , α 2 )2 J′φ(0) Jφ(1)− Jφ ( −1 B−1 )
Proof. Let Σ̃∗ = ( Σ∗ c∗11T
c∗11T Σ∗
) be the CBSB1 fixed point in Thm H.6. Take a smooth path
Σ̃τ = ( Στ Ξτ ΞTτ Σ ′ τ ) ∈ S2B , τ ∈ (− , ) such that Σ̃0 = Σ̃∗, ˙̃Σ0 := ddτ Σ̃ = ( 0 Υ ΥT 0 ) for some
Υ ∈ RB×B . Then with Ωτ = ( sΣGτ √ stΞGτ√
st(ΞGτ ) T tΣ′τ G
) and Πτ = D−1Ωτ (I + 2Ωτ )−1D−1 with
D = √ sIB ⊕ √ tIB = (√ sIB 0 0 √ tIB ) ,
d
dτ E[Bφ(z)⊗ Bφ(z′) : (z, z′) ∼ N (0, Σ̃)] ∣∣∣∣ τ=0
= d
dτ BαΓ(α/2)−2 ∫ ∞ 0 ds ∫ ∞ 0 dt (st)α/2−1 det(I2B + 2Ω) −1/2Vφ(Π)1:B,B+1:2B ∣∣∣∣ τ=0
= BαΓ(α/2)−2 ∫ ∞
0
ds ∫ ∞ 0 dt (st)α/2−1 ( d dτ det(I2B + 2Ω) −1/2 ∣∣∣∣ τ=0 Vφ(Π)1:B,B+1:2B
+ det(I2B + 2Ω) −1/2 d
dτ Vφ(Π)1:B,B+1:2B ∣∣∣∣ τ=0 ) = BαΓ(α/2)−2
∫ ∞ 0 ds ∫ ∞ 0 dt (st)α/2−1 ( − det(I + 2Ω0)−1/2 tr ( (I + 2Ω0) −1Ω̇0 ) Vφ(Π)
+ det(I + 2Ω0) −1/2(D−α)⊗2 ◦ J
{ d
dτ Ω(I + 2Ω)−1 ∣∣∣∣ τ=0 }) 1:B,B+1:2B
(58)
by chain rule and Lemma F.26, where J = dVφdΣ ∣∣∣∣ Σ=Ω0(I+2Ω0)−1 . We compute
Ω0 = µ ∗(sG⊕ tG) Ω̇0 = √ st ( 0 ΥG
(ΥG)T 0 ) (I + 2Ω0) −1 = ((1 + 2sµ∗)−1G+ 1
B 11T )⊕ ((1 + 2tµ∗)−1G+ 1 B 11T )
det(I + 2Ω0) = (1 + 2sµ ∗)B−1(1 + 2tµ∗)B−1
Ω0(I + 2Ω0) −1 =
sµ∗
1 + 2sµ∗ G⊕ tµ
∗
1 + 2tµ∗ G.
So then J = (( sµ∗
1 + 2sµ∗
)(α−1)/2 IB ⊕ ( tµ∗
1 + 2tµ∗
)(α−1)/2 IB )⊗2 ◦ ( B − 1 B )α−1 dVφ dΣ ∣∣∣∣ Σ=BSB1(1, −1B−1 ) ⊕2
With d
dτ Ω(I + 2Ω)−1 ∣∣∣∣ τ=0 = (I + 2Ω0) −1Ω̇0(I + 2Ω0) −1
(by Lemma F.27)
=
√ st
(1 + 2sµ∗)(1 + 2tµ∗)
( 0 ΥG
(ΥG)T 0 ) we have
J { d
dτ Ω(I + 2Ω)−1 ∣∣∣∣ τ=0 } = ( B − 1 B )α−1 √ st (1 + 2sµ∗)(1 + 2tµ∗) cαJ ′ φ(0)
× ( sµ∗
1 + 2sµ∗
)(α−1)/2( tµ∗
1 + 2tµ∗
)(α−1)/2( 0 ΥG
(ΥG)T 0 ) = ( B − 1 B )α−1 (st)α/2µ∗α−1cαJ′φ(0) (1 + 2sµ∗)(α+1)/2(1 + 2tµ∗)(α+1)/2 ( 0 ΥG (ΥG)T 0 ) .
Thus the product (I + 2Ω0)−1Ω̇0 has zero diagonal blocks so that its trace is 0. Therefore, only the second term in the sum in Eq. (58) above survives, and we have
d
dτ E[Bφ(z)⊗ Bφ(z′) : (z, z′) ∼ N (0, Σ̃)] ∣∣∣∣ τ=0
= BαΓ(α/2)−2 ∫ ∞
0
ds ∫ ∞ 0 dt (st)α/2−1(1 + 2sµ∗)− B−1 2 (1 + 2tµ∗)− B−1 2 ×
(st)−α/2 ( B − 1 B )α−1 (st)α/2µ∗α−1cαJ′φ(0) (1 + 2sµ∗)(α+1)/2(1 + 2tµ∗)(α+1)/2 ΥG
= B(B − 1)α−1Γ(α/2)−2 ∫ ∞
0
ds ∫ ∞ 0 dt (st)α/2−1 µ∗α−1cαJ ′ φ(0) (1 + 2sµ∗)(B+α)/2(1 + 2tµ∗)(B+α)/2 ΥG
= B(B − 1)α−1Γ(α/2)−2µ∗−1cαJ′φ(0) (∫ ∞
0
d(µ∗s) (µ∗s)α/2−1
(1 + 2sµ∗)(B+α)/2
)2 ΥG
= B(B − 1)α−1Γ(α/2)−2µ∗−1cαJ′φ(0) ( 2−α/2 Beta ( B
2 , α 2
))2 ΥG
= B
2
( B − 1
2
)α−1 P ( B
2 , α 2
)−2 µ∗−1cαJ ′ φ(0)Υ G
I CROSS BATCH: BACKWARD DYNAMICS
For k ≥ 2, we wish to study the following generalization of Eq. (51), Π̃l = VB⊕kφ ′(Σ̃l)†{Π̃l+1}.
As in the single batch case, we approximate it by taking Σ̃l to its CBSB1 limit Σ̃∗, so that we analyze
Π̃l = VB⊕kφ ′(Σ̃∗)†{Π̃l+1} (59)
Its dynamics is then given by the eigendecomposition of VB⊕kφ ′(Σ̃∗), in parallel to the single batch scenario.
WLOG, we only need to consider the case k = 2.
VB⊕2φ ′(Σ̃∗) = E (x,y)∼N (0,Σ̃∗) [(B′φ(x)⊕ B′φ(y))⊗2]
VB⊕2φ ′(Σ̃∗)
{( Σ Ξ
ΞT Λ
)} = E
(x,y)∼N (0,Σ̃∗) [( B′φ(x)ΣB′φ(x)T B′φ(x)ΞB′φ(y)T B′φ(y)ΞTB′φ(x)T B′φ(y)ΛB′φ(y)T )] From this one sees that V[(B⊕2φ )
′] acts independently on each block, and consequently so does its adjoint. The diagonal blocks of Eq. (59) evolves according to Eq. (51) which we studied in Appendix G. In this section we will study the evolution of the off-diagonal blocks, which is given by
Ξl = E (x,y)∼N (0,Σ̃∗) [B′φ(x)⊗ B′φ(y)]†{Ξl+1} = E (x,y)∼N (0,Σ̃∗) [B′φ(y)T ⊗ B′φ(x)T ]{Ξl+1} (60)
= E (x,y)∼N (0,Σ̃∗) [B′φ(y)TΞl+1B′φ(x)T ]
Definition I.1. Let MB := RB×B be the set of all B × B real matrices, and let MGB := {Ξ ∈ MB : GΞG = Ξ}.
Define T := E(x,y)∼N (0,Σ̃∗)[B′φ(x)⊗B′φ(y)]. As in the single batch case, after one step of backprop, ΞL−1 = V†{ΞL} is inMGB . Thus it suffices to study the eigendecomposition of G⊗2 ◦ T † MGB : MGB →MGB . Below, we will show that its adjoint T ◦G⊗2 acting onMGB is in fact multiplication by a constant, and thus so is it.
We first compute
T = E (ξ,η)∼N (0,G⊕2Σ̃∗G⊕2)
[ dφ ◦ n(z)
dz
∣∣∣∣ z=ξ ⊗ dφ ◦ n(z) dz ∣∣∣∣ z=η ] ◦G⊗2
= E (ξ,η)∼N (0,µ∗(G 00 G ))
[ dφ ◦ n(z)
dz
∣∣∣∣ z=ξ ⊗ dφ ◦ n(z) dz ∣∣∣∣ z=η ] ◦G⊗2
since Σ̃∗ is CBSB1 with diagonal blocks Σ∗ (Corollary H.4). Then ξ is independent from η, so this is just
E (x,y)∼N (0,Σ̃∗) [B′φ(x)⊗ B′φ(y)] = E ξ∼N (0,µ∗G)
[ dφ ◦ n(z)
dz
∣∣∣∣ z=ξ ]⊗2 ◦G⊗2
= E ξ∼N (0,µ∗G)
[√ BDr−1(I − vvT ) ]⊗2 ◦G⊗2
(by Proposition G.4)
where D = Diag(φ′( √ Bv)), r = ‖ξ‖, v = ξ/‖ξ‖. But notice that T := Eξ∼N (0,µ∗G) [ Dr−1(I − vvT ) ] is actually BSB1, with diagonal
E rv∼N (0,µ∗G)
r−1(1− v2i )φ′( √ Bvi) = E
rv∼N (0,µ∗G) r−1(1− v21)φ′(
√ Bv1),∀i ∈ [B]
and off-diagonal
E rv∼N (0,µ∗G)
r−1(−vivj)φ′( √ Bvi) = E
rv∼N (0,µ∗G) r−1(−v1v2)φ′(
√ Bv1),∀i 6= j ∈ [B].
Thus by Lemma E.51, T has two eigenspaces, {x : Gx = x} and R1, with respective eigenvalues T11 − T12 and T11 + (B − 1)T12. However, because of the composition with G⊗2, only the former eigenspace survives with nonzero eigenvalue in G⊗2 ◦ T .
Theorem I.2. T on MGB is just multiplication by λ↓M̃ := 1 2πBµ ∗−1 (∫ π 0 dθ sinB−1 θφ′(− √ B − 1 cos θ) )2 .
Proof. By the above reasoning, the sole eigenspace with nonzero eigenvalue is {x : Gx = x}⊗2 = MGB . It has eigenvalue
B ( E[r−1(1− v21 + v1v2)φ′( √ Bv1) : rv ∼ N (0, µ∗G)] )2 = Bµ∗−1 ( E[r−1(1− v21 + v1v2)φ′( √ Bv1) : rv ∼ N (0, G)]
)2 = Bµ∗−1 ( E[r−1(1− (ew)21 + (ew)1(ew)2)φ′( √ B(ew)1) : rw ∼ N (0, IB−1)]
)2 = Bµ∗−1 ( Γ((B − 2)/2) Γ((B − 3)/2)2
−1/2π−1∫ π 0 dθ1 ∫ π 0 dθ2 (1− ζ(θ1)2 + ζ(θ1)ω(θ1, θ2))φ′( √ Bζ(θ1)) sin B−3 θ1 sin B−4 θ2 )2 where we applied Lemma E.36 with ζ(θ) = − √ B−1 B cos θ and ω(θ1, θ2) = 1√ B(B−1)
cos θ1 −√ B−2 B−1 sin θ1 cos θ2.
We can further simplify∫ π 0 dθ1 ∫ π 0 dθ2 (1− ζ(θ1)2 + ζ(θ1)ω(θ1, θ2))φ′( √ Bζ(θ1)) sin B−3 θ1 sin B−4 θ2
= ∫ π 0 dθ1 sin B−3 θ1φ ′( √ Bζ(θ1)) (( 1− ζ(θ1)2 + ζ(θ1) 1√ B(B − 1) cos θ1 )∫ π 0 dθ2 sin B−4 θ2
− √ B − 2 B − 1 sin θ1ζ(θ1) ∫ π 0 dθ2 sin B−4 θ2 cos θ2 )
= ∫ π 0 dθ1 sin B−3 θ1φ ′( √ Bζ(θ1)) (( 1− ζ(θ1)2 + ζ(θ1) 1√ B(B − 1) cos θ1 ) Beta ( B − 3 2 , 1 2 )) (by Lemma E.29)
= ∫ π 0 dθ1 sin B−3 θ1φ ′( √ Bζ(θ1)) ( sin2 θ1 Beta ( B − 3 2 , 1 2 )) = Beta ( B − 3
2 ,
1
2 )∫ π 0 dθ1 sin B−1 θ1φ ′(− √ B − 1 cos θ1)
so the cross-batch backward off-diagonal eigenvalue is Bµ∗−1 (
1√ 2π ∫ π 0 dθ1 sin B−1 θ1φ ′(− √ B − 1 cos θ1) )2 = 1
2π Bµ∗−1 (∫ π 0 dθ1 sin B−1 θ1φ ′(− √ B − 1 cos θ1) )2
For positive homogeneous functions we can evaluate the eigenvalues explicitly. Theorem I.3. If φ is positive-homogeneous of degree α with φ(c) = aρα(c) − bρα(−c), then T :MGB →MGB is just multiplication by
λ↓ M̃ = 1 8π B(B − 1)α−1µ∗−1α2(a+ b)2 Beta
( α
2 , B 2
)2 .
Proof. As in the proof of Corollary H.4,∫ π/2 0 dθ (cos θ)α−1 sinB−1 θ = ∫ π π/2 dθ (− cos θ)α−1 sinB−1 θ = 1 2 Beta ( α 2 , B 2 ) So
1
2π Bµ∗−1 (∫ π 0 dθ1 sin B−1 θ1φ ′(− √ B − 1 cos θ1) )2 = 1
2π Bµ∗−1(B − 1)α−1α2
( a ∫ π π/2 dθ (− cos θ)α−1 sinB−1 θ + b ∫ π/2 0 dθ (cos θ)α−1 sinB−1 θ )2
= 1
2π Bµ∗−1(B − 1)α−1α2(a+ b)22−2 Beta
( α
2 , B 2 )2 = 1
8π B(B − 1)α−1µ∗−1α2(a+ b)2 Beta
( α
2 , B 2
)2
It is also straightforward to obtain an expression of the eigenvalue in terms of Gegenbauer coefficients.
Theorem I.4. If φ( √ B − 1x) has Gegenbauer expansion ∑∞l=0 al 1cB−1,lC(B−32 )l (x), then T : MGB →MGB is just multiplication by
λ↓ M̃ = 1 2π B(B − 1)µ∗−1a21 Beta
( B
2 ,
1
2 )2 = 1 2πB(B − 1)a21 Beta ( B 2 , 1 2
)2∑∞ l=0 a 2 l 1 cB−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1
)) = λ↑
M̃ .
Proof. It suffices to express the integral in Thm I.2 in Gegenbauer coefficients. Changing coordinates x = − cos θ, so that dx = sin θ dθ, we get∫ π
0
dθ sinB−1 θφ′(− √ B − 1 cos θ) = ∫ 1 −1 dx(1− x2)B−22 φ′( √ B − 1x).
By Eq. (35),
φ′( √ B − 1x) = (B − 1)−1/2 d dx φ( √ B − 1x)
= (B − 1)−1/2 ∞∑ l=0 al 1 cB−1,l C (B−32 ) l ′(x)
= (B − 1)−1/2 ∞∑ l=1 al B − 3 cB−1,l C (B−12 ) l−1 (x).
Then by the orthogonality relations among {C( B−1 2 )
l−1 }l≥1,∫ π 0 dθ sinB−1 θφ′(− √ B − 1 cos θ) = ∫ 1 −1 dx(1− x2)B−22 φ′( √ B − 1x)
= (B − 1)−1/2a1 B − 3 cB−1,1 ∫ 1 −1 [ C (B−12 ) 0 (x) ]2 (1− x2)B−22 dx
= (B − 1)−1/2a1 B − 3 cB−1,1
√ πΓ ( B 2 ) Γ ( B+1
2 ) = a1 √ B − 1 Beta ( B
2 ,
1
2
) .
To summarize,
Theorem I.5. The backward cross batch dynamics Eq. (59) over HGkB is separable over each individual B × B block. The diagonal blocks evolve as in Eq. (51), and this linear dynamics has eigenvalues λ↓G, λ ↓ L, λ ↓ M as described by Appendix G. The off-diagonal blocks dynamics is multiplication by λ↓ M̃ = λ↑ M̃ .
Note that
λ↓ M̃ = λ↑ M̃
= 1
2πB(B − 1)a21 Beta ( B 2 , 1 2 )2∑∞ l=0 a 2 l 1 cB−1,l ( C (B−32 ) l (1)− C (B−32 ) l ( −1 B−1
)) ≤ 1 2πB(B − 1) Beta ( B 2 , 1 2 )2 1
cB−1,1
( C (B−32 ) 1 (1)− C (B−32 ) 1 ( −1 B−1 )) = B − 1
2 P
( B
2 ,
1
2 )−2 which increases to 1 from below as B →∞. Thus, Corollary I.6. For any φ inducing Eq. (54) to converge to a CBSB1 fixed point, λ↓
M̃ = λ↑ M̃ <
1. For any fixed batch size B, λ↓ M̃ is maximized by φ = id. Furthermore, for φ = id, λ↓ M̃ = B−1 2 P ( B 2 , 1 2 )−2 and limB→∞ λ ↓ M̃ = 1, increasing to the limit from below.
This corollary indicates that stacking batchnorm in a deep neural network will always cause chaotic behavior, in the sense that cross batch correlation, both forward and backward, decreases to 0 exponentially with depth. The φ that can maximally ameliorate the exponential loss of information is linear.
J UNCOMMON REGIMES
In the above exposition of the mean field theory of batchnorm, we have assumed B ≥ 4 and that φ induces BSB1 fixed points in Eq. (8).
Small Batch Size What happens when B < 4? It is clear that for B = 1, batchnorm is not well-defined. For B = 2, Bφ(h) = (±1,∓1) depending on the signs of h. Thus, the gradient of a batchnorm network with B = 2 is 0. Therefore, we see an abrupt phase transition from the immediate gradient vanishing of B = 2 to the gradient explosion of B ≥ 4. We empirically see that B = 3 suffers similar gradient explosion as B = 4 and conjecture that a generalization of Thm 3.9 holds for B = 3.
Batch Symmetry Breaking What about other nonlinearities? Empirically, we observe that if the fixed point is not BSB1, then it is BSB2, like in Fig. 1, where a submatrix of Σ (the dominant block) is much larger in magnitude than everything else (see Defn K.1). If the initial Σ0 is permutationinvariant, then convergence to this fixed point requires spontaneous symmetry breaking, as the dominant block can appear in any part of Σ along the diagonal. This symmetry breaking is lost when we take the mean field limit, but in real networks, the symmetry is broken by the network weight randomness. Because small fluctuation in the input can also direct the dynamics toward one BSB1 fixed point against others, causing large change in the output, the gradient is intuitively large as well. Additionally, at the BSB2 fixed point, we expect the dominant block goes through a similar dynamics as if it were a BSB1 fixed point for a smaller B, thus suffering from similar gradient explosion. Appendix K discusses several results on our current understanding of BSB2 fixed points. A specific form of BSB2 fixed point with a 1 × 1 dominant block can be analyzed much further, and this is done in Appendix K.1.
Finite width effect For certain nonlinearities, the favored fixed point can be different between the large width limit and small width. For example, for φ = C( B−3 2 ) 5 (x/ √ B − 1)−C( B−3 2 ) 7 (x/ √ B − 1) where B = 10, one can observe that for width 100, Eq. (43) favors BSB2 fixed points, but for width 1000 and more, Eq. (43) favors BSB1 fixed points.
K TOWARD UNDERSTANDING BSB2 FIXED POINTS
Definition K.1. For B1, B2 ≥ 2, a BSB2 matrix is one of the form
BSB2B1,B2(d1, f1, d2, f2, c) :=
( BSB1B1(d1, f1) c
c BSB1B2(d2, f2) ) up to simultaneous permutation of rows and columns. A B̂SB2 matrix is a specific kind of BSB2 matrix B̂SB2 B′
B (d, f, c, b) = BSB2B′,B−B′(d, f, c, b, b), where B ′ ∈ [2, B − 1] (where for B′ =
B−1 the lower right hand block is a scalar b). We call the upper left hand block of B̂SB2 B′
B (d, f, c, b) its main block.
When φ grows quickly, Eq. (43) can converge to BSB2 fixed points of the form B̂SB2 B′
B (d, f, c, b), up to simultaneous permutation of rows and columns. In this section we present several results on BSB2 fixed points that sheds light on their structure. We leave the study of the forward Eq. (43) and backward dynamics Eq. (51) to future work.
First, we give the eigendecomposition of BSB2 matrices. Theorem K.2. Let Σ = BSB2B1,B2(d1, f1, d2, f2, c), where B1, B2 ≥ 2. Then G⊗2{Σ} = GΣG = ΣG has the following eigenspaces and eigenvalues.
1. (B1 − 1)-dimensional eigenspace Z1 := {(x1, . . . , xB1 , 0, . . . , 0) : ∑B1 i=1 xi = 0}, with
eigenvalue d1 − f1.
2. (B2 − 1)-dimensional eigenspace Z2 := {(0, . . . , 0, y1, . . . , yB2) : ∑B2 j=1 yj = 0}, with
eigenvalue d2 − f2.
3. 1-dimensional Rq with eigenvalue (d2−f2)B1+(d1−f1)B2+(f1+f2−2c)B1B2B1+B2 , where q =
( B1︷ ︸︸ ︷ B−11 , . . . , B −1 1 , B2︷ ︸︸ ︷ −B−12 , . . . ,−B−12 ).
4. 1-dimensional R1 with eigenvalue 0.
Proof. We can verify all eigenspaces and their eigenvalues in a straightforward manner. These then must be all of them by a dimensionality argument.
Specializing to B̂SB2 matrices, we get
Corollary K.3. Let Σ = B̂SB2 B′
B (d, f, c, b) with B ′ ≥ 2. Then G⊗2{Σ} = GΣG = ΣG has the
following eigenspaces and eigenvalues
• (B′ − 1)-dimensional eigenspace Z1 := {(x1, . . . , xB′ , 0, . . . , 0) : ∑B′ i=1 xi = 0}, with
eigenvalue d− f .
• 1-dimensional Rq with eigenvalue (d−f)(B−B ′)+(f−c)B(B−B′) B , where q =
( B′︷ ︸︸ ︷ B′−1, . . . , B′−1, B2︷ ︸︸ ︷ −B−12 , . . . ,−B−12 ) where B2 = B −B′.
• (B−B′)-dimensional eigenspace {( B′︷ ︸︸ ︷ −µ, . . . ,−µ, y1, . . . , yB−B′ : µ = 1B−B′ ∑B−B′ j=1 yj}
with eigenvalue 0.
Note also that when B′ = B, then Z1 and Rq become the usual eigenspaces of BSB1(d, f).
For fixed B′ < B, specializing the forward dynamics Eq. (43) to B̂SB2 B′
B fixed points yields a 2-dimensional dynamics on the eigenvalues for the eigenspaces Z1 and Rq. This dynamics in general is not degenerate, so that the fixed point seems difficult to obtain analytically. Moreoever, the Gegenbauer method, essential for proving that gradient explosion is unavoidable when Eq. (43) has a BSB1 fixed point, does not immediately generalize to the BSB2 case, since Z1 and Rq in general do not have the same eigenvalues so that we cannot reduce the integrals to that on a sphere. For this reason, at present we do not have any rigorous result on the BSB2 case.
However, we expect that the main block of the B̂SB2 fixed point should undergo a similar dynamics to that of a BSB1 fixed point, leading to similar gradient explosion.
K.1 B̂SB2 1
B FIXED POINTS
When φ grows extremely rapidly, for example φ(x) = relu(x)30, and the width of the network is not too large, then we sometimes observe B̂SB2 1
B fixed points
B̂SB2 1
B(d, c, b) =
( d c1T
c1 b11T ) where 1 ∈ RB−1 is the all 1s column vector. In these situations we can in fact see pathological gradient vanishing, in a way reminiscent of the gradient vanishing for B = 2 batchnorm, as we shall see shortly.
We can extend Corollary K.3 to the B′ = 1 case.
Theorem K.4. B̂SB2 1
B(d, c, b) G is rank 1 and equal to λq̂⊗2, where λ = B−1B (d − 2c + b) and
q̂ = (1, −1B−1 , . . . , −1 B−1 ) √ B−1 B .
Proof. Straightforward verification.
Because B̂SB2 1
B under G-projection is rank 1, such a fixed point of Eq. (43) can be determined
(unlike the general B̂SB2 B′
B case).
Theorem K.5. There is a unique B̂SB2 1 B fixed point of Eq. (43), B̂SB2 1 B(d ∗, c∗, b∗) where
d∗ = 1
2
( φ( √ B − 1)2 + φ(− √ B − 1)2 ) c∗ = 1
2
( φ( √ B − 1)φ( −1√
B − 1) + φ(− √ B − 1)φ( 1√ B − 1) )
b∗ = 1
2
( φ(
−1√ B − 1) 2 + φ( 1√ B − 1) 2
) .
Additionally, B̂SB2 1
B(d ∗, c∗, b∗)G = λ∗q̂⊗2, where
λ∗ = B − 1
2B
[( φ( √ B − 1)− φ( −1√ B − 1) )2 + ( φ(− √ B − 1)− φ( 1√ B − 1) )2]
Proof. If Σ = B̂SB2 1
B(d, c, b), then
VBφ(Σ) = E y∼N (0,GΣG)
φ(n(y))⊗2
= 1
2
[ φ( √ Bq̂)⊗2 + φ(− √ Bq̂)⊗2 ] since GΣG = λq̂⊗2 for some λ by Thm K.4. The rest then follows from straightforward computation and another application of Thm K.4.
The rank-1 nature of the B̂SB2 1
B fixed point under G-projection is reminiscent of the case of B = 2 batchnorm. As we will see, gradients of a certain form will vanish similarly.
Let δ be a gradient vector. Then by Proposition G.4, for Σ∗ being the B̂SB2 1
B fixed point, one step of backpropagation yields
dBφ dΣ ∣∣∣∣† Σ=Σ∗ δ = √ Br−1G(I − vvT )Dδ
where D = Diag(φ′(n(y))), r = ‖y‖, v = y/‖y‖, and y ∼ N (0, GΣ∗G). Because GΣ∗G = λ∗q̂⊗2 by Thm K.5, n(y) = √ Bv = ± √ Bq̂ with equal probability and r ∼ |N (0, λ∗)| (the absolute value of a Gaussian). Furthermore, G(I − q̂⊗2) = G− q̂⊗2 because Gq̂ = q̂. Thus
dBφ dΣ ∣∣∣∣† Σ=Σ∗ δ = √ B|r|−1(G− q̂⊗2)(φ′( √ B sgn(r)q̂) δ), r ∼ N (0, λ∗)
Now notice the following Lemma K.6. GB − q̂⊗2 = (
0 0 0 GB−1
) =  0 0 0 · · · 0 0 1− 1B−1 −1B−1 · · · −1B−1 0 −1B−1 1− 1B−1 · · · −1B−1 0 −1B−1 −1 B−1 · · · 1− 1B−1  . Proof. Straightforward computation.
Thus if δ = (δ1, δ2)T ∈ R1 × RB−1, then
dBφ dΣ ∣∣∣∣† Σ=Σ∗ δ = √ Bφ′ (− sgn(r)√ B − 1 ) |r|−1(0, GB−1δ2)T , r ∼ N (0, λ∗)
If δ2 = 0, then this is deterministically 0. Otherwise, this random variable does not have finite mean because of the |r|−1 term, and this is solely due to Σ∗ being rank 1. Thus as long as the gradient δ is not constant in δ2, then we expect severe gradient explosion; otherwise we expect severe gradient vanishing. Remark K.7. We would like to note that, in pytorch, with either Float Tensors or Double Tensors, the gradient empirically vanishes rapidly with generic gradient δ, when φ is rapidly increasing and leads to a B̂SB2 1
B fixed point, e.g. φ(x) = relu(x) 30. This is due to numerical issues rather than
mathematical issues, where the computation of (G− q̂⊗2)δ in backprop does not kill δ1 completely.
L WEIGHT GRADIENTS
Recall weight gradient and the backpropagation equation Eq. (16) of the main text.
∂L ∂W lαβ = ∑ i δlαix l−1 βi , δ l αi = ∑ βj ∂xlαj ∂hlαi W l+1βα δ l+1 βj
where we have used xlβi to denote Bφ(hlβ)i, δli = ∇hliL, and subscript i denotes slice over sample i in the batch, and subscript α denotes slice over neuron α in the layer. Then
E ( ∂L ∂W lαβ )2 = E (∑
i
δlαix l−1 βi )2 = E
∑ i,j δlαix l−1 βi δ l αjx l−1 βj  .
By gradient independence assumption (Appendix B),
E δlαix l−1 βi δ l αjx l−1 βj = E[δ l αiδ l αj ]E[x l−1 βi x l−1 βj ]
= ΠlijΣ l−1 ij .
Thus
E ( ∂L ∂W lαβ )2 = ∑ i,j ΠlijΣ l−1 ij = 〈Πl,Σl−1〉.
If l is large and φ is well-behaved, then we expect Σl−1 ≈ Σ∗, the unique BSB1 fixed point of Eq. (43). Assuming equality, we then have
E ( ∂L ∂W lαβ )2 = 〈Πl,Σ∗〉 = 〈G⊗2{Πl},Σ∗〉 = 〈Πl, G⊗2{Σ∗}〉 = 〈Πl, µ∗G〉 = µ∗ tr Πl.
Thus in general, we expect the magnitude of weight gradients to follow the magnitude of hidden unit gradient δli.

Review summaries:
This paper develops a mean field theory for batch normalization (BN) in fully-connected networks with randomly initialized weights. There are a number of interesting predictions made in this paper on the basis of this analysis. The main technical results of the paper are Theorems 5-8 which compute the statistics of the covariance of the activations and the gradients.