A Unified Paths Perspective for Pruning at Initialization
Authors: Thomas Gebhart
Udit Saxena
Paul R. Schrater
Venue:  (2021)
Keywords: Pruning
Paths
Neural Networks
Neural Tangent Kernel

Abstract: A number of recent approaches have been proposed for pruning neural network parameters at initialization with the goal of reducing the size and computational burden of models while minimally affecting their training dynamics and generalization performance. While each of these approaches have some amount of well-founded motivation, a rigorous analysis of the effect of these pruning methods on network training dynamics and their formal relationship to each other has thus far received little attention. Leveraging recent theoretical approximations provided by the Neural Tangent Kernel, we unify a number of popular approaches for pruning at initialization under a single path-centric framework. We introduce the Path Kernel as the data-independent factor in a decomposition of the Neural Tangent Kernel and show the global structure of the Path Kernel can be computed efficiently. This Path Kernel decomposition separates the architectural effects from the data-dependent effects within the Neural Tangent Kernel, providing a means to predict the convergence dynamics of a network from its architecture alone. We analyze the use of this structure in approximating training and generalization performance of networks in the absence of data across a number of initialization pruning approaches. Observing the relationship between input data and paths and the relationship between the Path Kernel and its natural norm, we additionally propose two augmentations of the SynFlow algorithm for pruning at initialization.

Under review as a conference paper at ICLR 2021
A UNIFIED PATHS PERSPECTIVE FOR PRUNING AT INITIALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
A number of recent approaches have been proposed for pruning neural network parameters at initialization with the goal of reducing the size and computational burden of models while minimally affecting their training dynamics and gener- alization performance. While each of these approaches have some amount of well-founded motivation, a rigorous analysis of the effect of these pruning meth- ods on network training dynamics and their formal relationship to each other has thus far received little attention. Leveraging recent theoretical approximations provided by the Neural Tangent Kernel, unify number of popular approaches for pruning at initialization under a single path path-centric framework. We introduce the Path Kernel the data-independent factor in decomposition of the Neural Tangent Kernel and show the global structure of the Path Kernel can be computed efficiently. This Path Kernel decomposition separates the architectural effects from the data-dependent effects within the Neural Tangent Kernel, providing a means to predict the convergence dynamics of network from its architecture alone. We analyze the use of this structure in approximating training and generalization performance of networks in the absence of data across number of initialization pruning approaches. Observing the relationship between input data and paths and the relationship between the Path Kernel and its natural norm, we additionally propose two augmentations of the SynFlow algorithm for pruning at initialization.
INTRODUCTION
wealth of recent work has been dedicated to characterizing the training dynamics and generaliza- tion bounds of neural networks under linearized approximation of the network depending on its parameters at initialization (Jacot et al. 2018 Arora et al. 2019, Lee et al. 2019a]] WWodworth et al., 2020). This approach makes use of the Neural Tangent Kernel, and under infinite width assumptions, the training dynamics of gradient descent over the network become analytically tractable. In this paper, we make of the Neural Tangent Kernel theory with the goal of approximating the effeets of various initialization pruming methods on the resulting training dynamics and performance of the network. Focusing networks with homogeneous activation functions (ReLU, Leaky-ReL.U, Linear), we introduce a novel decomposition of the Neural Tangent Kernel which separates the effects of network architecture from effects due the data on the training dynamics of the network. We find the data-independent factor of the Neural Tangent Kernel to have particularly nice structure as symmetric matrix representing the covariance of path values in the network which term the Path Kernel. We subsequently show that the Path Kernel offers a data iata-indeeendent approximation of the network's convergence dynamics during training.
To validate the empirical benefits of this theoretical approach, we turn to the problem of pruning initialization. While the problem of optimally pruning deep networks is nearly as old as deep networks themselves (Reed) 1993), interest in this problem has experienced revival in recent years. This revival is likely a product of a number of underlying factors, but much of the recent interest could be ascribed to the Lottery Ticket Hypothesis (Frankle & Carbin 2018) which states that sparse, trainable networks-whin dense counterparts can be found at initialization. The Lottery Ticket Hypothesis implies that the over- paramttrrization of neural networks is incidental in finding a trainable solution, the topology of which often exists at initialization. However, finding these lottery ticket networks currently requires
achieve task performance that matches or exceeds those of its

---

Under review as a conference paper at ICLR 2021
some amount of iterative re- e--training of the network at increasing levels of sparsity which is inefficient and difficult to analyze theoretically.
The resurgence of interest in optimal pruning has spurred the development of number of recent approaches for pruning deep neural networks at initialization (Lee et al. 2019b Liu & Zenke 2020
Wang et al. 2020: Tanaka et al.. 2020) in supervised, semi-supervised, and unsupervised settings, borrowing theoretical motivation from linearized training dynamics (Jacot et al. 2018), mean- isometry (Saxe et al. 2013), and saliency (Dhamdhere et al. 2019). While each of these methods have their own theoretical motivations, little work has been dedicated to formally describing the effect of these pruning methods on the expected performance of the pruned network. Also, the diversity in theoretical motivations that give rise to these pruning methods makes it difficult to observe their similarities.
In this paper, we observe that number of initialization pruning approaches are implicitly dependent on the path covariance structure captured by the Path Kernel which, in turn, affects the network's training dynamics. We show that we can approximate these training dynamics in general, and our approximation results for a number of initialization pruning approaches suggests that it is possible to estimate, prior to training. the efficacy of a particular initialization pruning approach on given architecture by investigating the eigenstructure of its Path Kernel. Motivated by our theoretical results and the unification of number of initialization pruning methods in this Path Kernel framework, we investigate the close relationship between the SynFlow (Tanaka et al.) (0200) pruning approach and our path decomposition. This leads to our suggestion of two new initialization pruning approaches which we predict to perform well under various assumptions on the stability of the Path Kernel and the input distribution of the data. We then validate these predictions empirically by comparing the performance of these pruning approaches across a number of network architectures.
The insights on initialization pruning provided by the Path Kernel decomposition are only one of a number of potential application domains which could benefit from this path-centric framework. Importantly, the coviariance structure over paths encoded by the Path Kernel is general and may be computed at any point in time, not just at initialization. We anticipate that this representation will provide insight into other application areas like model interpretation, model comparison, or transfer learning across domains.
The sections of the paper proceed as follows. We start with a brief introduction to the Neural Tangent Kernel in Section before introducing the Path Kernel decomposition in Section and its relationship to approximations of network convergence properties. In Section 4 we reformulate in this path framework three popular initialization pruning approaches and introduce two additional initialization pruning approaches inspired by this path decomposition. We validate these convergence approximations and the behavior of these pruning approaches in Section 5 and conclude with discussion of the results and opportunities for future work.
THE NEURAL TANGENT KERNEL
Recent work by Jacot al. (0188) has shown that the exact dynamics of infinite-width network outputs through gradient descent training corresponds to kernel gradient descent in function space with respect the Neural Tangent Kernel. More formally, for a neural network f parameterized by f). y) denote the empirical loss function. Here, X' is the training set, y is the associated set of class labels. For multiple inputs, denote f(X, 0) E V the outputs of the network where K is the output dimension and N the number of training examples. In continuous-time gradient descent, the evolution of parameters and outputs can be expressed as
0 and loss function : R x IR K + R. let L = E y y) e(ff
0.
(1)
= Vof(X,0,)0, -x, 5(
(2)
where the matrix (X X) E NKNNK is the Neural Tangent Kernel at time step f, defined as the covariance structure of the Jacobian of the parameters over all training samples:
(2 x)= = Vof
T.
For infinitely wide networks, the NTK exactly captures the output space dynamics through training, and (x X)) remains constant throughout. Lee et al. 2019a) have shown that neural networks of
2

---

Under review as a conference paper at ICLR 2021
any depth tend to follow the linearized training dynamics predicted by the NTK. Moreover, approximate the outputs of the network linearly through a one-step Taylor expansion given imput a as:
(x, 0.) = f(x 0o) + Vof [x, 0o)
(4)
where w, 0, Oo is the change in parameters from their initial values. The first term of Equation is constant while the second term captures the dynamics of the initial outputs during training. Substituting f for f in Equations | and 2 the dynamics of the linearized gradient flow become
W =
V (2
(5)
(z, = (,, X,V f (2
(6)
Under MSE loss, the above ODEs have closed-form solutions as
W. = 0o) (I (I- - e (f 0o) - -))
(7)
(X, ( e y + - f(X, Oo).
(8)
In other words, through the tangent kernel and initial outputs of the network, we can compute the training convergence of linearized neural network before running any gradient descent steps. We will show in Section 3 that, through the Path Kernel, we can reliably approximate the linearized convergence rate of the network in the absence of data and without computing the full NTK.
3 THE PATH KERNEL
We will now provide a reformulation of the output behavior of networks in terms of activated paths and their values. This reformulation provides access unique decomposition of the NTK that separates the data-dependent output dynamics of the network from those dependent on the architecture and initialization. This reformulation of network behavior in terms of active paths motivated by Meng et al. (2019) wherein the authors show that gradient descent on networks with homogenous activations may be computed completely in path space. The decomposition provided in this section makes ex explicit the relationships between the wruning-at-iniiaaizzan based approaches described in Section 4 and allows for estimation of the convergence behavior of networks at initialization and how pruning affects this behavior.
Let E RT denote the network parameters in vector form and let x E RRd denote an input. Assume the network has K output nodes. We define path from input output
binary vector p such that p, = 1 when 0, is an edge along path p. otherwise p, = O. Denote by P the enumerable set of all paths and P. P the subset of paths that go from imput node s to output node k. Let P [P] be the number of paths. Given the enumeration in P, we will abuse notation slightly and refer to paths by their index p in P. The value of path p can be calculated
product of parameters with binary exponents Up ()) = I m 0"' such that vp is the product along the 0 weights that are in path p. The activation status of path is (z 0) = II p,=1] II(op, (I, 0) > 0). Here Op, is the output of the hidden node which path p, passes through immediately after parameter j. We can then define the output of the network at node k as
(x = Σ Σ
pp((0) 0 0)..-
==1pEP,-.
The derivative of (z, 0) with respect to O will goes to zero in expectation. Using the chain rule, we can break the derivative of the output of the network with respect to 0 into two parts, the partial derivative of the output with respect to the path values and the partial derivative of the path values with respect to the parameters:
0f(z, 0) 0f(z, 0) do(0)
Vef(æ, 0)
Já (z)
(10)
00
0v( 00
Note the inner product structure formed over paths as
0f(=,)) = Jf(x) E RXP and (e(0) d
20
Jj Pxm, The change in output with respect to path values now only depends on the activation status of each path leading to the output and the input to that path. Each entry of this matrix has value
(2f(e.0)
0)
d
(f(x)))
I(p E P.-ss)) 0) 0)x.
00o(0)
kp
l=1
3

---

Under review as a conference paper at ICLR 2021
Similarly, the change in path values with respect to parameters is a function of only the parameters and their relational structure through paths:
Up 0
Pj
(J8) p
0
otherwise.
Given this reparameterization of the output of the network in terms of activated paths and their values, we see the NTK similarly decomposes nicely along this structure:
θ @ x) Vof(z (z)
J (x)
= [ (x)
where H. Jg. is the Path Kernel. The Path Kernel HG is positive semidefinite with max- imal values on the diagonal Hp(,, = ンド= 2,(0) ya and off-diagonal elements (p, =
E (=(()) ((pr() We can view Hl as covariance matrix on the weighted paths defined by the network architecture and parameter initialization. Note that JØ (a) entirely captures the depen- dence of f on the input by choosing which paths are active and re-weighting by the input, while HI is completely determined by the architecture and initialization. Therefore, we can expand our one-sample NTK to the entire training set through the appropriate expansion of dimensions such that J (x) P
0: 0:
In the following section, we will show that the Path Kernel decomposition of the NTK allows us to approximate, at initialization, the convergence behavior of the network during training. Additionally, we find we can compute the trace of the path kernel efficiently through an implicit computation over the parameter gradients of particular loss function. This trace computation serves as an approximation to full eigenstructure of the NTK.
3.1 NEURAL TANGENT KERNEL CONVERGENCE
We can decompose (x) = VDWT and II = USU and rewrite the NTK in Equation as
,, = (2) (x))
(11)
VVWTUSUTWDD DVT
(12)
VVD DVT.
(13)
From this perspective, we see the NTK acts as a rotation of inputs onto set of paths along with an imput-weighting those paths. computation of similarity in path space, followed by rotation of the result back to output space. The eigenvectors of the Path Kernel Ho, are a formal sum of paths with the eigenvector corresponding to the largest eigenvalue having interpretation as a unit-weighted set of paths producing highest flow through the network. The eigenstructure of the entire NTK is therefore determined by how eigenvectors of Ie, are mapped onto by the eigenvectors J (X).
Theorem 1. Let A be the eigenvalues of (, X), Vi the eigenvalues of J (x), and n. the eigenvalues of NI. Then > S V4TT, and EiST d E V/
NK
The proof of this theorem is given in the Appendix. Under the NTK assumptions of infinite width or infinite parameter scale (Woodworth et al.) 2020). Theorem
2 provides bounds on the network training convergence through the linearized dynamics in Equations 5 and 6 Specifically, if O, can be diagonalized by eigenfunctions with corresponding cigenvalues A, then the exponential e - has the same eigenfunctions with eigenvalues e e Therefore, ΣN* 17TT; provides an estimate of the network's training convergence at initialization as upper bound on the scale of e - which drives convergence in Equations 7 and S In Section[5.1] we show that this estimate does in fact predict convergence convergence performance of a number of initialization pruning algorithms.
wide range of finite network architectures, allowing us predict the
The effect of Jf( (X') the eigenvalues of the NTK will be scaling of the eigenvalues of II. If Vi is small, the input data map onto numerous paths within the network, increasing complexity and slowing convergence. Conversely, large v implies fewer paths activated by the input, leading to
4

---

Under review as a conference paper at ICLR 2021
lower complexity and faster convergence. In other words, the interaction between J (x) and HI will be high if they share eigenvectors. See the Section A.2 for further discussion of the effect of J (X) on the covariance structure of the network.
Given an initialized network, we can approximate this training convergence as the trace of the Path Kernel
m ΣΣ p=1 ==
0p(
Tr(TTo) Ho(p.))
(14)
8
This trace can be computed efficiently leveraging an implicit computation over the network's gradients. To this, define the loss function
L
Rpk a 0) = = 1T
III
0
x.
(15)
1=1
Letting x the vector of I's and noting that v, (0²) gradient of this loss
mh
0,,r
we can compute the (0)
3R (1,((1,))
Up
(0
003
0
p=1
j
ÖRpk (1, 0)
P
Up(0)
Tr((1l))
002
0g
4 PRUNING AND THE PATH KERNEL
The decomposition of the NTK into data-dependent and generalized view through which analyze prior approaches to pruning at initialization. Nearly all of the recent techniques for pruning at initialization are derived from notion of feature saliency, measure of importance of particular parameters with respect some feature of the network. Letting F represent this feature, we define saliency as the Hadamard product:
pieces provides
OF 3 0.
S(0)
The following methods for pruning initialization may all be viewed type of saliency measure of network parameters.
4.1 SNIP
Perhaps the most natural feature of the network target is the loss. This approach, known as skeletonization (Mozer & Smolensky 1989) and recently reintroduced as SNIP (Lee et al. 2019b). scores network parameters based on their relative contribution the loss
SsNIP ()) = 00
3 0
While intuitive, this pruning approach- which we will refer to as SNIP-is overexposed scale differences within the network. In other words, large differences in the magnitude of parameters can saturate the gradient, leading to unreliable parameter saliency values, potentially resulting in layer collapse (Tanaka et al. 2020). We rewrite the SNIP score using our path-based notation:
SSNIP (0) (x 0).
(Z( y) 0f 1,x,0 0) do(()
80
0f(X,0) 0)
Ov(0) 00
ac(f(X, y)
J (X)
0f(x,0) 0)
Clearly, SNIP scoring dependent network paths, network outputs fully described by the combination paths with the activation structure of those paths determined by input x. The precise interaction between paths and the loss is dependent the loss function, but it is clear from the J
5

---

Under review as a conference paper at ICLR 2021
term that the loss will be more sensitive to high-valued paths than low-valued paths. This points the source of overexposure to scale differences within the network, wherein particular high-valuce paths likely to dominate the loss signal, especially if initialization parameter scales are large (Lee et al.) 2020). The effect of large scale or variance in parameter initialization multiplicative along paths, and given the direct dependence of the network output on paths, the loss is likely be overexposed to the highest-magnitus paths.
4.2 SYNFLOWW
Tanaka et al. 2020) proposed SynFlow, an algorithmic approach to pruning neural networks without
data initia lization. SynFlow also relies saliency, but computes saliency with respect particular loss function:
-
0:1
1
(16)
such that the synaptic saliency score is SQ) 3Rsr(1.0) 0 which the authors call the "Synaptic Flow". Note the structural similarity of Equation 16 to Equation 15 we can view SynFlow as an l approximation to the trace of the Path Kernel and thus approximation of the sum of its eigenvalues.
This observation leads to our proposal of two variants of the SynFlow pruning approach. This first variant, which we call SynFlow-L2, is derived from the observation that SynFlow is an & approximation of the effect of the parameters on the Path Kernel structure. We score parameters in this variant as (0) (Rx((1.0) @ O,
Due to the parameter squaring, expect SynFlow-L2 to score more highly parameters along paths with high weight magnitudes relative to SynFlow. Given the squared interaction of path values defining the eigenstructure of the Path Kernel (Equation [T], we expect the Synflow-L2 pruning approach to better capture the dominant training dynamics as the architecture of the network becomes wider and better approximates the NTK assumptions of infinite width. We find empirical evidence for this fact in Section 5.2
06
The second SynFlow variant comes from observing the effect of the free input variable in Equation 15) The SynFlow algorithm sets x = which has the effect of computing scores over all possible paths. But all paths may be used given particular input distribution (see Section A.2 for an analysis of this fact for a 2-layer network). Therefore, if we have knowledge at the time of initialization of which input dimensions are more task-relevant, we can weight to reflect this discrepancy in relevance across input dimensions. This re- re-weighting increases the input-weighted value of paths calculated by Equations [5] and [6] For the datasets used in Section we assume we do not have a priori knowledge of which input dimensions are most relevant for classification success on each dataset. Therefore, we use the mean μ as a proxy for the "importance" of each input dimension under the assumption that important paths may be associated to high-magnitude input dimensions in expectation. We refer to this distributional variation SynFlow as SynFlow- Dist defined as S(p, O)SF-D (Rs(H,.) 80
c
0 and the distributional variation of SynFlow-L2 as
SynFlow-L.2-Dist defined as S(n, 0)SL2D
8R A) 00
O.
4.3 GRASP
Wang et al. (2020) make use of the NTK by pruning according to a measure of preservation of gradient flow after pruning by analyzing the change in loss AL after pruning, represented as a small deviation å in the weight:
Sumsp AC + 5) AC(O))
28 ((,) +(10))
2 + (())))
where H = V2L(0n) is the Hessian. When H = this score recovers SNIP, up to absolute value ((6TV)())))) This pruning approach, called GraSP, scores each weight according to the change in gradient flow after pruning the weight with the Hessian approximating the dependencies between weights with respect to the gradient. We can also embed GraSP within the saliency framework as (GR.SP(()
H 30 ( 0.
dc

---

Under review as a conference paper at ICLR 2021
We rewrite H using path notation
H == V = mat (5
(17)
The function (a) takes vector RT and forms a matrix A R and Hÿ - ( JBT)
Rm²xx is the second derivative of the path value matrix. See Section A.3 for the derivation. Equation 17 makes clear the relationship between paths and the GraSP scores for parameters. GraSP scoring takes SNIP scores and re-scales these scores by their interacting path values. The Hÿ is especially interesting as it reflects the second-derivativy along paths. In short, the addition of the Hessian in GraSP scoring makes the pruning scores much more path dependent compared to SNIP, albeit in a convoluted manner depending on both first and second-order effects of parameter changes on path values.
the path values according to pairs of parameters
EXPERIMENTS AND RESULTS
now bring together a number of ideas from the previous sections to show the empirical validity of the path reformulation of the NTK presented in the preceding sections for analyzing initialization pruning approaches. We begin by showing that the trace of the Path Kernel indeed a reasonable approximation of the linearized convergence dynamics approximated by Equations I and 2 and observe how different pruners affect these dynamics. Given the similarities between SynFlow and the computation of Tr(Tl,), we proposed in Section 4 three SynFlow-like pruning variants for pruning at initialization with and without knowledge of the data distribution. We test each of these variants on number of model-dataset combinations and observe, as predicted in Section 4 effects of model width on these pruning variants.
5.1
APPROXIMATING CONVERGENCE DYNAMICS
CIFAR10
CIFAR10C
83-
100)
(4)
71-
60
401
30
21-
0-
0-
1000
1500
2500
5000
2500
10000
12500
2000
1000
9000
5000
10.00
302
130
1000
200
201
100
30
500
1000
1500
2500
50C0
2500
10000
12500
2000
1000
6000
8000
10.00
TrTla,
Tr/lT,,)
Trillp
SynFllew
SynFlow
SynFlow-12
SynFlow
SNIP
GuSP
Figure 1: Aggregate weight change from initialization and output evolution aligns with Path Kernel estimate near initialization. Each point represents a particular FC-X, ResNet-20 or WideResNet-20 model pruned at initialization at range of compression ratios [0.5-3.0], colored by each pruner. The plotted curve the continuous-ting convergence estimation Equations and
approximation of the cigenstructure the NTK at initialization Oe,- As detailed Equations 7 and this eigenstructure will dominate the convergence dynamics of the network under the NTK assumptions. Figure plots number of architectures at varying compression ratios by pruner along with the exponential curve esti- mated as ||w>||| = a (1 - e Tr(Tle)))) and f (X,6 0,) | = (1 - - )r) a (c-n Tr(TT,, >)). Here, 77 0.001 and free parameters and b for each point capture the multiplicative constants in Equations and 2 fit at epoch 60. We choose the Path Kernel trace at epoch I for better visualization clarity as Io,)) has much larger variance (see Figure 2). Clearly, the Path Kernel acts useful
Section 3.1 introduced the trace of the path kernel initialization Tr
7

---

Under review as a conference paper at ICLR 2021
CIRAR1O
CIFARICO
MNIST
100
83-
60
00
03
42-
30-
10
20
10
15
20
25
2.5
5.0
5.5
4.0
4.5
5.0
0000/0
(4000)
0000
10000
(000)
2000
2000
20
3.0
De 30
log: 1:00
FC
FC-500 FC-1002 FC-2000
5C-5000
VGG-11
VGG-26
ResNet-20
WideResNes220
Figure 2: Aggregate weight change from initialization and output evolution is predicted by Tr(HI,))- Each point corresponds to a particular model pruned at initialization at a range of com- pression ratios [0.5-3.0], colored by model. The line of best fit is estimated on Tr(Ha,)) versus each aggregate convergence value ([ww)] || and I (X, ( with coefficient of determination inset.
proxy for the underlying NTK contribution to the estimation of the linearized dynamics as the empirical models across a range of seeds, compression ratios, and datasets lie along the predicted curve. The peweetccallyapppoxiimee
curve under-estimates the parameter and output changes for high values of Tr( (e,,) ). This is expected as the multiplicative data-dependent effects on the global missing from the Path Kernel estimation. Figure I also depicts the effects of
NTK eigenstructure the pruning approaches on convergence dynamics as described in Section 4 The 1.2 SynFlow variants align more closely with the predicted curve than the other pruning approaches. SNIP tends to produce models that have relatively larger Path Kernel eigenvalues while GraSP produces the opposite. The SynFlow pruners tend to lie between these extremes across most of the model-dataset combinations.
Figure 2 depicts the log-linear relationship between the Path Kernel eigenstructure and the estimated linearized dynamics across models. Each point again represents model varying compression ratios. We fit simple linear model to the natural logarithm of the trace of the Path Kernel at initialization, predicting each model's associated aggregate convergence value (||w>]l Il (X, O.)l). We observe generally linear relationship between log( (Tr((TT))) )) with different models exhibiting approximately equivalent slopes with different intercepts. These analyses point towards the general power of the
Path Kernel in predicting convergence both within and across models and across compression ratios or pruning approaches.
5.2 PATH KERNEL PRUNING VARIANTS
Figure 3 plots the Top-1 Accuracy of a number of models three datasets pruned at number of compression ratios, averaged across three random initializations. We leverage the simplicity of MNIST to train four fully-connected networks with six hidden layers that vary in width between models. Model FC has hidden layers of width 100 while the other models FC-W have hidden layers of width W. As we can see from the top row of Figure 3 the L2 variants of SynFlow underperform on skinny networks. However, SynFlow-L2 becomes much more competitive as network width increases, surpassing SynFlow at high compression ratios on FC-1000 and FC-2000. The distributional variants generally underperform the non- con-distributional pruners. Given the relative sparsity of MNIST in its input dimensionality, one might expect a number of paths picked out by SynFlow or SynFlow-L. to be irrelevant to the dataset. This does not seem to be the case given the underperformance of these distributional pruners. This observation provides further motivation for focusing on the path value term of the Path Kernel for estimating network training dynamics and generalization performance.
The second and third rows of Figure 3 again plot the averaged Top-1 Accuracy across three random initializations on CIFAR 10 and CIFAR 100, respectively. Notably, VGG-11 and VGG-16 both suffer from layer collapse at high compression ratios on at least one of the datasets and are generally noisy in their pruning performance. The ResNet20 models much more stable across compression ratios and datasets. Again, we see an increase in the performance of the SynFlow-L2 pruning approach as increase model width moving from ResNet20 to WideResNNr20 predicted in Section
8

---

Under review as a conference paper at ICLR 2021
FC
FC-500
FC FC-1000
8900
MIVIST
57.75
1xy-
07.00
0.0
VGG-11
VGG 16
Ree Vet 20
Vinieen 20
110
Tap- Aouraay
CIFRRI
11
1.5
1.0
2.0
VGGG-1
VOG-1E
RecVee-20
WidoRusNer 20
CI
10
1.0
21
300
0.5
25 Cur Sressso Rata
Corgres Ratio
Compresia Rade
Carpresion Ratio
Svnf low
SvnFlow: Dist
SvnFlow-L2 L2
Svnflow-L2.Ditt
Figure 3: Top-1 accuracy of models pruned by SynFlow and Path Kernel variants. Plotted is the mean 1 accuracy across three seeds for each combination of variant, model, and dataset. SynFlow-LA underperforms SynFlow on skinnier networks but outperforms on wider networks as predicted in Section The distributional variants perform better at lower pruning thresholds but generally underperform their non-distributional counterparts.
DISCUSSION
Recent work on the linear approximation of neural network training dynamics through the Neural Tangent Kernel have shown that these dynamics may be solved for analytically under the assumption of infinite width. However, many practical models currently in use are well-approximated by this theory, despite their finite architectures (Lee et al., 2019a). Leveraging this theory, we showed that for networks with homogeneous activation functions the Neural Tangent Kernel can be decomposed into a product of input-dependent and architecture cecheectureedeeeeddent terms. Specifically, at initialization, our results show that this decomposition provides approximation of the network's 's linearized convergence dynamics without any reference to input data. With this understanding, we applied this theory the problem of optimal network pruning at initialization and showed that number of popular initialization pruning approaches may be embedded within this path framework, and that their effects on training dynamies may be understood in terms of their effect on the network's path structure. There are a number natural routes for future work stemming from this application of the Path Kernel to pruning at initialization. Most pertinent of these in bounding the data-dependent effects of the NTK eigenstructure as discussed in Theorem 1. An approximation of the data-dependent contribution to the eigenstructure would provide increased accuracy in the approximation of convergence dynamics for any
pair.
The Path Kernel decomposition likely has many more uses outside of the initialization pruning context presented here. The previous sections have shown the strong bias imported into neural network training and generalization performance by their composite paths. These biases comprise the central features of study in the representation and transfer learning literature. Importantly, we can calculate the Path Kernel for a network at any stage in training, even at convergence. The Path Kernel encodes these representational structures a form that can further exploited to characterize the implicit representations embedded networks, compare learned representations across networks, predict the performance of network given particular data distribution, or prioritize the presence of particular path- determined biases throughout the course of training.
type of covariance matrix over composite paths which provides

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint (9Xi::/907.08884, 2019.
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron? International Conference Learning Representations, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprini arXiv: 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. In Advances in neural information processing systems, pp. 8572-8583, 2019a.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference Learning Representations, 2019b. URL httptttpe//opeeeeeiiieee net forum? ii 1w
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. A signal propagation perspective for pruning neural networks at initialization. International Conference on Learning Representations, 2020. URL
Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent transfer. arXiv preprint arXiv:2006.08228, 2020.
Qi Meng, Shuxin Zheng, Huishuai Zhang, Wei Chen, Zhi-Ming Ma, and Tie- Liu. G-SGD: Optimizing reLU neural networks in its positively scale-invariant space. In International Confer- ence on Learning Representations, 2019. URL https: /atpss:///peeeeiewww net forum?id SyxfEn09Y7
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. In Advances in neural information processing systems, pp. 107-115, 1989.
Russell Reed, Pruning algorithms-a survey. IEEE transactions on Neural Networks, 4(5):740-747, 1993.
Andrew Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprini 13 2013.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, arXiv:2006, 200.:0600005567 [cond-mat, q-bio, stat), arXiv: 2006.05467.
June 2020. URL URLLLEpp///aaxtt.oo////////// 05467
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow. arXiv:2002.0793 /cs, stat), February 2020. URL htt.:http://at.
arggabbs2002...o 07376
arXiv: 2002.07376.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. arXiv preprint ar 2tt.::200200927... 2020.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.I PROOF OF THEOREM
We repeat the theorem below for convenience:
Theorem 2. Let X be the eigenvalues of (x (X, the eigenvalues of JJ (X), and a, the eigenvalues of No. Then D3= S -ViTT and Ei=1 Σ NK
Proof. This fact is evident from the decomposition in and 13 Ho, is positive semidefinite and will not alter the eigenvalues of J (x). To see this, consider eigenvector of θ, Define yv Vy. then y VDWTII, DVVVy yv y SU''
A.2 RELATING INPUTS TO PATH ACTIVATION
Arora et al. 2019) showed that under the constant NTK assumption and given labels y E RKK the training dynamics for squared error loss for a two-layer network may be written:
| 3(xx0,)]2
Σ 11 ())))(())))) +
(18)
N Noaaoy is the eigendecomposition of where is bounded error term and 0 VAVT Enst the Gram matrix defined as
H Bw-Noonn
0, w x 2 0)]
(19)
x ( (2.
(20)
2x
for inputs 2;, V, and column w of 0. Equation 18 shows that the training dynamics of a two- layer network may be approximated as the higenvalue-weighter sum of the inner product of NTK eigenvectors on the output labels, with convergence being driven by the directions of maximal overlap of network inputs.
In this case of a two-layer, fully- network with single output dimension 19) the effect of J (X) on the network output covariance structure becomes clear. Recall the definition of path activation
(x, 0)
II PP=1)
I[(op, (xx, 0) > 0).
For network with ReLU activations, normalized inputs ~V I). and weights in the first layer drawn from unit normal distribution 0, ~ AN(0, I). the node-wise output activation function for hidden node h is completely described by the hth column of (0)).,
and has expectation E [op, (z. w,)] = E [w] = = O. Therefore, half of the paths in the path kernel will be activated expectation since each u is also normally distributed with zero mean. The structure of H is equivalent to the NTK under the architectural assumptions described above. Therefore, the network outputs are completely determined after the inner product on the first layer parameters, and due to the homogenous activations, the value of these outputs is only dependent the sign of this inner product. The path values determine the output values after the active paths are picked out by the inner products.
A.3 PATH KERNEL DERIVATION OF HESSIAN
We can re-write a network's 's Hessian defined as the second derivative of the loss with respect 8² C in terms of its path decomposition values. Recall the 30,00,
to parameters H VoGu,
11

---

Under review as a conference paper at ICLR 2021
decomposition of the loss given path view:
VoL= = aL(f(x, 0). v)
@L(f(a, 0).91 0f 0) do(0) 0f(x.0) dv(0)
00
00
0l(f(x.0). J (z)JØ
0f(x.0 9)
C (m)
Therefore, we are looking to take the second-derivative of this composition of three functions Vgc= 80 JJ which, using the product rule, becomes
V2c 30 100
a x v JØ Jo J
8 00
v vc)
Define the function mal (a) which takes a vector a E RR and reshapes into matrix in R=xn, Note that o (a))T) will go zero expectation as each D0 (2, 0) has measure zero everywhere but a single point. We can now define the Hessian
8 V 00
Vac mat (H)J J (z)
mat H
(
J
(z)
where we have defined H = JØ R=2xpp
A.4
EXPERIMENTAL DETAILS
A.4.1 PRUNING ALGORITTMS
Apart from pruning algorithms already mentioned before (Random, SNIP, GraSP, SynFlow) also implement the pruning algorithms we defined earlier: SynFlow-I2, SynFlow-I.2-Dist and SynFlow-Dist by extending the two step method described in (Tanaka et al. 2020) scoring and masking parameters globally across the network and pruning those with the lowest scores.
A.4.2 MODEL. ARCHITECTURES
We used standard implementations of VGG-11 and VGG-16 from OpenLTH, and FC- ResNet-16 18 and WideResNet18 from PyTorch models using the publicly available, open sourced code from (Tanaka et al., 2020) As described in (Tanaka al., 2020) we pruned the convolutional and linear layers of these models prunable parameters, but not biases nor the parameters involved in batchnorm layers. For convolutional and linear layers, the weights were initialized with Kaiming normal strategy and biases to be zero.
A.4.3 TRAINING HYPERPARAMETERS
We provide the hyperparameters we used to train our models as listed in Table andd]
12

---

Under review as a conference paper at ICLR 2021
Table Hyperparameters for ResNet-220, Wide-ResNet-20, VGG11 and VGG16 models. These hyperparameters were consistent across pruned and unpruned models and were chosen to optimize performance of the unpruned models. For the pruned models, we pruned for 100 epochs except for SNIP and GraSP, where we only pruned for a single epoch.
ResNet-20
Wide- ResNet-20
VGG-II/VGG-16
CIFAR-10/I00
CIFAR-10/100
CIFAR-I0
CIFAR-100
Optimizer
adam
adam
adam
adam
Training Epochs
100
100
160
160
Batch Size
64
64
64
64
Learning Rate
0.001
0.001
0.001
0.0005
Learning Rate Drops
60
60,120
60,
Drop Factor
0.1
0.1
0.1
0.1
Weight Decay
10
10
10 "
Table 2: Hyperparameters for the FC, FC-500, FC-1000, FC-2000 models. Each of the FC models have 6 fully connected layers with the indicated number of parameters in each layer. These hyperparameters were consistent across pruned and unpruned models and were chosen to optimize performance of the unpruned models. For the pruned models, pruned for 100 epochs except for SNIP and GraSP, where we only pruned for a single epoch.
FC/FC-500 FC-1000 FC-2000
MNIST CIFAR-10/100
Optimizer
adam
adam
Training Epochs
50
100
Batch Size
64
64
Learning Rate
0.01
0.001
Learning Rate Drops
60
Drop Factor
0.1
0.1
Weight Decay
10
10
13


Reviews:
[{"review_id": "VFeps_4XO-p", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new kernel named Path Kernel to understand deep neural network training. The key idea is to reparameterize the network with respect to the active path in the network. In this way, they can decompose the Tangent Kernel into data-dependent and architecture-dependent pieces. They authors rewrite the formulations of the existing pruning at initialization methods with respect to their Path Kernel and provide some new understandings. \n\nThe positive aspects:\n\n1. The proposed Path Kernel is novel.\n\n2. Based on Path Kernel, the authors provide some new understandings to the existing pruning methods.\n\n3. This paper is well written and easy to read.\n\nMy concerns are:\n\n1. We know that NTK focus on the small neighbourhood near the initialization. However, in deep neural network training, the weights could be quite far from the initialization. So I am not sure whether it is reasonable to view the existing pruning at initialization methods from the perspective of NTK.\n\n2. I think this paper may be a bit over claimed. The authors provide some new understandings to the existing pruning methods instead of unify them into a single framework.  The authors essentially rewrite the formulations in existing pruning methods with respect to active paths and show that they all have Path Kernel. I think this is not surprising since the network is reparameterized with respect  to the active paths. Such rewriting can be performed on lots of other models besides pruning and we can always show there exists Path Kernel in them. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ph1mdHAIBw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the path kernel which decomposes NTK in terms of path kernel. It decouples the NTK into two parts, the data-dependent part, and the data-independent part, which is the Path Kernel.\n\nThe new perspective to explain several recent pruning algorithms by Path Kernel is novel and valuable. The experiment of the proposed variation of  SynFlow also coincides with the theory.\n \nPros: a good theoretical foundation on the proposed algorithm which separates the data-independent part to Path Kernel.\n\nCons:\n1. The experiment's setup is not clear to me,  can I ask what is the formal definition of compression ratio. I thought it might be the original size/ compressed size, however, it can be small than 1 or even 0 in Figure 2 which confused me. \n\n2. Recent researches show that pruning at initialization is not better than random pruning when keeping the non-zero ratio among layers. Could the authors provide a comparison between the proposed algorithm and keep-ratio pruning? Here's the reference.\nhttps://arxiv.org/abs/2009.11094\nhttps://arxiv.org/pdf/2009.08576.pdf", "extra_scores": {}, "extra_text": {}}, {"review_id": "O6xUOuTOPV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "__summary.__\nThe paper studies the problem of neural network pruning at initialization through the lens of neural tangent kernels (NTK). As a result, the paper delivers a unified perspective on SNIP, GRASP, and SynFlow. Based on the framework, the paper provides a method to approximate the convergence dynamics of pruned models. The paper also motivates a SynFlow-variant from the theoretical framework.\n\n__overall opinion.__\nI could not find a very big empirical benefit from adopting the proposed theoretical perspective. Given the limited novelty and theoretical significance, I do not think this paper makes enough contribution to be shared at ICLR.\n\n__originalty.__\nAs a matter of fact, I did not find the proposed approach strikingly novel. The fact that the considered pruning methods (which are being unified under the given framework) are all based on the first-order approximations is not very surprising; I would rather say that the fact has been quite well known. The use of first-order approximation itself has been prevalent in the network pruning society, at least since the [optimal brain damage paper](https://papers.nips.cc/paper/250-optimal-brain-damage.pdf) at 1989 and has been used until quite recently, in [layerwise obs](https://arxiv.org/abs/1705.07565) or [lookahead pruning](https://openreview.net/forum?id=ryl3ygHYDB). Also, the combination with the NTK theory for the case of pruning at initialization was already outlined with details in the GRASP paper.\n\n__clarity.__\nThe paper is clearly written.\n\n__significance.__\nI do not see a big merit, especially about the pruning variants. The performance gain of using the proposed variant is nonexistent or marginal, compared to the original SynFlow. The materials presented in section 5.1 seems to be potentially useful, but the paper does not elaborate for what purpose such predictability can be used.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1rvh_1nZVZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper is twofold: 1) the authors propose a way of computing a NTK by decomposing a piecewise linear network into paths; 2) the authors propose to use this decomposition to detect the least useful weights at initialization.\n\n### Details\nFor convenience, I will call the theoretical part \"Path-NTK\" in the following.\n\nThe main usable result of Path-NTK is a proxy for computing the speed of convergence of the neural network. This result is based on two approximations: 1) the influence of the eigenvalues $\\nu_i$, related to the data, is neglected; 2) the sum of the eigenvalues $\\lambda_i$ is assumed to reflect the value of each $\\lambda_i$ (i.e., if $\\sum_i \\lambda_i$ is large, then each $\\lambda_i$ is large).\nThe justification of 1 has been made in Annex A.2, but, once again, at the cost of more approximations, that is, inputs are assumed to be N(0, 1), which is not true in MNIST.\nThe justification of 2 is experimental, and has been made in Section 5.1. However, the graphs are only qualitative, and not quantitative. We can see that the larger $\\sum_i \\lambda_i$, the faster the convergence is. But there is no quantitative estimation of the convergence rate, which could have been compared to $\\sum_i \\lambda_i$ in order to justify completely the approximation. And, last but not least, the dataset MNIST is missing in these experiments.\n\nAbout the pruning experiments, I can see in the graphs at the bottom-right of Figure 2 that the proposed method outperforms slightly the precedent ones, but this seems very subtle to me.\n\n### Significance\nPath-NTK is original, and the idea is interesting and probably reusable. However, this is limited to piecewise linear networks, and the only practical result is based on many approximations that have been certainly discussed, but they could be improved.\n\nThe Lottery Ticket Hypothesis is a certainly a hot topic, but the specific topic \"pruning at initialization\" seems to be marginal. Moreover, the overall significance of the results is not encouraging.\n\n### Additional comments\nThe justification of all approximations made in the theoretical part seems to be itself a research topic. \n\nEdit:\n### Rebuttal\nRemark 1: \n * Figures are much clearer;\n * about Figure 1, I see that MNIST has been added, which is good;\n\nRemark 2:\n * \"We can always normalize inputs such that they are distributed as N(0,1)\": this is not true. At best, we can put the mean to 0 and the variance to 1. It does not mean anything in terms of distribution *shape*;\n * I maintain that experiments with MNIST were missing in Figure 1 in the first version of the paper. This is more important than pretended in the rebuttal, because, in the MNIST dataset, the pixels are not N(0, 1) (even when centered and normalized), so MNIST does no meet the approximation made in Annex A.2. However, the results are not too different from those obtained with the other datasets.\n\nThe remark about the limitation to piecewise linear NNs has not been correctly addressed by the authors. The setup used in this paper corresponds to \"Piecewise linear NNs\", which is *really* limiting, since the biases are supposed to be zero. This is very different from \"Piecewise *affine* NNs\", which are actually widely used (biases can be non-zero). Adding the influence of the biases would break the entire computation made in this paper (or, at least, would necessitate more effort to take it into account). I assert that this limitation is not discussed in the paper, and my attempt to discuss it here has been eluded in the rebuttal.\n\nDespite the new and more fashionable figures the authors presented in the rebuttal, I am disappointed by its lack of accuracy and vagueness, especially when discussing the \"piecewise linear NNs\". So, I lower my rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a very interesting decomposition of the neural tangent kernel, which promises
to decouple effects of the parameters and data. The authors illustrate the effects of this decomposition
by considering pruning strategies for initialization.
While the approach looks promising, the current paper is somewhat premature: The only "hard" 
theoretical result, Theorem 1, is a direct consequence of the decomposition.  Its consequences for 
training discussed in the subsequent paragraph involve quite a few approximations, yet the effects 
of these approximations remain unclear. This general, high-level tone is kept when discussing the 
initializations.  
Finally, the N(0,1)-response to Reviewer 3 worries me.
Decision: Reject