How many degrees of freedom do we need to train deep networks: a loss landscape perspective
Authors: Brett W Larsen
Stanislav Fort
Nic Becker
Surya Ganguli
Venue: ICLR 2022 Poster (2022)
Keywords: loss landscape
high-dimensional geometry
random hyperplanes
optimization

Abstract: A variety of recent works, spanning pruning, lottery tickets, and training within random subspaces, have shown that deep neural networks can be trained using far fewer degrees of freedom than the total number of parameters. We analyze this phenomenon for random subspaces by first examining the success probability of hitting a training loss sublevel set when training within a random subspace of a given training dimensionality.  We find a sharp phase transition in the success probability from $0$ to $1$ as the training dimension surpasses a threshold. This threshold training dimension increases as the desired final loss decreases, but decreases as the initial loss decreases. We then theoretically explain the origin of this phase transition, and its dependence on initialization and final desired loss, in terms of properties of the high-dimensional geometry of the loss landscape.  In particular, we show via Gordon's escape theorem, that the training dimension plus the Gaussian width of the desired loss sublevel set, projected onto a unit sphere surrounding the initialization, must exceed the total number of parameters for the success probability to be large.  In several architectures and datasets, we measure the threshold training dimension as a function of initialization and demonstrate that it is a small fraction of the total parameters, implying by our theory that successful training with so few dimensions is possible precisely because the Gaussian width of low loss sublevel sets is very large. Moreover, we compare this threshold training dimension to more sophisticated ways of reducing training degrees of freedom, including lottery tickets as well as a new, analogous method: lottery subspaces. 

Published as a conference paper at ICLR 2022
How MANY DEGREES OF FREEDOM DO WE NEED TO TRAIN DEEP NETWORKS: A LOSS LANDSCAPE PERSPECTIVE
Brett W. Larsen', Stanislav Fort', Nie Becker', Surya Ganguli
Stanford University, MMeta Al Correspondence bw a rarseen wwrssestttaaoodd.cooo
ABSTRACT
A variety of recent works, spanning pruning, lottery tickets, and training within random subspaces, have shown that deep neural networks can be trained using far fewer degrees of freedom than the total number of parameters. We analyze this phenomenon for random subspaces by first examining the success probability of hitting a training loss sublevel set when training within a random subspace given training dimensionality. We find sharp phase transition in the success probability from O to as the training dimension surpasses a threshold. This threshold training dimension increases as the desired final loss decreases, but decreases as the initial loss decreases. We then theoretically explain the origin of this phase transition, and its dependence on initialization and final desired loss, in terms of properties of the high ditheniiinionl geometry of the loss landscape. In particular, we show via Gordon's escape theorem, that the training dimension plus the Gaussian width of the desired loss sublevel set, projected onto unit sphere surrounding the initialization, must exceed the total number of parameters for the success probability to be large. In several architectures and datasets, we measure the threshold training dimension as a function of initialization and demonstrate that it is small fraction of the total parameters, implying by theory that successful training with so few dimensions is possible precisely because the Gaussian width low loss sublevel sets very large. Moreover, we compare this threshold training dimension to more sophisticated ways of reducing training degrees of freedom, including lottery tickets as well new, analogous method: lottery subspaces.
INTRODUCTION
How many parameters are needed to train neural network to a specified accuracy? Recent work on two fronts indicates that the answer for given architecture and dataset pair is often much smaller than the total number of parameters used modern large-scale neural networks. The first is successfully identifying lottery tickets sparse trainable subnetworks through iterative training and pruning cycles (Frankle & Carbin 2019). Such methods utilize information from training to identify lower- dimensional parameter spaces which optimize
similar accuracy the full model. The second the observation that constrained training within a random, low-dimension affine subspace is often successful at reaching high desired train and test accuracy on variety of tasks, provided that the training dimension of the subspace is above an empirically-observe al. 2018). These results, however, leave open the question of why low-dimensional training so successful and whether we can theoretically explain the existence of threshold training dimension.
threshold training dimension (Li
In this work, we provide such an explanation in terms of the nigh-dimensional geometry of the loss landscape, the initialization, and the desired loss. In particular, we leverage a powerful tool from high nigh-dimensional probability theory, namely Gordon's 's escape theorem, to show that this threshold training dimension is equal to the dimension of the full parameter space minus the squared Gaussian width of the desired loss sublevel set projected onto the unit sphere around initialization. This theory can then be applied in several ways to enhance our understanding of neural network loss landscapes. For a quadratic well or second-order approximation around a local minimum, we derive an analytic bound on this threshold training dimension in terms of the Hessian spectrum and the distance of the initialization from the minimum. For general models, this relationship can be used in reverse to measure important high-dimensional properties of loss landscape geometry. For example, by

---

Published as a conference paper at ICLR 2022
performing tomographic exploration of the loss landscape, i.e. training within random subspaces of varying training dimension, we uncover a phase transition the success probability of hitting a given loss sublevel set. The threshold training dimension is then the phase boundary in this transition, and our theory explains the dependence of the phase boundary on the desired loss sublevel set and the initialization, in terms of the Gaussian width of the loss sublevel projected onto sphere surrounding the initialization.
Motivated by lottery tickets, we furthermore consider training not only within random dimensions, but also within optimized subspaces using information from training in the full space. Lottery tickets can be viewed as constructing an optimized, axis-aligned subspace, i.e. where each subspace dimension corresponds to a single parameter. What would constitute an optimized choice for general subspaces? We propose two new methods: burn-in subspaces which optimize the offiset of the subspace by taking a few steps along a training trajectory and lottery subspaces determined by the span of gradients along full training trajectory (Fig. []. Burn-in subspaces in particular can be viewed lowering the threshold training dimension by moving closer to the desired loss sublevel set. For all
loss sublevee) seaa
Weight Space R
padom vilgaro
optimization
pertrainod
labsyy ohar utbepace
inis
tabsln alloss sabrpacs
three methods, we empirically explore the threshold training dimension across a range of datasets and architectures.
An important motivation of our work is the observation that training within a random, low-dimensional affine subspace can suffice to reach high training and test ac- curacies on variety of tasks, provided the training dimension exceeds a threshold that was called the intrinsic dimension by (L. et al.) 2018) and which is closely related to what we call the threshold training dimension. Li et al. (018) however did not explore the dependence of this threshold on the quality of the initialization or provide a theoretical explanation for its existence. Our primary goal then is to build on this work by providing such an explanation in terms of the geometry of the loss landscape and the quality of initialization. Indeed under- standing the geometry of high-dimensional error landscapes has been a subject of intense interest in deep learning, see e.g. Dauphin et al. (2014); Goodfellow et al. (2014); Fort & Jastrzebski (2019);
Related Work:
Figure 1: Illustration of finding point in the intersection between affine subspaces and low-loss sub- level set. We use three methods: I) random affine subspaces (blue) containing the initialization, 2) burn- in affine subspaces (red) containing a pre-trained point on the training trajectory, and 3) lottery subspaces (purple) whose span is defined by the steps of full training trajectory.
Ghorbani et al. (019); Sagun et al. (2016; 20 Yao et al.((20) 8); Fort & Scherlis (2019); Papyan (2020) Gur-Art Ari al. 2018); Fort & Ganguli (2019); Papyan (2019): et al. (2020). or Bahri
et al. 2020) for a review. But to our knowledge, the Gaussian width of sublevel sets projected onto a
sphere surrounding initialization, a key quantity that determines the threshold training dimension, has not been extensively explored in deep learning.
Another motivation for our work is contextualizing the efficacy of diverse more sophisticated network pruning methods like lottery tickets (Frankle & Carbin 2019 Frankle et al. (019).. Further work in this area revealed the advantages obtained by pruning networks not at initialization (Frankle & Carbin 2019; Lee et al. 2018 Wang et al., 2020: Tanaka et al. 2020) but slightly later in training (Frnnlle et al. 2020), highlighting the importance of early stages of training (Jastrzebski et al.) 2020 Lewkowycz al. 2020). We find empirically, well as explain theoretically, that even when training within random subspaces, one can obtain higher accuracies for a given training dimension if one starts from a slightly pre-trained or burned-in initialization as opposed to random initialization.
2 AN EMPIRICALLY OBSERVED PHASE TRANSITION IN TRAINING SUCCESS
We begin with the empirical observation of a phase transition in the probability of hitting loss sublevel set when training within a random subspace of a given training dimension, starting from some initialization. Before presenting this phase transition, we first define loss sublevel sets and two different methods for training within a random subspace that differ only in the quality of the initialization. In the next section, develop theory for the nature of this phase transition.
2

---

Published as a conference paper at ICLR 2022
Loss sublevel sets. Let ý (x) be neural network with weights E R2 and inputs R For a given training set (xx: V Valn= and loss function f, the empirical loss landscape is given by [(w) = x Ex ( (Xn),,
Though our theory is general, we focus on classification for our experiments, where y E [0,1] is a one-hot encoding of C' class labels, y is a vector of class probabilities, and E(y. y) is the cross-entropy loss. In general, the loss sublevel set S(e) at desired value of loss e is the set of all points for which the loss is less than or equal to E:
S(c) = (w E RR: [(w)= S {c..
(2.1)
Random affine subspace. Consider d-dimensional random affine hyperplane contained in D- dimensional weight space, parameterized by Rd: w(0) A6 + Wo- Here A Dxd is a random Gaussian matrix with columns normalized to and wo E RDd a random weight initialization by standard methods. To train within this subspace, we initialize 0 = O, which corresponds to randomly initializing the network at Wo, and we minimize ((w(O)) with respect to O.
Burn-in affine subspace. Alternatively, we can initialize the network with parameters wo and train the network in the full space for some number of iterations t, arriving the parameters Wt- We can then construct the random burn-in subspace
w ( = A0 + Wt.
(2.2)
with A chosen randomly as before, and then subsequently train within this subspace by minimizing L( (w())) with respect O. The random affine subspace is identical to the burn-in affine subspace but with L = O. Exploring the properties of training within burn-in as opposed to random affine subspaces enables us to explore the impact of improving the initialization, via some information from the training data, on the success of subsequent restricted training.
Success probability in hitting a sublevel set. In either training method, achieving & (w(0)) = e implies that the intersection between our random or burn-in affine subspace and the loss sublevel set s(e') is non-empty for all é 2 c. As both the subspace and the initialization wa leading to w, are random, we are interested in the success probability (. () that burn-in (or random when / = 0) subspace of training dimension actually intersects loss sublevel set S(e):
Pa e) = P ((((t) n {w, spantssaaa)))aa (A)] + 0].
(2.3)
Here, span(A) denotes the column space of A. Note in practice we cannot guarantee that we obtain
the minimal loss in the subspace so we use the best value achieved by Adam (Kingma & Bal 2014) as an approximation. Thus, the probability of achieving a given loss sublevel set via training constitutes
an approximate lower bound on the probability in 2.3) that the subspace actually intersects the loss sublevel set.
Threshold training dimension as phase transition boundary. We will find that for any fixed 6, the success probability P. (d, e, E) in the by plane undergoes a sharp phase transition. In particular for a desired (not too low) loss it transitions sharply from 0 to 1 as the training dimension d increases. To capture this transition we define:
Definition 2.1. [Threshold training dimension] The threshold training dimension d' (c, t, 5) is the minimal value of d such that Ps c, t) 2 1 for some small 0.
For any chosen criterion ö (and fixed t) we will see that the curve d' (c, 6) forms a phase boundary in the by d plane separating two phases of high and low success probability. This definition also gives an operational procedure to approximately measure the threshold training dimension: run either the random or burn-in affine subspace method repeatedly over a range of training dimensions d and record the lowest loss value found in the plane when optimizing via Adam. We can then construct the empirical probability across runs of hitting a given sublevel set S(e) and the threshold training dimension is lowest value of d for which this probability crosses 1 - å (where we employ d 0.1).
2. AN EMPIRICAL DEMONSTRATION OF A TRAINING PHASE TRANSITION
In this section, we carry out this operational procedure, comparing random and burn-in affine subspaces across range of datasets and architectures. We examined 3 architectures: 1) Conv-2
3

---

Published as a conference paper at ICLR 2022
Phuse tramabors in acrensbble accursey versus tramiig dimersecn
08
04
0.
02
1/0
08
0.0
0.8
04
0.0
27 21
2 21
272
250
204
512
Tranking
Traning
Traking
Baseline acourity
Affino Subopace
Burr-in
(16)
Frobabiltty achioving accuraty
Figure 2: An empirical phase transition in training success on 4 datasets (4 rows) for Conv-2 comparing random affine subspaces (column 1) and burn-in affine subspaces with t = 4, 8, 16 burn- in steps (columns 2,3,4). The black-whit color maps indicate the empirically measured success probability (d c, t) in (2.3) in hitting a training loss sublevel set (or more precisely a training accuracy super-level set). This success probability is estimated by training 10 runs every training dimension d and burn-in time L. The horizontal dashed line represents the baseline accuracy obtained by training the full model for the same number of epochs. The colored curves indicate the threshold training dimension d' (c, t, 8) in definition 2. for 5 = 0.1. The threshold training dimensions for the 4 training methods are copied and superimposed in the final column.
which is simple 2-layer CNN with 16 and 32 channels, ReLU activations and maxpool after each convolution followed by a fully connected layer; 2) Conv-3 which is 3-layer CNN with 32, 64. and 64 channels but otherwise identical setup to Conv-2, 2: and 3) ResNet 20v1 as described in He jt al. (2016) with on-the-fly batch normalization (loffe & Szegedy 2015). We perform experiments on datasets: MNIST (LeCun et al.) 2010), Fashion MNIST (Xiao et al. 2017). CIFAR- 10 and CIFAR-100 (Krizhevsky al. al.12014) and SVHN (Netzer al. 2011). Baselines and experiments were run for the same number of epochs for each model and dataset combination; further details on architectures, hyperparameters, and training procedures are provided
appendix. The code for
the experiments was implemented in JAX (Bradbury et al. 2018).
Figure shows results on the training loss for datasets for both random and burn-in affine subspaces with Conv-2. We obtain similar results for the two other architectures (see Appendix). Figure exhibits several broad and important trends. First, for each training method within a random subspace, there is indeed sharp phase transition in the success probability (sd c, t) in the (or equivalently accuracy) by dl plane from 0 (white regions) to (black regions). Second, the threshold training dimension d' (e, L, 0) (with 8 0.1) does indeed track the tight phase boundary separating these two regimes. Third, broadly for each method, to achieve a lower loss, or equivalently higher accuracy, the threshold training dimension is higher; thus one needs more training dimensions to achieve better performance. Fourth, when comparing the threshold training dimension across all 4 methods on the same dataset (final column of Figure 2) we see that at high accuracy (low loss c), increasing the amount of burn in lowers the threshold training dimension. To see this, pick a high accuracy for each dataset, and follow the horizontal line constant accuracy from left to right to find the threshold training dimension for that accuracy. The first method encountered with the lowest threshold training dimension burn-in with = 16. Then burn-in with and on, with random affine having the highest. Thus the main trend is, for accuracies, burning more information into the initialization by training on the training data reduces the number of subsequent training dimensions required achieve the desired accuracy.
has a higher threshold training dimension
range of desired
4

---

Published as a conference paper at ICLR 2022
Figure shows the threshold training dimension for each train and test accuracy level for all three models on MNIST, Fashion MNIST and CIFAR-10 10. The broad trends discussed above hold robustly for both train and test accuracy for all three models.
Conv-3
ResNeet-2
Rull 10/00000
Text Pocurny
trai
Tost
0.30
1.00
03:00
1.00
0.25
Traiing CITAAOIOO
Turning demoraloo
Tranning Sirertiinn
Traning dirimuo
Tarning cirersinn
Tuining dewomsioon
SaTH spEcio
Suboppao
(4)
(38)
Lettery Sutrspoo
Here we focus on small dimensions and lower desired accuracies emphasize the differences threshold training dimension across different training methods. The purple curves are generated via a novel lottery subspace training method which we introduce in section 4 The curves summarize data for 10 runs for Conv-2, 5 rums for Conv-3, and runs for ResNet 2 0; the choice of 5 will determine how many runs must successfully hit the sublevel set when reading off d" The dimensions of the full parameter space for the experiments with CIFAR-10 are 25.6k for Conv-2, 2, 66.58 for Conv-3, and 272.5k for Restet20. On the other two datasets, the full parameter space is 20.5k for Conv-2, 61.5k for Conv-3, and 272.2k for ReaNet20 The black dotted line is the accuracy obtained by training the full model for the same number of epochs.
Figure 3: The threshold training dimension d' (e, t, 6) in definition 2.1
A THEORY OF THE PHASE TRANSITION IN TRAINING SUCCESS
Here we aim to give theoretical explanation for the major trends observed empirically above, namely: (1) there exists a phase transition in the success probability P. (d, e, t) yielding a phase boundary given by threshold training dimension d' (e, 8); (2) fixed and this threshold increases as the desired loss decreases (or desired accuracy increases), indicating more dimensions are required perform better; (3) at fixed e and this threshold decreases the burn-in time increases, indicating fewer training dimensions are required to achieve given performance starting from a better burned-in initialization. Our theory will build upon several aspects of high-dimensional geometry which we first review. In particular we discuss, in turn, the notion of the Gaussian width of set
a set, then Gordon's 's escape theorem, and then introduce notion local angular dimension about point. Our final result, stated informally, will be that the threshold training dimension plus the local angular dimension of a desired loss sublevel set about the initialization must equal the total number of parameters D. As will see, this succinct statement will conceptually explain the major trends observed empirically. First we start with the definition of Gaussian width:
Definition 3.1 (Gaussian Width). The Gaussian width of a subset RPD is given by (see Figure
1 2B sup (g.x y). g N~N.0..
w(
2
x.ES
As simple example, let S solid la ball of radius r and dimension d & embedded in R.P. Then its Gaussian width for large is well approximated by w(S) rvdd
Gordon's escape theorem. The Gaussian width w(S) of a set S, least when that set is contained in a unit sphere around the origin, in turn characterizes the probability that a random subspace intersects that set, through Gordon's escape theorem (Gordon 1988):
5

---

Published as a conference paper at ICLR 2022
Vuit
alticr saleppece
Projertire
g.y x)
ga)
Neer
Far ManifAl
Muntlidl
Figure 4: Left panel: An illustration of measuring the width of a set S (in green) in a direction by identifying x, y E S in maxx,yes g (y x). The expectation of this width using random vectors g (0, IDD) instead of g twice the Gaussian width w(S). Intuitively, it is the characteristic extent of the set T over all directions rescaled by factor between D/ panel: Illustration of projecting manifolds on the unit sphere and Gordon's 's escape theorem. The same manifold far from the sphere will have a smaller projection to it than the one that close, and therefore it will be harder to intersect with an affine subspace.
1 and vD. Right
Theorem 3.1. (Escape Theorem] Let S be closed subset of the unit sphere in IR. If > w S) then d D dimensional subspace Y drawn uniformly from the Grassmannian satisfies (Goddon 1988):
P(Yns=0) 3.5 - (((k///+T
A clear explanation of the proof can be found in Mixon 2014
Thus, the bound says when (S)) the probability of no intersection quickly goes to e for any 0. Matching lower bounds which state that the intersection occurs with high probability when & I wS)) have been proven for spherically convex sets (Amelunxen et al. 2014). Thus, this threshold is sharp except for the subtlety that you only guaranteed to hit the spherical convex hull of the set (defined on the sphere) with high probability.
When expressed in terms of the subspace dimension d = D - k. rather than its co-dimension k these results indicate that a d-dimensional subspace will intersect a closed subset of the unit sphere around the origin with high probability if and only if d + w(S)² > D, with a sharp transition the threshold d' = D w(S)². This is a generalization of the result that two random subspaces in IR of dimension d and d2 intersect with high probability if and only d + de > D. Thus we can think of w( 2 as playing a role analogous to dimension for sets on the centered unit sphere.
3.1 INTERSECTIONS OF RANDOM SUBSPACES WITH GENERAL SUBSETS
To explain the training phase transition, we must now adapt Gordon's escape theorem to a general loss sublevel set S in R._ and must take into account that the initialization is not at the origin in weight space. To do so, we first define the projection of a set S onto a unit sphere centered at w,:
projw, (S) = ((x wt)/llx well2 x S).
(3.1)
Then we note that any affine subspace Y of the form in eq. (.22) centered w, intersects S if and only if it intersects projw. (5). Thus we can apply Gordon 's escape theorem to projw, (S) to compute the probability of the training subspace in eq. (2.2) intersecting sublevel set S. Since the squared Gaussian width of set in a unit sphere plays a role analogous to dimension, we define:
Definition 3.2 (Local angular dimension). The local angular dimension of a general set S C R
about point wa is defined
(S, w.) w² (projw, (S)).
(3.2)
An escape theorem for general sets S and affine subspaces now depends on the initialization W, also, and follows from the above considerations and Gordon's original escape theorem:
Theorem 3.2. MMain Theorem] Let S be closed subset of R.D. If k. (S))) then
==1 D k dimensional affine subspace drawn uniformly from the Grussmannian and centered w. satisfies:
(k/V/ P PYYys- -0) -1- -x.5eexp
[
ww(proj. (S))) /18)

---

Published as a conference paper at ICLR 2022
To summarise this result in the context of our application, given an arbitrary loss sublevel set S(c), training subspace of training dimension d starting from an initialization w, will hit the (convex hull) of the loss sublevel set with high probability when d + dhoca((((e) w.) > D, and will miss it (i.e
have empty intersection) with high probability when d + dhocal (S(e), w.) < D. This analysis thus
establishes the existence of phase transition in the success probability P. (. e, moreover establishes the threshold training dimension d' (e, t, 9) for small values of ó in definition 2.1
in eq. (2.3) and
d" (S(c), w:) = D diocal (S(c), Wt).
(3.3)
Our theory provides several important insights on the nature of threshold training dimension. Firstly, small threshold training dimensions can only arise if the local angular dimension of the loss sublevel set S(c) about the initialization W. is close to the ambient dimension. Second, as e increases, S(c) becomes larger, with a larger diocal (S(c). we), and consequently a smaller threshold training dimension. Similarly, if w. is closer to S(e), then (S(e), w:) will be larger, and the threshold training dimension will also be lower (see fig. 4). This observation accounts for the observed decrease in threshold training dimension with increased burn-in time t. Presumably, burning in information into the initialization w. for a longer time brings the initialization closer to the sublevel set S(c), making it easier to hit with random subspace of lower dimension. This effect is akin to staring out into the night sky in a single random direction and asking with what probability we will see the moon; this probability increases the closer are to the moon.
3.2 A PARADIGMATIC LOSS LANDSCAPE EXAMPLE: THE QUADRATIC WELL
Smuubtedd boosar spectuun
Demire-11
1.0
Summary
Dirance-11
2.8
0.0
Eirance-1
0.6
0.4
2.4
10
02
2.2
0.0
Train dimendood d
Thaiing dmeralnnd
Traning dimemiion
Cutting plane mana
muuMiaa spotrum
Dotercenth
Dslancem30
Summary
92
sareica
0.0
208
0.4
0.4
0.4
12
band
Drttnnce-10
0.5
Trasing danasan
C.ting plave marro
Figure 5: A comparison between simulated results and our analytic upper bound for threshold training dimension of sublevel sets on a synthetic quadratic well. The middle 3 columns show the success probability (d, e, R) as a function of and for three different values of the distance R between initialization and the global minimum, clearly exhibiting a phase transition (black and white maps). This success probability is estimated from a numerical experiment across 10 runs and the estimated threshold training dimensions are shown as solid curves. Our analytic upper bounds on threshold training dimension obtained from our lower bound on local angular dimension in Eq. 3.4 are shown as dashed curves. The top row shows the case of quadratic basin with a bimodal split of eigenvalues; the local angular dimension is approximately the number of long directions (small eigenvalues) and we start hitting low-loss sublevel sets at D/2 as expected. The bottom row shows the case of continuous bulk spectrum. In both cases, threshold training dimension is lowered as the distance R is decreased. The upper bound is tighter when is close 0, the regime of are most interested in.
To illustrate our theory, we work out the paradigmatic example of a quadratic loss function L(w) 7 w Hw where w E Idd and H E DxD is a symmetric, positive definite Hessian matrix. A sublevel set S(e) of the quadratic well is an ellipsoidal body with principal axes along the eigenvectors é; of H. The radius r; along principal axis é; obeys { = e where XE is the eigenvalue. Thus r. = v //2//, Ar. and so large (small) Hessian eigenvalue leads to narrow (wide) radius along each principal axis of the ellipsoid. The overall squared Gaussian width of the sublevel set obeys (S(())) ~ 2e Tr(H-1) > where ~ denotes bounded above and below by this expression times positive constants (Vershynin 2018)
7

---

Published as a conference paper at ICLR 2022
We next consider training within random subspace of dimension d starting from some initialization Wo- To compute the probability the subspace hits the sublevel set S(e), as illustrated in Fig. 4 must project this ellipsoidal sublevel set onto the surface of the unit sphere centered at Wo- The Gaussian width of this projection proiw, (S(e)) depends on the distance R = ||wol| from the initialization to the global minimum at = 0 (i.e. it should increase with decreasing R). We can develop a crude approximation to this width as follows. Assuming D > 1, the direction é; will be approximately orthogonal to Wo, so that |é, Xoo < R. The distance between the tip of the ellipsoid at radius r; along principal axis e, and the initialization Wo is therefore P3 = V
R2 + r The ellipse's s radius r; then gets scaled down to approximately ri/ / R +r? when projected onto the surface of the unit sphere. Note the subtlety in this derivation is that the point actually projected onto the sphere is where a line through the center of the sphere lies tangent to the ellipse rather than the point of fullest extent. As result, r R2 + provides lower bound to the projected extent on the circle. This is formalized in the appendix along with an explanation as to why this bound becomes looser with decreasing R. Taken together, lower bound on the local angular dimension of (()) about Wo is:
diocal (c, R) = (projjo ~
Σ R
(3.4)
where again r3 = /2c/A, In Fig. we plot the corresponding upper bound on the threshold training dimension, Lee. D diocal (c, R) alongside simulated results for two different Hessian spectra.
4 CHARACTERIZING AND COMPARING THE SPACE OF PRUNING METHODS
Training within random subspaces is primarily a scientific tool to explore loss landscapes. It further has the advantage that we can explain theoretically why the number of degrees of freedom required to train can be far fewer than the number of parameters, as described above. However, there are many other pruning methods of interest. For example, the top row of Table I focuses on pruning to axis-aligned subspaces, starting from random weight pruning, to lottery tickets which use information from training to prune weights, and/or choose the initialization not rewound to init. As one moves from left to right, one achieves better pruning (fewer degrees of freedom for given accuracy). Our analysis can be viewed as relaxing the axis-aligned constraint to pruning to general subspaces (second row of Table either not using training all (random affine subspaces) or using information from training to only to choose the init (burn in affine subspaces). This analogy naturally leads to the notion of lottery subspaces described below (an analog of lottery tickets with axis-alignment relaxed to general subspaces) either rewound to init or not (last two entries of Table D]. We compare the methods we have theoretically analyzed (random and burn-in affine subspaces) to popular methods like lottery tickets rewound to init, and our new method of lottery subspaces, in an effort understand the differential efficacy of various choices like axis-alignment, initialization, and the use of full training information to prune. A full investigation of table however is the subject of future work.
Table I: Taxonomy of Pruning Methods.
Training
Training used
Training used for pruning only Lottery tickets, rewound to init.
Training used for init. and pruning
not used
for init. only
Axis-aligned subspaces
Random weight pruning
Random weight pruning at step Burn-in affine at step /
Lottery tickets. rewound to step Lottery subspaces at step
General
Random affine
Lottery subspaces
subspaces
subspaces
Lottery subspaces. We first train the network in the full space starting from an initialization Wo- We then form the matrix Ua E Dxd whose d columns are the top d principal components of entire the training trajectory Wo.7 (see Appendix for details). We then train within the subspace w(O)= = 0,0 + w, starting from a rewound initialization w, (f = 0 is rewinding to the original init).
Since the subspace is optimized to match the top d dimensions of the training trajectory, we expect lottery subspaces to achieve much higher accuracies for a given training dimension than random or potentially even burn-in affine subspaces. This expectation is indeed borne out in Fig. 3 (purple lines above all other lines). Intriguingly, very few lottery subspace training dimensions (in the range of 20 to 60 depending the dataset and architecture) are required attain full accuracy, and thus lottery
8

---

Published as a conference paper at ICLR 2022
subspaces can set a (potentially optimistic) target for what accuracies might be attainable by practical network pruning methods as a function of training dimension.
Conv-2
Com
ResNet--0
Full tram
acEureey
1.00
1945T
0.50
0.25
0.25
0.75
0.22
10'
10
10"
12"
10
10"
10"
10
16
10"
10'
10
10"
10
10"
10"
10
12"
10"
10"
Comppessaion ratlo
Compression on ratlo
Comp
Comprssston catto
Compress radlo
Comp Wedion 190to
Full modd, come spochs
Rondom Affice Subssaac
Bum-in (4)
Sum-in (8)
Damii (16)
Lostery Subopace
Lottery Ticket
Figure 6: Accuracy vs. compression ratio for the same data. Compression ratio is defined the number parameters in the full model over the dimension the subspace (D/d). The dimensions of the full parameter space for the experiments with CIFAR-10 10 are 25.6k for Conv- 2, 66.5k for Conv-3, and 272.5k for ResNet On the other two datasets, the full parameter space is 20.5k for Conv-2, 61.5k for Conv-3, and 272.2k for ResNet20. The curve for each lottery ticket experiment summarizes data for at least runs. For all other experiments, the curve summarizes data for 10 runs for Conv-2, accuracy of the full model run for the same number of epochs.
2, 5 runs for Conv and 3 runs for ResNet20. Black dotted lines are the
Empirical comparison of pruning methods. Figure 6 presents empirical results comparing a subset of the methods in table [ random affine subspaces, burn-in affine subspaces, lottery subspaces, and lottery tickets plotted against model compression ratio (defined as parameters in full model over parameters, or training dimension, in restricted model). The lottery tickets were constructed by training for 2 epochs, performing magnitude pruning of weights and biases. rewinding to initialization, and then training for the same number of epochs as the other methods. Note that lottery tickets are created by pruning the full model (increasing compression ratio) in contrast to all other methods which are built up from a single dimension (decreasing compression ratio). We observe lottery subspaces significantly outperform random subspaces and lottery tickets at low training dimensions (high compression ratios), and we explore the spectrum of these spaces in more detail in the Appendix.
The comparison to lottery tickets at low compression ratios is limited by the fact that it is com- putationally expensive to project to higher dimensional subspaces and thus the highest training dimension we used was 4096. In the regions where the experiments overlap, the lottery tickets do not outperform random affine subspaces, indicating that they are not gaining an advantage from the training information they utilize. A notable exception is Conv-2 on CIFAR-10 in which the lottery tickets do outperform random affine subspaces. Finally, we note lottery tickets do not perform well at high compression ratios due the phenomenon of layer collapse, where entire layer gets pruned.
5 CONCLUSION
The surprising ability of pruning methods like lottery tickets achieve high accuracy with very few well-chosen parameters, and even higher accuracy if not rewound to initialization but to a later point training. has garnered great interest in deep learning but has been hard to analyze. In this paper, we focused on gaining theoretical insight into when and why training within random subspace starting at different initializations (or burn-ins) along a full training trajectory can achieve a given low loss e. We find that this can occur only when the local angular dimension of the loss sublevel set S(c) about the initialization is high, or close to the ambient dimension D. Our theory also explains geometrically why longer burn-in lowers the number of degrees of freedom required to train to given accuracy, This is analogous to how rewinding to a later point in training reduces the size of lottery tickets, and indeed may share similar mechanism. Overall, these theoretical insights and comparisons begin to provide a high-dimg fith-dimensional geometric framework to understand and assess the efficacy wide range of network pruning methods at beyond initialization.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
B.W.L. was supported by the Department of Energy Computational Science Graduate Fellowship program (DE-FG00-97ER25306 S.G. thanks the Simons Foundation, NTT Research and an NSF Career award for funding while at Stanford.
REFERENCES
Dennis Amelunxen, Martin Lotz, Michael B McCoy, and Joel A Tropp. Living on the edge: Phase transitions in convex programs with random data. Information and Inference: A Journal of the IMA, 3(3):224-25) 294, 2014.
Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstenn and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, March 2020.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:/ //gitub. com ctmm /omm//oooe/aaxxx
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in Neural Information Processing Systems, pp. 2933 1931,994,110 2014.
Stanislav Fort and Surya Ganguli. Emergent properties of the local geometry of neural loss landscapes. arXiv preprint arXiv:1910.05000 2019.
Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes, 2019.
Stanislav Fort and Adam Scherlis. The goldilocks zone: Towards better understanding of neural network loss landscapes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3574-3581, 2019.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution the neural tangent kernel, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint (xXi:22009.08776, 2020.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density, 2019.
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv: 1412.6544 2014.
Yehoram Gordon. On milman's 's inequality and random subspaces which escape through mesh in R In Geometric aspects of functional analysis, pp. 106. Springer, 1988.
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04F 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456.. PMLR. 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http:///ww. cs. toronto. edu/kriz/cifar html, 55:5, 2014.
Yann LeCun, Corinna Cortes, and Christopher J Burges. Mnist handwritten digit database. 2010. URL http://yann. lecun. com/exdl/mmnsst, 7:23, 2010.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H S Torr. SNIP: Single-shot network pruning based connection sensitivity. October 2018.
Aitor Lewkowyez, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism, 2020.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes, 2018
Dustin
G Mixon.
Gordon's escape
through mesh theorem, Feb
2014. gordons-escape
URL https://dustinggngg xon. wordpress ccom/201402//0
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Vardan Papyan. Measurements three-level hierarchical structure in the outliers the spectrum of deepnet hessians, 2019.
Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1-64, 2020.
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond, 2016.
Levent Sagun, Utku Evei, V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis the hessian of wnerpparametrized neural networks, 2017.
Hidenori Tanaka. Daniel Kunin, Daniel L K Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. February 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1708.07777 2017.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W. Mahoney. Hessian-based analysis of large batch training and robustness to ad adversaries, 2018
11

---

Published as a conference paper at ICLR 2022
A EXPERIMENT SUPPLEMENT
The core experiment code is available on Github: https://github htw.:/tttu..ooom/gggggggug- degrees of freedom The three top-level scripts are burn_in subsspac_e py, py. Random affine experiments were run in the burn-in subspace code. The primary automatic
lottery_ suseppce_sbppceeeeeee py, and lottery_tickeene by setting the parameter init _iters differentiation framework used for the experiments was JAX BBradbury et al. (2018). The code was developed and tested using JAX v0.1.74, JAXlib v0. 1.52, and Flax v0.2.0 and run on an internal cluster using NVIDIA TITAN Xp GPU's.
Phone mandsces In advecable socirscy WETJ taiiing dimension
Thresheld traiiig direedon
06
0.0
0.0
1.0
0.8
0.0
Booid acORs0y
Ror Kandon Affiwe Subssace
Burnit
Ber
Butn (24)
Probald by adlaning hdted accaACy
Figure 7: An empirical phase transition in training success on 3 datasets (3 rows) for Conv-3 comparing random affine subspaces (column and burn-in affine subspaces with i = 4, 8, 16 burn- in steps (columns 2,3,4). The black-white color maps indicate the empirically measured success probability (d t) in (2.3) in hitting a training accuracy super-level set. This success probability estimated by training
runs every training dimension d. The horizontal dashed line represents the baseline accuracy obtained by training the full model for the same number of epochs. The colored curves indicate the threshold training dimension df (e, t, in definition 2.1 for d 0.2.
Phese nations achacable
Thrashold
0.8
02
0.8
0.0
302
Figure 8: An empirical phase transition in training success 3 datasets (3 rows) for ResNet20 comparing random affine subspaces (column 1) and burn-in affine subspaces with / = 8, 16 burn- in steps (columns 2.3). The black-white color maps indicate the empirically measured success probability (d e, t) in (2.3) in hitting training accuracy super-level set. This success probability estimated by training on runs at every training dimension d. The horizontal dashed line represents the baseline accuracy obtained by training the full model for the same number of epochs. The colored curves indicate the threshold training dimension d" (c. t, in definition 2.1 for 8 = 0.33.
12

---

Published as a conference paper at ICLR 2022
Figures 7 and & show the corresponding empirical probability plots for the two other models con- sidered in this paper: Conv-3 and ResNet20, These plots are constructed in the same manner as fig. 2 except larger value of 8 was used since fewer runs were conducted (6 was always chosen such that all but one of the runs had to successfully hit training accuracy super-level set). The data in these plots is from the same runs figs. and
A.1 COMPARISON TO LINEARIZED NETWORKS (NEURAL TANGENT KERNEL)
For general neural networks, do not expect be able bound the local angular dimension; instead, we use the relationship between the threshold training dimension and local angular dimension to empirically probe this important property of the loss landscape in the experiments of fig. 3 For a single basin, we can consider the second-order approximation to the landscape at the optimum which yields a quadratic well based the spectrum of the Hessian at this point, corresponding the experiments presented in fig. 5 Jusing a well with the appropriate spectrum. In this section, we consider how linearizing via the neural tangent kernel (NTK) can be used tool better approximate the landscape of the network around single basin while being potentially more amenable to theoretical characterization than the full network.
1.0
prijls ept (knosloytise poin))
2012
2p12
Food lest AECUREY
Origiaal
0.6
0.4
0.2
0.0
10
10
10
metion
Training
acturacy
Linerized 54t Cred P9a0
Conv2
erigin cot 221
orggn
Orighal
lood text uncry
FashionhhSS
Weo deeche
0.G
0.4
10P
122
102
107
Weggh Sraccer
Tratiing lout 204901
Figure 9: Empirical comparison of the threshold training dimension for the full landscape vs. the linearized model around specific local optimum. These experiments were run on the Conv-2 model; the top row is on MNIST and the bottom is on Fashion MNIST. The column show a 2- dimensional cut of the test loss landscape defined by the initialization point and two optimum found via training. The second column shows the same cut but for the linearized model around optimum 1. Finally, the right column shows the threshold training dimension for both the original and linearized model.
For this experiment we first train in the full network starting from initialization Wo until we find a local optimum Wopt- Instead of using the second-order approximation around this optimum given by the Hessian, we linearize the network around Wape via the NTK aaacot al., 2018). In essence, if f(w, x) is the function that outputs the ith logit for a given input x instead consider the following approximation which is a linear function in w:
f(Wopt + w. RJ / (Wape, x) + / f x)) w = A(Wopt, x) + B ((Wopt, x) w
At Wopt: the full and linearized network are identical; however, in the linearized network there is only one basin which is around Wopt- We then compare these networks by returning to the initialization point Wo and perform the experiment training within random affine subspaces across range of dimensions in both the full and linearized network.
13

---

Published as a conference paper at ICLR 2022
Figure 9 shows the results of this experiment for both MNIST and Fashion MNIST using the model Conv-2. 2. In these two settings, the threshold training dimension of the linearized model approximates this property of the full model fairly well, indicating promise useful approximation to the true loss landscape around a basin. Thus, we consider theoretically characterizing the local angular dimension of these linearized models interesting future work.
A.2 SPECTRA OF LOTTERY SUBSPACES
feshionMNIST
CUFAR-100
1.00
1.00
0.75
3.700
0.00
0.20
000
0.00
1.00
1.50
154
251
3M
$19
126
206
3F1
510
130 250 304 312
120
201
5/2
130
206
311
212
120
202
201 242
1ho0 Samm Dirmension
syperpians Dimerato
Hoperpara Demeration
Hepertinn Demaratoe
Figure 10: In the main text, we plot a running max accuracy applied to the individual runs because the subspaces are nested and we are only concerned with the existence of an intersection. The accuracies are plotted here without this preprocessing step for comparison with the spectra. Left: Singular values for the lottery subspace experiments on MNIST. Right: Singular values for the lottery subspace experiments on Fashion MNIST and an additional run on CIFAR- 100. Only the first spectra (out of 10) are shown for Conv-2. Directions were added in order of descending singular values.
In experiments, we formed lottery subspaces by storing the directions traveled during a full training trajectory and then finding the singular value decomposition of this matrix. As we increased the subspace dimension, directions were added in order of descending singular values. Figure 10 and
the left panel of fig. 11 show the associated spectra for the results presented in figs. 3 and 6 Note that in the main text figures we plot the accuracies as running max over the current and smaller dimensions. This is because the subspaces are nested such that if we increase the dimension and find a point of lower accuracy, it indicates failure of optimization to find the intersection as the higher accuracy point is still in the subspace. In these supplement figures, we plot the recorded accuracies without this processing step for completeness. We see that in several cases this optimization failure did occur as we moved to higher dimensions; we suspect this is related to how quickly the singular values fall off meaning the higher dimensions we add are much less informative.
The spectra are aligned with the train and test accuracy plots such that the value directly below a point on the curve corresponds to the singular value of the last dimension added to the subspace. There were 10 runs for Conv-2, 5 for Conv-3, 3, and 3 for ResNet20. Only the first 5 out of 10 runs are displayed for the experiments with Conv-2. No significant deviations were observed in the remaining runs.
From these plots, we observe that the spectra for a given dataset are generally consistent across architectures. In addition, the decrease in accuracy after a certain dimension (particularly for CIFAR- 10) corresponds to the singular values of the added dimensions falling off towards 0.
The right panel of fig. II shows tangential observation that lottery subspaces for CIFAR- 10 display a sharp transition in accuracy at d = 10. This provides additions evidence for the conjecture explored by Gur-Ari et al. (2018) Fort & Ganguli (2019), and Papyan 2020) that the sharpest directions of the
14

---

Published as a conference paper at ICLR 2022
CIFAR-10
Con-3 3
0.75
025
0.2
000
1.20
0.4
2:53
0.0
2:23
1.20
0.4
-20
250
012
-20
250
204
208 120
304
Hypertaaee dinnension
Hapergtone divennsion
Figure 11: In the main text, we plot running max accuracy applied to the individual runs beacuse the subspaces are nested and we are only concerned with the existence of an intersection. The accuracies plotted here without this preprocessing step for comparison with the spectra. Left: Singular values for the lottery subspace experiments on CIFAR- 10. Only the first spectra (out of 10) are shown for Conv-2. Directions were added in order of descending singular values. Right: Lottery subspaces display accuracy transition around d = 10 for the dataset CIFAR-10. Provides additional evidence for the conjecture that the shaprest directions of the Hessian are each associated with class but no learning happens in them.
Hessian and the most prominent logit gradients are each associated with class. Very little learning happens in these directions, but during optimization you bounce up and down along them so that the prominent in the SVD of the gradients. This predicts exactly the behavior observed.
A.3 ACCURACY OF BURN-IN INITIALIZATION
Figure 12 shows a subset of the random affine and burn-in affine subspace experiments with a value plotted at dimension 0 to indicate the accuracy of the random or burn-in initialization. This is to give context for what sublevel set the burn-in methods are starting out, enabling us to evaluate whether they are indeed reducing the threshold training dimension of sublevel sets with higher accuracy. In most cases, as we increase dimension the burn-in experiments increase in accuracy above their initialization and at faster pace than the random affine subspaces. A notable exception is Conv-3 on MNIST which the burn-in methods appear to provide no advantage.
A.4 HYPERPARAMETERS
Random hyperplanes were chosen by sampling a D x matrix of independent, standard Gaussians and then normalizing the columns to 1. This equivalent sampling uniformly from the Grassmanian as required by theorem 3.2 Optimization restricted to an affine subspace was done using Adam Kingma & Ba (2014) with 3 0.9. 82 0.999, and e = 10-7. We explored using 5 10-2 and 10 for the learning rate but 5 10 worked substantially better for this restricted optimization and was used all experiments; a batch size of 128 was used. The full model runs used the better result of 5- 1002 and 1 for the learning rate. ResNet20vi was run with on-the-fly batch normalization loffe & Szegedy (2015), meaning we simply use the mean and variance of the current batch rather than maintaining running average. 2 the number of epochs used for each dataset and architecture combination across all experiments. 3 epochs was chosen by default and then increased if the full model was not close convergence.
15

---

Published as a conference paper at ICLR 2022
Comv-2
Conv-3
Realvet-2
0.75
Prppeplorn
dimeraaon
Figure 12: First 128 dimensions for a subset of the random affine and burn-in affine subspace experiments. The plots include value at dimension 0 which indicates the accuracy of the random initialization or the burn-in initialization.
Table 2: Epochs Used for Experiments.
Dataset
MNIST
Fashion MNIST
CIFAR-10
CIFAR-100
SVHN epochs
Conv-2
3 epochs
epochs
4 epochs
Conv-3
4 epochs
5 epochs
5 epochs
5 Spochs
ResNet20
3 epochs
3 epochs
4 epochs
B THEORY SUPPLEMENT
In this section, provide additional details for study of the threshold training dimension of the sublevel sets of quadratic wells. We also derive the threshold training dimension affine subspaces to provide further intuition.
B.1 PROOF: GAUSSIAN WIDTH SUBLEVEL SETS THE QUADRATIC WELL
In our derivation of eq. 3.4 we employ the result that the Gaussian width squared of quadratic well sublevel sets is bounded as w (S(c)) ~ 2c 1r(H-1)) Σ, r?. i.e. bounded above and below by this expression times positive constants. This follows from well-established bounds the Gaussian width of an ellipsoid which we now prove.
In our proof, will use equivalent expression for the Gaussian width of set S:
w(S)
sup (g.x y) (g.x). x.yes
xe3 g NOO,IDx
16

---

Published as a conference paper at ICLR 2022
Lemma B.1 (Gaussian width of ellipsoid), Let & be an ellipsoid in RR defined by the vector r R with strictly positive entries as:
:
E XERD Σ
Then w(E)? or the Gaussian width squared of the ellipsoid satisfies the following bounds:
2
2
w(8) -
j=1
Proof.
Let ~ IDxD. Then we upper-bound w( by the following steps:
D
w(E)= = Eg
sup
g:ay
X
F2
Eg
sup
9:
1
X
1/2
1/2
Eg
sup X
Σ
7
2 2 Σ
Cauchy-Schwarz inequality
i=1
< Eg
95
ii
2
Definition of E
D
Σ
Jensen's inequality
1/2
B Blwa]
-1
giving the upper bound in the lemma. For the lower bound, we will begin with a general lower bound for Gaussian widths using two facts. The first is that if e; are i.i.d. Rademacher random varaibles and, then G/ggs| ~ (O, 1). Second, we have:
Blg9:1| =
1
lyle 12l-v
ye-v 12/2 2
2m
/2 Jo
Then for the Gaussian width of a general set:
(S))
E
sup SES
W
E,
E
supp ZES
Eim
Using esgg.l NO 1)
i=1
E,
sup
c E Egg Iaa
Jensen's Inequality
E
supp
c
ZES i=1
17

---

Published as a conference paper at ICLR 2022
All that remains for our lower bound to show that for the ellipsoid E supeee Ei= G
1/2
EP r3
We begin by showing it an upper-
E
sup
sup
[
Using & symmetric
.EE
i=1
T3T's
F:
sup
EE i=1
Vi
T
D
1/2
D
1/2
< sup
Σ
Cauchy-Schwarz inequality
xEE
1/2
Definition of &
In the first line, we mean that & is symmetric about the origin such that we can use = 1 for all i without loss of generality. Finally, consider x such that 2; = = (Eir have E & and:
1/2
For this choice we
1/2
D
D
Σ
Σ
2
1/2
Li=
showing that equality is obtained in the bound. Putting these steps together yields the overall desired lower bound:
D
1/2
2
(( 2
E
sup
C3
=
DE
With this bound in hand, immediately obtain the following corollary for quadratic well defined by Hessian H. The Gaussian width invariant under affine transformation so we can shift the well to the origin. Then note that S(e) is an ellipsoid with r; = 22e and thus Σ, r c cTr(H-l)
Corollary B.1 (Gaussian width of quadratic sublevel sets). Consider a quadratic well defined by Hessian H D2Dx Then the Gaussian width squared of the associated sublevel sets S(e) obey the following bound:
2 20 2e 26 &
12 π
B.2 DETAILS ON THRESHOLD TRAINING DIMENSION UPPER BOUND
In section 3.2
we consider the projection of ellipsoidal sublevel sets onto the surface of a unit sphere centered Wo- The Gaussian width of this projection proj (S(e)) will depend on the distance 0 (i.e. it should increase with
R = Wwl| from the initialization to the global minimum at decreasing R). We used crude approximation to this width as follows. Assuming D > 1, the direction é; will be approximately orthogonal to wo. so that (ē; Xol < R. The distance between the tip of the ellipsoid at radius r; along principal axis e, and the initialization Wo therefore
P. = RR² + r.. The ellipse's radius ry then gets scaled down to approximately r / R + + when projected onto the surface of the unit sphere.
We now explain why this projected size always a lower bound by illustrating the setup in two dimensions in fig. [3] As shown, the linear extent of the projection will always result from line that tangent to the ellipse. For an ellipse (z/a)² + ((y R)/b)² = and a line y = cr in two-dimensional space (we set the origin at the center of the unit circle), line tangent to the ellipse
18

---

Published as a conference paper at ICLR 2022
must satisfy a/vR2 - 62. That means that the linear extent of the projection unit circle will be a/v v/2va22 R 82. For = /x; and R = R, this is exactly Eq. 3.4 provided b 0. The b o will always make the linear projections larger, and therefore Eq. [.4]3.4ww will be lower bound on the projected Gaussian width. Furthermore, this bound will be looser with decreasing R. We then obtain a corresponding upper bound on the threshold training dimension, i.c. D diocal(e, Rt).
a
R
Tangent line
a²
0² +
R
a²
Unit circle
a + R B
Illustration in dimensions why the projection of the principal axes of ellipse the unit circle will lower bound the size of the projected set. The linear extent of the projection will result from a line that lies tangent to the ellipse.
Figure 13:
B.3 THRESHOLD TRAINING DIMENSION OF AFFINE SUBSPACES
we considered the threshold training dimension of the sublevel sets of a quadratic well and showed that it depends on the distance from the initialization to the set, formalized in eq. (3.4). As point of contrast, we include derivation of the threshold training dimension of random affine subspace in ambient dimension D and demonstrate that this dimension does not depend on distance to the subspace. Intuitively this is because any dimension in the subspace is of infinite or zero extent, unlike the quadratic sublevel sets which have dimensions of finite extent.
In Section 3.2
Let us consider D-dimensional space for which we have randomly chosen d-dimensional affine subspace defined by a vector offset X IRP and set of d orthonormal basis vectors {f that we encapsulate into a matrix M E I D. Let us consider another random nedimensional affine subspace B. Our task is to find point x i..::
E A that has the minimum & distance to the subspace B,
x = argmin xea
x - ---rgminn''n ||x - x' x2112 ll
In words, we looking for point in the d-dimensional subspace A that is close as possible to its closest point in the n- n-dimensional subspace B. Furthermore, points within the subspace A can be parametrized by a d-dimensional vector 0 TRd x(0) 0M xo A; for all choices of 0, the associated vector x is in the subspace A.
Without loss of generality, let us consider the where the basis vectors of the subspace B aligned with the dimensions D - n, D + 1,...,D D of the coordinate system (we can rotate our D n the short directions of
coordinate system such that this true). Call the remaining axes the subspace B. A distance from a point x to the subspace B now depends only on its coordinates 1, 2. ......... s. Under our assumption of the alignment of subspace B we then have:
(xx B) argmin eeB ||x
x
The only coordinates influencing the distance are the first s values, and thus let us consider a R* subspace of the original RDD only including those without loss of generality. Now 0 Rd. M Rdxs
19

---

Published as a conference paper at ICLR 2022
and Xy E R and the distance between a point within the subspace A parameterized by the vector is given by:
(xx(0),B) B) = |0M + xoll?.
The distance I attains its minimum for
Ogl² (x(0), B) 2- (OM + Xo) M -0
yielding the optimality condition 0"M -x- There are 3 cases based on the relationship between d and
The overdetermined case, 8. case d 8 = n, the optimal 0 = -x,M- belongs to (d - s + D)-dimensional family of solutions that attain 0 distance to the plane B. In this case the affine subspaces A and B intersect and share a (d + n D)-dimensional intersection.
A unique solution case, s. In case of After plugging this back to the distance equation, we obtain 0 is
D-n,
n. the solution a unique 0 M
P (xx(00), B) = || M M + x
|| -0 x 2 0.
The matrix M is square in this case and cancels out with its inverse M
3. An underdetermined case, d < s. In case of d there is generically intersection between the subspaces. The inverse of M is now the Moore-Penrose inverse M Therefore the closest distance 0:
2 (x((')) B) = =II-xoM* + x
Before our restriction from D + dimensions, the matrix M consisted of d D-dimensional, mutually orthogonal vectors of unit length each. We will consider these vectors to be component- wiss random, each component with variance 1/ 1/VD / satisfy this condition average. After restricting space dimensions, M's vectors are reduced to s components each, keeping their variance 1/ L/VD. They are still mutually orthogonal in expectation, but their length are reduced to V The transpose of the inverse M consists of vectors of the same directions, with their lengths scaled up to vD/ Vs. That means that expectation, MM+ + is diagonal matrix with d diagonal components set 1, and the remainder being O. The matrix (I M*M) M) contains (s d) ones on its diagonal. The projection [Ix, - M M)]l² is therefore of the expected value of | (s d) D. The expected distance between the d-dimensional subspace and the d-dimensional subspace B is:
vD-n-d vD
n n D, n + d D.
E B[d(A, x
10
To summarize, for space of dimension D. two affine subspaces generically intersect provided that their dimensions d and dg add up to at least the ambient (full) dimension of the space. The exact condition for intersection is d + dæ 2 D, and the threshold training dimension of subspace B is D d. This result provides two main points of contrast to the quadratic well:
Even extended directions are not infinite for the quadratic well. While in the case of the affine subspaces even slight non-coplanarity of the target affine subspace and the random training subspace will eventually lead to an intersection, this is not the case for the sublevel sets of the quadratic well. Even its small eigenvalues, i.e. shallow directions, will still have finite extent for all finite
Distance independence of the threshold training dimension. As a result of the dimen- sions having finite extent, the distance independence of threshold training dimension for affine subspaces does not carry over to the case of quadratic wells. In the main text, this dependence on distance is calculated by projecting the onto the unit sphere around the initialization enabling us to apply Gordon's 's Escape Theorem.
20


Reviews:
[{"review_id": "q0SMpLxi8u", "reviewer": "Reviewer_krRB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n-- A new theoretical result that explains the existence of threshold training dimension.\n\n-- The paper is well motivated and well structured. In particular, the authors showed in a toy example that the theoretical lower bound is close to the empirically observed value. The authors provided extensive experimental analysis of transition bounds for different random subspaces.\n\nCons:\n\n-- The lottery subspaces, despite having smaller transition dimensions, encode a lot of information into the linear transformation matrix, therefore applications to compression can be limited.\n\n-- The considered theory does not take into account optimization methods.\n\nQuestions and notes:\n\n-- Figure 2, last column. The training threshold dimensions look similar for burn-in subspaces with different starting iterations. Therefore the argument that a better starting point decreases the threshold dimension does not look convincing. Does one burn-in step correspond to one batch update?\n\n-- Section 2. Loss sublevel sets paragraph. l(f_w(x), y) -> l(f_w(x_n), y_n)\n\n", "summary_of_the_paper": "The modern deep neural networks are over-parameterized and it is possible to construct a smaller parameter space that delivers a similar loss value. If one investigates the dependency of the probability of achieving the desired loss values on training subspace size, one will observe a sharp transition. The dimension where this transition happens the authors call a threshold training dimension. The authors of this paper proposed a theoretical explanation of the existence of the threshold training dimension. In particular, using Gordon’s escape theorem the authors describe the dependence of the threshold training dimension on the initialization and final desired loss. The authors proposed new lottery subspaces for which the threshold training dimension is much higher than for other random subspaces.", "summary_of_the_review": "Overall, I would recommend accepting the paper. It contains interesting theoretical results explaining the existence of threshold training dimensions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "lEGIQnXDdI_", "reviewer": "Reviewer_fVom", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nTheory:\nThis paper introduces a novel concept using the Gordon escape theorem to understand the effect of training in lower-dimensional random subspace of the NN weight space, especially after a burn-in period. Specifically the theorem shows the existence of a phase transition in the success probability of training and this is supported by the experiments as well. \n\nExperiments:\nThey also provide a new training/pruning method based on this phenomena, which performs better than other existing training methods, especially in low-training dimensions\n\nWeaknesses:\n\nThe theorem is a first such characterization, but it is not clear as to the difficulty of estimating the threshold dimensions for more than quadratic loss functions, potentially when involving deep ReLU networks for instance. It would be nice to have a discussion of the challenges for at least shallow networks (1 hidden layer ReLU network for example).\n\nAnother clarification is in Eq 2.2, the randomness of A is from a Gaussian distribution. In the statement of Gordon escape theorem it would be useful to clarify what \"uniformly from a Grassmanian mean\"? Are there any restrictions on how the random subspace is generated? \nAs a continuation, in the Lottery subspaces algorithm, the matrix U is generated by the dynamics of GD on the loss function. Does the Gordon escape theorem still apply for this random process?\n\nIs it possible to infer in the pruning algorithm and the Gordon escape theorem, how the trainable network depends on depth or width of the original network? It would be interesting to see at least experimentally the change in depth/width for large classes of problems, as this ties in with numerous results on expressivity that say in general , increasing depth is more useful than increasing width (from the lens of expressivity alone!).\n\nIt would be useful if the authors can clarify the dependence on the ratio D/d, in the w.h.p statements.\n\n", "summary_of_the_paper": "The paper aims to provide a theoretical explanation for recent observations (lottery tickets, training in random subspaces, spanning pruning) that deep neural networks can be trained using fewer parameters than necessary. They provide the theoretical explanation using the so called Gordon escape theorem from high-dimensional geometry, which says that there is a phase transition in the success probability of training as the training dimension exceeds a threshold and this threshold is rather tight. This is supported by experiments on various benchmark datasets that seem to exhibit this phenomena of this phase transition. ", "summary_of_the_review": "In summary, the authors provide a theoretical explanation for many recent results that support training on a lower-dimensional subspace of the NN weight space and the back the theoretical claim via experimental results that show the existence of such phase transitions. Although, a lot remains to be said, this paper raises more interesting questions and I think this will be of value to the deep learning community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gcsIxNigYWS", "reviewer": "Reviewer_17kR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Gordon's escape theorem characterizes the probability that a given subset of the unit sphere $S\\in \\mathbb{S}^{D-1}$ has an empty intersection with a random $d$-dimensional subspace of $\\mathbb{R}^D$. The randomness here is given by a standard Gaussian matrix and the random subspace is distributed uniformly on the Grassmann manifold w.r.t. the associated Haar measure. The probability of escape (i.e., no intersection) is characterized by a squared version of the Gaussian width of $S$. The Gaussian width is the expected value of the dual norm of a standard Gaussian vector, and for sets containing the origin it coincides with the more familiar notion of Gaussian complexity in statistical learning theory. \n\nThe innovation in this paper is a straightforward adaptation of Gordon's result for estimating the success probability of hitting the convex hull of a loss sublevel set when training within a random subspace of the original parameter space. For the success probability to be large, the subspace codimension $D-d$ must be less than the squared Gaussian width of the desired training loss sublevel set projected onto a sphere around the initialization. The authors identify the latter as the critical threshold training codimension in relation to the observations of Li et al (2018) and empirically demonstrate a phase transition in the success probability in the loss-subspace dimension plane around this threshold. For a quadratic approximation of the loss, the authors give a bound on the threshold dimension in terms of the spectrum of the Hessian and distance to initialization.\n\nThe main strength of the paper lies in exploring the ramifications of the above for training within random and optimized subspaces starting from either a random initialization or after using some information from training the network in the full space (which the authors call \"burn-in\") that apparently brings the initialization closer to a desired loss sublevel set. This explains for instance why the threshold dimension decreases when more information is burned into the initialization. \n\nQuestion: \n\nCan you comment on the estimation of the threshold dimension for general losses (beyond the quadratic approximation)?\n\nMinor nitpicking: \n\n-- The reference to Gordon's theorem both in Gordon (1988) which I believe is Corollary 3.4 and also in the Mixon (2014) blog state the result with coefficient for the exponential as 3.5 instead of 2.5. I can see how one gets the coefficient 2.5, but perhaps for the benefit of the reader, the authors should furnish a reference that accurately reflects their statement of Gordon's theorem in relation to the above.\n\n-- In the second paragraph in page 1, the statement should be \"... this threshold training dimension is equal to the dimension of the full parameter space minus the \"squared\" Gaussian width of the desired loss sublevel set projected onto the unit sphere around initialization.\" (include the squared?).\n", "summary_of_the_paper": "This paper builds on the empirical observation of Li et al (2018) that training neural networks in a low-dimensional random subspace of the parameter space often achieves similar levels of train and test accuracies as training in the original parameter space when the subspace dimension is above a critical threshold. The main result is an application of Gordon's escape theorem showing that the critical threshold training codimension is the squared Gaussian width of loss sublevel sets projected onto a sphere around the initialization.", "summary_of_the_review": "The contribution is a straightforward application of Gordon's result weighing mainly on the empirical side. I believe nonetheless that the paper gives new insights for training in random subspaces. Overall, I like the paper and would be happy to raise my score based on the authors' response.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HwEMjr568PP", "reviewer": "Reviewer_L6aD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The article is well written and clear. It studies an interesting phenomenon with mix of empirical and theoretical results.\n\nThe empirical experiments are detailed and illustrate clearly the transition they are studying.\n\nThe theoretical results are nice and explained well. It is not much more than a corollary of Gordon's escape theorem, but I know of no previous work using it in this context.\n\nThe example analysis for quadratic loss is important, it gives an idea of how one could compute the local angular dimension in practice. I think the authors have missed an opportunity here to draw a link with the loss of DNNs. With the MSE loss and in the NTK regime (with a specific initialization and sufficiently many neurons in the hidden layers), it has been shown that the loss landscape around the training path is approximately quadratic (https://openreview.net/pdf?id=SkgscaNYPS), with spectrum equal to the spectrum of the NTK Gram matrix. In this setting, the local angular dimension of the sublevel sets of the DNN loss should be well approximated by the local angular dimension of the quadratic approximation. This would make it possible to compare the empirical transition with your theory in the DNN case directly.", "summary_of_the_paper": "The paper studies DNNs training when restricted to a random affine subspace (centered either at the initialization point $x_0$ or at a point $x_t$ obtained by training an unrestricted network for $t$ steps, which they call burn-in), and the probability that a loss lower than $\\epsilon$ can be reached with a subspace of dimension $d$. They observe a sharp transition between pairs $(\\epsilon,d)$ where this probability is either almost $0$ or $1$. They observe that the lower the dimension, the less likely one is to reach a loss $\\epsilon$.\n\nThe authors show that this transition can be understood in terms of the Gaussian width of the projection of the sublevel set $S(\\epsilon)$ to the unit ball around the center point $x_0$ (or $x_t$), which they call the local angular dimension. Gordon's escape theorem makes it possible to bound the transition between the two phases as a function of the local angular dimension.\n\nCalculating the local angular dimension of the sublevel sets of the loss of DNNs is difficult, instead the authors study a quadratic loss, giving an approximation for the local angular dimension in this case. They then compare the empirical transition and the bound on the transition in terms of the local angular dimension between the two phases and find a good agreement with the theory.\n\nFinally they propose a notion of lottery subspaces inspired by the lottery tickets paper and compare their results.", "summary_of_the_review": "The article is clear and well written. It studies an interesting phenomenon with mix of empirical and theoretical results. The theoretical results are clear and interesting though they are direct consequence of Gordon's escape theorem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents new insights for training on random subspaces of low dimension, with several theoretical and experimental contributions. This is a paper that would be interesting to many people doing research in deep learning, both from the theoretical and practical side.
Decision: Accept (Poster)