The Early Phase of Neural Network Training
Authors: Jonathan Frankle
David J. Schwab
Ari S. Morcos
Venue:  (2020)
Keywords: empirical
learning dynamics
lottery tickets
critical periods
early

Abstract: Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state and its updates during these early iterations of training, and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are label-agnostic, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning.

Published as a conference paper at ICLR 2020
THE EARLY PHASE OF NEURAL NETWORK TRAINING
Jonathan Frankle' MIT CSAIL
David J. Schwab
Ari S. Morcos
CUNY ITS
Facebook AI Research
Facebook Al Research
ABSTRACT
Recent studies have shown that many important aspects of neural network learn- ing take place within the very earliest iterations or epochs of training. For exam- ple, sparse, trainable sub-networks emerge (Frankle et 2019), gradient descent moves into a small subspace (Gur-Ari et al. 2018). and the network undergoes a critical period (Achille et al., 2019). Here we examine the changes that deep neural networks undergo during this early phase of training. We perform exten- sive measurements of the network state during these early iterations of training and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only few hundred iterations. Despite this behavior, pre-training with blurred inputs or an auxiliary self-supervised task can approx- imate the changes in supervised networks, suggesting that these changes are not inherently deeendde,,,,en,, though labels significantly accelerate this process. To- gether, these results help to elucidate the network changes occurring during this pivotal initial period of learning.
INTRODUCTION
Over the past decade, methods for successfully training big, deep neural networks have revolution- ized machine learning. Yet surprisingly, the underlying reasons for the success of these approaches remain poorly understood, despite remarkable empirical performance (Santurkar et al.) 2018 /Zhang al.) 2017). A large body of work has focused on understanding what happens during the later stages of training (Neyshabur et al. 2019: Yaida 2019 Chaudhuri & Soatto, 2017 Wei & Schwab, 2019). while the initial phase has been less explored. However, number distinct observations indicate that significant and consequential changes are occurring during the most early stage of train- ing. These include the presence of critical periods during training (Achille et al. 2019), the dramatic reshaping of the local loss landscape (Sagun et al. 2017 Gur-Ari et al. 2018), and the necessity of rewinding in the context of the lottery ticket hypothesis (Frankle et al. 2019). Here we perform a thorough investigation of the state of the network in this carly stage.
To provide unified framework for understanding the changes the network undergoes during the early phase, we employ the methodology of iterative magnitude pruning with rewinding (IMP), as detailed below, throughout the bulk of this work (Frankle & Carbin 2019 Frankle et al. 2019). The initial lottery ticket hypothesis, which was validated on comparatively small networks, proposed that small, sparse sub-networks found via pruning of converged larger models could be trained to high performance provided they were initialized with the same values used in the training of the unpruned model (Frankle & Carbin, 2019). However, follow-up work found that rewinding the weights to their values some iteration early in the training of the unpruned model, rather than to their initial values, was necessary to achieve good performance on deeper networks such as ResNets (Frankle je al., 2019). This observation suggests that the changes in the network during this initial phase are vital for the success of the training of small, sparse sub-networks. As result, this paradigm provides a simple and quantitative scheme for measuring the importance of the weights at various points early in training within an actionable and causal framework.
Work done while intern Facchook AI Research.

---

Published as a conference paper at ICLR 2020
We make the following contributions, all evaluated across three different network architectures:
1. We provide an in-depth overview of various statistics summarizing learning over the early part of training.
2. We evaluate the impact of perturbing the state of the network in various ways during the early phase of training, finding that:
counter to observations in smaller networks (Zhou et al. 2019), deeper networks are not robust to reinitializion with random weights, but maintained signs
iii) the distribution of weights after the early phase of training is already highly non-i.i.d.,
as permuting them dramatically harms performance, even when signs are maintained
iiii) both of the above perturbations can roughly be approximated by simply adding noise to the network weights, though this effect is stronger for (ii) than (i)
3. We measure the data data-dependence of the early phase of training, finding that pre-training using only p(z) can approximate the changes that occur in the early phase of training, though pre-training must last for far longer (~32x longer) and not be fed misleading labels.
2 KNOWN PHENOMENA IN THE EARLY PHASE OF TRAINING
Lottery ticket rewinding: The original lot- tery ticket paper (Frankle & Carbin 2019) re- wound weights to initialization, I.c., h 0, during IMP. Follow up work on larger models demonstrated that it is necessary rewind to later point during training for IMP to succeed, i.e., k <<<<>> T. where T is total training itera- tions (Frankle et al. 2019). Notably, the benefit of rewinding to a later point in training satu- rates quickly. roughly between 500 and 2000 iterations for ResNet-20 on CIFAR-10 (Figure
Resns-30 (CIFAR-10)
-90
Evei Avcunayy
Rewinl tu
Rowind in
36
Rowinl to 250
Rewind tn
Rewind 1000
Rewind to 2000
100.0
66.0
41.0
26.2
16.5
10.7
6.9
4.1
Porcent of Weight Remoining
D. This timescale is strikingly similar to the changes in the Hessian described below.
Figure I: Accuracy of IMP when rewinding to various iterations of the early phase for ResNet- 20 sub-networks as function of sparsity level.
The shape of the loss landscape around the network state also appears to change rapidly during the early phase of training (Sagun et al., 2017: Gur-Ari et al. 2018). At initialization, the Hessian of the loss contains a number of large positive and negative eigenvalues. However, very rapidly the curvature is reshaped in few marked ways: few large eigenvalues emerge, the bulk eigenvalues are close to zero, and the negative eigenvalues become very small. Moreover, once the Hessian spectrum has reshaped, gradient descent appears to occur largely within the top subspace of the Hessian (Gur-Ari et al., 2018). These results have been largely confirmed in large scale studies (Ghorbani et 2019). but note they depend to some extent on architecture and (absence of) batch normalization dlffe & Szegedy 2015). A notable exception to this consistency the presence of substantial L, energy of negative eigenvalues for models trained on ImageNet.
Hessian eigenspectrum:
Critical periods in deep learning: Achille et al. (2019) found that perturbing the training process by providing corrupted data early on training can result in irrevocable damage to the final perfor- mance of the network. Note that the timescales over which the authors find critical period extend well beyond those we study here. However, architecture, learning rate schedule, and regularization all modifyy the timing of the critical period, and follow-up work found that critical periods were also present for regularization, in particular weight decay and data augmentation (Golatkar et al. 2019).
3 PRELIMINARIES AND METHODOLOGY
Networks: Throughout this paper, we study five standard convolutional neural networks for CIFAR- 10. These include the ResNet-20 and ResNet-56 architectures designed for CIFAR-10 (He ct al. 2015), the ResNet-18 18 architecture designed for ImageNet but commonly used on CIFAR-TO (He et al. 2015), the WRN-16-8 wide residual network (Zagoruyko & Komodakis 2016), and the
2

---

Published as a conference paper at ICLR 2020
Acrndieed rww.rnannggeeeo Rrdeer
condertteonnngge suughyy ragittoe
Rapid motton in spag
Weight magritudes hormas Ineatty
Gridict ro regnitates minon nughtyccnent coneane
dign changes for
Profomance dereirotss but repil
legin, hight
foffur stablling 10% hightr
Mation in redggt Lipics slowa, buti st Irspid
hoffemmanue slonly aymolohh tereft of rowiiigg souriees
haffrmass say
ieuching evalsenisey
1-10
10-500
500-2000
2000+
Training iterations
Figure 2: Rough timeline of the early phase of training for ResNet-20 on CIFAR-10.
VGG-13 network (Simonyan & Zisserman (011)) as adapted by Liu et al. (2019)). Throughout the main body of the paper, we show ResNet-20; in Appendix [ we present the same experiments for the other networks. Unless otherwise stated, results were qualitatively similar across all three net- works. All experiments in this paper display the mean and standard deviation across five replicates with different random seeds. See Appendix A for further model details.
Iterative magnitude pruning with rewinding: In order to test the effect of various hypotheses about the state of sparse networks early in training, we use the Iterative Magnitude Pruning with rewinding (IMP) procedure of Frankle al. (2019) extract sub-networks from various points in training that could have learned on their own. The procedure involves training a network to com- pletion, pruning the 20% of weights with the lowest magnitudes globally throughout the network, and rewinding the remaining weights to their values from an earlier iteration & during the initial, pre-pruning training run. This process is iterated produce networks with high sparsity levels. As demonstrated in Frankle et al. (2019) IMP with rewinding leads to sparse sub-networks which can train high performance even at high sparsity levels > 90%.
Figure T shows the results of the IMP with rewinding procedure, showing the accuracy of ResNet- 20 at increasing sparsity when performing this procedure for several rewinding values of k. For k 2 500, sub-networks can match the performance of the original network with 16.8% of weights remaining. For 2000, essentially no further improvement observed (not shown).
4 THE STATE OF THE NETWORK EARLY IN TRAINING
Many of the aforementioned papers refer to various points in the "early" part of training. In this section, we descriptively chart the state of ResNet-20 during the earliest phase of training to provide context for this related work and subsequent experiments. We specifically focus the first 4,000 iterations (10 epochs). See Figure A3 for the characterization of additional networks. We include summary of these results for ResNet-20 timeline including results from several previous papers for ResNet- 18 in Figure Al
timeline in Figure and include broader
As shown in Figure during the earliest ten iterations, the network undergoes substantial change. experiences large gradients that correspond to rapid increase in distance from the initialization and a large number of sign changes of the weights. After these initial iterations, gradient magnitudes drop and the rate of change in each of the aforementioned quantities gradually slows through the remainder of the period we observe. Interestingly, gradient magnitudes reach minimum after the first 200 iterations and subsequently increase to a stable level by iteration 500. Evaluation accuracy, improves rapidly, reaching 55% by the end of the first epoch (400 iterations), more than halfway to the final 91.5%. By 2000 iterations, accuracy approaches 80%.
During the first 4000 iterations of training, we observe three sub-phases. the first phase, lasting only the initial few iterations, gradient magnitudes are very large and, consequently, the network changes rapidly. In the second phase, lasting about 500 iterations, performance quickly improves, weight magnitudes quickly increase, sign differences from initialization quickly increase, and gra- dient magnitudes reach a minimum before settling at a stable level. Finally, in the third phase, all of these quantities continue change in the same direction, but begin to decelerate.
3

---

Published as a conference paper at ICLR 2020
Dufinnace
1.54
1000
1001
5000
9000
1006
2005
300
900
1000
200
Thonsu Lamdice
Tranddy
Tiikkg Iredes
13.0aa
Croia Studiaityy
2.2
13:
0.6
0.24
+000 1000
2003
Figure 3: Basic telemetry about the state ResNet-20 during the first 4000 iterations (10 epochs). Top row: evaluation accuracy/loss; average weight magnitude; percentage of weights that change sign from initialization; the values of ten randomly-selecte tude; I.2 distance of weights from their initial values and final values the end of training; cosine similarity of weights from their initial values and final values at the end of training.
weights. Bottom row: gradient magni-
92
Evll Accunay
83
2nth signs: 0
Iur:: 0 Biginn 0
sigre:
siges: 2000
66
random init sigas: 500
mindom init sign: 2000
sigus;
Osigns
64
ast:: 500 aign:: 0
inl: 2000 signs:
64.0
26.2
16.8
100.0
64.0
41.0
26.2
10.7
4.4
Weights Remairing
Weights Remsining
Figure 4: Performance of an IMP-derived sub-network of ResNet-20 on CIFAR-IO initialized the
signs at iteration 0 or k and the magnitudes at iteration O or k. Left: k = 500. Right: k 2000.
PERTURBING NEURAL NETWORKS EARLY IN TRAINING
Figure shows that the changes in the network weights over the first 500 iterations of training are essential to enable high performance at high sparsity levels. What features of this weight transfor- mation are necessary to recover increased performance? Can they be summarized by maintaining the weight signs, but discarding their magnitudes implied by /hou al. (2019 Can they be represented distributionally? In this section, we evaluate these questions by perturbing the early state of the network various ways. Concretely, we either add noise or shuflle the weights of IMP sub-networks of ResNet-20 across different network sub-compenents and examine the effect on the network's 's ability to learn thereafter. The sub-networks derived by IMP with rewinding make it possible to understand the causal impact of perturbations on sub-networks that are as capable as the full networks more visibly decline in performance when improperly configured. comparisons between the experiments in Section and provide a common frame of reference, we measure the effective standard deviation of each perturbation, i.c. stddev(w, (Wpeeturerrrr Worig).
enable
5.1 ARE SIGNS ALL YOU NEED?
Zhou et al. (2019) show that, for set of small convolutional networks, signs alone are sufficient to capture the state of lottery ticket sub-networks. However, it is unclear whether signs are still sufficient for larger networks early in training. In Figure 4 we investigate the impact of combining the magnitudes the weights from one time-point with the signs from another. We found that the signs iteration 500 paired with the magnitudes from initialization (red line) from separate random initialization (green line) were insufficient to maintain the performance reached by using both signs and magnitudes from iteration 500 (orange line), and performance drops to that of using
4

---

Published as a conference paper at ICLR 2020
Det ActenNy
84
Redded
Brvind
Resind 5Q0
Rovina 2000
Sauffl Giubally
Shuffo-Gibsaly Clibally
Snuffi Layer
Shaffic Lagen
Snoffl
Shattls libmn
1900
6..0
41.0
2622
16.8
10.7
6.
102.0
64.0
4.0
262
U5A
10.7
6.9
Parcat of Walght Memarrz
Percat Weight Rerracing
Figure 5: Performance of an IMP-derived ResNet-20 sub-network on CIFAR-10 initialized with the weights at iteration permuted within various structural elements. Left: k 500. Right: & 2000.
11r Actecc (50
Rroted toll
Rrotinl tod
Resind 500
Reviial 2000
Sxiffl Glabilyy 500
Shufflo Clibvily. Siga- 2000
Smatfle Layen, Skgze
Shattlc Lagen, Sgga
Figure 6: The effect of training an IMP-derived sub-netword of ResNet-20 on CIFAR-10 initialized with the weights at iteration & as shuffled within various structural elements where shuffling only occurs between weights with the same sign. Left: k = 500. Right: k 2000.
both magnitudes and signs from initialization (blue line). However, while using the magnitudes from iteration 500 and the signs from initialization, performance is still substantially better than initialization signs and magnitudes. In addition, the overall perturbation to the network by using the magnitudes at iteration 500 and signs from initialization (mean: 0.0, stddev: 0,033) is smaller than by using the signs at iteration 500 and the magnitudes from initialization (0.0 + 0.042, mean + std). the first 500 iterations of training
These results suggest that the change in weight magnitudes are substantially more important than the change in the signs for enabling subsequent training.
By iteration 2000, however, pairing the iteration 2000 signs with magnitudes from initialization (red line) reaches similar performance to using the signs from initialization and the magnitudes from iteration 2000 (purple line) though not as high performance using both from iteration 2000. This result suggests that network signs undergo important changes between iterations 500 and 2000, as only 9% of signs change during this period. Our results also suggest that counter the observations of Zhou et al. (2019) in shallow networks, signs are not sufficient in deeper networks.
5.2 ARE WRIGHT DISTRIBUTIONS I.I.D.?
Can the changes in weights over the first & iterations be approximated distributionally? To mea- sure this, we permuted the weights at iteration within various structural sub-components of the network (globally, within layers, and within convolutional filters). If networks robust these permutations, it would suggest that the weights in such uub-compenents might be approximated and sampled from. As Figure 5)suss, however, weights globally (green line) or within layers (red line), and drops substantially to no better than that of the original initialization (blue line) both 500 and 2000 iterations Shufffing within filters (purple line) performs slightly better, but results in smaller overall perturbation (0.0 + 0,092 for 500) than shuffling layerwise (0.0 or globally (0.0+10.14), suggesting that this change in perturbation strength may simply account for the difference.
found that performance was not robust to shuffling
We also considered shuffling within the incoming and outgoing weights for each neuron, but performance was equivalent shuffling within layers. We elided these lines for readahility.
5

---

Published as a conference paper at ICLR 2020
90
DigI Accunicy
Rewied lo Rewind
Neevind sn
250
300
300
Weight Remening
Woghht Ronsaring
Figure 7: The effect of training an IMP-derived sub-network of ResNet-20 CIFAR- initialized with the weights at iteration k and Gaussian noise of na, where or is the standard deviation of the initialization distribution for each layer. Left: k 500. Right: k 2000.
Are the signs from the rewinding iteration, k, sufficient recover the damage caused by permuta- tion? In Figurel6) we also consider shuffling only amongst weights that have the same sign. Doing so substantially improves the performance of the filter-wise shuffle; however, also reduces the extent of the overall perturbation (0.0 + 0.049 for k 500). It also improves the performance of shuffling within layers slightly for & 500 and substantially for = 2000. We attribute the behavior for k = 2000 to the signs just in Figure when the magnitudes are similar in value (Figure 4 red line) or distribution (Figure and green lines), using the signs improves performance. Reverting back to the initial signs while shuffling magnitudes within layers (brown line), however, damages the network too severely (0.0 + 0.087 for = 500) to yield any performance improvement over random noise. These results suggest that, while the signs from initialization
not sufficient for high performance at high sparsity as shown in Section [5.] the signs from the rewinding iteration are sufficient recover the damage caused by permutation, least to some extent.
5.3 IT ALL JUST NOISE?
Some of our previous results suggested that the impact of signs and permutations may simply reduce to adding noise to the weights. To evaluate this hypothesis, we next study the effect of simply adding Gaussian noise to the network weights iteration k. To add noise appropriately for layers with different scales, the standard deviation the noise added for each layer was normalized to a multiple of the standard deviation of the initialization distribution for that layer. In Figure 7 see that for iteration & = 500, sub-networks can tolerate 0.5o to 1a of noise before performance degrades back to that of the original initialization higher levels of noise. For iteration k 2000, networks are surprisingly robust to noise up to la, and even 2a exhibits nontrivial performance.
In Figure we plot the performance of each network at fixed sparsity level function of the effective standard deviation of the noise imposed by each of the aforementioned perturbations. We find that the standard deviation of the effective noise explained fairly well the resultant performance 0.008; k 2000: -0.726, p = 0.003). As expected, perturbations
(k = 500: r = -0.67 -0.672, that preserved the performance of the network generally resulted in smaller changes to the state of the network at iteration k. Interestingly, experiments that mixed signs and magnitudes from different points training (green points) aligned least well with this pattern: the standard deviation of the perturbation is roughly similar among all of these experiments, but the accuracy of the resulting networks changes substantially. This result suggests that although the standard deviation of the noise is certainly indicative of lower accuracy, there are still specific perturbations that, while small in overall magnitude, can have large effect on the network's 's ability to learn, suggesting that the observed perturbation effects not, in fact, just a consequence of noise.
THE DATA SOTTA-DEENNDNCCE OF NEURAL NETWORKS EARLY IN TRAINING
Section 5 suggests that the change in network behavior by iteration k is not due to easily- ascertainable, distributional properties of the network weights and signs. Rather, appears that training is required to reach these network states. It is unclear, however, the extent to which various aspects of the data distribution are necessary. Mainly, is the change in weights during the early phase of training dependent on p(x) or p(ylar)? Here, attempt to answer this question by mea-

---

Published as a conference paper at ICLR 2020
8% Spundiy
Spaniit
Rewind
Rewind 2000
Fihers, 200000 1.0.
Garasian 0.5
01.0
Signs
20.5
50.
500 signi O
Bed Aeeas
Ganoian 1.0
30.0
@G Giazsii 2.0
@Stufttle Layer, Signs 2000 eshoftle Filters ShufTe Glubally, Signs
815
inio O sigre: 2000
89-8
Shuffle Filtrs
Shofil: Layers, Signs 500
Stuftle Globally, Signs at 200
* Sauftle Globalyy Shuttle
18.5
suit signs: 300
random iit signs Roall Glubalic Shaffli Laper
sundom inii sign: 500 e
3.0
800
a
0.30 01s
0.50
000 6.6
0.10
B. 0,00 635
0.50
St6dev of Eifeetieee Rowrdd
oi Netic Rosted 200
Figure 8: The effective standard deviation of various perturbations as a function of mean evaluation accuracy (across 5 seeds) at sparsity 26. 2%. The mean of each perturbation was approximately 0. Left: 500, r 0.672, p 0.008; Right: 2000. 0. 0.726, p 0.003.
Self-Saperiied
Rewind i
Rewind to
85
Bod NNCURD
58
Resind to 2000
Rewisd 2000
80
Primnic 2 Eppcct
Pretrnn 2 Eppacto
Uppccs
Sppachs
Pretrie Spocss
Protrin 10 Epocls
75
Pretnic Epocts
Pretran 40 Epoclo
100.0
64.0
41.0
262
16.8
30.7
6.9
4.4
102.0
64.0
4.00
262
16.5
10.7
6.9
4.4
Peamm of Waight Romaiting
Pereatt of Weighte Remaniing
4 Blaning
4a Blaninng Self Suparvved Retatios
02
Reeind to
Kewind
88
Resind to 500
End Nocuaa
Rewind 1001
Resind
Rewasl 2000
85
Epocia
96
Pretzin Epocit
Primir S Uljccs
Prarinn Sppachs
Pretrie Spocls
Pretrin Epucto
Spocks
Pretzin Epocto
82
100.0
64.0
41.0
262
10.7
Peaxatt Weights Remsining
Weight Remaiiing
Figure 9: The effect of pre-training ResNet-20 on CIFAR-I0 with random labels, self-supervised rotation, 4x blurring, and 4x blurring and self superviee rotation.
suring the extent to which can re-create a favorable network state for sub-network training using restricted information from the training data and labels. In particular, we consider pre-training the network with techniques that ignore labels entirely (self-supervised rotation prediction, Section 6.22) provide misleading labels (training with random labels, Section 6.1.. or eliminate information from examples (blurring training examples Section 6.3).
We first train a randomly-inittalized,, unpruned network on CIFAR-10 on the pre-training task for set number of epochs. After pre-training, we train the network normally as if the pre-trained state were the original initialization. We then use the state of the network at the end of the pre-training phase as the "initialization" to find masks for IMP. Finally, we examine the performance of the IMP- pruned sub-networks
initialized using the state after pre- pre-training. This experiment determines the extent to which pre- training places the network in a state suitable for sub-network training as compared to using the state of the network at iteration & of training on the original task.

---

Published as a conference paper at ICLR 2020
6.1 RANDOM LABELS
To evaluate whether this phase of training is dependent on underlying structure in the data, we drew inspiration from Zhang et al. (2017) and pre-trained networks on data with randomized labels. This experiment tests whether the input distribution of the training data is sufficient to put the network in a position from which IMP with rewinding can find sparse, trainable sub-network despite the presence of incorrect (not just missing) labels. Figure | (upper left) shows that pre-training on random labels for up to 10 epochs provides no improvement above rewinding to iteration and that pre-training for longer begins to hurt accuracy. This result suggests that, though it is still possible that labels may not be required for learning, the presence incorrect labels is sufficient to prevent learning which approximates the early phase of training.
6.2 SELF-SUPERVISED ROTATION PREDICTION
What if we remove labels entirely? p(a)) sufficient to approximate the early phase of training? Historically, neural network training often involved two steps: a self-supervised pre-training phase followed by supervised phase the target task (Erhan et al. 2010). Here, consider one such self-supervised technique: rotation prediction (Gidaris et al. 2018). During the pre-training phase, the network is presented with training image that has randomly been rotated 90n degrees (where E {0, 1,). The network must classify examples by the value of n. If self-supervised pre- training can approximate the early phase of training. it would suggest that p(z) is sufficient on its own. Indeed, as shown in Figure (aguuree(uun (upper right), this pre-training regime leads to well-trainable sub- networks, though networks must be trained for many more epochs compared to supervised training (40 compared to 1.25, factor of 32x). This result suggests that the labels for the ultimate task themselves are not necessary to put the network in such a state (although explicitly misleading labels detrimental). We emphasize that the duration of the pre-training phase required is order of magnitude larger than the original rewinding iteration, however, suggesting that labels add important information which accelerates the learning process.
6.3 BLURRING TRAINING EXAMPLES
To probe the importance of p(x) for the early phase of training, we study the extent which the training input distribution is necessary. Namely, we pretrain using blurred training inputs with the correct labels. Following Achille et al. (2019). we blur training inputs by downsampling by 4x and then upsampling back to the full size. Tigure[(bottonn succeeds: after 40 epochs of pre-training, IMP with rewinding can find sub-networks that are similar in performance to those found after training on the original task for 500 iterations (1.25 epochs).
left) shows that this pre-training method
Due to the success of the the rotation and blurring pre-training tasks, we explored the effect of combining these pre-training techniques. Doing so tests the extent to which we can discard both the training labels and some information from the training inputs. Figure 9 (bottom right) shows that amount of pre-training we considered makes
doing so provides the network too little information: it possible for IMP with rewinding to find sub-networks that perform tangibly better than rewinding to iteration O. Interestingly however, shown in Appendix [ trainable sub-netwe netwoes found for VGG-13 with this pre-training regime, suggesting that different network architectures have different sensitivities the deprivation of labels and input content.
6.4 SPARSE PRETRAINING
Since sparse sub-networks are often challenging to train from scratch without the proper initializa- tion (Han et al. 2013 Liu et al.) 2019: Frankle & Carbin 2019), does pre-training make it casier for sparse neural networks to learn? Doing so would serve as a rough form of curriculum learning (Bengio et al., 2009) for sparse neural networks. We experimented with training sparse sub-networks of 2esNet-22 (IMP sub-networks, randomly reinitialized sub-networks, and randomly pruned sub- networks) first on self-supervised rotation and then the main task, but found no benefit beyond rewinding to iteration 0 (Figure 10). Moreover, doing so when starting from a sub-network rewound to iteration 500 actually hurts final accuracy. This result suggests that while pre-training is sufficient to approximate the early phase of supervised training with an appropriately structured mask, it not sufficient to do with an inappropriate mask.
8

---

Published as a conference paper at ICLR 2020
Resast (CIFAR 10) Spane Pretrain
92-
Dvval Asciragy
86
86
Rewind 500
Pectrinn
Pretminn, Random Reinit
82
Pretmin, Random Roirit, Rundomize Misks
1000
64.0
41.0
26.2
16.8
10.7
6.9
Weights Remaining
Figure 10: The effect of pretraining sparse sub-networks of Resnet-20 (rewound to iteration 500) with 40 epochs of self-supervised rotation before training on CIFAR-10.
7 DISCUSSION
In this paper, we first performed extensive measurements of various statistics summarizing learning over the early part of training. Notably, we uncovered 3 sub-phases: in the very first iterations, gra- dient magnitudes
anomalously large and motion is rapid. Subsequently, gradients overshoot to smaller magnitudes before leveling off while performance increases rapidly. Then, learning slowly begins to decelerate. We then studied a suite of perturbations to the network state in the early phase finding that, counter to observations in smaller networks (Zhou et al. 2019), deeper networks are not robust reinitializing with random weights with maintained signs. We also found that the weight distribution after the early phase of training is highly non-independent. Finally, we measured the deta-dependence the early phase with the surprising result that pre-training on a self-supervised task yields equivalent performance to late rewinding with IMP.
These results have significant implications for the lottery ticket hypothesis. The seeming necessity of late rewinding calls into question certain interpretations of lottery tickets as well as the ability to identify sub-networks at initialization. Our observation that weights are highly non-independent at the rewinding point suggests that the weights at this point cannot be easily approximated, making approaches which attempt to "jump" directly to the rewinding point unlikely to succeed. However, our result that labels are not necessary approximate the rewinding point suggests that the learning during this phase does not require task-specific information, suggesting that rewinding may not be necessary if networks are pre-trained appropriately.
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. 2019.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48. ACM, 2009.
Pratik Chaudhuri and Stefano Soatto. Stochastic gradient descent performs variational inference, converges limit cycles for deep networks. 2017. URL wtt./tttpa://axiw.orrg/
11029
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11(Feb):625-660, 2010.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:// openreview net. 2ttp:/ooruumm1xrr--333333
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: 1903301611 2019.

---

Published as a conference paper at ICLR 2020
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Proceedings of the 36th International Conference on Machine Learning. volume 97, pp. 2232-2244 2019. URL http://proceeiings mlrr . press/
v97/ghorbani 9b. html
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL
net Lnet/ooruunia--SvAAA.....
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep net- works: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. 2019.
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arXiv preprint vrXivv:812.04777 2018.
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. In Advances neural information processing systems, pp. 1135-1143, 2015.
K He, X Zhang, S Ren, and J Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recogntion (CVPR), volume 5, pp. 6, 2015.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference Inter- national Conference on Machine Learning Volume 37, ICML' pp. 448-456.. JMLR.org, 2015.
URL http::////. acm. wttgg/iiaaiion..oo cfm?ic 3045118. 3045167
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019. https:
//openreview...
wiw.mmmm-ddddnnnS5555 IJLnB3C5Ym
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of parammtrrttiioioonnn in generalization of neural networks. 2019.
Levent Sagun, Utku Evci, Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of werrppaameetrized neural networks. 2017. URL /tttps://arxi......t///////
abs/
7706.04454
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances in neural information processing systems, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. 2015.
Mingwei Wei and David Schwab. How noise during training affects the hessian spectrum in over- parameterized neural networks. 2019.
Sho Yaida. Fluctuation-dissipatii relations for stochastic gradient descent. 2019.
arXiv preprint Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXin:1605.07716. 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. 2019.
10

---

Published as a conference paper at ICLR 2020
A MODEL DETAILS
Network
Epochs
Batch Size
Learning Rate
Parameters
Eval Accuracy
ResNet-20
160
128
0.1 (Mom 0.9)
272K
91.5 + 0.2%
ResNer-55
160
128
0.1 (Mom 0.9)
856K
93.0 + 0.1%
ResNet-18
160
128
0.1 (Mom 0.9)
11.2M
86.8 + 0.3%
VGG-13
160
0.1 (Mom 0.9)
9.42M
93.5 = 0.1%
WRN-16-8
160
128
0.1 (Mom 0.9)
11.1M
94.8 = 0.1%
Table Al:
Summary of the networks we study in this paper. We present ResNet-20 in the main body of the paper and the remaining networks in Appendix B
Table A1 summarizes the networks. All networks follow the same training regime: we train with SGD for 160 epochs starting at learning rate 0.1 (momentum 0.9) and drop the learning rate by factor of epoch 80 and again epoch 120. Training includes weight decay with weight le- 4. Data is augmented with normalization, random flips, and random crops up to four pixels in any direction.
B EXPERIMENTS FOR OTHER NETWORKS
sige tlungee jou intlin 10
Papel motionin eeagrt
Weight mogettate decasss lnvelly
butoee dtstllinng 10% fightr al tonz 250
sugp by stant
Hisioon weight quer besi opii
hoffamann mursas repally
Fawerding sogero bnsomeufiicie
(100 Ieeaions Gavki 2010
Achill et 2019 do enuret 2010
Oréical
Raformanc slow expppoos
tavatons:
Eunadic of sownalin watanio
1-10
10-500
500-2000
2000+
Training iterations
Figure Al: Rough timeline of the early phase of training for ResNet-18 CIFAR-10, including results from previous papers.
11

---

Published as a conference paper at ICLR 2020
VGG-13 (CIFAR- 10)
Resnet-56 (CIFAR 10:
94
92
90
Eval Accuracy
Eval Accurazy
90
Rewind to 50
Rewind 100
Rewind to 100
86
Rewind to 250
Rewiid to 250
Rewind to 500
88
82
86
100.0
51.2
26.3
13.5
6.9
3.6
1.9
1.0
0.5
100.0
51.2
26.2
13.4
6.9
3.5
1.8
of Weights Remaining
Percent of Weights Remsining
Reanst-18 (CIFAR-10)
WRN-168 (CIFAR-10) 10)
95
Eval Accuracy (16)
87
Eva Accuracy (10)
94
Rewind to
Rewind to
Rewind to 10
85
Rewind to 100
93
Rewind to 25
92
83
Rewind to 500
Rewind
Rewind to 1000
91
Rewind to 250
Rewind to 2000
Rewind to 500
81
100.0
6.9
100.0
26.2
Parcent of Weights Remaining
Percent of Weights Remoining
Figure A2: The effect of IMP rewinding iteration the accuracy of sub-networks at various levels of sparsity. Accompanies Figure] [
12

---

Published as a conference paper at ICLR 2020
Bus Anemany
Leap Reip Medial
05
vid thed
dayy
100 110 coe
120 130
200220
530 150 PVD 30000000000
100 106 350
Buae End
Vogros
130 (80 5MA
193300
D1aen Ooobar Martokk
Draet
6:5
Scren-35 35- Dol
Suadd-h Ancage Nigge Mag
Swatch Wel
0421
0030
075
-642
0236
1000
250
300
Trabking Screis
0.5
20
Tbll
11l
136
2000
3030
ViD4 End Aoonng
ABc M.C. thet
V156 M.1. Jempg Pegge Megdhal
N0po Agg Dilarss
6.000
05
Tuetry Condic
490-1-10 Crudia Maptirr
A.5% 1S4 Clic Se ber
1.30
150 LA. CRN
180 130 50B 02 260 did
100 143200 Min
(06 150 250 /2001000
Figure A3: Basic telemetry about the state of all networks in Table Al during the first 4000 iterations of training. Accompanies Figure 3
13

---

Published as a conference paper at ICLR 2020
VGG (CIFAR-BE
VGG (CIFAR-
94
92
92
Eval Accuracy
Eval Accurazy
90
init: O signs:
90
init: O sigus::
Init: signs:
signs:
88
signs:
signs:
init: signs: 250
init: signs: 500
Init 250 signe:
init 500 signs:
86
86
100.0
51.2
26.3
13.5
6.9
3.6
1.9
1.0
0.5
100.0
51.2
26.3
13.5
6.9
3.5
1.9
1.0
0.5
Percent of Weights Kemaining
Percent of Weights Kemnining
Resnet-56 (CIFAR-10)
Resnet-56 (CIFAR-
94
92
90
Eval Accuracy
88
init: signs:
init: sigrs:
86
init: 500 signs: 500
86
inft: 2000 sigus: 2000
84
zandom init signs: 500
Eval
random init signs: 2000
82
init: signs:
82-
init: D signs: 2000
signs:
sigus:
100.0
51.2
26.2
13.4
6.9
3.5
100.0
51.2
26.2
13.4
6.9
3.5
1.8
Percent of Weights Remaining
Percent of Weights Remsining
Reans--18 (CIFAR-10)
Resnci-18 (CIFAR-10)
9.8
87
Eval Accurccy (16)
Eval Accurazy (16)
86
86
85
int: signs:
ini: signs:
signs
84
sigus:
83
random signs: 500
random 2000
82
init: signs: 500
init: signs: 2000
81
init 500 signs:
init 2000 signst
51.2
6.9
100.0
6.9
Percent of Weights Remaining
Percent of Weights Remnining
(CIFAR-10;
(CIFAR-10)
95
94
94
Exal A.cuncy
95
93
92
ind: O signs:
92
init: O signs:
91
init: 50 sigas: 50
91
Init: 250 sigas: 250
90
random signs: 50
random signs: 250
sign::
init: signs:
89
89
init: sigas:
init: signs:
88
88
100.0
51.2
26.2
13.
6.9
3.5
1.8
0.9
0.5
100.0
51.2
26.2
13.4
6.9
3.5
1.8
0.9
0.5
Weights Remaining
Weights Remaining
Figure A4: The effect of training an IMP-derived sub-network initialized the signs at iteration or l and the magnitudes at iteration 0 or k. Accompanies Figure A4
14

---

Published as a conference paper at ICLR 2020
VOG-15 (CIFAR-10) Rowiniing Demiio 250
VOG-I3 (CIFAR-10) Rewinslnn Demtien
Baal Acamcy (50
Bval Acamsy (50
00
90
Rewind to
Rewind to
Rewind
Shufte Globally
Shufte Globally
Shuffle Layeza
Shuflle Layera
Filten
86
Percon Weighro Rensiniing
Pescon Weights Rensaiting
Resan: 56 (CIFAR 13) Rewinding Deration 500
Reszet (CIFR 10) Rewaading Detation 2000
Bval (5)
Bval (50
Rewied to
Rewird to
44
Rewied
Rewied
Shuffls Globaly
Shaflle Globally
Shuffle Layara
Shuffle Layera
50
Filze
Peseon Weights Rensiniig
Pesuor Weights Rensiiting
Resac: 18 (CIFRR 10) Rewinding Deratio 500
Rasset (CIFAR 10) Rewadding Decatioo 2000
Dod (5)
86
Ev.l (50
85-
Rewied m
84
Rewird to
Revied
Shuffl: Clobaily
Shafle Clobally
Shuffle Layara
Shuffle Layera
Shuffla
Shafll
Pesoo Weights Rensiitin
Pesuor Weights Rensaiting
WRN (CIFAR-I0: Bostnding teeratonn 50
WEN-1644 (CIFAR 300 Restnding Toration 250
Erd Accitscy (5)
Erd (50)
91
Revind w
Rewied to
50
92
Shuffle Globally
Shufle Globally
Shofflo Layons
Shafffa Layors
Shuffis Filze
Shafll Filln
13.4 Weight Remsining
13.4 Weight Remainiig
Figure A5: The effect of training an IMP-derived sub-network initialized with the weights itera- tion & as shuffled within various structural elements. Accompanies Figure 5
15

---

Published as a conference paper at ICLR 2020
VOG
VOG
Dadd (55
92
D.d (8)
Revind w/O
Rewied to
Revild
Rewlld
Shufßle Clobally
Shuflle Clobally. Signs
Shofff:: Layens Signe al 250
Shaflfe. Layors Signe a: 500
Shuffle Filles Signs
Shaflle Fille, Signo
Shiffln Layens Signe
Shaflln Layins Signe
86
100.0
$1.2
26.3
13.5
3.6
1.9
1.0
0.5
100.0
51.2
26.3
13.5
5.6
1.9
1.0
0.5
Peruan Weights Retrainin
Pescar Weights Retraising
Resset (CIFAR
Resser (CIFAR
Bork (85
90
Rewind O
Rewied to
Rewina to 500
Rewind 2000
54
Shuffle Globally
Eir
Shuffle Globally. Sigas
Shuffls Layen, Signs 500
Shaflle Layors Signe a. 2000
Shatffo bilten Signs
Shalfle Rilces, Signs
Shuffli Layens Signe
Shaflln Layzz, Signe
100.0
$1.2
26.
4
5.9
3.5
100.0
51.2
26.2
13.4
6.9
Porcan Weights Romoining
Presar Weight Romaining
18 (CIFAR Rewitding Deraton 500
Resset (CFAR 10)
Rewinding Denation 2000
Dord Aconnyy (8)
86-
Dra. (8)
55
Revind Wo
Rewled to
Rewina to 500
84
Rewind to 2000
Shuffle Globally
Shuffte Globdly. Sigas
Shuflls Layen, Signs 500
Shaflle Layans, Signc a: 2000
Sheffln fillon, Signs at
Shefffn Pilker, Signs
Layens Signe
Shiflle Layaz, Signe
100.0
51.2
252
13.4
6.9
3.5
0.5
100.0
51.2
252
13.4
69
0.
Percen Weights Romoining
Presen Weight Remaining
WRN-16-4 (CIPAR- Rowinding Itention
WRN (CEFAR Rewinding Toration
Rewied to
Rewlle to
23
Revind 50
Rewind 2.0
Globally
Drral
Cleebilly Signs
Shuffli Layens Signs 50
Shaffle Layers, Signs a: 250
Shoffin Fille, Signe 50
Shaffle Filon, Signs
Shuflls Layens Signe
Shaflle Layez, Signe
100.11
26.
100
26.2
Pureen Weight Romoining
Presan Weight Remaining
Figure A6: The effect of training an IMP-derived sub-network initialized with the weights at iter- ation shufiled within various structural elements where shuffling only occurs between weights with the same sign. Accompanies Figure
16

---

Published as a conference paper at ICLR 2020
(CEAAR Rewinding Dension
Rewinding Denstoo
Dadd (55
92
D.d (8)
Revind w
Rewied to
Rewiod
Rewied
1.00
1.0g
86
100.0
51.2
25.3
13.5
6.9
3.5
1.9
1.0
0.5
100.0
51.2
26.3
13.5
6.9
3.6
1.9
1.0
0.5
Peruan Weights Retraining
Pescan Weights Rerraising
56 (CIFAR Rewizding
Resset (CIFAR 10) Rewiading Decation 2003
95-
Da-t (8)
Revind W
Rewied to
Rowied
Rewied 2000
80
1.00
1.00
88
97
100.0
64.0
41.0
25.2
15.5
10.7
6.9
100.0
64.0
41.0
25.2
15.5
10.7
6.0
Peraa Weights Remaiiing
Pescar Weights Retraiiing
18 (CIFAR Rewieding 500
Resset (CFFRR 10) Rewinding Decation 2000
Revind wo
Rewied to
44
Revind to 500
94
Rewind 2000
Pord
Dra.
1.00
1.00
100.0
51.2
252
13.4
6.9
3.5
1.5
0.5
100.0
51.2
252
13.4
6.9
1.5
0.
Prrcen Weights Remoiiing
Presen Weight Remaiiiin
WRN-16-4 (CIPAR 300 Rowinding
WRN-1666 Rewinding Toration
Dr.l (5)
Dor Aroray
23
Rewied to
Rewlee to
Rewind to 50
Rewind 25:
33
1.00
1.00
100.0
51.2
25-2
13.4
5.9
3.5
0.9
100.0
51.2
26.2
13.4
1.3
0.9
Teroem Weight Romoiiing
Presa Weight Remaining
Figure A7: The effect of training an IMP-derived sub-network initialized with the weights at itera- tion & and Gaussian noise of no, where the standard deviation the initialization distribution for each layer. Accompanies Figure
17

---

Published as a conference paper at ICLR 2020
VCr-TT Rewwtdd to 250- 0LS Sponity
VOG-3 Bowind Sparsie
Aowiad
52.
Revidd 500
Signest
Cuasviz 0.5
5-
Gunder U.S.
Sigie
103
293
Gaaminm 1.0
Gaaminm 1.M
fral Scn dgne
inly 9 jige 250
Shartta Layers Signt 290
519
T00: alrrc 500
signs
Shafile Fitess
$45-
Shutilic Filis
500
Gisbally Signn
Stadfi: Gluball
Cormito 2.0
Slatlle Cla bull Signs
150
Stuffe Layerk
mandken
Stufflc Guiilly
300
Gazaman
Cuassia
0.100
0.25
500
Lio
0.12 aie 6.5s
Sulder Mifcch Neaa 2r
Stidor Eilicii Noka
Rosded S8
Roast- 56 Revind to 502 1638 Spasity
Spenityy
90.15
RRokidd to 550
@ic Fltvon Signs 2000 2000 -iana:0
Hewan
00.30
Stuffic Filczs, Signr 500
110
*ui 500 signs. 0
Layen, Signo
03.75
91.00
Shufll Shulll-Ge Sigts at 2000
Guasiis (itullle Filto
100
515
#shattle togers
Sauttle Liaballyy
Eirbally Porertl Lagors
sizrs
lsit digs 2000 medom txit siges:
SSuuffi Gloaals
Granvizz 3.
#Conndn
ods old oid 6i5 $20
060 tis Goa bis 02: a2s
Sulee tfttiieeer Brwiddddd D
Rosect-B- Reviaal to 500 3.5% Spuzddy
Rewet- 18 Bevind 2000 3.5% Spansiy
Rextted
eQuasia
#Reind 2000
Qquallle Filen, Signa 500
175
Shafll Glenalyy Sign 300
Shaflll Layen, Sigza at Canadan Pagdenn ins aggs 300
GGnnda 2D
1522: D sigae Sqg Pizk, 500 sigos
@Grnsin 2.0
Skaffle Cioelly
Grasika 3
einer 24031 agte O
165
isk.
signs: 2000
2000 Guaskka
eSuillle Givelly, Sig d 2000 #Stuffic Layas
astreti
Slauli Fillos
Stafll Clvinlle
0.00
062
0/4
oLe
612
814
0.5
632
GMM
Eds
ais
0.4
Litectrs Hote Wr
Ulfecte Melic Roncadoo
WRN-15-8: Besind 50- 5.% Spaaity
WRN US S Rewiad 250 6.95 Spunkk
44.5
dgts
Filen, Sigze 50
Bewand
@Gazniza 0.5
941
443
lnt: 0 signs:
exir slgic 290
942
*Cressia 1.0
540
918
agen 250
Gansiess
Grasinn 2.0
rendeer Alpnn 10
@ctertl Gilddls Rign Eratt Gintaly
90
92.
Stad
EdC
Lie
sis
0.1
B12
660
960
6.4
6.0
012
Figure A8: The effective standard deviation of each of the perturbations studied in Section 5 as function of mean evaluation accuracy (across five seeds). Accompanies Figure g
18

---

Published as a conference paper at ICLR 2020
VGG 13 (CIFAR 10) Randem Labels
Restet 56 (CIPAR 10) Random Labels
80
Dral Accucccyy
Rewited 250
60
Rewind 500
Rewind
Evail
lipochs Pectmin Epochs
Pretmin lipochs
40
Pectmia Epochs
10 Epochs
to Epoccts
Pestralo 40 Epocis
Pretrála 40 Epocht
51.2
Percest of Weights Remaining
Perceat of Weights Remaining
Resnc-11 (CIFAR-B) 10) Random: Labels
(CI-A Random Labels
Dral Aconnzy
86
Rewind 300
Etai Acomncy
92
Rewind
Rewind 250
Rewind 2000
Rewiad 500
86
Epochs
90
lipochs
Protrain Epochs
Pretrali Epochs
Epochs
Epochs
Pestman 40 Epochs
Pretmin 40 Epochs
85
26.2
13.4
100.0
$1.2
26.2
13.4
Percent of Weights Remaiinigg
Percent of Weighes Remaiinig
Figure A9: The effect of pre-training CIFAR-10 with random labels. Accompanies Figure
Self-Superviood
Self-Sipervised
92
92
Rewied to
Dral Asturny
90
Rewitd to
500
90
Rewind 500
86
Rewind 2000
Dral
Pretmin Epochs
Pretmin Epochs
88
Epuchs
Epochs
Pretnin 10 Epochs
R2
Pretnin 1o Epochs
Upochs
Upochs
86
100.0
51.2
26.3
13.5
6.9
3.5
1.9
1.0
0.5
100.0
51.2
26.2
13.4
6.9
3.5
Percent Weights Remaining
Percent
Remzining
Resne-188 (CIFAR-10) Sopl-Saperviooo Rotstion
WRN-164 (CIFAR-10)- Self-Superviee Rotation
86
Rewind to
94
Row ind to
Dral Accuncy
Rewind 500
Rewind 250
93
500
Pretnin Fpochs
Dul
92
Protnin Epochs
Epochs
Epochs
82
Protnin 10 Epoche
91
Protmin 10 Epocch
Pretmin 40 Epochs
Preimin Epochs
100.0
51.2
26.2
13.4
6.9
3.5
1.8
0.9
0.5
100.0
51.2
26.2
13.4
6.9
3.5
Weights Remaining
Weights Remsining
Figure A10: The effect of pre-training CIFAR-10 with self-supervised rotation. Accompanies Figure
9
19

---

Published as a conference paper at ICLR 2020
VGG 13 (CIFAR 10) da Bluming
Resset 56 (CIFAR 10) 4x Blurting
94-
90
Dbll Accurity
Rewind 250
88
Rewind 500
90
86
Pectmin Epochs
Dal
Protnin Epochs
88
Pretmin Epochs
Pretmin Epochs
Pretmin 10 Epochs
Pretmin 10 Epochs
Pectmin 40 Epochs
80
Pectnia 40 Epochs
1000
13.5
0.5
Percent Wrights Remaining
Percent Weights Remsining
Resnet 18 (CIFAR 10) -bx Dlurring
WRN 16 (CIPAR 10) 4x Bluming
86
Dval Accunay
Rewind 500
Dval Accimcy
Rewind 250
84
Pretmin lipochs
92
Pretmin lipochs
Peetnin Epochs
Protnén Epochs
Epochs
91
Opochs
Peotrain 40 Epochs
Peetrain 40 Epochs
13.4
Percent of Weights Remaining
Percent of Weights Remaining
Figure A11: The effect of pre-training CIFAR-10 with 4x blurring. Accompanies Figure
VGG-13 (CIFAR-1U) 10) da Blurring Self Superivsed Rotation
Resnet 56 (CEARR-00) 4x Blloringg
Self: Superiiseed Rotation
92-
92
Dral Accucuy
Rewind 250
Dval Accumiy
88
Rew ind 500
90
Rewind
Pretminn lipochs
Pretmin lipochs
Pectnin 5 Epochs
84
Pretnin Epochs
Epochs
Epochs
Peetrain 40 Epochs
80
Pretrann 40 Epochs
6.9
Percezt of Weights Remaining
Percent of W Weights Remaining
Resnet (CIFAR-10): 4x Blumiing Self- Superivedd Rotation
WRN- 16-8 (CIFAR- 10 4xx Blarring -Sll Sell--Spprrived Robsion
Dral Accuncy
500
Eval Accuncy
93
250
Rewind 2000
Rewind 500
84
Epochs
92
Epochs
Pictnin Epochs
Pictnia Epochhs
Pretmin Epoche
91
Pretmin Epochs
Pretmin 40 Upoccss
Pretmin 40 Upochs
26.2
19.4
13.4
Percent of Weights Remaining
Percent of Weights Remaining
Figure A12: The effect of pre-training CIFAR-10 with 4x blurring and self-supervise rotation. Accompanies Figure
20


Reviews:
[{"review_id": "SJlvCqBatr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at exploring the properties of neural network training during the early phase. By some studies on the lottery ticket hypothesis, something important happens during the early phase of training so rewinding the network should go to these early phases instead of the initial phase. So, what is important during training? The paper explores this problem from four aspects through empirical studies: \n1. By showing the various statistics through different training iterations, especially the gradient magnitude, the training phase is divided into three parts, and each part has different behaviors.\n2. The paper explores what is more important for the early phase of training: signs of the weights or magnitude of the weights.\n3. The paper explores what is more important for the early phase of training if we redistribute the weights, signs or magnitudes.\n4. The paper explores how training depends on data. Giving weak information such as self-supervised information may work but giving wrong information such as random labels will hurt the performance.\n\nThis paper studies the properties of deep neural networks. Through a series of carefully designed experiments, the paper shows what is important for the weights (magnitude or signs), and what is important for the data (weak information or random information). It enables a deep understanding of neural networks and may motivate new neural network compression methods to be proposed. Generally, the paper is well-written, although some parts can be improved. I would vote for acceptance of the paper.\n\nSome questions/suggestions to make the paper clearer:\n1. It is better to have a table summarizing various results of the paper, to give readers an overall impression after going through so many detailed experimental results.\n2. The results of Fig. 4 and Fig.6 can be inconsistent. In Figure 4, it says that signs are less important than magnitudes. In Figure 6, it says that signs are more important than magnitudes if shuffling filters and keep signs. Any explanation on the inconsistency?\n3. There is no explanation of the “weight trace” in Figure 3.\n4. It is not clear what is the difference between “Init” and “Final” in “L2 Dist” and “Cosine Similarity” in Figure 3. \n\nFinally,  it is said in the “call for papers”, “Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages”. This paper is nearly nine pages, and higher standards should be applied. \n\n--------------------------------------\nI am satisfied with the rebuttal. Since the paper is now within the 8 pages limit, I would not apply a high standard and increase my score. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJx_-RnntS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\n\nThis paper is dedicated to examining the changes that networks undergo during the early phase of the network training. The author conducts extensive measurements of the network state and its updates during the early iterations of training. Based on the observations, they find that: i) deep network is not robust to reinitialization with random weights, but maintained signs; ii) the distribution is highly non-i.i.d after the early phase of network training. This is why permuting weight dramatically harms performance. iii) the changes in the supervised networks are label-agnostic. The author claims these results can play an important role in explaining the network changes in the initial critical period.\n\nStrength Bullets:\n\n1. The paper performs exhaustive experiments in the early phase of network training. And it has some interesting implications for lottery tickets. i.e. to some extent, then signs from the rewinding iteration are sufficient to recover the damage caused by permutation.\n2. I am very like the analysis of whether weight distributions are i.i.d.. The results in the paper are aligned with my intuition that the weights in the early stage are highly dependent and they share some similarities in the distribution level. And weight in different training stages supposes in a different distribution. Networks aren't robust to these permutations. They also show that the perturbations can be roughly be approximated by adding Gaussian noise to network weights.s\n3. The author offers detailed results to analysis the data-dependence of the early phase of network training. The experiment organization is complete and convincing.\n\nWeakness Bullets:\n\n1. Although the paper gives extensive measurements and observations about the early phase about network training, it doesn't provide useful and efficient guidance for the lottery tickets hypothesis. In other words, the observation is interesting but the novelty is arguable. How can we use these \"implications\" to find better tickets? I mean it should be efficient. It is not reasonable cost more the find the best weight magnitude point or sign point for a better initialization.\n2. [Minor] The legend in figure 9 is missing. I guess it may be the same as the upper left. But it is still confusing and hard to read.\n\nRecommendation:\n\nI think this paper provides plenty of insightful observations. I still hope the author can solve the above weakness bullets. This is a weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ggWiM15r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros\n+ This work provided a good summary of observations and network properties that worth studying during the early stage of network’s training.\n+ The authors conducted extensive and detailed experiments to study the statistics of weights and their gradients. Ablation studies also considered network’s accuracy under perturbation of weight signs, weight shuffling, and different standard deviations.\n+ The authors also verified the effectiveness of weak labels used in self-supervised learning.\n\nCons\n- The work itself did not propose any new network properties or any new metric to measure. Most experiments are designed for previous observations and mostly for verification purpose. I am concern about the core motivation of this work, like to identify or solve any new problems, in addition to experimentally verify the observations during network’s training.\n- The conclusion in this work is still very empirical: it remains uncertain whether in other vision tasks and with more complex networks (e.g. multi-branch network) these conclusions would hold.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies numerous ways in which the statistics of network weights evolve during network training.  Reviewers are not entirely sure what conclusions to make from these studies, and training dynamics can be strongly impacted by arbitrary choices made in the training process.  Despite these issues, the reviewers think the observed results are interesting enough to clear the bar for publication.
Decision: Accept (Poster)